Ranking in Google’s Free Organic Product Listings & Grids [2025]

The free Shopping listings currently take the form of

  • Organic Product Grids
  • Product knowledge panels
  • Image search results
  • Shopping tab

Reports to Check for Free Shopping Listings

GA4 – Acquisition – User Acquisition > organic SHopping (Keep in mind the recent non-PDP auto-tagging changes – might need customisation in GA4)

Search Console – > Performance Report > Add Filter > Merchant Listings
(There are reports for Search type: Web and Search Type: Image – just add the Search Appearance filter

Optimizing Your Product Feed

A well-optimized product feed is crucial for ranking in free Shopping listings. Focus on these key attributes:

  • Image link
  • Description
  • Brand
  • GTIN
  • MPN
  • Color
  • Size
  • Material
  • Title (highly influential for free listings and CTR)

Leveraging Merchant Features

To stand out in free shopping grids, optimise and complete the following merchant features:

  • Seller ratings (aim for 5-star reviews within the last 12 months)
  • Top quality score
  • Promotions (e.g., coupon codes)
  • Local free listings
  • “Buy Now” button
  • Payment options (PayPal, Google Pay)
  • Loyalty programs
  • Competitive pricing
  • Clear shipping and returns policies

Free Shopping Grids & Structured Data

Products with the best price tend to rank highest

Shipping policy & returns policy – can affect rankings significantly

In Search console there are warnings that show if u don’t have these set up in merchant center or in structured data.can ignore the warnings sometimes tho

Structured data – although we have seen google may unify product feeds and structured data in the near future

Influence for Free Shopping Grids & Listing – 

*Most influential* Website content →Product Feeds→ structured data. *Least influential*

Case Study: Improving Eligibility with Schema Changes

A recent case study demonstrated the impact of schema optimization:

  • Problem: Aggregate offer schema prevented merchant listing eligibility
  • Solution: Switched to ProductGroup schema
  • Result: Increased eligibility for product grids and boosted revenue

Additional Tips for Organic Google Shopping Success

  1. Optimize product titles with relevant keywords & attributes
  2. Use high-quality, unique product images – lifestyle usually best but A/B test
  3. Keep your product feed up-to-date and as accurate as possible
  4. Encourage customer reviews to improve seller ratings – might be wise to pick and choose if possible!
  5. Implement proper category mapping in your product feed
  6. Use custom labels to organize and group products effectively
  7. Optimize your landing pages – A/B test them
  8. Monitor and address any warnings in Google Search Console
  9. A/B test descriptions to improve click-through rates
  10. Regularly analyze competitor listings

Key Takeaways

  1. Use the tools – Search Console, Merchant Center, and GA4
  2. Sort out Search Console first, then monitor with GA4 and Merchant Center
  3. Implement an extensive set of product feed attributes
  4. Optimize product titles for maximum impact
  5. Highlight your unique selling propositions (price, returns, shipping, etc.) & strong selling points too
  6. Fix and address any warnings in Google Search Console
  7. Maintain consistency across your website, product feed, and structured data
  8. A/B test everything you can – if you have enough data etc!

eCommerce Product Page – SEO & UX Checklist

SEO Checklist:
Product Title Optimization
Include the main keyword naturally.
Keep titles concise but descriptive.
Meta Title and Meta Description
Write a compelling meta title (60-70 characters) with the main keyword.
Create a unique and engaging meta description (150-160 characters) with a call to action and relevant keywords.
URL Structure
Use clean and descriptive URLs with relevant keywords.
Avoid unnecessary parameters and ensure URLs are short.
Header Tags (H1, H2, H3, etc.)
Use one H1 tag for the product title.
Structure content with subheadings (H2, H3) for different sections.
High-Quality Product Descriptions
Write unique, engaging, and detailed product descriptions (300+ words).
Naturally incorporate relevant keywords.
Highlight features, benefits, and use cases.
Product Images and Alt Text
Use high-resolution, web-optimized images.
Add descriptive, keyword-rich alt text for images.
Include multiple angles or lifestyle images.
Structured Data Markup (Schema)
Implement product schema for price, availability, reviews, etc.
Use JSON-LD for structured data.
Internal Linking
Link to related products or categories.
Include “frequently bought together” and “related products” sections.
Mobile-Friendliness
Ensure the product page is fully responsive and optimized for mobile users.
Test on multiple devices.
Page Load Speed
Optimize images and use compression.
Minimize CSS, JavaScript, and HTML.
Use a content delivery network (CDN).
Canonical Tags
Implement canonical tags to prevent duplicate content issues.
Keyword Usage
Use keywords naturally in the title, description, URL, headings, and body text.
Avoid keyword stuffing.
User Reviews and Ratings
Display user-generated content like reviews and star ratings.
Use schema markup for review ratings.
Social Sharing Buttons
Provide social sharing options for easy promotion.
Breadcrumbs
Use breadcrumb navigation to enhance site structure and user experience.
UX Checklist:
Clear Call to Action (CTA)
Use prominent and persuasive “Add to Cart” or “Buy Now” buttons.
Ensure CTA buttons are highly visible.
Product Images
Include multiple images and zoom functionality.
Offer a gallery of images showing different angles and context.
Product Information
Display essential details (price, size, color, availability) prominently.
Offer accurate and up-to-date stock information.
Shipping Information
Provide clear and transparent shipping details, including delivery times and costs.
Product Variations
Offer a seamless selection process for colors, sizes, or other variations.
Use clear dropdown menus or buttons.
Customer Reviews and Ratings
Allow customers to leave reviews.
Include user-generated content for social proof.
Related Products
Display recommended, cross-sell, or upsell products.
Use “Customers Also Viewed” or “Frequently Bought Together” sections.
Return Policy Information
Clearly outline the return policy on the product page.
Trust Signals
Include security badges, accepted payment icons, and other trust symbols.
Show testimonials or guarantees if applicable.
Easy Navigation
Ensure intuitive breadcrumb trails.
Use filters and sorting options for larger catalogs.
User-Friendly Design
Use white space effectively to improve readability.
Ensure buttons and interactive elements are appropriately sized.
FAQ Section
Provide answers to common questions about the product.
Accessibility Compliance
Follow accessibility guidelines (e.g., alt text, ARIA labels).
Test for screen reader compatibility.
Wishlist or Save for Later Feature
Offer options to save the product for future reference.
Live Chat or Support Options
Provide real-time assistance through chat or customer support links.
Easy Checkout Process
Minimize steps to purchase.
Offer guest checkout and multiple payment options.
Stock Notifications
Include “Back in Stock” or “Low Stock” alerts.

SEO News & Tips – September 2024

Actionable Points:

  1. Impact Assessment:
    • Analyze Core Update Impacts: If your site was affected, investigate potential causes such as relevance adjustments, intent shifts, or quality issues.
    • Run Delta Reports: Isolate dropped queries and landing pages using delta reports to identify the reasons for traffic decline.**
  2. Content Strategy:
    • Reassess Content Strategy: Move away from generic content aimed at search engines (SEO-first) and focus on user-first content that aligns with intent and provides real value.
    • Avoid Overloading Pages: Do not overload pages with excessive information. Make them clear and relevant to user intent.
  3. Quality Over Quantity:
    • Holistic Quality Evaluation: Improve overall site quality, including UX, ads, presentation, and sources. It’s not just about content but the experience as a whole.
    • Long-Term Quality Improvements: Use a “kitchen sink” approach—continuous, holistic improvements to demonstrate significant quality enhancement over time.
  4. Utilize User Engagement Data:
    • Measure User Engagement: Focus on engagement metrics like click-through rates (CTR), time on page, and interactions (Navboost signals) to improve rankings.
  5. Consider the Intent Shifts:
    • Study Intent Shifts: Understand changes in user intent. Ensure your content targets the appropriate search intent, whether transactional, informational, or navigational.
    • Use Natural Language Processing (NLP) Tools: Leverage NLP tools to analyze how well your content aligns with searcher intent.*
  6. RankBrain and Vector Search:
    • Optimize for RankBrain: Vector search re-ranks results based on intent. Focus on improving content relevancy through quality rater feedback and actual user interactions.
  7. Monitor Industry-Specific Trends:
    • Industry Trends: Be aware of specific industry shifts (e.g., pet insurance, tech products) and adjust your SEO strategy accordingly. Specialized retailers and informational sites are gaining over commercial giants like Amazon.
  8. Consider Site Authority:
    • Balance Domain and Brand Authority: Tom Capper’s study suggests sites with higher brand authority (BA) performed better than those relying solely on domain authority (DA). Focus on building brand recognition alongside traditional SEO efforts.
  9. Core Update Response:
    • Future-Proofing: Continue producing high-quality, intent-driven content as Google aims to reward helpful content over SEO-first strategies. Sites that improve post-HCU or core updates will continue to see gains.
  10. Action for Content Creators:
  • Engage with the Community: Marie plans to collaborate with SB Pro members to develop a system for scoring intent alignment, potentially enlisting community support to study core update effects.

Marie also encourages community engagement, suggesting that producing authentic, user-focused content will be a key driver of success in the evolving search landscape.

supplements are bad

*1. Use Natural Language Processing (NLP) Tools to Analyze Content Alignment with Searcher Intent

What is NLP?

Natural Language Processing (NLP) is a branch of artificial intelligence that focuses on the interaction between computers and humans through natural language. In the context of SEO and content creation, NLP tools help analyze and understand the nuances of language, enabling better alignment of your content with what users are searching for.

Why Align Content with Searcher Intent?

Searcher intent refers to the primary goal a user has when typing a query into a search engine. Understanding and aligning your content with this intent ensures that your pages meet the needs of your audience, leading to higher engagement, better rankings, and increased conversions.

How to Leverage NLP Tools:

  1. Identify Searcher Intent:
    • Types of Intent:
      • Informational: Users seeking information (e.g., “How to fix a leaky faucet”).
      • Navigational: Users looking to reach a specific website or page (e.g., “Facebook login”).
      • Transactional: Users intending to make a purchase or complete a transaction (e.g., “Buy iPhone 14”).
      • Commercial Investigation: Users researching products or services before making a decision (e.g., “Best DSLR cameras 2024”).
  2. Analyze Your Content:
    • Keyword Analysis:
      • Use NLP tools to analyze the keywords and phrases within your content to determine if they align with the identified searcher intent.
    • Content Structure:
      • Ensure your content is structured in a way that directly addresses the user’s query. For instance, use clear headings, bullet points, and concise paragraphs for better readability and relevance.
    • Semantic Analysis:
      • Assess the context and relationships between words to ensure your content comprehensively covers the topic.
  3. Optimize Content Based on Insights:
    • Content Gap Analysis:
      • Identify areas where your content may be lacking in addressing certain aspects of the searcher intent and enhance those sections.
    • Content Enrichment:
      • Add relevant information, multimedia elements, or interactive features to better satisfy user needs.
    • Language Refinement:
      • Adjust the tone, complexity, and style of your content to match the preferences and expectations of your target audience.

Recommended NLP Tools:

  1. Google Natural Language API:
    • Offers powerful text analysis capabilities, including sentiment analysis, entity recognition, and syntax analysis.
    • Google Cloud Natural Language
  2. IBM Watson Natural Language Understanding:
    • Provides comprehensive text analysis features such as keyword extraction, concept tagging, and emotion analysis.
    • IBM Watson NLU
  3. Ahrefs Content Gap Tool:
    • While not a pure NLP tool, it helps identify content gaps by comparing your site with competitors, which can be complemented with NLP analysis.
    • Ahrefs Content Gap
  4. Surfer SEO:
    • Integrates NLP techniques to optimize content structure and keyword usage based on top-ranking pages.
    • Surfer SEO
  5. Clearscope:
    • Uses NLP to provide keyword suggestions and content optimization recommendations to improve relevancy.
    • Clearscope

Implementation Steps:

  1. Choose the Right NLP Tool:
    • Select a tool that fits your technical expertise and specific needs. For instance, Google Natural Language API is highly customizable but may require more technical setup compared to user-friendly platforms like Surfer SEO.
  2. Conduct a Content Audit:
    • Use the chosen NLP tool to analyze your existing content. Look for metrics like keyword density, semantic relevance, and sentiment alignment with user intent.
  3. Map Content to Search Intent:
    • Categorize your pages based on the primary search intent they fulfill. Ensure each page clearly aligns with its intended user goal.
  4. Optimize and Update Content:
    • Based on the analysis, make necessary adjustments to improve alignment. This could involve rewriting sections, adding new information, or restructuring content for better clarity and relevance.
  5. Monitor and Iterate:
    • After optimization, continuously monitor your content’s performance. Use analytics to assess improvements in rankings, engagement metrics, and conversions, and iterate as needed.

*2. Run Delta Reports: Isolate Dropped Queries and Landing Pages to Identify Reasons for Traffic Decline

What are Delta Reports?

Delta reports in SEO are comparative analyses that track changes in your website’s performance metrics over specific periods. By comparing data before and after a Google core update (or any significant change), you can identify which queries and landing pages experienced fluctuations in traffic.

Why Run Delta Reports?

Running Delta reports helps you pinpoint the exact areas of your website that are affected by algorithm changes. Understanding these changes allows you to take targeted actions to recover lost traffic and optimize your site for future updates.

How to Run Delta Reports:

  1. Gather Data:
    • Time Frames:
      • Pre-Update Period: Data from a period before the core update (e.g., two weeks before).
      • Post-Update Period: Data from a period after the core update has been fully rolled out (e.g., two weeks after).
    • Metrics to Collect:
      • Organic traffic
      • Impressions
      • Click-through rates (CTR)
      • Average position
  2. Use SEO Tools:
    • Google Search Console (GSC):
      • Provides comprehensive data on search performance, including queries and landing pages.
      • Google Search Console
    • Google Analytics:
      • Offers insights into overall traffic patterns and user behavior.
      • Google Analytics
    • Third-Party SEO Tools:
      • Tools like SEMrush, Ahrefs, and Moz can provide additional data and more sophisticated reporting features.
      • SEMrush
      • Ahrefs
      • Moz
  3. Generate Comparative Reports:
    • Google Search Console Steps:
      1. Access Performance Report:
        • Navigate to Performance > Search Results.
      2. Set Date Range:
        • Select a date range that covers both pre-update and post-update periods.
      3. Export Data:
        • Export the performance data for both periods to a spreadsheet for easier comparison.
      4. Calculate Delta:
        • Use formulas to calculate the percentage change in impressions, clicks, CTR, and average position for each query and landing page.
  4. Analyze the Data:
    • Identify Drops and Gains:
      • Highlight queries and pages that saw significant drops or gains in traffic.
    • Segment the Data:
      • Break down the data by categories such as content type, topic, or user intent to identify patterns.
    • Look for Correlations:
      • Determine if there are common factors among the affected queries/pages, such as content quality, relevance, or technical issues.
  5. Identify Potential Causes:
    • Relevancy Issues:
      • Content may no longer align with the updated search intent or relevancy criteria.
    • Technical Problems:
      • Issues like slow page speed, mobile usability problems, or broken links can impact rankings.
    • Content Quality:
      • Thin content, outdated information, or lack of depth can lead to declines.
    • Competitor Improvements:
      • Competitors may have enhanced their content, earning higher rankings.
    • User Experience (UX):
      • Poor UX, intrusive ads, or difficult navigation can negatively affect rankings.
  6. Develop an Action Plan:
    • Content Optimization:
      • Revise and enhance content to better match searcher intent and improve quality.
    • Technical SEO Fixes:
      • Address any identified technical issues to improve site performance.
    • Improve UX:
      • Enhance the overall user experience to keep visitors engaged and reduce bounce rates.
    • Monitor Competitors:
      • Analyze top-performing competitors to understand what changes they made that may have contributed to their gains.
    • Continuous Monitoring:
      • Regularly update and monitor Delta reports to track the effectiveness of your optimizations and stay ahead of future updates.

Tools and Techniques for Delta Reporting:

  1. Google Search Console (GSC):
    • Export and Compare:
      • Export performance data before and after the update and use spreadsheet tools to calculate differences.
    • Filtering Options:
      • Use filters to narrow down specific queries or pages for detailed analysis.
  2. Google Analytics:
    • Custom Reports:
      • Create custom reports to compare traffic sources, user behavior, and conversion metrics across different periods.
    • Annotations:
      • Use annotations to mark the date of the core update, helping correlate changes in data with the update rollout.
  3. SEO Platforms:
    • SEMrush’s Position Tracking:
      • Track keyword rankings over time and compare pre- and post-update positions.
    • Ahrefs’ Site Explorer:
      • Analyze backlink profiles and organic search traffic changes.
    • Moz’s Rank Tracker:
      • Monitor keyword performance and identify ranking fluctuations.
  4. Spreadsheet Analysis:
    • Pivot Tables:
      • Use pivot tables to summarize and analyze large datasets effectively.
    • Visualization:
      • Create charts and graphs to visualize traffic trends and identify significant changes.

Best Practices for Delta Reporting:

  • Consistent Time Frames:
    • Ensure the comparison periods are equivalent in length and cover similar days of the week to account for any variability.
  • Account for Seasonality:
    • Be aware of seasonal trends that might affect traffic independently of the core update.
  • Focus on Significant Changes:
    • Prioritize queries and pages with substantial traffic drops or gains to address the most impactful issues first.
  • Iterative Analysis:
    • Continuously refine your analysis by incorporating additional data points or adjusting your reporting criteria as needed.

Putting It All Together: Enhancing Your SEO Strategy

By integrating NLP-based content analysis and Delta reporting, you can create a robust SEO strategy that not only aligns your content with user intent but also allows you to proactively identify and address issues arising from algorithm updates. Here’s a step-by-step approach to combining these strategies:

  1. Initial Content Audit:
    • Use NLP tools to assess the alignment of your existing content with searcher intent.
    • Identify content areas that need enhancement or restructuring.
  2. Post-Update Performance Analysis:
    • After a core update, run Delta reports to identify affected queries and pages.
    • Cross-reference these findings with your NLP analysis to understand if misalignment with search intent contributed to the declines.
  3. Optimization and Implementation:
    • Optimize affected content based on insights from both NLP analysis and Delta reports.
    • Enhance content quality, adjust for better intent alignment, and fix any technical issues identified.
  4. Continuous Monitoring and Improvement:
    • Regularly use NLP tools to keep your content aligned with evolving search intents.
    • Continuously monitor your site’s performance with Delta reports to stay ahead of any negative trends.
  5. Community Engagement and Learning:
    • Engage with SEO communities (like the Search Bar Pro Community mentioned in your summary) to share insights and learn from others’ experiences.
    • Stay updated with the latest SEO trends and algorithm changes to adapt your strategy accordingly.

Additional Resources

  • Google’s Search Quality Evaluator Guidelines:
  • Moz’s Beginner’s Guide to SEO:
  • Ahrefs’ SEO Learning Center:
    • Offers tutorials and articles on various SEO topics, including content optimization and performance analysis.
    • Ahrefs Learning Center
  • Surfer SEO’s Guides:
    • Detailed guides on using their tools for content optimization and SEO analysis.
    • Surfer SEO Guides

Implementing these strategies effectively can lead to improved search rankings, better user engagement, and sustained organic growth. By continuously aligning your content with user intent and proactively monitoring your site’s performance, you can navigate algorithm updates with greater confidence and resilience.

If you have any further questions or need assistance with specific tools or techniques, feel free to ask!

eCommerce Structured Data

E-commerce Example for Structured Data Carousels

Below are step by step instructions

You will need:

  • A “summary” or “Product Listings Page” with structured data
  • Structured data on the product pages themselves

At the time of writing, this feature is only available in EU countries.

Step 1: Choose and Implement a Summary Page

Create a summary page that lists multiple products, each linking to a detailed product page.

For example, a category page that lists “Top 5 Winter Coats.”

Step 2: Add Required Properties to the Summary Page

Add the ItemList structured data to your summary page. Here’s an example in JSON-LD format:

<html>
  <head>
    <title>Top 5 Winter Coats</title>
    <script type="application/ld+json">
      {
        "@context": "https://schema.org",
        "@type": "ItemList",
        "itemListElement": [
          {
            "@type": "ListItem",
            "position": 1,
            "item": {
              "@type": "Product",
              "name": "Puffy Coat Series by Goat Coat",
              "image": [
                "https://example.com/photos/1x1/photo.jpg",
                "https://example.com/photos/4x3/photo.jpg",
                "https://example.com/photos/16x9/photo.jpg"
              ],
              "offers": {
                "@type": "AggregateOffer",
                "lowPrice": 45.00,
                "highPrice": 60.00,
                "priceCurrency": "USD"
              },
              "aggregateRating": {
                "@type": "AggregateRating",
                "ratingValue": 4.9,
                "reviewCount": 50
              },
              "url": "https://www.example.com/puffy-coats"
            }
          },
          {
            "@type": "ListItem",
            "position": 2,
            "item": {
              "@type": "Product",
              "name": "Wool Coat Series by Best Coats Around",
              "image": [
                "https://example.com/photos/1x1/photo.jpg",
                "https://example.com/photos/4x3/photo.jpg",
                "https://example.com/photos/16x9/photo.jpg"
              ],
              "offers": {
                "@type": "AggregateOffer",
                "lowPrice": 189.00,
                "highPrice": 200.00,
                "priceCurrency": "USD"
              },
              "aggregateRating": {
                "@type": "AggregateRating",
                "ratingValue": 4.7,
                "reviewCount": 827
              },
              "url": "https://www.example.com/wool-coats"
            }
          },
          {
            "@type": "ListItem",
            "position": 3,
            "item": {
              "@type": "Product",
              "name": "Antarctic Coat by Cold Coats",
              "image": [
                "https://example.com/photos/1x1/photo.jpg",
                "https://example.com/photos/4x3/photo.jpg",
                "https://example.com/photos/16x9/photo.jpg"
              ],
              "offers": {
                "@type": "Offer",
                "price": 45.00,
                "priceCurrency": "USD"
              },
              "aggregateRating": {
                "@type": "AggregateRating",
                "ratingValue": 4.9,
                "reviewCount": 1290
              },
              "url": "https://www.example.com/antarctic-coat"
            }
          }
        ]
      }
    </script>
  </head>
  <body>
  </body>
</html>

Step 3: Validate Your Code

  1. Rich Results Test: Validate your structured data using the Rich Results Test.
  2. URL Inspection Tool: Use the URL Inspection tool in Google Search Console to ensure your page is accessible and not blocked by robots.txt or noindex tags.

Step 4: Deploy Your Page

  1. Deploy your summary page and a few detail pages.
  2. Use the URL Inspection tool to request indexing of your new pages.

Step 5: Submit a Sitemap

  1. Create a sitemap including your summary and detail pages.
  2. Submit the sitemap through Google Search Console to ensure Google knows about all your pages.

Step 6: Follow Guidelines

  1. Ensure compliance with Search Essentials.
  2. Adhere to general structured data guidelines and specific requirements for carousel rich results.

Example Detail Pages

Each product linked from the summary page should have its own detail page with structured data. For example:

<html>
  <head>
    <title>Puffy Coat Series by Goat Coat</title>
    <script type="application/ld+json">
      {
        "@context": "https://schema.org",
        "@type": "Product",
        "name": "Puffy Coat Series by Goat Coat",
        "image": [
          "https://example.com/photos/1x1/photo.jpg",
          "https://example.com/photos/4x3/photo.jpg",
          "https://example.com/photos/16x9/photo.jpg"
        ],
        "description": "A stylish and warm puffy coat perfect for winter.",
        "offers": {
          "@type": "Offer",
          "price": 45.00,
          "priceCurrency": "USD",
          "url": "https://www.example.com/puffy-coats"
        },
        "aggregateRating": {
          "@type": "AggregateRating",
          "ratingValue": 4.9,
          "reviewCount": 50
        }
      }
    </script>
  </head>
  <body>
  </body>
</html>

That should be it!



Finding & Fixing 404s with Screaming Frog [2026] – Site-Wide 404s

Export the 404 Inlinks Report into Excel

The best report in Screaming Frog to see the source and destination of all 404s – is to go to Bulk Export at the top menu:

And then Response Codes – Client Error Inlinks

In the Exported Excel Sheet:

Copy the “destination” column

Paste into a new sheet/tab – In Column B

Remove duplicates

Back in the first sheet – Paste “Source”, into column D to the right of “destination” in first tab/sheet


In the second sheet –
Do a vlookup in the second sheet/tab – to import a source URL to each unique “destination” URL
Add the word “source” to cell C1 in the second sheet

I think you have to click the file name to download the example sheet below:

Sheet / Tab 1 should look like this:

The Second Sheet / Tab should look like this:

The Lookup Value in Cell C2, in the sheet above is:

=VLOOKUP(Sheet1!B2,’1 – Client Error (4xx) Inlinks’!C:D,2,0)

Just double click / drag it down to complete the lookup

You can add more lookups and columns to provide more details about the link location.

Just copy and paste values on the last Vlookup.

Paste another column like “Anchor” into column D on the first Tab

Paste the vlookup into the adjacent column

“0” means nothing found – i.e. no anchor text etc

Update –

404s found in the Navigation or “Aside” tend to be site-wide, so you can find one instance of this and update the 404 and Bob’s your uncle.

If a 404 is within the page content/copy however, they are possibly unique and need fixing individually. So take a look the Content 404s separately. They probably only have 1 source URL so no need to faff with vlookups etc.

Semantic Search & SEO – Using Python and Google Colab

Semantic search adds context and meaning to search results. For example, if someone is searching for “Lego” – do they want to buy Lego toys, or see a Lego movie or TV show (Ninjago is great). Another example might be “Tesla” – do people want to see the latest self-driving car, or learn more about Tesla the scientist and inventor?

  • Make sure you understand search intent and any confusing searches like Tesla(inventor or car?), Jaguar (car or animal?), etc
  • Look for structured data opportunities
  • Optimise internal links – especially if you are using a “Pillar Post” and “Cluster Page” structure
  • Follow traditional on page SEO best practices with headers, meta titles, alt tags etc

SMA Marketing have done a cool YouTube video about Semantic Search and they recommend tools including:

  • Wordlift
  • Frase
  • Advanced Custom Fields for WordPress
  • Google Colab with a SpaCy

Before you publish a post – look at the search results for the keyword(s) you are optimising the post for. Check in incognito in Chrome to remove most of the personalisation of the results.

For any answer boxes or snippets, you can click the “3 dots” to get information about the results:

As well as the snippets, you can click the 3 dots next to any organic result. Here’s another result for “MMA training program pdf” with some additional information:

With this in mind – if you are looking to rank for “MMA training program pdf” then you will want to include the search terms highlighted in the “About this result” box: mma, training, program, pdf and ideally LSI keywords “workout” and “plan”.

It’s also a good idea to scroll down to the bottom of the SERP and check out the “related searches”

Take a look too at any breadcrumb results that pull through below the organic listings. Combining all this information will give you a good idea as to what Google understands by your search query and what people are looking for too.

Semantic Search & NLP

This is a bit techy, but thankfully, the guy at SMA Marketing (thank you if you’re reading this) has put together a file/load of python code that does most of the work for us. You can find it here – https://colab.research.google.com/drive/1PI6JBn06i3xNUdEuHZ9xKPG3oSRi1AUm?usp=sharing#scrollTo=uatWEoHp5nxZ

Hover over [1] and click the play icon that appears (highlighted yellow in screenshot below)

When that section has finished loading and refreshing, scroll down to the “Installation tensorflow + transformers + pipelines” section and click the play icon there.

When that’s finished doing it’s thing, scroll down again, and add your search query to the uQuery_1: section:

add your query and then press the “play” button on the left hand side opposite the uQuery_1 line

You should then see the top 10 organic results from Google on the left hand side – in the form of a list of URLs

Next, you can scrape all the results by scrolling down to the “Scraping results with Trafilatura” section and hover over the “[ ]” and press play again:

Next, when the scraping of results is done – scroll down to “Analyze terms from the corpus of results” section and click the play button that appears when you hover over “[ ]”

Next! when that’s done click the play button on the section full of code starting with:

“df_1[‘top_result’] = [‘Top 3’ if x <= 3 else ‘Positions 4 – 10’ for x in df_1[‘position’]] # add top_result = True when position <=3 “

Finally – scroll down and click the play button on the left of the “Visualizing the Top Results” section.

On the right hand side where it says “Top Top 3” and lists a load of keywords/terms – these are frequent and meaningful (apparently) terms used in the top 3 results for your search term.

Below that, you can see the terms used in the results from 4-10

Terms at the top of the graph are used frequently in the top 3 results e.g. “Mini bands”

Terms on the right are used frequently by the results in positions 4-10

From the graph above, I can see that for the search term “resistance bands” the top 3 results are using some terms, not used by 4-10 – including “Mini bands”, “superbands” “pick bodylastics”

  • If you click on a term/keyword in the graph – a ton of information appears just below:

e.g. if I click “mini bands”

Google Colab TOol

It’s interesting that “mini bands” is not featured at all in the results positioned 4-10

If you were currently ranking in position 7 for example, you’d probably want to look at adding “mini bands” into your post or product page

You can now go to the left-side-bar and click “Top 25 Terms” and click the “play icon” to refresh the data:

Semantic SEO tool

Obviously – use your experience etc and take the results with a pinch of salt – some won’t be relevant.

Natural Language Processing

next click on “Natural Langauge Processing” in the side-menu

Click the “play” icons next to “df_entity =df_1[df_1[‘position’] < 6]” and the section below.

When they have finished running click the play icon next to “Extracting Entities”

Click “play” on the “remove duplicates” section and again on the “Visualising Data” section

This should present you with a colourful table, with more terms and keywords – although for me most of the terms weren’t relevant in this instance 😦

You can also copy the output from the “Extracting the content from Top 5” section:

Python Google Colab
Then paste it into the DEMO/API for NLP that Google have created here:

https://cloud.google.com/natural-language#section-2

You can then click the different tabs/headings and get some cool insights

Google NLP API

Remember to scroll right down to the bottom, as you’ll find some additional insights about important terms and their relevance

The Google NLP API is pretty interesting. You can also copy and paste your existing page copy into it, and see what Google categories different terms as, and how “salient” or important/relevant it thinks each term is. For some reason, it thinks “band” is an organisation in the above screenshot. You can look to improve the interpretations by adding relevant contextual copy around the term on the page, by using schema and internal links.

Quickly Scrape SERP Results for SEO

Go to Google.com

Press CTRL+D to bookmark the page

Add it to your bookmarks bar if possible

Copy the code below:

javascript:var a = document.getElementsByTagName('a'), arr = '';for(var i=0; i<a.length; i++) if (a[i].ping && !a[i].href.includes('google'))arr +=('<p>' + a[i].href + '</p>');var newWindow = window.open();newWindow.document.write(arr);newWindow.document.close();


Right click on the bookmark and click “Edit”

Where it says “URL” paste in the code

Rename the bookmark “URL Extractor”

Save the bookmark

That’s it!

Test the URL Extractor – do a Google Search and then click on the Bookmark you’ve just created

You should now have a new tab with all of the URLs listed in it

Change your Google Search settings to show 100 results – you can just copy and paste the tab full of extracted URLS into Google Sheets

Once in Google Sheets, you can easily extract the Meta Titles, Descriptions and keywords

You just need to find and replace the URL in these formulas:

=importxml("https://blackbeltwhitehat.com","//title")
=importxml("https://blackbeltwhitehat.com/","//meta[@name='description']/@content")
=importxml("https://blackbeltwhitehat.com/","//meta[@name='keywords']/@content")

=IMPORTXML("https://blackbeltwhitehat.com/","//h1")

Data Studio Fields & Filters – Notes

Once you’ve created a field – add it to your chart as a Dimension

Mixed Case URLs

Having a mix of URL cases in the letters can faff with your data as D.S. might think each capitalisation variation is a new URL:

  • Click on the Data Tab to the right – then “Create new field” or “Add a Field”
  • Name the field “Lower Source” type in “LOWER(“
  • Then add “Source” from “Available Fields” to the left
  • Click “Save” and add “Lower Source” to the table as a Dimension

Concatenate Data

  • Create or “Add a Field”
  • In the Formula box type “CONCAT” and then select the fields you want to use
  • Close the formula with the final “)” and save it

Internal Site Searches – Extract Search Query

Make the data look nicer and get the search term on its own in the table.

  • Select to a “Add a Field”
  • Use REGEXP_EXTRACT formula to pull out the search terms and get rid of “/search?q”
  • Use the REGEX shown below:

Search Queries – Pull Out Questions – What, Why, When?

  • Create / Add a New Field
  • Add REGEX as shown below and save
  • Add New Field as a Dimension to the table
  • Create a table/data Filter so that you include only table rows that equate to “True” in the new dimension/column

CASE Formulas in Data Studio

CASE formulas are basically “If this, then do that” formulas

When X happens, Then do Y

You can use the CASE Formula to classify and group channels together

For example:

WHEN query matches “who”, then display text “Who?” in the table

  • In the Formula – you need a “catch all” default for when nothing is true in the criteria
  • If the search doesn’t contain “who, what, why” etc. then:
    ELSE “others”

    So if the search term, doesn’t match any of the REGEX criteria – classify it as “others”
– Save the CASE statement
  • Add the new field – to the table or report

Date Ranges & Filter Controls

Taken me ages to work this out – I’ve only just twigged:

You can add Comparisons, Search Boxes, Drop Down Menus and all sorts to your Data Studio Reports

For controls and filters to work – you need the chart or table to have “Default Date Range” set to “Auto”

See my post about REGEX for SEO here.

Ta

If you want a pre-made Data Studio template, you can send me some money and hope for the best.

How to Create a Topic Cluster for SEO

Using Ahrefs with Wikipedia

  • Find the most relevant Wikipedia page, for example:

https://en.wikipedia.org/wiki/Association_football

  • Add the Wikipedia URL and click “submit”
  • Click “Copy to Clipboard”
  • Paste all the keywords/topics into a Google Sheet and remove any that aren’t relevant
  • In Ahrefs (or a similar tool like SEMRush’s KW Manager) – go to KW explorer
  • Paste in the keywords/topics
  • In Ahrefs – click “matching terms” then “Questions”

Using SEMRush with UGC

  • Copy the homepage URL of a UGC* website such as Pinterest or Quora
  • Put the URL in the main “search bar” in SEMRush
  • Go to “Organic Research” using the sidebar
  • Click “Positions”
  • Add a Keyword to the “Filter by Keyword” search box
  • Click a relevant Keyword in the table of results
  • Under “Questions” click “View all….”
  • Go back and click other relevant keywords
  • Click the “Questions” tab again – make a note or download all the relevant questions
  • Organise KWs and questions into main pillar posts and “cluster content”

See Hubspot’s article about Pillar Pages and Cluster Content – here

*UGC – User Generated Content

Google Ads Reporting Script

This script is no longer live, but I had it in my account, I’m pasting it here for safekeeping and future reference as it is pretty handy:

to use the script:

Change the Google sheet/doc URL and the email address

  • Sign into Google Ads
  • Click on “Tools & Settings” on the top menu near the right hand side
  • Click “scripts”
  • Click the addition (+) sign to add a new script
  • Paste in the below script – save – preview and then run

// Copyright 2015, Google Inc. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
//     http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

/**
 * @name Account Summary Report
 *
 * @overview The Account Summary Report script generates an at-a-glance report
 *     showing the performance of an entire Google Ads account. See
 *     https://developers.google.com/google-ads/scripts/docs/solutions/account-summary
 *     for more details.
 *
 * @author Google Ads Scripts Team [adwords-scripts@googlegroups.com]
 *
 * @version 1.1
 *
 * @changelog
 * - version 1.1
 *   - Add user-updateable fields, and ensure report row ordering.
 * - version 1.0.4
 *   - Improved code readability and comments.
 * - version 1.0.3
 *   - Added validation for external spreadsheet setup.
 * - version 1.0.2
 *   - Fixes date formatting bug in certain timezones.
 * - version 1.0.1
 *   - Improvements to time zone handling.
 * - version 1.0
 *   - Released initial version.
 */

var RECIPIENT_EMAIL = 'drewgriffiths@live.com';

var SPREADSHEET_URL = 'https://docs.google.com/spreadsheets/d/1mNjc7iJWOIq580DMLf6rYKT8xRAcl2MynnGSY29UiXY/edit#gid=3';

/**
 * Configuration to be used for running reports.
 */
var REPORTING_OPTIONS = {
  // Comment out the following line to default to the latest reporting version.
  apiVersion: 'v201809'
};

/**
 * To add additional fields to the report, follow the instructions at the link
 * in the header above, and add fields to this variable, taken from the Account
 * Performance Report reference:
 * https://developers.google.com/adwords/api/docs/appendix/reports/account-performance-report
 */
var REPORT_FIELDS = [
  {columnName: 'Cost', displayName: 'Cost'},
  {columnName: 'AverageCpc', displayName: 'Avg. CPC'},
  {columnName: 'Ctr', displayName: 'CTR'},
  {columnName: 'Impressions', displayName: 'Impressions'},
  {columnName: 'Clicks', displayName: 'Clicks'}
];

function main() {
  Logger.log('Using spreadsheet - %s.', SPREADSHEET_URL);
  var spreadsheet = validateAndGetSpreadsheet();
  spreadsheet.setSpreadsheetTimeZone(AdsApp.currentAccount().getTimeZone());
  spreadsheet.getRangeByName('account_id_report').setValue(
      AdsApp.currentAccount().getCustomerId());

  var yesterday = getYesterday();
  var date = getFirstDayToCheck(spreadsheet, yesterday);

  var rows = [];
  var existingDates = getExistingDates();

  while (date.getTime() <= yesterday.getTime()) {
    if (!existingDates[date]) {
      var row = getReportRowForDate(date);
      rows.push([new Date(date)].concat(REPORT_FIELDS.map(function(field) {
        return row[field.columnName];
      })));
      spreadsheet.getRangeByName('last_check').setValue(date);
    }
    date.setDate(date.getDate() + 1);
  }

  if (rows.length > 0) {
    writeToSpreadsheet(rows);

    var email = spreadsheet.getRangeByName('email').getValue();
    if (email) {
      sendEmail(email);
    }
  }
}

/**
 * Retrieves a lookup of dates for which rows already exist in the spreadsheet.
 *
 * @return {!Object} A lookup of existing dates.
 */
function getExistingDates() {
  var spreadsheet = validateAndGetSpreadsheet();
  var sheet = spreadsheet.getSheetByName('Report');

  var data = sheet.getDataRange().getValues();
  var existingDates = {};
  data.slice(5).forEach(function(row) {
    existingDates[row[1]] = true;
  });
  return existingDates;
}

/**
 * Sorts the data in the spreadsheet into ascending date order.
 */
function sortReportRows() {
  var spreadsheet = validateAndGetSpreadsheet();
  var sheet = spreadsheet.getSheetByName('Report');

  var data = sheet.getDataRange().getValues();
  var reportRows = data.slice(5);
  if (reportRows.length) {
    reportRows.sort(function(rowA, rowB) {
      if (!rowA || !rowA.length) {
        return -1;
      } else if (!rowB || !rowB.length) {
        return 1;
      } else if (rowA[1] < rowB[1]) {
        return -1;
      } else if (rowA[1] > rowB[1]) {
        return 1;
      }
      return 0;
    });
    sheet.getRange(6, 1, reportRows.length, reportRows[0].length)
        .setValues(reportRows);
  }
}

/**
 * Append the data rows to the spreadsheet.
 *
 * @param {Array<Array<string>>} rows The data rows.
 */
function writeToSpreadsheet(rows) {
  var access = new SpreadsheetAccess(SPREADSHEET_URL, 'Report');
  var emptyRow = access.findEmptyRow(6, 2);
  if (emptyRow < 0) {
    access.addRows(rows.length);
    emptyRow = access.findEmptyRow(6, 2);
  }
  access.writeRows(rows, emptyRow, 2);
  sortReportRows();
}

function sendEmail(email) {
  var day = getYesterday();
  var yesterdayRow = getReportRowForDate(day);
  day.setDate(day.getDate() - 1);
  var twoDaysAgoRow = getReportRowForDate(day);
  day.setDate(day.getDate() - 5);
  var weekAgoRow = getReportRowForDate(day);

  var html = [];
  html.push(
    '<html>',
      '<body>',
        '<table width=800 cellpadding=0 border=0 cellspacing=0>',
          '<tr>',
            '<td colspan=2 align=right>',
              "<div style='font: italic normal 10pt Times New Roman, serif; " +
                  "margin: 0; color: #666; padding-right: 5px;'>" +
                  'Powered by Google Ads Scripts</div>',
            '</td>',
          '</tr>',
          "<tr bgcolor='#3c78d8'>",
            '<td width=500>',
              "<div style='font: normal 18pt verdana, sans-serif; " +
              "padding: 3px 10px; color: white'>Account Summary report</div>",
            '</td>',
            '<td align=right>',
              "<div style='font: normal 18pt verdana, sans-serif; " +
              "padding: 3px 10px; color: white'>",
               AdsApp.currentAccount().getCustomerId(), '</h1>',
            '</td>',
            '</tr>',
          '</table>',
          '<table width=800 cellpadding=0 border=0 cellspacing=0>',
            "<tr bgcolor='#ddd'>",
              '<td></td>',
              "<td style='font: 12pt verdana, sans-serif; " +
                  'padding: 5px 0px 5px 5px; background-color: #ddd; ' +
                  "text-align: left'>Yesterday</td>",
              "<td style='font: 12pt verdana, sans-serif; " +
                  'padding: 5px 0px 5px 5px; background-color: #ddd; ' +
                  "text-align: left'>Two Days Ago</td>",
              "<td style='font: 12pt verdana, sans-serif; " +
                  'padding: 5px 0px 5x 5px; background-color: #ddd; ' +
                  "text-align: left'>A week ago</td>",
            '</tr>');
  REPORT_FIELDS.forEach(function(field) {
    html.push(emailRow(
        field.displayName, field.columnName, yesterdayRow, twoDaysAgoRow,
        weekAgoRow));
  });
  html.push('</table>', '</body>', '</html>');
  MailApp.sendEmail(email, 'Google Ads Account ' +
      AdsApp.currentAccount().getCustomerId() + ' Summary Report', '',
      {htmlBody: html.join('\n')});
}

function emailRow(title, column, yesterdayRow, twoDaysAgoRow, weekAgoRow) {
  var html = [];
  html.push('<tr>',
      "<td style='padding: 5px 10px'>" + title + '</td>',
      "<td style='padding: 0px 10px'>" + yesterdayRow[column] + '</td>',
      "<td style='padding: 0px 10px'>" + twoDaysAgoRow[column] +
          formatChangeString(yesterdayRow[column], twoDaysAgoRow[column]) +
          '</td>',
      "<td style='padding: 0px 10px'>" + weekAgoRow[column] +
          formatChangeString(yesterdayRow[column], weekAgoRow[column]) +
          '</td>',
      '</tr>');
  return html.join('\n');
}


function getReportRowForDate(date) {
  var timeZone = AdsApp.currentAccount().getTimeZone();
  var dateString = Utilities.formatDate(date, timeZone, 'yyyyMMdd');
  return getReportRowForDuring(dateString + ',' + dateString);
}

function getReportRowForDuring(during) {
  var report = AdsApp.report(
      'SELECT ' +
          REPORT_FIELDS
              .map(function(field) {
                return field.columnName;
              })
              .join(',') +
          ' FROM ACCOUNT_PERFORMANCE_REPORT ' +
          'DURING ' + during,
      REPORTING_OPTIONS);
  return report.rows().next();
}

function formatChangeString(newValue,  oldValue) {
  var x = newValue.indexOf('%');
  if (x != -1) {
    newValue = newValue.substring(0, x);
    var y = oldValue.indexOf('%');
    oldValue = oldValue.substring(0, y);
  }

  var change = parseFloat(newValue - oldValue).toFixed(2);
  var changeString = change;
  if (x != -1) {
    changeString = change + '%';
  }

  if (change >= 0) {
    return "<span style='color: #38761d; font-size: 8pt'> (+" +
        changeString + ')</span>';
  } else {
    return "<span style='color: #cc0000; font-size: 8pt'> (" +
        changeString + ')</span>';
  }
}

function SpreadsheetAccess(spreadsheetUrl, sheetName) {
  this.spreadsheet = SpreadsheetApp.openByUrl(spreadsheetUrl);
  this.sheet = this.spreadsheet.getSheetByName(sheetName);

  // what column should we be looking at to check whether the row is empty?
  this.findEmptyRow = function(minRow, column) {
    var values = this.sheet.getRange(minRow, column,
        this.sheet.getMaxRows(), 1).getValues();
    for (var i = 0; i < values.length; i++) {
      if (!values[i][0]) {
        return i + minRow;
      }
    }
    return -1;
  };
  this.addRows = function(howMany) {
    this.sheet.insertRowsAfter(this.sheet.getMaxRows(), howMany);
  };
  this.writeRows = function(rows, startRow, startColumn) {
    this.sheet.getRange(startRow, startColumn, rows.length, rows[0].length).
        setValues(rows);
  };
}

/**
 * Gets a date object that is 00:00 yesterday.
 *
 * @return {Date} A date object that is equivalent to 00:00 yesterday in the
 *     account's time zone.
 */
function getYesterday() {
  var yesterday = new Date(new Date().getTime() - 24 * 3600 * 1000);
  return new Date(getDateStringInTimeZone('MMM dd, yyyy 00:00:00 Z',
      yesterday));
}

/**
 * Returned the last checked date + 1 day, or yesterday if there isn't
 * a specified last checked date.
 *
 * @param {Spreadsheet} spreadsheet The export spreadsheet.
 * @param {Date} yesterday The yesterday date.
 *
 * @return {Date} The date corresponding to the first day to check.
 */
function getFirstDayToCheck(spreadsheet, yesterday) {
  var last_check = spreadsheet.getRangeByName('last_check').getValue();
  var date;
  if (last_check.length == 0) {
    date = new Date(yesterday);
  } else {
    date = new Date(last_check);
    date.setDate(date.getDate() + 1);
  }
  return date;
}

/**
 * Produces a formatted string representing a given date in a given time zone.
 *
 * @param {string} format A format specifier for the string to be produced.
 * @param {date} date A date object. Defaults to the current date.
 * @param {string} timeZone A time zone. Defaults to the account's time zone.
 * @return {string} A formatted string of the given date in the given time zone.
 */
function getDateStringInTimeZone(format, date, timeZone) {
  date = date || new Date();
  timeZone = timeZone || AdsApp.currentAccount().getTimeZone();
  return Utilities.formatDate(date, timeZone, format);
}

/**
 * Validates the provided spreadsheet URL to make sure that it's set up
 * properly. Throws a descriptive error message if validation fails.
 *
 * @return {Spreadsheet} The spreadsheet object itself, fetched from the URL.
 */
function validateAndGetSpreadsheet() {
  if ('YOUR_SPREADSHEET_URL' == SPREADSHEET_URL) {
    throw new Error('Please specify a valid Spreadsheet URL. You can find' +
        ' a link to a template in the associated guide for this script.');
  }
  var spreadsheet = SpreadsheetApp.openByUrl(SPREADSHEET_URL);
  var email = spreadsheet.getRangeByName('email').getValue();
  if ('foo@example.com' == email) {
    throw new Error('Please either set a custom email address in the' +
        ' spreadsheet, or set the email field in the spreadsheet to blank' +
        ' to send no email.');
  }
  return spreadsheet;
}

Bob’s your uncle

The code was originally on the Google Developer’s site here – https://developers.google.com/adwords/scripts/docs/solutions/ad-performance

Please Google, don’t bankrupt me for using the script on here! I’ll take it down if it upsets thee.