SEO & Digital Marketing News Round-Up 5 October to 19 October 2026

TL;DR

The September spam update finished on 8 October after 13 days and 16 hours. Three waves of ranking volatility hit sites along the way. Google then quietly tightened its helpful content guidance, published its first official crawl, index and recovery timing ranges, and added more inline links, carousels and creator profiles to AI Mode. OpenAI shipped GPT 6 with Intelligent UI. Google followed with Gemini 4 Argon. Bing opened a Topics Portfolio beta that may be the best free AI visibility report currently available.

New since Friday: Fresh ranking volatility spiked over the weekend of 17 to 19 October. Google denies a new update. The industry suspects one. OpenAI confirmed a ChatGPT bug that dropped UTM tags on outbound links from 8 to 15 October. Nozzle revealed that roughly 8% of Google SERPs served to rank trackers since mid September were poisoned with fake results. Google also pushed Gemini 4 Argon into Antigravity and added a Follow button to Discover videos.

Three jobs this week: date impact analysis from 24 September to 8 October, mark 8 to 15 October in analytics as a ChatGPT data glitch, and claim the Bing Topics Portfolio beta.

What’s new since Friday 16 October

  • 15 October: Google rolled more inline links, source blocks, article carousels and creator profile links into AI Mode and AI Overviews.
  • 16 October: A ranking volatility spike hit the SERPs. Barry Schwartz flagged an unconfirmed update. Google’s dashboard still shows nothing.
  • 14 to 15 October: OpenAI confirmed and fixed a bug that stripped UTM tags from ChatGPT outbound links between 8 and 15 October. Adobe tracked roughly 31,000 ChatGPT visits per day misclassified as direct traffic across one 500 page panel.
  • 10 October: Nozzle published research showing 8% of SERPs served to rank trackers since 13 September were poisoned, versus a baseline of 0.17%. The share peaked near 27% between 27 and 29 September.
  • 16 October: Gemini 4 Argon reached Google Antigravity for AI Pro subscribers. Google Discover added a Follow button to its video content viewer.
  • Over the weekend of 17 to 19 October: WebmasterWorld threads filled with reports of near zero traffic, stale News and Discover feeds, and the strong suspicion of a core update that Google has not confirmed.

Fourteen days, three waves of volatility, two major AI launches, one giant data glitch. This round up covers 5 October to 19 October 2026, what it means, and what to do about it.


What To Actually Know

Six developments shape planning for the rest of Q4:

  1. The September 2026 spam update finished on 8 October at 04:37 ET. Rollout time: 13 days and 16 hours. Normal spam updates in 2026 took two days. Google called this one “a normal spam update.” Barry Schwartz, Glenn Gabe and Lily Ray all described effects closer to a core update.
  2. Fresh volatility hit on 16 October. It kept running through the weekend. Google’s Search Status Dashboard shows no new update. Forum chatter leans heavily towards one being in progress.
  3. Google published official crawl, index and recovery timing ranges at Search Central Live Deep Dive Europe on 2 October. Core update recovery typical: 3 to 6 months. Slowest: 6 months to a year.
  4. Helpful content guidance tightened on authorship and AI on 1 October and again on 5 October. Fake author profiles are now called deception. Unreviewed AI content gets called out by name. Spam policy did not change. The quality lens did.
  5. OpenAI shipped GPT 6 and Intelligent UI on 7 and 8 October. Google followed with Gemini 4 Argon on 13 October. Answers now contain interactive tools, forms and charts. Plain text answers are becoming the exception.
  6. Rank tracking data is unreliable. Nozzle found Google serving fake results to tracker tools on 8% of SERPs since mid September. Combined with the ChatGPT UTM bug from 8 to 15 October, nothing outside Search Console or Bing Webmaster Tools can be fully trusted right now.

1. The September 2026 Spam Update finished, but the aftershocks haven’t

Google Search Central confirmed completion on 8 October. The quote: “The rollout was complete as of October 8, 2026.” Rollout time: 13 days and 16 hours. That’s longer than any 2026 spam update by an order of magnitude.

The 2026 spam update timeline

Update Started Finished Rollout time
March 2026 24 Mar 25 Mar About 19.5 hours
June 2026 24 Jun 26 Jun About 2 days 1 hour
August 2026 18 Aug 21 Aug About 2 days 16 hours
September 2026 24 Sep 8 Oct About 13 days 16 hours

The three waves of volatility

Relative volatility across the rollout. Aggregated signal from Semrush Sensor, Mozcast, AccuRanker and WebmasterWorld threads.

24 Sep
Launch, moderate

25 to 27 Sep
Wave 1, very high

28 to 29 Sep
Quiet

30 Sep to 1 Oct
Wave 2, high

2 to 3 Oct
Elevated

4 to 6 Oct
Wave 3, strongest signal

7 Oct
Winding down

8 Oct
Rollout complete

16 to 19 Oct
New spike, source unconfirmed

Sources: Barry Schwartz (Search Engine Roundtable), Glenn Gabe (G Squared Interactive), aggregated via Numinix and Distribb.

What got hit, what came back

Lily Ray flagged a representative casualty. A local services site had 143 county landing pages built from a single template. Traffic rose fast, held, then evaporated. The pattern repeated across several site categories:

  • Programmatic or templated pages with little unique value.
  • Unreviewed AI content at scale.
  • Doorway pages that funnel traffic to a different destination.
  • Parasite SEO setups piggybacking on domains with trust they did not earn.

On the recovery side, Glenn Gabe reported Reddit’s AI translated threads climbing again in Italy, Germany, Brazil and France after being crushed by the May and June updates. Edward Sturm posted that Reddit’s estimated organic traffic in Semrush hit an all time high. One site hit by the August 2026 update went from close to zero to roughly 8,000 clicks per day during this rollout and reappeared in AI Overviews and AI Mode.

What the community is saying

“Google called this ‘a normal spam update’ on LinkedIn. Taking two weeks to roll out is not normal, and the impact felt core update adjacent for a lot of sites.”

Barry Schwartz, Search Engine Roundtable, 8 October

“Several site owners emailed me about large drops starting 30 September. This update rolled out in stages. The industry just saw them as one update.”

Glenn Gabe, G Squared Interactive

“Every single website I monitor or have analytics access to is seeing barely any traffic today.”

WebmasterWorld member, 16 October

“Sigh, was hoping that the end of the update would stabilize things, but it’s just made them worse.”

WebmasterWorld member, 16 October

“Seems like a Core update on the way.”

WebmasterWorld member, 19 October

Charles Floate flagged a different problem on LinkedIn. The major SEO suites look out of sync with Search Console. Google’s anti scraping changes are hiding the real picture from third party tools. Ahrefs posted a notice that some SERP data may be inconsistent during the rollout.


2. Google published its first official crawl, index and recovery times

At Search Central Live Deep Dive Europe in Barcelona on 2 October, Google’s Gary Illyes shared the first on record “typical” and “slowest” timing ranges for the full search pipeline. Search Engine Journal published them on 4 October. Think of these as the numbers finally safe to quote to a client.

Crawling and indexing

Event Typical Slowest
New URL discovered About 20 hours Weeks to never
Known URL refreshed About 30 days Weeks to never
Sitemap processing About 24 hours Up to 14 days, or never
End to end indexing About 1.5 hours Months, or never (quality)
Canonicalisation change 1 to 3 weeks Months
Site move 1 to 3 months 6 months to a year or more

Search results and recovery

Event Typical Slowest
Title change picked up 1 to 2 days Several weeks to months
Snippet change picked up 1 to 2 days Several weeks to months
GSC URL removal (owner) About 2 hours 24 hours
Manual action removal 1 to 2 weeks 4 to 6 weeks, longer for dormant sites
Core update recovery 3 to 6 months 6 months to a year

Illyes was clear that these are reference points, not deadlines. The typical core update recovery of 3 to 6 months is the single most useful stat Google has volunteered this year. It settles plenty of nervous conversations.


3. Google quietly tightened its helpful content and AI guidance

No new algorithm was announced. Instead, Google edited two Search Central documents on 1 October and again on 5 October. The pages: helpful content, and generative AI content. Barry Schwartz surfaced the authorship change on 6 October.

Three changes that matter

  1. Fake author profiles are now explicitly deception. The updated page calls out AI headshots, invented names and false credentials. Deception makes a page untrustworthy to users and to automated quality systems. The last phrase is new.
  2. Unreviewed AI content gets named directly. The generative AI guide tells teams to manually fact check AI output before publishing. Titles, meta descriptions, structured data and image alt text all included.
  3. Main Content got elevated. The helpful content page now defines Main Content as the primary thing that answers the user. Four quality markers follow: effort, originality, skill and experience, and accuracy.

At a separate European event on 5 October, Google reportedly told attendees that spam updates now use AI to detect AI spam. The most 2026 sentence of the year, and a strong hint at what the September rollout targeted.

Second order effects came fast. On 2 and 3 October, affiliate networks began asking publishers for proof of human authorship. By week two, some unionised newsrooms reportedly signed contracts banning synthetic bylines. Any content supply chain that includes third party writers now faces a compliance headache for Q4.


4. AI search: GPT 6, Gemini 4 Argon, and the end of plain text answers

The biggest week for AI search product news in 2026. Six releases in nine days:

Date Release Why it matters
5 Oct OpenAI visual ads in ChatGPT Native ad units alongside image generation. First US advertisers testing this month.
7 Oct GPT 6 and Intelligent UI for paid tiers Answers now contain interactive tools, forms, charts and mini apps.
8 Oct GPT 6 extends to Free and Go tiers About 1.2 billion weekly users now on Intelligent UI.
13 Oct Gemini 4 Argon rolls out in AI Mode AI Pro and Ultra get it worldwide. More direct answers, less hedging.
13 Oct Bing Topics Portfolio beta The first free report naming AI topic clusters where competitors show and other sites don’t.
15 Oct AI Mode and AI Overviews get more inline links, source blocks, article carousels and creator profiles Google says preferred source usage grew 7x this year.
16 Oct Gemini 4 Argon reaches Google Antigravity AI Pro subscribers only. Agentic coding workflows get the latest model.

Intelligent UI matters most

For most of ChatGPT’s history the output has been text. GPT 6’s Intelligent UI changes that. The model now picks the format. Plain text, buttons, forms, calculators, labelled diagrams, cooking timelines, playable games. Ask for a dinner for twelve and a quantity calculator appears. Ask about splitting a bill and a bill splitter builds itself in the conversation.

OpenAI also claims GPT 6 Instant starts answering web search questions 44% sooner than GPT 5.6 Instant on average, with better instruction following and less hedging. The Chat tab moved to GPT 6 Sol (paid) and GPT 6 Luna (free) across 7 and 8 October. Work and Codex are unchanged for now.

The SEO problem Intelligent UI creates

When the answer is a built in tool rather than a sentence with a citation, less surface area remains for a link to appear. Expect a two to three month lag before anyone has solid data on how much this reduces citations. Any brand actively tracking AI visibility in ChatGPT should assume the citation to tool ratio shifts against sites, not in favour of them.

Bing Topics Portfolio is the sleeper hit

For practical value, the quietest release of the fortnight was Microsoft’s. Bing Topics Portfolio sits inside Bing Webmaster Tools’ AI performance reports. It groups the grounding queries where a site appears into thematic clusters. Each topic shows:

  • Topic Size: total citations across all sites for that topic.
  • Sites Cited: how many sites appear in that topic cluster.
  • Queries: how many distinct AI queries sit within the topic.

The killer feature: Top Opportunity Queries. A list of strong AI queries where competitors appear and a given site does not. One free report, one claim, one Q4 content brief half written.

What Google added to AI Mode on 15 October

Google rolled a fresh set of changes into AI Mode and AI Overviews:

  • More inline links. Links appear inside words and phrases within AI responses, and higher up than before.
  • Source blocks. When only a few official sites answer a query, a dedicated section at the top lists them with a short explanation.
  • Article carousels. Shown in more locations, more languages and for a wider range of queries. Preferred sources get the spotlight.
  • Related articles with thumbnails. Images and card style layouts aim to feel more clickable.
  • Creator profile links in AI Mode videos. Clicking a creator’s name sends traffic to a Google Search profile page, not the creator’s own site.

The headline figure: Google reports that preferred source usage grew 7x this year. The caveat: no click data has been published, and Barry Schwartz suspects the overall effect on publisher clicks is small.


5. The measurement layer broke twice this fortnight

Two separate stories, same conclusion. Trusting anything outside first party data right now is risky.

Nozzle: 8% of SERPs served to rank trackers were fake

Nozzle published research on 10 October analysing 17.8 million SERPs since 13 September. The headline numbers:

Metric Baseline 13 Sep onwards
Share of SERPs poisoned 0.17% 8.0% average, peaking at 27% on 29 Sep
Organic results kept from previous SERP 88.5% 20.7%
Tracked site falls out of top 10 2.7% 58.6%

In short, Google appears to be serving fake pages to rank trackers rather than refusing their requests. Social sites and forums fill the gap. Reddit visibility rose by 8.0 points on poisoned pages. 77.9% of poisoned SERPs returned to normal within three days. Nozzle credits Google’s broader fight against AI scraping, with rank trackers caught in the collateral damage.

OpenAI’s UTM bug misclassified ChatGPT traffic as direct from 8 to 15 October

Between 8 and 15 October, ChatGPT stopped adding its utm_source=chatgpt.com tag to outbound links. Because ChatGPT also normally strips the HTTP referrer, GA4 and Adobe recorded those visits as direct traffic.

OpenAI acknowledged the error on 14 October and finished the fix roughly 24 hours later. Tagged links are now back to about 95 to 100% of normal levels.

Adobe’s Cedric Huesler analysed 500 pages where ChatGPT normally drives at least 5% of visits. Reported ChatGPT traffic fell by around 31,000 visits per day. Direct traffic rose by around 26,000. Two in every three pages that lost ChatGPT traffic also saw direct traffic rise.

The practical fix: mark 8 to 15 October as a known data glitch in GA4 or Adobe Analytics. Do not change attribution models or channel rules based on that week. Keep UTM parameters on anything AI driven.


6. Google Discover is becoming an “AI concierge,” plus the quick hits

On 12 October, Search Engine Land, PPC Land and Search Engine Journal all picked up a Google job listing for a Principal UX lead on Discover. Salary: up to $382,000. The brief: turn Discover into a proactive, hyper personalised concierge with tighter AI integration. The signal: Discover is being rebuilt as the feed side entry point to AI Mode, in the same way Search was rebuilt as the query side one.

Any publisher relying heavily on Discover traffic should watch this more carefully than any upcoming core update. Previous Discover rebuilds have been brutal on traffic. A Follow button quietly appeared in Discover’s video content viewer on 16 October, which hints at the direction of travel.

Quick hits (dated)

Date Story Action
5 Oct Google appeals the US search monopoly ruling No near term SEO impact. Watch Q1 2027.
5 Oct Dark mode confirmed for Google Search Console Cosmetic. Enjoy.
7 Oct Review rich results disappearing for healthcare pages YMYL health sites should re check schema and expect lower CTR.
8 Oct UGC Fresh Data Program opens applications Only relevant to UGC platforms. Paywall and login content is out.
10 to 12 Oct GSC Performance report lagged about 60 hours Longest delay since Dec 2025. Back to normal by 12 Oct.
13 Oct Google: site migrations can run beyond 4 months Factor into migration planning. Old URLs may still rank for months.
16 Oct Google Ads: online gambling category certification rollout starts Affected advertisers should begin applications.
16 Oct Google Ads: lead forms support lead scoring (up to 12 qualifying answers per question) Review lead form questions for high intent filters.
16 Oct Google Ads API moves to twice monthly releases Factor into dev team sprint planning.
16 Oct Google Dataset Search outage (restored same day) No lasting effect reported.
16 to 19 Oct Fresh ranking volatility, unconfirmed update rumours Check Search Console. Do not act on third party data alone.

What To Actually Do

A short list for the next ten working days, in rough priority order:

  1. Date impact analysis from 24 September to 8 October. Pull clicks, impressions and average position by page type for that window. Compare against the previous four weeks. Pages that lost traffic across both waves are the real spam signal casualties. Pages that dropped in wave one and recovered are noise.
  2. Mark 8 to 15 October as a ChatGPT data glitch in GA4 and Adobe Analytics. Any apparent ChatGPT traffic drop that week was the UTM bug. Direct traffic rose by roughly the same amount it fell. Do not change attribution models based on it.
  3. Run a fake author and AI hallucination audit on the top 20 pages. For each: is the author real and verifiable? If AI assisted, is a documented human review step on record? Flag any page that fails either test.
  4. Claim the Bing Topics Portfolio beta inside Bing Webmaster Tools. Pull the Top Opportunity Queries list. That is a Q4 AI visibility content brief, half written.
  5. Set recovery expectations using Google’s own numbers. 3 to 6 months typical for core update recovery, 6 months to a year worst case. 1 to 2 weeks for manual action removal. Up to 14 days for sitemap processing. All on record now.
  6. Add a GPT 6 citation check to the AI visibility tracker. Run the fixed prompt set after 8 October and compare citation share to the pre GPT 6 baseline. Falling citations alongside rising tool based answers signal what happened.
  7. Do not trust rank tracking data alone for the next 3 to 4 weeks. Nozzle’s research puts 8% of SERPs served to trackers as fake. Cross check against Search Console before any ranking based decision.
  8. Diversify away from Discover. The move to an AI concierge model is coming, and prior rebuilds have hurt traffic badly.
  9. Watch the Search Status Dashboard this week. If Google confirms the 16 October volatility as an update, impact analysis should run from that date, not the end of the spam update.

The fortnight’s summary in one line: Google finished the harshest spam update of the year, raised the bar on authorship and AI content on paper, broke the measurement layer twice, and watched OpenAI and its own Gemini team rewire what an answer looks like.

Next round up in two weeks. Comments welcome on recovery playbooks for the spam update, GPT 6’s effect on AI citations, or how to use the Bing Topics Portfolio properly.


Sources: Google Search Central (@googlesearchc, 8 October); Google Search Status Dashboard, checked 19 October 2026; Search Engine Roundtable (Barry Schwartz, 8, 13 and 16 October); Search Engine Land (crawl timing ranges reporting; AI Mode and AI Overviews changes, 15 October; Bing Topics Portfolio, 13 October; ChatGPT UTM story, 16 October); Search Engine Journal (Google timing ranges, 4 October); Search Engine Watch (unconfirmed October update, 16 to 19 October); 9to5Google (Gemini 4 Argon, 13 October); Unite.AI (GPT 6 Intelligent UI, 7 and 8 October); TechCrunch (OpenAI visual ads, 5 October); PPC Land (fake author profiles; Google Discover job listing); SearchHerald (October 2026 archive); DesignRush SEO Roundup (8 October week); Distribb (September 2026 spam update retrospective); AEOSoup (helpful content guidance updates); WebmasterWorld (September and October 2026 observations threads); Numinix (October 2026 SEO news); Nozzle (Poisoned Google SERPs research, 10 October); Marketing4eCommerce (OpenAI UTM bug analysis).

How to Crawl a Sub-Folder Only in Screaming Frog [2025] – example.com/example-sub-folder/

If You’re Folder URL Resolves with a 200 with a trailing slash /

For example – to crawl mysite.com/buyers-guides/

To crawl only URLs in the /buyers-guides/ folder on mysite.com using Screaming Frog, follow these steps:

  1. Open Screaming Frog SEO Spider and enter the URL: https://www.mysite.com/buyers-guides/
  2. Go to Configuration > Spider > Crawl
  3. **Under “Crawl Limits,” select “Limit Crawl Depth” and set it to 1 to crawl only the specified folder
  4. In the “Include” box, add this pattern: https://www.mysite.com/buyers-guides/.*
  5. Start the crawl by clicking the “Start” button

This configuration will ensure Screaming Frog only crawls URLs within the /buyers-guides/ folder on networldsports.co.uk, excluding other sections of the website.

**Note – if you are looking to crawl a country folder – e.g. https://www.apple.com/us/ – and the country-folder has URLs deeper than one folder down – e.g. https://www.apple.com/us/phones/i-phone999/
Then don’t add/set Crawl Limit to 1.


If You’re Folder URL Resolves with a 200 without a trailing slash /

https://www.mysite.com/buyers-guides

Enter the URL of the domain plus sub/folder into the main address bar on Screaming Frog.

Choose Sub-folder option to the right of the address bar:

Go to Configuration (top menu bar, to the left) >Spider > crawl > Include
– Add the sub-folder (without trailing slash) to the include section:

Click “Start”. (Button to the right of the “Subfolder” drop down).


Crawling a sub-folder – plus all internal & external links that are contained within the sub-folder pages.

The protocol above would only check the status codes of any URLs actually held within the /buyers-guides/ folder.

For example, if a football goal guide, links to the FA’s website, and 404s, the above methods would not pick this up (as the FA doesn’t have /buyers-guides/ in the homepage etc)

  1. Crawl and check all the URLs within a sub-folder e.g.

https://www.example.com/buyers-guides



2. get the status codes of any internal and external link, that point to outside the folder?



For example:


Our football goal guide –

https://www.example.com/buyers-guides/football-pitch-size-guide

Contains links that point outside of the buyers-guides folder, like to our product pages and external links to thefa.com etc.

Sub-Folder with Trailing Slash /buyers-guides/

Crawl https://www.example.com/buyers-guides/ with ‘Crawl Outside of Start Folder’ disabled, but with ‘Check Links OutSide of Start Folder’ enabled.

Sub-Folder with NO Trailing Slash /buyers-guides

Perform the crawl outlined above (using the include), get the list of https://www.example.com/buyers-guides URLs

then switch to list mode (Mode > List), go to ‘Config > Spider > Limits‘ and change the ‘Limit Crawl Depth’ from ‘0’ to ‘1” and upload and crawl the URLs

Remember to delete in include /buyers-guides from the crawl config before doing the above
i.e. Config>Spider>Include – remove anything in the box/field.

Scraping JSON Schema with Screaming Frog Custom Extraction [2025]

Last updated – a few days ago (probably).

If you go to Menus – configuration > Custom > Custom Extraction

Click the +Add button on the bottom right and choose “Regex” from the drop-down menu, which is to the right of “Extractor 1” text box.

Add the code below, in the box/field to the right of “Regex”.

<script type=\"application\/ld\+json\">(.*?)</script>

I’m using the code below, to extract product schema only – I can export to excel and filter the URLs containing product schema, but don’t have the aggregaterating:

<script type=\"application\/ld\+json\">(.*?"@type":\s*"Product".*?)<\/script>

Scrape Product Schema & Identify Missing Fields
(Product Schema missing aggregateRating field in this case)

  • Turned out easier to use this regex to identify all the URLs that have aggregateRating fields:
"aggregateRating":\s*\{[^}]+\}


  • and set up a second custom extraction to check for URLs/pages with any reviews –
"review":\s*\[\s*\{[^]]+\}

If the page had review schema, but not aggregateRating – then we needed to fix them.

scrape schema screaming frog

Screaming Frog – How to Save & Load a Custom Crawl Coniguration

Handy if you have loads of custom extractions set up etc.

  • Saving Custom Config File:

Under the Configuration Menu – go to – Profiles>Save As

  • Importing The Custom Config File

Configuration > Profiles > Load…

SF custom Config File Import
  • Save as default

    Also worth knowing – if you use a similar setup each time you run Screaming Frog – let’s say, you exclude anything with “account” or “customer” in the URL, you can save your setup/config as default:

Which can be a nice time saver too!

I pretty much always add the following to the excludes ‘file’, so it’s easier to keep it in the default setup:

^https?://[^/]+/customer/account/.*
^https?://[^/]+/checkout/cart/.*

^.\?.

.js$

.css$

Screaming Frog – Custom Extraction – Extract Specific Page Copy [2026]

Last Updated – a few days ago (probably)

  • Open Screaming Frog
  • Go to Configuration in the top menu
  • Custom > Custom Extraction
  • Use Inspect Element (right click on the copy and choose “inspect” if you use Chrome browser) – to identify the name, class or ID of the div or element the page copy is contained in:

    In this example the Div class is “prose” (f8ck knows why)

  • You can copy the Xpath instead – but it appears to do the same thing as just entering the class or id of the div:
  • The following will scrape any text in the div called “prose”:

*Click Image to enlarge^

Once you are in the Custom Extraction Window – Choose:

  • Extractor 1
  • X Path
  • In the next box enter –> //div[@class=’classofdiv‘] —->

    in this example – //div[@class=’prose’]
  • Extract Text

//div[@class='prose']
^Enter the above into the 3rd 'box' in the custom extraction window/tab.
Replace "prose" with the name of the div you want to scrape.


If you copy the Xpath using Inspect Element – select the exact element you want. For example, don’t select the Div that contains text you want to scrape – select the text itself:

Here are some more examples:

How to Extract Common HTML Elements

//div[@class='read-more']
XPathOutput
//h1Extract all H1 tags
//h3[1]Extract the first H3 tag
//h3[2]Extract the second H3 tag
//div/pExtract any <p> contained within a <div>
//div[@class=’author’]Extract any <div> with class “author” (remember to check ‘ quote marks are correct)
//p[@class=’bio’]Extract any <p> with class “bio”
//*[@class=’bio’]Extract any element with class “bio”
//ul/li[last()]Extract the last <li> in a <ul>
//ol[@class=’cat’]/li[1]Extract the first <li> in a <ol> with class “cat”
count(//h2)Count the number of H2’s (set extraction filter to “Function Value”)
//a[contains(.,’click here’)]Extract any link with anchor text containing “click here”
//a[starts-with(@title,’Written by’)]Extract any link with a title starting with “Written by”

 

How to Extract Common HTML Attributes

XPathOutput
//@hrefExtract all links
//a[starts-with(@href,’mailto’)]/@hrefExtract link that starts with “mailto” (email address)
//img/@srcExtract all image source URLs
//img[contains(@class,’aligncenter’)]/@srcExtract all image source URLs for images with the class name containing “aligncenter”
//link[@rel=’alternate’]Extract elements with the rel attribute set to “alternate”
//@hreflangExtract all hreflang values

 

How to Extract Meta Tags (including Open Graph and Twitter Cards)

I recommend setting the extraction filter to “Extract Inner HTML” for these ones.

Extract Meta Tags:

XPathOutput
//meta[@property=’article:published_time’]/@contentExtract the article publish date (commonly-found meta tag on WordPress websites)

Extract Open Graph:

XPathOutput
//meta[@property=’og:type’]/@contentExtract the Open Graph type object
//meta[@property=’og:image’]/@contentExtract the Open Graph featured image URL
//meta[@property=’og:updated_time’]/@contentExtract the Open Graph updated time

Extract Twitter Cards:

XPathOutput
//meta[@name=’twitter:card’]/@contentExtract the Twitter Card type
//meta[@name=’twitter:title’]/@contentExtract the Twitter Card title
//meta[@name=’twitter:site’]/@contentExtract the Twitter Card site object (Twitter handle)

How to Extract Schema Markup in Microdata Format

If it’s in JSON-LD format, then jump to the section on how to extract schema markup with regex.

Extract Schema Types:

XPathOutput
//*[@itemtype]/@itemtypeExtract all of the types of schema markup on a page

References:

Update:

If the ‘shorter code’ in the tables above doesn’t work for some reason, you may have to right click – inspect and copy the full Xpath code to be more specific with what you want to extract:

For sections of text like paragraphs and on page descriptions, select the actual text in the inspect window before copying the Xpath.

Update 2

We wanted to compare the copy and internal links before and after a site-migration to a new CMS.

To see the links in HTML format – you just need to check “Extract Text” to “Extract Inner HTML” in the final drop down:
(click image to enlarge)

On the new CMS, it was easier to just copy the XPath

Why Use Custom Extraction with Screaming Frog?

I’m glad you asked.

We used it to check that page copy had migrated properly to a new CMS.

We also extracted the HTML within the copy, to check the internal links were still present.

One cool thing you can do – is scrape reviews and then analyse the reviews to see key feedback/pain points that could inform superior design.

Here’s a good way to use custom extraction/search to find text that you want to use for anchor text for internal links:

custom search screaming frog

I’m still looking into how to analyse the reviews – but this tool is a good starting point: https://seoscout.com/tools/text-analyzer

Throw the reviews in and see what words are repeated etc

This tool is also very good:

https://voyant-tools.org

Or – just paste into Chat GPT and ask for insights and pain-points to help develop a better product.

I asked Chat GPT and Google Gemini why I would want to scrape copy from sites and got these responses:


Monitoring Competitors

Content Strategy Insights: Gather headings, titles, and meta descriptions from competitor websites to gain insights into their content strategies and how they optimize for search engines.

Pricing Information: For e-commerce platforms, you can use custom extraction to keep an eye on competitors’ pricing strategies by pulling prices for similar products.

Promotional Strategies: Collect details on promotions, discounts, or special offers from competitors’ sites to evaluate how they stack up against your own promotional efforts.

Keyword Usage: Identify specific keywords from competitors’ content to understand how they are targeting particular topics or niches.

Backlink Analysis: While Screaming Frog isn’t primarily a backlink tool, you can utilize custom extraction to pinpoint external links on a page, providing insights into a competitor’s backlink strategy.

Optimizing Your Own Website

Content Optimization: Employ custom extraction to review your own site’s headings, titles, meta descriptions, and keyword usage to spot areas that need improvement.

Structural Data: Extract and assess structured data (like Schema markup) across your site to ensure proper implementation and identify potential enhancement opportunities.

Internal Linking: Analyze internal linking structures to understand the flow of page authority throughout your site and to find orphan pages or those with insufficient internal link support.

Audit Forms and Calls to Action (CTAs): Extract form action URLs and CTA links to verify they are working correctly and to optimize their placement and wording.

Identify Outdated Content: Use custom extraction to locate dates and timestamps on your content, which can help you identify articles or pages that may require updates.

Gemini said:

Reasons to Use Custom Extraction in Screaming Frog:

The custom extraction feature in Screaming Frog provides numerous advantages for website analysis and data collection. Here are some important reasons to utilize it:

  1. Extract Specific Data:

Go beyond basic SEO metrics: While Screaming Frog offers valuable insights, custom extraction enables you to focus on specific data points that may not be easily accessible, such as product prices, article publish dates, or user reviews.

Gain deeper competitor insights: Extract information on competitor pricing strategies, product descriptions, or content structures to compare with your own website.

Gather information for internal analysis: Monitor specific elements on your site, like call to action (CTA) button text or internal linking structures, to observe changes and assess their impact.

  1. Automate Data Collection:

Save time and effort: Collecting data manually can be labor-intensive and prone to errors. Custom extraction streamlines the process, allowing you to efficiently gather information across numerous pages.

Maintain consistent data: By setting up automated data extraction, you ensure uniform data collection over time, which facilitates better trend analysis and comparisons.

  1. Enhance Reporting and Analysis:

Combine extracted data with existing Screaming Frog metrics: Merge the extracted data with other SEO parameters such as page titles, meta descriptions, and internal links for a more thorough analysis.

Create custom reports: Use the extracted data to generate tailored reports for specific purposes, like competitor pricing comparisons or evaluations of content performance.

Monitoring Competitors:

Custom extraction serves as a valuable tool for competitor monitoring in various ways:

Extract competitor pricing data: Keep track of competitor pricing trends, identify potential gaps in your own pricing strategy, and make informed pricing decisions.

Analyze competitor content structure and keywords: Learn how competitors format their content, pinpoint their targeted keywords, and gain insights to enhance your own strategy.

Note for self – for Magento 2, Hyva theme Sub-category page copy – scrape using:

//div[@id='descriptionDiv']
Product page descriptions upper and lower divs -
//div[@class="product-description"]
//*[@id="specifications"]/div/div[2]/div/div/div/div/div
//*[@id="description"]/div/div[2]

For personal reference –

//div[@class=’product-description’]

//div[@class=’prose’]

/html/body/div[1]/div[3]/main/div[4]/div/div

//div[@class=’category-description read-more’]

Screaming Frog Custom Extraction

Configuration → Custom → Extraction → Add (one per row). All extract as Extract Text unless noted.

Priority extractors (most likely to contain untranslated English)

NameTypeSelectorExtractor
Meta DescriptionXPath//meta[@name='description']/@contentExtract Text
Meta KeywordsXPath//meta[@name='keywords']/@contentExtract Text
OG TitleXPath//meta[@property='og:title']/@contentExtract Text
OG DescriptionXPath//meta[@property='og:description']/@contentExtract Text
H1 FullCSSPathh1Extract Text
All H2sXPath//h2Extract Text
All H3sXPath//h3Extract Text
Product Description BodyCSSPath.product-descriptionExtract Text
USP / Special Feature BlocksCSSPath.special-feature-itemExtract Text
Review TitlesCSSPath[itemprop='name']Extract Text
Review BodiesCSSPath[itemprop='reviewBody']Extract Text
BreadcrumbsCSSPathnav.breadcrumbsExtract Text
Image AltsXPath//img/@altExtract Text
Image TitlesXPath//img/@titleExtract Text
Schema.org descriptionXPath//script[@type='application/ld+json']Extract Inner HTML

English-word detection (fires only if it finds English)

These use XPath’s contains() to return a value only when specific English words appear in body text. Handy for flagging pages fast:

NameTypeSelector
Has “the “XPath//body[contains(., ' the ')]/@class
Has “and “XPath//body[contains(., ' and ')]/@class
Has “with “XPath//body[contains(., ' with ')]/@class
Has “made “XPath//body[contains(., ' made ')]/@class
Has “quality “XPath//body[contains(., ' quality ')]/@class
Has “features “XPath//body[contains(., ' features ')]/@class
Has “delivery “XPath//body[contains(., ' delivery ')]/@class

Creating an Actionable 404 Report from Screaming Frog

Update – I don’t think all the process below is required.

Just download the 404 – inlinks report from Screaming Frog

Bulk Export (very top, slightly to the left on the GUI)> Response Codes > Internal > Client Error 4**s

Copy the “Destination” (column C on report) column and paste into an new Excel tab/sheet and remove duplicates

In the first sheet, copy and paste the source column into column D

In the second sheet, do a vlookup using the first destination URL, and “lookup” in the first sheet – columns C and D, to return the relevant source URL

Copy the vlookup and Paste – values – into column A into the second sheet

You can also copy and paste the anchor text and location into column C

Follow this protocol, to produce a sheet you can send to devs etc, to remove 404s

  • This will get rid of the site-wide 404s and some individual 404s

Run a crawl with Screaming Frog

Export the report –> Screaming Frog – Bulk Export  – Response Codes – Internal – Internal Client Error (4xxs)  (check 500s too)

In Excel – Copy and paste the “destination” URLs into a new sheet – into column A

Remove duplicates from the destination URLs that you’ve just copied into a new sheet

rename the colum – 404s Destination

  • Copy and paste the Source URLs and the Anchor Text into a new sheet.

Paste Source URLs in column A, and Anchor Text into column C

In cell B1 type – ” | ”

In cell D1 – give the column the heading “Source | Anchor”

In cell D2 concatenate – =CONCATENATE(A2,$B$1,C2)

Drag the formula down.

You’ll now have the anchor text and the source URL together, so you can vlookup the destination (404) URL

  • create a new sheet
  • Copy and paste all of the source URLs | Anchor Text (from the concatentate formula – paste special -values only
  • Copy & Paste Destination URLs from the original sheet into columns B and C in the new sheet you just made.


You need “destination” in column B and “Source | Anchor Text” in column C, as vlookup has to go left to right

  • So you’ll have – 404s Destination – Destination – Source | Anchor Text

Name column D in the new sheet “Example Source URL & Anchor Text” and in cell D2 enter the lookup – VLOOKUP(B2,B:C,2,0) (put “equals sign” before the V. Drag the formula down

Copy column A and paste into a new sheet. Name the sheet “Final”.

Copy column D with the vlookup and paste values into column B in the “Final Spreadsheet”

In “final”, you should now have all the unique 404s and an example of a page that links to those 404s with the anchor text.

  • You can use “text to columns” to seperate the source URLS and anchor text if you wish
  • If you’re sending the spreadsheet onto a dev or someone to fix the 404s, you are probably best sending the full sheet with all the inlinks to the 404s, plus the one you’ve just made. It depends how they go about fixing the 404s.

    Once 404s have been fixed, rerun a crawl and recheck them.

look out for 404s that are classed as  HTTP Redirects in the “type” column – these don’t seem to have a unique source URL. You may have to search for the URL in the search box in Screaming Frog and click the “inlinks” tab to see original link to the non-secure http page

If you like, before you send off the report to someone, you can double check the “destination” URLs definitely are 404s, by pasting them into screaming frog in “list” mode

Finding & Fixing 404s with Screaming Frog [2026] – Site-Wide 404s

Export the 404 Inlinks Report into Excel

The best report in Screaming Frog to see the source and destination of all 404s – is to go to Bulk Export at the top menu:

And then Response Codes – Client Error Inlinks

In the Exported Excel Sheet:

Copy the “destination” column

Paste into a new sheet/tab – In Column B

Remove duplicates

Back in the first sheet – Paste “Source”, into column D to the right of “destination” in first tab/sheet


In the second sheet –
Do a vlookup in the second sheet/tab – to import a source URL to each unique “destination” URL
Add the word “source” to cell C1 in the second sheet

I think you have to click the file name to download the example sheet below:

Sheet / Tab 1 should look like this:

The Second Sheet / Tab should look like this:

The Lookup Value in Cell C2, in the sheet above is:

=VLOOKUP(Sheet1!B2,’1 – Client Error (4xx) Inlinks’!C:D,2,0)

Just double click / drag it down to complete the lookup

You can add more lookups and columns to provide more details about the link location.

Just copy and paste values on the last Vlookup.

Paste another column like “Anchor” into column D on the first Tab

Paste the vlookup into the adjacent column

“0” means nothing found – i.e. no anchor text etc

Update –

404s found in the Navigation or “Aside” tend to be site-wide, so you can find one instance of this and update the 404 and Bob’s your uncle.

If a 404 is within the page content/copy however, they are possibly unique and need fixing individually. So take a look the Content 404s separately. They probably only have 1 source URL so no need to faff with vlookups etc.

Bulk checking Canonical URLs & Hreflang Tags for SEO on an eCommerce Store (With Screaming Frog)

TLDR – Easiest way to check hreflang is to just go to reports > Hreflang > non200 Hreflang URLs:

Filter “Source” column to your current domain and then you should see all the hreflang that 300 or 404.

  • However – you ideally want to get all of your indexable, 200 status code URLs – to have the relevant hreflang tags.
  • So first find all the indexable, 200, URLs with exact-match canonicals
  • Then check the hreflang on these only
  • Get all the URLs that 200 and are indexable from “internal_all” report
  • Paste in list mode, all of these URLs – check hreflang tab
  • Perform a Crawl Analysis
  • Check The overview panel

hreflang screaming Frog
  • Order the Hreflang tab by occurrences to double check tall URLs have at least 1 hreflang

Checklist here – https://docs.google.com/spreadsheets/d/1IHRZNFravVKaZ5T9uOEOg_zQ5ee8dHcRP8m92vkn4KY/edit#gid=0

Configuration – spider – crawl and store Hreflang

Find URLs that should have a hreflang tag – Indexable URLs

  • Crawl the website with Screaming Frog – in configuration crawl and store Hreflang
  • Download the “internal” report on first tab
  • Filter relevant columns to find all URLs that return 200 (Status Code), are indexable (Indexability)

Check Canonical URLs are Correct and Indexable

  • From the internal Report – filter status code to 200 only and Indexable
  • Paste the filtered columns into a new tab/sheet and delete irrelevant columns so you just have URL and Canonical columns
  • Use =Exact formula to find all URLs that match the canonical URL – add an extra column for this and call it “exact?”
  • For canonical checks – filter to “FALSE” and see which URLs don’t have self ref canonicals

Back in Internal – Original Spreadsheet

  • unfilter the internal report.
  • Filter to show all the canonical URLs. Canonical Column – filter – untick “blanks”
  • Copy and paste into Screaming Frog , list mode – check status code of all these canonical URLs

  • For Hreflang – Filter Canonical URL – “exact?” column to “true”
  • You should be left with all URLs that are indexable (and should therefore have hreflang if an international site) – i.e. 200 status codes and URLs that exactly match canonical URLs.

Hreflang Checks

For the indexable URLs found above, check hreflang using:

https://technicalseo.com/tools/hreflang/

  • Go to screaming frog GUI – and the Hreflang Window

Order by Occurrences – check if any URLs have less hreflang then they should have

  • Check all the reports in the drop down menu on the left

Non-200 Hreflang URLs – URLs within the rel=”alternate” markup that don’t result in a 200 status code

Download the non-200 hreflang report – by going to the “hreflang” tab in Screaming Frog.

Delete all the columns except those containing the actual hreflang URLs

Then in the column adjacent to all the hreflang URLs add the formula:
=INDEX($A$1:$G$126,CEILING(ROW()/7, 1),MOD(ROW()-1,7)+1)

The formula above would be used if the hreflang URLs are in columns A to G, and the last row that is populated in number 126.
You’ll also need to update the number 7 in the formula – for example if you have 5 columns of hreflang URLs, then change 7 to 5.

Drag this formula down until you start seeing #REF! errors, indicating you’ve covered all the data in your original columns

  • You should now have all the hreflang URLs listed in the non-200 hreflang report
  • To find the specific URLs that are non-200 – paste the last column containing all the URLs, into screaming frog in “List” mode

Export the List mode ‘crawl’ from Screaming Frog and filter by status code

  • Unlinked Hreflang URLS – Page sthat conain one or more hrelgang tag / URL that’s only linked to by a hreflang tag and not in the actual webpages

  • Missing Return Links – Hreflang should be recipricol.
  • Inconsistent Language & Region Return Links – This filter includes URLs with inconsistent language and regional return links to them. This is where a return link has a different language or regional value than the URL is referencing itself
  • Non Canonical Return Links – URLs with non canonical hreflang return links. Hreflang should only include canonical versions of URLs.
  • Noindex Return Links – Return links which have a ‘noindex’ meta tag. All pages within a set should be indexable, 
  • Incorrect Language & Region Codes – This simply verifies the language (in ISO 639-1 format) and optional regional (in ISO 3166-1 Alpha 2 format) code values are valid
  • Missing Self Reference – URLs missing their own self referencing rel=”alternate” hreflang annotation. It was previously a requirement to have a self-referencing hreflang, but Google has updated their guidelines to say this is optional. It is however good practice and often easier to include a self referencing attribute.
  • Not Using Canonical – URLs not using the canonical URL on the page, in it’s own hreflang annotation. Hreflang should only include canonical versions of URLs.
  • Missing – URLs missing an hreflang attribute completely. These might be valid of course, if there aren’t multiple versions of a page.
  • Outside <head> – Pages with an hreflang link element that is outside of the head element in the HTML. The hreflang link element should be within the head element, or search engines will ignore it.

Official documentation from Screaming Frog here.

You can also use this Google sheet to check hreflang further