SEO & Google Ads/PPC: News – Sept 2026

On September 7, 2026September 7, 2026 By DaveIn Business, seoLeave a comment

SEO & Digital Marketing News Round-Up: What UK Small Businesses Need to Action Right Now (Late August / Early September 2026)

If you’ve blinked in the last three weeks, you’ve missed a lot. Here’s the plain-English rundown of what actually changed, what it means for your business, and more importantly: what you should actually do about it.

Laptop screen showing digital marketing analytics dashboard with charts and data — *Photo by Unsplash. Swap for your own or upload to your Media Library.*

TL;DR The Five Things That Actually Matter This Month

If you only read one thing, read this table. Everything else is context.

#	The Change	Deadline	Who It Affects	Urgency
1	Google Ads AI Max auto-migration	1 Sept 2026	Anyone running Google Search Ads with broad match or auto-created assets	🔴 Critical
2	Google Performance Max AI video auto-resize	4 Sept 2026 (opt-out)	Anyone running PMax with video assets	🔴 Critical
3	Cloudflare AI crawler default changes	15 Sept 2026	Anyone hosting a site on Cloudflare (especially free-tier)	🟠 High
4	Google August 2026 Spam Update (finished rolling out)	Complete – 21 Aug 2026	Sites relying heavily on AI/programmatic content	🟠 High
5	Bing Webmaster Tools old API retired	31 Aug 2026	Anyone using dashboards/tools that pull Bing data	🟡 Medium

1. Google Ads AI Max: The Silent Migration You Might Have Missed

This is the single biggest paid-media change of the year, and Google has been remarkably quiet about it.

From 1 September 2026, Google is automatically migrating Search campaigns using either “Automatically Created Assets” (ACA) or the campaign-level broad match setting over to its new AI Max system. There is no advance decline – you either configured it yourself before the deadline or Google configures it for you.

What actually changes

When AI Max flips on, Google’s AI can:

Rewrite your ad text – new headlines and descriptions generated based on your website and existing ads (words you didn’t write and didn’t approve)
Match queries beyond your keyword list – using “search term matching” similar to how Performance Max already works
Swap landing pages – potentially pointing traffic to different pages on your site

Your brand inclusions/exclusions, budget and bid strategy do carry over.

The uncomfortable data

Google says AI Max delivers a +7% conversion lift. Independent testing tells a different story:

Source	Finding
Google’s own aggregate data	+7% conversions at similar CPA (excludes retail)
Smarter Ecommerce (250+ retail campaigns)	Median 13% increase in conversion value, 0% median ROAS change
Independent test of 30,000 search terms	99% of AI Max impressions produced zero conversions
Lunio (nine-month study)	Invalid-traffic rates rose from 2.46% to 5.28%

Not exactly a slam dunk.

Your action list

☐ Before 1 September: Log into every Google Ads account and audit whether campaigns use ACA or campaign-level broad match
☐ Decide per campaign whether to opt in deliberately (Campaigns → Settings → AI Max) or disable the legacy feature first
☐ Warning: Once you disable the legacy ACA feature, you can’t turn it back on. AI Max itself is reversible; ACA disablement is a one-way door.
☐ Export current search term reports and conversion data now so you’ve got a clean before/after baseline
☐ Audit “Limited by budget” campaigns – Google explicitly said it won’t adjust bid targets for you

Regulated industries (health, finance, legal): AI-rewritten ad copy could breach your compliance rules. Take this seriously.

2. Performance Max: Google’s AI Will Reframe Your Videos

If you didn’t already have enough to worry about, there’s a separate PMax change with an even earlier deadline.

From 4 September 2026, Google will use generative AI to auto-create missing aspect ratios (vertical, square) from your existing horizontal video assets in Performance Max campaigns. This applies to video you’ve already uploaded and approved, not just new uploads.

Why this matters

A 16:9 original cut down to 9:16 vertical keeps roughly a third of the width. That means:

Burnt-in text overlays likely disappear
Logos at the edges get cropped out
Legally required disclaimers can vanish
Generative AI invents the rest

How to opt out

Contact your Google account team or submit Google’s opt-out form before 4 September. Video settings can still be changed inside your account after that date, but the default from that point is “on”.

If your PMax video has your premises, staff, or actual product on screen – look at what the AI produces before letting it run in your name.

3. Google’s August 2026 Spam Update: What Got Hit

Google’s third spam update of the year ran from 18–21 August 2026. It rolled out globally in every language over about two days and 16 hours.

What Google said

No new spam policies were added
It re-scores sites against existing rules
Link spam and site reputation abuse were explicitly excluded from scope
Recovery is measured in months, not weeks

What actually got hit

According to SE Ranking’s study of 100,000 keywords across 20 industries:

16.71% of URLs that had ranked in Google’s top 10 fell beyond position 100 during the update
That’s roughly 1.8 times more than a comparable baseline period
Common patterns among impacted sites: programmatic content at scale, AI-generated content published without meaningful human review, thin affiliate pages, near-duplicate content hubs

What to do if you were hit

Priority	Action
1	Wait until rollout completed (it has – 21 Aug) before making changes
2	Check Search Console dates. Line up traffic drops with 18–21 August
3	Audit pages with low engagement, near-duplicate content, or high AI-probability scores
4	Consolidate or remove thin content rather than mass-rewriting good pages
5	Don’t file a reconsideration request, those are for manual actions, which this isn’t

One thing NOT to do: Don’t panic-rewrite perfectly good pages because an AI detector scored them highly. Detection tools are noisy and rewriting well-performing content in a panic is how you make things worse.

4. Cloudflare AI Crawler Changes: 15 September Deadline

This is the one everyone’s got slightly wrong. Let’s clear it up.

From 15 September 2026, Cloudflare’s new defaults kick in for:

New domains being onboarded
New sites added by existing customers
Existing free-tier accounts that haven’t touched their settings

On pages that show ads, Training and Agent crawlers will be blocked by default. Search crawlers stay allowed.

The actual risk (this is important)

Multi-purpose crawlers like Googlebot, Applebot and Bingbot crawl for both Search AND Training under a single user agent. Cloudflare’s new rule is: the most restrictive setting wins.

Translation: If someone on your team ticked the old “Block AI Bots” toggle in 2025, from 15 September you could accidentally be blocking Googlebot too, which would quietly disappear you from Google Search.

Your ten-minute check

Log into Cloudflare → Security → Settings → AI bot policies
If you see the old blanket “Block AI Bots” toggle, that’s your risk. Change it to the granular Search / Agent / Training controls
Allow Search crawlers unless you have a very specific reason not to
Make a deliberate choice on Training (are you happy for your content to train AI models?) and Agent (do you want ChatGPT, Perplexity etc to fetch your pages when users ask?)
Save. Even saving your current choice explicitly protects you from the new defaults being applied

If your growth loop includes getting cited in AI answers, blocking Training and Agent bots is an SEO own-goal. Most small businesses should probably allow all three.

5. ChatGPT Ads: Now Live in the UK (and 30 Other European Markets)

OpenAI has quietly turned ChatGPT into a self-serve ad platform, and as of 31 August 2026, self-service Ads Manager access is now live across 31 European markets, with the UK already included in the earlier launch cohort.

The state of play

Fact	Detail
Where ads appear	Free and Go tier users, logged in, adult accounts
UK availability	Live (self-serve Ads Manager)
Bidding	oCPC (conversion-optimised) now available
Formats	Sponsored product cards, multi-product carousel (new in August)
Partners	Etsy, Shopify integrations for in-chat checkout
Early UK/US CPM	Around $30 (down from initial $60 estimate)

Should you actually spend on this?

Honestly? Not yet, for most small UK businesses. Here’s why:

Ads only reach Free and Go tier users (Plus and Pro users see none)
Independent tests have been mixed at best – one agency test recorded 8,940 impressions, 58 clicks and zero leads
The channel behaves more like brand awareness than direct response
Measurement stack is still immature

But: get your OAIQ pixel installed and Conversions API wired up now, so you’re ready to test cheaply in Q1 2027 when the data will be clearer.

6. Quick-Fire Round-Up: The Other Stuff That Happened

Update	What Happened	What To Do
Bing Webmaster Tools API	SOAP/POX interfaces retired 31 Aug 2026	Migrate any dashboards to REST/JSON API
Google Search Console	Third-party views for IG, YouTube, TikTok now showing	Check your GSC – new data source for social attribution
GA4 & Google Ads	AI Overviews and “Ask Advisor” rolling out for benchmarking	Explore the new prompts for quick insights
Google Assistant	Being replaced by Gemini on Android from 4 Sept	Test how your business appears in Gemini responses
Reddit citations in ChatGPT	86% drop reported by Promptwatch in mid-August	Diversify – Reddit was a huge citation source for many niches
EU AI Act	Watermarking rules for AI content now in force	If you use AI-generated marketing content in EU markets, comply with disclosure rules
DaVinci Resolve, CapCut, Premiere Pro	All shipped major AI-powered updates	Genuine time-savers for video content teams

The Bigger Picture: Three Themes Worth Internalising

1. “Doing nothing” is now an active choice

Google, OpenAI and Cloudflare are all shipping changes with defaults that apply if you don’t act. Small businesses without dedicated marketing ops are the most exposed. Put September deadlines in your diary this week.

2. AI search rewards the same things good SEO always has

Generative Engine Optimisation (GEO) – getting cited by ChatGPT, Claude, Google’s AI Overviews and Perplexity – rewards clear answers, structured content, real expertise and trustworthy brand entities. The tactics haven’t changed as much as the terminology.

Data point worth knowing: SE Ranking found AI-referred visitors spend around 68% more time on site, and Semrush has reported they convert roughly 4.4 times higher than standard organic. AI referrals are still a trickle – but they’re a trickle of pre-qualified people.

3. Your Google Business Profile decays faster than you think

Recent local SEO data suggests GBP impressions drop noticeably for businesses that haven’t posted an update or added a photo in over 30 days. If you haven’t touched yours in a while, that’s a 10-minute Sunday-evening job.

Your September Action Checklist

Screenshot this, save it, tick it off.

Before 1 September:

☐ Audit Google Ads accounts for ACA and campaign-level broad match usage
☐ Export current search term reports as your baseline

Before 4 September:

☐ Decide on Performance Max video auto-resize opt-out
☐ Contact Google account team or submit opt-out form if needed

Before 15 September:

☐ Log into Cloudflare and set explicit AI bot preferences (don’t leave on defaults)
☐ Confirm Googlebot is not accidentally blocked

By end of September:

☐ Review Search Console for August spam update impact
☐ Post fresh content to your Google Business Profile
☐ Test how your business appears in ChatGPT, Claude and Google AI Overviews
☐ Audit your service pages for clear, extractable answers to buyer questions

Final Thought

Most of the damage in a month like this comes from the reaction rather than the news itself. Panic-rewriting content, changing settings without measuring, buying tools that “fix” problems you don’t have. Check your data against Search Console before you change anything. Put the September dates in your diary. Then get back to running your business.

The businesses that win in AI-era search aren’t the ones with the biggest content machines. They’re the ones with clear money pages, real expertise, strong local signals, and settings they’ve actively chosen rather than defaults they’ve inherited.

Any questions on any of this or want help auditing your setup ahead of the September deadlines? Drop a comment below.

Sources for this round-up include Search Engine Roundtable, Search Engine Land, Search Engine Journal, Google’s own Ads and Commerce Blog, OpenAI, Cloudflare, SE Ranking, Yoast, and reporting from Two Octobers, Seafoam, ROI Revolution, Ignite Visibility and Digital Loop. Snapshot as of early September 2026 – this stuff moves fast, always sense-check dates before acting.

Using Command Prompt & CURL for SEO Testing [2026]

On May 26, 2026 By DaveIn Business, seoLeave a comment

Why Every SEO Should Learn Curl (And How to Start in 5 Minutes)

Most SEOs rely on Screaming Frog, Sitebulb, or Chrome DevTools to check how a website is performing technically. All brilliant tools, but there’s something faster, free, and already installed on your computer: a tiny program called curl.

First, what’s a “terminal”?

Before computers had pretty desktops and clickable icons, you controlled them by typing instructions. That text-based interface is called a terminal, or on Windows, Command Prompt. Think of it as texting your computer directly: you type a command, it does the thing, and prints text back at you.

It’s still around today because for certain technical jobs, typing one line is faster than clicking through ten menus.

What is curl?

curl stands for “Client URL.” It’s a built-in tool on Windows, Mac, and Linux that fetches webpages without rendering them. Instead of seeing the pretty version your browser shows you, you see the raw response the server sends back. Exactly what Googlebot sees when it crawls your site.

That’s the gold for SEO. Your browser hides things; curl shows everything.

Your first curl check in 4 steps

Open Command Prompt. Press the Windows key, type cmd, hit Enter.
Type the command: curl -I https://yourdomain.com (the capital I means “headers only”).
Hit Enter. The server’s response headers print out instantly.
Read the results. Status code is on line one — 200 means OK, 301 means redirect, 404 means missing.

That’s it. You’ve just made the same request Google makes to your site thousands of times a day.

Why this matters for SEO

A single curl check reveals things a browser hides: status codes (is the page actually returning 200?), caching rules (is the CDN configured properly?), security headers (HSTS, X-Frame-Options), server fingerprints (what platform is the site built on?), and redirect chains (are there unnecessary hops slowing down crawling?).

It’s also the fastest way to spot staging environments accidentally leaking into Google’s index, broken canonicals, or compression that’s silently switched off.

Save the full response for deeper analysis

Want the entire HTML page too? Use this:

curl -i --compressed https://yourdomain.com > response.txt

That saves the complete response, headers and HTML- to a text file you can open in VS Code or Notepad++. Brilliant for auditing canonicals, hreflang, structured data, and meta tags in seconds.

The takeaway

Curl is the closest thing you’ll get to seeing your site through Google’s eyes. Five minutes to learn, a lifetime of faster audits. Open Command Prompt today and try it on your homepage, you’ll be amazed what it tells you.

It’s a way of pulling info from websites, without a browser. I think

Website Status Codes – All of them in a table [2025]

On January 28, 2026 By DaveIn UncategorizedLeave a comment

Here you go –

most common – 404 – Not found, 301 – redirect (permanently moved the webpage to this new one)

How Google Bot Crawls JavaScript [2025]

On November 17, 2025November 17, 2025 By DaveIn seoLeave a comment

Ever wondered how Google turns your lovingly handcrafted website into a ranking somewhere below a Reddit thread from 2013? It’s not magic, it’s just a long queue of tiny robot librarians fetching HTML, executing JavaScript, and occasionally having nervous breakdowns when they hit your React app.

This is the life cycle of a webpage inside Google’s digestive system: crawl, render, index, panic. Let’s go step by step before your sitemap starts crying.

1. Crawling: getting the raw HTML

1.1 URL discovery & crawl queue

Googlebot first has to discover your URLs. That can happen via:

Links from other pages
XML sitemaps
Manual submit / “Inspect URL → Request indexing” in Search Console
Other Google systems (e.g. feeds, previous crawls)

Discovered URLs go into a crawl queue with priority based on things like page importance and your site’s crawl budget.

1.2 robots.txt and basic checks

Before requesting the URL, Googlebot:

Fetches robots.txt
Checks if the URL (and key resources like JS/CSS) are allowed
Applies host load limits and crawl budget rules

If the page or important JS/CSS files are blocked in robots.txt, Google:

Won’t crawl them
Won’t be able to fully render your JS content later

Practical implication: Never block /js/, /static/, /assets/, etc. in robots.txt.

1.3 Fetching the HTML (“first wave”)

Googlebot makes a normal HTTP request (like a browser without UI):

Gets the initial HTML (without having run JS yet)
Parses head tags (title, meta description, canonical, meta robots, hreflang, etc.)
Extracts links from the HTML and adds them to the crawl queue
Notes references to resources (JS, CSS, images)

At this stage, only what’s in the raw HTML is visible. If your content is 100% client-side rendered (React, Vue, etc.), Google might see almost nothing yet.

Google can sometimes do basic indexing directly from the HTML (e.g. if content is already there), but JS-heavy pages need the next phase.

2. Rendering: running your JavaScript

Google describes the JS pipeline as: Crawling → Rendering → Indexing. Rendering happens in a separate system using an evergreen version of Chromium (a headless Chrome kept relatively up-to-date) called the Web Rendering Service.

2.1 The render queue (“second wave”)

After the initial crawl:

Google adds the page to a render queue.
When resources allow, that queue feeds URLs into the rendering system.
Until rendering happens, Google only “knows” what was in the raw HTML.

This is why people talk about “two waves of indexing” for JavaScript:

Wave 1: Index from HTML (if possible)
Wave 2: Index updated content after JS has run

Modern research suggests the process is smoother and faster than years ago, but there is still a render queue and potential delay for JS content.

2.2 How Google’s renderer behaves

When a page reaches the renderer:

Google loads it in an evergreen Chromium environment (no UI).
It fetches JS, CSS, and other resources (subject to robots.txt, CORS, etc.).
It executes JavaScript for a limited amount of time (shorter than a user session).
JS can:
- Modify the DOM
- Inject content
- Fetch JSON/XHR/data and add it to the page
- Add structured data (application/ld+json) dynamically

Important constraints (from Google’s docs & tests):

No user interactions: Google doesn’t click, type, or scroll like a user.
Time limits: Long chains of async calls may never complete before the renderer stops.
Resource limits: Heavily blocking scripts or endless network calls can break rendering.
“Noindex = no render” effect: If a page is noindex, Google generally won’t bother rendering it.

2.3 Post-render snapshot

Once JS finishes (or time runs out), Google:

Takes the final DOM snapshot (what a user would see after JS).
Extracts:
- Visible text content
- Links added by JS (e.g. SPA navigation)
- Structured data present in the DOM
- Meta tags if they are changed or added by JS

This rendered snapshot is what feeds into the real indexing stage.

3. Indexing: storing & scoring the rendered content

With the rendered HTML/DOM in hand, Google moves to indexing.

3.1 Understanding the content

From the rendered DOM, Google:

Tokenizes and stores text (words, phrases, headings).
Maps entities, topics, and relationships.
Reads links (anchor text + target URL) added by your JS navigation.
Parses structured data (schema.org, etc.) that JS may have injected.

This is the version of the page that can now rank for queries matching that content.

3.2 Canonicals, duplicates & signals

Indexing also handles:

Canonical selection (HTML tags, redirects, link signals).
Duplicate / near-duplicate detection, especially if JS rewrites similar pages.
Applying meta robots and HTTP headers from the final state after JS (for most cases).

If Google decides another URL is the canonical, your rendered JS content might be stored but not shown as the main result.

3.3 Final result: searchable document

After indexing, the document is:

Stored in Google’s index with:
- Content (from rendered DOM)
- Links
- Structured data
- Various quality & relevance signals
Ready to be retrieved and ranked when a user searches for related queries.

4. Where JavaScript sites usually break this flow

Because JS adds extra moving parts, a bunch of things can go wrong between crawl → render → index:

Blocked JS/CSS in robots.txt
Google can’t render layout or content if the files are disallowed.
Content only after interaction
If key text appears only after a click/scroll or in a modal that never opens, Google won’t see it.
Too slow or broken rendering
Heavy JS, long waterfalls, or failing XHR calls mean the DOM never contains the content when Google takes the snapshot.
Infinite scroll / SPA routing without proper URLs
If content is loaded endlessly on one URL without crawlable links or pagination (e.g. no ?page=2, no anchor links), Googlebot may only see the initial “page”.
Client-side only structured data that doesn’t materialise in time
If JS injects JSON-LD but too slowly (or fails), rich results won’t trigger.

5. How to see what Google sees (JS-specific)

To understand how your JS is being processed:

Use URL Inspection → “View crawled page” & “Screenshot” in Search Console to see the rendered DOM.
Compare “HTML” vs “Rendered HTML” to spot what content only appears post-JS.
Use “Test live URL” if you suspect render-queue delay.
Check Coverage / Pages report for “Crawled – currently not indexed” patterns that often show render/index issues.

So there you have it — from lazy bots fetching half your HTML to a headless Chrome pretending to be a real user for 0.3 seconds. Somewhere in that chaos, your content might actually get indexed.

If your JavaScript site isn’t showing up, don’t blame Google just yet — try unblocking your own files and giving the crawler half a chance. Think of it as SEO mindfulness: eliminate obstacles, breathe deeply, and let the bots eat your content in peace.

Explained in simpler terms – How Googlebot Crawls Javascript –

Stage 1 – Discovery & Crawling: “Finding your page and grabbing the first copy”

1. Google finds your URL

Google finds pages from things you already know:

Links on other sites
Your internal links
Your XML sitemap
Stuff you submit in Search Console

It puts those URLs in a big to-do list (crawl queue).

2. robots.txt check

Before visiting a URL, Google checks your robots.txt file:

If the page or important files (JS/CSS) are blocked, Google is basically told: “Don’t look here.”
If they’re allowed, it moves on.

Simple rule for you:
Never block your JS/CSS folders in robots.txt.

3. Google downloads the HTML (Wave 1)

Google now requests the page, just like a browser:

It gets the basic HTML (before any JavaScript runs).
From that HTML it grabs:
- Title, meta description, canonical, meta robots, etc.
- Any plain text that’s already there
- Links to other pages
- Links to JS/CSS/images

At this point, Google has not run your JavaScript yet.

If your important content is already in the HTML (e.g. server-side rendered), Google can often index it right away from this “first wave”.

Stage 2 – Rendering: “Actually running your JavaScript”

Now Google needs to know what your page looks like after JS runs – like a real user would see it.

Because this is heavy work, Google doesn’t do it instantly for every URL.

4. Render queue (waiting line)

After the first crawl, JavaScript pages go into a render queue:

Think of it like: “We’ve saved the HTML. When we have time, we’ll come back and run the JS.”

So for a while, Google might only know the bare HTML version of your page.

5. Headless Chrome renders the page

When your page reaches the front of the queue, Google loads it in something like Chrome without a screen (headless browser).

This browser:

Downloads JS/CSS (if not blocked)
Executes the JS for a short amount of time
Lets JS:
- Change the DOM (the page structure)
- Insert more text
- Add more links
- Inject structured data (JSON-LD)

Then it takes a snapshot of the final page – the “after JS” version.

This is basically:

“What a user would see if they opened your page and waited a bit.”

6. Things that can go wrong here

This is where JS sites often break:

Blocked JS/CSS → Google can’t see the layout or content properly.
Very slow JS → content appears after Google stops waiting.
Content only after a click/scroll → Google doesn’t usually click buttons or scroll like a human.
Broken scripts / errors → content never appears at all.

Result: Google’s snapshot may miss your main content.

Stage 3 – Indexing: “Filing your page in the library”

Now Google has:

Version 1: HTML-only (first wave)
Version 2: Rendered DOM (after JS runs)

7. Understanding the rendered page

From the rendered snapshot Google:

Reads all the visible text
Sees headings and structure
Follows any extra links added by JS
Reads structured data (schema)
Applies canonical tags / meta robots, etc.

This updated information is used to update your page in the index (second wave).

8. Search results

When someone searches:

Google looks through its index of pages (which contains that rendered version).
It decides which pages are most relevant.
It shows your URL in the results.
When the user clicks it, they go to your live site, not to Google’s stored copy.

Quick “JS SEO” checklist for you

If you remember nothing else, remember these:

Can I see my main content in the raw HTML?
- If yes → you’re mostly safe (e.g. SSR or hybrid).
- If no → you’re relying heavily on rendering; be extra careful.
Are JS & CSS allowed in robots.txt?
- They should be.
Does important content require a click/scroll?
- Try to have key text and links visible without interaction, or use proper URLs for loaded content.
Is the page reasonably fast?
- If it takes ages for content to appear, Google may bail before seeing it.
Use Search Console’s URL Inspection → “View crawled page”
- Compare:
  - HTML Google saw
  - Rendered HTML
- If you don’t see your text in the rendered version → that’s a problem.

Checking product Schema On a Raspberry Pi

On October 29, 2025October 29, 2025 By DaveIn seoLeave a comment

Goal of the mini-project

The aim here is to –

Verify that product schema (JSON-LD) is implemented correctly on example.co.uk after the migration to Adobe Commerce (Magento).
The script crawls your chosen product URLs and reports if required fields like price, brand, sku, and availability are present.

Step 1 – Open a terminal

Click the black terminal icon on the Pi desktop.

Step 2 – Check Python 3

python3 --version

You should see something like Python 3.9.2 (any 3.7+ is fine).

Step 3 – Install libraries

sudo apt update
pip3 install requests beautifulsoup4

Step 4 – Create a working folder

mkdir ~/schema_check
cd ~/schema_check

Step 5 – Create the script file

nano check_schema.py

Then paste this entire script:

import requests, json, csv, time
from bs4 import BeautifulSoup

# ---------- configuration ----------
# Put your product URLs here (you can add as many as you like)
urls = [
    "https://www.example.co.uk/example-product-1",
    "https://www.example.co.uk/example-product-2"
]

# Fields you want to confirm exist in the Product schema
required_fields = ["name", "brand", "sku", "price", "priceCurrency", "availability"]

# Optional delay between requests (seconds)
delay = 2

# ---------- functions ----------
def extract_product_schema(url):
    try:
        r = requests.get(url, timeout=15)
        soup = BeautifulSoup(r.text, "html.parser")
        for tag in soup.find_all("script", type="application/ld+json"):
            try:
                data = json.loads(tag.string)
                if isinstance(data, list):
                    for item in data:
                        if item.get("@type") == "Product":
                            return item
                elif data.get("@type") == "Product":
                    return data
            except Exception:
                continue
    except Exception as e:
        print(f"Error fetching {url}: {e}")
    return None

def check_fields(product_json):
    found = json.dumps(product_json)
    return [f for f in required_fields if f not in found]

# ---------- main ----------
results = []
for u in urls:
    print(f"Checking {u} ...")
    product = extract_product_schema(u)
    if not product:
        print(f"❌ No Product schema found: {u}")
        results.append([u, "No Product schema", ""])
    else:
        missing = check_fields(product)
        if missing:
            print(f"⚠️ Missing: {', '.join(missing)}")
            results.append([u, "Missing fields", ", ".join(missing)])
        else:
            print(f"✅ All key fields present")
            results.append([u, "All fields present", ""])
    time.sleep(delay)

# ---------- save to CSV ----------
with open("schema_results.csv", "w", newline="") as f:
    writer = csv.writer(f)
    writer.writerow(["URL", "Status", "Missing Fields"])
    writer.writerows(results)

print("\nDone! Results saved to schema_results.csv")

Save and exit:

Ctrl + O, Enter → save
Ctrl + X → exit

Step 6 – Edit your URLs

Later, open the script again (nano check_schema.py) and replace the two example links with your 10–50 product URLs.
Each URL must be inside quotes and separated by commas.

Step 7 – Run the script

python3 check_schema.py

It will:

Fetch each page
Extract the Product JSON-LD
Report any missing fields
Save a summary to schema_results.csv in the same folder

Step 8 – View the results

cat schema_results.csv

or open the file in LibreOffice Calc / Excel.

Example output:

URL,Status,Missing Fields
https://www.example.co.uk/football-goal.html,All fields present,
https://www.example.co.uk/tennis-net.html,Missing fields,priceCurrency availability
https://www.example.co.uk/baseball-bat.html,No Product schema,

Optional tweaks

Increase delay = 2 to 5 if you test hundreds of URLs (avoids rate limits).
You can import hundreds of URLs from a CSV by editing the script — ask later if you’d like that version.
Re-run anytime to confirm schema fixes.

Quick recap

Step	Action	Command
1	Open terminal	(click icon)
2	Check Python	`python3 --version`
3	Install deps	`pip3 install requests beautifulsoup4`
4	Make folder	`mkdir ~/schema_check && cd ~/schema_check`
5	Create script	`nano check_schema.py`
6	Edit URLs	inside script
7	Run it	`python3 check_schema.py`
8	View results	`cat schema_results.csv`

That’s it. Job done.

You’ve now got a simple tool that checks your product schema in seconds. No fancy platforms. No monthly fees. Just a Raspberry Pi doing proper work.

Run it whenever you push changes. Catch broken schema before Google does. Keep your rich results intact.

The script sits there, ready to go. Update your URLs. Hit run. Get answers.

This is what proper validation looks like – fast, local, and under your control.

Next steps?

Bookmark this guide for when you migrate sites
Test 50 products now, then spot-check monthly
If you need the CSV import version, you know where I am

Your structured data matters. Now you can actually prove it’s working.

Go check your products. Then sleep better knowing your schema’s solid.

Questions? Issues? The comments are open. MOFOs

Handy – GA 4 Report for Organic Traffic

On April 17, 2025 By DaveIn Analytics, seoLeave a comment

Looking at organic traffic for all our our products containing the word “hygiene” in the product name

Reports > Acquisition >- Traffic Acquisition
Add filter – Landing page + query string > contains > Hygieine > click “Apply” (bottom right)
Click + next to “Session Primary…” and add Landing page + query string
Use magnifying glass to search for “organic”
Use date picker to do comparison of date ranges.

ProductGroup Schema Example

On April 11, 2025April 23, 2025 By DaveIn seoLeave a comment

Used when you have product variants, like colours and sizes.

Some helmets, call this “product variant schema”. Not sure why though.

Example:

<html>
  <head>
    <title>Polyester winter football top</title>
    <script type="application/ld+json">
    [
      {
        "@context": "https://schema.org",
        "@type": "ProductGroup",
        "@id": "#footy-top",
        "name": "football shirt",
        "description": "Nice football shirt for playing football",
        "url": "https://www.example.com/footy-shirt",
        // ... other ProductGroup-level properties
        "brand": {
          "@type": "Brand",
          "name": "Ace Footy Kits"
        },
        "productGroupID": "44E01",
        "variesBy": [
          "https://schema.org/size",
          "https://schema.org/color"
        ]
      },
      {
        "@context": "https://schema.org",
        "@type": "Product",
      "isVariantOf": { "@id": "#footy-top" },
        "name": "Small green top",
        "description": "Small wool green top for the winter season",
        "image": "https://www.example.com/top_small_green.jpg",
        "size": "small",
        "color": "green",
        // ... other Product-level properties
        "offers": {
          "@type": "Offer",
          "url": "https://www.example.com/top?size=small&color=green",
          "price": 39.99,
          "priceCurrency": "USD"
          // ... other offer-level properties
        }
      },
      {
        "@context": "https://schema.org",
        "@type": "Product",
        "isVariantOf": { "@id": "#footy-top" },

        "name": "Small dark blue top",
        "description": "Small wool light blue top for the winter season",
        "image": "https://www.example.com/top_small_lightblue.jpg",
        "size": "small",
        "color": "light blue",
        // ... other Product-level properties
        "offers": {
          "@type": "Offer",
          "url": "https://www.example.com/top?size=small&color=lightblue",
          "price": 39.99,
          "priceCurrency": "USD"
          // ... other offer-level properties
        }
      },
      {
        "@context": "https://schema.org",
        "@type": "Product",
       "isVariantOf": { "@id": "#footy-top" },
        "name": "Large light blue top",
        "description": "Large wool light blue top for the winter season",
        "image": "https://www.example.com/top_large_lightblue.jpg",
        "size": "large",
        "color": "light blue",
        // ... other Product-level properties
        "offers": {
          "@type": "Offer",
          "url": "https://www.example.com/top?size=large&color=lightblue",
          "price": 49.99,
          "priceCurrency": "USD"
          // ... other offer-level properties
        }
      }
    ]
    </script>
  </head>
  <body>
  </body>
</html>

https://developers.google.com/search/docs/appearance/structured-data/product-variants

Here’s an example for boxing gloves –

Possible ProductGroup Schema


<script type="application/ld+json">
{
  "@context": "https://schema.org",
  "@type": "ProductGroup",
  "name": "BlackBeltWhiteHat Boxing Gloves",
  "description": "High-quality BlackBeltWhiteHat boxing gloves designed for training and sparring. Available in various sizes and colours to suit beginners and advanced boxers.",
  "image": "https://www.nicemma.com/media/catalog/product/m/e/BlackBeltWhiteHat-boxing-gloves.jpg",
  "brand": {
    "@type": "Brand",
    "name": "BlackBeltWhiteHat"
  },
  "manufacturer": {
    "@type": "Organization",
    "name": "BlackBeltWhiteHat"
  },
  "url": "https://www.nicemma.com/BlackBeltWhiteHat-boxing-gloves.html",
  "aggregateRating": {
    "@type": "AggregateRating",
    "ratingValue": "4.7",
    "reviewCount": "83",
    "bestRating": "5",
    "worstRating": "1"
  },
  "review": [
    {
      "@type": "Review",
      "author": {
        "@type": "Person",
        "name": "Sarah T."
      },
      "datePublished": "2023-09-14",
      "description": "Excellent gloves for the price! Great fit and perfect for light sparring.",
      "name": "Highly recommend!",
      "reviewRating": {
        "@type": "Rating",
        "bestRating": "5",
        "ratingValue": "5",
        "worstRating": "1"
      }
    }
  ],
  "offers": {
    "@type": "AggregateOffer",
    "lowPrice": "14.99",
    "highPrice": "29.99",
    "priceCurrency": "GBP",
    "itemCondition": "https://schema.org/NewCondition",
    "availability": "https://schema.org/InStock"
  },
  "hasMerchantReturnPolicy": {
    "@type": "MerchantReturnPolicy",
    "applicableCountry": "GB",
    "returnPolicyCategory": "https://schema.org/MerchantReturnFiniteReturnWindow",
    "merchantReturnDays": 30,
    "returnMethod": "https://schema.org/ReturnByMail",
    "returnFees": "https://schema.org/FreeReturn"
  },
  "shippingDetails": {
    "@type": "OfferShippingDetails",
    "shippingRate": {
      "@type": "MonetaryAmount",
      "value": "4.99",
      "currency": "GBP"
    },
    "shippingDestination": {
      "@type": "DefinedRegion",
      "addressCountry": "GB"
    },
    "deliveryTime": {
      "@type": "ShippingDeliveryTime",
      "handlingTime": {
        "@type": "QuantitativeValue",
        "minValue": 0,
        "maxValue": 1,
        "unitCode": "d"
      },
      "transitTime": {
        "@type": "QuantitativeValue",
        "minValue": 1,
        "maxValue": 3,
        "unitCode": "d"
      }
    }
  },
  "hasVariant": [
    {
      "@type": "Product",
      "name": "BlackBeltWhiteHat Boxing Gloves - Pink - 6oz",
      "description": "Pink BlackBeltWhiteHat Boxing Gloves, 6oz size -- ideal for young boxers or light training sessions. Offers excellent comfort and protection.",
      "image": "https://www.nicemma.com/media/catalog/product/m/e/BlackBeltWhiteHat-boxing-gloves.jpg",
      "url": "https://www.nicemma.com/BlackBeltWhiteHat-boxing-gloves.html",
      "color": "Pink",
      "size": "6oz",
      "sku": "BlackBeltWhiteHat-BG-PINK-6OZ",
      "gtin8": "12345678",
      "offers": {
        "@type": "Offer",
        "price": "14.99",
        "priceCurrency": "GBP",
        "availability": "https://schema.org/InStock",
        "itemCondition": "https://schema.org/NewCondition"
      },
      "review": {
        "@type": "Review",
        "reviewRating": {
          "@type": "Rating",
          "ratingValue": "5",
          "bestRating": "5",
          "worstRating": "1"
        },
        "author": {
          "@type": "Person",
          "name": "Sarah T."
        },
        "reviewBody": "Brilliant gloves for the price! Comfortable fit and ideal for light sparring."
      }
    },
    {
      "@type": "Product",
      "name": "BlackBeltWhiteHat Boxing Gloves - Pink - 8oz",
      "description": "Pink BlackBeltWhiteHat Boxing Gloves, 8oz size -- perfect for training and sparring, offering balanced protection and a snug fit.",
      "image": "https://www.nicemma.com/media/catalog/product/m/e/BlackBeltWhiteHat-boxing-gloves.jpg",
      "url": "https://www.nicemma.com/BlackBeltWhiteHat-boxing-gloves.html",
      "color": "Pink",
      "size": "8oz",
      "sku": "BlackBeltWhiteHat-BG-PINK-8OZ",
      "gtin8": "12345679",
      "offers": {
        "@type": "Offer",
        "price": "16.99",
        "priceCurrency": "GBP",
        "availability": "https://schema.org/InStock",
        "itemCondition": "https://schema.org/NewCondition"
      }
    },
    {
      "@type": "Product",
      "name": "BlackBeltWhiteHat Boxing Gloves - Pink - 10oz",
      "description": "Pink BlackBeltWhiteHat Boxing Gloves, 10oz size -- a versatile glove size suitable for pad work, bag work, and light sparring.",
      "image": "https://www.nicemma.com/media/catalog/product/m/e/BlackBeltWhiteHat-boxing-gloves.jpg",
      "url": "https://www.nicemma.com/BlackBeltWhiteHat-boxing-gloves.html",
      "color": "Pink",
      "size": "10oz",
      "sku": "BlackBeltWhiteHat-BG-PINK-10OZ",
      "gtin8": "12345680",
      "offers": {
        "@type": "Offer",
        "price": "18.99",
        "priceCurrency": "GBP",
        "availability": "https://schema.org/InStock",
        "itemCondition": "https://schema.org/NewCondition"
      }
    },
    {
      "@type": "Product",
      "name": "BlackBeltWhiteHat Boxing Gloves - Black - 12oz",
      "description": "Black BlackBeltWhiteHat Boxing Gloves, 12oz size -- designed for adult training and sparring, providing optimal wrist and knuckle support.",
      "image": "https://www.nicemma.com/media/catalog/product/m/e/BlackBeltWhiteHat-boxing-gloves.jpg",
      "url": "https://www.nicemma.com/BlackBeltWhiteHat-boxing-gloves.html",
      "color": "Black",
      "size": "12oz",
      "sku": "BlackBeltWhiteHat-BG-BLK-12OZ",
      "gtin8": "12345681",
      "offers": {
        "@type": "Offer",
        "price": "22.99",
        "priceCurrency": "GBP",
        "availability": "https://schema.org/InStock",
        "itemCondition": "https://schema.org/NewCondition"
      }
    },
    {
      "@type": "Product",
      "name": "BlackBeltWhiteHat Boxing Gloves - Black - 14oz",
      "description": "Black BlackBeltWhiteHat Boxing Gloves, 14oz size -- suitable for heavy bag work and sparring, offering enhanced padding for protection.",
      "image": "https://www.nicemma.com/media/catalog/product/m/e/BlackBeltWhiteHat-boxing-gloves.jpg",
      "url": "https://www.nicemma.com/BlackBeltWhiteHat-boxing-gloves.html",
      "color": "Black",
      "size": "14oz",
      "sku": "BlackBeltWhiteHat-BG-BLK-14OZ",
      "gtin8": "12345682",
      "offers": {
        "@type": "Offer",
        "price": "25.99",
        "priceCurrency": "GBP",
        "availability": "https://schema.org/InStock",
        "itemCondition": "https://schema.org/NewCondition"
      }
    },
    {
      "@type": "Product",
      "name": "BlackBeltWhiteHat Boxing Gloves - Black - 16oz",
      "description": "Black BlackBeltWhiteHat Boxing Gloves, 16oz size -- ideal for sparring sessions, providing maximum hand protection and durability.",
      "image": "https://www.nicemma.com/media/catalog/product/m/e/BlackBeltWhiteHat-boxing-gloves.jpg",
      "url": "https://www.nicemma.com/BlackBeltWhiteHat-boxing-gloves.html",
      "color": "Black",
      "size": "16oz",
      "sku": "BlackBeltWhiteHat-BG-BLK-16OZ",
      "gtin8": "12345683",
      "offers": {
        "@type": "Offer",
        "price": "29.99",
        "priceCurrency": "GBP",
        "availability": "https://schema.org/InStock",
        "itemCondition": "https://schema.org/NewCondition"
      }
    }
  ]
}
</script>

The above example, was based on this product schema:

Current product schema

<script type="application/ld+json">
{
"@context": "http://schema.org",
"@type": "Product",
"description": "Best boxing gloves for any level martial arts fighter. Thanks to the range of sizes on offer, these sparring gloves are suitable as adults or kids boxing gloves. Crafted from premium Maya hide leather material, these are long-lasting boxing training gloves. Pink or Black available.", "name": "BlackBeltWhiteHat Boxing Gloves",
"image": "https://nwscdn.com/media/catalog/product/cache/h265xw265/b/o/boxing-gloves-black_1.jpg",

"sku": "BlackBeltWhiteHat-boxing-gloves",

"url": "https://www.nicemma.com/BlackBeltWhiteHat-boxing-gloves.html",

"brand": "BlackBeltWhiteHat",

"offers": [

{

"@type": "Offer",

"itemCondition": "http://schema.org/NewCondition",

"price": "7.99",

	"availability": "InStock",

	"priceCurrency": "GBP",

	"url" :"https://www.nicemma.com/BlackBeltWhiteHat-boxing-gloves.html"

}

]

,

"review": [

{


"@type": "Review",


"author": {


"@type": "Person",


"name" : "Danny "

},


"datePublished": "2020-08-20",


"description": "These gloves are great- really comfortable and easy to handle. I have got a good amount of use out of these so far and would highly recommend these to anyone looking for a pair of both long lasting and high quality gloves!

",

"name": "Amazing All Round",
"reviewRating": {
"@type": "Rating",
"bestRating": "5",
"ratingValue": "5",
"worstRating": "1"

}


} ]

,

"aggregateRating": {

"@type": "AggregateRating",

"ratingValue": "5",

"reviewCount": "1"

}

}

</script>

Please note:

You want the full productGroup Schema on the “main product” page.

On the variant pages e.g. nicemma.com/mma-t-shirt?red-XL – You still want the ‘normal’ product schema NOT productGroup

“@type”: “Product”

With the details just for the variant.

Beginner’s Guide to Carl Jung Psychology [2025]

On March 13, 2025September 26, 2025 By DaveIn Mental Health2 Comments

Good day.

Here’s a brief overview of some of Jung’s essential concepts. Enjoy…

Jung’s Model of the Psyche

The Structure of the Psyche that big Carl Jung introduced, was a three-part, cool as fuck model that collectively kicked his old pal Fraud square in the ball sack:

Ego: This is our conscious mind, where we keep thoughts, memories, and emotions (those things weak people have) that we’re aware of.

Personal Unconscious: This part holds forgotten or repressed memories and experiences, like the time your ma dropped you on your head when you were 1.

Collective Unconscious: A shared, universal layer of the unconscious that connects all humans. In theory. This could explain why all ancient cultures have similar myths and stories about dragons and mermaids and stuff.

Core Concepts

Archetypes: These are universal, inherited patterns of thought or symbolic imagery that come from the collective unconscious.

What does that mean?

Imagine your mind is a big library, and it has books with characters that everyone in the world can see and shares in this library.

These characters show up in loads of different ancient stories, even back in the day when the world was very segregated – because we had to walk or get on a horse to travel places…so people in the Scottish Highlands for example, probably didn’t share information that much with the French. Cos you know, fuck walking that far. Also languages and translating iPhones weren’t all that back in the 11th century.

Despite this, similar myths and stories with similar characters emerged across the world at this, ancient/old time – we’re talking a few thousand years ago. Well a few hundred anyway.

The hero for example, still shows up in modern stories. The hero’s journey – bloke is living day to day life, gets a call to action, kills loads of baddies, discovers himself, goes home.

Jung thought that these similar stories and characters emerged across the world, because of the “collective unconscious”.

Basically, we all have some kind of default programming that we are not conscious of. This is shown in the stories we tell, and they shape how we see the world and possibly, how we classify everyone.

Side note- this sounds similar to “morphic resinence” theory. Which states that when a species learns something new on one side of the world, the new knowledge is somehow used and adopted by the same species then other side of the world.

Examples include: The Mother, The Hero, The Wise Old Man, The Child, The bellend, and The Trickster.

Individuation: This is the journey of integrating both conscious and unconscious elements.

You know that you are aware when someone else is a complete c*nt? Well, we are not so good at being objective with ourselves.

Individuation kinda starts with knowing yourself, your own issues, your traumas, and how you project them.

Individuation is a process of self-discovery – becoming self aware and less of a knobhead that runs on default and auto-pilot.

People need to get to know themselves better, and stop distracting themselves with social media, TV, gaming etc if they want to develop in this way.

Keeping a dream journal can help – as this can, in theory, tell you what your subconscious is processing and ‘doing’. Incidentally, taking l-theanine before bed really gives you vivid dreams.

Understand what triggers you, why you feel the need to do certain things like judge people and get angry.

Face your mother fucking shadow. The part of you that you don’t like and try to hide. Everyone gets jealous, angry etc – this is fine, as long as you don’t act on it. It’s normal, move into and explore these feelings.

Be honest with yourself and accept your faults, so you can then work on them.

Complexes: Often caused by trauma, complexes can be good or bad.

Sometimes seen as clusters of thoughts, feelings, and memories centred around a specific idea within the personal unconscious.

Common complexes relate to parents, inferiority and power/status. Being aware of these complexes can help us grow and be ‘better’. Remember – You are enough and all that, but you can still grow and learn as along as you live.

Psychological Types

Jung identified two attitudes and four functions of consciousness:

Attitudes: Extraversion (like attention) and Introversion (likes to be alone).

Functions: Thinking, Feeling, Sensation, and Intuition.

Shadow: This represents the unconscious, often negative or repressed sides of our personality. Lots of memes about this fucker. Probably why you criticise people based on your own insecurities.

Anima/Animus: The feminine aspect found in males (Anima) and the masculine aspect found in females (Animus).

Persona: This is the “mask” or public persona we show to the world.

Self: The central archetype that symbolizes the unified psyche and the ultimate goal of individuation. Key Principles The psyche is self-regulating and always striving for balance and wholeness.

Dreams and symbols play a crucial role in helping us understand the unconscious. The collective unconscious shapes behaviours and experiences across different cultures.

The Jungian Approach to Therapy takes a fresh look at mental health by focusing on the present and future, rather than getting stuck in past experiences. It highlights how crucial dreams and active imagination are for tapping into the unconscious mind.

The goal is to guide individuals toward individuation and self-realization, helping them become their true selves. It also acknowledges the healing power of engaging with images and symbols, which can be incredibly transformative.

Diagram (above^)

Jung’s Model of the Psyche text [Collective Unconscious] | v [Personal Unconscious] | v [Ego] This simple diagram shows the layered structure of the psyche based on Jung’s theory.

At the deepest level is the collective unconscious, followed by the personal unconscious, with the ego sitting at the surface of our conscious awareness. The ego is a right wanker.

Jung’s psychology is all about integrating every part of our personality, including the shadow and spiritual aspects, to achieve a sense of psychological wholeness. His ideas have left a significant mark on the field of psychology, influencing areas like personality assessment (think Myers-Briggs Type Indicator) and dream analysis.

Carl Jung’s idea of synchronicity

Carl Jung’s idea of synchronicity refers to those coincidences that happen to us when we are more self aware.

Jungian psychologists think that these coincidences connect our inner thoughts & feelings with the ‘outside ‘real’ world, exposing how our minds and the material universe are intertwined.

What is Jungian Synchronicity?

Acausal Connection: Synchronicity connects our internal psychological experiences—like dreams and thoughts—with external events based on meaning rather than a straightforward cause-and-effect relationship.

Example: Imagine a patient dreaming about a golden scarab, only to have a real scarab beetle show up at Jung’s window during their therapy session.

Collective Unconscious: This phenomenon occurs when personal unconscious elements resonate with universal archetypes that are common to all of humanity.

Relativity of Time/Space: Jung proposed that synchronicity reveals a “psychically relative space-time continuum,” where our unconscious experiences blur the usual boundaries of time and space.

(Western Science) | Synchronicity (Jungian View)

Based on cause-effect chains | Meaningful, acausal parallels Governs physical phenomena
Connects psyche and matter Objective, measurable | Subjective, symbolic

How to Generate Synchronicity

While you can’t force synchronicity, there are practices that can help you become more open to those meaningful coincidences:

Heighten Awareness – Pay attention to patterns in your dreams, symbols, numbers, or recurring themes. Example: If you keep encountering a specific animal or phrase, it might be a sign that you’re tapping into an archetypal message.

Engage the Unconscious – Try using active imagination (Jung’s technique of conversing with your unconscious) or journaling to delve into your inner imagery.

Synchronicity often pops up during transitional states, like meditation or when you’re in a creative flow. Interpret Symbolism Look for personal or archetypal meanings in coincidences.

Example: A chance meeting could reflect something unresolved in your emotional landscape. Cultivate Openness Try to step back from relying solely on rational thought; allow yourself to track patterns

Keep a synchronicity journal to spot recurring themes or symbols as they emerge over time.

Clinical and Philosophical Context Therapeutic Use: About 70% of therapists see the value of synchronicity in uncovering unconscious material, although patients often feel a bit misunderstood when they share these experiences.

Individuation: Synchronicity plays a key role in Jung’s idea of self-realization by helping to blend conscious and unconscious elements.

Criticism: Focusing too much on synchronicity can sometimes veer into superstition or delusion, so it’s important to interpret it with balance.

A Modern Perspective Jung teamed up with physicist Wolfgang Pauli to connect synchronicity with the non-local aspects of quantum theory. While it hasn’t been scientifically proven, it still serves as a fascinating tool for exploring the unity of psyche and matter, as well as our search for existential meaning. By being mindful of coincidences and archetypal symbols, people can harness synchronicity’s insights for personal growth and creativity.

As Jung wisely said:

“Synchronicity is an ever-present reality for those who have eyes to see”.

As others say

“Shut up you mad cunt, it’s just coincidence”.

I had synchronicity happen once – which could be just the law of averages, but I dreamt about seeing an old mate in a specific street in Wrexham, and he was wearing a green jacket…this then actually happened a few weeks later. Which freaked me out.

Scraping Reviews with Screaming Frog? [2025]

On March 7, 2025March 11, 2025 By DaveIn UncategorizedLeave a comment

You can scrape reviews if they are :

– In Schema
– Have their own specific class of <p> tag

e.g. class=”review-tex text-base font-secondary”

Can you scrape the reviews then?

Yes, you can scrape the reviews if they are formatted in schema markup (like Review, AggregateRating, etc.) or if they belong to a specific class. Screaming Frog’s Custom Extraction feature will allow you to do this.

✅ How to Scrape Reviews in Schema or Specific HTML Classes Using Screaming Frog

1️⃣ Enable Structured Data Extraction (for Schema Reviews) If the reviews are in schema markup (JSON-LD, Microdata, or RDFa), Screaming Frog can directly extract them:

✔ Open Screaming Frog SEO Spider.

✔ Navigate to Configuration > Spider > Enable Structured Data.

✔ Start a crawl, and Screaming Frog will pull JSON-LD, Microdata, and RDFa formats.

✔ Once the crawl is finished, go to Reports > Structured Data > Review Schema to export the data.

2️⃣ Extract Reviews from a Specific Class (e.g., class=”review-text text-base font-secondary”) If the reviews are located within a specific tag that has a class, you can use Custom Extraction to get them:

✔ Open Screaming Frog and go to Configuration > Custom Extraction.

✔ Select XPath or CSS Selector based on your data structure. ✔ Use these extraction methods: 📝 Option 1: Using CSS Selector (Ideal for Class-Based Extraction) css Copy Edit p.review-text.text-base.font-secondary ✔

This will retrieve all tags that have the class review-text text-base font-secondary. 📝 Option 2: Using XPath (For More Complex Queries) xpath Copy Edit //p[contains(@class, ‘review-text text-base font-secondary’)] ✔

This will locate all elements with a class that includes “review-text text-base font-secondary”.

3️⃣ Run the Crawl and Export the Review Data

✔ Start the crawl and allow Screaming Frog to extract the reviews.

✔ After it’s done, go to Custom Extraction > Export Data.

✔ Open the resulting file in Excel or Google Sheets for further analysis. 🚀 Alternative Approach: Scraping JSON-LD Schema Reviews If the reviews are included in JSON-LD schema markup, you can extract them using: Custom Extraction (JSON-LD): xpath //script[@type=’application/ld+json’]/text()
✔ This extracts all JSON-LD data, which you can filter for reviews.
✔ Use JSON formatting tools (like jq or Python json.loads()) to clean and parse the data.

🔹 Summary: Can Screaming Frog Scrape Reviews?
✅ YES, if reviews are inside Schema Markup (JSON-LD, Microdata, RDFa).
✅ YES, if they have a specific class, using Custom Extraction.
✅ YES, if the reviews are static HTML, meaning they exist in the page source.
❌ NO, if reviews are loaded dynamically via JavaScript (use Selenium instead).