Semantic Search & SEO – Using Python and Google Colab

Semantic search adds context and meaning to search results. For example, if someone is searching for “Lego” – do they want to buy Lego toys, or see a Lego movie or TV show (Ninjago is great). Another example might be “Tesla” – do people want to see the latest self-driving car, or learn more about Tesla the scientist and inventor?

  • Make sure you understand search intent and any confusing searches like Tesla(inventor or car?), Jaguar (car or animal?), etc
  • Look for structured data opportunities
  • Optimise internal links – especially if you are using a “Pillar Post” and “Cluster Page” structure
  • Follow traditional on page SEO best practices with headers, meta titles, alt tags etc

SMA Marketing have done a cool YouTube video about Semantic Search and they recommend tools including:

  • Wordlift
  • Frase
  • Advanced Custom Fields for WordPress
  • Google Colab with a SpaCy

Before you publish a post – look at the search results for the keyword(s) you are optimising the post for. Check in incognito in Chrome to remove most of the personalisation of the results.

For any answer boxes or snippets, you can click the “3 dots” to get information about the results:

As well as the snippets, you can click the 3 dots next to any organic result. Here’s another result for “MMA training program pdf” with some additional information:

With this in mind – if you are looking to rank for “MMA training program pdf” then you will want to include the search terms highlighted in the “About this result” box: mma, training, program, pdf and ideally LSI keywords “workout” and “plan”.

It’s also a good idea to scroll down to the bottom of the SERP and check out the “related searches”

Take a look too at any breadcrumb results that pull through below the organic listings. Combining all this information will give you a good idea as to what Google understands by your search query and what people are looking for too.

Semantic Search & NLP

This is a bit techy, but thankfully, the guy at SMA Marketing (thank you if you’re reading this) has put together a file/load of python code that does most of the work for us. You can find it here – https://colab.research.google.com/drive/1PI6JBn06i3xNUdEuHZ9xKPG3oSRi1AUm?usp=sharing#scrollTo=uatWEoHp5nxZ

Hover over [1] and click the play icon that appears (highlighted yellow in screenshot below)

When that section has finished loading and refreshing, scroll down to the “Installation tensorflow + transformers + pipelines” section and click the play icon there.

When that’s finished doing it’s thing, scroll down again, and add your search query to the uQuery_1: section:

add your query and then press the “play” button on the left hand side opposite the uQuery_1 line

You should then see the top 10 organic results from Google on the left hand side – in the form of a list of URLs

Next, you can scrape all the results by scrolling down to the “Scraping results with Trafilatura” section and hover over the “[ ]” and press play again:

Next, when the scraping of results is done – scroll down to “Analyze terms from the corpus of results” section and click the play button that appears when you hover over “[ ]”

Next! when that’s done click the play button on the section full of code starting with:

“df_1[‘top_result’] = [‘Top 3’ if x <= 3 else ‘Positions 4 – 10’ for x in df_1[‘position’]] # add top_result = True when position <=3 “

Finally – scroll down and click the play button on the left of the “Visualizing the Top Results” section.

On the right hand side where it says “Top Top 3” and lists a load of keywords/terms – these are frequent and meaningful (apparently) terms used in the top 3 results for your search term.

Below that, you can see the terms used in the results from 4-10

Terms at the top of the graph are used frequently in the top 3 results e.g. “Mini bands”

Terms on the right are used frequently by the results in positions 4-10

From the graph above, I can see that for the search term “resistance bands” the top 3 results are using some terms, not used by 4-10 – including “Mini bands”, “superbands” “pick bodylastics”

  • If you click on a term/keyword in the graph – a ton of information appears just below:

e.g. if I click “mini bands”

Google Colab TOol

It’s interesting that “mini bands” is not featured at all in the results positioned 4-10

If you were currently ranking in position 7 for example, you’d probably want to look at adding “mini bands” into your post or product page

You can now go to the left-side-bar and click “Top 25 Terms” and click the “play icon” to refresh the data:

Semantic SEO tool

Obviously – use your experience etc and take the results with a pinch of salt – some won’t be relevant.

Natural Language Processing

next click on “Natural Langauge Processing” in the side-menu

Click the “play” icons next to “df_entity =df_1[df_1[‘position’] < 6]” and the section below.

When they have finished running click the play icon next to “Extracting Entities”

Click “play” on the “remove duplicates” section and again on the “Visualising Data” section

This should present you with a colourful table, with more terms and keywords – although for me most of the terms weren’t relevant in this instance 😦

You can also copy the output from the “Extracting the content from Top 5” section:

Python Google Colab
Then paste it into the DEMO/API for NLP that Google have created here:

https://cloud.google.com/natural-language#section-2

You can then click the different tabs/headings and get some cool insights

Google NLP API

Remember to scroll right down to the bottom, as you’ll find some additional insights about important terms and their relevance

The Google NLP API is pretty interesting. You can also copy and paste your existing page copy into it, and see what Google categories different terms as, and how “salient” or important/relevant it thinks each term is. For some reason, it thinks “band” is an organisation in the above screenshot. You can look to improve the interpretations by adding relevant contextual copy around the term on the page, by using schema and internal links.

Quickly Scrape SERP Results for SEO

Go to Google.com

Press CTRL+D to bookmark the page

Add it to your bookmarks bar if possible

Copy the code below:

javascript:var a = document.getElementsByTagName('a'), arr = '';for(var i=0; i<a.length; i++) if (a[i].ping && !a[i].href.includes('google'))arr +=('<p>' + a[i].href + '</p>');var newWindow = window.open();newWindow.document.write(arr);newWindow.document.close();


Right click on the bookmark and click “Edit”

Where it says “URL” paste in the code

Rename the bookmark “URL Extractor”

Save the bookmark

That’s it!

Test the URL Extractor – do a Google Search and then click on the Bookmark you’ve just created

You should now have a new tab with all of the URLs listed in it

Change your Google Search settings to show 100 results – you can just copy and paste the tab full of extracted URLS into Google Sheets

Once in Google Sheets, you can easily extract the Meta Titles, Descriptions and keywords

You just need to find and replace the URL in these formulas:

=importxml("https://blackbeltwhitehat.com","//title")
=importxml("https://blackbeltwhitehat.com/","//meta[@name='description']/@content")
=importxml("https://blackbeltwhitehat.com/","//meta[@name='keywords']/@content")

=IMPORTXML("https://blackbeltwhitehat.com/","//h1")

Google Analytics 4 – Quick Notes & Introduction 

The new version of Google Analytics – Google Analytics 4 – has less pre-made reports and users are prompted to customise their own dashboards and use the search function.

You can literally ask questions, using the search bar. Which is fun.

Instead of pageviews and sessions, GA 4 is built around events.

Pre-configured reports are limited in GA 4. It’s a good idea to have Universal Analytics in addition to GA4

  • You can’t yet link GA 4 to Search Console
  • You can Can store raw data in BigQuery

when you install GA4 on your site, the reports don’t import data from GA universal – start from scratch – no historical data is passed over to GA4 when you install it on your site.

  • GA4 is Built Around Events.

Events:

You can track pageloads, elements clicked, product details, and loads more

Parameters:

Parameters are info that are sent to GA4 with the events. 

Eg. Pageview is sent to GA4 with URL of the page, page title and the referral details

Automatic Events:

First Visit – first time someone visits site (this even populates the new user report too)

Page View – same as normal pageviews

Session Start – new session after 30 mins of inactivity

User Engagement – starts whens someone on your site for at least 10 secs

Enhancement Measurement

GA4  automatically tracks:

Scrolls – 90% of a page

Outbound Clicks –

Site Search – search queries

Video Engagement – for embedded YouTube videos

File Downloads

To edit the Enhancement Measurement reports go to “Data Streams” in the Admin area:

Click on data stream and check to see that “Enhanced Measurement” is switched to the on position. If you click on the cog icon, you can then switch different measurements on and off.

They’ll be set to “on” by default.

There are advanced settings for pageviews and site search – just click “Show advanced settings”. 

Google provides a list of recommended events for all websites and apps, and then by industry.

Retail & eCommerce

Jobs, Education, Local Deals & Real Estate

Travel 

Games

Custom Events

Custom events allows you to name your events.

Google’s naming convention is probably best to copy – [action]_[object/item]

For example, if you create an event to track when people rate your website, call it:

Rate_item 

Instead of

Product-rate

Add parameters such as product, item_id and rating – so you know which product is rated and what rating the user gave it.

You’ll need to register the parameters as custom dimensions or metrics

To make a new event – in the left hand nav/side bar – click – 

Events – Create Event – Name the event

Enter the parameters to tell Google when to trigger the new event

E.g. page_location – contains – thank-you

Click “create” in top right corner.

You can create brand new events in Tag Manager – rather than basing them on existing events

Data Settings>Data Retention

By default GA4 only stores data for 2 months – in the Admin area, you can change this to 14 months.

Go to Admin>Data Settings>Data Retention and change it to 14 months on the drop down menu.

Brief Run Down of GA4 Reports (in the left hand side-menu)

Acquisition Reports – Where are visitors coming from?

Engagement Reports – What are visitors doing on your site?

Monetization – New term for eCommerce Reporting in GA 4

See what items people are buying and how much money their spending

Retention – Looks at new users/visitors and returning visitors. See what percentage of different cohorts come back to the site etc

Demographics – Where are people based and what type of people are coming to your site?

Tech – what computers, tablets and phones are people using?

EVENTS

Conversions – conversions can be customised. But typically include “Begin_checkout” and “purchase”

All Events – basically less important events – compared to conversions. Include clicks, scrolls etc.

EXPLORE

Analysis – There is a Template Gallery (top right of screen) 

You can look into funnels, acquisition etc and get cool reports

CONFIGURE 

Audiences – you can build different audiences by location, device etc and analyse them

Data Studio – Quick Tips (Advanced)

Speed Up Data Studio Reports (Significantly) – Extract Data

To speed up your reports – you can “Extract Data” and cache it.

It can help to have 2 copies of the report up – so you can see which metrics and dimensions you need to select when adding the data to extract and cache (also a good idea to test the extract data method on a copy of the report in case you faff anything up)

Go to “Add Data” in the top menu-bar

  • Click on “Extract Data”
  • Choose the data you need – eg Google Analytics
  • Add the dimensions and metrics you need for the report
  • On the Right hand side – click to turn “Auto Update” on
  • Select “daily”
  • Click “Save and Extract”

Sometimes you have to faff around a bit with the dimensions – Google Analytics doesn’t seem to like caching a dimension, but still goes super-quick if you cache the metrics only.

Edit in Bulk

If you want to edit all of the charts or tables on the page, in “Edit” mode, right click – go to “Select” and then choose “Tables on page” or whatever type of chart, scorecard or table you’ve selected.

This works instead of CTRL clicking or SHIFT clicking – but you can only change charts or visualisations of the same type at the same time. You can change the style, add a comparison date range etc.

Brand Colour Theme in Data Studio

Click on “Them and Layout” at the top of the screen and then, near the bottom right click “Extract Theme from Image” – you can then upload your logo and choose a theme with your brand colours.

If your shite at presentation like me, this is helpful.

Copy & Paste Styles

In Data Studio – If you want to copy a style from a chart or table, right click it, then choose “copy”

Click another chart/table and the right click – Paste Special – Paste Style Only

Add Chart Filters to an Entire Report

If you want to add a filter to all the data in a report, then it can be a pain going through the charts individually.

Right click on a blank part of the page –

  • Click “Current Page Settings”
  • On the right hand side – click “Create a Filter”
  • Choose or create a filter to apply to all the page

To add a filter to multiple pages

  • Right click on a blank part of the page
  • click “Report Settings”
  • click “Add a filter” in the right side-menu

Add Elements to All Pages of a Report in Data Studio

If you want to add a header and date range selector, for example, to all the pages in the report – add the elements to a page, then right click on the element – and choose “Make report-level”

Quickly Align Elements in Data Studio

Click and drag to select all the elements

Right click – choose “align” – “middle” to get everything inline horizontally

To get an equal space between all the elements, so they’re spaced evenly:

– click and drag to select the elements

– right click – select “Distribute”

– “horizontally” to space evenly across the page, or “vertically” to distribute evenly in a vertical manner.

You can also tidy up individual tables to align the columns vertically – right click and select “”Fit to data”

See our Data Studio Fields & Filters blog post – https://businessdaduk.com/2021/12/15/data-studio-fields-filters-notes/

Data Studio Fields & Filters – Notes

Once you’ve created a field – add it to your chart as a Dimension

Mixed Case URLs

Having a mix of URL cases in the letters can faff with your data as D.S. might think each capitalisation variation is a new URL:

  • Click on the Data Tab to the right – then “Create new field” or “Add a Field”
  • Name the field “Lower Source” type in “LOWER(“
  • Then add “Source” from “Available Fields” to the left
  • Click “Save” and add “Lower Source” to the table as a Dimension

Concatenate Data

  • Create or “Add a Field”
  • In the Formula box type “CONCAT” and then select the fields you want to use
  • Close the formula with the final “)” and save it

Internal Site Searches – Extract Search Query

Make the data look nicer and get the search term on its own in the table.

  • Select to a “Add a Field”
  • Use REGEXP_EXTRACT formula to pull out the search terms and get rid of “/search?q”
  • Use the REGEX shown below:

Search Queries – Pull Out Questions – What, Why, When?

  • Create / Add a New Field
  • Add REGEX as shown below and save
  • Add New Field as a Dimension to the table
  • Create a table/data Filter so that you include only table rows that equate to “True” in the new dimension/column

CASE Formulas in Data Studio

CASE formulas are basically “If this, then do that” formulas

When X happens, Then do Y

You can use the CASE Formula to classify and group channels together

For example:

WHEN query matches “who”, then display text “Who?” in the table

  • In the Formula – you need a “catch all” default for when nothing is true in the criteria
  • If the search doesn’t contain “who, what, why” etc. then:
    ELSE “others”

    So if the search term, doesn’t match any of the REGEX criteria – classify it as “others”
– Save the CASE statement
  • Add the new field – to the table or report

Date Ranges & Filter Controls

Taken me ages to work this out – I’ve only just twigged:

You can add Comparisons, Search Boxes, Drop Down Menus and all sorts to your Data Studio Reports

For controls and filters to work – you need the chart or table to have “Default Date Range” set to “Auto”

See my post about REGEX for SEO here.

Ta

If you want a pre-made Data Studio template, you can send me some money and hope for the best.

Online Marketing for New Businesses

Started a business?

Got a website?

Okay, now you need to look at building your brand and getting traffic and engagement on your website and social media accounts.

Website Checklist

  • Trust signals – include membership badges, university logos prominently
  • Humanize the site – include images and videos of people who work for the business
  • Social Proof – include reviews and testimonials. TrustPilot and video testimonials work well
  • Site Speed – make sure the site is quick and works perfectly on mobile devices
  • EAT – Expertise Authority & Trust – show your credentials on the about us page
  • Contact Form – Make it as easy as possible to be contacted
  • Trust & Transparency – Include full contact info if possible – address, tel number, email
  • Have you got Google Analytics & Search Console installed?

You will also need to think about the colour scheme and imagery.

Think about what mood you want to portray

https://www.bedtracks.com/blog/2017/6/15/how-to-coordinate-music-with-the-colour-in-your-video

  • Strong Call To Action

You will probably want a “Call to Action” or “CTA” button, such as “Buy Now”, “Learn More” or “Contact Us”

This CTA button is generally placed “above the fold” on most pages, so that people don’t have to scroll down or look for a way to get in touch or buy from you.

“Join Free for a Month” – is the CTA on Netflix’s homepage (at the time of writing)

For more expensive, high end or though-out purchases such as – buying a car or contacting a therapist, sometimes it’s better to have the CTA below the fold. The best thing to do is test it, with Google Optimize.

For more information about “Conversion Rate Opimization” (CRO), see this article:

https://blog.hubspot.com/marketing/conversion-rate-optimization-guide

For a full SEO (Search Engine Optimization) checklist for your website – to help get visibility on Google – see this article – https://backlinko.com/seo-checklist

Google My Business

Register your website and your office with Google My Business

You can go through the steps here:

https://www.google.com/intl/en_uk/business/

Google will send out a postcard to your office (or home) address

The postcard has a code – so you can confirm you are at that address

Local Directories

Register your business with high quality, local directories such as

  • Bing Places
  • Yelp
  • Yell
  • Free Index
  • Open Di

Try and get on any local government directories too.

Social Media & Captioned Videos

If relevant, register your business on:

  • Linkedin
  • Facebook
  • Instagram
  • TikTok


    Arguably the best way to get noticed on social media at the moment, is to create videos with captions – so they can be watched on mute.

Linkedin is said to have the greatest organic reach at the moment too – meaning you can get your video, image or text-post in front of more people, without paying for ads.

Social media sites like people posting videos too – because they drive a high rate of engagement and keep people on the site for longer.

  • Do NOT post to YouTube and then post a link on social media

Instead – upload your video direct to the platform.

Social media sites will tend to kill your reach if you post a link – they don’t want people to click and leave their website

YouTube is also showing on more and more Search Results Pages on Google.

Consider creating a YouTube channel with lots of informative, helpful and entertaining content. You can then edit the videos and post to specific social media platforms.

Jab, Jab, Jab – Right Hook

General principle of content and social media marketing by Gary V.

Identify your target market

Identify their issues and pain points

Post helpful content related to their pain points and problems

Do NOT constantly promote your business – slip in the odd “Right Hook”, every 3 or 4 posts

People do not want to be sold to constantly, they want helpful, insightful and funny content.

For example.

If you target market is small business owners, take a look on Quora and Reddit and see what people are talking about. If a common theme is Facebook advertising for example, make some helpful videos and blog posts about Facebook marketing.

SEO, PPC and More

The above is just a foundation.

If you have the time and resources, you will ideally produce lots of insightful blog content, earn lots of inbound links and work your way to the top of Google.

You will also want to consider “PPC” – Pay Per Click ads on Google, Facebook and Linkedin.

One beginner mistake to avoid with ads – is sending people to your homepage.

Have a specific “landing page” for each advertising campaign.

oh – make sure you have a good looking logo too. You can use Canva or hire someone on PeoplePerHour.com

A good place to start with SEO is to check your website using an On-Page SEO Checklist.

Videos are great for social media, and YouTube is also starting to show more and more often in the Google results. I would personally have a good go at gaining an online presence using videos and social media – particularly Linkedin at the moment.

Build a Brand

Here’s a good article that some hero wrote about building your brand as a small business

  • Nail down your USP
  • Identify other propositions “why use me/us and not the competitor?”
  • Write down your brand story
  • Use high quality photography & videography (avoid stock pictures)

Consider making customer support a key element of your brand – this can help with online reviews too. Pre-purchase, purchase and post-purchase consumer stages are all opportunities to impress and help.

Get Content Ideas from Competitor Websites (SEMRush)

Requires:

  • SEMRush
  • A computer
  • The internet

If competitor has articles and blog posts inside a subfolder e.g. “buyers-guides” or “/blog” – make a note of the sub-folder name

  • Add Competitor’s homepage URL in “Search Bar” on SEMRush Homepage
  • Click “Organic Research”
  • Click on the “Positions” tab
  • Click on “Advanced Filter” and add the subfolder name e.g. “blog”
  • Export Results into Excel
  • Create a Pivot Table
  • Use the settings below – you’ll need to change position to “average” instead of “sum of”:
  • Tick the check-box for “URL” at the top
  • Drag search volume, traffic and position into “Values” box at the bottom right
  • Click the little arrow on the right of “Sum of Position” – go to “Value Field Settings” and choose “average”
  • Analyse which articles get the most traffic (approximately) and have most potential

Obvs. the URLs of my competitor have been blacked-out in the image above

If the competitor has all their articles at the root domain level e.g.

https://barbend.com/best-weightlifting-belts/

Just use SEMRush – Organic Research – Positions tab and download and pivot the pages data – no need for advanced filter

How to Create a Topic Cluster for SEO

Using Ahrefs with Wikipedia

  • Find the most relevant Wikipedia page, for example:

https://en.wikipedia.org/wiki/Association_football

  • Add the Wikipedia URL and click “submit”
  • Click “Copy to Clipboard”
  • Paste all the keywords/topics into a Google Sheet and remove any that aren’t relevant
  • In Ahrefs (or a similar tool like SEMRush’s KW Manager) – go to KW explorer
  • Paste in the keywords/topics
  • In Ahrefs – click “matching terms” then “Questions”

Using SEMRush with UGC

  • Copy the homepage URL of a UGC* website such as Pinterest or Quora
  • Put the URL in the main “search bar” in SEMRush
  • Go to “Organic Research” using the sidebar
  • Click “Positions”
  • Add a Keyword to the “Filter by Keyword” search box
  • Click a relevant Keyword in the table of results
  • Under “Questions” click “View all….”
  • Go back and click other relevant keywords
  • Click the “Questions” tab again – make a note or download all the relevant questions
  • Organise KWs and questions into main pillar posts and “cluster content”

See Hubspot’s article about Pillar Pages and Cluster Content – here

*UGC – User Generated Content

Sorting XML Sitemap URLs by Folder Depth

This can be handy if for example, you only want a list of products and they reside on folders that are 3 “/” deep into your URLs/Domain

For example:

  • Myshop.com/categorypages/subcategorypage/productpage/

I only want the URLs that reside at the third level – i.e. /productpage/

  1. Go to your XML sitemap – usually at Myshop.com/sitemap.xml
  2. Right click and “save as” – save on your computer
  3. Open Excel
  4. Go to the Developer Tab (you might need to add this as it’s not there by default)
  5. Click “Import”
  6. Browse to find your sitemap.xml and import it into Excel
  7. This usually pulls all your URLs into column 1 and other info like priority into separate columns
  8. Delete all the columns except the first one with your URLs in it
  9. Remove the https:// from the URLs with “find and replace” – On “Home” tab under “Find & Select” on the right
  10. In cell B2 add the function:
=LEN(A2)-LEN(SUBSTITUTE(A2,"/",""))

11. Drag the formula down the rest of column B

12. You can now order column B by the number of “/” found in each URL

If different categories have different folder structures then you can conditionally format and use different colours for different categories and then do a multiple criteria sort – by colour, then folder depth (column B)

You can download an example spreadsheet with the formula in here

SEO Technical Audit Checklist (advanced)

The idea of technical SEO is to minimise the work of bots when they come to your website to index it on Google and Bing. Look at the build, the crawl and the rendering of the site.

Tools Required:

  • SEO Crawler such as Screaming Frog or DeepCrawl
  • Log File Analyzer – Screaming Frog has this too
  • Developer Tools – such as the ones found in Google Chrome – View>Developer>Developer Tools
  • Web Developer Toolbar – giving you the ability to turn off Javascript
  • Search Console
  • Bing Webmaster Tools – shows you geotargetting behaviour, gives you a second opinion on security etc.
  • Google Analytics – With onsite search tracking *

    *Great for tailoring copy and pages. Just turn it on and add query parameter

Tech SEO 1 – The Website Build & Setup

The website setup – a neglected element of many SEO tech audits.

  • Storage
    Do you have enough storage for your website now and in the near future? you can work this out by taking your average page size (times 1.5 to be safe), multiplied by the number of pages and posts, multiplied by 1+growth rate/100

for example, a site with an average page size of 1mb with 500 pages and an annual growth rate of 150%

1mb X 1.5 X 500 X 1.5 = 1125mb of storage required for the year.

You don’t want to be held to ransom by a webhost, because you have gone over your storage limit.

  • How is your site Logging Data?
    Before we think about web analytics, think about how your site is storing data.
    As a minimum, your site should be logging the date, the request, the referrer, the response and the User Agent – this is inline with the W3 Extended Format.
log file analyzer

When, what it was, where it came from, how the server responded and whether it was a browser or a bot that came to your site.

  • Blog Post Publishing
    Can authors and copywriters add meta titles, descriptions and schema easily? Some websites require a ‘code release’ to allow authors to add a meta description.
  • Site Maintenance & Updates – Accessibility & Permissions
    Along with the meta stuff – how much access does each user have to the code and backend of a website? How are permissions built in?
    This could and probably should be tailored to each team and their skillset.

    For example, can an author of a blog post easily compress an image?
    Can the same author update a menu (often not a good idea)
    Who can access the server to tune server performance?

Tech SEO 2 – The Crawl

  • Google Index

Carry out a site: search and check the number of pages compared to a crawl with Screaming Frog.

With a site: search (for example, search in Google for site:businessdaduk.com) – don’t trust the number of pages that Google tells you it has found, scrape the SERPs using Python on Link Clump:

Too many or too few URLs being indexed – both suggest there is a problem.

  • Correct Files in Place – e.g. Robots.txt
    Check these files carefully. Google says spaces are not an issue in Robots.txt files, but many coders and SEOers suggest this isn’t the case.

XML sitemaps also need to be correct and in place and submitted to search console. Be careful with the <lastmod> directive, lots of websites have lastmod but don’t update it when they update a page or post.

  • Response Codes
    Checking response codes with a browser plugin or Screaming Frog works 99% of the time, but to go next level, try using curl and command line. Curl avoids JS and gives you the response header.

Type in Curl – I and then the URL

e.g.

curl – I https://businessdaduk.com/

You need to download cURL which can be a ball ache if you need IT’s permission etc.

Anyway, if you do download it and run curl, your response should look like this:

Next enter an incorrect URL and make sure it results in a 404.

  • Canonical URLs
    Each ‘resource’ should have a single canonical address.

common causes of canonical issues include – sharing URLs/shortened URLs, tracking URLs and product option parameters.

The best way to check for any canonical issues is to check crawling behaviour and do this by checking log files.

You can check log files and analyse them, with Screaming Frog – the first 1,000 log files can be analysed with the free version (at time of writing).

Most of the time, your host will have your logfiles in the cPanel section, named something like “Raw Access”. The files are normally zipped with gzip, so you might need a piece of software to unzip them or just allow you to open them – although often you can still just drag and drop the files into Screaming Frog.

The Screaming Frog log file analyser, is a different download to the SEO site crawler – https://www.screamingfrog.co.uk/log-file-analyser/

If the log files are in the tens of millions, you might need to go next level nerd and use grep in Linux command line

Read more about all things log file analysis-y on Ian Lurie’s Blog here.

This video tutorial about Linux might also be handy. I’ve stuck it on my brother’s old laptop. Probably should have asked first.

With product IDs, and other URL fragments, use a # instead of a ? to add tracking.

Using rel-canonical is a hint, not a directive. It’s a work around rather than a solution.

Remember also, that the server header, can override a canonical tag.

You can check your server headers using this tool – http://www.rexswain.com/httpview.html (at your own risk like)


Tech SEO 3 – Rendering & Speed

  • Lighthouse
    Use lighthouse, but use in with command line or use it in a browser with no browser add-ons.If you are not into Linux, use pingdom, GTMetrix and Lighthouse, ideally in a browser with no add-ons.

    Look out for too much code, but also invalid code. This might include things such as image alt tags, which aren’t marked up properly – some plugins will display the code just as ‘alt’ rather than alt=”blah”
  • Javascript
    Despite what Google says, all the SEO professionals that I follow the work of, state that client-side JS is still a site speed problem and potential ranking factor. Only use JS if you need it and use server-side JS.

    Use a browser add-on that lets you turn off JS and then check that your site is still full functional.

  • Schema

Finally, possibly in the wrong place down here – but use Screaming Frog or Deepcrawl to check your schema markup is correct.

You can add schema using the Yoast or Rank Math SEO plugins

The Actual Tech SEO Checklist (Without Waffle)

Basic Setup

  • Google Analytics, Search Console and Tag Manager all set up

Site Indexation

  • Sitemap & Robots.txt set up
  • Check appropriate use of robots tags and x-robots
  • Check site: search URLs vs crawl
  • Check internal links pointing to important pages
  • Check important pages are only 1 or 2 clicks from homepage

Site Speed

Tools – Lighthouse, GTMetrix, Pingdom

Check – Image size, domain & http requests, code bloat, Javascript use, optimal CSS delivery, code minification, browser cache, reduce redirects, reduce errors like 404s.

For render blocking JS and stuff, there are WordPress plugins like Autoptimize and the W3 Total Cache.

Make sure there are no unnecessary redirects, broken links or other shenanigans going on with status codes. Use Search Console and Screaming Frog to check.

Site UX

Mobile Friendly Test, Site Speed, time to interactive, consistent UX across devices and browsers

Consider adding breadcrumbs with schema markup.

Clean URLs

Image from Blogspot.com

Make sure URLs – Include a keyword, are short – use a dash/hyphen –

Secure Server HTTPS

Use a secure server, and make sure the unsecure version redirects to it

Allow Google to Crawl Resources

Google wants to crawl your external CSS and JS files. Use “Fetch as Google” in Search Console to check what Googlebot sees.

Hreflang Attribute

Check that you are using and implementing hreflang properly.

Tracking – Make Sure Tag Manager & Analytics are Working

Check tracking is working properly. You can check tracking coed is on each webpage with Screaming Frog.

Internal Linking

Make sure your ‘money pages’ or most profitable pages, get the most internal links

Content Audit

Redirect or unpublish thin content that gets zero traffic and has no links. **note on this, I had decent content that had no visits, I updated the H1 with a celebrity’s name and now it’s one of my best performing pages – so it’s not always a good idea to delete zero traffic pages**

Consider combining thin content into an in depth guide or article.

Use search console to see what keywords your content ranks for, what new content you could create (based on those keywords) and where you should point internal links.

Use Google Analytics data regarding internal site searches for keyword and content ideas 💡

Update old content

Fix meta titles and meta description issues – including low CTR

Find & Fix KW cannibalization

Optimize images – compress, alt text, file name

Check proper use of H1 and H2

See what questions etc. are pulled through into the rich snipetts and answer these within content

Do you have EAT? Expertise, Authority and Trust?

https://www.semrush.com/blog/seo-checklist/

You can download a rather messy Word Doc Template of my usual SEO technical checklist here:

https://smallbusinessdad.files.wordpress.com/2021/11/drewseotemplate.docx

It uses Screaming Frog, SEMRush and Search Console