If You’re Folder URL Resolves with a 200 with a trailing slash /

For example – to crawl mysite.com/buyers-guides/

To crawl only URLs in the /buyers-guides/ folder on mysite.com using Screaming Frog, follow these steps:

1. Open Screaming Frog SEO Spider and enter the URL: https://www.mysite.com/buyers-guides/
2. Go to Configuration > Spider > Crawl
3. Under “Crawl Limits,” select “Limit Crawl Depth” and set it to 1 to crawl only the specified folder
4. In the “Include” box, add this pattern: https://www.mysite.com/buyers-guides/.*
5. Start the crawl by clicking the “Start” button

This configuration will ensure Screaming Frog only crawls URLs within the /buyers-guides/ folder on networldsports.co.uk, excluding other sections of the website.

---

If You’re Folder URL Resolves with a 200 without a trailing slash /

https://www.mysite.com/buyers-guides

Enter the URL of the domain plus sub/folder into the main address bar on Screaming Frog.

Choose Sub-folder option to the right of the address bar:

Go to Configuration (top menu bar, to the left) >Spider > crawl > Include
– Add the sub-folder (without trailing slash) to the include section:

Click “Start”. (Button to the right of the “Subfolder” drop down).

---

Crawling a sub-folder – plus all internal & external links that are contained within the sub-folder pages.

The protocol above would only check the status codes of any URLs actually held within the /buyers-guides/ folder.

For example, if a football goal guide, links to the FA’s website, and 404s, the above methods would not pick this up (as the FA doesn’t have /buyers-guides/ in the homepage etc)

1. Crawl and check all the URLs within a sub-folder e.g.

https://www.example.com/buyers-guides



2. get the status codes of any internal and external link, that point to outside the folder?



For example:


Our football goal guide –

https://www.example.com/buyers-guides/football-pitch-size-guide

Contains links that point outside of the buyers-guides folder, like to our product pages and external links to thefa.com etc.

Sub-Folder with Trailing Slash /buyers-guides/

Crawl https://www.example.com/buyers-guides/ with ‘Crawl Outside of Start Folder’ disabled, but with ‘Check Links OutSide of Start Folder’ enabled.

Sub-Folder with NO Trailing Slash /buyers-guides

Perform the crawl outlined above (using the include), get the list of https://www.example.com/buyers-guides URLs

then switch to list mode (Mode > List), go to ‘Config > Spider > Limits‘ and change the ‘Limit Crawl Depth’ from ‘0’ to ‘1” and upload and crawl the URLs

Remember to delete in include /buyers-guides from the crawl config before doing the above
i.e. Config>Spider>Include – remove anything in the box/field.