Screaming Frog

Advanced Techers

To exclude dynamic URLs

Click Configuration

This exclues all URLs that include a ? in the address.  The backslash before the question mark, escapes the question mark, so that it isn’t treated as a special character

Crawl a Folder

Spider Mode

Enter folder URL in the address bar – e.g.

Go to Configuration on the top nav bar


Enter URL with .* at the end e.g.*

Click “ok”

then “Start”

Find the Source of a 404

Go to the 404 URL and click on it in Screaming Frog itself (not on an exported sheet)

Click “Inlinks” tab at the bottom, and see what the anchor text is

If the “Inlink” is the same as the 404ing URL, then check the redirects – search screaming frog for the “From” URL that shows on the “Inlink” tab


Exporting InLinks in Screaming Frog

Go to “Bulk Export” at the very top menu
“All Inlinks” from the drop down menu