How To Configure Screaming Frog For Client Website Crawling

Screaming Frog is a tool used for crawling client sites (and our own websites) for performing Technical SEO analysis.

This guide shows you what the ideal crawl settings would look like for performing Initial Findings, for example.

Intro

By default, the crawl configuration is set up in a way that gives an incomplete picture of the state of the website. We can include more information into the crawl by changing some settings, but at the cost of crawl speed.

• To get to the settings you'd click on Configuration / Crawl Config in the top menu. Then, when you are done setting it up, you can go to the same menu and save your profile (see screenshot).

You can import/export profiles to adapt to your needs and deadlines, especially if the crawled site is enormous.

• Download this profile if you want to import ("Load") it instead of checking boxes manually.

• Getting more information and digging deeper can have a huge impact on crawl times.

• Internet speed & computer hardware also have an impact.

What your settings should look like, ideally:

Crawl:

Extraction:

Limits:

Rendering:

Advanced:

Content / Duplicates:

Content / Robots.txt

User Agent:

Crawl Analysis:

Additional Resources:

https://www.screamingfrog.co.uk/seo-spider/tutorials/

Crawl Configuration Profile (guaranteed only for version 23.2)

• This is hosted on our company Google Drive so you have to be logged in.

• The profile can be imported ("Loaded") instead of manually ticking boxes.