So yesterday was a great day to be a millennial in Digital Marketing with 2 very big bits of news coming out. Firstly Space Jam after all the talk is officially happening and secondly, one of our favourite tools has had a game-changing update. What a time to be alive.
Anyway, now I’m done freaking out about Bugs Bunny setting up LeBron for an Alley-Oop, we can get onto Screaming Frog! If you have worked with us here at Bulldog or have been keeping track of our blogs, you’ll know that Screaming Frog’s SEO Spider is one of our favourite tools to use on the SEO team. For this update (Codenamed Liger) there are a lot of exciting new features being added for us to breakdown. We’ll look at the new features added and let you know what we think of them. (If you want a more in-depth breakdown of the features purpose & functionality, I’d recommend checking out Screaming Frog’s blog post about it here)
Scheduling
This feature is something Myself & Richard have been praying for for a while now, the ability to set up crawls to run automatically, this only gets better when you realise that you can set it to automatically save the crawl data. This update makes our life so much easier, we can set up our clients to be crawled automatically once a month to keep an eye on everything going on with our clients sites.
Full Command Line Interface & -Headless Mode
Another great feature here, with the new Screaming Frog update you can now operate the SEO Spider entirely via command line. Screaming Frog have put together a full list of commands you can use in the command line here.
There’s a lot of potential for this feature and we’re sure some techy users will think up some new & unique ways to utilise this tool.
Indexability & Indexability Status
This isn’t a massive change in how the SEO Spider works, but the concept of indexability is important for any SEO. With this added into the Spider, every URL is either ‘Indexable’ or ‘Non-Indexable’.
These phrases are now commonplace within SEO, but the community hasn’t got an exact definition of what they mean. For the SEO Spider, an ‘Indexable’ URL means a page that can be crawled, responds with a ‘200’ status code and is permitted to be indexed.
This may not be entirely in line with how Search Engines can actually index pages, due to various reasons, but for efficiency in identifying indexable pages in an audit, this will be a great approximation.
XML Sitemap Crawl Integration
I know what you’re thinking “But Buzz, I’ve had Screaming Frog’s SEO Spider for years & always been able to crawl XML sitemaps!” well you have, but now you can crawl them & integrate them as part of a site crawl.
There’s a new sitemap tab (pictured below) which has filters that allow you to quickly analyse common issues with your sitemap. Also you can now put the XML Sitemap location into the URL bar at the top, and the SEO Spider will crawl that directly, too (instead of switching to list mode).
Internal Link Score
If you’re looking for a useful way to evaluate & improve your internal linking this is a feature for you! This new addition brings in ‘Link Score’ which is a 0-100 scale which essentially analyses & estimates the authority of individual pages on a site by looking at various factor.
The actual process of working out the authority is rather complex & in depth so I shan’t bore you with all the details, but this is a great feature to use for interlinking to make sure you use authoritative pages in your interlinking strategy.
Post Crawl Analysis
The SEO Spider can now perform additional analysis once a crawl has been completed (or when it’s stopped) for more insight into the crawl. This includes the new ‘Link Score’ metric we just mentioned and a number of other new filters that have been introduced.
The crawl analysis feature can be set to run automatically once a crawl has been completed, or it can be set up to run manually. This can be viewed under ‘Crawl Analysis > Configure’ and the crawl analysis can be started by selecting ‘Crawl Analysis > Start’. When the analysis is running, the SEO Spider can continue to be used as normal which is a massive bonus if you have a lot of sites to crawl.
Visualisations
There is nothing more satisfying than a nice bit of data visualisation in a visually pleasing chart or graph. Well now you can get the satisfaction of visualisation from this new feature, the SEO Spider can now provide useful site visualisations, allowing you to visually understand the architecture of a site, internal linking structure, etc without having data hidden from you.
There are a few different types of visualisations provided in this update, including 2 crawl visualisations and 2 directory tree visualisations.
AMP Crawling & Validation
With the new AMP Crawling & Validation feature you can automatically crawl & extract AMP (accelerated mobile pages) to analyse & validate them. Allowing you to quickly identify various AMP issues via the new AMP tab & filters (See below)
Furthermore the AMP validator has been integrated into the SEO Spider, allowing you to identify any large scale validation issues, this uses exact checks from the AMP validator.
Canonicals & Pagination Tabs & Filters
Canonicals and pagination were previously included within the directives tab. However, neither are directives and as such don’t belong in the directives tab.
When it comes to auditing a website both of these are useful to view in combination with each other, so there is now a tab dedicated to all canonical issues and another for pagination issues. So, both have their own new tabs with new & updated tabs that are more in depth.
Improved Redirect & Canonical Chain Reports
The SEO Spider now reports on canonical chains and ‘mixed chains’, which can be found in the renamed ‘Redirect & Canonical Chains’ report.
The updated reports means auditing redirects on a site is now massively more efficient, as you can quickly identify the start & end URL of the chain and the chain type, as well as the number of redirects & indexability of the final URL in the chain.
Other Updates
That’s right, there are still more updates, but most of these are smaller updates & bug fixes outlined below:
🐸 You’re now able to automatically load new URLs discovered via Google Analytics and Google Search Console, into a crawl. Previously new URLs discovered were only available via the orphan pages report, this now configurable. This option can be found under ‘API Access > GA/GSC > General’.
🐸 ‘Non-200 Hreflang URLs’ have now been moved into a filter under the ‘Hreflang‘ tab.
🐸 You can disable respecting HSTS Policy under the advanced configuration (to retrieve true redirect status codes easier, rather than an internal 307) .
🐸 The ‘Canonical Errors’ report has been renamed to ‘Non-Indexable Canonicals’ and is available under reports in the top level menu.
🐸 The ‘rel=”next” and rel=”prev” Errors’ report has been adjusted to ‘Pagination’ > ‘Non-200 Pagination URLs’ and ‘Unlinked Pagination URLs’ reports.
🐸 Hard disk space has been reduced by around 30% for crawls in database storage mode.
🐸 Re-spidering of URLs in bulk on larger crawls is faster and more reliable.
🐸 There are new ‘bulk exports’ for Sitemaps and AMP as you would expect.
🐸 The main URL address bar at the top is now much wider.
🐸 Donut charts and right click highlighting colours have been updated.
🐸 There’s a new ‘Always Follow Canonicals‘ configuration item for list mode auditing.
🐸 The 32k character limit for custom extraction has been removed.
🐸 ‘rel=”next” and rel=”prev” are now available in the ‘Internal’ tab.
🐸 ‘Max Redirects To Follow’ configuration has been moved under the ‘Limits’ tab.
🐸 There’s now a ‘resources’ lower window tab, which includes (you guessed it), resources.
🐸 The Google Search Console integration website profiles list is now searchable.
🐸 The include and exclude configuration, now have a ‘test’ tab to help test your regex pre a crawl.
🐸 There’s a new splash screen on start-up.
🐸 There’s a bunch of new right click options for popular checks with other tools such as PageSpeed Insights, the Mobile Testing Tool etc.
There you go, that’s everything covered in the latest update for Screaming Frog SEO Spider, if you’ve made it this far, good on you, now go over to the Screaming Frog site & download V10.0 of the Screaming Frog SEO Spider.
Oh, let’s not forget to include Richards love for Screaming Frog with a little snippet from our Behind The Bulldog Epsiode 3!
