Publishing content on the Internet is second nature for the majority of businesses. Publishing engaging content with the right keywords improves organic rankings, drives traffic and potentially increases conversions.

But little thought is given to how content is processed, so to speak - how search engines work. What other behind the scene factors happen that determine how web pages are ranked?

One of those factors is crawl budget.

Search engines don't have unlimited resources to crawl web pages, images, JavaScript and CSS files and documents etc.

You can't please everyone all the time. So to prioritise crawling efforts, search engines need to be smart by dividing attention across millions of websites and billions of URLs.  This is where crawl budget comes into play.

Essentially a crawl budget is the number of URLs a search engine crawls within a given timeframe. Every search engine calculates crawl budget based on:

  • Crawl limit
  • Crawl demand

Crawl limit

The crawl limit, or host load, is the frequency search engines crawl a website without causing any issues. The slower the server for instance, the more impact it will have on crawl budget. Other factors that determine crawl limit include:

  • Platform latency and errors: If a server returns errors when URLs are requested then the crawl limit is affected
  • Shared hosted platforms: If a website is hosted on a platform with other websites then the crawl budget is shared across all the websites hosted on that platform. This will affect crawl budget of larger websites

Crawl budgets are scheduled. Which websites, and how often should indexed URLs be re-crawled? This depends on various factors:

  • Page type: Different pages may need updating more than others. For example, a Privacy Notice page will most likely be updated annually, whereas a Product page would be updated more frequently, possibly daily. In this scenario, the Product page will take priority
  • Popularity: If URLs get a lot of traffic it means they’re popular and are ranking for certain search queries, and will be high in search results.  In addition, the amount of internal and external links play a role too.
  • Update status:  When were the URLs last updated?  Is the website's content fresh?
Enjoying this article? Sign up to our newsletter

Should crawl budget be important to SEO professionals?

Of course it should. Although the term 'crawl budget' isn't bandied around in content meetings and the like, the essence and the impact of crawl budget should always be considered.

What happens if you spend too much effort optimising pages on your site that don't drive much traffic? The crawl budget for those pages is being wasted.

On the other hand, focusing SEO efforts on those high impact web pages that deliver value and drive conversions means that they're more likely to be scheduled for a crawl.

Can you see crawl budget for your website?

Yes you can! In two ways in fact:

  1. Google Search Console: Login and select Crawl > Crawl stats.  You'll then be presented with a graph showing how many pages were crawled daily.
  2. Log files:  If you don't have security rights to access your log files ask your webmaster to look at the server logs to see how many pages were crawled. You can then compare GSC against your server logs. It will be interesting to see whether there's a difference between the two.

How to not waste crawl budget

This is all down to SEO best practice and optimising your web pages. Consider the following:

  • Address duplicate content (if necessary, set up canonicals on a number of URLs)
  • Have accessible URL parameters
  • Avoid publishing low quality content
  • Ensure the XML sitemap is showing the correct URLs
  • Optimise images and videos to avoid page load latency
  • Generate 'link juice' by having robust linking
  • Fix broken links
  • Avoid crawl errors! Don't have too many non-indexable pages (3xx, 4xx, 5xx)

How to get the best from crawl budget

If your website has high Domain Authority then it means it's trustworthy and authoritative. The more authority, the more the crawl budget allocation.

Increase your crawl budget. Ensure you build and maintain your website's authority by producing content with value, meets your audience intent and provides solutions and insights.

At Codehouse, our digital experience team, including Google certified experts, are available to help you get the best from your content. We start by analysing your technical SEO and provide you with important details on how to improve your site’s performance. Get in touch to find out more.