Updated
Crawl budget is the number of pages Googlebot can and wants to crawl on a website within a given time. It combines crawl capacity (how much your server can handle) and crawl demand (how important and fresh Google considers your content).
Who needs to care?
For most websites with a few hundred or a few thousand pages, crawl budget is not an issue. It matters for large online stores, portals with many filters and sites with huge numbers of automatically generated URLs.
What wastes crawl budget
- Filter and parameter URLs with no unique value
- Redirect chains and broken links
- Duplicate content without a canonical
- Slow servers and 5xx errors
- Endless calendars or internal search pages
How to optimise it
- Block unimportant areas in robots.txt
- Maintain a clean XML sitemap
- Point internal links at important pages
- Improve server response time
The Crawl stats report in Google Search Console shows how often Google visits your site.
Digitalman optimises crawling for large websites. Learn more on our on-page SEO page.
Digitalman can help
Want a technically sound website for Google?
We check indexing, structure, speed and content and implement the fixes for you. The first 60 minutes of consultation are free.