Crawl Budget Explained (and How to Fix It)

Crawl budget is the number of URLs Googlebot is willing to crawl on your site in a given day. If you have more pages than budget, some of them — often your important new ones — simply don't get crawled.

For most small sites it's a non-issue. But it's worth understanding, because wasting crawl budget is a silent way to delay your own indexing.

When crawl budget actually matters

Google's own guidance: crawl budget "mostly matters for big sites." If you have fewer than a few thousand pages, Google can generally crawl everything on schedule. The problems start when you have:

• Thousands of pages (e-commerce, directories, programmatic SEO).
• A large volume of low-value or duplicate URLs.
• A slow server that makes crawling expensive.

What wastes crawl budget

Four things consume budget without returning value:

• Faceted navigation — /?color=red&size=large generates near-infinite URL variations.
• Duplicate content — printer versions, session IDs, tracking parameters.
• Soft 404s — pages that return 200 but are effectively empty.
• Hacked or spam pages — Google slows down crawling a compromised site.

How to fix it

1. Block junk in robots.txt — disallow parameter URLs, search results, and anything that shouldn't be indexed.
2. Canonicalize duplicates — point every variation at the one URL that should rank.
3. Fix soft 404s — return a real 404 or 410, or add content.
4. Speed up the server — a slow response makes Google crawl fewer pages. Fast pages also help on large sites.
5. Keep the sitemap clean — list only URLs you want indexed.

The bigger lesson

Crawl budget is mostly a symptom of a messy site, not a thing you tune directly. Clean up the junk and your important pages get crawled sooner. And for the pages you need in Google now — a new post, a product launch — don't wait for budget to free up; push them. Notify Googlebot directly and skip the crawl queue.

FAQ

Does my small site need to worry about crawl budget?

Usually no. Google says a current sitemap and regular Page Indexing report checks are enough for smaller sites.

Can I increase my crawl budget?

You can't set it directly, but you can make better use of it: remove junk URLs, speed up the server, and keep your sitemap clean so Google spends its crawl on pages that matter.

What's a soft 404?

A page that returns HTTP 200 but has no real content (an empty category, a "no results" page). Google treats it as a waste of crawl budget.