Crawl budget: why most of your generated pages never get indexed
Indexation rates on a programmatic batch commonly sit between 20% and 40% in year one. That is not a technical inevitability: it is an economic decision Google makes, and every term in it is actionable.
What crawl budget is made of
- Capacity limit. What your server tolerates without degrading. Rising response times mechanically reduce crawl frequency.
- Crawl demand. How interested Google is in returning. It follows URL popularity, genuine freshness, and the observed value of pages already crawled.
On generated sites the binding constraint is almost always demand. The server copes; the interest is missing. Shaving milliseconds off a site Google does not want to crawl more changes nothing.
Read coverage statuses in the right order
| Status | Meaning | Dominant cause |
|---|---|---|
| Discovered — currently not indexed | URL known, never crawled | Insufficient budget or no internal links |
| Crawled — currently not indexed | Page read, judged not worth indexing | Perceived value too low |
| Duplicate without user-selected canonical | Content judged identical to another page | Template too uniform |
| Alternate page with proper canonical tag | Consolidation as instructed | Expected, not a problem |
The distinction between the first two rows is the whole diagnosis. "Discovered" is a discovery problem: sitemap, internal links, depth. "Crawled" is a content problem: the page was read and rejected. The remedies share nothing.
Six actions that raise indexation rate
1. Reduce click depth
Moving a page from depth five to depth three does more than any content edit. See internal linking at scale.
2. Remove worthless URLs
Every crawled URL that does not deserve indexing burns budget. Faceted filters, internal search results, sort parameters: block or consolidate them.
3. Publish in waves
A 300-page wave gets absorbed. A 20,000-page overnight drop creates a queue whose tail is never processed.
4. Prune zero-impression pages
After six months without a single impression, a page earns nothing and costs crawl. Removing it improves the rate for the rest of the batch — the ratio matters as much as the volume.
5. Strengthen external signals to the pillar
An external link to the pillar redistributes through the silo via internal links. On a young site this is the highest-leverage move available.
6. Stabilise response times
Not to shave milliseconds, but to avoid spikes: unstable response times lower the capacity limit durably.
Four things that do not work
- Resubmitting the sitemap. With no substantive change, the outcome is identical.
- Manual URL inspection requests. Useful for a handful of URLs, irrelevant for a batch.
- Adding footer links. The signal is too weak to change perceived depth.
- Republishing with a new date. Changing
lastmodwithout changing content degrades the signal sitewide — see correct lastmod usage.
How long before concluding
On an established domain a wave is largely processed within four to eight weeks. On a young domain, expect three to six months before the indexation rate is interpretable at all. Drawing conclusions at two weeks reliably leads to undoing what would have worked.
Instrumenting the wait is covered in measuring with the Search Console API.
Frequently asked questions
What does the status Discovered — currently not indexed mean?
Google knows the URL but has not crawled it. That is a discovery and prioritisation problem: weak internal linking, excessive click depth, or a saturated crawl budget.
Should I delete pages that are not indexed?
After six months with no impressions and no indexation, usually yes. They consume crawl budget and lower the average perceived quality of the site.
Does crawl budget matter for small sites?
Rarely. Below a few thousand URLs it is seldom the limiting factor. It becomes decisive on large generated sites.