Crawl budget
Updated June 22, 2026 · Reviewed by the Quratic editorial team
Definition
Crawl budget is the approximate number of pages search engine bots will fetch on a site within a given period — influenced by site size, health, server speed, and link signals. Large or low-quality sites may not get every URL crawled frequently, delaying indexation and updates.
Finite bot attention
Googlebot allocates crawl capacity per site. Faceted URLs, infinite calendar archives, and soft-404s waste budget on junk instead of money pages. Consolidation via canonical URLs, robots.txt blocks on low-value paths, and strong internal linking to priority URLs helps bots spend budget where it matters.
Most mid-size marketing sites are not crawl-budget constrained — but enterprise and ecommerce catalogs in Asia often are.
How crawl budget differs from indexability
Indexability is permission and eligibility; crawl budget is frequency and capacity. A page may be indexable yet crawled rarely — stale content in AI answers until refresh. Updated dateModified, sitemaps, and internal links accelerate recrawl.
In Asian markets
Multi-locale sites multiply URL count sixfold — budget dilution risk if every filter and tag generates indexed URLs. Audit locale sites separately; Japanese faceted navigation on marketplace-style catalogs is a common budget sink.
Example
An ecommerce brand blocks sorted query parameters in robots.txt and canonicalizes variants. Crawl stats show more daily fetches on product and FAQ URLs; AI-generated answers update pricing within weeks instead of months.