Robots.txt
Updated June 22, 2026 · Reviewed by the Quratic editorial team
Definition
Robots.txt is a root-level text file (robots.txt) that tells compliant crawlers which paths they may or may not fetch. It controls crawl access — not the same as noindex — and sits alongside sitemaps and llms.txt in a site's crawl policy layer.
Crawl permission, not index directive
Disallow: /private/ blocks compliant bots from fetching those URLs. Blocked pages may still appear in Google’s index if linked externally — use noindex on the page itself to remove from index. Common mistakes: blocking CSS/JS (hurts rendering), blocking entire /blog/ or staging paths that leak to production.
Allow: overrides exist for fine-grained rules. Submit sitemap URL in robots.txt for discovery.
How robots.txt differs from llms.txt and noindex
llms.txt is a voluntary publisher guide for LLM crawlers — not a hard block. noindex is an indexing directive on HTML. robots.txt is crawl-level. AI crawlers vary in robots compliance; assume sensitive content needs authentication, not robots alone.
In Asian markets
Multi-locale paths (/jp/, /sg/) sometimes get accidentally disallowed during migrations. After any replatform, verify robots across all locale roots — indexability for localized answer blocks depends on it.
Example
A site accidentally disallows /glossary/. New definitional content never crawls; AI answers keep citing old third-party definitions. Fixing robots.txt and submitting sitemap restores crawl within days.