Robots.txt

Robots.txt is a plain text file placed in a website’s root directory that provides instructions to web crawlers and search engine bots about which pages or sections of the site they are allowed—or not allowed—to access.

Why It’s Important

• Crawl Budget Management: Search engines allocate a limited amount of time and resources (a “crawl budget”) to scan a site. Blocking unnecessary or low-value pages keeps crawlers focused on high-priority content.
• Server Load Protection: Uncontrolled crawling can overload web servers, especially on large sites or during peak traffic hours. Directing bot behavior helps prevent site slowdowns or crashes.
• Preventing Indexing of Sensitive/Admin Areas: It prevents crawlers from wasting time scanning private directories, backend login screens (e.g., /wp-admin/), internal search result pages, or staging environments.
• Sitemap Location: It acts as a beacon for search engines by pointing crawlers directly to the site’s XML sitemap URL.

Common Rules & Syntax

• User-agent: Specifies which bot the rule applies to (e.g., User-agent: Googlebot or User-agent: * for all crawlers).
• Disallow: Tells bots which paths or files they should not crawl (e.g., Disallow: /admin/).
• Allow: Overrides a disallow rule for a specific sub-folder or file within a restricted directory.
• Sitemap: Declares the location of the site’s XML sitemap (e.g., Sitemap: [https://yourdomain.com/sitemap.xml](https://yourdomain.com/sitemap.xml)).

How It Works & Important Caveat

• First Stop for Crawlers: When a bot visits a site, it reads the robots.txt file at [yourdomain.com/robots.txt](https://yourdomain.com/robots.txt) before crawling any other page.
• Advisory, Not Security: robots.txt is an advisory protocol, not a security wall. Well-behaved search bots respect it, but malicious bots will ignore it.
• Not a Guarantee Against Indexing: If a page is blocked in robots.txt but has incoming links from elsewhere on the web, Google may still index the URL in search results without reading its content. To keep a page entirely out of search results, use a noindex meta tag or require user authentication instead.