robots.txt Basics: The First Step in Controlling Crawlers

Overview

Many teams treat robots.txt as a “universal switch”: want to hide a page? Write a robots.txt rule. Don’t want AI to see your site? Disallow everything. This understanding is often inaccurate, and it can actively hurt healthy search and GEO performance.

This lesson starts from the official guidance and explains robots.txt’s purpose, boundaries, and pitfalls in full. Google’s definition of robots.txt is very clear: it is a file that tells crawlers which URLs they may access and which they should not, and its primary use is to manage crawl traffic and keep the server from being overwhelmed by unnecessary requests. At the same time, Google stresses that robots.txt is not a mechanism for hiding a web page from Google—if you genuinely want to keep a page out of search results, you need to use noindex or password protection (Per: Google Search Central).

Core Concepts

This lesson is organized around five core concepts.

1. robots.txt is “crawl control,” not “index control”

This is the single most important sentence in the lesson. A page that is merely disallowed from crawling in robots.txt can still be discovered—through external links, for example—and may appear in search results in a “URL-only” form (Per: Google Search Central). In other words, blocking crawling does not equal preventing indexing.

2. What robots.txt can control

• Whether a given user-agent is allowed to crawl certain paths
• Whether to restrict access to specific resource directories
• Basic crawl isolation
• Different rules for different bots

3. What robots.txt cannot guarantee

• It cannot guarantee that every crawler will obey it
• It cannot guarantee that a page will absolutely never be indexed
• It cannot keep sensitive information secure
• It cannot replace login permissions, noindex, authentication, or response-header controls

Google’s official documentation makes clear that crawlers do not all support the same syntax, and that robots.txt relies on crawlers voluntarily complying (Per: Google Search Central).

A minimal robots.txt example looks like this:

User-agent: *
Disallow: /admin/
Allow: /

Sitemap: https://example.com/sitemap.xml

4. The new significance of robots.txt in GEO

In the past, the focus was mainly on Googlebot and Bingbot; now you also need to account for a wider range of visitors:

• AI search crawlers
• AI training crawlers
• User-triggered fetchers
• Special-purpose crawlers

This means designing robots.txt rules is no longer just a question of “indexed or not”—you need to think in layers, organized by purpose.

5. Common misconceptions

• Blocking CSS / JS / resource files site-wide, which makes pages hard to understand
• Trying to hide content but only writing a robots.txt rule
• Blocking AI bots across the board, which ends up hurting AI search visibility
• Updating robots.txt without verifying and monitoring the result afterward

Putting robots.txt back into the full picture of control mechanisms

What teams most often confuse is not “whether a tool exists” but “what each tool is actually for.” robots.txt is just one of several technical control mechanisms. It addresses “can it be crawled,” whereas noindex addresses “can it enter the index,” preview controls (such as nosnippet and max-snippet) address “how much can be displayed,” and llms.txt addresses “how models can understand the site more quickly.” Only by understanding this map of relationships can a team stop throwing every problem at robots.txt.

Exercise

Study five robots.txt examples: a corporate website version, a documentation site version, a SaaS product site version, a blog version, and an example of mistakes. Then determine which directories should be open, which should be handled with caution, and which rules might inadvertently harm SEO / GEO.

Deliverables

• “robots.txt Foundational Mental Model”
• “robots.txt Common Misconceptions Checklist”
• “Baseline robots Strategy Template”