The robots.txt file is one of the oldest and most misunderstood tools in SEO. A single wrong line can accidentally hide your entire website from Google, while a well-configured file helps search engines crawl efficiently. This guide explains exactly what robots.txt does, what it can't do, and how to set it up correctly.
What Is a Robots.txt File?
Robots.txt is a plain text file placed in the root of your website at yourdomain.com/robots.txt. It gives instructions to web crawlers (also called bots or spiders) about which parts of your site they may or may not crawl. When a search engine visits your site, the robots.txt file is one of the first things it checks.
How Robots.txt Works
The file is made up of simple rules grouped by user-agent (the name of the crawler). The two most common directives are:
- User-agent: specifies which crawler the rule applies to. An asterisk (
*) means all crawlers. - Disallow: tells the crawler not to access a specific path.
- Allow: explicitly permits a path, often used to create exceptions inside a disallowed folder.
For example, this blocks all crawlers from an admin folder while allowing everything else:
User-agent: *
Disallow: /admin/
What Robots.txt Can and Cannot Do
It's crucial to understand the limits of robots.txt. It controls crawling, not indexing. If another site links to a page you've disallowed, Google may still index the URL (without crawling its content). To truly keep a page out of search results, use a noindex meta tag on the page itself rather than relying on robots.txt.
Robots.txt also offers no security. Anyone can read your robots.txt file, so never use it to "hide" sensitive pages — that actually advertises their location. Protect private content with passwords or proper access controls instead.
How to Create a Robots.txt File
You can write robots.txt by hand, but it's easy to make a syntax mistake. Our free Robots.txt Generator builds a valid file for you based on simple choices. The typical process:
- Step 1: Decide which folders or pages you want to keep crawlers out of.
- Step 2: Use the Robots.txt Generator to produce the file.
- Step 3: Add your sitemap line so crawlers can find your XML sitemap.
- Step 4: Upload the file to your site's root directory.
A Safe Default Robots.txt
For most websites, you want search engines to crawl almost everything. A safe, permissive starting point looks like this:
User-agent: *
Disallow:
Sitemap: https://yourdomain.com/sitemap.xml
An empty Disallow: line means "nothing is blocked." This lets crawlers access your whole site while still pointing them to your sitemap.
The Most Dangerous Robots.txt Mistake
The single most damaging error is accidentally blocking your entire site, usually with this line left over from a development environment:
User-agent: *
Disallow: /
That single forward slash tells every crawler to stay away from every page. Sites have vanished from Google overnight because of it. Whenever you launch a new site or move from staging to production, double-check that this line is removed.
How to Test Your Robots.txt
After uploading, always verify your file works as intended. Visit yourdomain.com/robots.txt in a browser to confirm it loads. Then use Google Search Console's robots.txt report to check for errors and confirm that important pages aren't being blocked. It's far better to catch a mistake here than to discover it weeks later when traffic has dropped.
Robots.txt and Crawl Budget
On large sites, robots.txt helps manage "crawl budget" — the limited number of pages a search engine will crawl in a given period. By disallowing low-value sections such as faceted-search URLs, internal search results, or endless filter combinations, you steer crawlers toward the pages that actually matter. For small sites, crawl budget is rarely a concern, so keep your file simple.
Final Thoughts
Robots.txt is powerful precisely because it's simple — and that simplicity is also its danger. Use it to guide crawlers away from genuinely unimportant areas, always include your sitemap reference, and never rely on it for privacy or to control indexing. Build a clean file with our free Robots.txt Generator, test it, and you'll have one less technical SEO worry.