robots.txt Tester & Generator for Google and AI Crawlers

What robots.txt does (and what it doesn't)

A robots.txt file tells crawlers which parts of your site they may fetch. Google, Bing and the major AI companies read it before they crawl. It is a polite request, not a lock: reputable bots follow it, but nothing technically stops a bot that decides to ignore it.

The most important thing to understand: robots.txt controls crawling, not indexing. If you block a URL, Google won't read the page, but it can still show the URL in search results when other pages link to it. Such results usually appear without a description.

• Want a page out of Google? Use a noindex meta tag or HTTP header instead. The page must stay crawlable, or Google never sees the noindex.
• Want to hide private content? Use a password or login. robots.txt is public, so listing secret folders there actually points people to them.
• Want to save crawl budget on a huge site? That is where robots.txt shines, for example by blocking endless filter or search result URLs.

If a page you care about is missing from Google, robots.txt is one of the first things to check. Our guide on why a page is not indexed walks through the other usual causes.

Where the robots.txt file has to live

Crawlers only look in one place: the root of the host, at exactly /robots.txt. A file in a subfolder is ignored.

Each combination of protocol and host needs its own file. If your shop or booking system runs on a subdomain, give it its own robots.txt. The file must be plain text (UTF-8) and Google reads only the first 500 KiB, which is far more than any normal site needs.

robots.txt syntax with examples

A robots.txt is made of groups. Each group starts with one or more User-agent lines, followed by Disallow and Allow rules. Sitemap lines stand on their own and apply to the whole file.

# Rules for every crawler without its own group
User-agent: *
Disallow: /cart/
Disallow: /search
Allow: /search/help

# A separate group just for GPTBot
User-agent: GPTBot
Disallow: /

Sitemap: https://www.example.com/sitemap.xml

• User-agent names the crawler. * means every crawler that has no more specific group.
• Disallow lists a path the crawler should not fetch. An empty Disallow: blocks nothing.
• Allow makes an exception inside a blocked path.
• Sitemap gives the full URL of your XML sitemap. You can list several.
• # starts a comment. Field names are not case-sensitive, but paths are.

Crawl-delay is ignored by Google. Bing respects it, but most small sites don't need it. The old Noindex: line inside robots.txt has not worked in Google since 2019.

How robots.txt matching works

This tester follows RFC 9309, the official robots.txt standard, and the way Google applies it:

1. The crawler picks the group with the most specific matching User-agent. Only if none matches does it use the * group. It never combines both.
2. Inside that group, every rule whose path matches the start of the URL path is a candidate.
3. The longest matching rule wins. If an Allow and a Disallow are equally long, Allow wins.
4. * matches any sequence of characters, and $ marks the end of the URL.
5. /robots.txt itself is always allowed.

If you add a group for Googlebot, Googlebot ignores everything in the * group. Repeat any shared rules inside the Googlebot group.

Common robots.txt mistakes

• **Disallow: / left over from staging.** Developers often block the test site and copy the file to the live site at launch. One line hides the whole site from crawlers.
• Blocking CSS and JavaScript. Google renders pages like a browser. If it can't load your styles and scripts, it may misjudge layout and mobile friendliness.
• Blocking pages you want removed. A blocked page can't show Google its noindex, so it can linger in results. Allow crawling, add noindex, and wait.
• Wrong case. Disallow: /Images/ does not block /images/.
• Misreading wildcards. Disallow: /blog also blocks /blog-news and /blogroll. Add a trailing slash if you mean the folder only.
• A broken robots.txt. A 404 is fine and means “crawl everything”. But if the file returns a server error (5xx), Google may treat the whole site as disallowed for a while and pause crawling.

AI crawlers: training bots vs. search bots

AI companies run different bots for different jobs. Some collect content to train models. Others fetch pages live so an AI assistant can answer a question and link to you. Blocking the first kind does not have to cost you visibility in the second.

Google-Extended is not a separate crawler. Googlebot does the fetching, and the token only tells Google whether the content may be used for its AI models. Blocking it has no effect on Google Search rankings. Applebot-Extended works the same way for Apple: regular Applebot keeps powering Siri and Spotlight.

If you block GPTBot but allow OAI-SearchBot, your site can still appear and be linked in ChatGPT search. Bot names and behavior change, so check the official lists such as OpenAI's crawler docs now and then. Note that user-triggered fetchers like ChatGPT-User may not always follow robots.txt the same way as automatic crawlers.

Want AI assistants to recommend your business? Crawl access is step one. Our guide on getting recommended by ChatGPT covers the rest, and the llms.txt generator creates an optional summary file for AI tools. The RankTutor iPhone app can also check whether the ChatGPT, Claude and Perplexity crawlers can read your site.

A ready-made robots.txt for most small business sites

This version lets search engines and AI search bots in, keeps AI training bots out, and hides typical clutter. Adjust the paths to your site, or build your own with the generator above.

User-agent: *
Disallow: /cart/
Disallow: /checkout/
Disallow: /*?s=

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: meta-externalagent
User-agent: Bytespider
Disallow: /

Sitemap: https://www.example.com/sitemap.xml

Several User-agent lines in a row share one group, which keeps the file short. Bots without their own group, including OAI-SearchBot, Claude-SearchBot and PerplexityBot, fall back to the * group and may crawl everything except the cart, checkout and search pages.

How to test robots.txt in Google Search Console

Google retired its old robots.txt Tester in 2023. The replacement is the robots.txt report:

1. Open Google Search Console and select your property.
2. Go to Settings and open the robots.txt report.
3. Check which robots.txt files Google found, when it last fetched them, and any warnings or errors.
4. After a fix, use Request a recrawl so Google picks up the new version faster.
5. To test a single page, run the URL Inspection tool. It tells you if the URL is blocked by robots.txt.

Google usually caches robots.txt for up to 24 hours, so changes are not instant. For a broader look at your pages, run a free SEO check, and if you've just unblocked pages, a sitemap and IndexNow can help search engines notice sooner.