AI Robots.txt Tester
Paste a robots.txt and see how every AI and search crawler reads it, side by side. The verdict is the easy part. This shows the rule that won, the line it came from, and why it beat the others, plus the two mistakes a live access check cannot catch because your file is being obeyed exactly as written.
Most sites do not let a browser read their robots.txt directly, so Fetch succeeds on a minority of domains. When it fails, Open in tab shows the file so you can copy it.
How the winning rule is decided
Both steps come from Google's published robots.txt specification, and step one is where most surprises live.
- One group is picked and the rest are ignored. A crawler finds the group whose user-agent token is the most specific match for its own name, then reads only that group. Google's wording is blunt: other groups are ignored. Position in the file makes no difference, and where several groups name the same crawler, their rules merge.
- Inside that group, the longest path wins. Rules rank by the length of the rule path, so
Allow: /blog/helpbeatsDisallow: /blog/for/blog/help. Where two conflicting rules are the same length, including rules carrying wildcards, the least restrictive applies, which means Allow takes ties.
Also handled: a Disallow: line with no path is ignored rather than read as allow-all, * matches any run of characters, $ anchors to the end of the URL, and rule paths are case-sensitive while field names and user-agent tokens are not.
Three tokens that do not do what their names suggest
ChatGPT-User
OpenAI's crawler documentation says that because these actions are initiated by a user, robots.txt rules may not apply. It also says this is not the agent deciding whether your content can appear in ChatGPT search, and names OAI-SearchBot for that.
Google-Extended
Not a crawler at all. Google says it has no separate user-agent string, that crawling happens under existing Google user agents, and that the token has no effect on Google Search or other products. Blocking it changes nothing about crawling, Search, or AI Overviews.
AdsBot
Google states that AdsBot ignores the global * user agent with the ad publisher's permission. A blanket User-agent: * plus Disallow: / will not stop it, so only a group naming AdsBot does.
One more thing worth keeping in view: Disallow controls crawling, not indexing. Blocking a URL leaves any existing index entry standing and hides the noindex that would have removed it. That mechanism gets its own treatment in a separate piece.
Questions
Why does my 'User-agent: *' rule not apply to GPTBot?
Because a crawler obeys only the single most specific group matching its name, and every other group is ignored. If your file has a GPTBot group anywhere in it, GPTBot reads that group and nothing else, so the * rules do not stack on top of it. This tool names the ignored group and its line number whenever that happens.
Does blocking Google-Extended remove me from AI Overviews?
No. Google's documentation states that Google-Extended has no effect on Google Search or other products, and that it has no separate user-agent string of its own: crawling still happens under normal Google user agents, and the token acts only as a control for Gemini Apps and Vertex AI Agents. Disallowing it does not reduce crawling and does not change Search or AI Overviews.
Will disallowing ChatGPT-User stop ChatGPT from reading my page?
Not reliably. OpenAI's crawler documentation states that because ChatGPT-User actions are initiated by a user, robots.txt rules may not apply. OpenAI also says ChatGPT-User is not what determines whether content can appear in ChatGPT search, and points to OAI-SearchBot for managing that. The tool flags both points on the ChatGPT-User row.
Does a Disallow rule remove a URL from search results?
No. Disallow stops the crawl, which leaves any existing index entry standing and also hides the noindex tag that would have removed it. Blocking and de-indexing are different jobs with different tools.
Is my robots.txt uploaded anywhere?
No. Parsing and matching run entirely in your browser. The optional Fetch button makes a request straight from your browser to the domain you typed, and most sites do not permit that, which is why pasting is the primary path.
How does the tool decide which rule wins?
It follows Google's published robots.txt specification: the most specific rule by length of the rule path, and where rules conflict, including rules with wildcards, the least restrictive one. A Disallow line with no path is ignored, wildcards and end-anchors are supported, and rule paths are matched case-sensitively while field names and user-agent tokens are not.
Precedence rules and crawler behaviour verified 2026-08-18 against Google's robots.txt specification,Google's crawler overview,Google's common crawlers reference, and OpenAI's crawler documentation. Crawler tokens change; re-check one before making a high-stakes change on it.