AI Crawler Access & Speed Checker
Your robots.txt can say yes while your firewall says no. This sends the real user-agent strings that OpenAI, Anthropic, Perplexity and Google publish, then reports what your server actually returned and how long it took to start answering.
Test a real product or article page, not only the homepage. Bot rules often differ by path, which is the whole reason the robots.txt tester takes paths too.
How to read a refusal
Crawlers blocked, Chrome fine
Your server is treating those user agents differently on purpose or through a bot-management default. This is the case worth acting on, and it is invisible in robots.txt because robots.txt was never consulted.
Spoofed Googlebot also blocked
The site is checking more than the string, so it is fingerprinting. A real crawler arriving from a verified IP range may get through where this probe did not, which makes every verdict here a floor rather than a final answer.
Everything blocked, Chrome included
The finding is about the URL rather than about crawlers. Check the path, the scheme and whether the page needs a session before reading anything into the crawler rows.
The other half of the question
This tool reports what your server did. The AI Robots.txt Tester reports what your rules say, across 28 crawler tokens, naming the rule that won and the line it sits on. Run both when they disagree: a 403 against an allowing robots.txt means the block is in your infrastructure, and an allowed request against a disallowing robots.txt means you are relying on a crawler's good manners.
Questions
Why can't this run in my browser like the robots.txt tester?
Because User-Agent is a forbidden header name in the Fetch specification, so JavaScript on a page cannot set it. A browser asking to be GPTBot still arrives as the browser. Presenting a crawler identity requires a server, which is why this tool calls one and the robots.txt tester does not.
My robots.txt allows GPTBot but this says 403. Which is right?
Both. robots.txt is a request a well-behaved crawler chooses to honour; a 403 is your server or CDN refusing the connection before robots.txt matters. A firewall rule, a bot-management setting, or a WAF signature will block a crawler your robots.txt invites. That gap is the single most common reason a site is absent from AI answers while its owner believes it is open.
Is the speed number what ChatGPT actually experiences?
No, and nothing that runs outside OpenAI's network could tell you that. The measurement is from a Cloudflare edge location to your origin, with Cloudflare's own cache bypassed. It is a fair comparison between your own URLs and a reasonable proxy for whether your origin is slow, but it is not a vendor-side figure.
Why does it measure time to first byte rather than full load?
Because a live fetcher pulling your page mid-answer does not wait around. Time to first byte is when your server starts responding, which is the part a timeout would cut. The tool stops at the response headers and never downloads the body.
Does this send traffic to my site?
Yes: one real request per crawler identity, seven for the access check and three for the speed check, issued sequentially. They will appear in your server logs with those user agents, which is a useful way to confirm your own logging sees them.
User-agent strings taken from OpenAI, Anthropic, Perplexity and Google, read 2026-08-18. Requests are rate limited per IP.