# Blocking the AI crawlers did not stop the citations, and the bot table explains why

> Around 75% of sites blocking OpenAI or Google AI bots were still cited. Their own table shows why: nobody blocked the retrieval bot, only the training bots.

Canonical: https://brandonlazovic.dev/articles/blocking-training-bots-left-retrieval-open/  
Author: Brandon Lazovic  
Published: 2026-08-26

## The short version

- Publishers blocked the bots that gather training data and left the bot that feeds live retrieval wide open, then the citations were measured on the retrieval path.
- The study's own table reports 0% of cited domains blocking Googlebot, because blocking it would also remove them from Google search, and Google's AI surfaces cite from that index.
- Three figures in the study do not reconcile, including two different bots at two different companies reporting an identical citation count and an identical percentage.
- On my own log, 43% of requests asked for a URL that does not exist, so anyone counting AI crawler activity from raw server logs is inflating the number.

A study of citations drawn from 3,600 prompts across ChatGPT, Gemini, AI Overviews and AI Mode found that around three quarters of the sites blocking OpenAI's or Google's AI bots still appeared in AI citations. [1] That reads like proof that robots.txt has stopped working, and a duller explanation sits in the study's own bot table. Almost every site in the sample blocked the crawlers that gather training data, and none of them blocked the crawler that feeds live retrieval, because blocking that one also removes you from ordinary search results. Blocking a training crawler was never going to suppress a citation that arrives through a retrieval index. So the conclusion is narrower than "robots.txt is broken": match the control to the behavior you are trying to reach, because these numbers measure what happens when a crawl control is pointed at something that is not crawling you.

> The study reports 0% of cited domains blocking Googlebot, and Google's AI surfaces cite from that same index.

## Which bot does what, and why the distinction decides the answer?

Training crawlers gather corpus data for model training, and the study names GPTBot for OpenAI and Google-Extended for Google. Retrieval crawlers fetch pages to support a live answer, and there the study names ChatGPT-User and OAI-SearchBot for OpenAI, with Googlebot filling that role for Google's AI surfaces. Citations come from the retrieval path, which the study states directly: links and sources typically come from a retrieval pipeline rather than the training dataset. [1]

## What did the study find, and what do its numbers actually cover?

Across the top 50 news sites in the study's own breakdown, 70.6% of the sites blocking ChatGPT-User still appeared in the citations, 82.4% of those blocking OAI-SearchBot, and 88.2% of those blocking Google-Extended. [1] The overall summary is that around 75% of sites blocking OpenAI or Google AI bots still turned up. Vince Nero, who ran the analysis for BuzzStream using Citation Labs' XOFU tool, gives cnbc.com as the worked case: it blocked ChatGPT-User, GPTBot and OAI-SearchBot, and still appeared 1,298 times in the dataset. [1]

Then comes the row that explains the Google half of the study. Under the overall breakdown, Googlebot shows **0%** of cited domains blocking it, with the study's own parenthetical: "must be indexable". [1] Nobody blocks Googlebot, because doing so removes them from Google search entirely. Google's AI Overviews and AI Mode cite from that same index. So the publishers in this sample closed the training door and left the retrieval door open, and the study then measured the retrieval door. My read: that makes most of the headline arithmetic instead of a surprise, and the study states both halves of it without ever joining them.

![A figure stating that around three quarters of sites blocking OpenAI's or Google's AI training bots still appeared in AI citations, followed by a giant 0%, labeled of cited domains blocking Googlebot. A callout below quotes the study's own parenthetical, must be indexable, and explains that blocking Googlebot would remove a site from Google Search entirely, while Google's AI Overviews and AI Mode cite from that same index.](/figures/blocking-training-bots-left-retrieval-open/googlebot-zero-row.png)

*BuzzStream's analysis for Citation Labs, run by Vince Nero; overall breakdown table, read 2026-08-18.*

One date to keep in view. The page reads "Last Updated: April 8, 2026", so this is roughly four months old, and vendor robots documentation has moved since. [1]

## Which figures in the study do not reconcile?

Three, and each is visible to any reader who opens the post. First, the prose says "92.30% of the sites that block Google-Extended still appear", while the overall breakdown table gives 88.2% for the same bot. [1] Both numbers are in the same document, so I would cite the range rather than either figure.

Second, and this is the one that would change a conclusion. In the share-of-citations table, GPTBot shows 5,362 citations at 95.4%, and Google-Extended shows 5,362 citations at 95.4%. [1] An identical count and an identical percentage for an OpenAI training bot and a Google training bot is far more likely to be a duplicated cell than a coincidence. The two OpenAI retrieval rows differ from each other plausibly, at 3,932 and 3,913, so those are the figures I would lean on.

Third, the headline sample and the analyzed sample differ by three orders of magnitude. The study opens on "4 million citations from 3,600 prompts", and the share-of-citations table totals in the thousands because this section runs on the top 50 news sites. [1] The four million is the dataset behind the wider series. It is not the base for these percentages, and quoting it as though it were makes the finding sound far more heavily evidenced than it is.

None of this makes the direction wrong. Flagging it is how a soft number avoids being laundered into consensus by everyone who cites the summary without opening the tables.

## Why is a user-triggered fetch not a crawl?

A user-triggered fetch happens because a person asked a question, so the vendor treats it as a user action rather than as crawling. OpenAI states this on its own crawler documentation page, and the wording is worth reading exactly, because a paraphrase softens it. [2]

> "ChatGPT-User is not used for crawling the web in an automatic fashion. Because these actions are initiated by a user, robots.txt rules may not apply. ChatGPT-User is not used to determine whether content may appear in Search. Please use OAI-SearchBot in robots.txt for managing Search opt outs and automatic crawl."

Read 2026-08-18 from OpenAI's live page. [2] Three separate things sit in that paragraph. Your robots.txt may not bind this agent at all. The agent is not what decides whether you appear in ChatGPT search. And the token that does decide is a different one, which means a publisher who blocked ChatGPT-User to stay out of ChatGPT has quite possibly blocked nothing and left OAI-SearchBot untouched.

That single distinction would produce a null result by design, with no disobedience required anywhere. It is documented behavior rather than evasion, and it is checkable: the [AI Robots.txt Tester](/tools/ai-robots-txt-tester/) on this site flags ChatGPT-User as one of three tokens that do not do what their names suggest.

The study reaches for circumvention, citing a Reuters report and its own correlation. [1] A correlation measured across a sample where the retrieval path was never blocked cannot separate disobedience from a control that was pointed at the wrong bot, and the documented explanation needs no disobedience at all. My read: reach for the boring explanation first, and keep the interesting one for evidence that rules the boring one out.

The study offers two more explanations worth carrying. Some AI retrieval works from search-result titles, URLs and snippets without ever fetching the underlying page, which is why a cited source sometimes does not contain the sentence attributed to it. [1] A robots.txt edit cannot reach a snippet captured before the edit. And the easiest explanation, that these pages were crawled before the block went up, fails on the study's own data: only 15% of the cited publications existed before ChatGPT and about 30% before AI Overviews launched, so the citations are drawing on recent material. [1] Common Crawl does not rescue the story either, since about 70% of the sites block CCBot too. [1]

## Can you separate the bot types in your own logs?

Yes, and it takes one pass over the user-agent column. Over 33 days to 17 August 2026, across 24,125 logged bot requests on this site, on-demand assistant fetchers accounted for 2,324 requests, or 9.6%. Declared training crawlers accounted for 8,792, or 36.4%. Everything else, including ordinary search crawlers and everything my classifier does not name, made up 13,009 requests at 53.9%.

My classifier is a keyword match on user-agent strings, so any token missing from either list lands in the third bucket, which is why that bucket is the largest. And these figures moved while I was writing, because the log updates daily. Re-derive them rather than quoting mine.

## What does your log say once you throw out the requests for files you never had?

Something uncomfortable, and it changes every bot count you have ever read from a raw log. Of 21,432 requests to this site over 25 days, **9,207 of them, or 43%, asked for a URL that does not exist**, spread across 874 paths the build never emits. The leaders are `/api/v1/env`, `/.npmrc`, `/firebase-service-account.json`, `/config.env` and `/app-config.json`. Those are credential probes wearing crawler clothing.

**Ghost crawl traffic** is the handle worth having for this: logged requests for URLs a site has never published, inflating any bot-activity count taken from a raw log. On this site it is nearly half of everything.

The method matters more than the number. My first pass used a keyword denylist of attack shapes and reported 21% noise, having filed `/api/v1/env`, `/.npmrc`, `/firebase-service-account.json`, `/.boto`, `/app-config.json` and `/runtime-config.js` as legitimate crawling. A denylist can only ever be as complete as the last attack you have seen. The version I trust allowlists instead: a path counts as real only if the build actually emits it, which is exact and needs no judgment. If you are measuring AI crawler activity from server logs, join against your sitemap before you count anything, and expect your headline number to fall.

## Which control reaches which behavior?

Pick the control that reaches the layer you care about, then verify it rather than assuming. To stay out of training corpora, robots.txt is the right tool, and this study broadly supports that it works for that purpose. To stop live retrieval, robots.txt is the wrong layer, which is why publishers reach for CDN bot rules instead. To limit snippet-level reuse you need search-result controls, since no crawl directive reaches text that was captured months ago.

Vince Nero's own conclusion for practitioners is that whether a news site blocks AI crawlers should not change your strategy, and that chasing coverage on specific sites is the wrong focus. [1] I would add the narrower version for anyone auditing their own property. This site's earlier piece on [why Disallow is not noindex](/articles/robots-disallow-is-not-noindex/) showed that a Disallow stops the fetch and leaves the index entry standing. The same lesson applies one layer out. The file governs crawling, and crawling is only one of the ways your words end up in an answer.

## Sources

1. BuzzStream: Do News Publishers That Block AI Crawlers Get Cited Less Often by AI? — https://www.buzzstream.com/blog/news-block-ai-bots-citations/
2. OpenAI: Overview of OpenAI Crawlers — https://developers.openai.com/api/docs/bots