How ChatGPT search finds pages: its own crawler, plus Cloudflare's view of the web. On my site, that crawler trailed Googlebot on 18 of 24 new articles

ChatGPT search finds a page when OpenAI’s search crawler, a program that requests web pages so they can be read and stored, fetches it. In a research pilot announced in July 2026, Cloudflare, a network company that says more than 20% of the web sits behind it, helps that crawler choose.² Cloudflare tells OpenAI which addresses lead nowhere and which pages have changed, and says the pilot brought hundreds of millions of new pages into OpenAI’s index, the store of pages a search engine can answer from. The same post says OpenAI “independently evaluates and ranks” those pages.¹

On my own site, OpenAI’s crawler reached most new articles after Googlebot, Google’s crawler, or was not seen in the sampled log at all. So on this site, the wait for OpenAI’s crawler ran longer than the wait for Google’s.

On this site, OpenAI’s search crawler reached a new article on or before Googlebot for 6 of the 24 articles Googlebot reached, and after it, or not in the sampled log at all, for the other 18.

How does ChatGPT search find pages?

ChatGPT search finds pages through OpenAI’s search crawler, which requests a page so OpenAI can read it and add it to the index that ChatGPT’s search answers draw on. In my site’s logs that crawler identifies itself as OAI-SearchBot and links to openai.com/searchbot.³ A pilot with Cloudflare, announced in July 2026, also tells the crawler which pages are worth requesting.¹²

Cloudflare’s results post, published October 15 and written with two OpenAI authors, describes the job the signals do. They “help OpenAI’s search crawler discover more useful pages across the web to surface to ChatGPT users.”¹ The crawler still has to visit. The signals change which addresses it spends its visits on, and when it comes back.

OpenAI is not the only AI company with an index of its own to fill. DeepSeek opened its search and fetch service to everyone on October 17, describing it as “powered by our in-house search engine, serving hundreds of millions of requests daily worldwide.”⁴

What does Cloudflare tell OpenAI in the pilot?

Cloudflare’s pilot with OpenAI sends three kinds of signal about web pages, drawn from Cloudflare’s network. One kind filters out addresses that do not exist or only redirect, one says when a page has changed, and one, still in testing, flags potentially harmful pages. The press release that announced the pilot on July 8 also names traffic quality.¹²

The table below gives each signal and the words Cloudflare used for it.

Signal	What Cloudflare says it does	Source
Filtering	“keeps crawlers away from pages that don’t exist or that redirect elsewhere”	Results post1
Change	“tell them when a page is worth another visit, and when it isn’t”	Results post1
Harmful pages, in testing	“can help identify pages that should be kept out of an index”	Results post1
Freshness, traffic quality and page changes	Named as “real-time network insights” that aim to improve AI systems’ indexing and crawling	Press release2

The filtering signal targets patterns most site owners will recognize. “Bots repeatedly ask for files that aren’t there,” the post says, and “some sites redirect almost any address to their homepage.”¹ The post puts the pilot’s reach at “~1% of Cloudflare’s total network”, and it says the signals are metadata about the web, with no page content shared for model training.¹

Think of it like a mail carrier who gets a note each morning listing which houses on the route are empty and which ones have new residents. The carrier still walks to every door they visit, but the note decides which doors are worth the walk.

Does OpenAI rank pages using Cloudflare’s signals?

Cloudflare’s results post does not say whether OpenAI ranks pages using Cloudflare’s signals. It says the signals help OpenAI’s search crawler find pages, and that “OpenAI then independently evaluates and ranks how those pages appear based on how relevant and helpful they are.” The post gives finding pages to the signals and ranking to OpenAI, and says nothing more about how the two connect.¹

My reading, which neither Cloudflare page makes, is that the pilot gives OpenAI the kind of view a company running its own browser and search results might build for itself, a running sense of which addresses are real, which are dead and which just changed. Google runs both, though whether it uses either for crawling is outside what these sources cover. OpenAI is getting a comparable view from a network instead.

How ChatGPT picks among pages it has already found is a separate question, covered in the article on ChatGPT’s source selection policy.

Does OpenAI’s search crawler reach new pages before Googlebot?

On this site, OpenAI’s search crawler reached a new article before Googlebot, or on the same day, for 6 of 24 articles. Googlebot came first on 14, and OpenAI’s crawler was not seen in the sampled log on 4. The median wait after an article went live was two days for Googlebot and three for OpenAI’s crawler, across the 24 articles Googlebot reached from September 21 to October 20.³

The table below sorts the 28 articles in the log by who got there first. The late group went live October 21 to 23, so it had only two to four days of log after going live.

Group	Articles	Googlebot reached	OpenAI’s crawler first	Same day	Googlebot first	OpenAI’s crawler not seen	Median wait, Googlebot	Median wait, OpenAI’s crawler
September 21 to October 20	25	24	5	1	14	4	2 days (24 seen)	3 days (21 seen)
October 21 to 23	3	3	0	0	0	3	1 day	none seen

One article from the main group, published October 9, was reached by OpenAI’s crawler two days later and showed no Googlebot request in the sampled log. It sits outside the count of 24, which covers only the articles Googlebot reached.

In the first week the order ran the other way, which I noticed only after seeing the data. OpenAI’s crawler reached all five articles published September 21 to 25 before Googlebot did. Googlebot requested article pages on only nine log rows across the whole site from September 20 to 29, so that week says more about Googlebot being idle than about OpenAI being quick.

Once Googlebot was crawling normally, it reached the new article first on 14 of 19, tied once, and OpenAI’s crawler did not show up on four. Googlebot’s own waits are covered in the article on how long new pages wait for Googlebot.

How did I count the first visits?

I counted first visits from this site’s crawler log, a daily record of bot requests built from Cloudflare’s analytics. Each row holds a date, a user agent (the label a program sends to say what it is) and an address. For every article dated September 21 to October 23, I took the earliest date each crawler requested the article’s own address.³

Googlebot means a user agent containing Googlebot/2.1, which covers its phone and desktop forms and leaves out Google’s image crawler. OpenAI’s search crawler means a user agent containing OAI-SearchBot. Requests for an article’s Markdown copy and its images do not count. The wait is the first request’s date minus the publish date in whole days, and articles go live at 05:00 UTC (Coordinated Universal Time), the same clock the log uses.

I wrote the rule for a failed claim before running the count. If OpenAI’s crawler had arrived on or before Googlebot for half or more of the articles Googlebot reached, 12 of 24, the claim that it trails Googlebot would have failed. I checked two articles against the raw log rows by hand, and both matched.

What can one site’s crawler log not tell you about the pilot?

One site’s crawler log cannot show whether that site is in Cloudflare’s pilot. The pilot covers a cohort of about 1% of Cloudflare’s network, and nothing in this site’s log or settings says whether this site is part of it, so these results say nothing about whether the signals sped OpenAI’s crawler up or not.¹³

The log is also sampled. Cloudflare keeps a share of requests, so “not seen” means not seen in the sample, and five of the seven articles where OpenAI’s crawler was not seen went live October 19 or later, with six days of log or fewer. The log holds no network addresses, so a program pretending to be Googlebot would count as Googlebot. And 24 articles on one small site describe this site, not ChatGPT search as a whole.

What should a site owner do about ChatGPT search discovery?

A site owner who wants pages found by ChatGPT search can do two things. Measure when OpenAI’s search crawler first requests each new page, beside Googlebot’s first request, from your own server or network logs. And stop the patterns Cloudflare’s filtering signal is built to catch, such as missing pages that redirect to the homepage.¹

A page that does not exist should answer with a “not found” status (a 404) instead of a redirect to the homepage, because a redirect makes every dead address look like a live one. Cloudflare says the next phase will test “faster and more reliable change signals” and “measure what happens after content is discovered,” so the gap between Googlebot and OpenAI’s crawler is worth re-measuring as that work lands.¹