Do AI crawlers fetch JavaScript? In 22 days of my log, 14 claimed crawlers split three ways

The usual claim is that AI crawlers never touch JavaScript. This site’s crawler log splits them three ways instead. A crawler is a program that requests web pages automatically, and each request names the crawler with a label called a user agent. I call them claimed crawlers because any program can send any label.

Across 14 crawler labels over 22 days, seven requested scripts, one requested only stylesheets, and six requested neither. Scripts are the files that hold a page’s JavaScript code, and stylesheets are the files that hold its styling. Anyone planning around a blanket rule about JavaScript is planning around something this log does not show.¹

Requests identifying as GPTBot also asked for four addresses that exist only as unfilled placeholders inside a script. That suggests, suspected and not confirmed, that it read the script as text without running it.

Of 14 claimed crawlers in 22 days of this site’s log, seven requested scripts, one requested only stylesheets, and six requested neither.

What does a site’s crawler log cover, and what can it not show?

A crawler log is a site’s record of which automated visitors asked for which files. Mine covers September 20 to October 11, 2026. Each request carries a user agent, the label it uses to name its crawler. The counts are samples and the labels are claims, so the log describes this one site over 22 days and nothing wider.¹

Cloudflare, the network service in front of this site, records only a share of its traffic and scales each count back up. So one row of the log, which is one date and one path for one crawler label, can read 16 requests or 729. That scaled estimate is too coarse to compare crawlers, so I count rows. The whole file held 7,888 rows before I filtered anything, and none repeats a date, label and path.¹

Each crawler name is also a claim. The free Cloudflare plan cannot check which network sent a request. So I write “requests identifying as GPTBot” and never “GPTBot”.

I grouped requests by the crawler name in the label and kept Google’s smartphone and desktop labels apart. I kept only requests for pages and files this site has. I dropped the rest as probes, meaning requests for addresses that do not exist here.

A script is a path ending in .js or .mjs, and a stylesheet is a path ending in .css. For four crawlers, GPTBot, Bingbot, Applebot and OAI-SearchBot, I also read every script, stylesheet and placeholder row by hand.

Can you trust the crawler name in your log?

You cannot trust it, because a user agent, the label a request uses to name its crawler, is a claim that any program can make. Two names in this log show it. Requests identifying as Perplexity-User and as CCBot each appeared on a single row on September 22, 2026, asking for /i.php and /@fs/proc/self/cmdline. Neither path exists on this site.¹

Cloudflare, which logs only a sample of traffic and scales it back up, reported 729 requests for each. My read is that an attack scanner was using each crawler’s name, because a crawler built to read a site’s pages has no reason to ask for a path the site never had. A raw request total would have credited each name with 729 requests of crawling it never did.

Dropping every request for a path the site does not serve removes both. Perplexity-User and CCBot kept no rows. Neither did Claude-SearchBot, so none of the three appears in the table that follows.

Which AI crawlers requested JavaScript files, and which requested none?

Seven of the 14 claimed crawlers requested scripts, the .js and .mjs files that hold a page’s JavaScript code. One, the desktop label of Googlebot, requested only stylesheets, the .css files that style a page. Six requested neither. Each count is rows, where a row is one date and one path.¹

The table lists each claimed crawler in its group, with the rows it logged and how many of those rows asked for a script or a stylesheet.

Group	Claimed crawler	Rows kept	Script rows	Stylesheet rows
Fetched scripts	Applebot	421	28	35
Fetched scripts	Bingbot	250	3	9
Fetched scripts	GoogleOther	97	10	14
Fetched scripts	Googlebot, smartphone	248	21	23
Fetched scripts	GPTBot	91	3	2
Fetched scripts	meta-externalagent	443	23	40
Fetched scripts	OAI-SearchBot	391	11	17
Stylesheets only	Googlebot, desktop	36	0	2
Neither	Amazonbot	418	0	0
Neither	ChatGPT-User	134	0	0
Neither	Claude-User	75	0	0
Neither	Cloudflare AI Search	677	0	0
Neither	PerplexityBot	115	0	0
Neither	PetalBot	1,328	0	0

Within each group the order is alphabetical, because sampled counts cannot support a ranking. The table mixes AI labels with search labels such as Googlebot and Bingbot, because they share one log and the contrast between them is the point.¹

Did OpenAI’s crawlers request JavaScript?

Two of the three OpenAI crawler labels in this site’s log requested scripts, the files that hold a page’s JavaScript code. Requests identifying as OAI-SearchBot asked for a script on 11 rows and a stylesheet on 17. Requests identifying as GPTBot asked for a script on three rows and a stylesheet on two. ChatGPT-User asked for neither, so the claim that AI crawlers never touch JavaScript fails for two of OpenAI’s three.¹

Which rules in robots.txt, the file that tells crawlers which pages they may fetch, cover each OpenAI label is a separate question, and this site’s piece on blocking training bots covers it.

Which crawlers requested no scripts and no stylesheets?

Six claimed crawlers requested no script and no stylesheet in 22 days: ChatGPT-User, Claude-User, PerplexityBot, Amazonbot, PetalBot and Cloudflare AI Search. Scripts are the files that hold a page’s JavaScript and stylesheets are the files that hold its styling. Three of the six, ChatGPT-User, Claude-User and PerplexityBot, fetch a page to answer a person’s question.¹

Each of the six did visit. Their rows run from 75 for Claude-User to 1,328 for PetalBot, so a zero in the script column is a pattern in what they asked for and not an absence of visits.¹ Amazonbot, PetalBot and Cloudflare AI Search also fetched pages without ever fetching the files that build them.

Do Google’s two Googlebot labels behave the same way?

Googlebot’s two labels do not behave the same way in this log. Googlebot is Google’s search crawler, and it identifies itself with one label for smartphone and one for desktop. Requests identifying as the smartphone label asked for a script on 21 rows and a stylesheet on 23. Requests identifying as the desktop label asked for no script and for a stylesheet on two rows. I can report the difference but not explain it.¹

How Google’s own rendering works is covered in Google’s crawler split and in Fetch is not parse. Whether Google chooses to fetch a URL at all is a different question, and the piece on a Search Console sitemap “Couldn’t fetch” error covers one case where it skipped a file that loaded fine.

Why did requests identifying as GPTBot ask for addresses that do not exist?

On September 28, 2026, requests identifying as GPTBot asked for four addresses shaped like /_astro/${e.href}. A script is a file holding a page’s JavaScript code, and a placeholder is a blank that the code fills with a real address when it runs. They exist only as text inside a script. My reading, suspected and not confirmed, is that GPTBot read the script as plain text and added strings that looked like addresses to its list of pages to visit.¹

The four addresses were /_astro/${e.href}, /_astro/${e.article.href}, /_astro/${k(n.url)} and /_astro/${k(i.url)}. Together they made an estimated 10 requests over four rows. That same day, the same name fetched two of this site’s script files.

A crawler that had run the code would have asked for the filled-in address, so these requests look like strings pulled out of the file’s text. The log records what was asked for and says nothing about why, which is why I tag the reading suspected.

Do AI crawlers execute JavaScript, or only request it?

A request for a script does not mean the crawler ran it. The request shows that a crawler downloaded the file, and it does not show that the crawler ran the code inside. Running a page’s scripts and applying its stylesheets to build the visible page is called rendering.

The reverse also holds. A crawler that never requests a script might still have rendered the page, for example by reusing a saved copy of the file from an earlier visit.

The log shows requests and nothing else, so it cannot show what a crawler did with a file after downloading it. The GPTBot case is one example, because the same crawler name fetched script files and also queued text from inside one, which looks more like reading than running.

My read for site owners is to put anything every crawler must read in the HTML the server sends back, since six of the 14 labels gave no sign of fetching the files that build a page. That is a precaution drawn from one log and not a measured rule.

How can you check which crawlers request JavaScript on your own site?

To check your own site, drop every request for a path your site does not serve, group the remaining requests by crawler name, count rows (one date and one path for one crawler), and count how many end in .js, .mjs or .css. The same steps work on any server log or analytics export that records the user agent and the path.

The steps are these.

1. Filter to the crawler names you care about, matching a request when its user agent string contains the name.
2. Drop every request for a path your site does not serve, so scanners that borrow a crawler’s name fall out.
3. Count distinct date and path pairs for each crawler, which gives rows.
4. Count the rows whose path ends in .js, .mjs or .css.
5. Read the script rows by hand for any crawler that matters to you, and look for paths that exist only as text inside a script.

A sampled log over 22 days is a snapshot, so repeat the count after a few weeks before you build a rule on it.