Tool · AI & crawl monitoring
Who's crawling brandonlazovic.dev
As of 2026-07-31, 47% of the identified crawler traffic hitting brandonlazovic.dev comes from AI systems: model-training crawlers, AI answer engines, and on-demand assistants like OpenAI, Cloudflare, Perplexity and Google. This page reconstructs every bot request from Cloudflare's server-side logs (the traffic Google Analytics never sees, because crawlers don't run the tracking script). It then segments that traffic by what each crawl actually feeds.
| Category | Requests | Share |
|---|---|---|
| Search (organic & AI) | 1,186 | 23% |
| AI trainingAI | 708 | 14% |
| AI searchAI | 1,339 | 26% |
| AI assistantAI | 367 | 7% |
| Social | 274 | 5% |
| SEO tools | 260 | 5% |
| Other | 1,038 | 20% |
Live crawler activity
Interactive charts below are a visual layer over the same numbers. The values above and the methodology further down carry the full record.
Crawler activity over time
Daily request count by category. Search engines dominate raw volume, so click Search (organic & AI) in the legend to hide it and see the AI detail.
tip: click a chip to hide or show a category's line
Requests by crawler
Top vendors across the tracked window.
By category
What the crawl feeds. Search (organic & AI) is dual-use, because the same crawl powers blue links and AI answers.
Crawler breakdown
Each crawler with its total and daily activity over the selected window. Click a row to expand its day-by-day detail, or a header to sort.
| Crawler | Category | Daily activity | Last seen | Requests |
|---|
Enable JavaScript to explore the crawler breakdown and charts.
What crawlers read most
The site pages bots request most, summed over every crawler across 3 days of the path log (only live content pages are counted — assets, feeds, and exploit-probe paths are excluded; 406 page requests so far). This log keeps day-level detail, so the trend view deepens as history accrues.
| Page type | Requests | Share |
|---|---|---|
| Pulse | 142 | 35% |
| Articles | 120 | 30% |
| Home | 90 | 22% |
| Tools | 31 | 8% |
| Other | 13 | 3% |
| About | 5 | 1% |
| Writing | 5 | 1% |
What this is
Every website is crawled constantly by automated clients: search engines, AI companies training and grounding their models, social platforms fetching link previews, SEO tools, and a long tail of scripts. Almost none of that shows up in Google Analytics, because analytics fires from JavaScript a browser runs and crawlers generally don't. So this is the traffic most site owners never look at. This page is my own site's crawler log, made public and kept current.
Why this is interesting
The mix of automated traffic hitting this site has flipped toward AI. A meaningful and growing share of it is AI systems reading the site: OpenAI's GPTBot and OAI-SearchBot, Anthropic's ClaudeBot, PerplexityBot, xAI, DeepSeek, Amazon, Common Crawl, Google's Vertex crawler, and Cloudflare's AI Search fetcher. That's the supply side of AI citation: an answer engine can only cite a page it has actually read. Watching which AI crawlers arrive, and how often, is the closest first-party signal I have that the entity-hardening and answer-first work on this site is reaching the systems that increasingly sit between people and the open web.
The crawler taxonomy
Each crawler is sorted by what its crawl feeds, a crawl-intent taxonomy rather than a crawl-operator one. Two crawlers from the same company can land in different buckets, and that's the point:
- Search (organic & AI): Googlebot, Bingbot, Yandex, Baidu, Applebot. Grouped together and flagged dual-use on purpose (see the caveat below).
- AI training: crawlers that gather text to train or update models: GPTBot, ClaudeBot, CCBot, Bytespider, Amazonbot, Google-CloudVertexBot, DeepSeekBot.
- AI search: the indexers behind AI answer engines: OAI-SearchBot, PerplexityBot, xAI-SearchBot, Cloudflare-AI-Search.
- AI assistant (on-demand): fetches triggered live by a user's question: ChatGPT-User, Claude-User.
- Social: link-preview unfurlers: Facebook, LinkedIn, WhatsApp, Twitter/X.
- SEO tools: third-party backlink and rank crawlers: Semrush, Ahrefs, Moz, Majestic, DataForSEO, HubSpot.
- Other: security scanners and research monitors that don't fit the buckets above.
"Search (organic & AI)" is deliberately not counted as AI. The same Googlebot crawl feeds classic blue-link results and Google's AI Overviews and AI Mode, and Bingbot feeds Bing and Copilot. There is no separate user agent for the AI use, so splitting that traffic would be a guess. Keeping it in its own dual-use bucket means the "AI-crawler share" figure only counts crawlers whose sole purpose is AI. That makes it a conservative floor, not a ceiling.
Sourcing and data caveats
Read the numbers as a directional, recent snapshot, not an audited total. The specifics:
- Source. Cloudflare's GraphQL Analytics API (
httpRequestsAdaptiveGroups, grouped by user agent and day). This is edge data (requests Cloudflare served or challenged), not origin logs, which a static site on Cloudflare Pages doesn't expose. - Retention is short. The free Analytics tier keeps roughly three days of this breakdown, so the window is a rolling recent one. A daily job appends each day before it ages out, so long-run history accumulates only from that point forward.
- It's sampled and self-reported. Cloudflare's adaptive analytics can sample high-volume traffic, and the "vendor" is read from the user-agent string, which crawlers can spoof. Counts are close, not exact.
- Noise is excluded from this view. Generic HTTP libraries (curl, Go, Python), unattributed headless browsers, and a recurring WordPress exploit probe are logged but filtered out here, so the page answers "who is reading me" rather than "what random script hit an endpoint." In the current window that removes 795 requests. My local dashboard keeps them.
- Secret-scanner probes are excluded too. In the current path log, 226 requests were automated probes for files this site has never had (
/.env,/wp-config.php,/backup.sqland the like). They're dropped from the page tables above. - Reuse. The numbers on this page are shared under a Creative Commons Attribution (CC BY 4.0) license. Reuse them with credit and a link back.
How this page is built
A small zero-dependency Node script queries the Cloudflare GraphQL API once a day on a scheduled GitHub Action, appends the new day's rows to a committed CSV, and pushes. The push triggers a Cloudflare rebuild, which re-reads the CSV and regenerates this page, so what you're seeing tracks the log within a day, with no server to run. Classification happens at view time from a single shared catalog, which means a new category can be split out across the whole history just by editing one list. The same pipeline also feeds a fuller local dashboard I keep for myself.