Research · Issue 1 ·
Which AI bots read our sites? A month-by-month log, October 2026
More AI bots than search engines, and nothing stopped them. From 18 Oct to 21 Oct 2026, bots Cloudflare files as AI made 13,030 requests to our sites, 2.2 times the 5,893 from search engine crawlers. Our bot protection challenged 1,509 requests in those days, none of them from a verified AI bot. They read robots.txt and the sitemap, and almost never llms.txt.
Findings
7 numbers from 4 days of our own logs
Every request Cloudflare logged from 18 Oct to 21 Oct 2026 (UTC) on the three zones that hold our 10 sites: 107,901 in all. “AI bots” are the bots Cloudflare verifies as AI crawlers, AI search bots or AI assistants, plus OpenAI’s search crawler; figures on bots, sites, files and status codes are out of their 13,030 requests. The counts are Cloudflare’s estimates from a sampled dataset (see Method).
- 2.2 to 1
AI bots against search engine crawlers: 13,030 requests against 5,893. AI bots made 12.1% of all 107,901 requests Cloudflare logged on our three zones. Counting OAI-SearchBot as a search engine, as Cloudflare files it, the ratio is 1.8 to 1 (12,267 against 6,656).
- 81.2%
of AI requests came from four bots: ShapBot (3,801), GoogleOther (3,411), ClaudeBot (2,074), Amazonbot (1,289). GPTBot, OpenAI’s training crawler, made 600.
- 83.4%
of AI requests went to two of our 10 sites: the job board, gocmeden.hellobasalt.dev (6,007), and the coffee catalogue, kahve.hellobasalt.dev (4,858). The other 8 shared 2,165.
- 2
AI bot requests for llms.txt and 0 for llms-full.txt, against 502 for robots.txt and 397 for sitemap.xml, though all 10 of our sites serve an llms.txt (the 2 were both on outcrop.hellobasalt.dev). ClaudeBot made 351 of the robots.txt requests.
- 661
AI bot requests got a 404 (5.1%), 517 of them from GoogleOther, mostly on the coffee catalogue (395) and the job board (256). 79.7% got a 200 and 14.2% a redirect.
- 0
of 13,030 AI bot requests were challenged or blocked. Cloudflare’s Bot Fight Mode, on for hellobasalt.dev and kacpapel.app, challenged 1,509 requests in the same days, none from a bot Cloudflare verifies.
- 554
requests came from AI assistants fetching a page while answering a person: ChatGPT-User 390, an assistant with a browser user agent 86, Claude-User 68. 371 of ChatGPT’s were on the job board.
The bots
Which AI bots came
Bots with 10 or more requests, by family name. “Category” is Cloudflare’s. 10 more requests came from 3 bots with fewer than 10 each; they are in the CSV. Crawlers of search engine optimisation tools, a category of their own, made 13,592 requests, more than all the AI bots together.
| Bot | Run by | Category | Requests | Our sites read | 404s |
|---|---|---|---|---|---|
ShapBot | Parallel | AI Search | 3,801 | 6 | 15 |
GoogleOther | Google | AI Crawler | 3,411 | 10 | 517 |
ClaudeBot | Anthropic | AI Crawler | 2,074 | 10 | 84 |
Amazonbot | Amazon | AI Crawler | 1,289 | 9 | 18 |
OAI-SearchBot | OpenAI | Search Engine Crawler | 763 | 10 | 0 |
GPTBot | OpenAI | AI Crawler | 600 | 10 | 3 |
Applebot | Apple | AI Search | 452 | 10 | 10 |
ChatGPT-User | OpenAI | AI Assistant | 390 | 3 | 14 |
Unnamed (browser user agent) | – | AI Assistant | 86 | 3 | 0 |
Reflectionbot | – | AI Crawler | 86 | 2 | 0 |
Claude-User | Anthropic | AI Assistant | 68 | 1 | 0 |
Our sites
Which of our sites they read
All 10 are Basalt Studio’s own sites; we name no one else’s. Search engine crawlers here leave OAI-SearchBot out, since it is counted with the AI bots.
| Our site | What it is | AI bots | Search engines | All requests | AI share |
|---|---|---|---|---|---|
gocmeden.hellobasalt.dev | job board | 6,007 | 2,473 | 27,251 | 22.0% |
kahve.hellobasalt.dev | coffee catalogue | 4,858 | 2,450 | 16,955 | 28.7% |
kacpapel.app | 617 | 139 | 7,089 | 8.7% | |
cosmicpost.lol | 475 | 408 | 25,302 | 1.9% | |
outcrop.hellobasalt.dev | this site | 271 | 140 | 2,718 | 10.0% |
hellobasalt.dev | studio home | 240 | 78 | 10,058 | 2.4% |
lull.hellobasalt.dev | 176 | 62 | 3,027 | 5.8% | |
byewall.hellobasalt.dev | 169 | 41 | 2,619 | 6.5% | |
upright.hellobasalt.dev | 136 | 64 | 4,055 | 3.4% | |
folio.hellobasalt.dev | 81 | 34 | 2,079 | 3.9% |
What they fetched
robots.txt, sitemap.xml and llms.txt
Requests by AI bots for the files written for crawlers, on all our sites together. All 10 of our sites answered /llms.txt with a 200 when we checked on 22 Oct 2026.
| File | Requests | By bot | Status |
|---|---|---|---|
/robots.txt | 502 | ClaudeBot 351, OAI-SearchBot 78, Applebot 57, Reflectionbot 10, Claude-User 6 | 200: 492, 301: 10 |
/sitemap.xml | 397 | ClaudeBot 351, GPTBot 33, ShapBot 12, Applebot 1 | 200: 397 |
/llms.txt | 2 | ClaudeBot 1, GPTBot 1 | 200: 2 |
/llms-full.txt | 0 | none | – |
What they got back
Status codes for AI bots
79.7% of AI bot requests got a 200. 1,855 were redirected, 661 found nothing and 16 hit a server error.
| Status | Meaning | Requests | Most from | Why |
|---|---|---|---|---|
200 | OK | 10,390 | GoogleOther 2,807; kahve.hellobasalt.dev 4,319 | |
204 | No content | 25 | Unnamed (browser user agent) 25; hellobasalt.dev 13 | |
301 | Moved permanently | 21 | Applebot 19; www.kacpapel.app 7 | |
302 | Found (temporary redirect) | 109 | ChatGPT-User 107; gocmeden.hellobasalt.dev 109 | |
304 | Not modified | 4 | Unnamed (browser user agent) 4; outcrop.hellobasalt.dev 3 | |
307 | Temporary redirect | 37 | Claude-User 28; gocmeden.hellobasalt.dev 28 | |
308 | Permanent redirect | 1,688 | ShapBot 1,555; gocmeden.hellobasalt.dev 1,537 | Mostly job ads on the job board asked for without the trailing slash their address has, redirected to it. |
402 | Payment required | 53 | GPTBot 34; cosmicpost.lol 53 | All on /_next/image on cosmicpost.lol, which answered 402 to every kind of visitor; Cloudflare’s pay-per-crawl status for them is “not applicable”. |
404 | Not found | 661 | GoogleOther 517; kahve.hellobasalt.dev 395 | Mostly removed job ads (they answer 410 since 20 Oct), and catalogue addresses with a stray code after a colon that the catalogue does not serve. |
410 | Gone | 25 | Amazonbot 11; gocmeden.hellobasalt.dev 25 | Removed job ads on the job board, which answer 410 instead of 404 since 20 Oct. |
499 | Client closed the connection | 1 | ChatGPT-User 1; kahve.hellobasalt.dev 1 | |
500 | Server error | 4 | Applebot 4; kacpapel.app 4 | |
503 | Unavailable | 12 | Amazonbot 9; kahve.hellobasalt.dev 12 | Catalogue pages, one request each. |
Bot protection
What Bot Fight Mode did to verified AI bots
Nothing. 13,030 of 13,030 AI bot requests passed with no security action. Bot Fight Mode is Cloudflare’s free bot protection; it was on for hellobasalt.dev and kacpapel.app. In the same days it challenged 1,509 requests, all from visitors Cloudflare does not verify, and Cloudflare’s managed rules blocked 676 more. Turning Bot Fight Mode on did not cost us any verified AI crawler, search bot or assistant.
Method
How we counted
- Source
- Cloudflare’s GraphQL Analytics API, dataset
httpRequestsAdaptiveGroups, for the three Cloudflare zones that hold our sites: hellobasalt.dev (with its subdomains), kacpapel.app and cosmicpost.lol. One read-only query per zone, run on 22 Oct 2026 at 20:22 UTC. The query asks for five groupings of the same window: all requests, requests by bot category and security action, requests by host, AI bot requests by user agent, host, status and security action, and AI bot requests for the four files. - Window
- 4 whole days, 18 Oct 00:00 to 21 Oct 2026 24:00 UTC. The hellobasalt.dev zone joined Cloudflare on 17 Oct 2026, so this first issue starts the day after. Later issues cover a calendar month.
- Sampling
- The dataset is sampled: Cloudflare stores a share of requests and scales them up, so each count is its estimate. We asked for 95% intervals. All requests: 107,355 to 108,447. AI Crawler: 7,384 to 7,536; AI Search: 4,173 to 4,333; AI Assistant: 528 to 580; Search Engine Crawler: 6,610 to 6,702. We add up each zone’s bounds, which makes the ranges a little wider than they are. Counts under a few hundred were mostly not sampled at all.
- Labels
- “AI Crawler”, “AI Search”, “AI Assistant” and “Search Engine Crawler” are Cloudflare’s verified-bot categories, not ours. Cloudflare files GoogleOther under AI Crawler, Applebot under AI Search, and OAI-SearchBot, OpenAI’s search crawler, under Search Engine Crawler; we count OAI-SearchBot as AI (763 requests) and say so wherever it matters. Bot names are the user agent reduced to its family name.
- Security
- Each request carries the security feature that acted on it and what it did (
securitySource,securityAction). We read “challenged” and “blocked” from those. Bot Fight Mode was on for hellobasalt.dev and kacpapel.app and off for cosmicpost.lol when we checked on 22 Oct 2026; the API shows today’s setting, not its history. - Plan
- All three zones are on Cloudflare’s Free plan: one query covers at most 30 days and nothing older than 31 days, and the dashboard’s bot scores are not available, so a bot is either verified by Cloudflare or not.
Limits
What this log cannot see
- Only bots Cloudflare verifies. A crawler that is not on Cloudflare’s verified list, or one that only claims to be GPTBot, counts as an ordinary visitor here. The AI share is a floor.
- Fetching is not citing. A bot reading a page says nothing about whether an assistant named or linked us in an answer.
- Our sites, not the web. 10 small sites of one studio, two of which drew most of the traffic. 4 had fewer than 200 AI bot requests in 4 days, too few to compare bots on.
- 4 days. Crawlers come in bursts, so a few days can over- or under-count any one bot. Monthly issues will show which bots stay.
- One assistant cannot be named: 86 AI Assistant requests came with an ordinary browser user agent.
The script
Run the same count on your own site
Every number here comes from one read-only query that a small script sends to Cloudflare: one command, no dependencies, a token that can only read your zone’s analytics. It prints the bot categories, the AI bots by name, the hosts they read, their status codes, the files they fetched and what your security did to them, and writes the same CSV as ours. The script is published with the next issue, under the MIT licence.
Data
Download the data
The CSV has one row per zone, host, Cloudflare category, bot, path group (all, robots.txt, llms.txt, llms-full.txt, sitemap.xml), status and security outcome: 202 rows. Add up the “all” rows only; the file rows are part of them. summary.json holds every figure on this page and its base: all_requests, ai_requests_with_oai_searchbot and search_engine_crawler_without_oai_searchbot. The bot operators and what each site is come from us, not from the files.
summary.json and ai-bots-2026-10.csv
JSON and CSV, under CC BY 4.0: use them, quote them and build on them, with credit.
How to cite it
Outcrop (Basalt Studio), “Which AI bots read our sites? October 2026 log”, 23 Oct 2026. https://outcrop.hellobasalt.dev/research/ai-bots-2026-10/. Data: https://outcrop.hellobasalt.dev/research/ai-bots-2026-10/summary.json and https://outcrop.hellobasalt.dev/research/ai-bots-2026-10/ai-bots-2026-10.csv, CC BY 4.0.
Next issue: November 2026, covering the whole of October, with the script. Our other studies: can AI crawlers read small software sites? and pricing and comparison pages on 1,000 launches.