Outcropby Basalt

Guide ·

Can AI crawlers read your site? A 5-minute check

Three commands tell you whether AI crawlers get your pages, whether robots.txt lets them in, and whether your text is in the page without JavaScript. They use curl, grep and perl, which come with macOS and most Linux systems.

Why check

An assistant can only quote what its crawler could read

ChatGPT, Claude and Perplexity answer many questions from pages their crawlers fetched. If a crawler gets a block page, a challenge page or an empty shell that only fills in with JavaScript, your product cannot be described from your own site.

The worked example is Upright, a free Mac app and one of Basalt Studio’s own products. The output shown is real terminal output from 20 Oct 2026 at 01:38 Türkiye time (22:38 UTC on 19 Oct), trimmed to the parts that matter here.

Before you start

Open a terminal and set your address once

On a Mac, open Terminal. On Linux, any shell will do. The commands work the same in zsh, bash and sh. Set your site’s address once, without a slash at the end, and the three steps below use it:

site=https://example.com

Step 1

Fetch your home page as each crawler

This asks for your home page 11 times, each time with a different crawler’s name in the user agent, and prints the status code and the size of what came back. The first line uses a plain browser-style name to compare against.

site=https://example.com

for ua in Mozilla/5.0 GPTBot/1.1 OAI-SearchBot/1.0 ChatGPT-User/1.0 ClaudeBot/1.0 Claude-SearchBot/1.0 PerplexityBot/1.0 CCBot/2.0 Applebot/0.1 Googlebot/2.1 bingbot/2.0; do
  printf '%-22s ' "$ua"
  curl -s -o /dev/null -A "$ua" -w '%{http_code} %{size_download}\n' "$site/"
done

Worked example: Upright, 20 Oct 2026

Upright’s home page, fetched with 11 crawler user agents. That run also printed each page’s title; the command above prints the status and size only. That run included Google-Extended, which is not a crawler of its own (see step 2), so the command above leaves it out.

GPTBot/1.1             200 35857  title: Upright: a free Logi Options+ alternative for Mac
OAI-SearchBot/1.0      200 35857  title: Upright: a free Logi Options+ alternative for Mac
ChatGPT-User/1.0       200 35857  title: Upright: a free Logi Options+ alternative for Mac
ClaudeBot/1.0          200 35857  title: Upright: a free Logi Options+ alternative for Mac
Claude-SearchBot/1.0   200 35857  title: Upright: a free Logi Options+ alternative for Mac
PerplexityBot/1.0      200 35857  title: Upright: a free Logi Options+ alternative for Mac
Google-Extended        200 35857  title: Upright: a free Logi Options+ alternative for Mac
Googlebot/2.1          200 35857  title: Upright: a free Logi Options+ alternative for Mac
bingbot/2.0            200 35857  title: Upright: a free Logi Options+ alternative for Mac
CCBot/2.0              200 35857  title: Upright: a free Logi Options+ alternative for Mac
Applebot/0.1           200 35857  title: Upright: a free Logi Options+ alternative for Mac

How to read it

  • Every line shows 200 and the same size: no crawler was turned away by its name. That is what you want to see.
  • A 403, 429 or 503, or a size much smaller than the browser line, means that name got a block or challenge page instead of your content.
  • Sizes can differ a little between runs if the page has parts that change. Compare the lines within one run.
  • We ran the command exactly as printed here at 14:03 Türkiye time the same day: every user agent got a 200 and 37,000 bytes. The page had been updated in between.

Step 2

Read your robots.txt rules for AI crawlers

This prints the rules for all crawlers (User-agent: *) and for each AI crawler named in your robots.txt, with the three lines after each.

curl -s "$site/robots.txt" | grep -i -E -A3 '^user-agent: *(\*|gptbot|oai-searchbot|chatgpt-user|claudebot|claude-searchbot|perplexitybot|google-extended|ccbot|applebot-extended)'

Worked example: Upright, 20 Oct 2026

Upright’s robots.txt has one group for all crawlers and names no AI crawler.

User-agent: *
Allow: /

Sitemap: https://upright.hellobasalt.dev/sitemap.xml

How to read it

  • Allow: / under User-agent: * with no AI crawler named: every crawler may read every page.
  • Disallow: / under a crawler’s name blocks that crawler from the whole site. A group for a named crawler replaces the * group for that crawler.
  • OpenAI and Anthropic each run separate crawlers for training (GPTBot, ClaudeBot) and for search (OAI-SearchBot, Claude-SearchBot). You can block training and still allow search.
  • Google-Extended and Applebot-Extended are names used only in robots.txt. They control whether Google and Apple may use your content for their AI models, not whether you appear in search.

Step 3

Count the words that arrive without JavaScript

This downloads the raw HTML, removes scripts, styles and tags, and counts what is left: roughly the text a crawler that does not run JavaScript sees.

curl -s "$site/" | perl -0777 -pe 's/<(script|style)\b.*?<\/\1>//gis; s/<[^>]+>/ /g' | wc -w

To check several pages at once, change the paths in this loop to your own:

for p in / /pricing /docs; do
  printf '%6s words  %s\n' "$(curl -s "$site$p" | perl -0777 -pe 's/<(script|style)\b.*?<\/\1>//gis; s/<[^>]+>/ /g' | wc -w | tr -d ' ')" "$p"
done

Worked example: Upright, 20 Oct 2026

Words in the raw HTML of Upright’s 8 pages.

1413 words  https://upright.hellobasalt.dev/
1057 words  https://upright.hellobasalt.dev/logi-options-alternatives
1114 words  https://upright.hellobasalt.dev/upright-vs-mac-mouse-fix
1021 words  https://upright.hellobasalt.dev/upright-vs-bettermouse
1064 words  https://upright.hellobasalt.dev/upright-vs-steermouse
754 words  https://upright.hellobasalt.dev/mx-vertical
613 words  https://upright.hellobasalt.dev/privacy
523 words  https://upright.hellobasalt.dev/terms

How to read it

  • Every page has hundreds of words in the raw HTML, so the text is there without JavaScript.
  • If a page that reads as a full article in your browser comes back with a few dozen words, its text is drawn by JavaScript, and a crawler that does not run JavaScript sees an empty page.
  • The count includes your menu and footer, so compare pages with each other rather than against a fixed number.
  • Word counters differ. Run the same way at 14:03 Türkiye time, after the site had been updated, the command above counted 1,506 words on Upright’s home page. Compare runs made with the same command.

Limits

What this check cannot tell you

  • A user agent is a hint, not proof. Anyone can put “GPTBot” in a request, so Cloudflare and other bot-protection services check real crawlers by their IP addresses. Your fake GPTBot may get through while the real one is challenged, or the other way round. Check your bot-protection settings and your logs for requests from the real crawlers.
  • JavaScript. Google renders JavaScript before it indexes a page. Most AI crawlers do not, so step 3 matters more for AI answers than for Google.
  • Being read is not being cited. A crawler that can read your page may still never quote it. Whether assistants name you is a separate question, answered by asking them.
  • One page at a time. These commands check the pages you point them at. A full crawl finds the pages you forgot.