Outcropby Basalt

Research ·

Can AI crawlers read small software sites? We checked 1,000 launches

Mostly they can get in, but one in four home pages gives them almost nothing to read. Only 1.4% of the sites we reached block the bots behind AI answers, while 26.6% of home pages show fewer than 80 words without JavaScript.

Findings

Seven numbers from 1,000 launch sites

1,000 product sites launched on Show HN in the past year, checked once on 20 Oct 2026. Home-page figures are out of the 896 home pages we read; robots.txt and sitemap figures out of the 955 sites we reached. The median home page had 442 words without JavaScript.

  1. 26.6%

    of home pages show fewer than 80 words without JavaScript (238 of 896), and 17.6% show fewer than 20, so a crawler that does not run JavaScript finds almost nothing to describe.

  2. 1.4%

    of reachable sites block a bot that ChatGPT search, Claude or Perplexity use to answer questions (13 of 955); more block training crawlers (54, 5.7%), led by Bytespider (45), GPTBot (40) and CCBot (39).

  3. 20 of 40

    GPTBot blocks are on sites serving Cloudflare’s managed robots.txt (99 sites, 10.4%), so half come from a default setting rather than a decision.

  4. 23.7%

    of home pages say in structured data that they are software (212 of 896); 41.7% have any structured data and 28.1% name an Organization.

  5. 45.1%

    of reachable sites have no valid sitemap at the robots.txt Sitemap line or /sitemap.xml (431 of 955), and 33.1% have no robots.txt at all (316).

  6. 26.4%

    of the sites checked have an llms.txt (248 of 941), but 26 of them have a home page under 80 words without JavaScript, and the file does not fix an empty page.

  7. 32.8%

    pass all five basics (294 of 896): a home page readable without JavaScript (80+ words), no AI answer bot blocked, a meta description, structured data and a working sitemap, with no noindex; with llms.txt as a sixth, 18.2%.

What to do about it

Fix the page first, then the files

Blocking is rarely the problem. An empty page, a missing sitemap and no word on what the product is are. In order:

  • Count the words your home page sends without JavaScript. If it is a few dozen, render the main text on the server. Step 3 of the AI crawler check does it in one command, and steps 1 and 2 show whether a crawler is turned away.
  • Read your robots.txt as each AI crawler would. If your site is on Cloudflare, check whether its managed robots.txt is on and whether it blocks the bots you meant to block. Blocking training crawlers and allowing answer bots is a choice you can make on purpose.
  • Say what the product is. A SoftwareApplication, WebApplication or MobileApplication block in the home page’s structured data, and a meta description that names the product and who it is for.
  • Serve a sitemap and point robots.txt at it. The SEO checklist for a new software site puts this and the other basics in order, with a check for each.
  • Add llms.txt once the page has text. The llms.txt guide shows what we put in ours. It is a cheap extra, not a substitute for a readable page.

Method

How we checked

Sample
Every Show HN post from 21 Oct 2025 to 19 Oct 2026 with 10 or more points and a link: 3,834 posts. Links to code hosts, app stores, social networks, hosted blogs and free hosting subdomains were removed, leaving one row per host: 1,830 product sites. 1,000 were drawn at random. The median Hacker News score in the sample was 30.
Checks
On 20 Oct 2026, at most five requests per site, one at a time with a pause between them: robots.txt, the home page, /llms.txt and one sitemap. Our user agent named itself and linked to this site. We followed each site’s robots.txt for our own crawler: 14 sites disallowed us, so their home pages were not read. Of 955 sites reached, 896 home pages were read.
Definitions
“Blocked” means the bot may not fetch / under that site’s robots.txt, matched the way crawlers match it: the most specific named group, otherwise *, longest rule wins. “Words without JavaScript” counts the visible text in the raw HTML, leaving out scripts, styles and SVG.
Spot checks
Five random home pages under 20 words were fetched again: three were empty app shells and two drew everything from scripts. Four random llms.txt files were real Markdown files describing the product.

Limits

What this study cannot tell you

  • 41 sites could not be reached from our server, and 4 checks failed. Each percentage uses the sites we could read as its base, and says which base.
  • The sample leans technical. Show HN launches come from makers who launch in public and know about AI, not from all software sites. Their llms.txt rate is probably higher than average.
  • One fetch per site, from one location on one day. A site may serve real AI crawlers differently from us, by user agent or IP address; we did not pretend to be any bot. Only 4 home pages showed us a bot challenge.
  • Home page only. Inner pages may do better or worse.
  • Google-Extended is a robots.txt token, not a crawler. Blocking it keeps content out of Gemini training; it does not remove a site from Google Search or AI Overviews.

Data

Download the data

Every number on this page comes from one file of aggregate counts. It names no site. Bases: sample_sites, reachable_sites, home_pages_read and llms_txt_checked.

summary.json

JSON, under CC BY 4.0: use it, quote it and build on it, with credit.

Download summary.json

How to cite it

Outcrop (Basalt Studio), “Can AI crawlers read small software sites? 1,000 launches”, 20 Oct 2026. https://outcrop.hellobasalt.dev/research/ai-readiness-2026-10/. Data: https://outcrop.hellobasalt.dev/research/ai-readiness-2026-10/summary.json, CC BY 4.0.

Re-run every quarter; next in January 2027.