llms.txt Adoption Data: Server-Log Analysis (12-Day Study)

Introduction: The Hype vs. The Server Logs

AI hype promises a clean, structured internet.

Yet your actual server logs tell a different story about automated traffic. Tech circles positioned the new markdown-based directory file as a breakthrough for managing AI crawler behavior, but observing llms.txt in practice reveals a stark gap between industry excitement and raw reality. We wanted answers. So we ran a twelve-day measurement on our own site, logging all 9,645 requests that reached it.

Do bots actually care?

No, they do not. A brief window on one small site is one data point rather than a universal industry study, but the pattern we found matches every larger measurement published so far. AI crawlers bypass the polite file entirely. Hosting a text directory will not change your site wide traffic or alter which crawler user agents visit. To understand how AI bots crawl sites in reality, operators must look directly at their raw access log data.

What is llms.txt? Origin, Specifications, and the Robots Analogy

Operators must understand this proposal before judging it. The Answer.AI proposal arrived in September 2024: founder Jeremy Howard introduced the clean, markdown-based directory hosted at llmstxt.org as a friction-free way to hand language models a curated map of a site. Placed directly at the domain root, the file functions as a context-rich sitemap aimed at reasoning engines. Why would it matter? Traditional HTML pages carry navigation, scripts, and tag-heavy markup that consume model context, whereas llms.txt in practice is meant to point models directly at clean Markdown content (llmstxt.org).

How does the robots.txt vs llms.txt comparison break down against older digital publishing standards?

Consider these structural differences:

• Robots.txt blocks or allows path access.
• The llms.txt proposal organizes context.
• Robots.txt targets traditional crawler user agents.
• The llms.txt proposal feeds generative engines.

Webmasters originally designed the classic robots.txt standard to manage raw crawler request volume and protect server capacity. The new initiative instead tries to build a polite content discovery pathway: a modern counterpart that guides AI models rather than blocking them. Adoption remains low (SE Ranking). The concept is elegant on paper, but its success depends entirely on whether crawler operators ever request the file. That question is exactly what AI crawler server logs can answer under real-world conditions.

AWRSHIFT’s 12-Day Experiment: Zero Organic llms.txt Reads

Building on these structural theories, AWRSHIFT ran a 12-day measurement of llms.txt in practice. What did our server logs actually reveal? We parsed 9,645 Caddy JSON access requests recorded from July 19 to July 31, 2026, on our own publisher site (wp.awrshift.com). Caddy’s HTTP request logging is enabled with the log directive in the Caddyfile and captures every incoming request, providing an unfiltered view of crawler behavior (Caddy documentation).

Our final count was absolute. Zero external AI-crawler requests for /llms.txt occurred during the entire twelve-day window.

External AI-crawler requests for /llms.txt: 0
Total /llms.txt fetches in the log: 39 (all ours: monitoring curls, own tools, one browser)
Dataset scale: 9,645 Caddy JSON log lines

Not a single fetch came from GPTBot, ClaudeBot, OAI-SearchBot, Meta’s crawler, or any other AI user agent. Every recorded request for the file was our own monitoring, our own tooling, or a manual browser check.

And it is not because AI crawlers were absent. The same twelve days saw roughly 1,000 AI-crawler requests site-wide in our log:

The bots were here the whole time. They never asked for the file. Cloudflare’s Radar data explains what they are busy with instead: crawling for model training dominates AI bot activity: nearly 80% of AI crawling at its mid-2025 peak, and still roughly half of AI crawler requests under Cloudflare’s refined 2026 purpose taxonomy (Cloudflare Radar). Training crawlers scrape raw pages at scale. They do not hunt for polite, structured directories.

What Bots Pull Instead: The Technical Mechanics

If AI engines ignore the new directory, how do they navigate a site? Server log analysis gives a clear answer: they exploit the traditional content discovery pathways, and you can see it by filtering the log on each user agent string. Consider ClaudeBot. XML sitemap requests and robots.txt checks made up 209 of its 263 logged requests. GPTBot targeted different assets: 26 requests to the WordPress oEmbed endpoint, 21 to the homepage, and 12 to the XML sitemap. That is rapid metadata extraction without downloading heavy media. Meta’s crawler illustrated a third extreme. It spent 326 of its 492 requests pulling a single Open Graph image (an og:image PNG), hunting rich visual assets and structured metadata while bypassing text-only directories. What they want from the content itself is a separate question: AI engines retrieve self-contained passages, not curated directories.

The robots.txt comparison makes the verdict unambiguous. In the same window where /llms.txt collected zero crawler requests, crawlers fetched /robots.txt 181 times: ClaudeBot 103, OAI-SearchBot 23, Bytespider 19, Googlebot 13, Applebot 9, Bingbot 8, YandexBot 6. The thirty-year-old standard gets checked constantly. The proposed one never gets checked at all.

These patterns lead to a practical realization for digital publishers:

• ClaudeBot prioritizes standard index paths to map site architecture.
• GPTBot targets oEmbed endpoints for fast metadata ingestion.
• Meta-externalagent focuses heavily on visual media assets.

Relying on unadopted text standards will not protect server resources or improve how AI systems index your pages. Technical SEO must align with verified crawler behavior; the levers with actual evidence behind them are covered in our GEO citation guide. You can check your own site with a log-file analysis tool such as Screaming Frog Log File Analyser, or with a few lines of scripting over the raw access log, which is all we used.

External Evidence: The Adoption-vs-Readership Gap

Does our small-site result generalize? The published evidence says yes:

• No traffic effect: Search Engine Land ran a 10-site, 90-day tracking study by Ana Fernández, published January 20, 2026. It found no evidence that publishing the file drove AI-search traffic or citation gains.
• No citation effect: SE Ranking’s citation impact study analyzed a 300,000-domain dataset in November 2025. The file showed no effect on AI citations, and only one of the top-50 most AI-cited domains in the study had it at all, against a 10.13% baseline adoption.
• Readership near zero: direct measurements of crawler requests for the file, including ours, keep finding approximately nothing.

Meanwhile the llms.txt adoption rate keeps climbing. The Rankability adoption rate tracker, updated July 17, 2026, shows 8.7% of the Tranco top-1,000 domains publishing the file (15.8% of the reachable sites in that cohort). That is the adoption-vs-readership gap in one picture: webmasters keep building directories that machines, so far, do not read.

Official Search Engine Stance

How do search engine operators treat llms.txt officially? They decline it. Google’s John Mueller stated in April 2025 that none of the AI services have said they are using llms.txt, comparing it to the outdated keywords meta tag (a site owner’s self-declaration that consumers have no reason to trust) and pointing site owners at exactly the evidence we used: “you can tell when you look at your server logs that they don’t even check for it” (Search Engine Journal). Gary Illyes went further at Google’s Search Central Deep Dive event (Bangkok, July 2025): Google does not support llms.txt and is not planning to (Search Engine Land, via Kenichi Suzuki’s recap).

One genuine nuance persists inside Google: the Chrome Lighthouse team documented the file in its new agentic-browsing checks even as Google Search refuses to consume it. A proposed standard can be simultaneously indexed as content, referenced by browser tooling, and ignored by every production crawler. That is what our logs show too.

The Practical Verdict: Why We Keep Our llms.txt Anyway

Why keep a file that crawlers ignore? We are keeping ours. The direct traffic to it is zero today. Our own logs say so. But the cost side of the ledger is also zero. The file took minutes to create, costs nothing to serve, and carries no downside we can measure. If AI vendors ever start requesting it, early adopters lose nothing by having waited with the file in place. Skeptics lose nothing by adding it later. Some practitioners call the file a dud on current evidence (Kai Spriestersbach), and on readership data they are right.

Our practical verdict is simple:

• Create the markdown file.
• Upload it to your root directory.
• Ignore it in your SEO planning until server logs (yours or anyone’s) show something actually reading it.

You can check today. Grep your own access log for llms.txt and count the requests that are not you.

Frequently Asked Questions

Did any AI crawler read the llms.txt file in your measurement?

No. Across 12 days and 9,645 logged requests on wp.awrshift.com (July 19–31, 2026), external AI crawlers requested /llms.txt exactly zero times. All 39 fetches of the file were our own monitoring and tooling. The same crawlers made roughly 1,000 other requests to the site in that window.

What are the key elements of the adoption-vs-readership paradox regarding llms.txt in practice?

Sites keep adopting a file that machines do not request. Rankability’s tracker (updated July 17, 2026) shows 8.7% of the Tranco top-1,000 domains publishing llms.txt, while SE Ranking’s 300,000-domain study (November 2025) found no effect on AI citations, and direct readership measurements, including our AI crawler server logs, find approximately zero requests.

How do AI crawlers actually parse site assets like XML sitemaps, oEmbed endpoints, and Open Graph images?

In our logs: ClaudeBot spent 209 of 263 requests on the XML sitemap and robots.txt; GPTBot’s top paths were the WordPress oEmbed endpoint, the homepage, and the sitemap; meta-externalagent devoted 326 of 492 requests to a single og:image file. Sitemaps, feeds, metadata endpoints, and images are how AI bots crawl sites today.

What is the official Google stance on supporting the llms.txt standard?

Google has said on the record that it does not use llms.txt and has no plans to. John Mueller compared it to the keywords meta tag in April 2025, and Gary Illyes stated at the July 2025 Search Central Deep Dive event that Google does not support the file. No major AI vendor has publicly committed to reading it either.

What is the origin, specification, and original purpose of the llms.txt proposal?

Jeremy Howard of Answer.AI introduced the llms.txt proposal in September 2024 (Source: llmstxt.org). The specification describes a markdown file at the site root that gives language models a curated, low-noise map of a site’s key content for use at inference time.

Standards keep making promises the logs do not keep. The same check applied to a different specification turned up four documented MCP behaviours that were not true on the host I run.