A Good Engineering Idea Sold as a Bad Marketing One

llms.txt was proposed in late 2024 as a clean machine-readable map of a website, and has since been sold as a route to visibility in AI answers. Reported analysis across roughly three hundred thousand domains found no correlation with citation, around 97 per cent of the files were never fetched by anything, and Google has stated publicly that it does not use the file. Its real purpose is agent readiness — documentation for software working with your site, which is a developer concern.

If somebody has quoted you for an llms.txt implementation, this article will take eight minutes and save you the money.

But the file is more interesting than a straightforward debunking allows, because it began as a sensible proposal and it still has a real use. It is simply not the use being sold.

Where it came from

The idea was proposed in late 2024 by Jeremy Howard of Answer.AI, and the reasoning behind it was sound.

A language model reading a web page has to work through navigation, cookie notices, advertising markup, related-post widgets and footer links to find the content. That is wasteful, and context is finite. So: put a markdown file at a fixed location on the site, listing the pages that matter with short descriptions, in clean text. A map, written for machines, maintained by the site owner.

Anyone who has tried to extract content from a modern web page will recognise the problem it addresses. It is a good instinct.

What happened next was not the proposal’s fault. As businesses began worrying about visibility in AI answers, the file was picked up and sold as the way to be seen — a technical deliverable, quick to produce, easy to invoice, and satisfying to tick off.

What the evidence says

Our pillar on AI search states the headline. Here is the whole picture in one place.

Adoption sits near ten per cent of measured domains. That alone is neutral.

But adoption is reported to be lower among high-traffic sites than among low-traffic ones. That is the number worth sitting with. Techniques that work spread from the top down — large sites have more to gain, more resource, and better information. A measure adopted most by the smallest sites is a measure spreading through marketing rather than through results.

Roughly 97 per cent of these files are never fetched by anything at all. Not fetched and ignored. Not requested in the first place.

Analysis across roughly three hundred thousand domains reportedly found no correlation between having one and being cited in AI answers.

Among the fifty most-cited domains in AI search, one had the file.

And Google has said publicly that it does not use llms.txt, has no plans to, and that its presence will neither help nor harm visibility in Search.

That is a fuller and more consistent evidence base than we have for almost anything else in this subject, and it points one way.

“There is no harm, so why not”

This is the argument that keeps the file being sold, and it deserves a proper answer rather than a shrug.

Three reasons why not.

It is not free. Not the file — the afternoon, the invoice, and the attention. Every hour on this is an hour not spent on the things our pillar identifies as actually controllable, or on the slower work of being mentioned.

It creates a false sense of having acted. A business that has implemented llms.txt believes it has addressed AI search. It has not, and it will now stop thinking about it for six months, which is the expensive part.

And it decays. A file listing pages that have since moved, changed or been deleted is a document asserting things about your site that are no longer true. If anything does eventually read it, it will read the stale version. Nobody who sells the implementation sells the maintenance.

There is a parallel worth drawing with markup for features that were deleted — the same shape of problem, where the artefact outlives the reason it was created and nobody notices because it is invisible.

What the file is actually for

Now the part the debunking articles miss, and it matters more each quarter.

The file is agent infrastructure.

OpenAI uses it in its Agents SDK and in its agentic commerce work. Anthropic recommends it in guidance on writing documentation that agents will read. The use case is not a search engine deciding whether to quote you. It is a piece of software navigating your site to complete a task — reading your documentation, understanding your product structure, potentially transacting.

Which turns the question into a different and more useful one. Not whether llms.txt helps you rank, but whether you want software agents to be able to work with your site.

For most Indian businesses the answer today is no, or not yet. A services firm taking enquiries by form and phone has nothing for an agent to do.

For some, the answer is already yes. If you sell software, publish an API, or maintain documentation that developers use, a clean machine-readable map of that documentation is worth having and has nothing to do with marketing. If you run ecommerce, agentic purchasing is a real direction of travel and being legible to it is a reasonable bet, though it is a bet rather than a certainty.

So: implement it if you are a documentation-heavy product or preparing for agent commerce. Do not implement it because somebody told you it affects AI visibility. Those are two different projects with two different owners, and only one of them belongs to marketing.

It is not the other file

The comparison people reach for is the long-established file that tells crawlers what they may and may not visit. The two get discussed together because they share a naming convention and a location, and they are doing different jobs.

The older file is a permission statement. It says what a visiting crawler should not fetch. It is honoured by convention rather than enforced by anything, and the well-behaved crawlers do respect it — which makes it consequential, because a mistake in it can remove you from search entirely.

llms.txt is a content map. It asserts nothing about permission and grants nothing. It is an offer of a tidier route, and the evidence says almost nothing takes the offer.

Which decision belongs to which file also differs. Whether to allow the AI crawlers onto your site at all is a real decision with real trade-offs, and it is the subject of a separate article in this cluster. It is made in the permission file and it has nothing to do with this one.

The practical upshot: getting llms.txt wrong costs you an afternoon. Getting the permission file wrong can cost you your visibility. Keep them separate in your head and in whoever’s hands you put them.

The full-text variant, and why it is worse

A companion file is often proposed alongside — a single document containing the complete text of your site rather than a map of it.

For a small documentation set this is reasonable and genuinely useful to a developer.

At the scale of an ordinary business website it is a maintenance liability. You are creating a second full copy of your content, in a different format, that nothing updates automatically. Within a few months it describes a version of your site that no longer exists.

Duplicated content that drifts out of sync is a problem in every system that has ever had it. There is no reason to expect this one to be different, and a stale complete copy is worse than a stale map because there is more of it to be wrong.

If you already have either file, the decision is straightforward. Keep it if somebody owns updating it and can say why it exists. Remove it if nobody can answer both.

What to say to whoever quoted you

If the quote describes llms.txt as a way to appear in AI answers, ask three questions.

Which system fetches the file, and can they show you it being requested in your server logs.

What Google has said about it.

And what happens to the file when your site changes.

An honest supplier will tell you it is a developer artefact, that Google has ruled it out, and that it needs maintaining. Anyone who answers otherwise is either behind the evidence or hoping you are.

The test that works on the next one

There will be another. There is always another — a file, a tag, a directory, a submission, something that can be completed quickly and demonstrated in a report.

Here is a test that catches most of them.

If a visibility measure can be completed in twenty minutes by somebody who has never met your business and knows nothing about your customers, it is not a visibility measure.

Visibility comes from being useful to a specific set of people in a specific market, and no artefact placed at a fixed path on a server can supply that. The things that work in this subject are slow, particular to you, and impossible to deliver without understanding what you do.

That is inconvenient for everyone selling and for everyone buying. It also holds up remarkably well as a filter.

What we cannot tell you

We cannot tell you what OpenAI, Anthropic or Perplexity do with the file at retrieval time. None has published a clear statement about whether their assistants fetch it when answering a question, and their positions may differ from each other.

So the honest reading is that Google has ruled it out explicitly, the measured behaviour shows almost nothing fetching these files, and the remaining systems have not said. That is not the same as proof of nothing, and we would rather state it that way than overclaim in the direction we happen to favour.

If that changes, this page changes, and we will say what changed rather than editing quietly.

Final thoughts

The proposal was good. The product built on it is not.

If you are a developer with documentation, this is your file and it has nothing to do with search. If you are a business hoping to be quoted in AI answers, the file will not do it, and the money is better spent on being worth quoting.

If you have a quote in front of you and want a second opinion, you can reach out to us on whatsapp at +91 7738844851 .