llms.txt explained: what it is and whether your site needs one

In short

• llms.txt is a proposed Markdown file, placed at an address such as yoursite.com/llms.txt, that gives AI tools a short summary of a website and a list of its most useful pages.
• It is not an official standard. Google says no AI text file is needed to appear in its AI features, and a large 2026 log study found that 97% of llms.txt files received no requests at all in a month.
• For most Hong Kong businesses it is a nice-to-have. Crawler access, consistent facts and clear bilingual answers come before it.

What llms.txt is

llms.txt was proposed by Jeremy Howard in September 2024, and the proposal is published at llmstxt.org. The idea is simple: AI tools work best with concise, expert-level information gathered in one place, while normal web pages are full of menus, scripts and adverts. A plain Markdown file can point an AI assistant straight to the pages that matter.

The proposal describes a fixed shape:

1. A title (H1) with the name of the site or project. This is the only required part.
2. A short summary in a blockquote, saying what the site is about.
3. Optional notes in plain paragraphs or lists.
4. Sections (H2) of links, each written as a link with an optional note after a colon, such as "Services: what we offer and how projects run".
5. An "Optional" section for secondary links that an AI tool can skip when it needs less context.

The file can sit at the root of a site or under a sub-folder, covering the pages beneath it. The proposal also suggests offering clean Markdown copies of important pages, for example by adding ".md" to the page address.

Who uses it today

The file is most common on software documentation sites. The payment company Stripe, for example, publishes a documentation llms.txt that gives coding assistants instructions and a list of key guides. That use case makes sense: a developer asks an AI coding tool a question, and the tool can fetch a tidy index instead of crawling hundreds of pages.

For ordinary business websites the picture is different. Common Crawl analysed the llms.txt files it found in its July 2026 crawl, 584,107 non-empty files in all. It found that 68.27% were generated from templates, with one website builder alone producing 41.34%. About half (49.90%) followed the full shape of the proposal, but only 32.94% of link entries included the descriptive note, and 22.56% of files had no links at all. In other words, many files exist because a platform switched them on, not because someone wrote them with care.

Do AI tools actually read it?

This is the key question, and the evidence so far says: rarely.

• Google states on its AI features page that you don't need to create new machine-readable files, AI text files or markup to appear in AI Overviews or AI Mode, and that there is no special schema to add.
• Ahrefs studied 137,210 domains in May 2026. About 28% had a valid llms.txt, an upper bound because its customers skew technical. Of those files, 97% received zero requests that month. AI retrieval bots made up only 1.1% of requests to the files; SEO audit tools made far more.
• The same study noted that AI agents did fetch the file when they were pointed to it, which is why it can still be useful on documentation sites.

So llms.txt is not a ranking factor and not a shortcut into AI answers. It is a convenience file that some agents may use when they already know it is there.

Risks to keep in mind

Two problems come up in the data. First, some site owners use llms.txt as if it were robots.txt, writing rules such as "do not train on this site". Common Crawl found that 6.59% of files contained policy language of this kind, and it called this "a robots.txt by mistake". The file does not block anything; access rules belong in robots.txt.

Second, because AI tools read the file as plain instructions, it can carry prompt-injection text. Common Crawl found a small number of files with deliberate injections. Ahrefs recommends treating the file like code: keep it under version control, limit who can edit it, and avoid wording that reads like commands to the AI.

A practical approach for Hong Kong businesses

1. Fix the basics before anything else. Make sure robots.txt allows Googlebot, OAI-SearchBot and PerplexityBot, and that your company facts match everywhere. Our GEO guide covers these steps.
2. If your platform offers llms.txt, read what it generates. Template files often list random pages or none at all. Edit it so the summary describes your business accurately and the links point to your services, case studies and contact page.
3. Keep it short and factual. One title, one summary, a few sections of links with a one-line note each. Include both your English and Chinese pages if you publish in both languages.
4. Do not put rules or instructions in it. Use robots.txt for access, and keep the tone descriptive.
5. Check your server logs. See whether any AI tool requests the file before spending more time on it. Our article on measuring AI visibility explains what is worth tracking.

If you are not sure where llms.txt fits in your plan, our AI marketing and GEO service starts with a check of exactly these technical basics.

The short version

llms.txt is a sensible idea with little proof of impact so far. It costs little to add a clean, accurate file, but it will not make up for blocked crawlers, inconsistent facts or thin content. Put your effort into the pages AI tools already read, and treat llms.txt as a tidy extra.