The Short Answer

Block the wrong crawler in robots.txt and your pages drop out of ChatGPT search, however good your llms.txt is. The three files do different jobs. robots.txt tells crawlers what they may fetch. The sitemap lists the pages you want found and when each last changed. llms.txt is a proposed Markdown summary of a site for language models. As of 17 November 2026, Google says its Search does not use such files, and the crawler pages from OpenAI and Perplexity say nothing about their crawlers reading one. Fix them in that order: access first, the page list second, the summary last.

Short on time? Ask an AI to summarise this post:

Developers often meet llms.txt described as "the robots.txt for AI". The comparison costs something. A team adds the new file, feels finished, and leaves a Disallow line in the old one that keeps ChatGPT's search crawler out. The three files sit side by side at the root of a site and they do three different jobs.

This post is for the person who edits those files. It says what each one controls, what Google, OpenAI and Perplexity document about reading it, and where the published record stops. Every quote below was read on 17 November 2026 on the page named beside it.

What does each file do, in one sentence?

robots.txt governs access, the sitemap helps discovery, and llms.txt offers a summary.

  • robots.txt tells a crawler which paths it may fetch. It is a request that well-behaved crawlers honour.
  • sitemap.xml lists the URLs you want found, with the date each last changed. It is a hint and it grants nothing.
  • llms.txt is a Markdown file that describes the site and links to its main pages, written for a language model to read. It is a proposal from 2024.

The first can remove you from an answer. The second can help a new page get found. The third has no documented effect on either.

What does robots.txt control, and what does it leave alone?

robots.txt controls whether a crawler fetches a page. It does not control whether the page is indexed. Google's introduction to robots.txt says the file "is used primarily to manage crawler traffic to your site" and that "it is not a mechanism for keeping a web page out of Google". The same page says a disallowed page "can still be indexed if linked to from other sites". Google's remedy for a page you want kept out is a noindex rule or a password.

For AI answers the file decides more, because each company runs several crawlers and reads the file once per crawler. OpenAI's crawler page lists four of them as of 17 November 2026: OAI-SearchBot for ChatGPT search, GPTBot for model training, ChatGPT-User for fetches a person triggers, and OAI-AdsBot for checking ad landing pages. The page says "Each setting is independent of the others" and that sites opted out of OAI-SearchBot "will not be shown in ChatGPT search answers". Perplexity's crawler page lists PerplexityBot and Perplexity-User, and recommends allowing PerplexityBot.

So one file carries two separate decisions: whether a company may train on your pages, and whether its search product may show them. A site can refuse the first and allow the second. Our post on the shared training signal that Google and OpenAI announced on 30 October 2026 covers that choice.

Two limits apply. Google's page says "other crawlers might not" obey the file, so it is no defence against a scraper. And a change is slow to land: OpenAI's page says "it can take ~24 hours from a site's robots.txt update for our systems to adjust", and Perplexity's page says "it may take up to 24 hours".

What does a sitemap do, and does a small site need one?

A sitemap tells a search engine which URLs exist and when they changed, and a small, well-linked site can live without one. Google's sitemap overview defines it as "a file where you provide information about the pages, videos, and other files on your site". The same Google page says a site may not need one when it is small, which Google puts at "about 500 pages or fewer", and "comprehensively linked internally". Google also says a sitemap "doesn't guarantee that all the items in your sitemap will be crawled and indexed".

Google's page on building a sitemap settles the details that waste the most time:

  • Google's page says "All formats limit a single sitemap to 50MB (uncompressed) or 50,000 URLs."
  • Google says it ignores the priority and changefreq values, so leave them out.
  • Google uses the lastmod date only when it is consistently accurate, so set it when the page changed and at no other time.
  • Google's page says "Use fully-qualified, absolute URLs in your sitemaps."
  • Tell Google where the file is with a Sitemap line in robots.txt, or submit it in Search Console.

The AI engines say less. As read on 17 November 2026, the crawler pages from OpenAI and Perplexity do not mention sitemaps. Google's guide to its generative AI features says that to be eligible "a page must be indexed". So for Google, a sitemap helps with AI answers the same way it helps with ordinary search: it helps pages get found and indexed.

What is llms.txt, and who reads it?

llms.txt is a proposal that Jeremy Howard published at llmstxt.org on 3 September 2024, and as of 17 November 2026 neither Google, OpenAI nor Perplexity documents using it to choose sources for an answer. The proposal reads: "We propose adding a /llms.txt markdown file to websites to provide LLM-friendly content." The format is short. An H1 carries the name of the site, and the proposal says "This is the only required section". A blockquote summary follows, then H2 sections that hold lists of links.

The proposal separates the file from the sitemap in its own words: "While sitemaps list all pages for search engines, llms.txt offers a curated overview for LLMs."

Here is what each company's page says.

  • Google. Its generative AI guide says "You don't need to create new machine readable files, AI text files, markup, or Markdown to appear in Google Search", and adds that Google Search "itself doesn't use them". The guide also says "It's completely fine if you decide to create and maintain LLMS.txt files" for other services that use them.
  • OpenAI. The crawler page names llms.txt once, as a link to the index of OpenAI's own documentation. It says nothing about OAI-SearchBot reading the file on your site.
  • Perplexity. The pattern is the same. The page points to Perplexity's own llms.txt for its documentation and says nothing about PerplexityBot reading yours.

The proposal's site notes that "The AI labs themselves publish llms.txt files for their own developer docs". Publishing a file and reading other people's files are different acts, and only the first is documented.

We give no adoption percentage here, because we have found no primary source for one. The measurement you can trust is on your own server. Search the access log for requests to /llms.txt and read the user agents that made them.

How do the three files work together?

The files point to each other, and they should never contradict each other.

  1. robots.txt carries the Sitemap line, so any crawler that reads the first file learns where the second one is.
  2. Every URL in the sitemap should be one that robots.txt allows. A URL listed in one file and blocked in the other tells the crawler two things at once.
  3. Every link in llms.txt should resolve, should be allowed in robots.txt, and should describe the page as it is today. A summary with last year's prices is a second version of your facts for a machine to choose from.

Which file should you fix first?

Fix access, then the page list, then the summary.

  1. Read robots.txt. Open yourdomain.com/robots.txt and look for Disallow lines under the catch-all user agent and under each named crawler. When this is done you will see Googlebot, OAI-SearchBot and PerplexityBot free of any rule that covers your public pages.
  2. Make the training decision on purpose. GPTBot and Google-Extended are training controls. Allow them or refuse them, and write the reason in a comment. When this is done each one has a deliberate line in the file.
  3. Check the sitemap. Open the sitemap URL, find your newest page in it, and confirm the lastmod dates are true. Submit it in Search Console once. When this is done the Sitemaps report shows the file as read, with a count of the pages found.
  4. Wait a day and look again. OpenAI and Perplexity both document a delay of up to about 24 hours before a robots.txt change takes effect. When the day has passed, the rules you wrote are the rules in force.
  5. Write llms.txt if you want one. An H1, a blockquote that says who you are and what you do, and links to the pages that matter. When this is done the file loads as plain text at yourdomain.com/llms.txt.
  6. Read the access log after a month. When this is done you will know which user agents asked for the file, which is more than any published source can tell you.

What do we run on nexum-sol.com?

We run all three, and we can vouch for two. Our robots.txt was last edited on 11 June 2026. It allows every crawler by default, names the AI crawlers with explicit Allow lines, and ends on the Sitemap line. Allowing the training crawlers was a decision for our own business, and yours may differ. Our sitemap is rebuilt by the deploy script each time a post goes out. Our llms.txt gains one line per post from the same build.

The first two do documented work. We keep the third because it costs one line per post, and we have no measurement that shows it changed any answer. If a client's budget covered two of the three files, the two would be robots.txt and the sitemap.

Four mistakes worth checking for

  • Using Disallow to hide a page. Google says a disallowed page can still be indexed when other sites link to it. Use noindex or a password.
  • A blanket block. A rule that refuses every user agent it does not recognise takes the search crawlers out along with the scrapers. Name the crawlers you want and allow them.
  • Treating llms.txt as a permission file. It allows and refuses nothing. Access lives in robots.txt.
  • A sitemap full of dead ends. Redirected, deleted and blocked URLs do not belong in it. List the pages you want found, at their final addresses.

Three small files, read in that order, cover the ground. The oldest of them is the one that can cost you an answer today.

Want the full list of checks?

Our free AI Visibility checklist covers 12 checks, the crawler rules and the sitemap among them, starting with what ChatGPT, Gemini, and Google say when someone asks about a business like yours. Enter your email and we will send it over.

✓ Check your inbox, your checklist is on the way!

We respect your privacy. Unsubscribe at any time.

Key Takeaway

robots.txt decides whether a crawler may read you, the sitemap helps it find your pages, and llms.txt is an optional summary with no documented effect on any answer engine. Check the first two before you spend an hour on the third.

One forward before you go: open yourdomain.com/robots.txt in a browser and read it from top to bottom. If Googlebot, OAI-SearchBot or PerplexityBot falls under a Disallow line that covers your public pages, that is the first fix, and it comes before any new file.

Frequently asked questions

Is llms.txt the same thing as robots.txt?

No. robots.txt tells crawlers which paths they may fetch, and the main crawlers document that they follow it. llms.txt is a Markdown summary of a site with links to its main pages, proposed in 2024 for language models to read. It grants and refuses nothing, so it cannot replace a rule in robots.txt.

Does ChatGPT read llms.txt?

OpenAI does not say so. As of 17 November 2026, OpenAI's crawler page mentions llms.txt once, as a link to the index of OpenAI's own documentation, and says nothing about OAI-SearchBot reading the file on other sites. The way to know for your own site is to search your server's access log for requests to /llms.txt and read which user agents made them.

Does Google use llms.txt?

Google says it does not. Its guide to optimising for generative AI features says you do not need to create new machine readable files, AI text files, markup, or Markdown to appear in Google Search, and that Google Search itself does not use them. The same guide says it is fine to keep such a file for other services that use it.

Does a small website need a sitemap?

Often it does not. Google's sitemap overview says a site may not need one when it has about 500 pages or fewer and is comprehensively linked internally. A sitemap is still cheap to generate, and it tells Google when a page last changed, so most sites that publish regularly keep one.

Can robots.txt keep a page out of Google?

No. Google's introduction to robots.txt says the file is not a mechanism for keeping a web page out of Google, and that a disallowed page can still be indexed if other sites link to it. Google's remedy on that page is a noindex rule or password protection.

If I block GPTBot, do I disappear from ChatGPT search?

Not according to OpenAI's crawler page, which says each setting is independent of the others. GPTBot is the training crawler. OAI-SearchBot is the one that surfaces sites in ChatGPT search, and OpenAI says sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers.

Does a sitemap help with ChatGPT or Perplexity?

Neither company documents it. As of 17 November 2026, the crawler pages from OpenAI and Perplexity do not mention sitemaps. For Google, a sitemap helps pages get found and indexed, and Google says a page must be indexed before it can appear in its generative AI features.

References & sources

  1. Google Search Central, Introduction to robots.txt (read 17 November 2026)
  2. Google Search Central, Learn about sitemaps (read 17 November 2026: definition, when a site may not need one, no guarantee of crawling or indexing)
  3. Google Search Central, Build and submit a sitemap (read 17 November 2026: size limits, lastmod, priority and changefreq, the Sitemap line)
  4. Google Search Central, Guide to optimizing for generative AI features on Google Search (read 17 November 2026: no new machine readable files needed, indexing as the condition)
  5. OpenAI, OpenAI bots documentation (OAI-SearchBot, OAI-AdsBot, GPTBot, ChatGPT-User) (read 17 November 2026)
  6. Perplexity, Perplexity crawlers (PerplexityBot, Perplexity-User) (read 17 November 2026)
  7. Howard, J., The /llms.txt file, llmstxt.org (3 September 2024) (the proposal; read 17 November 2026)
A
Abdulmuiz Adebowale
AI Product Manager & Builder, Nexum-sol

Abdulmuiz helps Nigerian businesses build websites and digital systems that get found, by people and by AI. He writes about practical SEO, AI search, and growth systems for SMEs at Nexum-sol.