How ChatGPT indexes your website

Things are changing fast, and what SEO asks of us is changing with them. It is no longer only about whether your page is well indexed in Google. With ChatGPT, Claude and other AI models around, people are not only searching through Google — they ask the AI directly. So how do you get ChatGPT to surface your business when someone asks about it? Let us go through it.

There are several ways to get your pages into ChatGPT:

• Indexing prompts
• ChatGPT crawlers
• llms.txt

Indexing prompts

With this method you are trying to get a mention into ChatGPT that your site exists at all. You can ask it directly to find your domain — and once it has searched with your domain through its search tools, ChatGPT stores it in what is called a knowledge graph, its database of what it knows. Next time it therefore “knows” about your page and can offer it to other users.

ChatGPT crawlers

ChatGPT was trained on text from across the internet. To gather that text data, the platform uses what are called web crawlers. Search engines like Google, Bing and Seznam.cz use the same method. To have ChatGPT remember our page, we can allow OpenAI's specific crawlers to read it. These permissions are always defined in the robots.txt file.

The robots.txt file sits at the root of the domain (for example https://yourdomain.com/robots.txt) and has a simple structure — for each robot (User-agent) you say what it may (Allow) and may not (Disallow) do, and at the end you point to the sitemap (Sitemap):

# OpenAI / ChatGPT
User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

# Bing (the search engine behind ChatGPT)
User-agent: bingbot
Allow: /

Sitemap: https://yourdomain.com/sitemap.xml

The individual robots serve different purposes: GPTBot collects data for training, while OAI-SearchBot and ChatGPT-User handle searching and browsing pages while an answer is being produced. You can read more about OpenAI's crawlers in their documentation: developers.openai.com/api/docs/bots.

The other thing is that ChatGPT uses search tools in the background. Because OpenAI is closely tied to Microsoft, the one it uses is Bing. So in robots.txt we should also allow Bing's crawler (bingbot) to read our page — as in the example above.

llms.txt

llms.txt is a newly proposed standard that helps ChatGPT and other AI models better understand what is on your page. You put the structure of the site and its content into it, along with a short description of what the site is about. Think of it as a purpose-built reader for language models — instead of chewing through the whole HTML, they get a clear map of what matters most.

The file goes at the root of the domain (https://yourdomain.com/llms.txt) and is written in Markdown. The structure is simple — a heading with the name of the site, a short description, and then links sorted into sections:

# Site name

> One sentence on what the site does and who it is for.

## Main pages

- [Home](https://yourdomain.com/): A brief description of the page
- [Services](https://yourdomain.com/services): What you offer
- [About](https://yourdomain.com/about): Who is behind the site

## Blog

- [Article title](https://yourdomain.com/blog/article): What the article is about

## Contact

- [Contact](https://yourdomain.com/contact): How to reach you

The above is only an illustration of the structure — you will of course adapt the content to your own site. What matters is that every link has a short, accurate description, so the model can tell what it will find on that page.