Can AI crawlers read my website?

Only if you let them. AI answer engines send their own crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, OAI-SearchBot) and they obey your robots.txt. If your site blocks them, or hides content behind a login or JavaScript-only rendering, the engines can't fetch your pages. A page they can't fetch can't be cited.

Which AI crawler user-agents should I allow?

The main ones are GPTBot and OAI-SearchBot (ChatGPT), ClaudeBot and anthropic-ai (Claude), PerplexityBot (Perplexity), and Google-Extended (which governs Gemini and Google's AI answers). Allow them in robots.txt so your public pages stay eligible to be cited.

Does robots.txt affect whether AI cites me?

Yes. Compliant AI crawlers respect robots.txt. If you disallow a page or block the crawler, the engine won't fetch it, so it can't quote or cite it. Crawlability is the precondition for citation.

What's the difference between crawlability and GEO?

Crawlability is whether AI can reach and read your page at all. GEO is whether, once it can read you, it actually names and recommends you. Crawlability is step zero; GEO is the work that follows.

Do I need a sitemap for AI search?

It helps. A sitemap.xml tells crawlers which pages you want found and when they changed, so they spend crawl budget on the content meant to be cited instead of guessing. Pair it with a robots.txt that welcomes AI bots.

Can AI Even Read Your Site? The Technical Foundation of GEO

The short answer

Before an AI engine can recommend your brand, it has to be able to read your site. And plenty of sites quietly stop that from happening. AI answer engines send their own crawlers, and they obey your robots.txt. If those crawlers are blocked, or your content only appears after a login or a pile of JavaScript, the engine never sees the page. A page it can't read can't be cited. This is the part of GEO nobody talks about: crawlability comes before content.

New to the topic? Start with What Is GEO? The Complete Guide.

Crawlability before content

Most GEO advice jumps straight to "write better content." Fair, but it skips a step. An AI engine builds its answer from pages it can actually fetch. If it can't fetch yours, the quality of your writing is beside the point, because you're not even in the running.

So the first question isn't "is my content good enough to be cited." It's "can the engine reach it at all." Get that wrong and everything downstream is wasted effort.

The crawlers you need to let in

When someone asks ChatGPT or Perplexity a question, the answer draws on content those companies' crawlers have already fetched. Each engine runs its own, and you control access to all of them from one file: robots.txt.

The user-agents worth knowing by name:

GPTBot and OAI-SearchBot: OpenAI's crawlers, for ChatGPT and ChatGPT search.
ClaudeBot and anthropic-ai: Anthropic, for Claude.
PerplexityBot: Perplexity.
Google-Extended: Google's control for Gemini and its AI answers, separate from the regular Googlebot.

Naming them matters, and not only for your robots.txt. Being specific and concrete is also how content gets cited in the first place. More on that in How to Get Cited by AI.

How to make sure AI can read you

The checklist is short and mostly one-time:

Welcome AI bots in robots.txt. Don't disallow the crawlers above for your public pages. Blocking them is the fastest way to go invisible in AI answers. Wall off only what shouldn't be public anyway: your app, admin, and internal APIs.
Ship a real sitemap.xml. It tells crawlers which pages you want found and when they last changed, so they spend their limited crawl budget on the pages meant to be cited, not on favicons and app routes.
Declare an absolute canonical URL. Make sure every page states its real, full address, so engines don't get confused by duplicates or relative links.
Don't hide public content behind a login or JavaScript-only rendering. If a human has to click "sign in" or run heavy JavaScript to see the words, assume the crawler sees nothing. Server-render the content you want quoted.
Forward-looking bonus: llms.txt. An emerging convention, like robots.txt but a curated map of your most useful content for LLMs. Not required yet, but cheap insurance as the standards settle.

We had to check this on our own site

Here's the honest part. We build a tool that measures AI visibility, and we still had to make sure theogeo.ai itself was readable by the engines. So we did it on purpose. Our robots.txt welcomes GPTBot, ClaudeBot, PerplexityBot, Google-Extended, and OAI-SearchBot by design, and walls off only the logged-in app and internal routes. Our sitemap lists every public page and picks up new blog posts automatically, including this one.

If a GEO product shipped without doing that, it would be a bad look. More to the point, it wouldn't practice what it sells.

Crawlability is step zero. GEO is the rest.

Being readable doesn't mean you'll be recommended. It means you're eligible. Once an AI engine can reach your pages, the real question is whether it actually names you when a buyer asks, and that's what the rest of GEO is for. (See How to Get Cited by AI for the content side of the work.)

See where you stand

Crawlability gets you into the room. Whether AI actually recommends you is the thing you can measure. theogeo.ai runs a free audit across ChatGPT, Perplexity, Gemini, and Claude and shows you where you're named, where you're not, and who's getting picked in your place. No credit card, no sales call, about five minutes.

Make sure the engines can read you. Then go find out what they say.

Sources

OpenAI, GPTBot and OAI-SearchBot crawler documentation. platform.openai.com/docs/bots
Google, "Overview of Google crawlers and fetchers" (Google-Extended). developers.google.com