How AI Crawlers Decide What to Index (And What They Skip)

A domain owner turned on Cloudflare's "block AI scrapers" feature last year. It was one checkbox. Took two seconds. Felt like a smart move. You don't want your content scraped without attribution. Who would?

Six months later, they were searching for their own services in Perplexity. Typing their exact specialty. The thing their site covered in more detail than anyone else in the space. Not one result mentioned them.

They asked a competitor in a forum thread: "How do you show up in Perplexity answers?" The competitor had no idea what they were talking about. They'd never thought about it. They'd also never blocked AI crawlers.

One checkbox. Six months of invisibility. The difference wasn't content quality or authority or history. It was a two-second decision about access.

The AI Crawler Ecosystem

Google's crawler is 25 years old. There's extensive documentation, established behavior, and an entire industry built around understanding it. You probably know Googlebot even if you've never thought about crawlers.

AI crawlers are a different world. They're new, they're growing fast, and their behavior is still being documented as it changes. But the main ones are now identifiable, and knowing who they are matters:

GPTBot is OpenAI's crawler. OpenAI has documented it in their own help pages, including the exact user agent string and how to block it via robots.txt. GPTBot is used both for training data collection and for ChatGPT's Browse capability. These are technically separate crawls but use related infrastructure.

ClaudeBot is Anthropic's crawler, used for training data. Documented in Anthropic's publicly available documentation.

PerplexityBot is Perplexity's real-time indexer. This crawler is how Perplexity stays current with web content and serves live answers. Unlike training crawlers, PerplexityBot is visiting your site regularly to stay updated.

Google-Extended is Google's opt-out mechanism for AI training data specifically. Standard Googlebot still crawls your site. Google-Extended is the crawler you're blocking when you want to prevent your content from being used in Gemini's training data but still want regular Google indexing.

CCBot is Common Crawl's crawler. Common Crawl is a nonprofit archive of the web that many AI companies use as training data. As of 2024, Common Crawl has crawled approximately 250 billion web pages, but only 15 to 20 billion make it into high-quality training sets. The quality filters are: original content, sufficient length, language identification, and deduplication. If your site makes it into Common Crawl's quality set, it becomes part of the training data many AI systems draw from.

What Makes Cloudflare's Default a Problem

Cloudflare reported in 2025 that AI crawler volume had reached 7 to 8 times the volume of traditional search crawl activity. That number drove a response: Cloudflare began offering, and in some configurations enabling by default, a "block AI scrapers" feature.

I get why that feature exists. If you don't want AI companies training on your content, blocking their crawlers is the direct way to prevent that. It's a legitimate choice.

But here's what the feature doesn't tell you: it doesn't distinguish between training crawlers and citation crawlers. PerplexityBot, which is what gets your site cited in live Perplexity answers, gets blocked alongside training crawlers. GPTBot's Browse crawler, which feeds ChatGPT's real-time answers, gets blocked too.

The result: sites using aggressive Cloudflare bot blocking for protection are often inadvertently opting out of AI discovery entirely. They wanted to protect their content. They ended up hiding it.

This is the defining tension in 2026: the same mechanisms designed to protect your content from being used without attribution are also the mechanisms that prevent AI systems from citing you with attribution. You have to make an intentional choice about which problem you're solving.

What AI Crawlers Actually Look For

So what does "making your site readable to these systems" actually mean? Knowing the crawlers by name is useful, but what matters more is understanding what they're looking for when they arrive. And what makes them leave.

Structured, parseable content. AI crawlers want content they can extract meaning from. Semantic HTML with clear heading hierarchies, schema markup that identifies the content type, and plain-text body content all make extraction easier. JavaScript-heavy pages where content only renders after scripts execute are harder to parse. Many AI crawlers don't execute JavaScript the same way browsers do, meaning content that requires JS rendering may simply be invisible to them.

Markdown-friendly formatting. Research from 2025 indicated that AI crawlers show preference for markdown-formatted or cleanly structured content. Lists, headers, code blocks, and direct prose are more parseable than complex CSS-styled layouts where the visual design obscures the content hierarchy.

Clear topic focus. A page clearly about one thing is easier to categorize and cite than a page that meanders across multiple topics. AI systems building knowledge graphs prefer clear topical signals.

Freshness indicators. Pages with clear publication and modification dates tell crawlers whether the content is current. A page with no date metadata is harder to place in time, which matters for AI systems trying to provide current information.

Sufficient content depth. Common Crawl's quality filtering includes a "sufficient length" criterion. Very short pages, thin pages with minimal text, and pages that are primarily images or video with little text may be filtered out of quality training sets.

No blocking signals. The most basic thing: a crawler that sees Disallow: / in your robots.txt for its user agent will simply not crawl. No exceptions. If you're blocked, you're not indexed.

What AI Crawlers Skip

Understanding what gets skipped is just as important as understanding what gets indexed. Sometimes more important, because you might be doing everything right on the content side and still getting skipped for one of these reasons.

Pages blocked in robots.txt. This is the explicit skip. Any page or path covered by a Disallow rule for a crawler's user agent is not visited. If your entire site is blocked, the crawler visits your robots.txt, reads the block, and leaves. It's like putting a "closed" sign on a store that's fully stocked.

Pages with thin or duplicate content. Quality filtering removes pages below certain length thresholds and pages with content that appears elsewhere on the web (deduplication). If your pages are very short or contain syndicated content that's published elsewhere, they're at higher risk of being filtered from quality training sets.

JavaScript-only content. If your page content exists only after JavaScript executes, many AI crawlers won't see it. Content that requires a browser-like JS execution environment to appear may be invisible to crawlers that don't run scripts.

Dark or parked domains. A domain with no content, a default parking page, or only a coming-soon placeholder presents nothing to index. The crawler visits, finds no meaningful content, and leaves. This isn't a penalty. It's simply nothing to index. For the specific question of what this means for domain owners, Can a Domain With No Content Show Up in AI Search? covers it directly.

Pages without clear language identification. Common Crawl and many AI training pipelines include language identification as a quality filter. Pages where the language can't be reliably identified (or pages with very mixed-language content) may be filtered.

How to Configure AI Crawler Access Intelligently

The goal isn't to block everything or allow everything. The goal is to make an intentional choice about who gets in and why.

Here's a configuration framework that gives you control without giving up visibility:

Allow citation crawlers, decide on training crawlers.

PerplexityBot drives Perplexity citations. GPTBot drives some ChatGPT Browse results. These are citation-mode crawlers. If your goal is to appear in AI-generated answers, these should be allowed to access your public content.

Training crawlers (CCBot, the training-specific portion of GPTBot) are a different question. If you have concerns about your content being used in AI training data without attribution or compensation, you can block CCBot specifically without necessarily blocking all AI discovery.

Your robots.txt can be specific:

# Allow Perplexity
User-agent: PerplexityBot
Allow: /

# Allow OpenAI's Browse
User-agent: GPTBot
Allow: /

# Block Common Crawl training scraper if preferred
User-agent: CCBot
Disallow: /

This isn't an official recommendation. It's an illustration that the decision can be more nuanced than "block all" or "allow all."

Check your Cloudflare settings explicitly.

If you use Cloudflare, log into your dashboard and look at your Bot Management or Security settings. Any "block AI scrapers" toggle should be reviewed. Make a deliberate choice rather than accepting the default.

Allow access to your public content.

Whatever you decide about training data, the pages you want cited should be accessible to the crawlers that drive citations. Your homepage, your key topic pages, your FAQ content, your published writing. These are the pages worth being intentional about.

The Block vs. Welcome Decision

Here's the thing. The blocking instinct is understandable. AI companies have been using web content to build commercial products. Site owners have legitimate questions about whether that's fair. Those questions deserve honest answers, and the industry hasn't provided them yet.

But blocking AI crawlers as a protest or a default isn't neutral. It has a direct, measurable consequence: you stop appearing in the AI-generated answers that are increasingly where discovery happens. The domain owner who flipped that Cloudflare checkbox learned this over six months of silence. You can learn it in six seconds by reading your robots.txt.

Welcoming AI crawlers on your terms, with intentional robots.txt configuration and structured content that invites parsing and citation, is how you get cited. Blocking everything is how you disappear. And the difference between those two outcomes is a deliberate decision, not a default setting.

Why AI Can't Read Your Parked Domain covers the extreme case of this. The Internet Now Has Two Audiences frames why both audiences matter. Two Audiences of Every Domain extends that frame. Domain Parking Is Not Neutral explains why the parking default sends active skip signals to these crawlers.

The full machine readability foundation is in The Beginner's Guide to Making Your Website Machine-Readable. AI crawler access is one layer. Schema markup, semantic structure, and entity signals are the other layers. They all work together.

SiteWarming is fundamentally about welcoming the right machines on your terms. Intentional access, structured content, and clear signals that say: here's what this domain is, here's why it's worth citing, here's how to read it. An active domain has its robots.txt configured correctly. Its public content is open to citation crawlers. Its schema markup tells those crawlers exactly what they're reading. The invitation is built into the structure. What is SiteWarming?

Related Reading

• The Beginner's Guide to Making Your Website Machine-Readable
• Can a Domain With No Content Show Up in AI Search?
• Why AI Can't Read Your Parked Domain
• Two Audiences of Every Domain

Image Credits

• Hero: Photo by Tamas Pap on Unsplash