Cloudflare AI Crawl Control: How to Allow, Block, or Charge AI Crawlers

If your site sits behind Cloudflare, the first place AI crawlers meet your policy is not your robots.txt file. It is the edge. Cloudflare sees every request before your origin does, which means a single setting can decide whether GPTBot, ClaudeBot, PerplexityBot, or a search-oriented AI crawler ever reaches a page.

That is powerful, and it is also how sites accidentally disappear from AI search. This guide explains what AI Crawl Control does, how the allow, block, and charge actions work, what the managed robots.txt adds, and how to set a policy that matches your goals instead of a blanket "block all AI."

Everything below is based on Cloudflare's own documentation, checked in November 2026. Dashboard labels change from time to time, so treat the exact menu names as a guide and confirm them in your account.

What AI Crawl Control is

AI Crawl Control is a section of the Cloudflare dashboard for zones on Cloudflare. According to Cloudflare, it lets you:

• see which AI services access your content and how their request volume changes;
• set allow or block rules for individual crawlers;
• track which crawlers follow your robots.txt directives and which ignore them;
• explore monetization through pay per crawl, which Cloudflare describes as a private beta;
• use it without configuration on every Cloudflare plan.

It builds on Cloudflare's verified bots program. Verified bots are bots whose operators have registered them with Cloudflare, and each one belongs to a category such as AI crawler, AI assistant, search engine crawler, or archiver. Historically, verified bots were excluded from default bot configurations on all plans. Cloudflare now lets every customer define separate allow or block policies for AI crawlers and AI agents.

The practical consequence: Cloudflare can distinguish between bots that train models, bots that build a search index, and assistants that fetch a page because a person asked. Your policy should too.

Three kinds of AI traffic

Before touching any switch, decide which traffic you want. The same company often runs several bots with different jobs.

Purpose	What it does	Typical examples	Usual decision for publishers
Model development	Collects content that may be used to train or tune models	GPTBot, ClaudeBot, CCBot, Bytespider	Block, allow, or charge, depending on your content strategy
AI search and retrieval	Builds an index that AI search products cite and link to	OAI-SearchBot, Claude-SearchBot, PerplexityBot	Usually allow if you want citations and referrals
User-initiated fetch	Fetches a page because a user asked an assistant to open it	ChatGPT-User, Claude-User, Perplexity-User	Usually allow; blocking breaks "summarize this page" requests

Cloudflare's own wording for when to allow is helpful here: allow crawlers "that offer services which provide value through citations, referrals, or existing agreements," and block them "when their behavior does not align with your content strategy, or violate your policies." Charging is suggested for crawlers when your content has training value and you want to explore monetization.

Where to find the controls

According to the Cloudflare documentation:

1. Log in to the Cloudflare dashboard and select your account and domain.
2. Open AI Crawl Control.
3. Use the Crawlers view for the per-crawler table, the Directives view for robots.txt insights, and the Settings view for block-response options.

The Crawlers table lists each crawler and its operator, its category, the number of allowed and unsuccessful requests with a trend chart, the number of robots.txt violations, and the action you have chosen. You can filter by crawler name, operator, or category.

One detail in the documentation deserves attention: the Requests column counts unsuccessful requests that may come from any rule or response error, not only the block action in AI Crawl Control. If a number looks high, check your other WAF rules before assuming the crawler is misbehaving.

The three actions: allow, block, charge

For each AI crawler you choose one action.

Allow

Allow lets the crawler fetch your content. You can still enforce robots.txt on top of an allow action, which is useful when you want a crawler to read some sections but not others.

Block

Block stops the crawler at the edge. When you block a crawler in AI Crawl Control, Cloudflare creates or updates a WAF custom rule on your zone to enforce it. That matters for two reasons:

• the block is a real network-level control, unlike robots.txt, which relies on voluntary compliance;
• you can extend the generated rule in the WAF yourself, for example to add path-based exceptions or extra user agents.

On paid plans you can also configure the block response. You can choose between a 403 Forbidden status, which signals "do not access this content," and a 402 Payment Required status, which signals that the crawler would need to pay. You can also write a plain-text response body. Cloudflare suggests using the body to explain to a crawler operator how to license your content properly.

Charge (private beta)

Charge sets a price for each successful crawl. Cloudflare labels pay per crawl as a private beta, so most sites cannot use it today. If you are in the beta, the Crawlers table exposes a Charge action next to Allow and Block. For everyone else, it is mainly useful to know it exists: a 402 block response is the closest signal you can send today.

Managed robots.txt

robots.txt is voluntary. Cloudflare's documentation says so plainly: some crawler operators may disregard directives such as Disallow: /. Still, a clear robots.txt is the standard way to publish your preferences, and Cloudflare offers a managed version.

When the managed robots.txt is enabled:

• if your origin already serves a robots.txt with HTTP 200, Cloudflare prepends its managed block before your original content and serves the two as a single file;
• if your origin has no robots.txt, Cloudflare creates one containing managed Disallow rules for known AI crawlers.

The managed content in Cloudflare's own example includes a Content Signals preamble, a Content-signal: search=yes, ai-train=no line for all user agents, and Disallow: / rules for crawlers such as Amazonbot, Applebot-Extended, Bytespider, CCBot, ClaudeBot, Google-Extended, GPTBot, and meta-externalagent.

Notice what that example does not do: it does not block Googlebot, Bingbot, or search-oriented AI crawlers. It targets crawlers used for training.

Turning it on

The documentation describes this path:

1. In the Cloudflare dashboard, open Security Settings for the zone.
2. Filter by Bot traffic.
3. Select AI Crawler and Agent access policies.
4. Set Training to Disallow.
5. Turn on Enable Bot Preference Sync.
6. Save.

With Bot Preference Sync on, setting Training to Disallow adds the managed directives and enables edge blocking for AI training crawlers. Cloudflare notes that mixed-use crawlers are excluded from that edge block. The Search and Agent policies add their own managed directives when they are set to Block or "Block on pages with ads."

Read that last sentence twice. Setting the Search policy to Block is the setting most likely to remove your pages from AI answers. Do not touch it unless you mean to.

Content Signals in plain language

Content Signals are machine-readable lines in robots.txt that say how your content may be used. There are three:

Signal	Meaning
search	Building a search index and returning links and short excerpts. Does not include AI-generated search summaries.
ai-input	Feeding content into AI models for real-time answers, such as retrieval-augmented generation or grounding.
ai-train	Training or fine-tuning AI models.

A value of yes grants permission for that use, no denies it, and an omitted signal means the site neither grants nor restricts that use. Cloudflare's preamble also states that the restrictions are an express reservation of rights under Article 4 of the EU Directive 2019/790 on copyright in the digital single market. Whether that is legally effective for your situation is a question for a lawyer, not a configuration file.

One more documented detail: zones on the Free plan that have no robots.txt of their own and do not use the managed feature will display the Content Signals Policy text when a crawler requests robots.txt. That policy defines the categories but does not state any preference. To state a preference such as ai-train=no, configure the access policies and enable Bot Preference Sync.

Reading the Directives view

The Directives view is the best place to confirm that your robots.txt is healthy. It reports:

• whether Cloudflare is managing your robots.txt;
• a Robots.txt availability table with request counts, the HTTP status code of the file, and whether the file contains Content Signals;
• a Robots.txt violations table listing crawlers that requested paths your file disallows, with the exact directive they broke and the number of violations.

The documentation gives practical next steps. If the status is 404, create a robots.txt. If the file exists but crawlers report errors, look for upstream WAF rules or security settings that block it. If a crawler violates your directives, you can block it from the Crawlers view, write a path-specific WAF rule, or use Redirect Rules to steer it toward an appropriate part of the site.

There is also an Agent Readiness card that scans your site against emerging standards for AI agents, including a valid robots.txt, Markdown content negotiation, and Content Signals. It is powered by the isitagentready.com scanner and returns recommendations.

A sensible policy for most content sites

If your goal is visibility in AI search with some control over training, this configuration fits most publishers:

1. Keep search and retrieval crawlers allowed. OAI-SearchBot, PerplexityBot, and similar crawlers are how AI products find pages to cite.
2. Keep user-initiated fetchers allowed. ChatGPT-User, Claude-User, and Perplexity-User act on behalf of a person who asked about your page.
3. Decide separately about training crawlers. Disallow them if you do not want your text used for model development. Remember that this is a business choice, not a security measure.
4. Use the managed robots.txt if you have no file of your own. It is a fast way to publish a policy and Content Signals.
5. Set a block response on a paid plan if you block crawlers, so operators see a clear message.
6. Review the violations table monthly and escalate only the crawlers that actually ignore your rules.

If you run a paid product, documentation behind a login, or licensed content, the calculation changes. Block training crawlers, protect private content with authentication, and consider the charge action if you qualify for the beta.

Common mistakes

Blocking everything labeled AI. Several AI products use separate bots for training and for search. A blanket block removes you from the products you probably want.

Assuming robots.txt is enforcement. It is a request. For a hard stop you need the WAF rule that a Block action creates, or your own firewall rule.

Forgetting that a bot challenge can block good bots. If you run strict bot fight settings or custom rules in front of the same paths, verify with a real fetch that allowed crawlers still receive a 200.

Not checking after the change. After any policy change, request a page with the crawler's user agent and confirm the response. A user-agent check does not prove the request would pass Cloudflare's verification, but it quickly catches a rule that returns the wrong status. Then watch the Crawlers table for a few days.

Ignoring the origin. Cloudflare protects the edge. If your origin or another CDN has its own bot rules, the combined behavior may differ from what AI Crawl Control shows.

How to verify your result

1. Open https://yourdomain.com/robots.txt in a browser and confirm that the Content Signals and Disallow blocks you expect are present.
2. Run the llmsmap audit on your domain to see which AI bots your robots.txt allows and whether llms.txt is reachable.
3. In AI Crawl Control, confirm that the actions in the Crawlers table match your policy.
4. After a week, compare the request counts for allowed crawlers. A sudden drop to zero for a crawler you allowed usually means another rule is blocking it.
5. In your analytics, watch for referrals from AI products. The guide to tracking AI traffic in GA4 explains how.

FAQ

Does AI Crawl Control work on the Free plan?

Cloudflare documents AI Crawl Control as working automatically on all plans, and the managed robots.txt as available on all plans. Some options, such as configuring the block response, are documented as paid-plan features.

Is blocking in AI Crawl Control the same as Disallow in robots.txt?

No. Disallow is a published preference that compliant crawlers follow. A Block action creates a WAF rule that stops matching requests at the edge, whether or not the crawler respects robots.txt.

Will blocking training crawlers hurt my Google rankings?

The managed robots.txt shown in Cloudflare's documentation targets training crawlers and does not block Googlebot. Google-Extended is a control token for use of content in certain Google AI products; it does not change how Googlebot crawls for Search. Always confirm the current behavior in Google's documentation before relying on that.

Can I allow a crawler on some paths and block it on others?

Yes. Use the generated WAF rule as a starting point and add path-based conditions, or use robots.txt path rules together with an allow action.

What is pay per crawl?

It is Cloudflare's mechanism for charging a crawler operator per successful request. Cloudflare describes it as a private beta, so check your dashboard to see whether you have access.

Should I also publish llms.txt?

They solve different problems. Cloudflare policies control access. llms.txt is an emerging convention that points agents to your most useful pages. Publishing it does not guarantee that any assistant will use it. See the llms.txt vs robots.txt vs sitemap comparison.

Summary

AI Crawl Control gives you per-crawler allow, block, and charge actions, a managed robots.txt with Content Signals, and visibility into which crawlers ignore your rules. Use it to separate training crawlers from search and user-initiated fetchers, keep the ones that send you citations, and verify the result with a real request. Treat robots.txt as a statement of preference and the WAF rule as enforcement.