Should you block AI crawlers? A decision framework

TL;DR: Block training-only crawlers with low or no referral value (ClaudeBot’s training crawler, CCBot, Bytespider) unless you have a specific licensing reason to allow them. Allow crawlers tied to a citation or search surface that sends you visits back (OAI-SearchBot, PerplexityBot’s search crawler, Google-Extended if you want AI Overviews inclusion). The right call is per-bot, not all-or-nothing, and it depends on your crawl-to-refer ratio, your server cost, and whether you would take a licensing check if one were offered.

---

Table of contents

• The one-sentence answer
• Why this decision got complicated in 2026
• The crawl-to-refer ratio: the number that should drive your decision
• The bots, one by one
• The decision framework
• What robots.txt actually does and does not stop
• The Perplexity stealth-crawling problem
• Pay-per-crawl: getting paid instead of choosing block or allow
• Licensing deals: the other way this pays off
• Cloudflare’s default block and what it means for your site
• Server cost: the case nobody talks about
• How to implement your decision
• Frequently asked questions
• Key takeaways

The one-sentence answer

Blocking every AI crawler protects your content from training use but also removes you from the answer engines an increasing share of your audience uses to find things, so the decision has to be made bot by bot, weighing the crawl-to-refer ratio of that specific crawler against whatever value it sends back.

That ratio is not a guess. Cloudflare Radar tracked it directly: from January through March 2026, Anthropic’s ClaudeBot crawled 23,951 pages for every single referral it sent back to the website it took content from (Cloudflare Radar). By May-June 2026 that ratio had improved to 11,122:1, still worse than almost any other bot on the web. OpenAI’s GPTBot sat at 1,276:1 earlier in the year and fell to 857:1 by late May, a real improvement but still nowhere near Google’s traditional Googlebot, which runs at roughly 5:1.

Why this decision got complicated in 2026

Three years ago, “block or allow” was a simple binary because there was one kind of AI bot: a training scraper. That is no longer true. A single AI company now runs multiple, functionally different crawlers, and treating them as one decision throws away information you need.

This split is also why sites rank fine on Google but stay invisible in ChatGPT and Perplexity: a blocked search-citation crawler produces exactly that symptom even when Googlebot has full access. OpenAI alone operates GPTBot (training), OAI-SearchBot (indexing for ChatGPT search and citations), and ChatGPT-User (fetches a page live when a user asks ChatGPT to look something up). Blocking GPTBot keeps your content out of future model training. Blocking OAI-SearchBot removes you from ChatGPT’s live search citations, the surface that actually sends visits back. These are opposite trade-offs living under one company’s name, and a blanket robots.txt rule against “OpenAI” collapses them into the same decision.

Cloudflare’s own product now reflects this split. Starting September 15, 2026, new domains onboarding to Cloudflare get Training and Agent crawlers blocked by default on pages that carry ads, while Search crawlers stay allowed (Cloudflare Blog). That is Cloudflare admitting the old single “Block AI bots” toggle was too blunt for what publishers actually wanted.

The crawl-to-refer ratio: the number that should drive your decision

If you take one number away from this article, take this one. The crawl-to-refer ratio tells you how many times a bot reads your content for every visitor it sends you. A high ratio means the bot is a pure cost center. A low ratio means the bot is closer to a partner.

Cloudflare’s Q1 2026 and May-June 2026 measurements, cross-referenced with SEOmator’s independent tracking, show a wide spread (Cloudflare Radar; SEOmator):

Crawler	Crawl-to-refer ratio (approx., 2026)	Direction of travel
Googlebot (for comparison)	5:1	Stable, mature
DuckDuckGo	1.5:1	Near parity
PerplexityBot	111:1 (Jan-Mar), 190:1 (late May)	Got worse
GPTBot / OAI-SearchBot combined	1,276:1 (Jan-Mar), 857:1 (late May)	Improved
ClaudeBot	23,951:1 (Jan-Mar), 11,122:1 (May-Jun)	Improved but still extreme

Read the direction column as carefully as the ratio itself. Perplexity’s ratio moved the wrong way over the same window that GPTBot and ClaudeBot improved, which matters if you are deciding whether a bot is trending toward being a fair trade or away from it.

None of this tells you what to do with a single bot in isolation. It tells you the price of allowing it: for every referral ClaudeBot’s training crawler sends you, you serve roughly 11,000 requests with no return traffic attached. If your server costs scale with request volume, that is a number you can put a dollar figure on.

The bots, one by one

Generic advice to “block AI bots” ignores that these crawlers serve different jobs. Here is what each one actually does, sourced from vendor documentation and the reference tables tracked across 2026 (guptadeepak.com AI Crawler Reference; Cloudflare Blog):

GPTBot (OpenAI): crawls for model training data. Blocking it stops your content from entering future GPT training runs. It does not affect whether ChatGPT can browse and cite a page live, that is a separate bot.

OAI-SearchBot (OpenAI): indexes pages specifically to surface them as citations in ChatGPT’s search feature. Blocking this removes you from ChatGPT search results and citations, the highest-value AI referral source: ChatGPT commands 92.4% of trackable LLM referral traffic (Previsible, July 2026 AI Traffic Report).

ChatGPT-User: fires when a live user asks ChatGPT to fetch a specific page. This is closer to a proxy for a real reader than a bulk scraper.

ClaudeBot (Anthropic): training crawler with the worst crawl-to-refer ratio measured in 2026 (11,122:1 as of May-June). Anthropic does not currently operate a comparable citation-driven search crawler at Perplexity’s or OpenAI’s scale, so there is little upside to weigh against the cost.

PerplexityBot: crawls to power Perplexity’s answer engine and its citations. Perplexity generates real referral traffic and converts it at a documented 10.5% rate (Cloudflare Radar cited via digitalapplied.com), but Cloudflare has separately accused the company of ignoring blocks entirely (more on that below).

Google-Extended: controls whether your content trains Gemini and appears in AI Overviews and AI Mode. This is separate from Googlebot, which handles your regular organic ranking; blocking Google-Extended does not touch your position in classic blue-link results (Google Search Central). If your traffic already dropped after AI Overviews rolled out on your queries, see the AI Overviews traffic recovery plan before you change this setting, since blocking it now does not undo a ranking-independent visibility change already in effect.

CCBot (Common Crawl): not an AI company’s own crawler, but the open dataset many AI labs train on downstream. Blocking it is a training-data decision with no direct referral trade-off either way, since Common Crawl sends no traffic to anyone.

Bytespider (ByteDance): training crawler for ByteDance’s AI products. No meaningful citation surface for most non-Chinese-market sites, so the trade-off looks similar to ClaudeBot’s: mostly cost, little upside.

The decision framework

Run each bot through four questions in order. Stop at the first “no” that matters to you.

flowchart TD
    A[New AI crawler you have not decided on] --> B{Does it power a live<br/>citation or answer surface<br/>you want visibility in?}
    B -- No, training only --> C{Do you have or want<br/>a licensing relationship<br/>with this company?}
    B -- Yes --> D{Is its crawl-to-refer<br/>ratio reasonable<br/>for your traffic goals?}
    C -- No --> E[Block]
    C -- Yes, or open to one --> F[Allow, or use pay-per-crawl<br/>to get paid either way]
    D -- No, mostly cost --> G{Is the referral traffic<br/>it does send high quality?}
    D -- Yes --> H[Allow]
    G -- No --> E
    G -- Yes --> H

Applied to the four highest-volume bots as of August 2026:

• GPTBot / OAI-SearchBot: allow OAI-SearchBot (feeds the biggest AI referral channel, 92.4% share), decide GPTBot on your appetite for training-data use since the crawl-to-refer ratio improved to 857:1 but is still far worse than Googlebot’s 5:1.
• ClaudeBot: block, unless Anthropic offers you a direct licensing arrangement. The 11,122:1 ratio makes the pure-crawl case weak.
• PerplexityBot: allow for the citation surface, but pair it with the enforcement caveat in the next section, because Cloudflare’s forensic findings mean your “block” setting may not actually be honored.
• Google-Extended: allow if you want AI Overviews and AI Mode inclusion, and understand it does not change your regular Google ranking either way.

Turning the four highest-volume bots into a plain allow/block read:

Crawler	Sends citation traffic	Crawl-to-refer ratio is reasonable	Reliably honors blocks	Recommended default
GPTBot	✗ (training only)	✗ (857:1 to 1,276:1)	✗ (39.5% of bans unenforced)	Block unless you want training inclusion
OAI-SearchBot	✓ (92.4% of AI referral share)	N/A, citation bot	✓	Allow
ClaudeBot	✗ (training only)	✗ (11,122:1)	Not separately measured	Block
PerplexityBot	✓	✗ (111:1 to 190:1)	✗ (documented stealth crawling)	Allow with network-level monitoring
Google-Extended	✓ (AI Overviews, AI Mode)	N/A, no comparable ratio published	✓	Allow if you want AI Overviews visibility
CCBot	✗ (no citation surface)	✗	Generally honors robots.txt	Block

What robots.txt actually does and does not stop

robots.txt is a request, not a lock. It is a plain-text file that tells well-behaved crawlers which paths they should not fetch, and it works only because the crawler chooses to honor it.

Compliance varies by bot and is measurably imperfect even among crawlers that publish their intent to follow it. TechnologyChecker.io’s July 2026 analysis of 10,894 domains found that 39.5% of GPTBot bans set in robots.txt were not actually honored by OpenAI’s crawler at the network level (TechnologyChecker.io). That figure alone should change how much confidence you put in a robots.txt-only strategy.

The same voluntary-compliance question comes up with llms.txt: it is another file crawlers can choose to read or ignore, not a binding control, and treating either file as a guarantee is the mistake to avoid. Given that, a robots.txt rule against a crawler you actively want to keep out should be treated as step one, not the whole plan. Server-level blocking (user-agent and IP-range rules at the CDN or WAF layer) is the enforcement mechanism; robots.txt is the polite version other crawlers can choose to respect.

The Perplexity stealth-crawling problem

In August 2025, Cloudflare published forensic evidence that Perplexity was crawling websites that had explicitly blocked it, using undeclared user agents and rotating IP addresses and Autonomous System Numbers to avoid detection (Cloudflare, reported by Daring Fireball). Cloudflare’s researchers wrote that “although Perplexity initially crawls from their declared user agent, when they are presented with a network block, they appear to obscure their crawling identity in an attempt to circumvent the website’s preferences” (Cloudflare, via Computing.co.uk).

Perplexity’s public documentation states it “respects robots.txt directives,” a claim Cloudflare’s test directly contradicted by demonstrating content access that should have been impossible under a robots.txt-only block. Cloudflare responded by de-listing Perplexity as a verified bot and adjusting its rules to catch the stealth traffic (Computing.co.uk).

Separately, Wired reported Perplexity’s CEO characterizing robots.txt as not a legal framework, a position that lines up with Cloudflare’s forensic findings rather than contradicting them (reported via The Register forums summary). The practical takeaway for your decision: if you decide to block PerplexityBot specifically for its low referral value or its content-use policy, do it at the network level, not only in robots.txt, and monitor your logs for user agents you did not explicitly allow. If you are weighing Perplexity against ChatGPT and Gemini specifically, the source-selection comparison across all three engines shows how differently each one picks what to cite once it can reach your content.

Pay-per-crawl: getting paid instead of choosing block or allow

Cloudflare launched Pay Per Crawl in July 2025 as a third option beyond block and allow: charge. Site owners set one of three rules per crawler in the Cloudflare dashboard: Allow (free access), Charge (per-request payment), or Block (deny, with an option to negotiate) (Search Engine Land).

Cloudflare’s own data behind the launch showed the scale of the imbalance that made this necessary: OpenAI’s crawler scraped websites 1,700 times for every referral it sent back at the time of launch, and Anthropic’s scraped 73,000 times per referral (eMarketer). Cloudflare estimated high-traffic publishers could earn $50,000 to $200,000 a month from AI crawler charges under the model (Okoone, citing Cloudflare). Major publishers including TIME, Conde Nast, The Associated Press, The Atlantic, and Fortune aligned with the approach at launch (Okoone). As of August 2026, Pay Per Crawl remains in private beta, so it is not yet a general-availability option for most sites, but it changes the framing: for a site with real negotiating power, “charge” can replace “block or allow” as the actual decision.

Licensing deals: the other way this pays off

Separate from per-crawl payment, direct licensing deals have paid real, sourced sums to publishers large enough to negotiate one. OpenAI’s largest publicly reported deal is roughly $250 million over five years with News Corp, covering The Wall Street Journal, Barron’s, MarketWatch, the New York Post, and more than a dozen other titles (LLM Pulse). Reddit disclosed $203 million in aggregate data-licensing contract value in its IPO filing, with combined API licensing revenue from Google and OpenAI running near $130 million a year (LLM Pulse). News Corp separately signed a deal with Meta in March 2026 worth up to $50 million a year over three years (LLM Pulse).

As of August 2026, OpenAI alone had signed roughly two dozen publisher and data deals, and market tracking put the average deal size at $24 million per publisher across the Big 4 AI buyers, OpenAI (53% of commitments), Google (12%), Microsoft (9%), and Meta (6%), totaling $2.92 billion in committed spend (LLM Pulse). This tier is realistic only for publishers with a recognized brand and a large enough archive to matter as training data; the same market tracking notes the licensing market pays the brand-name corpus and largely strands the long tail. If you are not in that tier, pay-per-crawl and the block/allow decision are the tools actually available to you, alongside the unlinked-mention strategy covered in brand mentions as the new backlinks for AI search.

Cloudflare’s default block and what it means for your site

Since July 1, 2025, every new domain onboarding to Cloudflare has GPTBot, ClaudeBot, and PerplexityBot blocked by default, at the network level, before your site’s robots.txt is ever consulted (Cloudflare Blog). If your site sits behind Cloudflare and you have not touched this setting, you may already be blocking these bots without having made an active decision to.

That defaults change again on September 15, 2026: new domains get Training and Agent categories blocked by default only on pages that carry ads, while Search stays allowed, replacing the single blanket toggle with the three-category system described earlier (Cloudflare Blog). If you want AI Overviews, ChatGPT search citations, or Perplexity citations to reach your content, check your Cloudflare dashboard’s AI Crawl Control settings directly. Do not assume the default matches your intent.

The scale of this default matters beyond any one site: GPTBot is the single most-blocked AI crawler across robots.txt files measured in Q1 2026, appearing in 5.52% of all DISALLOW rules network-wide, ahead of CCBot at 5.08% and ClaudeBot at 4.88% (TechnologyChecker.io). Among the top 1,000 websites specifically, 25% now block GPTBot outright, and among 107 prominent sites checked in June 2026, 44.9% block at least one major AI crawler (multiple sources cross-referenced). News publishers block at a much higher rate than the general web: 79% of top news sites block AI training bots via robots.txt (BuzzStream, publisher block study). If your crawler settings look right but citations still are not showing up, run the checks in the AI answer engine optimization mistakes list before assuming a block is the cause.

Server cost: the case nobody talks about

Every crawl request, whether it is training-only or citation-driven, costs you bandwidth and compute the same way a human visitor does. On a large site, this adds up in a way that has nothing to do with content rights.

If ClaudeBot’s training crawler alone generates 11,122 requests for every referral it sends, and your site gets crawled thousands of times a day by that bot, you are paying real infrastructure cost for a return that, for most sites, rounds to zero. This is a distinct argument from the licensing or citation argument, and it applies even to sites that have no opinion at all about AI training ethics: it is a straightforward cost-control decision, closer to blocking a scraper that is hammering your server than to a content-policy stance. It is also the same underlying economics behind the zero-click search survival guide: traffic that never converts to a visit or a referral is a cost, not neutral, regardless of whether it comes from a search snippet or a training crawler.

This is a separate concern from whether AI-written content on your own site gets penalized, which is a policy question rather than a crawler-access one; see does AI content get penalized by Google for what Google’s actual guidance says on that front.

How to implement your decision

Fold this into your regular SEO KPI reporting rather than treating crawler policy as a one-time setup task.

1. Audit your current state first. Check your Cloudflare dashboard (or equivalent CDN/WAF) for existing AI bot rules, since defaults changed twice in the last 18 months and you may be blocking or allowing bots you never explicitly chose.
2. Separate training crawlers from search/citation crawlers per company, using the per-bot breakdown above. Do not write one rule for “OpenAI” or “Anthropic.”
3. Set server-level rules for anything you actually want blocked. robots.txt alone left 39.5% of GPTBot bans unenforced in the 2026 measurement cited above; treat it as a signal, not a wall.
4. Watch your logs for undeclared user agents and unusual ASN patterns, especially if you are relying on a PerplexityBot block, given Cloudflare’s stealth-crawling findings.
5. Reassess quarterly. Crawl-to-refer ratios moved meaningfully in both directions within a single year (GPTBot improved, Perplexity got worse), so a decision made in January can be stale by summer.
6. Track the traffic the allowed bots actually send you. Setting up GA4 to separately track ChatGPT and Perplexity referral traffic is the only way to confirm whether an “allow” decision is paying off, rather than assuming it is based on the industry-wide averages above.
7. Report the outcome against share of answer, not share of voice. A crawler-allow decision succeeds or fails based on whether you show up in the answers, not on raw traffic alone.

Frequently asked questions

Does blocking AI crawlers hurt my Google ranking?

No. Google-Extended, which controls Gemini training and AI Overviews inclusion, is separate from Googlebot, which handles standard organic indexing and ranking. Google has stated this control is not used as a ranking signal outside generative AI Search features, so blocking Google-Extended leaves your classic search ranking untouched (Google Search Central).

What is the difference between GPTBot and OAI-SearchBot?

GPTBot crawls to gather training data for future OpenAI models. OAI-SearchBot indexes pages specifically to surface them as live citations inside ChatGPT’s search feature. Blocking GPTBot only affects training use; blocking OAI-SearchBot removes you from ChatGPT’s citation results, the largest single AI referral channel at 92.4% share (Previsible).

Is robots.txt legally enforceable against AI crawlers?

No. robots.txt is a voluntary standard with no legal backing of its own; a crawler operator can choose to ignore it, and enforcement depends on the crawler’s own policy plus whatever separate legal claims (copyright, terms of service, computer-fraud statutes) a site owner might pursue. Perplexity’s leadership has publicly characterized robots.txt as not a legal framework, a stance that lines up with Cloudflare’s forensic findings of Perplexity crawling blocked sites.

Which AI crawler has the worst crawl-to-refer ratio?

ClaudeBot, Anthropic’s training crawler, measured at 23,951 pages crawled per referral sent in Q1 2026, improving to 11,122:1 by May-June 2026, according to Cloudflare Radar data. That is worse than GPTBot (857:1 to 1,276:1 across the same period) and vastly worse than Googlebot’s roughly 5:1 (Cloudflare Radar).

Can I charge AI companies to crawl my site instead of blocking them?

Yes, through Cloudflare’s Pay Per Crawl, which lets you set a per-request charge instead of a binary allow or block for each crawler. It launched in July 2025 and remained in private beta as of August 2026. Cloudflare estimated high-traffic publishers could earn $50,000 to $200,000 a month under the model, based on the crawl volumes it measured at launch (Okoone).

Do news publishers block AI crawlers more than other sites?

Yes, significantly more. 79% of top news sites block AI training bots via robots.txt, compared with 25% of the broader top 1,000 websites blocking GPTBot specifically (BuzzStream; multiple 2026 sources). This tracks with news publishers’ greater exposure to content-scraping economics and their outsized presence in AI training corpora.

If I block GPTBot, does that stop ChatGPT from ever mentioning my site?

Not necessarily. ChatGPT can still surface information about your site if it was previously trained on content that mentioned you, or if a user pastes your URL directly and ChatGPT-User fetches it live. Blocking GPTBot stops future training-data collection going forward; it does not retroactively remove content already used in past training runs.

What percentage of my internet traffic actually comes from AI search referrals right now?

Small, but not zero and growing. The global average for AI referral traffic sits at roughly 0.15% to 0.25% of total internet traffic as of 2026, with ChatGPT taking 92.4% of that trackable share (tryanalyze.ai; Previsible). For most sites this is a fraction of organic search traffic, but it converts at a notably high rate where it does arrive.

Do AI referral visitors convert better or worse than regular search traffic?

Better, on the data available so far. Claude-referred traffic converts at 16.8%, the highest of the tracked platforms, followed by ChatGPT at 14.2% to 15.9% and Perplexity at 10.5% (Cloudflare Radar data, via digitalapplied.com). Low volume, high intent is the emerging pattern: fewer visitors arrive this way, but the ones who do have already had their question partially answered by the AI system before clicking through.

Should a small blog with no licensing negotiating power bother blocking anything?

Consider it mainly for cost and content-reuse control rather than expecting revenue. Licensing deals in the tens or hundreds of millions go to publishers with brand-name archives large enough to matter as training corpora; a small site is not in that negotiating position. But the server-cost argument, avoiding thousands of crawl requests per referral, applies at any site size, and blocking pure training crawlers with near-zero citation value (ClaudeBot, Bytespider, CCBot) costs you almost nothing in visibility.

Does Cloudflare block AI crawlers even if I never touch the setting?

Yes, if your domain is new enough. Every Cloudflare domain created after July 1, 2025 blocks GPTBot, ClaudeBot, and PerplexityBot by default, at the network level, before your robots.txt file is even read. Check your Cloudflare dashboard’s AI Crawl Control panel directly rather than assuming your intent matches the default (Cloudflare Blog).

What changes on September 15, 2026?

Cloudflare replaces its single blanket “block AI bots” toggle with three separately controllable categories: Search, Agent, and Training. New domains onboarding after that date get Training and Agent blocked by default only on pages that display ads, while Search crawlers stay allowed by default, a more granular approach than the July 2025 blanket block (Cloudflare Blog).

Is it true that some AI crawlers ignore robots.txt bans entirely?

Yes, measurably. A July 2026 analysis of 10,894 domains found that 39.5% of GPTBot bans set in robots.txt were not actually honored by the crawler at the network level (TechnologyChecker.io). This is why server-level or CDN-level blocking is the more reliable enforcement layer for a crawler you genuinely want to keep out.

What exactly did Cloudflare accuse Perplexity of doing?

Cloudflare published forensic evidence in August 2025 that Perplexity crawled sites that had explicitly blocked it, using undeclared user agents and rotating IP addresses and Autonomous System Numbers to evade network blocks and firewall rules, despite Perplexity’s public documentation stating it respects robots.txt. Cloudflare responded by de-listing Perplexity as a verified bot (Cloudflare, via Computing.co.uk).

Does blocking Google-Extended remove me from Google’s regular search results?

No. Google-Extended only controls Gemini training data collection and inclusion in AI Overviews and AI Mode. Google has stated explicitly that this control is separate from, and does not act as a ranking signal for, standard organic search results (Google Search Central).

Is CCBot (Common Crawl) the same as an AI company’s crawler?

No. CCBot is Common Crawl’s own open-web dataset crawler, not owned by any single AI lab. Multiple AI companies train on Common Crawl’s dataset downstream, so blocking CCBot is a training-data decision, but it sends no referral traffic of its own either way since Common Crawl is not a consumer-facing product.

How much has GPTBot’s crawl-to-refer ratio actually improved in 2026?

From roughly 1,276:1 in the January-March 2026 measurement window down to about 857:1 by late May 2026, according to Cloudflare Radar tracking. That is a real improvement but still far worse than Googlebot’s roughly 5:1 ratio, meaning GPTBot still crawls hundreds of pages for every visitor it sends back (Cloudflare Radar).

What’s the realistic revenue from Pay Per Crawl for an average site?

Cloudflare’s public estimate of $50,000 to $200,000 a month applies to high-traffic publishers, not average sites. The program remained in private beta as of August 2026, and eligibility, terms, and typical payouts for mid-size sites have not been publicly disclosed in comparable detail (Okoone, citing Cloudflare).

Can I allow a bot for search citations but block it for training, if it’s the same company?

Yes, and this is the central point of the whole framework here. Companies like OpenAI run separate crawlers for separate jobs (GPTBot for training, OAI-SearchBot for search citations), each with its own user agent string. You set independent rules for each user agent rather than one rule per company.

Does structured data affect whether an AI crawler can cite my page?

Not directly. Structured data helps a crawler parse and understand a page once it has already been allowed to reach it; it does not override a robots.txt or network-level block. If your crawlers are allowed but you are still not seeing citation lift, see is structured data required for AI search for what actually moves that outcome versus what Google says is optional.

Key takeaways

• The crawl-to-refer ratio is the single most useful number for this decision: Googlebot runs near 5:1, GPTBot improved to roughly 857:1 by late May 2026, and ClaudeBot still sits around 11,122:1 even after improving from 23,951:1.
• Treat training crawlers and search/citation crawlers from the same company as separate decisions. Blocking GPTBot does not have to mean blocking OAI-SearchBot.
• robots.txt is a request, not an enforcement mechanism. 39.5% of GPTBot bans in robots.txt went unhonored in a 2026 measurement, and Cloudflare has documented Perplexity actively evading network-level blocks.
• Cloudflare blocks GPTBot, ClaudeBot, and PerplexityBot by default on every new domain since July 1, 2025, and shifts to a three-category Search/Agent/Training model on September 15, 2026. Check your actual dashboard settings rather than assuming.
• Licensing deals (News Corp’s ~$250M with OpenAI, Reddit’s $203M disclosed contract value) are real but reserved for publishers with brand-name archives; Pay Per Crawl and the block/allow decision are the tools available to everyone else.

If your visibility problem is less about which crawlers to allow and more about why AI search doesn’t cite you even when it can reach your content, the next step is a structured AI visibility audit rather than another crawler policy change.