Which AI Crawlers Actually Respect robots.txt in 2026? A Real IP/Proxy Test

You disallow GPTBot, ClaudeBot, and half a dozen other names in your robots.txt file, and the dashboard shows it as handled. Blocked. Done. But a robots.txt file is a request, not a lock, and whether it actually stops anything depends entirely on whether the thing reading it bothers to check the file at all. That turns out to vary a lot more than most site owners assume.

Guessing from server logs does not settle it, because any request can carry any user-agent string. The method we use, and recommend any technical SEO team run on its own site, replaces the guesswork with a controlled test: give each AI its own private, unlinked URL, disallow that URL in robots.txt, log every request that hits it at the edge, then ask the AI directly, in a fresh conversation, to open the link and report back what it finds.

Run that test and the differences are stark. One major AI checks the rule and walks away without touching the page. One ignores it on the very first visit, then appears to learn the rule within minutes. One shows no sign of reading robots.txt at all before fetching. And one arrives disguised as an ordinary Chrome browser, from dozens of different IP addresses scattered across residential ISPs and commercial proxy networks, which makes blocking it by name close to meaningless.

What follows is that methodology, what it found for each major AI platform cross-checked against each company’s own current documentation rather than taken at face value, and what to actually do about it. The answer is not to give up on robots.txt. It is to stop treating it as the whole defense.

Key Takeaways

• Claude’s crawlers follow robots.txt by Anthropic’s own documented policy, and testing confirms it in practice: disallow a page and Claude declines to fetch it.
• OpenAI documents ChatGPT-User, the fetcher behind user-requested links, as one where robots.txt rules may not apply. Testing found it skipping the rule on a brand-new site’s first request, then respecting it on every request after OpenAI’s own search crawler had read the file minutes later.
• Google’s documentation places AI tools that fetch a page on a person’s behalf into a category that generally ignores robots.txt rules. Testing found behavior consistent with that category for a Gemini-presenting fetcher, though Google does not publish a named, Gemini-specific policy to confirm it directly.
• xAI has not published official crawler or robots.txt documentation, unlike OpenAI, Anthropic, Google, and Perplexity. Independent testing, corroborated by a separately published analysis, has repeatedly found a fetcher using an ordinary browser user agent and arriving from dozens of rotating IP addresses rather than any range xAI has confirmed as its own.
• Perplexity draws an explicit, documented line: its indexing crawler is expected to respect robots.txt, while its user-triggered fetcher is documented as generally ignoring it.
• User-agent robots.txt rules only work against a crawler that identifies itself honestly. OpenAI, Anthropic, and Perplexity all publish IP ranges so a request’s real source can be cross-checked; where a company does not publish ranges, as with xAI currently, behavioral monitoring is the realistic fallback.

Why Server Logs Alone Can’t Answer This

The obvious first move is checking server logs for familiar bot names and assuming anything that shows up is telling the truth. That assumption is the whole problem. A user-agent string is just a header value the requester chooses to send, nobody verifies it in transit, and a fetcher with no reason to identify itself honestly has no reason to. Logs full of “Chrome on Windows” traffic could be real visitors, or could be an AI system’s fetcher deliberately avoiding attribution. There is no way to tell the difference from the user-agent field alone.

That is the actual reason a more deliberate test is worth running, not as a one-time curiosity but as a standing part of how a site verifies its own crawler-blocking assumptions are still true.

The Canary-URL Method We Use to Test This

The setup is simple enough to run without any paid tooling, and it is the same approach we recommend to clients who want a real answer instead of a guess:

• Create a separate, unique URL for each AI being tested. The URL should never be linked from anywhere else on the site or shared publicly, so the only way anything reaches it is by being handed that exact address.
• Disallow each canary URL specifically in robots.txt, scoped to that one path, so a hit against it is unambiguous: whatever respects robots.txt should never request it.
• Put logging in front of the URL, a Cloudflare Worker or equivalent edge function works well, capturing every request’s timestamp, IP address, and full user-agent string, including requests to robots.txt itself.
• In a fresh conversation with each AI system, paste only the canary link and ask it to open the page and report back a short code that is printed nowhere except on that page.
• Compare what actually hit the server against what the AI claims to have read, and check whether a robots.txt request preceded the page request, came after it, or never happened at all.

One detail that matters for a clean result: open every canary link yourself in your own browser first to confirm each one resolves correctly, then clear or exclude that traffic before pasting the links into each AI conversation. Otherwise your own visit becomes indistinguishable in the logs from the AI’s.

What Each AI Actually Did, Verified Against Its Own Documentation

Claude: Checks the File, Honors It

Claude checked robots.txt before making any other request to a new site. Once the canary URL was disallowed, Claude declined to retrieve it and said directly that it would not attempt to work around the restriction. That is not just a testing result. It matches Anthropic’s own support documentation, which states that its bots, including ClaudeBot and the user-triggered Claude-User, honor industry-standard robots.txt directives.

ChatGPT: Fetches First, Catches Up Fast

ChatGPT was the one with a genuine timing story. On a canary URL it had never seen before, the first ChatGPT-triggered fetch went straight to the disallowed page without reading robots.txt at all. About seventy-six seconds later, a separate OpenAI crawler, OAI-SearchBot, showed up and read the robots.txt file. From that point forward, every further ChatGPT request to the same disallowed path was refused, with no fetch attempt at all, as though the rule had since been learned. OpenAI’s own bot documentation explains the mechanism behind the first part: ChatGPT-User fetches are described as user-initiated, and OpenAI states that robots.txt rules may not apply to them. The “learns the rule afterward” pattern is not something OpenAI documents explicitly, so treat it as a real, testable, repeatable observation rather than official policy, but it fits cleanly with OpenAI’s own description of OAI-SearchBot as the crawler responsible for managing search opt-outs.

Gemini: No robots.txt Check Observed

A fetcher presenting a bare “Google” user agent, not a named, documented crawler string, retrieved a disallowed canary page without ever requesting robots.txt first, and its source IP addresses did not match Google’s own published crawler ranges. Google’s user-triggered fetchers documentation covers the general category this kind of request falls into, tools where a person’s own action triggers the fetch, and states that because the fetch was requested by a user, these fetchers generally ignore robots.txt rules. Google’s published list under that category names several specific tools but does not name a Gemini-specific browsing fetcher directly, so this finding should be read as consistent with Google’s documented category rather than a named, company-confirmed Gemini policy.

Grok: No Published Policy, and Traffic That Avoids Attribution

This is where testing diverged most sharply from anything a company has actually documented, because xAI has not published a crawler policy to compare against in the first place. Every canary request attributed to Grok arrived as a burst of nine to twelve separate IP addresses hitting the page within seconds of each other, every one presenting as a standard Chrome or Safari browser rather than any bot-identifying string. Across six separate test requests, 59 distinct IP addresses were logged, none reused, spread across hosting providers and small residential ISPs in multiple countries, with at least one range traceable to a known commercial proxy provider. robots.txt was never requested at any point.

That pattern is not an isolated result. A February 2026 crawler analysis ran its own independent test and reported the same core finding using a different method: none of xAI’s own documented crawler names ever appeared in its logs, with requests instead presenting Chrome, Safari, and plain HTTP-library strings, arriving in bursts from IP ranges registered to proxy and hosting providers. A separate third-party crawler reference independently confirms the underlying gap driving all of this: xAI publishes no IP ranges and no reverse-DNS hostnames for its crawler, which means there is currently no reliable way for a site owner to confirm that a request claiming, or not claiming, to be Grok actually is.

Perplexity: A Documented Split Between Crawler and Fetcher

Perplexity’s canary test produced the cleanest match between documentation and behavior of any AI tested. A crawler read robots.txt and the homepage, consistent with how Perplexity’s own bot documentation describes PerplexityBot, its indexing crawler, which it recommends allowing through robots.txt rather than blocking outright. The documentation separately describes Perplexity-User, the fetcher triggered by a direct user question, as one that generally ignores robots.txt rules, since the request came from a person rather than a background crawl. One extra behavior showed up that is not in Perplexity’s published documentation: the canary link was flagged back to the user as a possible phishing risk before being fetched, an added caution step worth knowing about if a legitimate but unfamiliar page ever gets this treatment.

Real Current robots.txt Posture, Verified Against Each Company’s Own Documentation

Treat the “what testing observed” column as exactly that, a documented test result, not a universal guarantee. Crawler behavior changes, and the official-documentation column is the more durable reference of the two.

Why Blocking by User-Agent Alone Doesn’t Work Anymore

A robots.txt rule only stops a crawler that both identifies itself truthfully and chooses to respect the file. Most of the AI systems tested here do exactly that, which is genuinely good news and worth not losing in the noise. But at least one does neither, and no user-agent rule, no matter how comprehensive the blocklist, can stop traffic that never announces what it is.

Where a company publishes IP address ranges, that becomes the more durable control. OpenAI, Anthropic, and Perplexity each publish JSON files listing the real IP ranges their crawlers and fetchers use, specifically so a site owner can check a request’s actual source against the company’s own list rather than trusting a header. Where a company has not published ranges, which is currently xAI’s position, IP-range checking simply is not available as an option, and user-agent rules carry no real weight either, since the traffic in question is not using any of xAI’s documented names to begin with.

What is left in that situation is behavioral detection: watching for request bursts from unrelated networks arriving within seconds of a single prompt, checking whether a visitor requests supporting assets like CSS, JavaScript, and favicon files the way a real browser session normally does, and treating a canary-style test as a recurring check rather than a one-time result, since crawler behavior is not fixed and can change without any public announcement.

What to Actually Do About This on Your Own Site

• Keep robots.txt entries current for the crawlers that do publish and honor a documented name, rather than relying on an old blocklist that may already contain retired bot tokens.
• Where a company publishes IP ranges, such as OpenAI, Anthropic, and Perplexity, cross-check server or CDN logs against those ranges rather than trusting the user-agent header alone.
• Treat a disallowed robots.txt entry and a training opt-out as two separate decisions, since most companies document them as separate tokens with different effects.
• Re-run a canary-style test periodically rather than once. The point of this research is that behavior varies by company and can change; a result from today is not a permanent guarantee.
• For traffic that does not identify itself honestly, rely on behavioral signals at the edge, request bursts, missing asset requests, and IP reputation, rather than expecting a robots.txt rule to do that job.

Where This Fits Into a Broader AI Visibility Strategy

Blocking or allowing AI crawlers is only one side of this. The other is making sure the crawlers you do want reading your site, the ones that genuinely respect robots.txt and are building the index or training data behind AI answers, actually find complete, well-structured content when they arrive. That is the core of the work under South Asia Digital’s AI SEO services and generative engine optimization services, not a separate topic from this one.

If your product pages also need a check for whether AI crawlers can actually see price, stock, and variant data once they arrive, not just whether they’re allowed in, see our companion piece on auditing the JS-rendering gap for AI crawlers. And if llms.txt came up in your own research as a possible answer to any of this, it’s worth reading what llms.txt actually does and doesn’t do before relying on it here, since it has no bearing on whether a crawler respects robots.txt.

Frequently Asked Questions

Does Claude’s crawler respect robots.txt?

Yes. Anthropic’s own documentation states that its bots, including ClaudeBot and Claude-User, honor industry-standard robots.txt directives. Testing using the canary-URL method below confirmed this in practice: Claude checked robots.txt before touching a new site and declined to fetch a page once it was disallowed, matching exactly what Anthropic publishes.

Does ChatGPT respect robots.txt?

It is mixed, and OpenAI’s own documentation reflects that directly. ChatGPT-User, the fetcher that runs when someone asks ChatGPT to open a specific link, is described by OpenAI as user-initiated, and OpenAI states that for this kind of fetch, robots.txt rules may not apply. Testing found exactly that pattern on a brand-new site: the first ChatGPT-triggered request went straight to a disallowed page without reading robots.txt at all. About a minute later, OAI-SearchBot, OpenAI’s separate search crawler, read the robots.txt file, and every ChatGPT request after that point was refused without even attempting the fetch. That improvement-after-first-contact detail is based on testing rather than something OpenAI documents explicitly, but it lines up with how OpenAI describes the two fetchers dividing the work.

Does Gemini check robots.txt before fetching a page?

Google’s crawler documentation places tools that fetch a page at a person’s direct request into a user-triggered fetchers category and states plainly that, because a person requested the fetch, these fetchers generally ignore robots.txt rules. Google does not publish a specific, named user agent for Gemini’s own browsing feature the way it does for some other user-triggered tools, so a Gemini-specific policy cannot be confirmed from Google’s documentation by name. Testing consistent with that general category found a fetcher presenting a bare Google user agent that never requested robots.txt before retrieving a disallowed page, with source IPs outside Google’s own published crawler ranges.

Does Grok respect robots.txt?

There is no confident answer, because xAI is the one major AI company in this comparison that has not published official crawler or robots.txt documentation at all, unlike OpenAI, Anthropic, Google, and Perplexity, which all publish bot names and, in most cases, IP address ranges for verification. What independent testing has repeatedly found, including canary-URL runs and a separate analysis published in February 2026, is a fetcher that never announces any of the bot names third-party crawler trackers associate with xAI. Instead it presents a standard Chrome or Safari user agent and arrives from a large number of IP addresses tied to residential ISPs and commercial proxy networks, rather than any address range xAI has confirmed as its own.

Why can’t I just block AI crawlers using a robots.txt user-agent rule?

Because a user-agent string is only a header the requester chooses to send, and nothing technical stops a fetcher from sending a different one, or from declining to read robots.txt in the first place. A disallow rule only works against a crawler that identifies itself honestly and chooses to honor the file, which several of the crawlers covered here do. It does nothing against a fetcher that presents itself as an ordinary Chrome or Safari browser, which is exactly the pattern testing found for one of them.

What is a canary URL, and how does it test AI crawler behavior?

A canary URL is a unique, unguessable page address that exists for one reason: it is never linked from anywhere, so the only way anything can reach it is by being handed that exact address directly. Giving each AI its own private canary URL, disallowing it in robots.txt, logging every request that hits it at the edge, and then asking that AI in a fresh conversation to open the link and report back a code shown only on that page turns a guess about server logs into a controlled, repeatable test. Whatever shows up at that address had to be that specific AI, not generic bot traffic or a coincidence.

Does Perplexity respect robots.txt?

It depends which part of Perplexity made the request, and Perplexity’s own documentation draws that line clearly. PerplexityBot, the crawler that builds Perplexity’s search index, is positioned as a standard crawler that site owners can allow or disallow. Perplexity-User, the fetcher that fires when someone asks Perplexity a direct question, is documented by Perplexity as generally ignoring robots.txt rules, since a real person triggered the request rather than a background crawl. Testing matched that documented split, with the added observation that an unfamiliar link got flagged as a possible phishing risk before being fetched, a caution step not mentioned in Perplexity’s published bot documentation.

If an AI crawler is blocked by robots.txt, does that stop it from training on my content?

Not necessarily, and that is a separate question from whether one fetch request succeeds. robots.txt only controls whether a crawler retrieves a page going forward; it says nothing about content already collected before the rule existed, and it cannot address whether a blocked page’s content reaches a model indirectly, for example through other sites that quote or summarize it. Blocking a crawler and opting a site out of model training are related but distinct controls, which is why most AI companies document them as separate tokens rather than one setting.

How do I actually verify that a request claiming to be an AI bot is genuine?

A user-agent string alone proves nothing, since any request can carry any string. OpenAI, Anthropic, and Perplexity all publish IP address ranges for their crawlers specifically so a site owner can cross-check a request’s real source IP against the company’s own published list, not just its claimed name. Where a company does not publish IP ranges, which is currently the case for xAI, both user-agent rules and IP-range checks lose most of their value, and the realistic fallback is behavioral: logging request bursts from rotating IPs, checking whether static assets like CSS and favicons get requested the way a real browser session would, and treating a single prompt followed seconds later by a wave of requests from unrelated networks as a pattern worth investigating.

Should I still disallow AI crawlers in robots.txt if some of them might ignore it?

Yes. robots.txt remains the correct, standard way to state a crawling preference, and several major AI crawlers, by their own companies’ documentation and by direct testing, genuinely honor it. A rule that one fetcher ignores is not a rule that stops working for the ones that do not. robots.txt is a declared preference, backed in some cases by real published verification infrastructure and in others by little more than goodwill, not a technical wall, and a site’s defenses should assume both are true at the same time rather than picking one.