The bot filter on the bot list: Anthropic's crawler IP feed returns 403 to Python's default User-Agent

Who reported it, and when we saw it

Key facts

• claude.com/crawling/bots.json returned 403 to User-Agent "Python-urllib/3.14" and 200 to Mozilla/5.0, to no User-Agent header, and to curl's default, all checked 2026-08-09; the 200 body carried 20 IPv4 prefixes and creationTime 2026-05-01T20:46:04Z.
• The seven other assistant-agent feeds returned 200 to the same urllib User-Agent on 2026-08-09: openai.com/chatgpt-user.json, gptbot.json, searchbot.json, adsbot.json, search.developer.apple.com/applebot.json, and the two Perplexity files (which 302 from www.perplexity.com to www.perplexity.ai).
• Our fetch loop catches the 403 with a bare "except Exception: continue", so Anthropic drops out of the verified-ranges dict with no log line; the same file's comment, dated 09.08.2026, records the feed as not responding with an HTTP error.
• The tool verifies six agents across three vendors (ChatGPT-User, GPTBot, OAI-SearchBot, PerplexityBot, Perplexity-User, Applebot); Anthropic is the one published feed it does not use, and the 403 is why.
• Of the eight assistant-agent feeds fetched on 2026-08-09, none carried an IPv6 prefix: 0 of 258, 0 of 21, 0 of 35, 0 of 2, 0 of 20, 0 of 12, 0 of 8, 0 of 4.
• Google publishes five IP-range files, not three: common-crawlers.json 315 prefixes (146 IPv6), special-crawlers.json 270 (135), user-triggered-fetchers.json 1056 (528), user-triggered-fetchers-google.json 494 (247), user-triggered-agents.json 20 (8 IPv6), all creationTime 2026-08-07T14:45, checked 2026-08-09.
• The canonical host the Google docs name, www.gstatic.com/crawling/ipranges/, returned 404 on 2026-08-09 for all three URL variants we tried, while developers.google.com/static/crawling/ipranges/ served all five files.

Two commands, two different answers from the same URL

Run these on any machine. Ask claude.com/crawling/bots.json for Anthropic's published crawler IP ranges with the User-Agent that Python's urllib sends unless you override it, and you get HTTP 403. Ask for the same URL with a browser User-Agent, with no User-Agent header at all, or with curl's default, and you get HTTP 200 and a JSON body containing 20 IPv4 prefixes and a creationTime of 2026-05-01T20:46:04Z. Checked 2026-08-09. End to end in Python the failure is unambiguous: urllib.request.urlopen on the bare URL raises HTTPError 403, and the identical script with a User-Agent header of Mozilla/5.0 returns the 20 prefixes. Nothing about the network, the TLS handshake or the JSON changes between the two calls. The only variable is a string the client chooses. We checked the other seven assistant-agent feeds on the same day with the same urllib User-Agent, and all seven returned 200: OpenAI's four files for GPTBot, OAI-SearchBot, ChatGPT-User and OAI-AdsBot, Apple's Applebot file, and Perplexity's two files, which redirect 302 from www.perplexity.com to www.perplexity.ai and then serve. So this is not a general property of vendor feeds. It is one feed, and it is the one an IP verifier most needs in order to say anything about Claude traffic at all.

Why the 403 produced a number instead of an error

A verifier loads each vendor's prefix file into a dictionary keyed by agent, then asks whether a given source address falls inside one of that agent's networks. Our loader wraps each fetch in try/except and, on any exception, continues to the next feed. The relevant two lines are literally "except Exception: continue". Anthropic therefore never enters the dictionary. Downstream, the lookup for ClaudeBot or Claude-User finds no key and returns the third state, not-verifiable, which is the correct answer to the wrong question: the agent was not unverifiable in principle, our fetch was refused. The visible output is a verified column that is smaller than it should be and an unverified column that is larger, with no line anywhere saying a fetch failed. That is the shape of the defect worth naming. A swallowed exception in a measurement pipeline does not produce an error, it produces a plausible number, and a plausible number gets published. We found this the same way anyone would: a comment in the file said Anthropic's feed does not respond, dated 09.08.2026, and the obvious question was whether that was still true. It was true, and it was true for a reason nobody had written down. The honest version of the comment is not "the vendor does not publish a feed" but "the vendor publishes a feed that refuses our client".

What the feeds actually contain, with dates

Verification asks two independent questions. The first is what the client calls itself, which it controls. The second is where the connection came from, which it does not. Only the second is evidence, and it requires a current list. As of 2026-08-09 the machine-readable lists are these. OpenAI documents four agents on its bots page with one file each: chatgpt-user.json carried 258 IPv4 prefixes with creationTime 2026-08-07T03:03:31, gptbot.json 21 prefixes from 2025-10-30, searchbot.json 35 from 2026-01-02, adsbot.json 2 from 2026-05-12. Anthropic's single file covering ClaudeBot, Claude-User and Claude-SearchBot carried 20 prefixes from 2026-05-01, once we got past the 403. Apple's applebot.json carried 12 prefixes with a creationTime of 2023-10-27, and Apple also documents reverse DNS in applebot.apple.com. Perplexity's two files carried 8 and 4 prefixes, from 2025-02-07 and 2025-10-17, and its docs recommend combining user-agent matching with IP verification. Google publishes five files, not three: common-crawlers.json with 315 prefixes of which 146 are IPv6, special-crawlers.json with 270 of which 135, user-triggered-fetchers.json with 1056 of which 528, user-triggered-fetchers-google.json with 494 of which 247, and user-triggered-agents.json with 20 of which 8 are IPv6. All five carried creationTime 2026-08-07T14:45. The last of those covers agent-initiated fetches and is the one most relevant to anyone counting AI assistants. Worth noting for the same reason as the 403: Google's documentation gives the canonical location as www.gstatic.com/crawling/ipranges/, and that host returned 404 on 2026-08-09 across the three URL variants we tried, while developers.google.com/static/crawling/ipranges/ served all five files.

Five ways this check quietly fails

First, the fetcher itself, as above. Before trusting a verified count, log the HTTP status of every feed fetch and fail loudly when one does not return 200. If your client library sets a default User-Agent, that string is part of your measurement apparatus. Second, the CDN. Both of our sites sit behind Cloudflare, which anyone can confirm with a single request: curl -sI https://dabyte.ai/ returned Server: cloudflare and a CF-RAY header on 2026-08-09, as did dablock.ai. If the origin logs the connecting address without restoring the visitor IP from CF-Connecting-IP, every logged address belongs to the CDN, nothing matches any vendor prefix, and the verifier reports that all of your AI traffic is forged. The failure looks exactly like a finding. Third, IPv6. None of the eight assistant-agent feeds listed a single IPv6 prefix on 2026-08-09: 0 of 258, 0 of 21, 0 of 35, 0 of 2, 0 of 20, 0 of 12, 0 of 8, 0 of 4. Google is the exception among the vendors we checked. Python's ipaddress reports an IPv6 address as not contained in an IPv4 network, so a naive implementation labels every IPv6 request a forgery. It is not a forgery, it is unverifiable, and those are different columns. Fourth, feed age. Applebot's creationTime reads 2023-10-27 and GPTBot's 2025-10-30, both checked 2026-08-09. A list not regenerated recently produces false negatives when the vendor adds capacity. That is the argument for keeping the documented reverse DNS path as a second route where a vendor offers one. It is also an interpretation of a self-reported field: the list may be current and simply not restamped. Fifth, your own heuristics. Our discard rule drops a request whose path matches a pattern an assistant supposedly never requests. That pattern includes ^/\. and /\.(env|git|aws|ssh|docker), secrets?\.json, config\.json, credentials, id_rsa, \.pem$, wp-, phpmyadmin, /actuator, /checkout, /pricing$, \.sql$, \.bak$, /vendor/ and /cgi-bin/. Two of those branches are wrong for our purpose. /pricing and /checkout are pages an assistant answering a buying question has an obvious reason to fetch, and on two sites about SaaS and crypto products that is not a corner case. The ^/\. branch catches /.well-known/ai-access.json, a file a well-behaved agent is supposed to request. The rule cannot separate a scanner probing /checkout from an assistant reading it, so any count derived from it is an upper bound on forgery, not a count of it.

What we can show from our own logs, and what we cannot

Our published product is the AI Visibility Index, and it is measured by submitting a fixed panel of prompts to three engines and recording which brands are named in the answers. It does not read server logs. No crawler figure has ever appeared on either site, and nothing in this article is a correction to anything we published. The error was internal and it is worth describing because the mechanism is general. An early log report of ours counted requests carrying assistant user agents and printed them under a heading calling them live sessions. Two problems at once: the counter increments per log line, so calling the total sessions overstates it, and the total included traffic that failed IP verification. An internal note written at the time records the raw figure as 12 on one site and 22 on the other, and records that IP verification against the OpenAI feeds reduced both to zero. We are not asking anyone to take those three numbers on faith, and we cannot back them. The raw nginx log is rotated away, and the daily snapshot store that would preserve a derived series was only added on 2026-08-09, hours before this was written. So there is no retained artifact, and a reader cannot re-derive them. That is precisely why the reproduce command for this article runs against your log and the live feed rather than asking you to believe ours. One more detail from that note is worth stating against our own interest. It cited /secrets.json, /.env.production, /.docker/config.json and /checkout as the paths that proved the traffic was scanners. Three of those are hard to explain innocently. The fourth, /checkout, is also on our own over-broad discard list, so citing it as evidence of scanning is partly circular: the rule that flagged it is the rule we now know over-rejects.

No published prefix file is not the same as unverifiable

It is tempting to write a rule that says: if the vendor publishes no JSON list, label the traffic unverified. That rule is too coarse, and we had a version of it. Yandex genuinely publishes no IP list, and says so directly, on the grounds that the addresses change frequently. But the same documentation names the autonomous systems its bots use, AS13238, AS208722 and AS212066, and documents reverse DNS verification against hostnames ending yandex.ru, yandex.net or yandex.com with a forward confirmation, and describes that method as preferable to managing access by IP precisely because it survives internal network changes. Apple and Google document reverse DNS alongside their files. So the absence of a prefix file tells you which method to use, not that no method exists. The unverified bucket belongs to agents with no documented verification method at all, not to agents whose documented method we have not implemented yet. Whichever method applies, the output should be three numbers and not two: verified, failed, and unverifiable. Collapsing the third into either of the first two is what turns an outage, a 403, an IPv6 address or a missing implementation into a claim about somebody else's traffic.

A user agent is not evidence, and the dispute over that proves it

The reason a scanner wears an assistant's name is that the name functions as an access token. Operators who want their pages quoted add allow rules keyed on assistant user agents at the edge, and anything that types the same string inherits the exemption. That holds as long as edge allowlists key on the user agent. The best-known public argument about this is unresolved and should be presented that way. Cloudflare published a post on 2025-08-04 alleging that Perplexity used undeclared crawlers with generic browser user agents and rotating source ASNs to reach content that blocked its declared bots. Perplexity replied on 2025-08-05 calling the analysis inadequate and arguing that Cloudflare had attributed to it traffic belonging to BrowserBase, a third-party cloud browser it says it uses occasionally, putting its own volume under 45,000 daily requests against the 3 to 6 million Cloudflare described. Cloudflare sells bot management, so it is an interested party; we have not independently verified either account. The narrow point survives the dispute intact and is stronger for it: the two sides disagree about who sent the traffic, which is only a question at all because the User-Agent header could not settle it. Cloudflare's verified-bots documentation now requires deterministic self-identification through a signature, a published IP list with a stable user agent, or reverse DNS. The signature route is being standardised: the IETF has a chartered working group, Web Bot Auth, in the Web and Internet Transport area, with charter-ietf-webbotauth-01 approved. The protocol document is draft-meunier-webbotauth-httpsig-protocol-01, active, last revised 2026-08-06, which replaces the earlier draft-meunier-web-bot-auth-architecture and builds on RFC 9421 HTTP Message Signatures. As of 2026-08-09 these are individual submissions, not yet working group documents, so the prefix lists remain the documented method for every agent named in this article.

Questions this answers

Machine access

• /api/articles.json — every piece we published, with the measurement each one rests on
• /api/aiv.json — the AI Visibility Index the numbers above come from
• How the index is measured