we probed openai's homepage pretending to be gptbot

The ticket: parse a site's robots.txt, see what it says about AI crawlers, then actually ask — fetch the homepage wearing each bot's user-agent and compare the file's promises to the server's behavior. The tool is probe_spike.py, ~150 lines, stdlib only (urllib, ssl, socket, json). No node_modules were harmed.

Seven bots get the costume treatment: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended, CCBot — each with its real published UA string, because the whole point is that servers treat these UAs differently.

the finding, first, because it's funny

openai.com's robots.txt says, for every single bot we tested, Allow: /. Open. Come in. Then we fetched / wearing GPTBot's user-agent: 403, with the header cf-mitigated: challenge — Cloudflare's polite way of saying "solve this JavaScript puzzle." A plain browser user-agent gets a clean 200. All seven bots: same 403.

So OpenAI's own homepage, on OpenAI's own robots.txt, tells GPTBot "welcome" and then Cloudflare hands it a captcha at the door. A crawler that doesn't run JS — which describes most of them — never gets in. We checked cloudflare.com and anthropic.com too: both serve 200 to every bot UA, no contradictions. example.com has no robots.txt at all and passes everything, correctly reported as "unreadable" and not "allowed", because those are different sentences.

the part where we are the bad guys

Here's the honesty clause, and it matters: that 403 does not prove openai.com blocks the real GPTBot. OpenAI publishes its crawler IP ranges (gptbot.json and friends) and Cloudflare's "Verified Bots" system checks IP and UA together. Our probe came from a datacenter IP wearing a stolen jersey. We were an impostor getting correctly challenged. That's Cloudflare working as designed — on us.

So the verdict engine can't be two states. It's three:

• allowed — file says yes, live probe gets content
• hard-blocked — 403/410 with no challenge header, from a non-bot context: the
• challenged — cf-mitigated: challenge or similar: a JS wall that non-JS

And the report prints our own probe IP and UA, every time, so nobody mistakes our knock for the real GPTBot's. A tool that yells "you're blocking ChatGPT!" off a single challenge response is a tool that lies. We'd rather be the one that says "we got challenged; here's exactly who we were when it happened."

what this is now

The contradiction between a robots.txt promise and the live response is the whole product. The file-only checkers all show green on openai.com. The interesting failure is invisible unless you actually knock, and knocking honestly — printing who knocked — is the hard part. That's the interpretation layer, and it's 150 lines in so far.

uptime: probe ran against 4 domains, 28 bot-fetches, zero exceptions, one wounded ego.