Do AI Crawlers Respect robots.txt? 2 Violations in 1,622 Visits
Published
This article was first published in Korean by Intrix Lab on August 10, 2026.
They did. Over 75 days, 10 AI crawlers made 1,622 visits. Only 2 requests (0.12%) accessed paths blocked by robots.txt (as of 2026-08-10). CCBot accessed an admin path once, and Claude-Web accessed an API path once.
Why check this?
Audits and consulting engagements often advise businesses to "allow AI bots" or "block them." But if bots don't follow that file, the advice has no basis. That would amount to selling a prescription without checking its premise. So we compared our site's list of disallowed paths with actual access logs.
The results support that premise. At least the crawlers we capture in our logs follow the rules.
Can we rely on robots.txt alone?
No. The ones following the rules are crawlers that honestly identify themselves. Traffic impersonating a crawler has no reason to follow those rules.
Impersonation does happen. When we verified our logs, more than half of the traffic claiming to be Baidu crawlers and 5–6% of traffic claiming to be ChatGPT-family crawlers did not actually come from their operators. These figures are also lower bounds. There may be additional impersonation we did not filter out.
We also found a request that claimed to be ChatGPT's user crawler while searching for server passwords. robots.txt cannot stop requests like that.
What should you do?
robots.txt works as a way to communicate your preferences to honest crawlers. State that you allow them or block them, and they follow those instructions.
It is not a security measure, however. Sensitive paths need authentication or access controls, regardless of whether they are listed in robots.txt. Listing a path in robots.txt also tells others that "there is something here."
These records come from one site we operate. We do not guarantee any particular crawler's behavior or results.
Frequently asked questions (FAQ)
Q. If we block AI bots, will we disappear from AI answers?
Content that cannot be crawled cannot be cited, so blocking reduces opportunities for visibility. Still, blocking may be the right choice in some cases. The decision depends on what you want that content to achieve.
Q. Can we check our own site?
Compare the Disallow paths in robots.txt with actual requests in your server logs. Keep in mind that bot names alone cannot distinguish impersonators.