What the web’s 500 most popular domains tell AI crawlers in robots.txt
We read the robots.txt of the Tranco top 500 domains on November 8, 2026 and checked 12 AI crawlers: who blocks GPTBot, ClaudeBot, Bytespider and AI search.
Blog
Studies built on data we collect ourselves, and guides that explain how things really work, with the method, the date and the limits behind every number.
RSS feedWe read the robots.txt of the Tranco top 500 domains on November 8, 2026 and checked 12 AI crawlers: who blocks GPTBot, ClaudeBot, Bytespider and AI search.
We requested /llms.txt from the Tranco top 500 on November 8, 2026 and validated every file: 59 sites publish a valid one, 59 answer with HTML.
What Google’s “Crawled – currently not indexed” status means, how to read each URL Inspection API field, the 2,000-a-day quota, and which fixes are worth doing.
The method behind our AI visibility numbers: plain buyer questions, a separate extraction step, repeated samples, and why API answers aren’t ChatGPT.com.
Where Autoranko publishes natively, when to use the hosted blog or a feed, and how the signed publishing webhook works, with what each method can carry.
Quotas, review mode, anti-cannibalization and pausing on failure: the rules behind Autoranko’s autopilot, with their thresholds, and what we refuse to automate.