How Do LLMs Crawl the Web?
A live experiment tracking AI bot behavior — which crawlers visit, how often they return, and which pages they prefer.
Research Questions
- Which AI crawlers visit most frequently? Tracking GPTBot, ClaudeBot, PerplexityBot, Google-Extended and others.
- How often do bots re-crawl the same page?
- Does content structure affect crawl frequency?
- What ratio of traffic is AI vs human?