AI crawler traffic statistics: AI bots vs Googlebot on a job board
Career App is a job board. Every request from a crawler that names itself in its user agent is counted as it reaches the site, and this page adds them up: AI crawlers next to Googlebot, by company, by purpose and by kind of page. It is one site’s logs, not the web.
Requests logged on Career App from 2026-10-06 to 2026-11-04 (UTC), 30 days.
- AI crawler requests
- 540,526
- Googlebot requests
- 16,915
- AI vs Googlebot
- 32 times
- Top AI crawler
- OAI-SearchBot
30 days to 2026-11-04
same days
as many requests
184,542 requests (OpenAI)
How many requests do AI crawlers make compared to Googlebot?
AI crawlers made 32 times as many requests as Googlebot: 540,526 against 16,915 in the 30 days to 2026-11-04. OAI-SearchBot alone made 11 times as many as Googlebot.
Which AI crawler crawls the most?
Download CSVOAI-SearchBot, OpenAI's crawler, made the most requests of any AI crawler: 184,542 in the 30 days to 2026-11-04, 34% of all AI crawler requests. Next came Amazonbot (171,947) and PerplexityBot (91,029).
By company, OpenAI's crawlers made 34% of AI crawler requests, Amazon's 32% and Perplexity's 17%.
| Crawler | Company | Type | Requests | Share of bot requests |
|---|---|---|---|---|
| Unrecognized bots | Unknown | Other | 247,159 | 30% |
| OAI-SearchBot | OpenAI | AI | 184,542 | 23% |
| Amazonbot | Amazon | AI | 171,947 | 21% |
| PerplexityBot | Perplexity | AI | 91,029 | 11% |
| ClaudeBot | Anthropic | AI | 57,063 | 7% |
| ExaSearchBot | Exa | AI | 21,531 | 3% |
| Googlebot | Search engine | 16,915 | 2% | |
| meta-webindexer | Meta | AI | 12,561 | 2% |
| YandexBot | Yandex | Search engine | 6,074 | 1% |
| Bingbot | Microsoft | Search engine | 4,014 | under 1% |
| ChatGPT-User | OpenAI | AI | 1,050 | under 1% |
| meta-externalagent | Meta | AI | 479 | under 1% |
| Baiduspider | Baidu | Search engine | 416 | under 1% |
| ExaUser | Exa | AI | 261 | under 1% |
| Applebot | Apple | Search engine | 100 | under 1% |
| DuckDuckBot | DuckDuckGo | Search engine | 75 | under 1% |
| GPTBot | OpenAI | AI | 59 | under 1% |
| Bytespider | ByteDance | AI | 3 | under 1% |
| Claude-User | Anthropic | AI | 1 | under 1% |
What share of crawler traffic is AI?
AI crawlers made 66% of the 815,279 requests we logged from bots in the 30 days to 2026-11-04; search engine crawlers made 3%, and 30% came from scrapers and bots that named no crawler we recognize.
Are AI crawlers collecting training data or answering users?
Each crawler is placed by what its operator says it is for. The operator's word is the only evidence of purpose a log has.
Crawlers their operators say collect training data made 42% of AI crawler requests; AI search indexers made 57%, and fetches made when a user asked an assistant under 1%. The other under 1% came from Bytespider, whose purpose we do not classify.
| Stated purpose | Crawlers | Requests | Share of AI requests |
|---|---|---|---|
| Collects training data | Amazonbot, ClaudeBot, meta-externalagent, GPTBot | 229,548 | 42% |
| Builds an AI search index | OAI-SearchBot, PerplexityBot, ExaSearchBot, meta-webindexer | 309,663 | 57% |
| Fetches a page when a user asks | ChatGPT-User, ExaUser, Claude-User | 1,312 | under 1% |
| Archive, or purpose not stated | Bytespider | 3 | under 1% |
Which pages do AI crawlers read most?
Page families are read from the URL path, the same grouping our Search Console reports use.
Individual job pages drew the most AI crawler requests, 30%, followed by skill pages (20%) and jobs-by-location pages (9%). Googlebot's most-requested family was individual job pages, at 43% of its requests.
| Pages | AI crawlers | Share | Googlebot | Share |
|---|---|---|---|---|
| Individual job pages | 161,132 | 30% | 7,300 | 43% |
| Skill pages | 107,991 | 20% | 3,267 | 19% |
| Jobs-by-location pages | 50,332 | 9% | 1,289 | 8% |
| Employer pages | 45,295 | 8% | 1,834 | 11% |
| Job search pages | 39,637 | 7% | 0 | 0% |
| Occupation pages | 39,248 | 7% | 97 | 1% |
| Other URLs | 29,244 | 5% | 12 | under 1% |
| Hobby pages | 29,009 | 5% | 1,603 | 9% |
| The home page | 11,298 | 2% | 7 | under 1% |
| Articles | 9,128 | 2% | 0 | 0% |
| Occupation-by-state pages | 6,958 | 1% | 0 | 0% |
| Hiring-trends pages | 5,159 | 1% | 0 | 0% |
| Help articles | 4,309 | 1% | 9 | under 1% |
| Robots.txt, sitemaps and llms.txt | 1,617 | under 1% | 1,496 | 9% |
| Member profiles | 169 | under 1% | 1 | under 1% |
Do AI crawlers read llms.txt?
No AI crawler requested /llms.txt or /llms-full.txt in the 30 days to 2026-11-04, against 699 AI crawler requests for robots.txt.
54,085 AI crawler requests were for the markdown version of a page (the same URL plus .md), 10% of all AI crawler requests.
Is AI crawling growing?
AI crawler requests rose 16% week over week: 139,899 in the week starting 2026-10-26, against 121,022 the week before. Across the last 8 complete weeks the weekly count ranged from 47,369 to 156,044.
The logs this page reads begin on 2026-07-23, 105 days of history; 8 complete weeks are compared here.
| Week starting | AI crawlers | Googlebot | All bots |
|---|---|---|---|
| 2026-09-07 | 73,104 | 6,455 | 93,833 |
| 2026-09-14 | 91,657 | 170 | 107,642 |
| 2026-09-21 | 106,503 | 214 | 137,004 |
| 2026-09-28 | 47,369 | 261 | 62,993 |
| 2026-10-05 | 88,697 | 1,104 | 112,162 |
| 2026-10-12 | 156,044 | 155 | 215,633 |
| 2026-10-19 | 121,022 | 3,648 | 194,860 |
| 2026-10-26 | 139,899 | 7,719 | 234,383 |
How this is counted
- A request is counted when its user agent names a crawler we recognize, before the page is served. Images, scripts and styles are not counted; pages, feeds, sitemaps, robots.txt and llms.txt are. Ordinary browser visits are never recorded.
- User agents are self-reported, and anyone can claim to be a crawler. AI crawlers are counted as they name themselves. 1,175 of Googlebot's 16,915 requests (7%) came from Google's own 66.249.x.x crawler addresses; the rest used the Googlebot name from other addresses.
- Google-Extended and Applebot-Extended are robots.txt switches, not crawlers, so they do not appear in logs. Google’s AI features read pages through Googlebot itself, so some Googlebot requests serve AI answers too; a log cannot tell which.
- Our robots.txt welcomes every crawler, AI crawlers included, and the site has tens of thousands of job, skill and employer pages. A smaller site, or one that blocks AI crawlers, will see different numbers. This is one job board, not the web.
- “Unrecognized” is traffic that looks automated but names no crawler we know, recorded since 30 August 2026. Only totals by crawler and by kind of page are published here: no addresses, no URLs, no single visits.
- Whole UTC days, ending yesterday; re-read every few hours. See hiring trends for the job-market numbers.
Cite these numbers
Free to reuse under CC BY 4.0 with a link to this page. Please give the dates: the window moves every day. Figures for 2026-10-06 to 2026-11-04, read 2026-11-05; the per-crawler table is in the CSV.
Career App, "AI crawler traffic statistics", career.thegoodapps.co/trends/ai-crawlers, requests from 2026-10-06 to 2026-11-04, accessed 2026-11-05.