AI & Bot Crawls Reports

gethttps://d1iwal3igywepk.cloudfront.net/bot-traffic

Which crawlers and automated browsers read your site: ChatGPT, Claude, Perplexity, Google and the rest. Use it to see whether AI assistants and search engines are reading your pages, and which pages they read most.

The question it answers: "Which bots read my site, how often, and which pages?"

What you get

Field	What it means
`crawlers[].server_requests`	Requests your server snippet reported for this crawler. Zero until the optional server snippet is installed
`crawlers[].verified_server_requests`	The part of `server_requests` that came from the vendor's published IP ranges
`crawlers[].tracker_pageviews`	Page views from crawlers that ran the tracker JavaScript
`crawlers[].last_seen`	The crawler's latest hit in the window
`pages[]`	The most-crawled paths, with the same two measures
`trend[]`	Daily `server_requests` and `tracker_pageviews` per category
`automated[]`	Bot page views excluded from web analytics, per `agent_reason` (`webdriver`, `datacenter`), with unique visitors
`agents.trend[]`	Daily page views and unique visitors from AI assistants browsing for a person (`ai_user`), excluded from web analytics
`agents.pages[]`	The top 50 pages those AI assistants viewed, with page views and unique visitors
`capture`	The last server crawler hit over the last 30 days (`null` when none). Ignores the date range and `page_url_scope`

server_requests and tracker_pageviews measure different requests: one crawler that runs JavaScript can produce one of each for the same page. Never add them.

Server requests need the optional server snippet (Setup → Web tracker → Capture AI crawlers). Without it, only crawlers that run JavaScript show up, in tracker_pageviews. Tracked page views split into human, agent and bot. A datacenter visitor who views 2+ distinct pages or is identified as a person (signed up or submitted a form) counts as human (a person on a VPN), so automated only lists the rest.

The pages block also returns pages_has_more and pages_next_offset. Status, category and verification filters run before ranking and pagination. Keep the same filters and limit when following offsets. The response can contain more rows than the limit because each path retains all its categories.

Parameters

Parameter	What it does	When you'd use it
`from`, `to` / `range`	The window. Defaults to the last 30 days	Compare a quarter
`fields`	Blocks to return: `crawlers`, `pages`, `trend`, `automated`, `agents`, `capture`	`fields=crawlers` for the crawler list only
`category`	Keep one crawler category: `ai_answer`, `ai_search`, `ai_training`, `search` or `other`	`category=ai_training` for model-training crawlers
`verified_only`	`true` keeps only hits from the vendor's published IP ranges	Drop bots that fake a crawler's user agent
`status_code`	Exact HTTP response code, 100–599, applied before ranking pages; omission includes all statuses, including unknown	`status_code=200` for successful responses; affects crawlers, pages and trend only
`limit`	Distinct paths per page, 1–200, default 50; all category rows for each path are kept	`limit=200` for a full page
`offset`	Number of ranked paths to skip, default 0	Pass `pages_next_offset` until `pages_has_more` is false
`page_url_scope`	`marketing` (default), `product` or `all`	Include product-site pages

Authorization

AuthorizationBearer <token>

API key as Bearer token

In: header

Query Parameters

offset?|

Number of ranked paths to skip. Default 0; use pages_next_offset for the next page.

Range0 <= value <= 4294967295

status_code?integer

Filter crawlers, pages and trend by HTTP response code before ranking pages. Omit for all responses, including unknown codes. Automated browsers and capture are unaffected.

Range100 <= value <= 599

page_url_scope?string

Which host to scope to. Defaults to marketing. all removes the host predicate entirely (every host in the workspace).

Value in

"marketing"
"product"
"all"

range?string

Preset date range. Canonical names (last_30_days, last_7_days, last_3_months, last_6_months, last_year, all_time, …) and dashboard short keys (30d, 7d, 3m, 6m, 1y, all) are both accepted. Mutually exclusive with from/to.

Value in

"last_7_days"
"last_14_days"
"last_28_days"
"last_30_days"
"last_90_days"
"current_month"
"this_month"
"last_3_months"
"last_6_months"
"last_year"
"all_time"
"7d"
"14d"
"28d"
"30d"
"90d"
"3m"
"6m"
"1y"
"all"

from?string

Custom start date (YYYY-MM-DD). Ignored if range is set.

Match^\d{4}-\d{2}-\d{2}$

to?string

Custom end date (YYYY-MM-DD). Ignored if range is set.

Match^\d{4}-\d{2}-\d{2}$

fields?string

Comma-separated response blocks to return. OMIT for every block of the resolved form (the default — this param only ever narrows a response, never widens one). The legal set is resolved AFTER form selection, so it depends on whether an addressing key was supplied: some blocks exist only on the single form, and a report whose resolved form declares none rejects fields with 400. An unknown or duplicated name is a 400 listing the legal blocks for that form.

category?string

Keep one crawler category: ai_answer, ai_search, ai_training, search or other.

Value in

"ai_answer"
"ai_search"
"ai_training"
"search"
"other"

verified_only?string

true = keep only hits from the vendor's published IP ranges.

Value in

"true"
"false"

limit?integer

pages block: rows to return (default 50, max 200).

Range1 <= value <= 200

Responses

Crawler and automated-browser traffic for the range

pages_has_more?boolean

pages_next_offset?number|null

host_scope*string|null

range*

crawlers?array<>

pages?array<>

trend?array<>

automated?array<>

agents?

capture?

is_cached?boolean

True when this response came from the gateway's cache (a fresh hit or a stale serve) rather than being computed for this request. A stale serve returns data that was correct at last_refresh and triggers a background refresh.

age_s?number

Seconds since the served figures were computed. 0 when they were computed for this request.

last_refresh?string

ISO-8601 UTC timestamp of the computation that produced these figures — the value to render as an 'as of' marker.

get/bot-traffic

curl -X GET "https://d1iwal3igywepk.cloudfront.net/bot-traffic?range=last_30_days&fields=crawlers%2Cpages"

Example Responses

{  "pages_has_more": true,  "pages_next_offset": 0,  "host_scope": "string",  "range": {    "from": "string",    "to": "string"  },  "crawlers": [    {      "provider": "string",      "crawler": "string",      "category": "string",      "server_requests": 0,      "verified_server_requests": 0,      "tracker_pageviews": 0,      "last_seen": "string"    }  ],  "pages": [    {      "path": "string",      "category": "string",      "server_requests": 0,      "tracker_pageviews": 0    }  ],  "trend": [    {      "date": "string",      "category": "string",      "server_requests": 0,      "tracker_pageviews": 0    }  ],  "automated": [    {      "agent_reason": "string",      "page_views": 0,      "unique_visitors": 0    }  ],  "agents": {    "trend": [      {        "date": "string",        "page_views": 0,        "unique_visitors": 0      }    ],    "pages": [      {        "page": "string",        "page_views": 0,        "unique_visitors": 0      }    ]  },  "capture": {    "server_last_hit_at": "string",    "server_last_crawler": "string"  },  "is_cached": true,  "age_s": 0,  "last_refresh": "string"}

application/json