crawl walks a site and returns every fetched page as Markdown. It is async:
submit → poll → fetch pages.
Lifecycle
POST /v1/crawl → 202 { data: { job_id, status_url } }
GET /v1/crawl/{jobId} → status / progress / credit_cost
GET /v1/crawl/{jobId}?include=pages&cursor=…&limit=…
DELETE /v1/crawl/{jobId} → cancel; pay only for pages already fetched
GET /v1/crawl → list jobs (keyset pagination, ?status= filter)Statuses: queued → running → completed | partial | failed | cancelled.
partial means the crawl stopped early because credits ran out — results are
incomplete; see Credits.
Submit options
| Field | Meaning |
|---|---|
url | entry URL (required) |
limit | max pages — also the up-front credit reservation (required, 1–10000) |
include_paths / exclude_paths | regex filters on discovered paths |
webhook | {url, events, secret?} — one-off event push for THIS job (crawl.* events) |
Idempotency-Key (header) | safe retries (a key hit ignores all other params) |
Request-level webhook: a one-off push for this job only — prefer a managed
channel (POST /v1/notifications/channels subscribed to crawl.*) for anything persistent,
since managed endpoints expose deliveries + resend and a rotatable secret while
per-request deliveries have no list/resend surface. Provide secret yourself:
an omitted secret is generated but never returned anywhere, making signature
verification impossible. crawl.page fires once per fetched page with the full
page markdown inlined — most receivers want crawl.completed/crawl.failed.
Polling discipline
Poll GET /v1/crawl/{jobId} every ~10 s — the endpoint is free and cheap, but
faster polling buys nothing. On a terminal status, call with ?include=pages
and follow next for additional batches.
Retention
Job results (page contents) are retained for a limited window — fetch and store what you need rather than relying on the API as an archive.