`crawl` walks a site and returns every fetched page as Markdown. It is async:
submit → poll → fetch pages.

## Lifecycle

```
POST /v1/crawl            → 202 { data: { job_id, status_url } }
GET  /v1/crawl/{jobId}    → status / progress / credit_cost
GET  /v1/crawl/{jobId}?include=pages&cursor=…&limit=…
DELETE /v1/crawl/{jobId}  → cancel; pay only for pages already fetched
GET  /v1/crawl            → list jobs (keyset pagination, ?status= filter)
```

Statuses: `queued → running → completed | partial | failed | cancelled`.
`partial` means the crawl stopped early because credits ran out — results are
incomplete; see [Credits](https://leancrawl.com/docs/credits).

## Submit options

| Field | Meaning |
|---|---|
| `url` | entry URL (required) |
| `limit` | max pages — also the up-front credit reservation (required, 1–10000) |
| `include_paths` / `exclude_paths` | regex filters on discovered paths |
| `webhook` | `{url, events, secret?}` — one-off event push for THIS job (`crawl.*` events) |
| `Idempotency-Key` (header) | safe retries (a key hit ignores all other params) |

Request-level `webhook`: a one-off push for this job only — prefer a managed
channel (`POST /v1/notifications/channels` subscribed to `crawl.*`) for anything persistent,
since managed endpoints expose deliveries + resend and a rotatable secret while
per-request deliveries have no list/resend surface. Provide `secret` yourself:
an omitted secret is generated but never returned anywhere, making signature
verification impossible. `crawl.page` fires once per fetched page with the full
page markdown inlined — most receivers want `crawl.completed`/`crawl.failed`.

## Polling discipline

Poll `GET /v1/crawl/{jobId}` every ~10 s — the endpoint is free and cheap, but
faster polling buys nothing. On a terminal status, call with `?include=pages`
and follow `next` for additional batches.

## Retention

Job results (page contents) are retained for a limited window — fetch and store
what you need rather than relying on the API as an archive.