/broken-links/crawl and the Link Extractor’s /link-extract/crawl. A crawl can take anywhere from a few seconds to several minutes, so starting one doesn’t block on the result — it hands you a job ID immediately, and you poll a status endpoint until it’s done.
The pattern
1
Start the crawl
POST the entry URL. You get back 202 Accepted and a jobId right away — this is the only step that costs credits.2
Poll for status
GET the status endpoint every 1–2 seconds. It always returns 200, whether the job is still running or finished — check the status field, not the HTTP status code.3
Stop when the job reaches a terminal phase
status: "done" means result is populated. status: "error" means error holds a message and there is no result. Any other status means keep polling.Polling and downloading are free — no credit cost, on any caller type. They do still count against your plan’s requests-per-minute limit, so poll every 1–2 seconds rather than in a tight loop.
Response shape
Echoes the ID from the URL.
Current phase. See the tables below — the two crawlers use slightly different phases.
Live counters — pages crawled, links found so far, whether a sitemap was located. Shape depends on the tool; see each endpoint’s reference page.
Present only once
status is "done". The full result, same shape as the synchronous single-page endpoint.Present only once
status is "error".Phases
Broken Link Checker (/broken-links/crawl):
Link Extractor (
/link-extract/crawl):
The Link Extractor has no
checking phase — it reports link metadata only and never makes a second request to check status codes.
Crawl limits
Both crawlers share the same tuning:
If a site is larger than the cap, the result marks the relevant
truncated* field true rather than failing — you get everything found up to the limit.
Job expiry
Job state lives in memory and is discarded 30 minutes after it last changed. Poll continuously while a crawl is running rather than storing ajobId to check back on later — a stale ID returns 404:
Broken Link Checker
Full parameter and response reference.
Link Extractor
Full parameter and response reference.