> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pocketutils.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Async Jobs & Polling

> How site-wide crawl endpoints work: start a job, poll its status, read the result.

Two tools crawl an entire site rather than a single page: the [Broken Link Checker](/api-reference/broken-links)'s `/broken-links/crawl` and the [Link Extractor](/api-reference/link-extract)'s `/link-extract/crawl`. A crawl can take anywhere from a few seconds to several minutes, so starting one doesn't block on the result — it hands you a job ID immediately, and you poll a status endpoint until it's done.

## The pattern

<Steps>
  <Step title="Start the crawl">
    `POST` the entry URL. You get back `202 Accepted` and a `jobId` right away — this is the only step that costs credits.

    ```json theme={null}
    { "success": true, "data": { "jobId": "c3d4e5f6-1234-4a5b-8c9d-0e1f2a3b4c5d" } }
    ```
  </Step>

  <Step title="Poll for status">
    `GET` the status endpoint every 1–2 seconds. It always returns `200`, whether the job is still running or finished — check the `status` field, not the HTTP status code.

    ```bash theme={null}
    curl https://api.pocketutils.com/v1/broken-links/crawl/c3d4e5f6-1234-4a5b-8c9d-0e1f2a3b4c5d \
      -H "Authorization: Bearer YOUR_API_KEY"
    ```
  </Step>

  <Step title="Stop when the job reaches a terminal phase">
    `status: "done"` means `result` is populated. `status: "error"` means `error` holds a message and there is no `result`. Any other `status` means keep polling.
  </Step>
</Steps>

<Note>
  Polling and downloading are free — no credit cost, on any caller type. They **do** still count against your plan's requests-per-minute limit, so poll every 1–2 seconds rather than in a tight loop.
</Note>

## Response shape

<ResponseField name="jobId" type="string">
  Echoes the ID from the URL.
</ResponseField>

<ResponseField name="status" type="string">
  Current phase. See the tables below — the two crawlers use slightly different phases.
</ResponseField>

<ResponseField name="progress" type="object">
  Live counters — pages crawled, links found so far, whether a sitemap was located. Shape depends on the tool; see each endpoint's reference page.
</ResponseField>

<ResponseField name="result" type="object">
  Present only once `status` is `"done"`. The full result, same shape as the synchronous single-page endpoint.
</ResponseField>

<ResponseField name="error" type="string">
  Present only once `status` is `"error"`.
</ResponseField>

## Phases

**Broken Link Checker** (`/broken-links/crawl`):

| Phase      | Meaning                                           |
| ---------- | ------------------------------------------------- |
| `queued`   | Job created, not yet started                      |
| `sitemap`  | Looking for and parsing `sitemap.xml`             |
| `crawling` | Following internal links breadth-first            |
| `checking` | Checking the HTTP status of every discovered link |
| `done`     | Finished — `result` is populated                  |
| `error`    | Failed — `error` holds the reason                 |

**Link Extractor** (`/link-extract/crawl`):

| Phase      | Meaning                                                             |
| ---------- | ------------------------------------------------------------------- |
| `queued`   | Job created, not yet started                                        |
| `sitemap`  | Looking for and parsing `sitemap.xml`                               |
| `crawling` | Following internal links breadth-first and collecting link metadata |
| `done`     | Finished — `result` is populated                                    |
| `error`    | Failed — `error` holds the reason                                   |

The Link Extractor has no `checking` phase — it reports link metadata only and never makes a second request to check status codes.

## Crawl limits

Both crawlers share the same tuning:

| Limit                         | Value             |
| ----------------------------- | ----------------- |
| Max pages crawled             | 250               |
| Time budget                   | 6 minutes         |
| Sitemap URLs considered       | 400               |
| Sitemap sub-sitemaps followed | 15                |
| Crawl concurrency             | 6 pages at a time |

If a site is larger than the cap, the result marks the relevant `truncated*` field `true` rather than failing — you get everything found up to the limit.

## Job expiry

Job state lives in memory and is discarded **30 minutes** after it last changed. Poll continuously while a crawl is running rather than storing a `jobId` to check back on later — a stale ID returns `404`:

```json theme={null}
{ "success": false, "error": "Crawl job not found or expired." }
```

<CardGroup cols={2}>
  <Card title="Broken Link Checker" icon="link-slash" href="/api-reference/broken-links">
    Full parameter and response reference.
  </Card>

  <Card title="Link Extractor" icon="link" href="/api-reference/link-extract">
    Full parameter and response reference.
  </Card>
</CardGroup>


## Related topics

- [Broken Link Checker](/api-reference/broken-links.md)
- [Link Extractor](/api-reference/link-extract.md)
- [Rate Limits & Credits](/rate-limits-and-credits.md)
- [Errors](/errors.md)