Limits
The allowance, concurrent runs, page sizes, retention and input limits.
The numbers that bound what you can do, in one place.
Account limits#
| Limit | Default | When you reach it |
|---|---|---|
| Unit allowance | 1,000 units a calendar month (UTC) | Starting a run is refused: 402 quota_exceeded. |
| Runs going at the same time | 2 | Starting a run is refused: 429 too_many_runs. Wait for a run to end. |
Both are set per account and can be changed for an account.
The console shows your account's own numbers; allowance_units is also in the answer of GET /v1/usage.
Request limits#
| What | Limit |
|---|---|
wait on POST /v1/scrapers/{id}/runs | 0 to 60 seconds; fractions are accepted. Default 0. |
limit on GET /v1/runs | 1 to 100 runs per page. Default 20. |
limit on GET /v1/runs/{id}/records | 1 to 1000 records per page. Default 100. |
The period of GET /v1/usage | At most 366 days. |
| Sign-up and log-in attempts | 20 a minute. More are answered 429 rate_limited with a Retry-After header. |
A value outside its range is refused with 422 invalid_input. Calls made with an API key are not rate limited;
what bounds them is the number of runs going at the same time.
Run limits#
| What | Limit |
|---|---|
| How long a run may take | 5 minutes. A run that reaches it ends partial or failed with the reason deadline_exceeded. |
Longer lists are read page by page with cursor — see
Fetch the records.
Retention#
| What | Kept for |
|---|---|
| A run's records | 7 days after the run ended |
| The run itself — status, reason, counts, diagnostics | Kept |
| A console session | 30 days |
Fetch records within 7 days of the run's end and store what you need. After that the records endpoint answers
with an empty list and "expired": true — see Fetch the records.
Input limits#
Each scraper states the limits of its own input. The limits on each scraper page are generated from
the scraper's schema and are always current; the ones asked about most are here. A value outside its limit is
refused with 422 invalid_input before any run is created.
careers.company and ats.board:
| Field | Limit |
|---|---|
max_detail_pages | 0 to 200. Default 30. |
board_url, careers_url | At most 2,000 characters. |
domain | At most 253 characters. |
company_name | At most 200 characters. |
linkedin.jobs_search:
| Field | Limit |
|---|---|
max_results | 1 to 1000. Default 25. LinkedIn serves at most 1,000 results for one search. |
posted_within_hours | 1 to 720. |
keywords, location | At most 200 characters each. |
web.fetch:
| What | Limit |
|---|---|
method | GET only. No request body can be sent. |
url | At most 2,048 characters; http or https; port 80, 443, 8080 or 8443. |
headers | At most 10, from the names on the scraper's page. Each value at most 1,024 characters. No cookie and no Authorization header. |
max_bytes | 1 to 10000000. Default 2000000 (2 MB): the body is cut after that, and the record's truncated is true. |
timeout_s | 1 to 120 seconds for the whole fetch. Default 30. |
| Redirects followed | At most 5, each to a public web address. |
| JavaScript | Not run. The body is what the server sent. |
What a target may be#
A scraper reads public web pages. An input is refused when the target is:
- an IP address rather than a host name;
- a name that only resolves inside a network —
localhost, or one ending in.local,.internal,.lan,.home,.corp,.test,.example,.invalid,.onionor.arpa; - not an
httporhttpsaddress; - an address that carries a login (
https://user:password@…).
We do not log in to sites and we do not read pages behind a login.
What we cannot read yet#
Pages that only show their content after running JavaScript. A careers run that meets one ends unsupported — or
partial, when other pages of the site gave records — with the reason render_required, and an unsupported run
costs nothing. web.fetch returns such a page as the server sent it, before any script has run.
How we fetch explains why.