How we fetch
What a scraper reads — public pages, never logged in — and what it cannot read yet.
What a scraper reads, what it does not read, and what it cannot read yet.
What we collect#
- Public pages only. A scraper reads pages that anyone can open without an account.
- Never logged in. We do not sign in to any site, we do not use anyone's credentials, and a target address may not carry a login.
- Public host names only. A target is a public web address — not an IP address, and not a name that only resolves inside a private network. See What a target may be.
- LinkedIn: logged-out pages only. The LinkedIn scrapers read what LinkedIn shows a visitor who is not signed in: its public job search, a posting's public page, a company's public page. What LinkedIn keeps behind a sign-in — the apply link, a company's employees — is not read, and names of recruiters on a posting are not returned.
Each scraper's page lists exactly which fields a record carries: see Scrapers.
What we cannot read yet: JavaScript rendering#
Some pages arrive almost empty and build their content in the browser by running JavaScript. Reading them takes a real browser — a render. We do not render pages yet.
So today a page that needs a render is not read, and the run tells you so:
| What the run found | Status | Reason |
|---|---|---|
| Nothing could be read without a render | unsupported | render_required |
| Some pages needed a render; others gave records | partial | render_required |
Known cases:
- Dayforce boards are drawn entirely by JavaScript.
ats.boardanswersunsupportedfor them at once. - Company careers sites built as single-page apps can need a render. When the scraper detects that such a site takes its jobs from an applicant tracking system it can read directly, you get those records anyway.
web.fetchmakes oneGETrequest and returns the page as the server sent it. No script is run, so a page that builds its content in the browser comes back as the shell it starts from — in anokrun, because the fetch itself succeeded.
An unsupported run costs nothing, and trying again gives the same answer until rendering is available.
When a site refuses us#
A site can answer a scraper with a refusal instead of the page: a 403, a "too many requests", a CAPTCHA, a sign-in
page. We do not solve CAPTCHAs and we do not sign in. Such a run ends failed — or partial, when records were
read before the refusal — with a reason that names the refusal (blocked, rate_limited, captcha, auth_wall),
and a failed run costs nothing. The codes are listed under
Reasons by scraper family.
This is different from a page the site itself answers with an error status. For web.fetch, a 404 or a 500 is
the page that was asked for: the run is ok, and the record carries the status.
How results are reported#
- A site we could not read is reported as
failedorunsupported— never as "no openings". - A result with a known gap is
partial, with areasonthat names the gap. - A value we do not know is
nullin a record — never an empty string and never0.
Every status and reason code, with what to do about each, is in Results and statuses.