Skip to the content
EnrichedNowDocs

    Type to search every page.

    How we fetch

    What a scraper reads — public pages, never logged in — and what it cannot read yet.

    What a scraper reads, what it does not read, and what it cannot read yet.

    What we collect#

    • Public pages only. A scraper reads pages that anyone can open without an account.
    • Never logged in. We do not sign in to any site, we do not use anyone's credentials, and a target address may not carry a login.
    • Public host names only. A target is a public web address — not an IP address, and not a name that only resolves inside a private network. See What a target may be.
    • LinkedIn: logged-out pages only. The LinkedIn scrapers read what LinkedIn shows a visitor who is not signed in: its public job search, a posting's public page, a company's public page. What LinkedIn keeps behind a sign-in — the apply link, a company's employees — is not read, and names of recruiters on a posting are not returned.

    Each scraper's page lists exactly which fields a record carries: see Scrapers.

    What we cannot read yet: JavaScript rendering#

    Some pages arrive almost empty and build their content in the browser by running JavaScript. Reading them takes a real browser — a render. We do not render pages yet.

    So today a page that needs a render is not read, and the run tells you so:

    What the run foundStatusReason
    Nothing could be read without a renderunsupportedrender_required
    Some pages needed a render; others gave recordspartialrender_required

    Known cases:

    • Dayforce boards are drawn entirely by JavaScript. ats.board answers unsupported for them at once.
    • Company careers sites built as single-page apps can need a render. When the scraper detects that such a site takes its jobs from an applicant tracking system it can read directly, you get those records anyway.
    • web.fetch makes one GET request and returns the page as the server sent it. No script is run, so a page that builds its content in the browser comes back as the shell it starts from — in an ok run, because the fetch itself succeeded.

    An unsupported run costs nothing, and trying again gives the same answer until rendering is available.

    When a site refuses us#

    A site can answer a scraper with a refusal instead of the page: a 403, a "too many requests", a CAPTCHA, a sign-in page. We do not solve CAPTCHAs and we do not sign in. Such a run ends failed — or partial, when records were read before the refusal — with a reason that names the refusal (blocked, rate_limited, captcha, auth_wall), and a failed run costs nothing. The codes are listed under Reasons by scraper family.

    This is different from a page the site itself answers with an error status. For web.fetch, a 404 or a 500 is the page that was asked for: the run is ok, and the record carries the status.

    How results are reported#

    • A site we could not read is reported as failed or unsupported — never as "no openings".
    • A result with a known gap is partial, with a reason that names the gap.
    • A value we do not know is null in a record — never an empty string and never 0.

    Every status and reason code, with what to do about each, is in Results and statuses.