Skip to the content
EnrichedNowDocs

    Type to search every page.

    Scraper · Web

    web.crawl

    Crawl a public website from a start page and read its pages as clean Markdown, obeying robots.txt.

    Id
    web.crawl
    Version
    1
    Billed per
    record — 1 unit for each record of a charged run
    Runs
    in the background; use wait or poll

    Start it with POST /v1/scrapers/web.crawl/runs. How runs work is in Running a scraper; what a run costs is in Usage and units.

    Inputs

    The fields of input. A field that is not listed here is refused.

    Inputs of web.crawl
    FieldTypeDescription
    main_content_onlybooleanWrite the page's own content only: no navigation, page header or footer, sidebars or cookie banners. False writes the whole page body.Default: true.
    include_linksbooleanWrite links as [text](address); false keeps their text only.Default: true.
    include_imagesbooleanWrite images as ![alt](address); false keeps their alt text only.Default: false.
    urlrequiredstringThe start page: http or https, on a public host name.Limit: at most 2048 characters.
    max_pagesintegerStop after this many pages (records).Limit: 1 to 100. Default: 10.
    max_depthintegerHow many links away from the start page a page may be; 0 reads the start page only. Pages from the sitemap count as 1.Limit: 0 to 10. Default: 3.
    include_pathsarray of stringRead only pages whose path starts with one of these (/docs/), or matches one with * (/blog/*/comments). The start page is always read.
    exclude_pathsarray of stringNever read pages whose path starts with or matches one of these.
    same_site_onlytrueOnly the start page's site is crawled; this cannot be turned off.Default: true.
    max_secondsintegerThe time budget of the whole crawl, in seconds.Limit: 10 to 900. Default: 300.
    • Only the start page's site is crawled, and its robots.txt is obeyed.
    • Billed per record: a charged run costs 1 unit for each page it returns. max_pages bounds that.
    • No JavaScript is run.

    Record fields

    Every record of a run has these fields. A value that is not known is null — never an empty string and never 0.

    Record fields of web.crawl
    FieldTypeNotes
    urlstringThe address that was asked for.
    final_urlstringThe address that answered, after redirects.
    statusintegerThe HTTP status of the answer.
    content_typestring or nullThe media type of the answer, e.g. text/html.
    titlestring or nullThe page's title.
    descriptionstring or nullThe page's meta description.
    languagestring or nullThe language the page declares (e.g. en, de-CH); null when it declares none.
    canonical_urlstring or nullThe canonical address the page names for itself.
    markdownstringThe page's content as Markdown.
    text_lengthintegerCharacters of plain text in the content that was written.
    word_countintegerWords of plain text in the content that was written.
    linksarray of objectThe links on the whole page, once each (the first 2000).
    metadataobject
    open_graph (in metadata)object of string valuesThe page's Open Graph properties (og:*, article:*).
    json_ld (in metadata)array of objectThe page's JSON-LD objects, as written.
    truncatedbooleanTrue when the page was longer than 5000000 bytes and was cut there.Default: false.
    depthinteger or nullweb.crawl only: links between the start page and this one.
    fetched_atstring (date-time)Always present.

    Example request

    Shell
    curl -X POST "https://api.enrichednow.com/v1/scrapers/web.crawl/runs?wait=60" \
      -H "Authorization: Bearer $ENRICHEDNOW_API_KEY" \
      -H "Content-Type: application/json" \
      -d '{"input":{"url":"https://example.com/","max_pages":10}}'

    The body on its own:

    JSON
    {
      "input": {
        "url": "https://example.com/",
        "max_pages": 10
      }
    }

    Example response

    The run, once it has ended. A run that has not ended within wait comes back with 202 and the status queued or running.

    JSON
    {
      "id": "run_6cce2b12aaf37a10b5e1b885",
      "scraper_id": "web.crawl",
      "scraper_version": 1,
      "status": "ok",
      "reason": null,
      "input": {
        "url": "https://example.com/",
        "max_pages": 10
      },
      "record_count": 1,
      "units": 1,
      "diagnostics": {
        "requests": 1,
        "bytes": 232779,
        "wall_ms": 565,
        "facts": {}
      },
      "created_at": "2026-11-16T20:27:29.009834Z",
      "started_at": "2026-11-16T20:27:29.013803Z",
      "finished_at": "2026-11-16T20:27:29.582787Z"
    }

    Its records, from GET /v1/runs/{id}/records (the values are an illustration; the fields are the ones above):

    JSON
    {
      "records": [
        {
          "url": "string",
          "final_url": "string",
          "status": 0,
          "content_type": null,
          "title": null,
          "description": null,
          "language": null,
          "canonical_url": null,
          "markdown": "string",
          "text_length": 0,
          "word_count": 0,
          "links": [],
          "metadata": {},
          "truncated": false,
          "depth": null,
          "fetched_at": "2026-01-01T00:00:00Z"
        }
      ],
      "next_cursor": null
    }

    A run does not always end ok. Results and statuses lists every status and the reasons every scraper shares; the reasons of the Web scrapers are in their own section there.