Scraper · Web
web.crawl
Crawl a public website from a start page and read its pages as clean Markdown, obeying robots.txt.
- Id
web.crawl- Version
- 1
- Billed per
- record — 1 unit for each record of a charged run
- Runs
- in the background; use wait or poll
Start it with POST /v1/scrapers/web.crawl/runs. How runs work is in Running a scraper; what a run costs is in Usage and units.
Inputs
The fields of input. A field that is not listed here is refused.
| Field | Type | Description |
|---|---|---|
main_content_only | boolean | Write the page's own content only: no navigation, page header or footer, sidebars or cookie banners. False writes the whole page body.Default: true. |
include_links | boolean | Write links as [text](address); false keeps their text only.Default: true. |
include_images | boolean | Write images as ; false keeps their alt text only.Default: false. |
urlrequired | string | The start page: http or https, on a public host name.Limit: at most 2048 characters. |
max_pages | integer | Stop after this many pages (records).Limit: 1 to 100. Default: 10. |
max_depth | integer | How many links away from the start page a page may be; 0 reads the start page only. Pages from the sitemap count as 1.Limit: 0 to 10. Default: 3. |
include_paths | array of string | Read only pages whose path starts with one of these (/docs/), or matches one with * (/blog/*/comments). The start page is always read. |
exclude_paths | array of string | Never read pages whose path starts with or matches one of these. |
same_site_only | true | Only the start page's site is crawled; this cannot be turned off.Default: true. |
max_seconds | integer | The time budget of the whole crawl, in seconds.Limit: 10 to 900. Default: 300. |
- Only the start page's site is crawled, and its
robots.txtis obeyed. - Billed per record: a charged run costs 1 unit for each page it returns.
max_pagesbounds that. - No JavaScript is run.
Record fields
Every record of a run has these fields. A value that is not known is null — never an empty string and never 0.
| Field | Type | Notes |
|---|---|---|
url | string | The address that was asked for. |
final_url | string | The address that answered, after redirects. |
status | integer | The HTTP status of the answer. |
content_type | string or null | The media type of the answer, e.g. text/html. |
title | string or null | The page's title. |
description | string or null | The page's meta description. |
language | string or null | The language the page declares (e.g. en, de-CH); null when it declares none. |
canonical_url | string or null | The canonical address the page names for itself. |
markdown | string | The page's content as Markdown. |
text_length | integer | Characters of plain text in the content that was written. |
word_count | integer | Words of plain text in the content that was written. |
links | array of object | The links on the whole page, once each (the first 2000). |
metadata | object | |
open_graph (in metadata) | object of string values | The page's Open Graph properties (og:*, article:*). |
json_ld (in metadata) | array of object | The page's JSON-LD objects, as written. |
truncated | boolean | True when the page was longer than 5000000 bytes and was cut there.Default: false. |
depth | integer or null | web.crawl only: links between the start page and this one. |
fetched_at | string (date-time) | Always present. |
Example request
curl -X POST "https://api.enrichednow.com/v1/scrapers/web.crawl/runs?wait=60" \
-H "Authorization: Bearer $ENRICHEDNOW_API_KEY" \
-H "Content-Type: application/json" \
-d '{"input":{"url":"https://example.com/","max_pages":10}}'The body on its own:
{
"input": {
"url": "https://example.com/",
"max_pages": 10
}
}Example response
The run, once it has ended. A run that has not ended within wait comes back with 202 and the status queued or running.
{
"id": "run_6cce2b12aaf37a10b5e1b885",
"scraper_id": "web.crawl",
"scraper_version": 1,
"status": "ok",
"reason": null,
"input": {
"url": "https://example.com/",
"max_pages": 10
},
"record_count": 1,
"units": 1,
"diagnostics": {
"requests": 1,
"bytes": 232779,
"wall_ms": 565,
"facts": {}
},
"created_at": "2026-11-16T20:27:29.009834Z",
"started_at": "2026-11-16T20:27:29.013803Z",
"finished_at": "2026-11-16T20:27:29.582787Z"
}Its records, from GET /v1/runs/{id}/records (the values are an illustration; the fields are the ones above):
{
"records": [
{
"url": "string",
"final_url": "string",
"status": 0,
"content_type": null,
"title": null,
"description": null,
"language": null,
"canonical_url": null,
"markdown": "string",
"text_length": 0,
"word_count": 0,
"links": [],
"metadata": {},
"truncated": false,
"depth": null,
"fetched_at": "2026-01-01T00:00:00Z"
}
],
"next_cursor": null
}A run does not always end ok. Results and statuses lists every status and the reasons every scraper shares; the reasons of the Web scrapers are in their own section there.