# AI Web Scraper – Any URL to Structured JSON (`sfoln/ai-web-scraper`)

Turn any web page into structured JSON with AI: give URLs plus a JSON Schema or a plain list of fields and get validated data per page (schema check with one automatic repair). Reads the main content and JSON-LD. $8 per 1,000 pages; unreadable pages are free.

- Family: Web toolkit for agents
- Categories: AI, Agents, Developer tools
- Availability: on request (private on Apify). Email contact@sfoln.com with subject "Access request: AI Web Scraper – Any URL to Structured JSON (sfoln/ai-web-scraper)" before calling; calls fail until it is public.
- JSON: https://www.sfoln.com/actors/ai-web-scraper.json
- MCP: https://mcp.apify.com?tools=sfoln/ai-web-scraper
- REST (sync): POST https://api.apify.com/v2/actors/sfoln~ai-web-scraper/run-sync-get-dataset-items
- REST (async): POST https://api.apify.com/v2/actors/sfoln~ai-web-scraper/runs

Turn any web page into structured JSON with AI. Give a list of URLs and say what you want: a JSON Schema, a plain list of fields such as price (number), or instructions in plain language. The Actor reads each page's main content (and the schema.org JSON-LD hidden in it), asks the model for exactly that data, validates the answer against your schema and, when it does not match, lets the model repair it once. You get one record per URL with data, valid and errors. Built for AI agents, enrichment pipelines and anyone who needs product, article, listing or contact data without writing a scraper per site. $8 per 1,000 pages; pages that cannot be read and pages with nothing to extract are free.

## Price per delivered record

| Event | Price | Per 1,000 | Charged for |
| --- | --- | --- | --- |
| Page extraction (`page-extraction`) | $0.008 | $8 / 1,000 | Charged once per URL whose page was read and from which the AI extracted data (at least one value), validated against your schema. |

Pay-per-event, USD, billed by Apify; no platform usage charge. Only records with status `ok` are charged. Cap a run with `maxTotalChargeUsd`. Apify free plan: up to 10 per run.

## Run

```bash
curl -X POST "https://api.apify.com/v2/actors/sfoln~ai-web-scraper/run-sync-get-dataset-items" \
  -H "Authorization: Bearer $APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"urls":["https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html"],"fields":["title","price (number): the price shown on the page","availability","upc: the product code"]}'
```

MCP client configuration:

```json
{
  "mcpServers": {
    "apify": {
      "type": "http",
      "url": "https://mcp.apify.com?tools=sfoln/ai-web-scraper"
    }
  }
}
```

Example prompt: Extract the title, price and availability from https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html as JSON.

## Input

Web page URLs plus what to extract from each: a JSON Schema, a list of fields, or plain-language instructions. Each URL gives one record with the extracted data.

| Field | Type | Default | Meaning |
| --- | --- | --- | --- |
| `urls` (required) | array | — | Page URLs: Web pages to extract data from, one per line (public http(s) pages; PDFs and other files are not read). Accounts on the Apify free plan can process up to 10 URLs per run. |
| `fields` | array | — | Fields to extract: The data to extract, one field per line: "name", "name: what it is" or "name (type): what it is". Types: string (default), number, integer, boolean, date, datetime, url, email, string[], number[], url[], object, object[]. A field the page does not state comes back as null. Use this or a JSON schema: when both are given, the schema is used and this list is ignored. |
| `schema` | object | — | JSON Schema: Instead of fields: a JSON Schema (draft-07 or 2020-12) of the result, e.g. {"type": "object", "properties": {"name": {"type": "string"}, "price": {"type": "number"}}, "required": ["name"]}. An array schema extracts a list (for example every product on a category page). At most 4,000 characters as compact JSON; only local $ref. |
| `instructions` | string | — | Instructions: Optional guidance in plain language (at most 2,000 characters), e.g. "Prices in EUR; dates as YYYY-MM-DD; ignore related products". On its own, without fields or a schema, the model decides the JSON keys. |
| `contentMode` | string | `"main"` | Page content: main: the page's main content (article, product details) as markdown, navigation and footers removed. full: all visible text of the page, for data that sits in headers, sidebars or footers (a page body above 800 KB, with more than 12,000 elements or nested deeper than 128 levels is read as main content instead, with a warning). One of `"main"`, `"full"`. |
| `includeContent` | boolean | `false` | Include page content: Add the page text that was sent to the model to each record (useful to check what the model saw). |
| `maxUrls` | integer | `1000` | Max URLs: Safety cap on how many URLs are processed in this run; the rest are ignored and counted in the OUTPUT record. |
| `concurrency` | integer | `3` | Concurrency: Pages processed in parallel (1–10). Pages on the same host are fetched at most 2 at a time. |
| `includeRaw` | boolean | `false` | Include raw model replies: Attach the model's raw replies (the first answer and the repair, if any) to each record. |

Example input:

```json
{
  "urls": [
    "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html"
  ],
  "fields": [
    "title",
    "price (number): the price shown on the page",
    "availability",
    "upc: the product code"
  ]
}
```

## Output

One record per input item with `status` (`ok` is charged; `not_found`, `error`, `skipped_*` are free) and `charged`. The run stores an `OUTPUT` summary record.

| Field | Type | Meaning |
| --- | --- | --- |
| `input` | string | object | Input: The URL as given in the input. |
| `url` | string | null | URL: The normalized URL that was requested. |
| `finalUrl` | string | null | Final URL: The URL that answered, after redirects. |
| `title` | string | null | Page title: The page's headline or document title. |
| `language` | string | null | Language: The page's declared language (BCP 47), when it has one. |
| `data` | object | array | string | number | boolean | null | Extracted data: The extracted data: an object shaped by your schema or field list (an array or other value when your schema's top level is one); null when the model failed. |
| `valid` | boolean | null | Valid: True when data validates against your schema (field lists always validate unless a value has the wrong type; instructions-only runs accept any JSON object). |
| `errors` | array | null | Validation errors: Why data does not validate: [{ path, message }] with a JSON Pointer path (at most 20); empty when valid. |
| `repaired` | boolean | null | Repaired: True when the first reply did not validate and the repair call's answer is the one returned. |
| `contentStatus` | string | null | Content status: ok \| blocked \| js_required \| not_html \| too_large \| error. The model is called only for ok pages. |
| `contentTruncated` | boolean | null | Content truncated: True when the page content was cut to fit the token budget before extraction. |
| `content` | string | null | Page content: The page text sent to the model (only with includeContent). |
| `model` | string | null | Model: The Mistral model that produced data. |
| `tokensUsed` | object | null | Tokens used: { input, output } tokens of the model calls for this page (informational; the price per page is fixed). |
| `status` | string | Status: ok (data extracted, charged) \| not_found (nothing on the page matched, or HTTP 404; free) \| error (page or model failure; free) \| skipped_invalid \| skipped_budget \| skipped_free_plan \| skipped_unmetered |
| `charged` | boolean | Charged: Whether this record incurred a pay-per-event charge. |
| `chargedEvent` | string | Charged event: page-extraction when charged. |
| `chargedCount` | integer | Events charged: Events actually charged when the charge came back short of the requested count (present only then). |
| `source` | string | Source: web+mistral: the page fetched by the Actor, extraction by Mistral. |
| `fetchedAt` | string | Fetched at: ISO timestamp. |
| `error` | object | Error: { code, message, provider?, status? } when status is not ok. Page codes: invalid_url, not_html, blocked, js_required, thin_content, too_large, not_found, http_error, timeout, network, dns_not_found, tls_error, blocked_url; model codes: nothing_extracted, output_too_long, invalid_model_reply, rate_limit, timeout, server, client; runtime codes: budget_exhausted, free_plan_limit, unmetered, aborted, charge_failed, record_too_large, invalid_record, dataset_push_failed. |
| `warnings` | array | Warnings: Notes about this record (content cut, schema mismatch after the repair, a failed repair). |
| `raw` | object | null | Raw model replies: The model's raw replies (only with includeRaw). |

Example record:

```json
{
  "input": "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html",
  "url": "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html",
  "finalUrl": "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html",
  "title": "A Light in the Attic",
  "language": "en-us",
  "contentStatus": "ok",
  "contentTruncated": false,
  "data": {
    "title": "A Light in the Attic",
    "price": 51.77,
    "availability": "In stock (22 available)",
    "upc": "a897fe39b1053632"
  },
  "valid": true,
  "errors": [],
  "repaired": false,
  "model": "mistral-small-2603",
  "tokensUsed": {
    "input": 642,
    "output": 48
  },
  "status": "ok",
  "charged": true,
  "chargedEvent": "page-extraction",
  "source": "web+mistral",
  "fetchedAt": "2026-10-14T10:00:02.000Z"
}
```

## FAQ

**Schema, fields or instructions: which should I use?**

A JSON Schema gives you full control (nested objects, arrays, enums, formats) and the result is validated against it. A field list ("price (number): the listed price") is the quick way: every field comes back, null when the page does not state it. Instructions alone let the model pick the keys; combine them with a schema or fields for formatting rules such as currencies or date formats.

**What do valid and errors mean, and is an invalid result charged?**

Every reply is validated against your schema. When it does not validate, the model gets one repair attempt with the schema and the validation errors (included in the price). valid is true when the final data matches the schema; otherwise errors lists each problem as { path, message }. A result with at least one extracted value is charged even when it is not fully valid (for example a required field the page does not contain); make such fields nullable to avoid it. Results with no value at all are returned as not_found and are free.

**Which pages work?**

Public HTML pages that show their content without JavaScript: product pages, articles, listings, docs, plain text and JSON endpoints. The page's main content is converted to markdown and its JSON-LD structured data is added, so prices and ratings from schema.org markup are found too. Pages that need JavaScript, sit behind bot protection or a login, and PDFs or other files are reported with contentStatus and not charged.

**How much of a page does the model read?**

About 4,800 tokens per page in total, of which the schema and instructions take up to 1,000; the page content (main content first) fills the rest, which is roughly 14,000 characters of text for a small schema. contentTruncated tells you when a long page was cut. For long listings, use contentMode main and ask for fewer items per page.


## Related

- [Website to Markdown Crawler (llms.txt)](https://www.sfoln.com/actors/website-to-markdown.md) (`sfoln/website-to-markdown`)
- [URL to Markdown & Article Extractor](https://www.sfoln.com/actors/url-to-markdown-summary.md) (`sfoln/url-to-markdown-summary`)
- [RAG Web Search – Google + LLM-Ready Markdown](https://www.sfoln.com/actors/rag-web-search.md) (`sfoln/rag-web-search`)
- [Document Parser – AI PDF & Image to JSON](https://www.sfoln.com/actors/document-to-json.md) (`sfoln/document-to-json`)
- [Sitemap URL Extractor](https://www.sfoln.com/actors/sitemap-url-extractor.md) (`sfoln/sitemap-url-extractor`)

Publisher: SFOLN LLC, contact@sfoln.com. Catalogue: https://www.sfoln.com/catalog.json. Agent guide: https://www.sfoln.com/agents.md.