AI Web Scraper – Any URL to Structured JSON

What it does

Turn any web page into structured JSON with AI. Give a list of URLs and say what you want: a JSON Schema, a plain list of fields such as price (number), or instructions in plain language. The Actor reads each page's main content (and the schema.org JSON-LD hidden in it), asks the model for exactly that data, validates the answer against your schema and, when it does not match, lets the model repair it once. You get one record per URL with data, valid and errors. Built for AI agents, enrichment pipelines and anyone who needs product, article, listing or contact data without writing a scraper per site. $8 per 1,000 pages; pages that cannot be read and pages with nothing to extract are free.

Price per delivered record

Apify free plan: up to 10 items per run. Paid plans have no per-run cap; set maxTotalChargeUsd to cap spend.

Call it from code, an agent or the Console

This API is private on Apify for now. Email us and we will publish it for you, usually within two working days. The calls below work once it is public. contact@sfoln.com

curl -X POST "https://api.apify.com/v2/actors/sfoln~ai-web-scraper/run-sync-get-dataset-items" \
  -H "Authorization: Bearer $APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"urls":["https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html"],"fields":["title","price (number): the price shown on the page","availability","upc: the product code"]}'

{
  "mcpServers": {
    "apify": {
      "type": "http",
      "url": "https://mcp.apify.com?tools=sfoln/ai-web-scraper"
    }
  }
}

POST https://api.apify.com/v2/actors/sfoln~ai-web-scraper/runs?maxTotalChargeUsd=1
GET  https://api.apify.com/v2/actor-runs/{runId}            # until status is SUCCEEDED
GET  https://api.apify.com/v2/datasets/{defaultDatasetId}/items?format=json

Authenticate with an Apify API token. MCP clients can instead sign in with OAuth when prompted. Example prompt for an agent: Extract the title, price and availability from https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html as JSON.

Input fields

Web page URLs plus what to extract from each: a JSON Schema, a list of fields, or plain-language instructions. Each URL gives one record with the extracted data.

{
  "urls": [
    "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html"
  ],
  "fields": [
    "title",
    "price (number): the price shown on the page",
    "availability",
    "upc: the product code"
  ]
}

One record per input item

Records land in the run's dataset; a synchronous call returns them directly. Each has a status (ok is charged; everything else is free) and a charged flag. The run's OUTPUT record totals items and charges.

{
  "input": "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html",
  "url": "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html",
  "finalUrl": "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html",
  "title": "A Light in the Attic",
  "language": "en-us",
  "contentStatus": "ok",
  "contentTruncated": false,
  "data": {
    "title": "A Light in the Attic",
    "price": 51.77,
    "availability": "In stock (22 available)",
    "upc": "a897fe39b1053632"
  },
  "valid": true,
  "errors": [],
  "repaired": false,
  "model": "mistral-small-2603",
  "tokensUsed": {
    "input": 642,
    "output": 48
  },
  "status": "ok",
  "charged": true,
  "chargedEvent": "page-extraction",
  "source": "web+mistral",
  "fetchedAt": "2026-10-14T10:00:02.000Z"
}

Questions

Schema, fields or instructions: which should I use?

A JSON Schema gives you full control (nested objects, arrays, enums, formats) and the result is validated against it. A field list ("price (number): the listed price") is the quick way: every field comes back, null when the page does not state it. Instructions alone let the model pick the keys; combine them with a schema or fields for formatting rules such as currencies or date formats.

What do valid and errors mean, and is an invalid result charged?

Every reply is validated against your schema. When it does not validate, the model gets one repair attempt with the schema and the validation errors (included in the price). valid is true when the final data matches the schema; otherwise errors lists each problem as { path, message }. A result with at least one extracted value is charged even when it is not fully valid (for example a required field the page does not contain); make such fields nullable to avoid it. Results with no value at all are returned as not_found and are free.

Which pages work?

Public HTML pages that show their content without JavaScript: product pages, articles, listings, docs, plain text and JSON endpoints. The page's main content is converted to markdown and its JSON-LD structured data is added, so prices and ratings from schema.org markup are found too. Pages that need JavaScript, sit behind bot protection or a login, and PDFs or other files are reported with contentStatus and not charged.

How much of a page does the model read?

About 4,800 tokens per page in total, of which the schema and instructions take up to 1,000; the page content (main content first) fills the rest, which is roughly 14,000 characters of text for a small schema. contentTruncated tells you when a long page was cut. For long listings, use contentMode main and ask for fewer items per page.

Related APIs

• website-to-markdown Website to Markdown Crawler (llms.txt)$1.5 / 1,000
• url-to-markdown-summary URL to Markdown & Article Extractor$1.5 / 1,000
• rag-web-search RAG Web Search – Google + LLM-Ready Markdown$1.5 / 1,000
• document-to-json Document Parser – AI PDF & Image to JSON$7 / 1,000
• sitemap-url-extractor Sitemap URL Extractor$0.20 / 1,000

This page for agents

• /actors/ai-web-scraper.json — this API as JSON
• /actors/ai-web-scraper.md — this API as Markdown
• /catalog.json — the whole catalogue