Website to Markdown Crawler (llms.txt)

What it does

Crawl a website or a docs section into clean, LLM-ready markdown, one row per page, and get llms.txt and llms-full.txt files for it. Give one or more start URLs; the Actor reads the start page, then the site's sitemap, then the links it finds (breadth-first, within the site or the section you started in), extracts each page's main content as markdown (menus, footers and ads removed, tables kept) and reports title, description, depth, word count and token estimate per page. Built for RAG pipelines, AI agents, documentation bots and anyone who needs a site's content as text. $1.50 per 1,000 pages; pages without content, duplicates and the llms.txt files are free.

Price per delivered record

Apify free plan: up to 10 items per run. Paid plans have no per-run cap; set maxTotalChargeUsd to cap spend.

Call it from code, an agent or the Console

curl -X POST "https://api.apify.com/v2/actors/sfoln~website-to-markdown/run-sync-get-dataset-items" \
  -H "Authorization: Bearer $APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"startUrls":["https://crawlee.dev/js/docs/introduction"],"maxPagesPerSite":3}'

{
  "mcpServers": {
    "apify": {
      "type": "http",
      "url": "https://mcp.apify.com?tools=sfoln/website-to-markdown"
    }
  }
}

POST https://api.apify.com/v2/actors/sfoln~website-to-markdown/runs?maxTotalChargeUsd=1
GET  https://api.apify.com/v2/actor-runs/{runId}            # until status is SUCCEEDED
GET  https://api.apify.com/v2/datasets/{defaultDatasetId}/items?format=json

Authenticate with an Apify API token. MCP clients can instead sign in with OAuth when prompted. Example prompt for an agent: Crawl https://crawlee.dev/js/docs/introduction (at most 10 pages) and give me the pages as markdown plus the llms.txt file.

Input fields

Start URLs of the sites (or site sections) to crawl. Each start URL gives one row per page read (markdown) plus one summary row with the crawl statistics and links to llms.txt and llms-full.txt.

{
  "startUrls": [
    "https://crawlee.dev/js/docs/introduction"
  ],
  "maxPagesPerSite": 3
}

One record per input item

Records land in the run's dataset; a synchronous call returns them directly. Each has a status (ok is charged; everything else is free) and a charged flag. The run's OUTPUT record totals items and charges.

[
  {
    "input": "https://crawlee.dev/js/docs/introduction",
    "recordType": "page",
    "position": 2,
    "url": "https://crawlee.dev/js/docs/introduction/setting-up",
    "requestedUrl": "https://crawlee.dev/js/docs/introduction/setting-up",
    "title": "Setting up",
    "siteName": "Crawlee",
    "description": "To run Crawlee on your computer, you need Node.js and npm.",
    "language": "en",
    "publishedAt": null,
    "canonicalUrl": "https://crawlee.dev/js/docs/introduction/setting-up",
    "depth": 1,
    "discoveredVia": "sitemap",
    "referrer": null,
    "markdown": "# Setting up\n\nIn this guide, we will set up Crawlee and run our first crawler. …",
    "markdownTruncated": false,
    "wordCount": 423,
    "tokenEstimate": 712,
    "contentStatus": "ok",
    "status": "ok",
    "charged": true,
    "chargedEvent": "crawled-page",
    "source": "website",
    "fetchedAt": "2026-10-14T10:00:04.000Z"
  },
  {
    "input": "https://crawlee.dev/js/docs/introduction",
    "recordType": "page",
    "url": "https://crawlee.dev/js/docs/introduction/old-page",
    "requestedUrl": "https://crawlee.dev/js/docs/introduction/old-page",
    "title": null,
    "depth": 1,
    "discoveredVia": "link",
    "referrer": "https://crawlee.dev/js/docs/introduction",
    "contentStatus": "error",
    "status": "error",
    "charged": false,
    "source": "website",
    "fetchedAt": "2026-10-14T10:00:05.000Z",
    "error": {
      "code": "not_found",
      "message": "HTTP 404 from crawlee.dev",
      "status": 404
    }
  },
  {
    "input": "https://crawlee.dev/js/docs/introduction",
    "startUrl": "https://crawlee.dev/js/docs/introduction",
    "finalStartUrl": "https://crawlee.dev/js/docs/introduction",
    "crawlScope": "path",
    "scopeLabel": "crawlee.dev/js/docs/introduction",
    "pagesFailed": 1,
    "duplicatePages": 0,
    "pagesFiltered": 0,
    "offsiteRedirects": 0,
    "urlsQueued": 10,
    "sitemapUrls": 9,
    "sitemap": {
      "discoveredVia": "robots",
      "sitemapsRead": 2,
      "sitemapsFailed": 0
    },
    "requests": 11,
    "maxDepthReached": 1,
    "llmsTxt": {
      "key": "llms-crawlee.dev-js-docs-introduction.txt",
      "url": "https://api.apify.com/v2/key-value-stores/<storeId>/records/llms-crawlee.dev-js-docs-introduction.txt",
      "pages": 9,
      "bytes": 1460
    },
    "llmsFullTxt": {
      "key": "llms-full-crawlee.dev-js-docs-introduction.txt",
      "url": "https://api.apify.com/v2/key-value-stores/<storeId>/records/llms-full-crawlee.dev-js-docs-introduction.txt",
      "pages": 9,
      "bytes": 61234,
      "truncated": false
    },
    "recordType": "summary",
    "resultsDelivered": 9,
    "stopReason": "end",
    "status": "ok",
    "charged": false,
    "source": "website",
    "fetchedAt": "2026-10-14T10:00:09.000Z"
  }
]

Questions

Which pages are crawled?

The start page first, then the URLs of the site's sitemap (found through robots.txt or /sitemap.xml) that are in scope, shallow paths first, then the links found on each page, breadth-first up to maxDepth. crawlScope auto stays under the start URL's path when it has one (https://example.com/docs crawls /docs and below) and otherwise covers the whole registrable domain, subdomains included. includePatterns and excludePatterns narrow it further. Files and media (PDF, images, archives) are skipped, and each page is read once even when several URLs lead to it.

What are llms.txt and llms-full.txt?

llms.txt (llmstxt.org) is a markdown index of a site for language models: the site's name and description, then one section per top-level path listing every page as a link with a short description. llms-full.txt contains the markdown of every page one after the other (up to 8 MB). Both are written to the run's key-value store (`llms-<site>.txt` and `llms-full-<site>.txt`); the summary row of each start URL links to them. They are free.

Why do some pages have contentStatus blocked or js_required?

No browser is used. blocked: the site answered with a bot check, an access-denied page or a rate limit. js_required: the page builds its content with JavaScript and the HTML has no text. Such pages are written as uncharged rows so you can see what was skipped; their links (if any) are still followed. For JavaScript-heavy sites the sitemap often still leads to readable pages.

How polite is the crawl?

At most 2 requests per second and 2 at a time per host, shared by every start URL of the run, with a 20-second timeout and a 5 MB cap per page. Each site gets at most 2 page requests per page it may deliver (maxPagesPerSite, 20 on the free plan) plus 20, so failing links cannot make a crawl run away.

Related APIs

• url-to-markdown-summary URL to Markdown & Article Extractor$1.5 / 1,000
• rag-web-search RAG Web Search – Google + LLM-Ready Markdown$1.5 / 1,000
• ai-web-scraper AI Web Scraper – Any URL to Structured JSON$8 / 1,000
• sitemap-url-extractor Sitemap URL Extractor$0.20 / 1,000
• website-change-monitor Website Change Monitor – Text Diff Alerts$1 / 1,000

This page for agents

• /actors/website-to-markdown.json — this API as JSON
• /actors/website-to-markdown.md — this API as Markdown
• apify.com/sfoln/website-to-markdown.md — the Store listing as Markdown
• /catalog.json — the whole catalogue