{"name":"website-to-markdown","id":"sfoln/website-to-markdown","title":"Website to Markdown Crawler (llms.txt)","description":"Crawl a website or docs section into clean, LLM-ready markdown: sitemap first, then links, one row per page with title, depth, word and token counts, plus llms.txt and llms-full.txt files for AI agents and RAG. Polite, no browser needed. $1.50 per 1,000 pages.","version":"0.1","status":"ready","family":{"id":"ai-agent-web-toolkit","name":"Web toolkit for agents","description":"Web search for retrieval, pages and sites converted to Markdown, AI extraction to JSON, URL metadata and safety checks, sitemaps, feeds, change monitoring and academic paper search.","actors":11},"categories":["AI","AGENTS","DEVELOPER_TOOLS"],"availability":"public","seoTitle":"Website to Markdown Crawler – llms.txt for AI & RAG","seoDescription":"Crawl any website or docs section into clean markdown, one page per row, and get llms.txt and llms-full.txt files for AI agents. $1.50 per 1,000 pages.","summary":"Crawl a website or a docs section into clean, LLM-ready markdown, one row per page, and get llms.txt and llms-full.txt files for it. Give one or more start URLs; the Actor reads the start page, then the site's sitemap, then the links it finds (breadth-first, within the site or the section you started in), extracts each page's main content as markdown (menus, footers and ads removed, tables kept) and reports title, description, depth, word count and token estimate per page. Built for RAG pipelines, AI agents, documentation bots and anyone who needs a site's content as text. $1.50 per 1,000 pages; pages without content, duplicates and the llms.txt files are free.","pricing":{"model":"pay-per-event","currency":"USD","billedBy":"Apify","freePlanMaxItemsPerRun":10,"events":[{"name":"crawled-page","title":"Crawled page","description":"Charged once per page delivered with readable content as markdown.","priceUsd":0.0015}]},"input":{"description":"Start URLs of the sites (or site sections) to crawl. Each start URL gives one row per page read (markdown) plus one summary row with the crawl statistics and links to llms.txt and llms-full.txt.","required":["startUrls"],"fields":[{"name":"startUrls","title":"Start URLs","type":"array","required":true,"description":"Websites or site sections to crawl, one per line, e.g. https://docs.example.com or https://example.com/blog. Accounts on the Apify free plan can process up to 10 start URLs per run (20 pages each).","prefill":["https://crawlee.dev/js/docs/introduction"],"editor":"stringList"},{"name":"maxPagesPerSite","title":"Max pages per site","type":"integer","required":false,"description":"Pages with content to deliver per start URL (1–10,000). Pages without content, duplicates and filtered pages do not count.","default":50,"prefill":3,"minimum":1,"maximum":10000},{"name":"maxDepth","title":"Max link depth","type":"integer","required":false,"description":"How many links away from the start page to follow (0–20). Sitemap URLs count as depth 1; 0 reads only the start page.","default":5,"minimum":0,"maximum":20},{"name":"crawlScope","title":"Crawl scope","type":"string","required":false,"description":"auto: under the start URL's path when it has one, else the whole domain. path: the start URL's host, under its path. host: the start URL's host only. domain: every subdomain of the start URL's registrable domain.","default":"auto","enum":["auto","path","host","domain"],"editor":"select"},{"name":"useSitemap","title":"Use the sitemap","type":"boolean","required":false,"description":"Queue the site's sitemap URLs that are in scope before the links found on pages (sitemap from robots.txt or /sitemap.xml).","default":true},{"name":"includePatterns","title":"Include URL patterns","type":"array","required":false,"description":"Deliver only pages whose URL matches one of these, one per line: a glob where * matches anything (e.g. */blog/*) or a regular expression in slashes (e.g. /\\/docs\\/v2\\//). Other pages are still read for their links. A regular expression that backtracks catastrophically is refused (prefer globs or character classes).","editor":"stringList"},{"name":"excludePatterns","title":"Exclude URL patterns","type":"array","required":false,"description":"Do not deliver pages whose URL matches one of these (same syntax as the include patterns), e.g. */tag/* or */page/*.","editor":"stringList"},{"name":"maxCharsPerPage","title":"Max characters per page","type":"integer","required":false,"description":"Markdown kept per page (1,000–200,000 characters), cut at a paragraph boundary. About 4 characters make one LLM token.","default":50000,"minimum":1000,"maximum":200000},{"name":"llmsTxt","title":"Write llms.txt and llms-full.txt","type":"boolean","required":false,"description":"Write llms.txt (an index of the delivered pages) and llms-full.txt (all their markdown, up to 8 MB) per start URL to the run's key-value store. Free.","default":true},{"name":"maxSites","title":"Max start URLs","type":"integer","required":false,"description":"Safety cap on how many start URLs are crawled in this run; the rest are ignored and counted in the OUTPUT record.","default":100,"minimum":1,"maximum":1000},{"name":"concurrency","title":"Sites in parallel","type":"integer","required":false,"description":"Start URLs crawled at the same time (1–5). Each site is read at most 2 requests per second.","default":2,"minimum":1,"maximum":5}]},"exampleInput":{"startUrls":["https://crawlee.dev/js/docs/introduction"],"maxPagesPerSite":3},"outputFields":[{"name":"input","type":"string | object","title":"Input","description":"The start URL this record belongs to."},{"name":"recordType","type":"string","title":"Record type","description":"page = one page read (charged when contentStatus is ok); summary = one record per start URL with the crawl statistics.","example":"page"},{"name":"position","type":"integer","title":"Position","description":"page rows with content: 1-based position among the pages delivered for the start URL."},{"name":"url","type":"string | null","title":"URL","description":"page: the page URL (after redirects)."},{"name":"requestedUrl","type":"string | null","title":"Requested URL","description":"page: the URL that was requested (differs from url after a redirect)."},{"name":"title","type":"string | null","title":"Title","description":"page: the page's headline or title."},{"name":"siteName","type":"string | null","title":"Site name","description":"page: the site's name (Open Graph), when the page states it."},{"name":"description","type":"string | null","title":"Description","description":"page: the meta description, else the opening of the content."},{"name":"language","type":"string | null","title":"Language","description":"page: the declared language (BCP 47), when the page has one."},{"name":"publishedAt","type":"string | null","title":"Published at","description":"page: ISO 8601 publication time, when the page states one."},{"name":"canonicalUrl","type":"string | null","title":"Canonical URL","description":"page: the page's rel=canonical URL, when it has one."},{"name":"depth","type":"integer | null","title":"Depth","description":"page: links away from the start page (0 = the start page; sitemap URLs are depth 1)."},{"name":"discoveredVia","type":"string | null","title":"Discovered via","description":"page: start | sitemap | link."},{"name":"referrer","type":"string | null","title":"Referrer","description":"page: the page whose link led here (null for the start page and sitemap URLs)."},{"name":"markdown","type":"string | null","title":"Markdown","description":"page: the main content as GitHub-flavoured markdown (navigation, ads and page chrome removed), at most maxCharsPerPage characters."},{"name":"markdownTruncated","type":"boolean | null","title":"Markdown truncated","description":"page: true when the content was cut at maxCharsPerPage."},{"name":"wordCount","type":"integer | null","title":"Word count","description":"page: words of the extracted content."},{"name":"tokenEstimate","type":"integer | null","title":"Token estimate","description":"page: rough LLM token count of the markdown (about 4 characters per token)."},{"name":"contentStatus","type":"string | null","title":"Content status","description":"page: ok | blocked | js_required | not_html | too_large | error. Only ok pages are charged."},{"name":"startUrl","type":"string | null","title":"Start URL","description":"summary: the start URL as given."},{"name":"finalStartUrl","type":"string | null","title":"Final start URL","description":"summary: the start page after redirects."},{"name":"crawlScope","type":"string | null","title":"Crawl scope","description":"summary: path | host | domain (what auto resolved to)."},{"name":"scopeLabel","type":"string | null","title":"Scope","description":"summary: what was in scope: host + path, a host, or a registrable domain."},{"name":"pagesFailed","type":"integer | null","title":"Pages without content","description":"summary: pages read without usable content (uncharged page rows)."},{"name":"duplicatePages","type":"integer | null","title":"Duplicate pages","description":"summary: pages whose URL or content repeated a page already read (not delivered, not charged)."},{"name":"pagesFiltered","type":"integer | null","title":"Filtered pages","description":"summary: pages left out by includePatterns / excludePatterns (read for their links, not delivered)."},{"name":"offsiteRedirects","type":"integer | null","title":"Off-site redirects","description":"summary: pages that redirected out of the crawl scope."},{"name":"urlsQueued","type":"integer | null","title":"URLs queued","description":"summary: URLs queued in scope (start, sitemap and links)."},{"name":"sitemapUrls","type":"integer | null","title":"Sitemap URLs","description":"summary: in-scope URLs queued from the sitemap."},{"name":"sitemap","type":"object | null","title":"Sitemap","description":"summary: { discoveredVia (robots | probe), sitemapsRead, sitemapsFailed }, or null when the sitemap was not used."},{"name":"requests","type":"integer | null","title":"Page requests","description":"summary: page requests sent (at most 2 × maxPagesPerSite + 20; 60 on the free plan)."},{"name":"maxDepthReached","type":"integer | null","title":"Deepest page","description":"summary: the largest depth read."},{"name":"llmsTxt","type":"object | null","title":"llms.txt","description":"summary: { key, url, pages, bytes } of the llms.txt record in the run's key-value store, or null."},{"name":"llmsFullTxt","type":"object | null","title":"llms-full.txt","description":"summary: { key, url, pages, bytes, truncated } of the llms-full.txt record, or null."},{"name":"resultsDelivered","type":"integer | null","title":"Pages delivered","description":"summary: page rows delivered (and charged) for this start URL."},{"name":"stopReason","type":"string | null","title":"Stop reason","description":"summary: limit (maxPagesPerSite reached) | end (nothing more in scope) | budget | free_plan | deadline | aborted | error (an include/exclude pattern was too slow on a URL: error.code pattern_too_slow)."},{"name":"status","type":"string","title":"Status","description":"ok | not_found | error | skipped_invalid | skipped_budget | skipped_free_plan | skipped_unmetered. Page rows without content have status error and charged false.","example":"ok"},{"name":"charged","type":"boolean","title":"Charged","description":"page rows: true when the page was charged (contentStatus ok); summary rows are never charged."},{"name":"chargedEvent","type":"string","title":"Charged event","description":"crawled-page on charged page rows."},{"name":"chargedCount","type":"integer","title":"Events charged","description":"Events actually charged when a charge came back short (present only then)."},{"name":"source","type":"string","title":"Source","description":"website: pages fetched by the Actor."},{"name":"fetchedAt","type":"string","title":"Fetched at","description":"ISO timestamp."},{"name":"error","type":"object","title":"Error","description":"{ code, message, status? } when status is not ok. Page codes: blocked, js_required, not_html, unexpected_type, too_large, thin_content, not_found, http_error, timeout, network, dns_not_found, tls_error, blocked_url; summary codes: invalid_url, blocked_url, dns_not_found, not_found, no_content; runtime codes: budget_exhausted, free_plan_limit, unmetered, aborted, charge_failed, record_too_large, invalid_record, dataset_push_failed."},{"name":"warnings","type":"array","title":"Warnings","description":"Notes about this record (caps, sitemap problems, llms.txt files)."}],"outputExample":[{"input":"https://crawlee.dev/js/docs/introduction","recordType":"page","position":2,"url":"https://crawlee.dev/js/docs/introduction/setting-up","requestedUrl":"https://crawlee.dev/js/docs/introduction/setting-up","title":"Setting up","siteName":"Crawlee","description":"To run Crawlee on your computer, you need Node.js and npm.","language":"en","publishedAt":null,"canonicalUrl":"https://crawlee.dev/js/docs/introduction/setting-up","depth":1,"discoveredVia":"sitemap","referrer":null,"markdown":"# Setting up\n\nIn this guide, we will set up Crawlee and run our first crawler. …","markdownTruncated":false,"wordCount":423,"tokenEstimate":712,"contentStatus":"ok","status":"ok","charged":true,"chargedEvent":"crawled-page","source":"website","fetchedAt":"2026-10-14T10:00:04.000Z"},{"input":"https://crawlee.dev/js/docs/introduction","recordType":"page","url":"https://crawlee.dev/js/docs/introduction/old-page","requestedUrl":"https://crawlee.dev/js/docs/introduction/old-page","title":null,"depth":1,"discoveredVia":"link","referrer":"https://crawlee.dev/js/docs/introduction","contentStatus":"error","status":"error","charged":false,"source":"website","fetchedAt":"2026-10-14T10:00:05.000Z","error":{"code":"not_found","message":"HTTP 404 from crawlee.dev","status":404}},{"input":"https://crawlee.dev/js/docs/introduction","startUrl":"https://crawlee.dev/js/docs/introduction","finalStartUrl":"https://crawlee.dev/js/docs/introduction","crawlScope":"path","scopeLabel":"crawlee.dev/js/docs/introduction","pagesFailed":1,"duplicatePages":0,"pagesFiltered":0,"offsiteRedirects":0,"urlsQueued":10,"sitemapUrls":9,"sitemap":{"discoveredVia":"robots","sitemapsRead":2,"sitemapsFailed":0},"requests":11,"maxDepthReached":1,"llmsTxt":{"key":"llms-crawlee.dev-js-docs-introduction.txt","url":"https://api.apify.com/v2/key-value-stores/<storeId>/records/llms-crawlee.dev-js-docs-introduction.txt","pages":9,"bytes":1460},"llmsFullTxt":{"key":"llms-full-crawlee.dev-js-docs-introduction.txt","url":"https://api.apify.com/v2/key-value-stores/<storeId>/records/llms-full-crawlee.dev-js-docs-introduction.txt","pages":9,"bytes":61234,"truncated":false},"recordType":"summary","resultsDelivered":9,"stopReason":"end","status":"ok","charged":false,"source":"website","fetchedAt":"2026-10-14T10:00:09.000Z"}],"faq":[{"question":"Which pages are crawled?","answer":"The start page first, then the URLs of the site's sitemap (found through robots.txt or /sitemap.xml) that are in scope, shallow paths first, then the links found on each page, breadth-first up to maxDepth. crawlScope auto stays under the start URL's path when it has one (https://example.com/docs crawls /docs and below) and otherwise covers the whole registrable domain, subdomains included. includePatterns and excludePatterns narrow it further. Files and media (PDF, images, archives) are skipped, and each page is read once even when several URLs lead to it."},{"question":"What are llms.txt and llms-full.txt?","answer":"llms.txt (llmstxt.org) is a markdown index of a site for language models: the site's name and description, then one section per top-level path listing every page as a link with a short description. llms-full.txt contains the markdown of every page one after the other (up to 8 MB). Both are written to the run's key-value store (`llms-<site>.txt` and `llms-full-<site>.txt`); the summary row of each start URL links to them. They are free."},{"question":"Why do some pages have contentStatus blocked or js_required?","answer":"No browser is used. blocked: the site answered with a bot check, an access-denied page or a rate limit. js_required: the page builds its content with JavaScript and the HTML has no text. Such pages are written as uncharged rows so you can see what was skipped; their links (if any) are still followed. For JavaScript-heavy sites the sitemap often still leads to readable pages."},{"question":"How polite is the crawl?","answer":"At most 2 requests per second and 2 at a time per host, shared by every start URL of the run, with a 20-second timeout and a 5 MB cap per page. Each site gets at most 2 page requests per page it may deliver (maxPagesPerSite, 20 on the free plan) plus 20, so failing links cannot make a crawl run away."}],"mcpExamplePrompt":"Crawl https://crawlee.dev/js/docs/introduction (at most 10 pages) and give me the pages as markdown plus the llms.txt file.","related":["url-to-markdown-summary","rag-web-search","ai-web-scraper","sitemap-url-extractor","website-change-monitor"],"urls":{"site":"https://www.sfoln.com/actors/website-to-markdown/","siteJson":"https://www.sfoln.com/actors/website-to-markdown.json","siteMarkdown":"https://www.sfoln.com/actors/website-to-markdown.md","store":"https://apify.com/sfoln/website-to-markdown","storeMarkdown":"https://apify.com/sfoln/website-to-markdown.md","runSync":"https://api.apify.com/v2/actors/sfoln~website-to-markdown/run-sync-get-dataset-items","runAsync":"https://api.apify.com/v2/actors/sfoln~website-to-markdown/runs","mcp":"https://mcp.apify.com?tools=sfoln/website-to-markdown"},"publisher":{"name":"SFOLN LLC","apifyUsername":"sfoln","url":"https://www.sfoln.com","apifyProfile":"https://apify.com/sfoln","email":"contact@sfoln.com"},"generatedAt":"2026-10-22T04:28:58.477Z"}