{"name":"sitemap-url-extractor","id":"sfoln/sitemap-url-extractor","title":"Sitemap URL Extractor","description":"Get every URL of a website from its XML sitemaps: robots.txt and /sitemap.xml discovery, sitemap indexes, .gz files, image, news, video and hreflang data. Filter by URL pattern or lastmod date. One JSON row per URL with lastmod, changefreq and priority. $0.20 per 1,000 URLs.","version":"0.1","status":"ready","family":{"id":"ai-agent-web-toolkit","name":"Web toolkit for agents","description":"Web search for retrieval, pages and sites converted to Markdown, AI extraction to JSON, URL metadata and safety checks, sitemaps, feeds, change monitoring and academic paper search.","actors":11},"categories":["DEVELOPER_TOOLS","SEO_TOOLS","AUTOMATION"],"availability":"on_request","seoTitle":"Sitemap URL Extractor – All URLs of a Website as JSON","seoDescription":"Extract every URL from a site's XML sitemaps: robots.txt discovery, indexes, .gz, images, news and hreflang. Filter by pattern or lastmod. $0.20 per 1,000 URLs.","summary":"Get every URL of a website from its XML sitemaps as clean JSON. Give a website (the sitemaps are found through robots.txt, then /sitemap.xml and /sitemap_index.xml) or a sitemap file, and you get one row per page URL with lastmod, changefreq, priority, the sitemap it came from and the image, news, video and hreflang data the sitemap carries. Sitemap indexes are followed, .gz files decompressed, broken or cut-off sitemaps read leniently. Built for SEO audits, crawl planning, content inventories, change tracking and AI agents that need the list of pages before they read them. $0.20 per 1,000 URLs; sites without a sitemap cost nothing.","pricing":{"model":"pay-per-event","currency":"USD","billedBy":"Apify","freePlanMaxItemsPerRun":10,"events":[{"name":"sitemap-url","title":"Sitemap URL","description":"Charged once per page URL from a site's sitemaps delivered as a dataset row.","priceUsd":0.0002}]},"input":{"description":"Website or sitemap URLs. Every site gives one dataset row per page URL plus one summary row.","required":["siteUrls"],"fields":[{"name":"siteUrls","title":"Website or sitemap URLs","type":"array","required":true,"description":"A website (its sitemaps are found through robots.txt, then /sitemap.xml and /sitemap_index.xml) or a sitemap file URL (.xml, .xml.gz or .txt; sitemap indexes are followed). One per line. Accounts on the Apify free plan can process up to 10 sites per run.","prefill":["https://apify.com"],"editor":"stringList"},{"name":"maxUrlsPerSite","title":"Max URLs per site","type":"integer","required":false,"description":"Page URLs delivered per site (1–1,000,000). Each delivered URL is one sitemap-url event.","default":1000,"prefill":10,"minimum":1,"maximum":1000000},{"name":"includePatterns","title":"Include URL patterns","type":"array","required":false,"description":"Keep only URLs matching one of these: a glob where * matches anything (\"*/blog/*\"), a plain substring (\"/products/\") or a regular expression in slashes (\"/\\/p\\/\\d+$/\"). Text in slashes is a regular expression only when it uses regex syntax (\\ ^ $ ( ) [ ] { } | + ? or .*) or flags such as /i; otherwise it is a substring, slashes included. Globs and substrings are case-insensitive. Empty = all URLs.","editor":"stringList"},{"name":"excludePatterns","title":"Exclude URL patterns","type":"array","required":false,"description":"Drop URLs matching one of these (same syntax as the include patterns).","editor":"stringList"},{"name":"lastmodSince","title":"Changed since","type":"string","required":false,"description":"Only URLs whose lastmod is on or after this date (\"2026-01-01\") or within this period (\"30 days\", \"2 weeks\", \"6 months\"). URLs without a lastmod are left out when this is set, and child sitemaps the index dates earlier are not read.","editor":"textfield"},{"name":"maxSitemaps","title":"Max sitemap files per site","type":"integer","required":false,"description":"Sitemap files read per site, indexes included (1–1,000).","default":100,"minimum":1,"maximum":1000},{"name":"maxDepth","title":"Max index depth","type":"integer","required":false,"description":"Levels of nested sitemap indexes followed below the start sitemaps (0–5).","default":3,"minimum":0,"maximum":5},{"name":"maxSites","title":"Max sites","type":"integer","required":false,"description":"Safety cap on how many URLs are processed in this run; the rest are ignored and counted in the OUTPUT record.","default":1000,"minimum":1,"maximum":10000},{"name":"concurrency","title":"Concurrency","type":"integer","required":false,"description":"Sites processed in parallel (1–10). Requests to one host are always limited to 4 per second.","default":3,"minimum":1,"maximum":10}]},"exampleInput":{"siteUrls":["https://apify.com"],"maxUrlsPerSite":10},"outputFields":[{"name":"input","type":"string | object","title":"Input","description":"The website or sitemap URL this record belongs to, as given."},{"name":"recordType","type":"string","title":"Record type","description":"sitemap-url = one page URL (charged); summary = one record per input (never charged).","example":"sitemap-url"},{"name":"position","type":"integer","title":"Position","description":"sitemap-url: 1-based position among the URLs delivered for its input."},{"name":"siteUrl","type":"string | null","title":"Site","description":"sitemap-url: the input normalized to an absolute URL."},{"name":"loc","type":"string | null","title":"URL","description":"The page URL (sitemap <loc>)."},{"name":"lastmod","type":"string | null","title":"Last modified","description":"ISO 8601 (UTC) of the sitemap's <lastmod>; null when missing or not a valid date."},{"name":"changefreq","type":"string | null","title":"Change frequency","description":"always | hourly | daily | weekly | monthly | yearly | never, as declared."},{"name":"priority","type":"number | null","title":"Priority","description":"0.0–1.0 as declared."},{"name":"sitemap","type":"string | null","title":"Sitemap","description":"The sitemap file the URL was listed in."},{"name":"images","type":"array","title":"Images","description":"Image extension: [{ loc, title, caption }]."},{"name":"news","type":"object | null","title":"News","description":"News extension: { publicationName, language, publicationDate, title, keywords }."},{"name":"videos","type":"array","title":"Videos","description":"Video extension: [{ title, description, thumbnailLoc, contentLoc, playerLoc, durationSecs, publicationDate, tags }]."},{"name":"alternates","type":"array","title":"Language alternates","description":"hreflang alternates: [{ hreflang, href }]."},{"name":"inputUrl","type":"string | null","title":"Input URL","description":"summary: the input normalized to an absolute URL."},{"name":"discoveredVia","type":"string | null","title":"Discovered via","description":"summary: input (a sitemap URL was given) | robots (robots.txt Sitemap: lines) | probe (/sitemap.xml or /sitemap_index.xml)."},{"name":"sitemapsRead","type":"integer | null","title":"Sitemaps read","description":"summary: sitemap files fetched and read, indexes included."},{"name":"sitemapsFailed","type":"integer | null","title":"Sitemaps failed","description":"summary: sitemap files that could not be fetched or read (see warnings)."},{"name":"sitemapsSkipped","type":"integer | null","title":"Sitemaps skipped","description":"summary: child sitemaps not read because of maxDepth, maxSitemaps or lastmodSince."},{"name":"duplicateUrls","type":"integer | null","title":"Duplicate URLs","description":"summary: URLs listed more than once (delivered once, never charged twice)."},{"name":"excludedByPattern","type":"integer | null","title":"Excluded by pattern","description":"summary: URLs dropped by includePatterns / excludePatterns."},{"name":"olderThanLastmodSince","type":"integer | null","title":"Older than lastmodSince","description":"summary: URLs whose lastmod is before lastmodSince."},{"name":"resultsDelivered","type":"integer | null","title":"URLs delivered","description":"summary: sitemap-url rows delivered (and charged) for this input."},{"name":"stopReason","type":"string | null","title":"Stop reason","description":"summary: limit (maxUrlsPerSite reached) | end (all sitemaps read) | budget | free_plan | deadline | aborted | error."},{"name":"status","type":"string","title":"Status","description":"ok | not_found | error | skipped_invalid | skipped_budget | skipped_free_plan | skipped_unmetered","example":"ok"},{"name":"charged","type":"boolean","title":"Charged","description":"true on every sitemap-url row (one sitemap-url event each); summary records are never charged."},{"name":"chargedEvent","type":"string","title":"Charged event","description":"sitemap-url on URL rows."},{"name":"chargedCount","type":"integer","title":"Events charged","description":"Events actually charged when a charge came back short (present only then)."},{"name":"source","type":"string","title":"Source","description":"Always \"sitemap\" (fetched from the site by the Actor)."},{"name":"fetchedAt","type":"string","title":"Fetched at","description":"ISO timestamp."},{"name":"error","type":"object","title":"Error","description":"{ code, message, provider?, status? } when status is not ok. Codes: not_a_sitemap (no sitemap found or not a sitemap), invalid_url, not_found, http_error, timeout, network, dns_not_found, blocked_url, too_large, unexpected_type, invalid_body, plus the runtime codes budget_exhausted, free_plan_limit, unmetered, aborted, charge_failed, record_too_large, invalid_record, dataset_push_failed."},{"name":"warnings","type":"array","title":"Warnings","description":"Notes about this record (sitemaps skipped or cut off, URLs without lastmod, caps)."}],"outputExample":[{"input":"https://www.theguardian.com","recordType":"sitemap-url","position":1,"siteUrl":"https://www.theguardian.com/","loc":"https://www.theguardian.com/science/2026/oct/14/example-article","lastmod":"2026-10-14T21:54:00.000Z","changefreq":"hourly","priority":null,"sitemap":"https://www.theguardian.com/sitemaps/news.xml","images":[{"loc":"https://i.guim.co.uk/img/media/example/master/5494.jpg","title":null,"caption":null}],"news":{"publicationName":"The Guardian","language":"en","publicationDate":"2026-10-14T21:51:35.000Z","title":"Example science article","keywords":"Science, Space"},"videos":[],"alternates":[],"status":"ok","charged":true,"chargedEvent":"sitemap-url","source":"sitemap","fetchedAt":"2026-10-14T10:00:00.000Z"},{"input":"https://www.theguardian.com","inputUrl":"https://www.theguardian.com/","discoveredVia":"robots","sitemapsRead":2,"sitemapsFailed":0,"sitemapsSkipped":0,"duplicateUrls":3,"excludedByPattern":0,"olderThanLastmodSince":0,"recordType":"summary","resultsDelivered":497,"stopReason":"end","status":"ok","charged":false,"source":"sitemap","fetchedAt":"2026-10-14T10:00:02.000Z"}],"faq":[{"question":"How are the sitemaps found?","answer":"For a website URL the Actor reads its robots.txt and uses every Sitemap: line; when there is none it tries /sitemap.xml and /sitemap_index.xml. Sitemap indexes are followed up to maxDepth levels and maxSitemaps files, gzip-compressed sitemaps (.xml.gz) are decompressed, and plain-text sitemaps (one URL per line) and RSS/Atom feeds used as sitemaps are read too. You can also pass a sitemap file URL directly. The summary row says how the sitemaps were found (discoveredVia) and how many files were read."},{"question":"Can I get only some of the URLs?","answer":"Yes: includePatterns and excludePatterns keep or drop URLs by glob (\"*/blog/*\"), substring or regular expression, and lastmodSince keeps only URLs changed after a date or within a period such as \"30 days\". Filtered URLs are never charged, and with lastmodSince the Actor also skips child sitemaps its index dates earlier, so large sites are read faster."},{"question":"What about very large sites?","answer":"Each sitemap file is read incrementally (up to 50,000 URLs and 50 MB per file, the limits of the sitemap protocol; less on 256 MB runs), so memory stays flat. Raise maxUrlsPerSite and maxSitemaps for full exports; the run stops cleanly at your maximum cost per run."}],"mcpExamplePrompt":"List the first 50 blog URLs from the sitemap of apify.com that changed in the last 30 days.","related":["rss-feed-reader","website-to-markdown","url-to-markdown-summary","rag-web-search","url-metadata","onpage-seo-audit"],"urls":{"site":"https://www.sfoln.com/actors/sitemap-url-extractor/","siteJson":"https://www.sfoln.com/actors/sitemap-url-extractor.json","siteMarkdown":"https://www.sfoln.com/actors/sitemap-url-extractor.md","runSync":"https://api.apify.com/v2/actors/sfoln~sitemap-url-extractor/run-sync-get-dataset-items","runAsync":"https://api.apify.com/v2/actors/sfoln~sitemap-url-extractor/runs","mcp":"https://mcp.apify.com?tools=sfoln/sitemap-url-extractor","requestAccess":"mailto:contact@sfoln.com?subject=Access%20request%3A%20Sitemap%20URL%20Extractor%20(sfoln%2Fsitemap-url-extractor)&body=Please%20publish%20sitemap-url-extractor.%20What%20we%20plan%20to%20use%20it%20for%3A"},"publisher":{"name":"SFOLN LLC","apifyUsername":"sfoln","url":"https://www.sfoln.com","apifyProfile":"https://apify.com/sfoln","email":"contact@sfoln.com"},"generatedAt":"2026-10-22T04:28:58.477Z"}