sfoln/rag-web-search · AI, Agents, Developer tools On request
Search Google and get the top results as clean, LLM-ready markdown in one call: title, URL, site, date, word and token counts per page, plus the results page (answer box, People Also Ask). Built for RAG pipelines and AI agents. $3 per 1,000 searches + $1.50 per 1,000 pages.
Search Google and read the top results as clean, LLM-ready markdown in one call. For every query you get one row per result page with its main content as markdown (menus, ads and page chrome removed, tables kept), title, site name, publication date, word count and token estimate, plus one summary row with the Google results page: organic results, answer box, knowledge panel, People Also Ask and related searches. Built for RAG pipelines, research agents and AI assistants that need fresh web context with sources. $3 per 1,000 searches + $1.50 per 1,000 pages read; pages that cannot be read are never charged.
Price
Price per delivered record
Event
Price
Per 1,000
Charged for
Web page page
$0.0015
$1.5 / 1,000
Charged once per search result page delivered with readable content as markdown.
Search query search
$0.003
$3 / 1,000
Charged once per query that Google answered (one search credit), with its results page in the summary row.
Apify free plan: up to 10 items per run. Paid plans have no per-run cap; set maxTotalChargeUsd to cap spend.
Run
Call it from code, an agent or the Console
This API is private on Apify for now. Email us and we will publish it for you, usually within two working days. The calls below work once it is public. contact@sfoln.com
curl · synchronous run, rows in the response
curl -X POST "https://api.apify.com/v2/actors/sfoln~rag-web-search/run-sync-get-dataset-items" \
-H "Authorization: Bearer $APIFY_TOKEN" \
-H "Content-Type: application/json" \
-d '{"queries":["what is web scraping"],"topN":2}'
POST https://api.apify.com/v2/actors/sfoln~rag-web-search/runs?maxTotalChargeUsd=1
GET https://api.apify.com/v2/actor-runs/{runId} # until status is SUCCEEDED
GET https://api.apify.com/v2/datasets/{defaultDatasetId}/items?format=json
Authenticate with an Apify API token. MCP clients can instead sign in with OAuth when prompted. Example prompt for an agent: Search the web for "what is retrieval augmented generation", read the top 3 results and summarize them with sources.
Input
Input fields
Search queries. Each query gives one row per result page read (markdown) plus one summary row with the search results.
Field
Type
Default
Meaning
queries *
array
required
Search queries Google search queries, one per line (Google operators such as site: and quotes work). Accounts on the Apify free plan can process up to 10 queries per run.
topN
integer
3
Pages per query Top organic results to read as markdown (1–10). A result that cannot be read is replaced by the next one. Range 1–10.
country
string
"us"
Country 2-letter country code for the Google search (gl), e.g. us, gb, de.
language
string
"en"
Language Interface language code for the Google search (hl), e.g. en, de, pt-br.
timeRange
string
"any"
Time range Only results from the past hour, day, week, month or year. One of "any", "hour", "day", "week", "month", "year".
maxCharsPerPage
integer
20000
Max characters per page Markdown kept per page (1,000–200,000 characters), cut at a paragraph boundary. About 4 characters make one LLM token. Range 1000–200000.
maxQueries
integer
1000
Max queries Safety cap on how many queries are processed in this run; the rest are ignored and counted in the OUTPUT record. Range 1–10000.
concurrency
integer
2
Concurrency Queries processed in parallel (1–5); each query reads its pages in parallel. Range 1–5.
includeRaw
boolean
false
Include raw search response Attach the upstream search response to each query's summary row (larger output).
Example input
{
"queries": [
"what is web scraping"
],
"topN": 2
}
Output
One record per input item
Records land in the run's dataset; a synchronous call returns them directly. Each has a status (ok is charged; everything else is free) and a charged flag. The run's OUTPUT record totals items and charges.
Field
Type
Meaning
input
string | object
Input The search query this record belongs to.
recordType
string
Record type page = one search result page (charged when contentStatus is ok); summary = one record per query with the search results (charged as the search). e.g. "page"
position
integer
Position page rows with content: 1-based position among the pages delivered for the query.
query
string | null
Query The search query.
rank
integer | null
Rank page: position of the page in Google's organic results.
url
string | null
URL page: the result URL.
finalUrl
string | null
Final URL page: the URL that answered, after redirects.
title
string | null
Title page: the article headline (else the search result title).
siteName
string | null
Site name page: Open Graph site name or the publisher.
domain
string | null
Domain
publishedAt
string | null
Published at page: ISO 8601 publication time from the page (meta, JSON-LD), else estimated from the date Google shows.
language
string | null
Language page: the page's declared language (BCP 47) when it has one; summary: the interface language used for the search.
snippet
string | null
Snippet page: the snippet Google shows for the result.
markdown
string | null
Markdown page: the main content as GitHub-flavoured markdown, at most maxCharsPerPage characters; null when contentStatus is not ok.
markdownTruncated
boolean | null
Markdown truncated page: true when the content was cut at maxCharsPerPage.
wordCount
integer | null
Word count page: words of the whole extracted article, before the maxCharsPerPage cut (see markdownTruncated).
tokenEstimate
integer | null
Token estimate page: rough LLM token count of the markdown (about 4 characters per token).
contentStatus
string | null
Content status page: ok | blocked | js_required | not_html | too_large | error. Only ok pages are charged.
country
string | null
Country summary: country code used for the search.
timeRange
string | null
Time range summary: Google time filter (qdr:h, qdr:d, qdr:w, qdr:m, qdr:y) or null.
correctedQuery
string | null
Corrected query summary: the query Google searched when it differs from the input.
organicResults
integer | null
Organic results summary: organic results Google returned.
pagesWithContent
integer | null
Pages with content summary: page rows delivered (and charged).
pagesFailed
integer | null
Pages failed summary: pages read without usable content (uncharged rows).
duplicatePages
integer | null
Duplicate pages summary: pages whose content repeated an earlier result (not delivered, not charged).
results
array
Search results summary: the organic results [{ rank, title, url, domain, snippet, date, contentStatus }]; contentStatus duplicate = same content as an earlier page, not_fetched = not needed.
People also ask summary: [{ question, snippet, title, url }].
relatedSearches
array
Related searches summary: related query strings.
resultsDelivered
integer | null
Pages delivered summary: page rows delivered (and charged) for this query.
stopReason
string | null
Stop reason summary: limit (topN pages read) | end (no more results) | budget | free_plan | aborted.
status
string
Status ok | not_found | error | skipped_budget | skipped_free_plan | skipped_unmetered. Page rows without content have status error and charged false. e.g. "ok"
charged
boolean
Charged page rows: true when the page was charged (contentStatus ok); summary: true when the search was charged.
chargedEvent
string
Charged event page on page rows; search on summary rows.
chargedCount
integer
Events charged Events actually charged when a charge came back short (present only then).
source
string
Source serper+web: Google results through the Serper API, pages fetched by the Actor.
fetchedAt
string
Fetched at ISO timestamp.
error
object
Error { code, message, status? } when status is not ok. Page codes: blocked, js_required, not_html, unexpected_type, too_large, thin_content, extract_failed, not_found, http_error, timeout, network, dns_not_found, tls_error; search codes: rate_limit, timeout, server, client, invalid_response; runtime codes: budget_exhausted, free_plan_limit, unmetered, aborted, charge_failed, record_too_large, invalid_record, dataset_push_failed.
warnings
array
Warnings Notes about this record (budget or free-plan caps, no results).
raw
object | null
Raw search response summary: the upstream search response (only with includeRaw).
Example recordExample record
[
{
"input": "what is web scraping",
"recordType": "page",
"position": 1,
"query": "what is web scraping",
"rank": 1,
"url": "https://en.wikipedia.org/wiki/Web_scraping",
"finalUrl": "https://en.wikipedia.org/wiki/Web_scraping",
"title": "Web scraping",
"siteName": "Wikipedia",
"domain": "en.wikipedia.org",
"publishedAt": "2005-09-17T18:57:30.000Z",
"language": "en",
"snippet": "Web scraping is data scraping used for extracting data from websites.",
"markdown": "# Web scraping\n\n**Web scraping**, **web harvesting**, or **web data extraction** is data scraping used for extracting data from websites. …",
"markdownTruncated": true,
"wordCount": 3071,
"tokenEstimate": 5005,
"contentStatus": "ok",
"status": "ok",
"charged": true,
"chargedEvent": "page",
"source": "serper+web",
"fetchedAt": "2026-10-14T10:00:03.000Z"
},
{
"input": "what is web scraping",
"recordType": "page",
"query": "what is web scraping",
"rank": 3,
"url": "https://www.example-news.com/what-is-web-scraping",
"finalUrl": "https://www.example-news.com/what-is-web-scraping",
"title": "What is web scraping? A beginner's guide",
"siteName": null,
"domain": "example-news.com",
"publishedAt": null,
"language": null,
"snippet": "Learn what web scraping is and how it works.",
"markdown": null,
"markdownTruncated": false,
"wordCount": 0,
"tokenEstimate": 0,
"contentStatus": "blocked",
"status": "error",
"charged": false,
"source": "serper+web",
"fetchedAt": "2026-10-14T10:00:03.000Z",
"error": {
"code": "blocked",
"message": "The site answered with a bot check or an access-denied page."
}
},
{
"input": "what is web scraping",
"query": "what is web scraping",
"country": "us",
"language": "en",
"timeRange": null,
"correctedQuery": null,
"organicResults": 9,
"pagesWithContent": 2,
"pagesFailed": 1,
"duplicatePages": 0,
"results": [
{
"rank": 1,
"title": "Web scraping - Wikipedia",
"url": "https://en.wikipedia.org/wiki/Web_scraping",
"domain": "en.wikipedia.org",
"snippet": "Web scraping is data scraping used for extracting data from websites.",
"date": null,
"contentStatus": "ok"
},
{
"rank": 2,
"title": "What is web scraping?",
"url": "https://blog.apify.com/what-is-web-scraping/",
"domain": "blog.apify.com",
"snippet": "Web scraping is the process of automatically extracting data from a website.",
"date": "Oct 15, 2024",
"contentStatus": "ok"
},
{
"rank": 3,
"title": "What is web scraping? A beginner's guide",
"url": "https://www.example-news.com/what-is-web-scraping",
"domain": "example-news.com",
"snippet": "Learn what web scraping is and how it works.",
"date": null,
"contentStatus": "blocked"
}
],
"answerBox": null,
"knowledgeGraph": null,
"peopleAlsoAsk": [
{
"question": "Is web scraping legal?",
"snippet": "…",
"title": "…",
"url": "https://www.example.com/web-scraping-legal"
}
],
"relatedSearches": [
"web scraping tools",
"web scraping python"
],
"recordType": "summary",
"resultsDelivered": 2,
"stopReason": "limit",
"status": "ok",
"charged": true,
"chargedEvent": "search",
"source": "serper+web",
"fetchedAt": "2026-10-14T10:00:04.000Z"
}
]
FAQ
Questions
What does contentStatus mean?
ok: the page was read and its main content is in markdown (charged as one page). blocked: the site answered with a bot check, an access-denied page or a rate limit. js_required: the page builds its content with JavaScript (no browser is used). not_html: a PDF, document or media file. too_large: more than 5 MB of HTML. error: the page failed to load or had too little text. Pages that are not ok are written as uncharged rows, and the next search result is read in their place (one extra round at most).
How is the markdown extracted?
Each page is parsed with Mozilla Readability on a full DOM, with links made absolute; its output is used only when it keeps the opening paragraphs and at least 85% of the text of the page's main region, otherwise a second extractor that keeps every section is used (also for very large pages, over 800 KB of HTML, 10,000 elements or 128 levels of nesting). Navigation, ads, share buttons and other page chrome are removed, tables stay GitHub-flavoured markdown tables, and the text is cut at maxCharsPerPage on a paragraph boundary. tokenEstimate counts roughly 4 characters per token.
Can an AI agent pay for this per call?
Yes. It runs as a standard pay-per-event Actor, so agents using the Apify MCP server can call it and pay per search and per page, with maxTotalChargeUsd as a hard cap. A run takes a few seconds to start; one query with three pages usually finishes in 5–15 seconds.