# RAG Web Search – Google + LLM-Ready Markdown (`sfoln/rag-web-search`)

Search Google and get the top results as clean, LLM-ready markdown in one call: title, URL, site, date, word and token counts per page, plus the results page (answer box, People Also Ask). Built for RAG pipelines and AI agents. $3 per 1,000 searches + $1.50 per 1,000 pages.

- Family: Web toolkit for agents
- Categories: AI, Agents, Developer tools
- Availability: on request (private on Apify). Email contact@sfoln.com with subject "Access request: RAG Web Search – Google + LLM-Ready Markdown (sfoln/rag-web-search)" before calling; calls fail until it is public.
- JSON: https://www.sfoln.com/actors/rag-web-search.json
- MCP: https://mcp.apify.com?tools=sfoln/rag-web-search
- REST (sync): POST https://api.apify.com/v2/actors/sfoln~rag-web-search/run-sync-get-dataset-items
- REST (async): POST https://api.apify.com/v2/actors/sfoln~rag-web-search/runs

Search Google and read the top results as clean, LLM-ready markdown in one call. For every query you get one row per result page with its main content as markdown (menus, ads and page chrome removed, tables kept), title, site name, publication date, word count and token estimate, plus one summary row with the Google results page: organic results, answer box, knowledge panel, People Also Ask and related searches. Built for RAG pipelines, research agents and AI assistants that need fresh web context with sources. $3 per 1,000 searches + $1.50 per 1,000 pages read; pages that cannot be read are never charged.

## Price per delivered record

| Event | Price | Per 1,000 | Charged for |
| --- | --- | --- | --- |
| Web page (`page`) | $0.0015 | $1.5 / 1,000 | Charged once per search result page delivered with readable content as markdown. |
| Search query (`search`) | $0.003 | $3 / 1,000 | Charged once per query that Google answered (one search credit), with its results page in the summary row. |

Pay-per-event, USD, billed by Apify; no platform usage charge. Only records with status `ok` are charged. Cap a run with `maxTotalChargeUsd`. Apify free plan: up to 10 per run.

## Run

```bash
curl -X POST "https://api.apify.com/v2/actors/sfoln~rag-web-search/run-sync-get-dataset-items" \
  -H "Authorization: Bearer $APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"queries":["what is web scraping"],"topN":2}'
```

MCP client configuration:

```json
{
  "mcpServers": {
    "apify": {
      "type": "http",
      "url": "https://mcp.apify.com?tools=sfoln/rag-web-search"
    }
  }
}
```

Example prompt: Search the web for "what is retrieval augmented generation", read the top 3 results and summarize them with sources.

## Input

Search queries. Each query gives one row per result page read (markdown) plus one summary row with the search results.

| Field | Type | Default | Meaning |
| --- | --- | --- | --- |
| `queries` (required) | array | — | Search queries: Google search queries, one per line (Google operators such as site: and quotes work). Accounts on the Apify free plan can process up to 10 queries per run. |
| `topN` | integer | `3` | Pages per query: Top organic results to read as markdown (1–10). A result that cannot be read is replaced by the next one. |
| `country` | string | `"us"` | Country: 2-letter country code for the Google search (gl), e.g. us, gb, de. |
| `language` | string | `"en"` | Language: Interface language code for the Google search (hl), e.g. en, de, pt-br. |
| `timeRange` | string | `"any"` | Time range: Only results from the past hour, day, week, month or year. One of `"any"`, `"hour"`, `"day"`, `"week"`, `"month"`, `"year"`. |
| `maxCharsPerPage` | integer | `20000` | Max characters per page: Markdown kept per page (1,000–200,000 characters), cut at a paragraph boundary. About 4 characters make one LLM token. |
| `maxQueries` | integer | `1000` | Max queries: Safety cap on how many queries are processed in this run; the rest are ignored and counted in the OUTPUT record. |
| `concurrency` | integer | `2` | Concurrency: Queries processed in parallel (1–5); each query reads its pages in parallel. |
| `includeRaw` | boolean | `false` | Include raw search response: Attach the upstream search response to each query's summary row (larger output). |

Example input:

```json
{
  "queries": [
    "what is web scraping"
  ],
  "topN": 2
}
```

## Output

One record per input item with `status` (`ok` is charged; `not_found`, `error`, `skipped_*` are free) and `charged`. The run stores an `OUTPUT` summary record.

| Field | Type | Meaning |
| --- | --- | --- |
| `input` | string | object | Input: The search query this record belongs to. |
| `recordType` | string | Record type: page = one search result page (charged when contentStatus is ok); summary = one record per query with the search results (charged as the search). |
| `position` | integer | Position: page rows with content: 1-based position among the pages delivered for the query. |
| `query` | string | null | Query: The search query. |
| `rank` | integer | null | Rank: page: position of the page in Google's organic results. |
| `url` | string | null | URL: page: the result URL. |
| `finalUrl` | string | null | Final URL: page: the URL that answered, after redirects. |
| `title` | string | null | Title: page: the article headline (else the search result title). |
| `siteName` | string | null | Site name: page: Open Graph site name or the publisher. |
| `domain` | string | null | Domain |
| `publishedAt` | string | null | Published at: page: ISO 8601 publication time from the page (meta, JSON-LD), else estimated from the date Google shows. |
| `language` | string | null | Language: page: the page's declared language (BCP 47) when it has one; summary: the interface language used for the search. |
| `snippet` | string | null | Snippet: page: the snippet Google shows for the result. |
| `markdown` | string | null | Markdown: page: the main content as GitHub-flavoured markdown, at most maxCharsPerPage characters; null when contentStatus is not ok. |
| `markdownTruncated` | boolean | null | Markdown truncated: page: true when the content was cut at maxCharsPerPage. |
| `wordCount` | integer | null | Word count: page: words of the whole extracted article, before the maxCharsPerPage cut (see markdownTruncated). |
| `tokenEstimate` | integer | null | Token estimate: page: rough LLM token count of the markdown (about 4 characters per token). |
| `contentStatus` | string | null | Content status: page: ok \| blocked \| js_required \| not_html \| too_large \| error. Only ok pages are charged. |
| `country` | string | null | Country: summary: country code used for the search. |
| `timeRange` | string | null | Time range: summary: Google time filter (qdr:h, qdr:d, qdr:w, qdr:m, qdr:y) or null. |
| `correctedQuery` | string | null | Corrected query: summary: the query Google searched when it differs from the input. |
| `organicResults` | integer | null | Organic results: summary: organic results Google returned. |
| `pagesWithContent` | integer | null | Pages with content: summary: page rows delivered (and charged). |
| `pagesFailed` | integer | null | Pages failed: summary: pages read without usable content (uncharged rows). |
| `duplicatePages` | integer | null | Duplicate pages: summary: pages whose content repeated an earlier result (not delivered, not charged). |
| `results` | array | Search results: summary: the organic results [{ rank, title, url, domain, snippet, date, contentStatus }]; contentStatus duplicate = same content as an earlier page, not_fetched = not needed. |
| `answerBox` | object | null | Answer box: summary: Google's answer box { title, answer, snippet, url, date }. |
| `knowledgeGraph` | object | null | Knowledge graph: summary: Google's knowledge panel. |
| `peopleAlsoAsk` | array | People also ask: summary: [{ question, snippet, title, url }]. |
| `relatedSearches` | array | Related searches: summary: related query strings. |
| `resultsDelivered` | integer | null | Pages delivered: summary: page rows delivered (and charged) for this query. |
| `stopReason` | string | null | Stop reason: summary: limit (topN pages read) \| end (no more results) \| budget \| free_plan \| aborted. |
| `status` | string | Status: ok \| not_found \| error \| skipped_budget \| skipped_free_plan \| skipped_unmetered. Page rows without content have status error and charged false. |
| `charged` | boolean | Charged: page rows: true when the page was charged (contentStatus ok); summary: true when the search was charged. |
| `chargedEvent` | string | Charged event: page on page rows; search on summary rows. |
| `chargedCount` | integer | Events charged: Events actually charged when a charge came back short (present only then). |
| `source` | string | Source: serper+web: Google results through the Serper API, pages fetched by the Actor. |
| `fetchedAt` | string | Fetched at: ISO timestamp. |
| `error` | object | Error: { code, message, status? } when status is not ok. Page codes: blocked, js_required, not_html, unexpected_type, too_large, thin_content, extract_failed, not_found, http_error, timeout, network, dns_not_found, tls_error; search codes: rate_limit, timeout, server, client, invalid_response; runtime codes: budget_exhausted, free_plan_limit, unmetered, aborted, charge_failed, record_too_large, invalid_record, dataset_push_failed. |
| `warnings` | array | Warnings: Notes about this record (budget or free-plan caps, no results). |
| `raw` | object | null | Raw search response: summary: the upstream search response (only with includeRaw). |

Example record:

```json
[
  {
    "input": "what is web scraping",
    "recordType": "page",
    "position": 1,
    "query": "what is web scraping",
    "rank": 1,
    "url": "https://en.wikipedia.org/wiki/Web_scraping",
    "finalUrl": "https://en.wikipedia.org/wiki/Web_scraping",
    "title": "Web scraping",
    "siteName": "Wikipedia",
    "domain": "en.wikipedia.org",
    "publishedAt": "2005-09-17T18:57:30.000Z",
    "language": "en",
    "snippet": "Web scraping is data scraping used for extracting data from websites.",
    "markdown": "# Web scraping\n\n**Web scraping**, **web harvesting**, or **web data extraction** is data scraping used for extracting data from websites. …",
    "markdownTruncated": true,
    "wordCount": 3071,
    "tokenEstimate": 5005,
    "contentStatus": "ok",
    "status": "ok",
    "charged": true,
    "chargedEvent": "page",
    "source": "serper+web",
    "fetchedAt": "2026-10-14T10:00:03.000Z"
  },
  {
    "input": "what is web scraping",
    "recordType": "page",
    "query": "what is web scraping",
    "rank": 3,
    "url": "https://www.example-news.com/what-is-web-scraping",
    "finalUrl": "https://www.example-news.com/what-is-web-scraping",
    "title": "What is web scraping? A beginner's guide",
    "siteName": null,
    "domain": "example-news.com",
    "publishedAt": null,
    "language": null,
    "snippet": "Learn what web scraping is and how it works.",
    "markdown": null,
    "markdownTruncated": false,
    "wordCount": 0,
    "tokenEstimate": 0,
    "contentStatus": "blocked",
    "status": "error",
    "charged": false,
    "source": "serper+web",
    "fetchedAt": "2026-10-14T10:00:03.000Z",
    "error": {
      "code": "blocked",
      "message": "The site answered with a bot check or an access-denied page."
    }
  },
  {
    "input": "what is web scraping",
    "query": "what is web scraping",
    "country": "us",
    "language": "en",
    "timeRange": null,
    "correctedQuery": null,
    "organicResults": 9,
    "pagesWithContent": 2,
    "pagesFailed": 1,
    "duplicatePages": 0,
    "results": [
      {
        "rank": 1,
        "title": "Web scraping - Wikipedia",
        "url": "https://en.wikipedia.org/wiki/Web_scraping",
        "domain": "en.wikipedia.org",
        "snippet": "Web scraping is data scraping used for extracting data from websites.",
        "date": null,
        "contentStatus": "ok"
      },
      {
        "rank": 2,
        "title": "What is web scraping?",
        "url": "https://blog.apify.com/what-is-web-scraping/",
        "domain": "blog.apify.com",
        "snippet": "Web scraping is the process of automatically extracting data from a website.",
        "date": "Oct 15, 2024",
        "contentStatus": "ok"
      },
      {
        "rank": 3,
        "title": "What is web scraping? A beginner's guide",
        "url": "https://www.example-news.com/what-is-web-scraping",
        "domain": "example-news.com",
        "snippet": "Learn what web scraping is and how it works.",
        "date": null,
        "contentStatus": "blocked"
      }
    ],
    "answerBox": null,
    "knowledgeGraph": null,
    "peopleAlsoAsk": [
      {
        "question": "Is web scraping legal?",
        "snippet": "…",
        "title": "…",
        "url": "https://www.example.com/web-scraping-legal"
      }
    ],
    "relatedSearches": [
      "web scraping tools",
      "web scraping python"
    ],
    "recordType": "summary",
    "resultsDelivered": 2,
    "stopReason": "limit",
    "status": "ok",
    "charged": true,
    "chargedEvent": "search",
    "source": "serper+web",
    "fetchedAt": "2026-10-14T10:00:04.000Z"
  }
]
```

## FAQ

**What does contentStatus mean?**

ok: the page was read and its main content is in markdown (charged as one page). blocked: the site answered with a bot check, an access-denied page or a rate limit. js_required: the page builds its content with JavaScript (no browser is used). not_html: a PDF, document or media file. too_large: more than 5 MB of HTML. error: the page failed to load or had too little text. Pages that are not ok are written as uncharged rows, and the next search result is read in their place (one extra round at most).

**How is the markdown extracted?**

Each page is parsed with Mozilla Readability on a full DOM, with links made absolute; its output is used only when it keeps the opening paragraphs and at least 85% of the text of the page's main region, otherwise a second extractor that keeps every section is used (also for very large pages, over 800 KB of HTML, 10,000 elements or 128 levels of nesting). Navigation, ads, share buttons and other page chrome are removed, tables stay GitHub-flavoured markdown tables, and the text is cut at maxCharsPerPage on a paragraph boundary. tokenEstimate counts roughly 4 characters per token.

**Can an AI agent pay for this per call?**

Yes. It runs as a standard pay-per-event Actor, so agents using the Apify MCP server can call it and pay per search and per page, with maxTotalChargeUsd as a hard cap. A run takes a few seconds to start; one query with three pages usually finishes in 5–15 seconds.


## Related

- [URL to Markdown & Article Extractor](https://www.sfoln.com/actors/url-to-markdown-summary.md) (`sfoln/url-to-markdown-summary`)
- [Google Search Results Scraper](https://www.sfoln.com/actors/google-search-results.md) (`sfoln/google-search-results`)
- [RSS Feed Reader – Blogs, News & Newsletters](https://www.sfoln.com/actors/rss-feed-reader.md) (`sfoln/rss-feed-reader`)
- [Website to Markdown Crawler (llms.txt)](https://www.sfoln.com/actors/website-to-markdown.md) (`sfoln/website-to-markdown`)
- [AI Web Scraper – Any URL to Structured JSON](https://www.sfoln.com/actors/ai-web-scraper.md) (`sfoln/ai-web-scraper`)
- [Sitemap URL Extractor](https://www.sfoln.com/actors/sitemap-url-extractor.md) (`sfoln/sitemap-url-extractor`)
- [Google News Scraper](https://www.sfoln.com/actors/google-news-search.md) (`sfoln/google-news-search`)

Publisher: SFOLN LLC, contact@sfoln.com. Catalogue: https://www.sfoln.com/catalog.json. Agent guide: https://www.sfoln.com/agents.md.