> For the complete documentation index, see [llms.txt](/llms.txt) # Smart Scrape API Intelligently scrape any URL using cascading strategies that automatically escalate from fast HTTP fetching to headless browsers and captcha solving as needed. Specify output formats to receive HTML, markdown, screenshots, PDFs, or extracted links, all in a single request. **Endpoint** - Method: `POST` - Path: `/smart-scrape` - Auth: `token` query parameter (`?token=`) - Content-Type: `application/json` - Response: `application/json` - A Browserless API token from your [account dashboard](https://browserless.io/account/) ## Quickstart **cURL:** ```sh curl --request POST \ --url 'https://production-sfo.browserless.io/smart-scrape?token=YOUR_API_TOKEN_HERE' \ --header 'Content-Type: application/json' \ --data '{ "url": "https://news.ycombinator.com/", "formats": ["html", "markdown", "links"] }' ``` **Javascript:** ```js const TOKEN = "YOUR_API_TOKEN_HERE"; const url = `https://production-sfo.browserless.io/smart-scrape?token=${TOKEN}`; const headers = { "Content-Type": "application/json" }; const data = { url: "https://news.ycombinator.com/", formats: ["html", "markdown", "links"] }; const smartScrape = async () => { const response = await fetch(url, { method: 'POST', headers: headers, body: JSON.stringify(data) }); const result = await response.json(); console.log(result); }; smartScrape(); ``` **Python:** ```python import requests TOKEN = "YOUR_API_TOKEN_HERE" url = f"https://production-sfo.browserless.io/smart-scrape?token={TOKEN}" headers = { "Content-Type": "application/json" } data = { "url": "https://news.ycombinator.com/", "formats": ["html", "markdown", "links"] } response = requests.post(url, headers=headers, json=data) result = response.json() print(result) ``` **Response** ```json { "ok": true, "statusCode": 200, "content": "...", "contentType": "text/html; charset=utf-8", "headers": { "content-type": "text/html; charset=utf-8", "cache-control": "private; max-age=0" }, "strategy": "http-fetch", "attempted": ["http-fetch"], "message": null, "actions": null, "screenshot": null, "pdf": null, "markdown": "# Hacker News\n\n[new](newest) | [past](front) | [comments](newcomments) | [ask](ask) | [show](show) | [jobs](jobs) | [submit](submit)\n\n1. [Motorola GrapheneOS devices will be bootloader unlockable/relockable](https://grapheneos.social/...)...", "rawText": null, "links": [ "https://news.ycombinator.com/news", "https://news.ycombinator.com/newest", "https://news.ycombinator.com/front" ], "metadata": { "title": "Hacker News", "description": null, "language": "en", "sourceURL": "https://news.ycombinator.com/", "statusCode": 200 } } ``` ## How it works The Smart Scrape API uses a cascading strategy pipeline to fetch content in the most efficient way possible. It starts with the fastest, cheapest approach and automatically escalates to heavier strategies only when needed: 1. **Fast HTTP fetch**: Makes a lightweight HTTP request that mimics a real browser's network fingerprint. This handles the majority of static and server-rendered sites in under 2 seconds. 2. **Proxied HTTP fetch**: If the initial request is blocked (e.g., by IP detection), the same request is retried through the selected proxy network (residential by default, or datacenter if `proxy: "datacenter"` is set in the request body). 3. **Headless browser**: If the page requires JavaScript rendering (single-page apps, client-rendered content), a full stealth browser is launched to render the page. 4. **Browser + captcha solving**: If a captcha or bot detection challenge is encountered, the browser automatically detects and solves it (supports reCAPTCHA, Cloudflare Turnstile, and others). The pipeline stops as soon as a strategy succeeds. The `strategy` field in the response tells you which approach was used, and the `attempted` array shows the full sequence of strategies tried. > **Captcha handling scope** > Smart Scrape only solves captchas that **gate access to the page itself** — for example, a Cloudflare Turnstile interstitial or a reCAPTCHA that blocks the page from loading. In these cases the browser solves the challenge automatically so the underlying content can be returned. > > Captchas that are **embedded in a form on the page** (e.g., a reCAPTCHA next to a "Submit" button on a contact or signup form) are not solved. Actions can interact with ordinary form controls, but they do not solve embedded form captchas. If you need to submit a form behind a captcha, use [BrowserQL](/browserql/start) with the [`solve` mutation](/browserql/bot-detection/solving-captchas) instead. ## Request body | Field | Type | Required | Default | Description | |---|---|---|---|---| | `url` | `string` | Yes | - | The URL to scrape. Must be `http://` or `https://`. | | `formats` | `string[]` | No | `["html"]` | Output formats to include. Options: `"html"`, `"markdown"`, `"rawText"`, `"screenshot"`, `"pdf"`, `"links"`. | | `proxy` | `string` | No | `"residential"` | Proxy network to route the scrape through: `"residential"` (6 units/MB) or `"datacenter"` (2 units/MB). | | `onlyMainContent` | `boolean` | No | `false` | Removes `nav`, `footer`, `aside`, `[role="navigation"]`, `script`, `style`, and `noscript` elements. Browserless defaults to `false`; Firecrawl's equivalent defaults to `true`. | | `includeTags` | `string[]` | No | `[]` | Keeps HTML webpage content matching up to 100 CSS selectors. Malformed entries are ignored; if no selector matches, unfiltered content is returned. It can't be combined with `excludeTags` or `onlyMainContent`. | | `excludeTags` | `string[]` | No | `[]` | Removes HTML webpage content matching up to 100 CSS selectors. Malformed selectors are ignored. It can't be combined with `includeTags`. | | `headers` | `object` | No | `{}` | Adds request headers sent to the target. Unsafe transport, proxy, authentication, and cookie headers are removed. | | `waitFor` | `number` | No | `0` | Waits after page load before reading content. Values are clamped to 0–30,000 milliseconds, and any positive value forces a browser strategy. | | `actions` | `object[]` | No | - | Sequential browser actions to run before the final page is captured. A non-empty array forces a browser strategy. | ## Actions Use `actions` to interact with a rendered page before Smart Scrape returns its final content. Actions run in submission order after page-level anti-bot checks have completed. The first failed action aborts the sequence, returns `ok: false`, and discards partial action results. | Type | Fields | Behavior | |---|---|---| | `wait` | Exactly one of `milliseconds` or `selector`; optional `timeout` | Wait for a duration or for a visible selector. | | `click` | `selector`; optional `timeout` | Click the first matching element. | | `write` | `selector`, `text`; optional `timeout` | Type into an element. A trailing `\n` sends Enter. | | `scroll` | Optional `selector`; optional `timeout` | Scroll the matching element into view, or scroll through the page when omitted. | | `scrape` | None | Save the current page URL and HTML in `actions.scrapes`. | | `executeJavascript` | `script`; optional `timeout` | Evaluate JavaScript in the page and save its scalar return value. | A request may contain at most 20 actions. Action timeouts default to 10 seconds and cannot exceed 30 seconds. Every action and the final network settle also share the request's overall `timeout` budget. `actions: []` is a no-op: it does not force a browser strategy, and the response contains `actions: null`. Combined action results are limited to 25 MiB of serialized UTF-8 JSON. Exceeding this limit aborts the sequence with `ok: false` and `actions: null`; results are not truncated, and later actions do not run. This bounds accumulated results, not the browser's memory while producing an individual result. This example dismisses a cookie banner, waits for the page to update, scrolls to a result, and captures the intermediate page: ```json { "url": "https://example.com/products", "formats": ["html", "links", "screenshot"], "actions": [ { "type": "click", "selector": "button[data-accept-cookies]" }, { "type": "wait", "selector": "main[data-ready]" }, { "type": "scroll", "selector": "#featured-product" }, { "type": "scrape" } ] } ``` The `scrape` action records the URL and HTML at that exact point. After the complete sequence, Smart Scrape waits for the page to settle and then re-reads the final status, headers, content type, HTML, and URL. Relative links and top-level screenshot/PDF formats therefore describe the final page after navigation or mutation. The final URL is used internally to resolve relative links; it is not a separate response field. `metadata.sourceURL` remains the requested URL, while each `actions.scrapes[].url` identifies the page at that particular `scrape` action. `executeJavascript` runs in the page context with the same network restrictions as BrowserQL. It cannot access Node.js globals or browser-level credential configuration; ordinary page APIs remain available. Scripts are limited to 10,000 characters. > **Firecrawl action differences** > Smart Scrape does not support a `press` action; append `\n` to `write.text` to send Enter. It also does not support mid-sequence `screenshot` or `pdf` actions. Add those values to top-level `formats` to capture the final page. ```json { "actions": { "scrapes": [ { "url": "https://example.com/products", "html": "..." } ], "javascriptReturns": [ { "type": "string", "value": "ready" }, { "type": "null", "value": null } ] } } ``` ## Output formats The `formats` array controls what data is returned. The `content` field contains raw HTML for webpages, parsed JSON for API endpoints, or extracted plain text for PDF targets. Additional formats populate their respective response fields. ### Markdown Converts the page content to clean markdown, stripping scripts, styles, and non-visible elements. **JSON body:** ```json { "url": "https://news.ycombinator.com/", "formats": ["markdown"] } ``` **cURL:** ```sh curl -s -X POST "https://production-sfo.browserless.io/smart-scrape?token=YOUR_API_TOKEN_HERE" -H "Content-Type: application/json" -d '{"url":"https://news.ycombinator.com/","formats":["markdown"]}' ``` **Response:** ```json { "ok": true, "statusCode": 200, "content": "...", "markdown": "# Hacker News\n\n[new](newest) | [past](front) | [comments](newcomments)...", "screenshot": null, "pdf": null, "links": null, "strategy": "http-fetch", "attempted": ["http-fetch"], "message": null } ``` ### Raw text Returns DOM text with `script`, `style`, and `noscript` elements removed and whitespace collapsed. PDF targets return extracted PDF text. Request it with `"formats": ["rawText"]`; the result is returned in the `rawText` response field. ### Screenshot Returns a full-page screenshot as a base64-encoded PNG string. Including `"screenshot"` in formats forces a headless browser to be used. **JSON body:** ```json { "url": "https://news.ycombinator.com/", "formats": ["screenshot"] } ``` **cURL:** ```sh curl -s -X POST "https://production-sfo.browserless.io/smart-scrape?token=YOUR_API_TOKEN_HERE" -H "Content-Type: application/json" -d '{"url":"https://news.ycombinator.com/","formats":["screenshot"]}' ``` **Response:** ```json { "ok": true, "statusCode": 200, "content": "...", "screenshot": "iVBORw0KGgoAAAANSUhEUgAA...", "pdf": null, "markdown": null, "links": null, "strategy": "browser", "attempted": ["browser"], "message": null } ``` ### PDF Returns the page as a base64-encoded PDF string. Like `"screenshot"`, including `"pdf"` forces a headless browser. **JSON body:** ```json { "url": "https://news.ycombinator.com/", "formats": ["pdf"] } ``` **cURL:** ```sh curl -s -X POST "https://production-sfo.browserless.io/smart-scrape?token=YOUR_API_TOKEN_HERE" -H "Content-Type: application/json" -d '{"url":"https://news.ycombinator.com/","formats":["pdf"]}' ``` **Response:** ```json { "ok": true, "statusCode": 200, "content": "...", "pdf": "JVBERi0xLjQKMSAwIG9iago8PA...", "screenshot": null, "markdown": null, "links": null, "strategy": "browser", "attempted": ["browser"], "message": null } ``` ### Links Extracts all links (``) from the page, resolves relative URLs to absolute, and filters to `http://` and `https://` links only. **JSON body:** ```json { "url": "https://news.ycombinator.com/", "formats": ["links"] } ``` **cURL:** ```sh curl -s -X POST "https://production-sfo.browserless.io/smart-scrape?token=YOUR_API_TOKEN_HERE" -H "Content-Type: application/json" -d '{"url":"https://news.ycombinator.com/","formats":["links"]}' ``` **Response:** ```json { "ok": true, "statusCode": 200, "content": "...", "links": [ "https://news.ycombinator.com/news", "https://news.ycombinator.com/newest", "https://news.ycombinator.com/front", "https://grapheneos.social/@GrapheneOS/116160393783585567" ], "screenshot": null, "pdf": null, "markdown": null, "strategy": "http-fetch", "attempted": ["http-fetch"], "message": null } ``` ## Content shaping For HTML webpages, use `onlyMainContent` to remove `nav`, `footer`, `aside`, `[role="navigation"]`, `script`, `style`, and `noscript` elements. Use `includeTags` when you want only specific CSS selector matches, or `excludeTags` when you want the full page minus selected elements. Each selector list accepts up to 100 entries. Malformed selector entries are ignored and do not return HTTP 400. Valid `includeTags` matches are still kept when another entry is malformed; Smart Scrape returns unfiltered content only when no include selector matches. Content shaping affects DOM-derived webpage outputs; parsed JSON `content` and parsed PDF output are unchanged. ```json { "url": "https://example.com/", "formats": ["html", "markdown", "rawText", "links"], "onlyMainContent": true, "excludeTags": ["nav", ".ad"] } ``` The following combinations return HTTP 400: - `includeTags` with `excludeTags`: `"includeTags" and "excludeTags" are mutually exclusive` - `includeTags` with `onlyMainContent: true`: `"includeTags" and "onlyMainContent" are mutually exclusive` `waitFor` is measured in milliseconds after page load and is clamped to the `[0, 30000]` range. Any positive value forces a browser strategy because an HTTP fetch can't honor a post-load delay. The `headers` object sends custom request headers to the target. Browserless first drops `host`, `authorization`, `proxy-authorization`, `cookie`, `set-cookie`, `x-forwarded-for`, `x-real-ip`, and `forwarded`. Among the remaining headers, invalid names, non-string values, and values containing CR, LF, or NUL characters return HTTP 400. Custom header names and values can total up to 65,536 bytes. > **Note** > Content shaping affects `content`, `markdown`, `rawText`, and `links`. It doesn't affect `screenshot` or `pdf`; those formats always capture the full rendered page. ## PDF targets A PDF target is a URL that points directly to an existing PDF. This is different from requesting the `"pdf"` output format above, which renders a webpage as a new PDF. Smart Scrape recognizes a PDF when the response uses `Content-Type: application/pdf`. For responses with a missing content type or `application/octet-stream`, the final URL path must end in `.pdf`. The body must also begin with the `%PDF-` signature. Recognized PDFs handled by the `http-fetch` and `http-proxy` strategies have their plain text extracted into `content`. When `"markdown"` is requested, `markdown` contains the extracted text with pages separated by blank lines. Because Smart Scrape does not extract links from inside PDFs, `links` is `null` for PDF targets. Password-protected PDFs, corrupt PDFs, PDFs larger than 25 MiB, PDFs over 10,000 pages, and PDFs whose extracted text exceeds 25 MiB return `ok: false` with an explanatory `message`. Requests that force a browser strategy remain unchanged and do not use PDF-target parsing. This includes requests for `"screenshot"` or `"pdf"` output and requests using a `profile`. A PDF target combined with a non-empty `actions` array returns a controlled failure because browser actions require an HTML page; it never returns Chromium's PDF-viewer HTML as scraped content. ## Response fields The response examples above are abbreviated. Every response includes all fields listed below. | Field | Type | Description | |---|---|---| | `ok` | `boolean` | Whether the scrape succeeded. | | `statusCode` | `number \| null` | The HTTP status code from the target site, or `null` on network errors. | | `content` | `string \| object \| null` | Page content as an HTML string, extracted PDF text, or a parsed JSON object if the target returns `application/json`. `null` on failure. | | `contentType` | `string \| null` | The content type of the scraped page. | | `headers` | `object` | HTTP response headers from the target site. | | `strategy` | `string` | The strategy that produced the result (or was being attempted on failure). | | `attempted` | `string[]` | All strategies attempted, in order. | | `message` | `string \| null` | Error message on failure, `null` on success. | | `actions` | `object \| null` | Ordered `scrapes` and `javascriptReturns` collected by successful actions. `null` when actions are omitted, empty, or the scrape fails. | | `screenshot` | `string \| null` | Base64-encoded PNG screenshot, when `"screenshot"` is in `formats`. | | `pdf` | `string \| null` | Base64-encoded PDF, when `"pdf"` is in `formats`. | | `markdown` | `string \| null` | Markdown conversion of a webpage, or page-separated extracted text for a PDF target, when `"markdown"` is in `formats`. | | `links` | `string[] \| null` | Extracted links, when `"links"` is in `formats`. Always `null` for PDF targets. | | `rawText` | `string \| null` | DOM text with scripts/styles removed and whitespace collapsed, or extracted PDF text, when `"rawText"` is in `formats`. | | `metadata` | `object \| null` | Page metadata on success, or `null` on failure. Contains `title`, `description`, `language`, `sourceURL` (the requested URL, not the final URL after redirects), and `statusCode`. | ## JSON auto-parsing When the target URL returns JSON content (e.g., an API endpoint with `Content-Type: application/json`), the `content` field will contain the parsed JSON object rather than a raw string: ```json { "ok": true, "statusCode": 200, "content": { "userId": 1, "id": 1, "title": "Example post title", "body": "Example post body..." }, "contentType": "application/json; charset=utf-8", "strategy": "http-fetch", "attempted": ["http-fetch"], "message": null } ``` ## Error handling On failure, the response still returns HTTP 200 with `ok: false` and a `message` describing the error: ```json { "ok": false, "statusCode": null, "content": null, "contentType": null, "headers": {}, "strategy": "browser-captcha", "attempted": ["http-fetch", "http-proxy", "browser", "browser-captcha"], "message": "Captcha was detected but could not be solved", "actions": null, "screenshot": null, "pdf": null, "markdown": null, "links": null, "rawText": null, "metadata": null } ``` ## Using a profile Scrape authenticated pages by passing a saved [profile](/baas/features/authenticated-profiles) via the `?profile=` query parameter. The browser loads the profile's cookies, `localStorage`, and `IndexedDB` before navigating, so the page is accessed as the logged-in user. **cURL:** ```sh curl --request POST \ --url 'https://production-sfo.browserless.io/smart-scrape?token=YOUR_API_TOKEN_HERE&profile=acme-prod' \ --header 'Content-Type: application/json' \ --data '{ "url": "https://app.example.com/dashboard", "formats": ["html", "markdown"] }' ``` **Javascript:** ```js const TOKEN = "YOUR_API_TOKEN_HERE"; const url = `https://production-sfo.browserless.io/smart-scrape?token=${TOKEN}&profile=acme-prod`; const headers = { "Content-Type": "application/json" }; const data = { url: "https://app.example.com/dashboard", formats: ["html", "markdown"] }; const smartScrape = async () => { const response = await fetch(url, { method: 'POST', headers: headers, body: JSON.stringify(data) }); const result = await response.json(); console.log(result); }; smartScrape(); ``` **Python:** ```python import requests TOKEN = "YOUR_API_TOKEN_HERE" url = f"https://production-sfo.browserless.io/smart-scrape?token={TOKEN}&profile=acme-prod" headers = { "Content-Type": "application/json" } data = { "url": "https://app.example.com/dashboard", "formats": ["html", "markdown"] } response = requests.post(url, headers=headers, json=data) result = response.json() print(result) ``` > **Tip** > Create and manage profiles via the [Authenticated Profiles](/baas/features/authenticated-profiles) workflow. The profile name is scoped to your API token — other tokens cannot access your profiles. ## Configuration options The `/smart-scrape` API supports a `timeout` query parameter to control the maximum time allowed for the scrape operation: ```http POST /smart-scrape?token=YOUR_API_TOKEN_HERE&timeout=30000 ``` The timeout value is in milliseconds and applies to each strategy attempt. If not specified, the server default timeout is used. ## FAQ & Troubleshooting
What is the Browserless Smart Scrape API? Smart Scrape uses a cascading strategy: it tries a fast HTTP fetch first and only launches a full browser if the initial request fails or returns incomplete content. This reduces cost and latency on pages that don't need a full browser.
When should I use Smart Scrape vs the /content API? Use Smart Scrape when you want automatic fallback from HTTP to browser rendering. Use /content when you know the page requires JavaScript rendering and want to skip the initial HTTP attempt.
Does Smart Scrape handle bot-protected pages? Yes. When the HTTP fetch is blocked, Smart Scrape falls back to a full browser with stealth capabilities. You can also enable proxies for additional protection.
## Next steps

REST API overview

introduction to all available endpoints

API playground

test endpoints interactively