Scrape API

Extract structured JSON from a fully rendered page using CSS selectors. Each request must include a url and an elements array that lists the selectors you want to capture.

Endpoint

• Method: POST
• Path: /scrape
• Auth: token query parameter (?token=)
• Content-Type: application/json
• Response: application/json

See the OpenAPI reference for complete details.

Prerequisites

• A Browserless API token from your account dashboard

Quickstart

cURL

curl --request POST \
  --url 'https://production-sfo.browserless.io/scrape?token=YOUR_API_TOKEN_HERE' \
  --header 'content-type: application/json' \
  --data '{
  "url": "https://browserless.io/",
  "elements": [
    {
      "selector": "h1"
    }
  ]
}'

Javascript

const TOKEN = "YOUR_API_TOKEN_HERE";
const url = `https://production-sfo.browserless.io/scrape?token=${TOKEN}`;
const headers = {
  "Cache-Control": "no-cache",
  "Content-Type": "application/json"
};

const data = {
  url: "https://browserless.io/",
  elements: [
    { selector: "h1" }
  ]
};

const scrapeContent = async () => {
  const response = await fetch(url, {
    method: 'POST',
    headers: headers,
    body: JSON.stringify(data)
  });

  const result = await response.json();
  console.log(result);
};

scrapeContent();

Python

import requests

TOKEN = "YOUR_API_TOKEN_HERE"
url = f"https://production-sfo.browserless.io/scrape?token={TOKEN}"
headers = {
    "Cache-Control": "no-cache",
    "Content-Type": "application/json"
}

data = {
    "url": "https://browserless.io/",
    "elements": [
        { "selector": "h1" }
    ]
}

response = requests.post(url, headers=headers, json=data)
result = response.json()

print(result)

Response

{
  "data": [
    {
      "results": [
        {
          "attributes": [
            { "name": "class", "value": "..." }
          ],
          "height": 120,
          "html": "Headless browser automation, without the hosting headaches",
          "left": 32,
          "text": "Headless browser automation, without the hosting headaches",
          "top": 196,
          "width": 736
        }
      ],
      "selector": "h1"
    }
  ]
}

How scraping works

The API uses document.querySelectorAll under the hood. Browserless loads the page, runs client-side JavaScript, and then waits (up to 30 seconds by default) for your selectors before scraping. Use more specific selectors to narrow down results.

Bot detection troubleshooting

If you're hitting Cloudflare challenges, "HeadlessChrome" blocks, or CAPTCHA pages, work up this ladder one step at a time. Each step adds cost or setup, so stop at the first one that gets you through. Every step also pays normal browser time (1 unit per 30 seconds).

Step	What to try	Extra cost
1	Add stealth=true to the request. See launch parameters.	None
2	Switch to the /unblock API with proxy=residential.	6 units per MB of residential proxy traffic
3	Connect over Browsers as a Service (BaaS) with solveCaptchas=true to solve CAPTCHAs that still appear.	10 units per successful solve
4	Use BrowserQL's solve mutation for full control over when and how challenges are solved.	10 units per successful solve

stealth=true alone often isn't enough for Cloudflare. If you still get a challenge with it on, move to step 2 rather than retrying. Failed solve attempts aren't charged. See unit consumption for the full cost breakdown, Bypass CAPTCHAs for the end-to-end approach, and Solving Cloudflare challenges for working code.

If scraped results are empty or missing expected elements, the site may be blocking automation. Signs include:

• Empty or missing results arrays in the response
• Partial page content missing key elements
• Different content compared to a regular browser
• Blocked requests or access denied messages

The /unblock API is specifically designed to bypass bot detection mechanisms like Datadome and passive CAPTCHAs.

Configuration options

The /scrape API supports shared request configuration options that apply across REST endpoints. In addition to elements and selectors, you can:

• Waiting for things: Wait for events, functions, selectors, or timeouts before scraping
• Navigation options: Customize navigation behavior with gotoOptions
• Rejecting undesired requests: Block resources with rejectResourceTypes and rejectRequestPattern
• Continue on error: Use bestAttempt to continue when async events fail or time out

FAQ & Troubleshooting

How does the Browserless /scrape API work?

Send a POST request with a URL and an elements array defining CSS selectors and what to extract (text, HTML, attributes). The API returns structured JSON matching your selectors.

Can I scrape multiple elements from a single page?

Yes. The elements array accepts multiple selector definitions. Each entry can target a different element and extract different properties, all returned in a single API response.

Does the /scrape API handle JavaScript-rendered content?

Yes. The API renders the page in a real headless browser, waits for JavaScript to execute, and then applies your selectors. You can also add custom wait conditions to handle lazy-loaded content.

Next steps

REST API overview

introduction to all available endpoints

API playground

test endpoints interactively