Scrape a website URL

Pull data off a live page with a single REST call, then run the same job over a WebSocket connection. After this page you'll have a working API token and know which of the two connection styles fits the job in front of you.

Steps

1. Get Your API Token

   Sign up for a free account, then copy your API token from the account dashboard.

   Every request carries the token as a ?token= query parameter. There are no other auth headers to set.

2. Scrape the Page

   Three ways to run the same job. They all reach the same managed browsers, so pick the one that fits how you already work. The tradeoffs are below.

   BAP SDK

   BAP is our own SDK, and the shortest path if you're starting fresh. It runs over BrowserQL rather than CDP, and extraction happens inside the browser, so you get back the content you asked for instead of a page to parse.

   TypeScript

   npm install @browserless.io/bap-ts

   import Browserless from "@browserless.io/bap-ts";
   
   const TOKEN = "YOUR_API_TOKEN_HERE";
   
   const browser = Browserless.connect({
     browserWSEndpoint: "wss://production-sfo.browserless.io/chromium/bql",
     token: TOKEN,
   });
   
   try {
     const page = await browser.newPage();
     await page.goto("https://example.com");
   
     const { text } = await page.text({ selector: "h1" });
     console.log(text);
   } finally {
     await browser.close();
   }

   Python

   python -m pip install bap-py

   import bap.sync_api as bap
   
   TOKEN = "YOUR_API_TOKEN_HERE"
   
   # The context manager closes the page and the browser when the block exits.
   with bap.Browserless.connect(
       browser_ws_endpoint="wss://production-sfo.browserless.io/chromium/bql",
       token=TOKEN,
   ) as browser:
       with browser.page() as page:
           page.goto("https://example.com")
           print(page.text(selector="h1")["text"])

   Output

   Example Domain

   Feeding a page to an LLM? Swap text() for markdown() and the conversion happens in the browser, so no markup crosses the wire. See extracting content.

   REST

   POST /scrape takes a URL and a list of CSS selectors, runs them against the fully rendered DOM, and returns what matched. Browserless launches and disposes of the browser for you, so there's nothing to install and nothing to clean up.

   cURL

   curl --request POST \
     --url 'https://production-sfo.browserless.io/scrape?token=YOUR_API_TOKEN_HERE' \
     --header 'Content-Type: application/json' \
     --data '{
     "url": "https://example.com",
     "elements": [{ "selector": "h1" }]
   }'

   JavaScript

   const TOKEN = "YOUR_API_TOKEN_HERE";
   
   const response = await fetch(
     `https://production-sfo.browserless.io/scrape?token=${TOKEN}`,
     {
       method: "POST",
       headers: { "Content-Type": "application/json" },
       body: JSON.stringify({
         url: "https://example.com",
         elements: [{ selector: "h1" }],
       }),
     }
   );
   
   console.log(await response.json());

   Python

   import requests
   
   TOKEN = "YOUR_API_TOKEN_HERE"
   
   response = requests.post(
       f"https://production-sfo.browserless.io/scrape?token={TOKEN}",
       json={
           "url": "https://example.com",
           "elements": [{"selector": "h1"}],
       },
   )
   
   print(response.json())

   Response

   {
     "data": [
       {
         "results": [
           {
             "html": "Example Domain",
             "text": "Example Domain"
           }
         ],
         "selector": "h1"
       }
     ]
   }

   Each result also carries the element's attributes and its position on the page. The full shape is on the /scrape reference.

   Puppeteer or Playwright

   The same job through Puppeteer or Playwright. Instead of describing the work in JSON, you drive the browser yourself over a Chrome DevTools Protocol (CDP) WebSocket, and the page stays open between commands.

   npm install puppeteer-core   # or: npm install playwright-core

   Puppeteer

   import puppeteer from "puppeteer-core";
   
   const TOKEN = "YOUR_API_TOKEN_HERE";
   
   // connect() replaces launch(). The browser runs on Browserless, not your machine.
   const browser = await puppeteer.connect({
     browserWSEndpoint: `wss://production-sfo.browserless.io?token=${TOKEN}`,
   });
   
   try {
     const page = await browser.newPage();
     await page.goto("https://example.com");
     console.log(await page.$eval("h1", (el) => el.textContent));
   } finally {
     // Always close, or the session runs until it times out and keeps burning units.
     await browser.close();
   }

   Playwright

   import { chromium } from "playwright-core";
   
   const TOKEN = "YOUR_API_TOKEN_HERE";
   
   // connectOverCDP(), not connect(). Browserless speaks CDP, not the Playwright server protocol.
   const browser = await chromium.connectOverCDP(
     `wss://production-sfo.browserless.io?token=${TOKEN}`
   );
   
   try {
     const page = await browser.newPage();
     await page.goto("https://example.com");
     console.log(await page.textContent("h1"));
   } finally {
     // Always close, or the session runs until it times out and keeps burning units.
     await browser.close();
   }

   Output

   Example Domain

Which path should you use?

All three hit the same managed browsers. They differ in who holds the browser open and how much you have to install.

Use BAP when you're writing new automation. It's our SDK over BrowserQL, and it ships stealth, CAPTCHA solving, and content extraction as methods rather than as things you assemble yourself. markdown() replaces the fetch-then-parse step that both other paths leave to you.

Use REST for stateless, one-shot work: fetch a page, grab some fields, render a PDF, take a screenshot. One request in, one JSON response out, nothing to manage.

Use Puppeteer or Playwright when you already have scripts running locally. Swapping launch() for connect() is the smallest possible change, and the rest of your code is untouched.

See the API comparison for the full breakdown, including where raw BrowserQL fits.

tip

The examples use the US West region (production-sfo). Browserless also runs in Europe. Pick the region closest to the sites you're targeting, since that's where the latency comes from. See connection URLs for the full list.

FAQ & Troubleshooting

Why am I getting a 401 Unauthorized or 403 Forbidden response?

The token is missing, mistyped, or sent in the wrong place. It goes in the URL as ?token=YOUR_API_TOKEN_HERE, not in a header and not in the JSON body. Confirm the value against your account dashboard.

The results array came back empty.

Either the selector didn't match, or the site served different content to the browser. Check the selector in your own browser's devtools first. If it matches there but not through the API, the site is likely blocking automation, so try the /unblock API or a stealth route.

Do I need puppeteer or puppeteer-core?

puppeteer-core. The full puppeteer package downloads a Chromium binary on install, which you don't need when the browser runs on Browserless. The API of the two is identical.

My script finished but the session kept running.

You didn't close the browser. Wrap the work in try/finally and call browser.close() in the finally block, as in the example above. An abandoned session stays open until it hits the timeout, and you're charged for that time.

Next steps

BAP quickstart

The SDK path, in TypeScript or Python

Scrape API reference

Every parameter the /scrape endpoint accepts

Choosing an API

Pick the right API for the job in front of you