Skip to main content

/unblock API

The /unblock API is designed to bypass bot detection mechanisms such as Datadome and other passive CAPTCHAs. It offers two main ways to access protected pages:

  • Direct data extraction: Grab the HTML content, screenshots, or cookies from protected pages with a simple REST call.
  • WebSocket endpoint generation: Get a WebSocket URL to connect your existing Puppeteer/Playwright code for custom automations.
BrowserQL Recommended

For generic and scale-intensive use-cases involving advanced bot detection, we recommend using BrowserQL instead. Its advanced stealth capabilities include human-like interactions, built-in CAPTCHA solving (Cloudflare, reCAPTCHA, and a variety of others), fingerprint masking, and GraphQL-based automation that outperforms traditional tools.

For Cloudflare turnstiles, use BrowserQL's verify mutation. For reCAPTCHA, use the solve mutation. See the BrowserQL CAPTCHA Solving Guide for details.

The /unblock API works best when combined with our Browserless Residential Proxy Service.

This API is particularly useful for developers who need to automate web interactions on sites that employ sophisticated bot detection and blocking techniques. It offers four different ways to wait for preconditions to be met before returning a response.

You can view the complete Unblock API OpenAPI specification for all properties and documentation.

Examples

The Unblock API is optimized to return only the data you request. Set fields to true for the data you need, and false for everything else to reduce execution time and resource usage.

Retrieve the HTML content of a page after bypassing bot detection:

{
"url": "https://www.example.com/",
"content": true,
"cookies": false,
"screenshot": false,
"browserWSEndpoint": false,
"ttl": 0
}

Retrieving HTML

If you'd like to retrieve the HTML of a page for scraping, Set content: true in the JSON payload. Here's a complete example with residential proxy enabled:

curl --request POST \
--url 'https://production-sfo.browserless.io/unblock?token=YOUR_API_TOKEN_HERE&proxy=residential' \
--header 'Content-Type: application/json' \
--data '{
"url": "https://example.com/",
"content": true
}'

Response:

{
"content": "<!DOCTYPE html><html>...</html>",
"cookies": [],
"screenshot": null,
"browserWSEndpoint": null
}

Process the returned HTML with scraping libraries like Scrapy or Beautiful Soup.

Getting a WebSocket Endpoint

The /unblock API can bypass bot detection, then return a WebSocket endpoint and cookies for you to connect with your own Puppeteer/Playwright code.

JSON Request Body:

Set browserWSEndpoint: true, cookies: true, and specify a ttl (time-to-live in milliseconds) to keep the browser session alive for your automation:

{
"url": "https://example.com",
"browserWSEndpoint": true,
"cookies": true,
"ttl": 30000
}

We recommend using a residential proxy for better bot detection bypass:

curl --request POST \
--url 'https://production-sfo.browserless.io/unblock?token=YOUR_API_TOKEN_HERE&proxy=residential' \
--header 'Content-Type: application/json' \
--data '{
"url": "https://example.com",
"browserWSEndpoint": true,
"cookies": true,
"ttl": 30000
}'

Response:

{
"browserWSEndpoint": "wss://production-sfo.browserless.io/p/53616c74...646b292c/devtools/browser/102ea3e9-74d7-42c9-a856-1bf254649b9a",
"cookies": [
{
"name": "session_id",
"value": "XYZ123",
"domain": "example.com",
"path": "/",
"secure": true,
"httpOnly": true
}
],
"content": null,
"screenshot": null
}

Connecting to the Browser

After receiving the response with the browserWSEndpoint and cookies, you can use Puppeteer, Playwright, or another CDP library to connect to the browser instance and inject the cookies to continue your automation:

import puppeteer from "puppeteer-core";

const TOKEN = "YOUR_API_TOKEN_HERE";

const unblock = async (url) => {
const response = await fetch(
`https://production-sfo.browserless.io/unblock?token=${TOKEN}&proxy=residential`,
{
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({
url: url,
browserWSEndpoint: true,
cookies: true,
ttl: 30000
})
}
);
return await response.json();
};

// Get the WebSocket endpoint after bot detection bypass
const { browserWSEndpoint, cookies } = await unblock("https://example.com/");

// Connect to the browser
const browser = await puppeteer.connect({
browserWSEndpoint: `${browserWSEndpoint}?token=${TOKEN}`
});
const page = (await browser.pages())[0];

// Inject cookies if needed
// await page.setCookie(...cookies);

await page.screenshot({ path: "screenshot.png" });
await browser.close();

Waiting for Things

Browserless offers 4 different ways to wait for preconditions to be met on the page before returning the response. These are events, functions, selectors and timeouts.

waitForEvent

Waits for an event to happen on the page before continuing:

Example

curl -X POST \
https://production-sfo.browserless.io/unblock?token=YOUR_API_TOKEN_HERE \
-H 'Cache-Control: no-cache' \
-H 'Content-Type': 'application/json' \
-d '{
"url": "https://example.com/",
"waitForEvent": {
"event": "fullscreenchange",
"timeout": 5000
}
}'

waitForFunction

Waits for the provided function to return before continuing. The function can be any valid JavaScript function including async functions.

Example

JS function

async () => {
const res = await fetch("https://jsonplaceholder.typicode.com/todos/1");
const json = await res.json();

document.querySelector("h1").innerText = json.title;
};
curl -X POST \
https://production-sfo.browserless.io/unblock?token=YOUR_API_TOKEN_HERE \
-H 'Cache-Control: no-cache' \
-H 'Content-Type': 'application/json' \
-d '{
"url": "https://example.com/",
"waitForFunction": {
"fn": "async()=>{let t=await fetch('https://jsonplaceholder.typicode.com/todos/1'),e=await t.json();document.querySelector('h1').innerText=e.title}",
"timeout": 5000
}
}'

waitForSelector

Waits for a selector to appear on the page. If at the moment of calling this API, the selector already exists, the method will return immediately. If the selector doesn't appear after the timeout milliseconds of waiting the API will return a non-200 response code with an error message as the body of the response.

The object can have any of these values:

  • selector: String, required — A valid CSS selector.
  • hidden Boolean, optional — Wait for the selected element to not be found in the DOM or to be hidden, i.e. have display: none or visibility: hidden CSS properties.
  • timeout: Number, optional — Maximum number of milliseconds to wait for the selector before failing.
  • visible: Boolean, optional — Wait for the selected element to be present in DOM and to be visible, i.e. to not have display: none or visibility: hidden CSS properties.

Example

curl -X POST \
https://production-sfo.browserless.io/unblock?token=YOUR_API_TOKEN_HERE \
-H 'Cache-Control: no-cache' \
-H 'Content-Type': 'application/json' \
-d '{
"url": "https://example.com/",
"waitForSelector": {
"selector": "h1",
"timeout": 5000
}
}'