Skip to main content
Version: v2

Site Unblocking

Modern websites employ sophisticated anti-bot systems that can detect and block automated browser sessions. The Browserless Unblock API takes a minimalistic approach to getting past these blocks without relying on traditional automation libraries that leave detectable traces. You can unblock sites via the /Unblock REST API or through BrowserQL.

Key Benefits

  • Works directly with browser's native remote interfaces (no library traces)
  • Launches browsers just like end-users would
  • Automatically detects and corrects common bot blockers
  • Returns browser endpoints for continued automation or direct content/cookies

How The Unblock API Works​

The Unblock API is flexible and optimized - you only get the data you request, reducing execution time and resource usage. By setting certain values to false, Browserless will optimize the execution to only return fields you asked for, and not spend time producing the other fields.

API Reference

For detailed usage examples, request parameters, and response formats, see the /unblock API reference page.

Connect the unblocked browser session to your preferred automation library for continued control. This approach allows you to unblock a site first, then connect your library of choice to the browser and continue your automation workflow.

import puppeteer from 'puppeteer-core';

// The underlying site you wish automate
const url = 'https://www.example.com/';

// Your API token retrievable from the dashboard.
const token = 'YOUR_API_TOKEN_HERE';

// Set a threshold, in milliseconds, to unblock
const timeout = 5 * 60 * 1000;

// What proxy type (residential), or remove for none.
const proxy = 'residential';

// Where you want to proxy from (GB === Great Britain), or remove for none.
const proxyCountry = 'gb';

// If you want to use the same proxy IP for every network request
const proxySticky = true;

const queryParams = new URLSearchParams({
  timeout,
  proxy,
  proxyCountry,
  proxySticky,
  token,
}).toString();

const unblockURL =
  `https://production-sfo.browserless.io/chromium/unblock?${queryParams}`;

const options = {
  method: 'POST',
  headers: {
    'content-type': 'application/json'
  },
  body: JSON.stringify({
    url: url,
    browserWSEndpoint: true,
    cookies: true,
    content: true,
    screenshot: true,
    ttl: 5000,
  }),
};

try {
  console.log(`Unblocking ${url}`);

  const response = await fetch(unblockURL, options);

  if (!response.ok) {
    throw new Error(`Got non-ok response:\n` + (await response.text()));
  }

  const { browserWSEndpoint } = await response.json();

  console.log(`Got OK response! Connecting puppeteer to "${browserWSEndpoint}"...`);
  const browser = await puppeteer.connect({
    browserWSEndpoint: `${browserWSEndpoint}?${queryParams}`
  });
  // Find the page by inspecting the URL and matching it
  const pages = await browser.pages();
  const page = pages.find((p) => p.url().includes(url));
  page.on('response', (res) => {
    if (!res.ok) {
      console.log(`${res.status()}: ${res.url()}`);
    }
  });
  console.log('Reloading page with networkidle0...');
  await page.reload({
    waitUntil: 'networkidle0',
    timeout,
  });
  console.log('Taking page screenshot...');
  await page.screenshot({
    path: 'temp.png',
    fullPage: true,
  });
  console.log('Done!');
  await browser.close();
} catch (error) {
  console.error(error);
}

Unblocking with BrowserQL​

You can also use our query-language, BrowserQL, which is designed to bypass sophisticated bot detection mechanisms effectively. This API allows you to specify a target URL and return data you care about: the HTML content, a .png screenshot or an unblocked browser session to use with Playwright or Puppeteer.

import puppeteer from "puppeteer-core";

const TOKEN = 'YOUR_API_TOKEN_HERE';
const url = "https://www.browserless.io/"

const unblockWithBrowserQL = async (url) => {
  const opts = {
    method: "POST",
    headers: {
      "Content-Type": "application/json",
    },
    body: JSON.stringify({
      "query": "mutation Reconnect($url: String!) { goto(url: $url, waitUntil: networkIdle) { status } reconnect(timeout: 30000) { browserWSEndpoint } }",
      "variables": { url }
    }),
  };

  const response = await fetch(
    `https://production-sfo.browserless.io/chromium/bql?token=${TOKEN}`,
    opts,
  );

  return await response.json();
};

// Reconnect
const { data } = await unblockWithBrowserQL(url);
const browser = await puppeteer.connect({
  browserWSEndpoint: data.reconnect.browserWSEndpoint + `?token=${TOKEN}`,
});
const pages = await browser.pages();

const page = pages.find((p) => p.url() === url);
await page.screenshot({ path: `screenshot-${Date.now()}.png` });
await browser.close();