General Bot Detection Advice

Each site and each bot detection mechanism targets different parts of the whole browser environment, but Browserless offers many different ways to work around detection mechanisms.

Here are some ways Browserless can help you bypass anti-bots systems:

IP based

Some detection mechanisms use an IP-reputation-based system to detect bots. In these cases, datacenter IPs are more likely to be detected as bots. Browserless offers first party residential proxies to avoid this.

Residential proxies route your browser traffic through real residential IP addresses, making your automation appear as if it's coming from genuine home internet connections. This significantly reduces the likelihood of detection compared to datacenter IPs, which are commonly flagged by anti-bot systems. Browserless provides seamless integration with residential proxies across multiple geographic regions, allowing you to maintain anonymity while accessing geo-restricted content or bypassing IP-based detection mechanisms.

import puppeteer from 'puppeteer-core';

(async () => {
  const browser = await puppeteer.connect({
    browserWSEndpoint: `wss://production-sfo.browserless.io?token=GOES-HERE&proxy=residential&proxyCountry=US`,
  });

  const page = await browser.newPage();
  await page.goto('https://example.com');

  const screenshot = await page.screenshot();
  console.log('Screenshot taken!');

  await browser.close();
})();

You can read more about residential proxies here.

Fingerprinting

Other anti-bot systems rely on fingerprinting. This is a technique that identifies browser and device characteristics to identify the uniqueness of a browser. Bots are often identified by their fingerprint, since they almost always have the same fingerprint. Using the /chromium/stealth endpoint, you can make your browser look more like a human browser and avoid being detected as a bot.

import puppeteer from 'puppeteer-core';

(async () => {
  const browser = await puppeteer.connect({
    browserWSEndpoint: `wss://production-sfo.browserless.io/chromium/stealth?token=GOES-HERE`,
  });

  const page = await browser.newPage();
  await page.goto('https://example.com');

  const screenshot = await page.screenshot();
  console.log('Screenshot taken with stealth mode!');

  await browser.close();
})();

You can read more about chromium stealth here.

Protocol based

Some sophisticated anti-bot systems rely on protocol-based detection. This is a technique that identifies if the browser is using the CDP (Puppeteer)/Playwright/Chromedriver protocol, by checking the values of multiple browser properties. Browserless offers an /unblock API endpoint that can be used to load the page using a legitimate browser without any library, and then allows you to use CDP/Playwright to interact with the page.

import puppeteer from 'puppeteer-core';

(async () => {
  // First, use the unblock API to bypass detection
  const response = await fetch('https://production-sfo.browserless.io/unblock', {
    method: 'POST',
    headers: {
      'Content-Type': 'application/json',
      'Authorization': 'Bearer GOES-HERE'
    },
    body: JSON.stringify({
      url: 'https://example.com'
    })
  });

  const { browserWSEndpoint } = await response.json();

  // Then connect with Puppeteer to the unblocked browser
  const browser = await puppeteer.connect({
    browserWSEndpoint
  });

  const page = await browser.newPage();
  const screenshot = await page.screenshot();
  console.log('Screenshot taken after unblocking!');

  await browser.close();
})();

You can read more about the unblock API here.

TL;DR

Here's our recommended approach for bypassing bot detection, in order of preference:

• In general, try /chromium/stealth first. That's the smoothest and fastest stealthy experience we can offer right now.

• If /chromium/stealth is not enough, try /chromium/stealth with residential proxies, unless speed is of the essence. Residential proxies on /chromium/stealth are guaranteed to have a better IP reputation on average.

• If that doesn't work, try using the /unblock API alongside residential proxies. It uses the same logic that /chromium/stealth does, but it "holds" and loads the page without connecting to CDP or chromedriver, thus sites and detection mechanisms don't see anything "weird" or out of place and whitelist the session.

• If faced with captchas, the general recommendation is to try to /unblock the page first (assuming the captcha is blocking the whole page, and doesn't come later on the session). /unblock + residential proxies + the Browserless.solveCaptcha command are the best and more bulletproof anti-bot combination we've found so far.