Browserless (2.23.0)

This service extends the Browserless open-source image with many features and enhancements for teams automating at scale. Notable features include:

• A Chrome-devtools-protocol based API for extending and enhancing libraries in a cross-language way.
• A new hybrid-automation toolkit with live session interactivity.
• Robust session management: connect, reconnect, kill and limit what a browser can do.
• Bleeding features like multiplexing numerous clients into a single Chrome process in an isolated way.
• The ability to upload and run custom extensions.
• Run multiple tokens, with access controls on each.
• Multi-browser with all the robust capabilities already in the open-source images.

There's a lot to cover here so let's get started!

The Residential proxy is only available for Enterprise and Cloud plans.

Browserless comes with a built-in mechanism to proxy to what's called "residential" IPs. These are IP addresses are sourced from real-users running a proxy server on their home networking machines. Residential proxying is especially useful for things like bypassing certain bot blockages and more.

Using a residential proxy is as straightforward as adding a few parameters to your library or API calls. Here's the required parameters and the values they support:

• ?proxy=residential: Specifies that you want to use the residential proxy for this request. Data-center coming soon.
• ?proxyCountry=us: Specifies a country you wish to use for the request. A two-digit ISO code.
• ?proxySticky=true: If you want to use the same IP address for the entirety of the session. Generally recommended for most cases.

Simply append these to your connection call, REST API calls, or any library call:

wss://production-sfo.browserless.io/chromium?token=YOUR-API-TOKEN&proxy=residential&proxyCountry=us&proxySticky

https://production-sfo.browserless.io/chromium/unblock?token=YOUR-API-TOKEN&proxy=residential&proxyCountry=us&proxySticky

Please do note that using a proxy will increase the amount of units consumed. Every megabyte of data transferred consumes 6 units.

In order to enhance the experience with open source libraries like Puppeteer, we decided to take a new approach to extending these libraries in a language-agnostic way. We call it the Browserless CDP API. Here's a quick list of what it can do:

• Generate and give back live URLs for hybrid automation.
• Solve Captchas.
• Return your page's unique identifier created by Chrome.
• Way more coming!

Since most libraries come with a way to issue "raw" CDP commands, it's an easy way to drop-in custom behaviors without having to write and maintain a library. Plus you can continue to enjoy using the same packages you've already come to know.

Getting started with this API is pretty simple. For instance, if you want to use the live viewer for a particular page, simply do the following:

import puppeteer from 'puppeteer-core';

(async () => {
  const browserWSEndpoint = 'wss://production-sfo.browserless.io/chromium';
  const browser = await puppeteer.connect({ browserWSEndpoint });
  const page = await browser.newPage();
  const cdp = await page.createCDPSession();
  await page.goto('https://example.com');
  const { liveURL } = await cdp.send('Browserless.liveURL');

  // liveURL = 'http://localhost:3000/live/?i=98e83bbfd396241a6963425b1feeba2f';
})();

You can then visit this URL in any browser to interact with the headless Chrome running someplace else.

See more below for a full list of the available APIs and features.

This API is only available for Enterprise plans. Contact us for more information here.

Returns a fully-qualified URL to load into a web-browser. This URL allows for clicking, typing and other interactions with the underlying page. This URL doesn't require an authorization token, so you're free to share it externally with your own users or employees. If security is a concern, you can set a timeout parameter to limit the amount of time this URL is valid for. By default no timeout is set and the URL is good as long as the underlying browser is open.

Programmatic control of the session is also available, so you can close the live session once your code has detected a selector, network call, or whatever else. See the below example for programmatic control.

Basic example

import puppeteer from 'puppeteer-core';

(async () => {
  const browserWSEndpoint =
    'wss://production-sfo.browserless.io/chromium?token=YOUR-API-TOKEN';
  const browser = await puppeteer.connect({ browserWSEndpoint });
  const page = await browser.newPage();
  const cdp = await page.createCDPSession();
  await page.goto('https://example.com');
  const { liveURL } = await cdp.send('Browserless.liveURL');

  // liveURL = 'https://production-sfo.browserless.io/live/?i=98e83bbfd396241a6963425b1feeba2f';
})();

Timeout example

import puppeteer from 'puppeteer-core';

(async () => {
  const browserWSEndpoint =
    'wss://production-sfo.browserless.io/chromium?token=YOUR-API-TOKEN';
  const browser = await puppeteer.connect({ browserWSEndpoint });
  const page = await browser.newPage();
  const cdp = await page.createCDPSession();
  await page.goto('https://example.com');
  const { liveURL } = await cdp.send('Browserless.liveURL', {
    timeout: 10000, // 10 seconds to connect!
  });

  // liveURL = 'https://production-sfo.browserless.io/live/?i=98e83bbfd396241a6963425b1feeba2f';
})();

Setting a Quality and Type

Setting a "quality" and "type" effects the streamed quality of the live URL's client-side resolution. By default, Browserless sets these to quality: 100 and type of "png". You can experiment different settings to get an ideal resolutions while keep latency slow. The close to 100 quality is, the potential for higher perceived latency.

import puppeteer from 'puppeteer-core';

(async () => {
  const browserWSEndpoint =
    'wss://production-sfo.browserless.io/chromium?token=YOUR-API-TOKEN';
  const browser = await puppeteer.connect({ browserWSEndpoint });
  const page = await browser.newPage();
  const cdp = await page.createCDPSession();
  await page.goto('https://example.com');
  const { liveURL } = await cdp.send('Browserless.liveURL', {
    quality: 100, // Can be 1 - 100
    type: 'png', // Can be 'jpeg' or 'png'
  });

  // liveURL = 'https://production-sfo.browserless.io/live/?i=98e83bbfd396241a6963425b1feeba2f';
})();

It's also helpful to "wait" until the user is done doing what's needed. For that reason, Browserless will fire a custom event when the page is closed as well:

Wait for close

Custom CDP Events are not supported in all libraries, including .NET Playwright.

import puppeteer from 'puppeteer-core';

(async () => {
  const browserWSEndpoint =
    'wss://production-sfo.browserless.io/chromium?token=YOUR-API-TOKEN';
  const browser = await puppeteer.connect({ browserWSEndpoint });
  const page = await browser.newPage();
  const cdp = await page.createCDPSession();
  await page.goto('https://example.com');

  const { liveURL } = await cdp.send('Browserless.liveURL');

  console.log(liveURL);

  // Wait for the Browserless.liveComplete event when the live page is closed.
  // Please not that not all libraries support custom CDP events.
  await new Promise((r) => cdp.on('Browserless.liveComplete', r));

  console.log('Done!');

  await browser.close();
})();

Programmatic Control

import puppeteer from 'puppeteer-core';

(async () => {
  const browserWSEndpoint =
    'wss://production-sfo.browserless.io/chromium?token=YOUR-API-TOKEN';
  const browser = await puppeteer.connect({ browserWSEndpoint });
  const page = await browser.newPage();
  const cdp = await page.createCDPSession();
  await page.goto('https://example.com');

  // Having the liveURLId is required in order to close it later
  const { liveURL, liveURLId } = await cdp.send('Browserless.liveURL');

  await page.waitForSelector('.my-selector');

  // Calling this CDP API with the "liveURLId" will close the session, and terminate the client
  // further usage of the liveURL will fail and no more human-control is possible
  await cdp.send('Browserless.closeLiveURL', { liveURLId });

  // Continue to process or interact with the browser, then:
  await browser.close();
})();

It's recommended that you double check the page prior to executing further code to make sure the page is where it should be, elements are present, and so forth. This approach makes it easy to solve hard things like second-factor authentication and more in a trivial fashion.

This API is only available for Enterprise plans. Contact us for more information here.

Reconnecting allows for the underlying Chrome or Chromium process to continue to run for a specified amount of time, and subsequent reconnecting back to it. With this approach you can also "share" this connection URL to other clients to connect to the same browser process, allowing you to parallelize via a single Browser process.

Once a reconnection URL is retrieved, Browserless will intercept close-based commands and stop them from terminating the browser process itself. This prevents clients from accidentally closing the process via browser.close or similar.

In order to use this API, simply call Browserless.reconnect as a CDP command. You can, optionally, set a timeout or an auth property. See the below examples for details

Basic example with timeout

import puppeteer from 'puppeteer-core';

(async () => {
  const browserWSEndpoint =
    'wss://production-sfo.browserless.io/chromium?token=YOUR-API-TOKEN';
  const browser = await puppeteer.connect({ browserWSEndpoint });
  const page = await browser.newPage();
  const cdp = await page.createCDPSession();
  await page.goto('https://example.com');

  // Allow this browser to run for 10 seconds, then shut down if nothing connects to it.
  // Defaults to the overall timeout set on the instance, which is 5 minutes if not specified.
  const { error, browserWSEndpoint } = await cdp.send('Browserless.reconnect', {
    timeout: 10000,
  });

  if (error) throw error;

  await browser.close();

  // browserWSEndpoint = 'https://production-sfo.browserless.io/reconnect/98e83bbfd396241a6963425b1feeba2f';
})();

If you want to enforce authentication, you can pass in an optional auth property that clients will need to use in order to connect with. Similar to how authentication works in general, a token query-string parameter will need to be applied.

Authentication example

import puppeteer from 'puppeteer-core';

(async () => {
  const browserWSEndpoint =
    'wss://production-sfo.browserless.io/chromium?token=YOUR-API-TOKEN';
  const browser = await puppeteer.connect({ browserWSEndpoint });
  const page = await browser.newPage();
  const cdp = await page.createCDPSession();
  await page.goto('https://example.com');

  // Set a custom authentication token that clients have to use in order to connect, or otherwise
  // receive a 401 Response.
  const { error, browserWSEndpoint } = await cdp.send('Browserless.reconnect', {
    auth: 'secret-auth-token',
  });

  if (error) throw error;

  await browser.close();

  // NOTE the URL here doesn't include the auth token!
  // browserWSEndpoint = 'https://production-sfo.browserless.io/reconnect/98e83bbfd396241a6963425b1feeba2f';
})();

Recursive Example

import puppeteer from 'puppeteer-core';

const job = async (reconnectURL) => {
  const browserWSEndpoint =
    reconnectURL ??
    'wss://production-sfo.browserless.io/chromium?token=YOUR-API-TOKEN';
  const browser = await puppeteer.connect({ browserWSEndpoint });
  const [page] = await browser.page();
  const cdp = await page.createCDPSession();
  await page.goto('https://example.com');

  // Anytime Browserless.reconnect is called, this restarts the timer back to the provided value,
  // effectively "bumping" the timer forward.
  const res = await cdp.send('Browserless.reconnect', {
    timeout: 30000,
  });

  if (res.error) throw error;

  await browser.close();

  // Continuously reconnect back...
  return job(res.browserWSEndpoint);
};

job().catch((e) => console.error(e));

This API is only available for Enterprise and Scale and above plans on Cloud. Contact us for more information here.. Only the /chrome and /chromium routes support Captcha solving.

Browserless comes with built-in captcha solving capabilities. We use a variety of techniques to try and mitigate the chances of captchas coming up, but if you happen to run into one you can simply call on our API to solve it.

Given the amount of possibilities during a captcha solve, the API returns many properties and information in order to help your script be more informed as to what happened. See the below code example for all details and fields returned by the API.

Please be aware that solving a captcha can take a few seconds up to several minutes, so you'll want to increase your timeouts accordingly for your scripts. Captcha's solved, or attempted to solve, cost 10 units.

import puppeteer from 'puppeteer-core';

(async () => {
  const browser = await puppeteer.connect({
    browserWSEndpoint:
      'wss://production-sfo.browserless.io/chromium?token=YOUR-API-TOKEN&timeout=300000',
  });

  const page = await browser.newPage();
  const cdp = await page.createCDPSession();
  await page.goto('https://www.google.com/recaptcha/api2/demo', {
    waitUntil: 'networkidle0',
  });

  const {
    // A simple boolean indicating whether the script can proceed
    ok,
    // Whether or not a captcha was found
    captchaFound,
    // A human-readable description of what occurred.
    message,
    // Whether a solve was attempted or not
    solveAttempted,
    // If the Captcha was solved, only true if found AND solved
    solved,
    // Any errors during execution are saved here:
    error,
  } = await cdp.send('Browserless.solveCaptcha', {
    // How long to wait for a Captcha to appear to solve.
    // Defaults to 10,000ms, or 10 seconds.
    appearTimeout: 30000,
  });

  console.log(message);

  if (ok) {
    await page.click('#recaptcha-demo-submit');
  } else {
    console.error(`Error solving captcha!`);
  }

  await browser.close();
})().catch((e) => {
  console.error(e);
  process.exit(1);
});

In general, if an ok response is sent back from this API, then your script is good to proceed with further actions. If a captcha is to suddenly appears after an action then you might want to listen for the Browserless.foundCaptcha event (see below) and retry solving.

This API is only available for Enterprise and Scale and above plans on Cloud. Contact us for more information here.. Only the /chrome and /chromium routes support Captcha solving.

Custom CDP Events are not supported in all libraries, including .NET Playwright.

Emitted whenever a captcha widget is found on the page. Useful for checking if there's a captcha and deciding whether or not to proceed with solving.

The example below stops until a captcha is found, which may or may not be the case with every website out there.

import puppeteer from 'puppeteer-core';

// Recaptcha
(async () => {
  const browser = await puppeteer.connect({
    browserWSEndpoint:
      'wss://production-sfo.browserless.io/chromium?token=YOUR-API-TOKEN&timeout=300000',
  });

  const page = await browser.newPage();
  const cdp = await page.createCDPSession();

  await page.goto('https://www.google.com/recaptcha/api2/demo', {
    waitUntil: 'networkidle0',
  });

  // Please note that not all libraries support custom CDP events.
  await new Promise((resolve) =>
    cdp.on('Browserless.captchaFound', () => {
      console.log('Found a captcha!');
      return resolve();
    }),
  );

  const { solved, error } = await cdp.send('Browserless.solveCaptcha');
  console.log({
    solved,
    error,
  });

  // Continue...
  await page.click('#recaptcha-demo-submit');
})().catch((e) => {
  console.error(e);
  process.exit(1);
});

This API is only available for Enterprise hosted and Starter and above plans on Cloud. Contact us for more information here..

Custom CDP Events are not supported in all libraries, including .NET Playwright.

A custom event emitted every several seconds, signaling a live connection. This is useful for a few reasons:

• It ensure that your connection with the browser is still good.
• Sending data can trigger some load-balancing technologies to not kill the connection.

Today this event is emitted every 30 seconds.

import puppeteer from 'puppeteer-core';

const browserWSEndpoint = `wss://production-sfo.browserless.io/chromium?token=YOUR-API-TOKEN`;

(async () => {
  const browser = await puppeteer.connect({ browserWSEndpoint });
  const page = await browser.newPage();
  await page.goto('https://example.com/');
  const client = await page.createCDPSession();

  client.on('Browserless.heartbeat', () => {
    console.log('Browserless.heartbeat');
  });
})();

This API is only available for Enterprise hosted and Starter and above plans on Cloud. Contact us for more information here..

A simple helper utility to return the page's unique ID. Since most libraries treat this ID as opaque, and some even hide it, knowing the page's ID can be of great help when interacting with other parts of Browserless.

import puppeteer from 'puppeteer-core';

(async () => {
  const browserWSEndpoint = 'wss://production-sfo.browserless.io/chromium';
  const browser = await puppeteer.connect({ browserWSEndpoint });
  const page = await browser.newPage();
  const cdp = await page.createCDPSession();
  const { pageId } = await cdp.send('Browserless.pageId');

  // pageId = 'ABC12354AFDC123';
})();

You can, optionally, try and "find" this ID in puppeteer or similar libraries. Given that puppeteer has this property underscored, it's likely to change or be unavailable in the future, and requires the infamous // @ts-ignore comment to allow TypeScript compilation.

const getPageId = (page: Page): string => {
  // @ts-ignore
  return page.target()._targetId;
};

BrowserQL is a new GraphQL-based API that allows you to interact with Browserless without having to write puppeteer or playwright code. We recommend using the BrowserQL editor to interact with this new API, but you can also use the API directly.

Goto action helps you navigate to a given URL.

mutation NewTab {
  goto(
    # The fully-qualified URL of the page you'd like to navigate to
    url: "https://browserless.io", 
    # When to consider the page fully-loaded and proceed with further execution
    waitUntil: networkIdle, 
    # The maximum amount of time, in milliseconds, to wait for the page to load, overriding any defaults
    timeout: 5000
  ) {
    # The status code response of the initial page-load.
    status
    # The status text of the response from the initial page-load. Generally 'ok'
    text
    # The final URL of the page after any potential redirects or URL rewrites
    url
    # The amount of time, in milliseconds, elapsed since the start of navigation to completion
    time
  }
}

SetCheckbox action helps you check or uncheck a checkbox or radio button.

mutation setCheckbox {
  goto(url: "https://demos.jquerymobile.com/1.4.5/checkboxradio-radio", waitUntil: networkIdle) {
    status
  }
  ok: setCheckbox(
    # The CSS selector of the element on the page you want to check/uncheck
    selector:"#radio-choice-0b", 
    # Whether or not the input should be checked
    checked:true, 
    # Whether or not to wait for the element to present in the DOM
    wait: true, 
    # Whether or not to scroll to the element prior to clicking, defaults to true
    scroll: true, 
    # Whether or not to check/uncheck the element only if it's visible
    visible:true, 
    # How long to wait for the element to appear before timing out on the handler, overriding any defaults
    timeout: 3000
  ){
    # The selector text
    selector
    # The X coordinate of the element, in pixels, on the page
    x
    # The Y coordinate of the element, in pixels, on the page
    y
    # The width of the element, in pixels
    width
    # The height of the element, in pixels
    height
    # The amount of time, in milliseconds, elapsed since the start of clicking to completion
    time
  }
  alreadyUnchecked: setCheckbox(selector:"#radio-choice-0a", checked:false){
    time
  }
  # returns error "Timed out waiting for selector" for not found selector
  notFound: setCheckbox(selector:"#unexistent", checked:true, timeout:3000){
    time
  }
  # returns error "Checked status did not change after clicking" if status did not change 
  disabled: setCheckbox(selector:"#radio-choice-7a", checked:true){
    time
  }
  # returns error "must be a checkbox or a radio button" if selector is not a radio button or checkbox
  noCheckOrRadio: setCheckbox(selector:"#ui-page-top", checked:true){
    time
  }
}