Running Crawlee on Browserless

Crawlee is a web scraping and crawling library for Node.js that handles request queues, retries, and session rotation. This guide connects PlaywrightCrawler or PuppeteerCrawler to Browserless, so your crawler keeps its queue and handlers while the browsers run in the cloud.

Prerequisites

• A Browserless API token from your account dashboard
• Node.js 22 or higher
• Crawlee 4.0.0-rc.1, which adds the remoteBrowser option. This page's examples use that release's API.
• Familiarity with Crawlee's request handlers

note

The remoteBrowser option ships in Crawlee 4, which is currently a release candidate, and its API is still changing between builds. The install commands below pin 4.0.0-rc.1. Crawlee 3.x has no built-in remote browser option. See the FAQ if you can't upgrade yet.

When to use Crawlee with Browserless

Use this setup when you already have Crawlee crawlers, or want Crawlee's queueing, retries, and storage, and need browsers that don't run on your own machines. Crawlee manages the crawl. Browserless runs the browsers and handles stealth, proxies, and CAPTCHAs.

If you only need rendered content, the /content API (one URL per request) or the /crawl API (a whole site, in beta on Cloud plans) is simpler and needs no crawler code. If the target site has aggressive bot detection, use BrowserQL.

Step-by-step setup

1. Set your API token

   Copy your API token from the account dashboard and export it:

   export BROWSERLESS_TOKEN=YOUR_API_TOKEN_HERE

2. Install Crawlee

   Install Crawlee 4 alongside playwright (or puppeteer). Crawlee loads these full packages by name, so the -core variants won't work here. The environment variable skips the local browser download, since Browserless supplies the browser:

   Playwright

   PLAYWRIGHT_SKIP_BROWSER_DOWNLOAD=1 npm install crawlee@4.0.0-rc.1 playwright

   Puppeteer

   PUPPETEER_SKIP_DOWNLOAD=true npm install crawlee@4.0.0-rc.1 puppeteer

3. Point the crawler at Browserless

   Create crawler.mjs. Pass your Browserless endpoint as remoteBrowser.endpoint instead of launching a local browser:

   Playwright

   import { PlaywrightCrawler } from "crawlee";
   
   const TOKEN = process.env.BROWSERLESS_TOKEN;
   
   const crawler = new PlaywrightCrawler({
     remoteBrowser: {
       endpoint: `wss://production-sfo.browserless.io?token=${TOKEN}`,
       // Keep this below your plan's concurrency limit; it's a soft cap, so leave headroom to avoid 429s.
       maxOpenBrowsers: 5,
     },
     maxRequestsPerCrawl: 20,
     async requestHandler({ page, request, enqueueLinks, pushData, log }) {
       const title = await page.title();
       log.info(`${request.loadedUrl} — "${title}"`);
       await pushData({ url: request.loadedUrl, title });
   
       // Stay on the same domain so the crawl doesn't wander off-site.
       await enqueueLinks({ strategy: "same-domain" });
     },
   });
   
   await crawler.run(["https://crawlee.dev"]);

   Puppeteer

   import { PuppeteerCrawler } from "crawlee";
   
   const TOKEN = process.env.BROWSERLESS_TOKEN;
   
   const crawler = new PuppeteerCrawler({
     remoteBrowser: {
       endpoint: `wss://production-sfo.browserless.io?token=${TOKEN}`,
       // Keep this below your plan's concurrency limit; it's a soft cap, so leave headroom to avoid 429s.
       maxOpenBrowsers: 5,
     },
     maxRequestsPerCrawl: 20,
     async requestHandler({ page, request, enqueueLinks, pushData, log }) {
       const title = await page.title();
       log.info(`${request.loadedUrl} — "${title}"`);
       await pushData({ url: request.loadedUrl, title });
   
       // Stay on the same domain so the crawl doesn't wander off-site.
       await enqueueLinks({ strategy: "same-domain" });
     },
   });
   
   await crawler.run(["https://crawlee.dev"]);

4. Run the crawler

   node crawler.mjs

   Response:

   INFO  PlaywrightCrawler: Starting the crawler.
   INFO  PlaywrightCrawler: https://crawlee.dev/ — "<page title>"
   INFO  PlaywrightCrawler: https://crawlee.dev/js/docs/quick-start — "<page title>"
   ...
   INFO  PlaywrightCrawler: Crawler reached the maxRequestsPerCrawl limit of 20 requests and will shut down soon.
   INFO  PlaywrightCrawler: Final request statistics: {"requestsFinished":20,"requestsFailed":0, ...}

   Scraped records land in ./storage/datasets/default, the same as a local Crawlee run.

How it works

• remoteBrowser.endpoint replaces the local launch. PlaywrightCrawler connects with connectOverCDP() by default; PuppeteerCrawler uses puppeteer.connect().
• Each browser Crawlee opens is a Browserless session. maxOpenBrowsers limits how many run at once, and newPage() waits for a free slot instead of failing. It's a soft limit, though; see Share one pool across crawlers.
• Pages always open in incognito contexts, because remote connections can't use persistent contexts. Use Crawlee's SessionPool if requests need to share cookies.

Configure the browser through the URL

Crawlee's headless, launchOptions, and userDataDir settings have no effect on a remote browser. Configure the Browserless session through the endpoint URL instead: routes such as /stealth go in the path, and options such as proxies and timeouts go in the query string.

Stealth

Use the stealth route for sites with bot detection:

endpoint: `wss://production-sfo.browserless.io/stealth?token=${TOKEN}`,

Residential proxies

Route traffic through residential proxies for geo-targeting and rate-limit avoidance. Leave Crawlee's proxyConfiguration unset when you do this, so the two don't conflict:

endpoint: `wss://production-sfo.browserless.io?token=${TOKEN}&proxy=residential&proxyCountry=us`,

Session timeout

Long crawls keep each browser open across many requests. Raise the per-session timeout (in milliseconds) if sessions close mid-crawl:

endpoint: `wss://production-sfo.browserless.io?token=${TOKEN}&timeout=300000`,

Regional and self-hosted endpoints

Deployment	Endpoint
US West (San Francisco)	wss://production-sfo.browserless.io?token=${TOKEN}
Europe (London)	wss://production-lon.browserless.io?token=${TOKEN}
Europe (Amsterdam)	wss://production-ams.browserless.io?token=${TOKEN}
Self-hosted Docker	ws://localhost:3000?token=${TOKEN}

See connection URL patterns for every supported parameter.

Share one pool across crawlers

If several crawlers run in the same process, build one remote pool with remotePlaywrightBrowserPool() (or remotePuppeteerBrowserPool()) and pass it as browserPool, so they share one session limit instead of each opening its own:

import { PlaywrightCrawler, remotePlaywrightBrowserPool } from "crawlee";

const TOKEN = process.env.BROWSERLESS_TOKEN;

// `await using` destroys the pool, and its Browserless sessions, when the block exits (Node.js 24+).
await using browserPool = remotePlaywrightBrowserPool({
  endpoint: `wss://production-sfo.browserless.io?token=${TOKEN}`,
  maxOpenBrowsers: 2,
  browserPoolOptions: { maxOpenPagesPerBrowser: 5 },
});

await new PlaywrightCrawler({ browserPool, requestHandler: async ({ page }) => { /* ... */ } })
  .run(["https://crawlee.dev"]);
await new PlaywrightCrawler({ browserPool, requestHandler: async ({ page }) => { /* ... */ } })
  .run(["https://apify.com"]);

On Node.js 22, declare the pool with const and call await browserPool.destroy() when you're done. Don't set both remoteBrowser and browserPool on the same crawler.

Builds after 4.0.0-rc.1 remove remotePlaywrightBrowserPool() and move its options into playwrightBrowserPool({ remoteBrowser: { endpoint, maxOpenBrowsers } }). On a newer build, follow Crawlee's remote browser guide for this section. The crawler-level remoteBrowser option works the same in both.

Treat maxOpenBrowsers as a soft limit. In 4.0.0-rc.1, several newPage() calls that start at the same moment can each see a free slot and open browsers past the limit. Browserless returns 429 for sessions over your plan's concurrency, so set maxOpenBrowsers with some headroom.

FAQ & Troubleshooting

I'm getting 429 Too Many Requests

You're opening more concurrent sessions than your plan allows. Set remoteBrowser.maxOpenBrowsers below your plan's concurrency limit. Crawlee queues pages until a slot frees up, but simultaneous newPage() calls can briefly overshoot the cap, so leave headroom.

Can I use Browserless with Crawlee 3.x?

Upgrade to Crawlee 4 (npm install crawlee@4.0.0-rc.1) to get remoteBrowser. Crawlee 3.x always launches a local browser and has no supported remote option. If you can't upgrade, call the /content API from a CheerioCrawler or HttpCrawler to get browser-rendered HTML without a local browser.

My headless or launchOptions settings are ignored

That's expected. Those options configure a local launch, and there isn't one. Set browser behavior through launch parameters on the endpoint URL instead.

Sessions close before the crawl finishes

Each Browserless session has a timeout. Add a larger timeout query parameter to the endpoint, as shown in Session timeout, or lower maxRequestsPerCrawl per run.

Does this work with Crawlee for Python?

This page covers Crawlee for JavaScript. Crawlee for Python's PlaywrightCrawler launches local browsers; for Python crawls, use Playwright for Python with connect_over_cdp or Scrapy with the /content API.

Next steps

Stealth routes

Get past bot detection on hardened sites

Crawl API

Crawl a whole site with one request and no crawler code

Connection URL patterns

Every path and query parameter the endpoint accepts