For AI agents: a documentation index is available at /llms.txt
Skip to main content

Network control with BAP

BAP gives you four levers over a session's network traffic: block requests before they leave, query everything the browser captured, wait for a specific request or response, and answer matching requests yourself. This guide shows each one and when to reach for it.

Prerequisites

Block requests with reject​

reject() stops matching requests before they're sent. Blocking images, media, and stylesheets is the usual first move for scraping, because pages settle faster and transfer less. Set it up before goto(), since rules only apply to requests made after the call.

Match by resource type, glob-style url patterns, regex, or HTTP method. The operator decides how multiple conditions combine: or (the default) rejects a request matching any condition, and requires all of them.

import Browserless from "@browserless.io/bap-ts";

const TOKEN = "YOUR_API_TOKEN_HERE";

const browser = Browserless.connect({
  browserWSEndpoint: "wss://production-sfo.browserless.io/chromium/bql",
  token: TOKEN,
});

const page = await browser.newPage();

// Rules apply to requests made after this call, so set them before goto().
await page.reject({
  type: ["image", "media", "stylesheet"],
  url: ["*doubleclick.net*", "*google-analytics.com*"],
  operator: "or",
});

await page.goto("https://example.com");
console.log(await page.title());
await browser.close();

Response:

Example Domain

Pass enabled: false to switch rejection off again mid-session.

Query captured traffic​

The browser records the session's traffic, and request() and response() query that capture with filters: glob url patterns, method, resource type, and for responses status. Both default to wait: true, which blocks until the first match and returns it. Pass wait: false to get everything captured so far instead, which is the mode you want after navigation has finished.

await page.goto("https://example.com");

// Everything captured so far; wait: false means "don't block for new matches".
const failures = await page.response({
  status: [404, 500, 502, 503],
  wait: false,
});

for (const res of failures) {
  console.log(res.status, res.url);
}

// The first XHR request to the API, waiting up to 10 seconds for it.
const [apiCall] = await page.request({
  url: ["*api.example.com*"],
  type: ["xhr", "fetch"],
  timeout: 10_000,
});

Response records carry url, status, method, type, headers, and body. Binary bodies come back base64-encoded with the base64Encoded flag set, so check it before decoding.

Wait for a request or response​

When you only care that one call happened, waitForRequest() and waitForResponse() block until a match arrives. They take a single glob url pattern rather than a list, and waitForResponse() also matches on statuses. Use these to synchronize on an API call a click triggers, instead of sleeping and hoping.

// Start waiting before the click so the response can't slip past.
const settled = page.waitForResponse({
  url: "*api.example.com/cart*",
  statuses: [200],
  timeout: 10_000,
});

await page.click("#add-to-cart");
const response = await settled;
console.log(response.status);

Unlike Puppeteer, these methods don't take predicate functions. The filters are sent to the server as part of the BrowserQL operation, so they must be data. When glob patterns can't express your condition, fetch candidates with request() or response() and filter client-side.

Mock responses with fulfill​

fulfill() answers matching requests yourself instead of letting them reach the network. Match the same way as reject(), then supply the status (default 200), body or base64Body for binary data, contentType, and extra headers. Like reject(), set it up before the navigation that triggers the requests.

// Serve a canned API payload so the page renders without the real backend.
await page.fulfill({
  url: ["*api.example.com/products*"],
  status: 200,
  contentType: "application/json",
  body: JSON.stringify([{ id: 1, name: "test product" }]),
});

await page.goto("https://example.com/catalog");

Set headers and HTTP credentials​

setExtraHTTPHeaders() attaches headers to every request the page makes from that point on. authenticate() answers HTTP authentication challenges (401/407) with the supplied credentials, optionally scoped to one origin so credentials don't leak to other hosts.

await page.setExtraHTTPHeaders({
  "X-Request-Source": "bap-worker-3",
});

// Scope credentials to one origin so they aren't offered elsewhere.
await page.authenticate("user", "pass", { origin: "https://internal.example.com" });

await page.goto("https://internal.example.com/dashboard");

FAQ & Troubleshooting​

My reject() rules aren't blocking anything

Call reject() before the navigation whose requests you want blocked. Rules apply to requests made after the call, so setting them after goto() only affects later traffic such as XHR polling.

request() hangs instead of returning captured traffic

Both request() and response() default to wait: true, which blocks until a new match arrives and times out after 30 seconds if none does. Pass wait: false to return what's already captured immediately.

Can I pass a predicate function like Puppeteer's waitForResponse(fn)?

No. Filters execute on the BrowserQL server, so they're plain data: glob URL patterns, methods, types, and statuses. For conditions globs can't express, pull candidates with response({ wait: false }) and filter in your own code.

The response body for an image is an unreadable string

Binary bodies come back base64-encoded, flagged with base64Encoded: true, so decode before use rather than treating the value as text. The bodyEncoding option on the response filter requires a BQL schema recent enough to support it.

Next steps​

Extracting content

Pull text, HTML, and structured data once the traffic you need has arrived

Stealth and humanization

Route through proxies and avoid detection while controlling traffic

Page reference

Full option and response types for every network method

Was this page helpful?