Extracting content with BAP
BAP has four extraction methods, and picking the right one saves you a parsing step: text() for readable content, html() for markup, markdown() for LLM-ready output, and mapSelector() for structured per-element data. This guide covers all four plus the clean options that strip a page down before it leaves the browser.
- A Browserless API token from your account dashboard
- The BAP SDK installed and connecting, from the BAP Quickstart
Pick a method
| You want | Use | Returns |
|---|---|---|
| Readable text, optionally scoped to a selector | text() | The text plus timing metadata |
| Markup, optionally scoped and cleaned | html() | The HTML plus timing metadata |
| LLM-ready conversion of the page | markdown() | The page converted to Markdown |
| One record per matched element | mapSelector() | id, class, innerText, and innerHTML for every match |
All four accept a selector to scope extraction, a timeout for how long to wait for that selector, and visible to skip elements that aren't rendered. Extraction happens inside the browser, so what comes back over the wire is already scoped and cleaned rather than a full page you post-process.
Extract text and HTML
- TypeScript
- Python
import Browserless from "@browserless.io/bap-ts";
const TOKEN = "YOUR_API_TOKEN_HERE";
const browser = Browserless.connect({
browserWSEndpoint: "wss://production-sfo.browserless.io/chromium/bql",
token: TOKEN,
});
const page = await browser.newPage();
await page.goto("https://example.com");
// Text of one element. The default selector is the whole document.
const { text } = await page.text({ selector: "h1" });
// Markup scoped to a selector.
const { html } = await page.html({ selector: "main" });
console.log(text, html?.length);
await browser.close();
import bap.sync_api as bap
TOKEN = "YOUR_API_TOKEN_HERE"
with bap.Browserless.connect(
browser_ws_endpoint="wss://production-sfo.browserless.io/chromium/bql",
token=TOKEN,
) as browser:
with browser.page() as page:
page.goto("https://example.com")
# Text of one element. The default selector is the whole document.
text = page.text(selector="h1")["text"]
# Markup scoped to a selector.
html = page.html(selector="main")["html"]
print(text, len(html or ""))
Playwright-style shorthands exist too: content() returns the full HTML as a plain string, and eval_on_selector(selector) returns an element's text.
Response:
Example Domain 220
Convert a page to Markdown
markdown() converts the rendered DOM to Markdown inside the browser, which is usually the cheapest way to feed a page to an LLM: markup noise is gone and the structure survives as headings and lists. Scope it with selector when you only want the article body rather than navigation and footers.
- TypeScript
- Python
const { markdown } = await page.markdown({ selector: "main" });
console.log(markdown);
markdown = page.markdown(selector="main")["markdown"]
print(markdown)
Response:
# Example Domain
This domain is for use in illustrative examples in documents.
Clean HTML before it leaves the browser
text() and html() accept a clean option that strips the page down server-side. That matters for LLM pipelines twice over: smaller payloads cross the wire, and fewer tokens reach the model. By default cleaning removes non-text nodes such as scripts, styles, and media, and collapses whitespace. The knobs:
| Option | Default | What it does |
|---|---|---|
removeNonTextNodes | true | Drops scripts, links, video, canvas, and other non-textual nodes |
selectors | — | Extra DOM selectors to remove, on top of the non-text nodes |
removeAttributes | false | Strips attributes from every node, keeping only structure |
attributes | — | With mode: "deny" (default), attributes to remove; with mode: "allow", the only attributes to keep |
removeRegex | true | Runs the regexes list over the output, replacing matches with a space |
regexes | whitespace + HTML comments | Regex patterns (no surrounding /) applied in order |
The generated CleanInput reference stays current with the schema when these options change.
- TypeScript
- Python
// Keep structure and href targets, drop everything else.
const { html } = await page.html({
selector: "main",
clean: {
removeAttributes: true,
mode: "allow",
attributes: ["href"],
selectors: ["nav", "footer", ".ads"],
},
});
# Keep structure and href targets, drop everything else.
# Input types are TypedDicts, so a plain dict with the schema's camelCase keys works.
html = page.html(
selector="main",
clean={
"removeAttributes": True,
"mode": "allow",
"attributes": ["href"],
"selectors": ["nav", "footer", ".ads"],
},
)["html"]
Get structured data with mapSelector
mapSelector() runs your selector against every match and returns one record per element with its id, class list, innerText, and innerHTML. One request replaces the query-then-loop pattern, which is the difference between one round trip and one per element.
- TypeScript
- Python
const links = await page.mapSelector("a");
for (const link of links) {
console.log(link.innerText, link.class);
}
// $$eval delegates to mapSelector, so this is the same call.
const items = await page.$$eval("ul li");
links = page.map_selector("a")
for link in links:
print(link["innerText"], link["class"])
# eval_on_selector_all delegates to map_selector, so this is the same call.
items = page.eval_on_selector_all("ul li")
Response, one line per matched element:
More information... null
By default mapSelector() waits for the selector to appear (wait: true). Pass wait: false to sample whatever is in the DOM right now, which returns an empty list instead of blocking when nothing matches.
FAQ & Troubleshooting
My extraction returns null instead of content
The selector didn't match within the timeout, or the element exists but isn't rendered and you passed visible: true. Widen the selector, raise the per-call timeout, or drop visible if hidden content is acceptable.
Why is $eval() not running my function?
BAP's $eval(selector) returns the matched element's text content and takes no function argument, unlike Puppeteer's. For structured properties across matches use mapSelector(). The migration guide lists every behavioral difference.
The cleaned HTML removed content I needed
removeNonTextNodes defaults to true and takes anchors' surrounding markup with it in aggressive setups. Set removeNonTextNodes: false and remove specific noise with selectors instead, or switch mode to "allow" and list only the attributes you need kept.
Should I use markdown() or html() with clean for LLM input?
Start with markdown(). It produces the smallest readable output and needs no tuning. Reach for html() with clean when the model needs markup semantics, such as table structure or attribute values, that Markdown conversion flattens.