Parsing HTML with External Libraries
This guide shows how to retrieve HTML with BrowserQL, handling any JavaScript rendering, bot protection, or interaction needed to reach it, then parse the result with Beautiful Soup, Scrapy, or Cheerio.
- A Browserless API token from your account dashboard
- Familiarity with BQL language basics
Example
The query below navigates to https://browserless.io/, clicks the Pricing link, and returns the page HTML.
mutation RetrieveHTML {
goto(url: "https://browserless.io/") {
status
}
click(selector: "a[href=\"/pricing\"]") {
time
}
html {
html
}
}
Beautiful Soup
Beautiful Soup is a Python library for parsing HTML and XML. Use it when you need to extract structured data from pages that require real browser interactions before the content is available.
import requests
from bs4 import BeautifulSoup
url = 'https://browserless.io/'
token = 'YOUR_API_TOKEN_HERE'
timeout = 5 * 60 * 1000
query = '''
mutation RetrieveHTML($url: String!) {
goto(url: $url) {
status
}
click(selector: "a[href=\"/pricing\"]") {
time
}
html {
html
}
}
'''
variables = {"url": url}
endpoint = f'https://production-sfo.browserless.io/chromium/bql?timeout={timeout}&token={token}'
payload = {"query": query, "variables": variables}
response = requests.post(endpoint, json=payload, headers={'content-type': 'application/json'})
html_content = response.json()['data']['html']['html']
soup = BeautifulSoup(html_content, 'html.parser')
plans = [tag.text.strip() for tag in soup.find_all('div', class_='tag_price margin-bottom margin-large')]
print(plans)
Scrapy
Scrapy is a Python web crawling framework with a built-in selector engine and data pipeline system. Pair it with Browserless to scrape pages that block traditional crawlers.
import requests
from scrapy.selector import Selector
url = 'https://browserless.io/'
token = 'YOUR_API_TOKEN_HERE'
timeout = 5 * 60 * 1000
query = '''
mutation RetrieveHTML($url: String!) {
goto(url: $url) {
status
}
click(selector: "a[href=\"/pricing\"]") {
time
}
html {
html
}
}
'''
variables = {"url": url}
endpoint = f'https://production-sfo.browserless.io/chromium/bql?timeout={timeout}&token={token}'
payload = {"query": query, "variables": variables}
response = requests.post(endpoint, json=payload, headers={'content-type': 'application/json'})
html_content = response.json()['data']['html']['html']
selector = Selector(text=html_content)
plans = selector.css('.tag_price.margin-bottom.margin-large::text').getall()
print(plans)
Cheerio
Cheerio is a jQuery-like HTML parser for Node.js. Use it to extract data from HTML returned by Browserless without running a second browser instance. If you prefer a no-code approach, the Smart Scrape API can return structured content directly.
import fetch from 'node-fetch';
import cheerio from 'cheerio';
const url = 'https://browserless.io/';
const token = 'YOUR_API_TOKEN_HERE';
const timeout = 5 * 60 * 1000;
const queryParams = new URLSearchParams({ timeout, token }).toString();
const query = `
mutation RetrieveHTML($url: String!) {
goto(url: $url) {
status
}
click(selector: "a[href=\"/pricing\"]") {
time
}
html {
html
}
}
`;
const endpoint = `https://production-sfo.browserless.io/chromium/bql?${queryParams}`;
const options = {
method: 'POST',
headers: { 'content-type': 'application/json' },
body: JSON.stringify({ query, variables: { url } }),
};
(async () => {
try {
const response = await fetch(endpoint, options);
const { data } = await response.json();
const $ = cheerio.load(data.html.html);
const plans = [];
$('.tag_price.margin-bottom.margin-large').each((_, element) => {
plans.push($(element).text().trim());
});
console.log(plans);
} catch (error) {
console.error('Error fetching or parsing HTML:', error);
}
})();
FAQ & Troubleshooting
Why am I getting a 403 Forbidden response?
Your API token is missing or malformed. Pass it as a ?token= query parameter. Check your account dashboard to verify the token is active.
My query times out before completing
Increase the timeout parameter on your mutation or use a more specific waitUntil condition. Long-running pages may need networkIdle instead of load.