Beautiful Soup (Python)
Beautiful Soup is one of the most popular scraping libraries for Python. It parses HTML strings so you can extract structured data. Use it alongside the /content API or BrowserQL to scrape any website.
Both APIs render the page in a browser before returning HTML. BrowserQL adds advanced stealth techniques to bypass bot detectors first.
- A Browserless API token from your account dashboard
- Beautiful Soup installed (
pip install beautifulsoup4)
Basic Usage
Like Cheerio, Beautiful Soup is only a parser: it does not fetch HTML on its own. To get HTML from a website, you would use the requests library to download the page:
import requests
from bs4 import BeautifulSoup
url = 'https://browserless.io/'
response = requests.get(url)
soup = BeautifulSoup(response.text, 'html.parser')
entries = soup.find_all('a')
for entry in entries:
print(entry.text.strip())
The problem: requests downloads only the HTML source, so it cannot return content that depends on JavaScript or user interaction to render.
The /content API solves this. It renders and evaluates the page in a browser before returning the HTML. You can call it with the requests library:
import requests
from bs4 import BeautifulSoup
response = requests.post("https://production-sfo.browserless.io/content",
params={ "token": "YOUR_API_TOKEN_HERE"},
json={
"waitForTimeout": 5000,
"url": "https://puppeteer.github.io/pptr.dev/"
}
)
soup = BeautifulSoup(response.text, 'html.parser')
entries = soup.find_all("a", class_="pptr-sidebar-item")
for entry in entries:
print(entry.text.strip())
In the example, we are using the old Puppeteer doc site, which relies heavily on JS to render its content. With a usual requests or cURL request, it would only download the page's source code, the JavaScript wouldn't be interpreted and the content wouldn't be rendered.
Using our API, you can use all the options available in the /content API, use stealth mode, our residential proxies and more! For more reference, please check our OpenAPI.
Bypass bot-blockers using /unblock
In cases where websites implement aggressive bot-detection mechanisms, you can use the /unblock API to bypass these. The /unblock API uses a variety of tools and strategies to override and hide the footprints that headless browsers leave behind, allowing you to access bot-protected websites from a remote interface.
Similar to the /content API, the /unblock API renders and evaluates the page in a browser, but with extra stealth features. This makes it ideal for scraping highly protected websites.
import json
import requests
from bs4 import BeautifulSoup
response = requests.post("https://production-sfo.browserless.io/unblock",
params={ "token": "YOUR_API_TOKEN_HERE"},
json={
"waitForTimeout": 5000,
"url": "https://puppeteer.github.io/pptr.dev/"
}
)
html_content = json.loads(response.text)['content']
soup = BeautifulSoup(html_content, 'html.parser')
entries = soup.find_all("a", class_="pptr-sidebar-item")
for entry in entries:
print(entry.text.strip())
FAQ & Troubleshooting
Why am I getting a 403 Forbidden error?
Your API token is missing or expired. Pass it as a ?token= query parameter in the WebSocket or HTTP URL. Verify the token in your account dashboard.
My script works locally but fails on Browserless
Local browser settings may differ from the Browserless environment. Use launch parameters to match your local setup (viewport, user agent, timezone). See launch parameters for the full list.