Scrapy (Python)

Use Scrapy alongside the /content API or BrowserQL to scrape data. Implement the start_requests() method to query /content or BrowserQL, and keep your scraping logic the same.

Prerequisites

• A Browserless API token from your account dashboard
• Scrapy installed (pip install scrapy)

Basic Usage

Scrapy uses HTTP requests to download HTML and build a DOM-like abstraction, so it can only return the initial render of a page without interacting with it. The /content API ensures the HTML is rendered and evaluated inside a browser, not downloaded raw. BrowserQL goes further, using advanced stealth techniques to bypass bot detectors before rendering.

Implement the start_requests() method to query the /content API, and keep your scraping logic the same.

import json
import scrapy

class PptrDocsSpider(scrapy.Spider):
    name = "pptr-docs"

    def start_requests (self):
        options = {
            "url": "https://puppeteer.github.io/pptr.dev/",
            "waitForTimeout": 5000
        }

        yield scrapy.Request(
            url="https://production-sfo.browserless.io/content?token=YOUR_API_TOKEN_HERE",
            method='POST',
            dont_filter=True,
            headers={"Content-Type": "application/json"},
            body=json.dumps(options)
        )

    def parse(self, response):
        entries = response.css('sidebar-component a.pptr-sidebar-item')
        for entry in entries:
            yield{
                'title' : entry.css('::text').get(),
                'url' : entry.css('::attr(href)').get(),
            }

You can use all the options available in the /content API, use stealth mode, our residential proxies and more! For more reference, please see this blogpost.

Bypass bot-blockers using /unblock

In cases where websites implement aggressive bot-detection mechanisms, you can use the /unblock API to bypass these. The /unblock API uses a variety of tools and strategies to override and hide the footprints that headless browsers leave behind, allowing you to access bot-protected websites from a remote interface.

Similar to the /content API, the /unblock API renders and evaluates the page in a browser, but with extra stealth features. This makes it ideal for scraping highly protected websites.

Make the /unblock request by yielding scrapy.Request(), and extract the HTML content from the JSON response in the parse() method. Here's an example:

import json
import scrapy

class PptrDocsSpider(scrapy.Spider):
    name = "pptr-docs"

    def start_requests (self):
        options = {
            "url": "https://puppeteer.github.io/pptr.dev/",
            "waitForTimeout": 5000
        }

        yield scrapy.Request(
            url="https://production-sfo.browserless.io/unblock?token=YOUR_API_TOKEN_HERE",
            method='POST',
            dont_filter=True,
            headers={"Content-Type": "application/json"},
            body=json.dumps(options)
        )

    def parse(self, response):
        # Extracts the content from the response
        html_content = json.loads(response.text)['content']
        dom = scrapy.Selector(text=html_content)

        # Continue as normal
        entries = dom.css('sidebar-component a.pptr-sidebar-item')
        for entry in entries:
            yield{
                'title' : entry.css('::text').get(),
                'url' : entry.css('::attr(href)').get(),
            }

FAQ & Troubleshooting

Why am I getting a 403 Forbidden error?

Your API token is missing or expired. Pass it as a ?token= query parameter in the WebSocket or HTTP URL. Verify the token in your account dashboard.

My script works locally but fails on Browserless

Local browser settings may differ from the Browserless environment. Use launch parameters to match your local setup (viewport, user agent, timezone). See launch parameters for the full list.

Next steps

BaaS Quickstart

get started with Puppeteer or Playwright on Browserless

Launch parameters

configure browser behavior and environment