Batch Scrape API

BETA

The Batch Scrape API is currently in beta. Parameters and response shapes may change in future releases.

The Batch Scrape API is only available for Cloud plans. Contact us for more information here.

Scrape a list of URLs in one asynchronous job. Submit up to your plan's page limit in a single request, then poll one job ID for paginated results or receive a webhook as each page finishes. Batch scrape runs on the same workers, storage, and limits as the Crawl API, but it scrapes exactly the URLs you send and never follows links.

Endpoints

Start a batch scrape: POST /batch-scrape
Get status and results: GET /batch-scrape/{id}
Cancel a batch scrape: DELETE /batch-scrape/{id}
Auth: token query parameter (?token=)
Content-Type: application/json
Response: application/json

Prerequisites

A Browserless API token from your account dashboard on a Cloud plan

When to use batch scrape

Use /batch-scrape when you already know which pages you want, for example product URLs from a feed or results from /search or /map. You get one job ID to poll instead of tracking hundreds of requests, and page concurrency, retries, and result storage are handled for you.

If you want to discover pages by following links from a starting URL, use /crawl instead. If you need one page right now, or need AI extraction, screenshots, or browser actions, call /smart-scrape per URL.

Quickstart

cURL
Javascript
Python

curl --request POST \
  --url 'https://production-sfo.browserless.io/batch-scrape?token=YOUR_API_TOKEN_HERE' \
  --header 'Content-Type: application/json' \
  --data '{
  "urls": [
    "https://example.com/",
    "https://books.toscrape.com/",
    "not-a-url"
  ],
  "formats": ["markdown"]
}'

const TOKEN = "YOUR_API_TOKEN_HERE";
const url = `https://production-sfo.browserless.io/batch-scrape?token=${TOKEN}`;
const headers = {
  "Content-Type": "application/json"
};

const data = {
  urls: [
    "https://example.com/",
    "https://books.toscrape.com/",
    "not-a-url"
  ],
  formats: ["markdown"]
};

const startBatchScrape = async () => {
  const response = await fetch(url, {
    method: 'POST',
    headers: headers,
    body: JSON.stringify(data)
  });

  const result = await response.json();
  console.log(result);
};

startBatchScrape();

import requests

TOKEN = "YOUR_API_TOKEN_HERE"
url = f"https://production-sfo.browserless.io/batch-scrape?token={TOKEN}"
headers = {
    "Content-Type": "application/json"
}

data = {
    "urls": [
        "https://example.com/",
        "https://books.toscrape.com/",
        "not-a-url"
    ],
    "formats": ["markdown"]
}

response = requests.post(url, headers=headers, json=data)
result = response.json()

print(result)

Response

{
  "success": true,
  "id": "batch_abc123",
  "url": "https://production-sfo.browserless.io/batch-scrape/batch_abc123",
  "invalidURLs": ["not-a-url"]
}

Batch job IDs always start with batch_. The url field is the status URL to poll. invalidURLs lists the entries that were skipped because they weren't valid public http:// or https:// URLs. The valid URLs still run.

Polling for results

Poll GET /batch-scrape/{id} to check progress and retrieve results. The response has the same shape as GET /crawl/{id}: progress counts, one entry per URL, and a pre-signed contentUrl for each completed page. Results are paginated, so follow next until it's null. The next URL doesn't include your API token, so add token=YOUR_API_TOKEN_HERE to it (or send your usual authentication) when you request it.

Query parameters

Parameter	Type	Default	Description
`token`	`string`	—	Your API token (required).
`skip`	`number`	`0`	Number of pages to skip for pagination (non-negative integer).

cURL
Javascript
Python

curl --request GET \
  --url 'https://production-sfo.browserless.io/batch-scrape/batch_abc123?token=YOUR_API_TOKEN_HERE'

const TOKEN = "YOUR_API_TOKEN_HERE";
const batchId = "batch_abc123";
const url = `https://production-sfo.browserless.io/batch-scrape/${batchId}?token=${TOKEN}`;

const getBatchStatus = async () => {
  const response = await fetch(url);
  const result = await response.json();
  console.log(result);
};

getBatchStatus();

import requests

TOKEN = "YOUR_API_TOKEN_HERE"
batch_id = "batch_abc123"
url = f"https://production-sfo.browserless.io/batch-scrape/{batch_id}?token={TOKEN}"

response = requests.get(url)
result = response.json()

print(result)

Response

{
  "status": "completed",
  "total": 2,
  "completed": 2,
  "failed": 0,
  "expiresAt": "2026-10-29T10:00:41.000Z",
  "next": null,
  "data": [
    {
      "status": "completed",
      "contentUrl": "https://crawl-artifacts.s3.us-east-1.amazonaws.com/crawls/batch_abc123/page_0a1b2c3d4e5f.json?X-Amz-Algorithm=AWS4-HMAC-SHA256&...",
      "metadata": {
        "title": "Example Domain",
        "description": null,
        "language": "en",
        "scrapedAt": "2026-10-28T10:00:38.000Z",
        "sourceURL": "https://example.com/",
        "statusCode": 200,
        "error": null
      }
    },
    {
      "status": "completed",
      "contentUrl": "https://crawl-artifacts.s3.us-east-1.amazonaws.com/crawls/batch_abc123/page_6f5e4d3c2b1a.json?X-Amz-Algorithm=AWS4-HMAC-SHA256&...",
      "metadata": {
        "title": "All products | Books to Scrape - Sandbox",
        "description": null,
        "language": "en-us",
        "scrapedAt": "2026-10-28T10:00:40.000Z",
        "sourceURL": "https://books.toscrape.com/",
        "statusCode": 200,
        "error": null
      }
    }
  ]
}

Pages that are still running have contentUrl: null. Each contentUrl expires after 1 hour, so poll again for a fresh link instead of storing it. When results have expired, this endpoint returns 410 Gone.

Batch scrape jobs don't appear in GET /crawl, and there's no list endpoint for batch jobs, so keep the id from the start response.

Scrape options

Batch scrape takes scrape options as top-level fields, the same way Firecrawl's batch scrape does. Every URL in the job uses the same options.

cURL
Javascript
Python

curl --request POST \
  --url 'https://production-sfo.browserless.io/batch-scrape?token=YOUR_API_TOKEN_HERE' \
  --header 'Content-Type: application/json' \
  --data '{
  "urls": [
    "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html",
    "https://books.toscrape.com/catalogue/tipping-the-velvet_999/index.html"
  ],
  "formats": ["markdown", "html"],
  "onlyMainContent": true,
  "excludeTags": ["nav", "footer"],
  "proxy": "datacenter"
}'

const TOKEN = "YOUR_API_TOKEN_HERE";
const url = `https://production-sfo.browserless.io/batch-scrape?token=${TOKEN}`;
const headers = {
  "Content-Type": "application/json"
};

const data = {
  urls: [
    "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html",
    "https://books.toscrape.com/catalogue/tipping-the-velvet_999/index.html"
  ],
  formats: ["markdown", "html"],
  onlyMainContent: true,
  excludeTags: ["nav", "footer"],
  proxy: "datacenter"
};

const startBatchScrape = async () => {
  const response = await fetch(url, {
    method: 'POST',
    headers: headers,
    body: JSON.stringify(data)
  });

  const result = await response.json();
  console.log(result);
};

startBatchScrape();

import requests

TOKEN = "YOUR_API_TOKEN_HERE"
url = f"https://production-sfo.browserless.io/batch-scrape?token={TOKEN}"
headers = {
    "Content-Type": "application/json"
}

data = {
    "urls": [
        "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html",
        "https://books.toscrape.com/catalogue/tipping-the-velvet_999/index.html"
    ],
    "formats": ["markdown", "html"],
    "onlyMainContent": True,
    "excludeTags": ["nav", "footer"],
    "proxy": "datacenter"
}

response = requests.post(url, headers=headers, json=data)
result = response.json()

print(result)

Sending a nested scrapeOptions object, as /crawl uses, returns HTTP 400 with scrape options are top-level fields on /batch-scrape; remove the scrapeOptions wrapper.

Invalid URLs

By default, ignoreInvalidURLs is true. Entries that aren't valid public http:// or https:// URLs, including URLs that point to private or internal network addresses, are skipped and returned in invalidURLs, and the job runs the rest. Duplicate URLs are scraped once.

Set "ignoreInvalidURLs": false to reject the whole request instead. The first invalid entry returns HTTP 400 with urls[INDEX] is not a valid public http(s) URL, and no job is created. A request in which no entry is valid always returns HTTP 400 with urls contains no valid public http(s) URLs.

Webhook notifications

Receive a notification as each page is scraped, or when the job completes or fails. Webhooks work the same way as on /crawl, but their type values use the batch_scrape. prefix.

cURL
Javascript
Python

curl --request POST \
  --url 'https://production-sfo.browserless.io/batch-scrape?token=YOUR_API_TOKEN_HERE' \
  --header 'Content-Type: application/json' \
  --data '{
  "urls": ["https://example.com/", "https://books.toscrape.com/"],
  "webhook": {
    "url": "https://your-server.com/webhook",
    "events": ["page", "completed", "failed"]
  }
}'

const TOKEN = "YOUR_API_TOKEN_HERE";
const url = `https://production-sfo.browserless.io/batch-scrape?token=${TOKEN}`;
const headers = {
  "Content-Type": "application/json"
};

const data = {
  urls: ["https://example.com/", "https://books.toscrape.com/"],
  webhook: {
    url: "https://your-server.com/webhook",
    events: ["page", "completed", "failed"]
  }
};

const startBatchScrape = async () => {
  const response = await fetch(url, {
    method: 'POST',
    headers: headers,
    body: JSON.stringify(data)
  });

  const result = await response.json();
  console.log(result);
};

startBatchScrape();

import requests

TOKEN = "YOUR_API_TOKEN_HERE"
url = f"https://production-sfo.browserless.io/batch-scrape?token={TOKEN}"
headers = {
    "Content-Type": "application/json"
}

data = {
    "urls": ["https://example.com/", "https://books.toscrape.com/"],
    "webhook": {
        "url": "https://your-server.com/webhook",
        "events": ["page", "completed", "failed"]
    }
}

response = requests.post(url, headers=headers, json=data)
result = response.json()

print(result)

Each event is sent as a JSON POST to your webhook URL:

Event	`type`	`data`
`page`	`batch_scrape.page`	`pageId`, `url`, `statusCode`, and a 1-hour `contentUrl` for the scraped page.
`completed`	`batch_scrape.completed`	`url` (the first URL in the job), `status`, `total`, and `completed`.
`failed`	`batch_scrape.failed`	Same fields as `completed`. `success` is `false`.

{
  "success": true,
  "type": "batch_scrape.page",
  "id": "batch_abc123",
  "data": {
    "pageId": "page_0a1b2c3d4e5f",
    "url": "https://example.com/",
    "statusCode": 200,
    "contentUrl": "https://crawl-artifacts.s3.us-east-1.amazonaws.com/crawls/batch_abc123/page_0a1b2c3d4e5f.json?X-Amz-Algorithm=AWS4-HMAC-SHA256&..."
  },
  "error": null
}

The webhook URL must use HTTPS and must not point to a private or internal network address. Browserless doesn't follow redirects from your receiver, so a 3xx response counts as a failed delivery. Respond with a 2xx status at the final URL.

Using a profile

Scrape authenticated pages by passing a saved profile with the ?profile= query parameter on POST /batch-scrape. Every URL in the job is loaded with the profile's cookies, localStorage, and IndexedDB. Requests for a profile that doesn't exist are rejected before the job is created.

curl --request POST \
  --url 'https://production-sfo.browserless.io/batch-scrape?token=YOUR_API_TOKEN_HERE&profile=acme-prod' \
  --header 'Content-Type: application/json' \
  --data '{
  "urls": ["https://app.example.com/reports/1", "https://app.example.com/reports/2"]
}'

Cancelling a batch scrape

Cancel a running job by sending a DELETE request with its ID. Pages that already finished stay available until the job expires.

cURL
Javascript
Python

curl --request DELETE \
  --url 'https://production-sfo.browserless.io/batch-scrape/batch_abc123?token=YOUR_API_TOKEN_HERE'

const TOKEN = "YOUR_API_TOKEN_HERE";
const batchId = "batch_abc123";
const url = `https://production-sfo.browserless.io/batch-scrape/${batchId}?token=${TOKEN}`;

const cancelBatchScrape = async () => {
  const response = await fetch(url, {
    method: 'DELETE'
  });

  const result = await response.json();
  console.log(result);
};

cancelBatchScrape();

import requests

TOKEN = "YOUR_API_TOKEN_HERE"
batch_id = "batch_abc123"
url = f"https://production-sfo.browserless.io/batch-scrape/{batch_id}?token={TOKEN}"

response = requests.delete(url)
result = response.json()

print(result)

Success response (200):

{
  "status": "cancelled"
}

If the job has already reached completed, failed, or cancelled, the API returns 409 Conflict:

{
  "id": "batch_abc123",
  "status": "completed",
  "message": "Batch scrape job is already completed"
}

Limits

Limit	Value
URLs per job	Your plan's pages-per-crawl limit: 25 on the Free plan, and 100, 500, or 1,000 on paid plans depending on plan size. Counted before invalid and duplicate URLs are removed. Larger requests return HTTP 400 with `urls must contain at most N entries on your plan`.
Concurrent jobs	Shared with `/crawl`. Running crawls and batch scrapes together count toward your plan's concurrent crawl limit. Starting one more returns `429` with `Maximum concurrent crawl and batch scrape jobs (N) reached for your plan.`
Result retention	The same as `/crawl` results for your plan, starting when the job finishes. That's 24 hours on the Free plan and longer on paid plans. `expiresAt` in the status response gives the exact time.
`contentUrl` lifetime	1 hour from each status response.

To scrape more URLs than one job allows, split the list into several jobs and run them within your concurrent-job limit.

Unit cost

Batch scrape is billed like /crawl. Each page is billed for its own scrape time at 1 unit per 30 seconds, rounded up per page, plus proxy traffic (6 units per MB residential, 2 units per MB datacenter) and 10 units per successful captcha solve. The Smart Scrape half-unit price for plain fetches doesn't apply to batch pages.

Starting a job and each status poll are billed as short requests, the same as /crawl. Use webhooks, or poll with a delay, rather than polling in a tight loop. Requests rejected with HTTP 400 aren't billed. See Unit consumption for how units are counted.

Request body

POST /batch-scrape

Field	Type	Required	Default	Description
`urls`	`string[]`	Yes	—	Public `http://` or `https://` URLs to scrape. Must be non-empty and contain only strings. Limited by your plan before deduplication.
`ignoreInvalidURLs`	`boolean`	No	`true`	Skip invalid URLs and list them in `invalidURLs`. Set to `false` to reject the request on the first invalid URL.
`formats`	`string[]`	No	`["markdown"]`	Output formats: `"markdown"`, `"html"`, `"rawText"`.
`onlyMainContent`	`boolean`	No	`true`	Whether to extract only the main content of each page.
`includeTags`	`string[]`	No	`[]`	HTML selectors to include.
`excludeTags`	`string[]`	No	`[]`	HTML selectors to exclude.
`waitFor`	`number`	No	`0`	Time in milliseconds to wait after page load before scraping, clamped to 0–30,000.
`headers`	`object`	No	—	Custom HTTP headers to send with each page request. `host`, `authorization`, `proxy-authorization`, `cookie`, `set-cookie`, `x-forwarded-for`, `x-real-ip`, and `forwarded` are removed.
`timeout`	`number`	No	`150000`	Navigation timeout per page in milliseconds, clamped to 1,000–180,000.
`proxy`	`string`	No	`"residential"`	Proxy network for page fetches: `"residential"` (6 units/MB) or `"datacenter"` (2 units/MB).
`webhook`	`object`	No	—	Webhook configuration. See below.

Unknown fields, including scrapeOptions, return HTTP 400.

webhook

Field	Type	Required	Default	Description
`url`	`string`	Yes	—	The public HTTPS URL to send webhook events to.
`events`	`string[]`	No	`["completed"]`	Which events to send: `"page"`, `"completed"`, `"failed"`.

Query parameters

Endpoint	Allowed parameters
`POST /batch-scrape`	`token`, `profile`
GET /batch-scrape/{id}	`token`, `skip`
DELETE /batch-scrape/{id}	`token`

Any other query parameter returns HTTP 400.

Response fields

POST /batch-scrape (start)

Field	Type	Description
`success`	`boolean`	Whether the job was created.
`id`	`string`	The batch scrape job ID, prefixed with `batch_`.
`url`	`string`	Status URL for GET /batch-scrape/{id}.
`invalidURLs`	`string[]`	Entries skipped because they weren't valid public HTTP(S) URLs. Empty when every URL was valid.

GET /batch-scrape/{id} (status and results)

The status response uses the same fields as GET /crawl/{id}: status, total, completed, failed, expiresAt, next, and data, where each data entry has status, contentUrl, and metadata. The next URL points to /batch-scrape/{id}?skip=N and doesn't include your API token, so append &token=YOUR_API_TOKEN_HERE when you request it.

Error responses

Status	Description
`400 Bad Request`	Invalid body or query parameters, a nested `scrapeOptions` object, too many URLs for your plan, no valid URLs, an invalid URL with `ignoreInvalidURLs: false`, a non-HTTPS or private webhook URL, or an ID that doesn't start with `batch_`.
`401 Unauthorized`	Missing or invalid API token.
`404 Not Found`	The job doesn't exist or belongs to another account.
`409 Conflict`	(DELETE only) The job is already `completed`, `failed`, or `cancelled`.
`410 Gone`	(GET only) The job's results have expired.
`429 Too Many Requests`	Your plan's concurrent crawl and batch scrape limit is reached, or the status endpoint is at capacity. Retry shortly.
`503 Service Unavailable`	The service is unavailable, temporarily degraded, or busy creating another job for your token. Retry shortly.

FAQ & Troubleshooting

Why do I get 400 with remove the scrapeOptions wrapper?

Batch scrape takes scrape options at the top level of the body, not inside scrapeOptions. Move formats, onlyMainContent, and the other options up one level next to urls. /crawl is the endpoint that uses the scrapeOptions wrapper.

Why do I get 429 when I start a batch scrape?

Crawls and batch scrapes share one concurrent-job limit for your plan. Wait for a running job to finish, or cancel one with DELETE, then retry. Sending one larger batch is usually better than several small ones running at the same time.

Can I poll a batch job through /crawl/{id}?

No. GET /crawl/batch_... returns 400 Invalid crawl ID, and /batch-scrape rejects crawl_ IDs the same way. Use the url returned when you started the job.

Why did fewer pages run than the URLs I sent?

Check invalidURLs in the start response. Invalid and private-network URLs are skipped, and duplicate URLs are scraped once. total in the status response counts the pages that actually run.

Next steps

Crawl API

Discover and scrape pages by following links from one URL

Smart Scrape API

Scrape a single page with AI extraction, screenshots, and actions

See the OpenAPI reference for complete parameter details.

Was this page helpful?

When to use batch scrape​

Quickstart​

Polling for results​

Scrape options​

Invalid URLs​

Webhook notifications​

Using a profile​

Cancelling a batch scrape​

Limits​

Unit cost​

Request body​

POST /batch-scrape​

webhook​

Query parameters​

Response fields​

POST /batch-scrape (start)​

GET /batch-scrape/{id} (status and results)​

Error responses​

FAQ & Troubleshooting​

Next steps​