/content API

The content API allows for simple navigation to a site and capturing the page's content (including the <head> section). Browserless will respond with a Content-Type of text/html, and string of the site's HTML after it has been rendered and evaluated inside the browser. This is useful for capturing the content of a page that has a lot of JavaScript or other interactivity.

You can check the full Open API schema here.

BrowserQL

We recommend using BrowserQL, Browserless' first-class browser automation API, to capture content from any website to be used in complex browser automation tasks.

Example

{
  "url": "https://example.com/"
}

curl -X POST \
  "https://production-sfo.browserless.io/content?token=YOUR_API_TOKEN_HERE" \
  -H 'Cache-Control: no-cache' \
  -H 'Content-Type: application/json' \
  -d '{
  "url": "https://example.com/"
}'

const TOKEN = "YOUR_API_TOKEN_HERE";
const url = `https://production-sfo.browserless.io/content?token=${TOKEN}`;
const headers = {
    "Cache-Control": "no-cache",
    "Content-Type": "application/json"
};
const data = {
    url: "https://example.com/"
};

const fetchContent = async () => {
    const response = await fetch(url, {
        method: 'POST',
        headers: headers,
        body: JSON.stringify(data)
    });

    const content = await response.text();
    console.log(content); 
};

fetchContent();

import requests

TOKEN = "YOUR_API_TOKEN_HERE"
url = f"https://production-sfo.browserless.io/content?token={TOKEN}"
headers = {
    "Cache-Control": "no-cache",
    "Content-Type": "application/json"
}
data = {
    "url": "https://example.com/"
}

response = requests.post(url, headers=headers, json=data)
print(response.text) 

import java.io.*;
import java.net.http.*;
import java.net.URI;

public class FetchContent {
    public static void main(String[] args) {
        String TOKEN = "YOUR_API_TOKEN_HERE";
        String url = "https://production-sfo.browserless.io/content?token=" + TOKEN;

        HttpClient client = HttpClient.newHttpClient();

        String jsonData = """
            {
                "url": "https://example.com/"
            }
        """;

        HttpRequest request = HttpRequest.newBuilder()
            .uri(URI.create(url))
            .header("Content-Type", "application/json")
            .header("Cache-Control", "no-cache")
            .POST(HttpRequest.BodyPublishers.ofString(jsonData))
            .build();

        try {
            HttpResponse<String> response = client.send(request, HttpResponse.BodyHandlers.ofString());
            System.out.println("Response: " + response.body());
        } catch (Exception e) {
            e.printStackTrace();
        }
    }
}

  using System.Text;
  using System.Text.Json;

  class Program
  {
      static async Task Main(string[] args)
      {
          string TOKEN = "YOUR_API_TOKEN_HERE";
          string url = $"https://production-sfo.browserless.io/content?token={TOKEN}";

          var payload = new
          {
              url = "https://example.com/"
          };

          var jsonContent = new StringContent(JsonSerializer.Serialize(payload), Encoding.UTF8, "application/json");

          using var httpClient = new HttpClient();
          try
          {
              var request = new HttpRequestMessage(HttpMethod.Post, url)
              {
                  Content = jsonContent
              };

              request.Headers.Add("Cache-Control", "no-cache");

              var response = await httpClient.SendAsync(request);
              response.EnsureSuccessStatusCode();
              var result = await response.Content.ReadAsStringAsync();
              Console.WriteLine("Response: " + result);
          }
          catch (Exception e)
          {
              Console.WriteLine("Error: " + e.Message);
          }
      }
  }

Additional Configuration Options

The /content API supports additional configuration options that are shared across multiple REST API endpoints. See the Request Configuration page for detailed documentation on:

Waiting for Things: Wait for events, functions, selectors, or timeouts before returning the response
Navigation Options: Customize navigation behavior using gotoOptions
Rejecting Undesired Requests: Block specific resources using rejectResourceTypes and rejectRequestPattern
Continue on Error: Use bestAttempt to proceed when async events fail or timeout

Bot Detection Troubleshooting

If you're experiencing issues with the /content API returning no data or partial data, this is typically due to bot detection mechanisms employed by the target website. Websites may use various techniques to detect and block automated browsers, which can result in empty responses or incomplete content.

Recognising failures from Anti Bot Mechanisms

Empty HTML response or minimal content
Partial page content missing key elements
Different content compared to what you see in a regular browser
Blocked requests or access denied messages

Alternative: Unblock API

When encountering bot detection issues, we recommend using the /unblock API as an alternative to the /content API. The /unblock endpoint is specifically designed to bypass bot detection mechanisms and can return HTML content directly in the response.

curl --request POST \
--url 'https://production-sfo.browserless.io/unblock?token=YOUR_API_TOKEN_HERE&proxy=residential' \
--header 'Content-Type: application/json' \
--data '{
  "url": "https://www.example.com/",
  "browserWSEndpoint": false,
  "cookies": false,
  "content": true,
  "screenshot": false
}'

The /unblock API is suitable for more adavanced bot detection bypass use-cases similar to:

Specialized unblocking: Designed specifically to bypass bot detection mechanisms like Datadome and passive CAPTCHAs
Direct content return: Returns HTML content directly in the response when content: true is set
Enhanced success rate: Works best when combined with residential proxies (&proxy=residential)
Simple integration: Provides the same content extraction functionality as the /content API

For more information about the /unblock API and its capabilities, see the /unblock API documentation.

Example​

Additional Configuration Options​

Bot Detection Troubleshooting​

Recognising failures from Anti Bot Mechanisms​

Alternative: Unblock API​

Example

Additional Configuration Options

Bot Detection Troubleshooting

Recognising failures from Anti Bot Mechanisms

Alternative: Unblock API