/content API
We recommended using BrowserQL, Browserless' first-class browser automation API, to capture content from any website.
The content API allows for simple navigation to a site and capturing the page's content (including the <head>
section). Browserless will respond with a Content-Type
of text/html
, and string of the site's HTML after it has been rendered and evaluated inside the browser. This is useful for capturing the content of a page that has a lot of JavaScript or other interactivity.
You can check the full Open API schema here.
Basic Usage
- JSON payload
- cURL
- Javascript
- Python
- Java
- C#
{
"url": "https://example.com/"
}
curl -X POST \
https://production-sfo.browserless.io/content?token=YOUR_API_TOKEN_HERE \
-H 'Cache-Control: no-cache' \
-H 'Content-Type: application/json' \
-d '{
"url": "https://example.com/"
}'
const TOKEN = "YOUR_API_TOKEN_HERE";
const url = `https://production-sfo.browserless.io/content?token=${TOKEN}`;
const headers = {
"Cache-Control": "no-cache",
"Content-Type": "application/json"
};
const data = {
url: "https://example.com/"
};
const fetchContent = async () => {
const response = await fetch(url, {
method: 'POST',
headers: headers,
body: JSON.stringify(data)
});
const content = await response.text();
console.log(content);
};
fetchContent();
import requests
TOKEN = "YOUR_API_TOKEN_HERE"
url = f"https://production-sfo.browserless.io/content?token={TOKEN}"
headers = {
"Cache-Control": "no-cache",
"Content-Type": "application/json"
}
data = {
"url": "https://example.com/"
}
response = requests.post(url, headers=headers, json=data)
print(response.text)
import java.io.*;
import java.net.http.*;
import java.net.URI;
public class FetchContent {
public static void main(String[] args) {
String TOKEN = "YOUR_API_TOKEN_HERE";
String url = "https://production-sfo.browserless.io/content?token=" + TOKEN;
HttpClient client = HttpClient.newHttpClient();
String jsonData = """
{
"url": "https://example.com/"
}
""";
HttpRequest request = HttpRequest.newBuilder()
.uri(URI.create(url))
.header("Content-Type", "application/json")
.header("Cache-Control", "no-cache")
.POST(HttpRequest.BodyPublishers.ofString(jsonData))
.build();
try {
HttpResponse<String> response = client.send(request, HttpResponse.BodyHandlers.ofString());
System.out.println("Response: " + response.body());
} catch (Exception e) {
e.printStackTrace();
}
}
}
using System.Text;
using System.Text.Json;
class Program
{
static async Task Main(string[] args)
{
string TOKEN = "YOUR_API_TOKEN_HERE";
string url = $"https://production-sfo.browserless.io/content?token={TOKEN}";
var payload = new
{
url = "https://example.com/"
};
var jsonContent = new StringContent(JsonSerializer.Serialize(payload), Encoding.UTF8, "application/json");
using var httpClient = new HttpClient();
try
{
var request = new HttpRequestMessage(HttpMethod.Post, url)
{
Content = jsonContent
};
request.Headers.Add("Cache-Control", "no-cache");
var response = await httpClient.SendAsync(request);
response.EnsureSuccessStatusCode();
var result = await response.Content.ReadAsStringAsync();
Console.WriteLine("Response: " + result);
}
catch (Exception e)
{
Console.WriteLine("Error: " + e.Message);
}
}
}
Rejecting Undesired Requests
You can use rejectResourceTypes
and rejectRequestPattern
to block undesired content, resources and requests.
- JSON payload
- cURL
- Javascript
- Python
- Java
- C#
// Will reject any images and .css files
{
"url": "https://browserless.io/"
"rejectResourceTypes": ["image"],
"rejectRequestPattern": ["/^.*\\.(css)"]
}
curl -X POST \
https://production-sfo.browserless.io/content?token=YOUR_API_TOKEN_HERE \
-H 'Cache-Control: no-cache' \
-H 'Content-Type: application/json' \
-d '{
"url": "https://browserless.io/"
"rejectResourceTypes": ["image"],
"rejectRequestPattern": ["/^.*\\.(css)"]
}'
const TOKEN = "YOUR_API_TOKEN_HERE";
const url = `https://production-sfo.browserless.io/content?token=${TOKEN}`;
const headers = {
"Cache-Control": "no-cache",
"Content-Type": "application/json"
};
const data = {
url: "https://example.com/",
rejectResourceTypes: ["image"],
rejectRequestPattern: ["/^.*\\.(css)"]
};
const fetchContent = async () => {
const response = await fetch(url, {
method: 'POST',
headers: headers,
body: JSON.stringify(data)
});
const content = await response.text();
console.log(content);
};
fetchContent();
import requests
TOKEN = "YOUR_API_TOKEN_HERE"
url = f"https://production-sfo.browserless.io/content?token={TOKEN}"
headers = {
"Cache-Control": "no-cache",
"Content-Type": "application/json"
}
data = {
"url": "https://example.com/",
"rejectResourceTypes": ["image"],
"rejectRequestPattern": ["/^.*\\.(css)"]
}
response = requests.post(url, headers=headers, json=data)
print(response.text)
import java.io.*;
import java.net.http.*;
import java.net.URI;
import com.google.gson.*;
public class FetchContentWithRejection {
public static void main(String[] args) {
String TOKEN = "YOUR_API_TOKEN_HERE";
String url = "https://production-sfo.browserless.io/content?token=" + TOKEN;
HttpClient client = HttpClient.newHttpClient();
String jsonData = new Gson().toJson(Map.of(
"url", "https://browserless.io/",
"rejectResourceTypes", new String[] { "image" },
"rejectRequestPattern", new String[] { "/^.*\\.(css)/" }
));
HttpRequest request = HttpRequest.newBuilder()
.uri(URI.create(url))
.header("Content-Type", "application/json")
.header("Cache-Control", "no-cache")
.POST(HttpRequest.BodyPublishers.ofString(jsonData))
.build();
try {
HttpResponse<String> response = client.send(request, HttpResponse.BodyHandlers.ofString());
System.out.println("Response: " + response.body());
} catch (Exception e) {
e.printStackTrace();
}
}
}
using System.Text;
using System.Text.Json;
class Program
{
static async Task Main(string[] args)
{
string TOKEN = "YOUR_API_TOKEN_HERE";
string url = $"https://production-sfo.browserless.io/content?token={TOKEN}";
var payload = new
{
url = "https://browserless.io/",
rejectResourceTypes = new[] { "image" },
rejectRequestPattern = new[] { "/^.*\\.(css)/" }
};
var jsonContent = new StringContent(JsonSerializer.Serialize(payload), Encoding.UTF8, "application/json");
using var httpClient = new HttpClient();
try
{
var request = new HttpRequestMessage(HttpMethod.Post, url)
{
Content = jsonContent
};
request.Headers.Add("Cache-Control", "no-cache");
var response = await httpClient.SendAsync(request);
response.EnsureSuccessStatusCode();
var result = await response.Content.ReadAsStringAsync();
Console.WriteLine("Response: " + result);
}
catch (Exception e)
{
Console.WriteLine("Error: " + e.Message);
}
}
}
Navigation Options
You can use the gotoOptions
to modify the default navigation behavior for the requested URL. The objects mirror Puppeteer's GoToOptions
interface.
- JSON payload
- cURL
- Javascript
- Python
- Java
- C#
{
"url": "https://example.com/",
"gotoOptions": { "waitUntil": "networkidle2" },
}
curl -X POST \
https://production-sfo.browserless.io/content?token=YOUR_API_TOKEN_HERE \
-H 'Cache-Control: no-cache' \
-H 'Content-Type: application/json' \
-d '{
"url": "https://example.com/",
"gotoOptions": { "waitUntil": "networkidle2" }
}'
const TOKEN = "YOUR_API_TOKEN_HERE";
const url = `https://production-sfo.browserless.io/content?token=${TOKEN}`;
const headers = {
"Cache-Control": "no-cache",
"Content-Type": "application/json"
};
const data = {
url: "https://example.com/",
gotoOptions: { "waitUntil": "networkidle2" },
};
const fetchContent = async () => {
const response = await fetch(url, {
method: 'POST',
headers: headers,
body: JSON.stringify(data)
});
const content = await response.text();
console.log(content);
};
fetchContent();
import requests
TOKEN = "YOUR_API_TOKEN_HERE"
url = f"https://production-sfo.browserless.io/content?token={TOKEN}"
headers = {
"Cache-Control": "no-cache",
"Content-Type": "application/json"
}
data = {
"url": "https://example.com/",
"gotoOptions": { "waitUntil": "networkidle2" }
}
response = requests.post(url, headers=headers, json=data)
print(response.text)
import java.io.*;
import java.net.http.*;
import java.net.URI;
import com.google.gson.*;
public class FetchContent {
public static void main(String[] args) {
String TOKEN = "YOUR_API_TOKEN_HERE";
String url = "https://production-sfo.browserless.io/content?token=" + TOKEN;
HttpClient client = HttpClient.newHttpClient();
String jsonData = new Gson().toJson(Map.of(
"url", "https://example.com/",
"gotoOptions", Map.of("waitUntil", "networkidle2")
));
HttpRequest request = HttpRequest.newBuilder()
.uri(URI.create(url))
.header("Content-Type", "application/json")
.header("Cache-Control", "no-cache")
.POST(HttpRequest.BodyPublishers.ofString(jsonData))
.build();
try {
HttpResponse<String> response = client.send(request, HttpResponse.BodyHandlers.ofString());
System.out.println("Response: " + response.body());
} catch (Exception e) {
e.printStackTrace();
}
}
}
using System.Text;
using System.Text.Json;
class Program
{
static async Task Main(string[] args)
{
string TOKEN = "YOUR_API_TOKEN_HERE";
string url = $"https://production-sfo.browserless.io/content?token={TOKEN}";
var payload = new
{
url = "https://example.com/",
gotoOptions = new { waitUntil = "networkidle2" }
};
var jsonContent = new StringContent(JsonSerializer.Serialize(payload), Encoding.UTF8, "application/json");
using var httpClient = new HttpClient();
try
{
var request = new HttpRequestMessage(HttpMethod.Post, url)
{
Content = jsonContent
};
request.Headers.Add("Cache-Control", "no-cache");
var response = await httpClient.SendAsync(request);
response.EnsureSuccessStatusCode();
var result = await response.Content.ReadAsStringAsync();
Console.WriteLine("Response: " + result);
}
catch (Exception e)
{
Console.WriteLine("Error: " + e.Message);
}
}
}
Continue on error
You can use bestAttempt
to make Browserless attempt to proceed when async events fail or timeout. This includes things like the goto
or waitForSelector
proprieties in the JSON payload.
- JSON payload
- cURL
- Javascript
- Python
- Java
- C#
{
"url": "https://example.com/",
"bestAttempt": true,
// This would fail without bestAttempt
"waitForSelector": { "selector": "table", "timeout": 500 }
}
curl -X POST \
https://production-sfo.browserless.io/content?token=YOUR_API_TOKEN_HERE \
-H 'Cache-Control: no-cache' \
-H 'Content-Type: application/json' \
-d '{
"url": "https://example.com/",
"bestAttempt": true,
"waitForSelector": { "selector": "table", "timeout": 500 }
}'
const TOKEN = "YOUR_API_TOKEN_HERE";
const url = `https://production-sfo.browserless.io/content?token=${TOKEN}`;
const headers = {
"Cache-Control": "no-cache",
"Content-Type": "application/json"
};
const data = {
url: "https://example.com/",
bestAttempt: true,
waitForSelector: { "selector": "table", "timeout": 500 }
};
const fetchContent = async () => {
const response = await fetch(url, {
method: 'POST',
headers: headers,
body: JSON.stringify(data)
});
const content = await response.text();
console.log(content);
};
fetchContent();
import requests
TOKEN = "YOUR_API_TOKEN_HERE"
url = f"https://production-sfo.browserless.io/content?token={TOKEN}"
headers = {
"Cache-Control": "no-cache",
"Content-Type": "application/json"
}
data = {
"url": "https://example.com/",
"bestAttempt": True,
"waitForSelector": { "selector": "table", "timeout": 500 }
}
response = requests.post(url, headers=headers, json=data)
print(response.text)
import java.io.*;
import java.net.http.*;
import java.net.URI;
import com.google.gson.*;
public class FetchContentWithWait {
public static void main(String[] args) {
String TOKEN = "YOUR_API_TOKEN_HERE";
String url = "https://production-sfo.browserless.io/content?token=" + TOKEN;
HttpClient client = HttpClient.newHttpClient();
String jsonData = new Gson().toJson(Map.of(
"url", "https://example.com/",
"bestAttempt", true,
"waitForSelector", Map.of(
"selector", "table",
"timeout", 500
)
));
HttpRequest request = HttpRequest.newBuilder()
.uri(URI.create(url))
.header("Content-Type", "application/json")
.header("Cache-Control", "no-cache")
.POST(HttpRequest.BodyPublishers.ofString(jsonData))
.build();
try {
HttpResponse<String> response = client.send(request, HttpResponse.BodyHandlers.ofString());
System.out.println("Response: " + response.body());
} catch (Exception e) {
e.printStackTrace();
}
}
}
using System.Text;
using System.Text.Json;
class Program
{
static async Task Main(string[] args)
{
string TOKEN = "YOUR_API_TOKEN_HERE";
string url = $"https://production-sfo.browserless.io/content?token={TOKEN}";
var payload = new
{
url = "https://example.com/",
bestAttempt = true,
waitForSelector = new
{
selector = "table",
timeout = 500
}
};
var jsonContent = new StringContent(JsonSerializer.Serialize(payload), Encoding.UTF8, "application/json");
using var httpClient = new HttpClient();
try
{
var request = new HttpRequestMessage(HttpMethod.Post, url)
{
Content = jsonContent
};
request.Headers.Add("Cache-Control", "no-cache");
var response = await httpClient.SendAsync(request);
response.EnsureSuccessStatusCode();
var result = await response.Content.ReadAsStringAsync();
Console.WriteLine("Response: " + result);
}
catch (Exception e)
{
Console.WriteLine("Error: " + e.Message);
}
}
}
Waiting for Things
Browserless offers 4 different ways to wait for preconditions to be met on page. These are events, functions, selectors and timeouts.
waitForEvent
Waits for an event to happen on the page before cotinue
Example
- JSON payload
- cURL
- Javascript
- Python
- Java
- C#
// Will fail since the event never fires
{
"url": "https://example.com/",
"waitForEvent": {
"event": "fullscreenchange",
"timeout": 5000
}
}
curl -X POST \
https://production-sfo.browserless.io/content?token=YOUR_API_TOKEN_HERE \
-H 'Cache-Control: no-cache' \
-H 'Content-Type: application/json' \
-d '{
"url": "https://example.com/",
"waitForEvent": {
"event": "fullscreenchange",
"timeout": 5000
}
}'
const TOKEN = "YOUR_API_TOKEN_HERE";
const url = `https://production-sfo.browserless.io/content?token=${TOKEN}`;
const headers = {
"Cache-Control": "no-cache",
"Content-Type": "application/json"
};
const data = {
url: "https://example.com/",
waitForEvent: {
event: "fullscreenchange",
timeout: 5000
}
};
const fetchContent = async () => {
const response = await fetch(url, {
method: 'POST',
headers: headers,
body: JSON.stringify(data)
});
const content = await response.text();
console.log(content);
};
fetchContent();
import requests
TOKEN = "YOUR_API_TOKEN_HERE"
url = f"https://production-sfo.browserless.io/content?token={TOKEN}"
headers = {
"Cache-Control": "no-cache",
"Content-Type": "application/json"
}
data = {
"url": "https://example.com/",
"waitForEvent": {
"event": "fullscreenchange",
"timeout": 5000
}
}
response = requests.post(url, headers=headers, json=data)
print(response.text)
import java.io.*;
import java.net.http.*;
import java.net.URI;
import com.google.gson.*;
public class FetchContentWithEvent {
public static void main(String[] args) {
String TOKEN = "YOUR_API_TOKEN_HERE";
String url = "https://production-sfo.browserless.io/content?token=" + TOKEN;
HttpClient client = HttpClient.newHttpClient();
String jsonData = new Gson().toJson(Map.of(
"url", "https://example.com/",
"waitForEvent", Map.of(
"event", "fullscreenchange",
"timeout", 5000
)
));
HttpRequest request = HttpRequest.newBuilder()
.uri(URI.create(url))
.header("Content-Type", "application/json")
.header("Cache-Control", "no-cache")
.POST(HttpRequest.BodyPublishers.ofString(jsonData))
.build();
try {
HttpResponse<String> response = client.send(request, HttpResponse.BodyHandlers.ofString());
System.out.println("Response: " + response.body());
} catch (Exception e) {
e.printStackTrace();
}
}
}
using System.Text;
using System.Text.Json;
class Program
{
static async Task Main(string[] args)
{
string TOKEN = "YOUR_API_TOKEN_HERE";
string url = $"https://production-sfo.browserless.io/content?token={TOKEN}";
var payload = new
{
url = "https://example.com/",
waitForEvent = new
{
@event = "fullscreenchange",
timeout = 5000
}
};
var jsonContent = new StringContent(JsonSerializer.Serialize(payload), Encoding.UTF8, "application/json");
using var httpClient = new HttpClient();
try
{
var request = new HttpRequestMessage(HttpMethod.Post, url)
{
Content = jsonContent
};
request.Headers.Add("Cache-Control", "no-cache");
var response = await httpClient.SendAsync(request);
response.EnsureSuccessStatusCode();
var result = await response.Content.ReadAsStringAsync();
Console.WriteLine("Response: " + result);
}
catch (Exception e)
{
Console.WriteLine("Error: " + e.Message);
}
}
}
waitForFunction
Waits for the provided function to return before cotinue. The function can be any valid JavaScript or EcmaScript function, and async
functions are supported.
Example
JS function
async () => {
const res = await fetch('https://jsonplaceholder.typicode.com/todos/1');
const json = await res.json();
document.querySelector("h1").innerText = json.title;
}
- JSON payload
- cURL
- Javascript
- Python
- Java
- C#
{
"url": "https://example.com/",
"waitForFunction": {
"fn": "async()=>{let t=await fetch('https://jsonplaceholder.typicode.com/todos/1'),e=await t.json();document.querySelector('h1').innerText=e.title}",
"timeout": 5000
}
}
curl -X POST \
https://production-sfo.browserless.io/content?token=YOUR_API_TOKEN_HERE \
-H 'Cache-Control: no-cache' \
-H 'Content-Type: application/json' \
-d '{
"url": "https://example.com/",
"waitForFunction": {
"fn": "async()=>{let t=await fetch('https://jsonplaceholder.typicode.com/todos/1'),e=await t.json();document.querySelector('h1').innerText=e.title}",
"timeout": 5000
}
}'
const TOKEN = "YOUR_API_TOKEN_HERE";
const url = `https://production-sfo.browserless.io/content?token=${TOKEN}`;
const headers = {
"Cache-Control": "no-cache",
"Content-Type": "application/json"
};
const data = {
url: "https://example.com/",
waitForFunction: {
fn: "async()=>{let t=await fetch('https://jsonplaceholder.typicode.com/todos/1'),e=await t.json();document.querySelector('h1').innerText=e.title}",
timeout: 5000
}
};
const fetchContent = async () => {
const response = await fetch(url, {
method: 'POST',
headers: headers,
body: JSON.stringify(data)
});
const content = await response.text();
console.log(content);
};
fetchContent();
import requests
TOKEN = "YOUR_API_TOKEN_HERE"
url = f"https://production-sfo.browserless.io/content?token={TOKEN}"
headers = {
"Cache-Control": "no-cache",
"Content-Type": "application/json"
}
data = {
"url": "https://example.com/",
"waitForFunction": {
"fn": "async()=>{let t=await fetch('https://jsonplaceholder.typicode.com/todos/1'),e=await t.json();document.querySelector('h1').innerText=e.title}",
"timeout": 5000
}
}
response = requests.post(url, headers=headers, json=data)
print(response.text)
import java.io.*;
import java.net.http.*;
import java.net.URI;
import com.google.gson.*;
public class FetchContentWithFunction {
public static void main(String[] args) {
String TOKEN = "YOUR_API_TOKEN_HERE";
String url = "https://production-sfo.browserless.io/content?token=" + TOKEN;
HttpClient client = HttpClient.newHttpClient();
String jsonData = new Gson().toJson(Map.of(
"url", "https://example.com/",
"waitForFunction", Map.of(
"fn", "async()=>{let t=await fetch('https://jsonplaceholder.typicode.com/todos/1'),e=await t.json();document.querySelector('h1').innerText=e.title}",
"timeout", 5000
)
));
HttpRequest request = HttpRequest.newBuilder()
.uri(URI.create(url))
.header("Content-Type", "application/json")
.header("Cache-Control", "no-cache")
.POST(HttpRequest.BodyPublishers.ofString(jsonData))
.build();
try {
HttpResponse<String> response = client.send(request, HttpResponse.BodyHandlers.ofString());
System.out.println("Response: " + response.body());
} catch (Exception e) {
e.printStackTrace();
}
}
}
using System.Text;
using System.Text.Json;
class Program
{
static async Task Main(string[] args)
{
string TOKEN = "YOUR_API_TOKEN_HERE";
string url = $"https://production-sfo.browserless.io/content?token={TOKEN}";
var payload = new
{
url = "https://example.com/",
waitForFunction = new
{
fn = "async()=>{let t=await fetch('https://jsonplaceholder.typicode.com/todos/1'),e=await t.json();document.querySelector('h1').innerText=e.title}",
timeout = 5000
}
};
var jsonContent = new StringContent(JsonSerializer.Serialize(payload), Encoding.UTF8, "application/json");
using var httpClient = new HttpClient();
try
{
var request = new HttpRequestMessage(HttpMethod.Post, url)
{
Content = jsonContent
};
request.Headers.Add("Cache-Control", "no-cache");
var response = await httpClient.SendAsync(request);
response.EnsureSuccessStatusCode();
var result = await response.Content.ReadAsStringAsync();
Console.WriteLine("Response: " + result);
}
catch (Exception e)
{
Console.WriteLine("Error: " + e.Message);
}
}
}
waitForSelector
Wait for a selector to appear in page. If at the moment of calling the method the selector already exists, the method will return immediately. If the selector doesn't appear after the timeout milliseconds of waiting, the function will throw.
The object can have any of these values:
selector
: String, required — A valid CSS selector.hidden
Boolean, optional — Wait for the selected element to not be found in the DOM or to be hidden, i.e. havedisplay: none
orvisibility: hidden
CSS properties.timeout
: Number, optional — Maximum number of milliseconds to wait for the selector before failing.visible
: Boolean, optional — Wait for the selected element to be present in DOM and to be visible, i.e. to not havedisplay: none
orvisibility: hidden
CSS properties.
Example
- JSON payload
- cURL
- Javascript
- Python
- Java
- C#
// Will fail since the event never fires
{
"url": "https://example.com/",
"waitForEvent": {
"event": "fullscreenchange",
"timeout": 5000
}
}
curl -X POST \
https://production-sfo.browserless.io/content?token=YOUR_API_TOKEN_HERE \
-H 'Cache-Control: no-cache' \
-H 'Content-Type: application/json' \
-d '{
"url": "https://example.com/",
"waitForEvent": {
"event": "fullscreenchange",
"timeout": 5000
}
}'
const TOKEN = "YOUR_API_TOKEN_HERE";
const url = `https://production-sfo.browserless.io/content?token=${TOKEN}`;
const headers = {
"Cache-Control": "no-cache",
"Content-Type": "application/json"
};
const data = {
url: "https://example.com/",
waitForEvent: {
event: "fullscreenchange",
timeout: 5000
}
};
const fetchContent = async () => {
const response = await fetch(url, {
method: 'POST',
headers: headers,
body: JSON.stringify(data)
});
const content = await response.text();
console.log(content);
};
fetchContent();
import requests
TOKEN = "YOUR_API_TOKEN_HERE"
url = f"https://production-sfo.browserless.io/content?token={TOKEN}"
headers = {
"Cache-Control": "no-cache",
"Content-Type": "application/json"
}
data = {
"url": "https://example.com/",
"waitForEvent": {
"event": "fullscreenchange",
"timeout": 5000
}
}
response = requests.post(url, headers=headers, json=data)
print(response.text)
import java.io.*;
import java.net.http.*;
import java.net.URI;
import com.google.gson.*;
public class FetchContentWithWaitForEvent {
public static void main(String[] args) {
String TOKEN = "YOUR_API_TOKEN_HERE";
String url = "https://production-sfo.browserless.io/content?token=" + TOKEN;
HttpClient client = HttpClient.newHttpClient();
String jsonData = new Gson().toJson(Map.of(
"url", "https://example.com/",
"waitForEvent", Map.of(
"event", "fullscreenchange",
"timeout", 5000
)
));
HttpRequest request = HttpRequest.newBuilder()
.uri(URI.create(url))
.header("Content-Type", "application/json")
.header("Cache-Control", "no-cache")
.POST(HttpRequest.BodyPublishers.ofString(jsonData))
.build();
try {
HttpResponse<String> response = client.send(request, HttpResponse.BodyHandlers.ofString());
System.out.println("Response: " + response.body());
} catch (Exception e) {
e.printStackTrace();
}
}
}
using System.Text;
using System.Text.Json;
class Program
{
static async Task Main(string[] args)
{
string TOKEN = "YOUR_API_TOKEN_HERE";
string url = $"https://production-sfo.browserless.io/content?token={TOKEN}";
var payload = new
{
url = "https://example.com/",
waitForEvent = new
{
@event = "fullscreenchange",
timeout = 5000
}
};
var jsonContent = new StringContent(JsonSerializer.Serialize(payload), Encoding.UTF8, "application/json");
using var httpClient = new HttpClient();
try
{
var request = new HttpRequestMessage(HttpMethod.Post, url)
{
Content = jsonContent
};
request.Headers.Add("Cache-Control", "no-cache");
var response = await httpClient.SendAsync(request);
response.EnsureSuccessStatusCode();
var result = await response.Content.ReadAsStringAsync();
Console.WriteLine("Response: " + result);
}
catch (Exception e)
{
Console.WriteLine("Error: " + e.Message);
}
}
}
waitForTimeout
Waits for a specified timeout before continuing.
Example
- JSON payload
- cURL
- Javascript
- Python
- Java
- C#
{
"url": "https://example.com/",
"waitForTimeout": 10000
}
curl -X POST \
https://production-sfo.browserless.io/content?token=YOUR_API_TOKEN_HERE \
-H 'Cache-Control: no-cache' \
-H 'Content-Type: application/json' \
-d '{
"url": "https://example.com/",
"waitForTimeout": 10000
}'
const TOKEN = "YOUR_API_TOKEN_HERE";
const url = `https://production-sfo.browserless.io/content?token=${TOKEN}`;
const headers = {
"Cache-Control": "no-cache",
"Content-Type": "application/json"
};
const data = {
url: "https://example.com/",
waitForTimeout: 10000
};
const fetchContent = async () => {
const response = await fetch(url, {
method: 'POST',
headers: headers,
body: JSON.stringify(data)
});
const content = await response.text();
console.log(content);
};
fetchContent();
import requests
TOKEN = "YOUR_API_TOKEN_HERE"
url = f"https://production-sfo.browserless.io/content?token={TOKEN}"
headers = {
"Cache-Control": "no-cache",
"Content-Type": "application/json"
}
data = {
"url": "https://example.com/",
"waitForTimeout": 10000
}
response = requests.post(url, headers=headers, json=data)
print(response.text)
import java.io.*;
import java.net.http.*;
import java.net.URI;
import com.google.gson.*;
public class FetchContentWithWaitForTimeout {
public static void main(String[] args) {
String TOKEN = "YOUR_API_TOKEN_HERE";
String url = "https://production-sfo.browserless.io/content?token=" + TOKEN;
HttpClient client = HttpClient.newHttpClient();
String jsonData = new Gson().toJson(Map.of(
"url", "https://example.com/",
"waitForTimeout", 10000
));
HttpRequest request = HttpRequest.newBuilder()
.uri(URI.create(url))
.header("Content-Type", "application/json")
.header("Cache-Control", "no-cache")
.POST(HttpRequest.BodyPublishers.ofString(jsonData))
.build();
try {
HttpResponse<String> response = client.send(request, HttpResponse.BodyHandlers.ofString());
System.out.println("Response: " + response.body());
} catch (Exception e) {
e.printStackTrace();
}
}
}
using System.Text;
using System.Text.Json;
class Program
{
static async Task Main(string[] args)
{
string TOKEN = "YOUR_API_TOKEN_HERE";
string url = $"https://production-sfo.browserless.io/content?token={TOKEN}";
var payload = new
{
url = "https://example.com/",
waitForTimeout = 10000
};
var jsonContent = new StringContent(JsonSerializer.Serialize(payload), Encoding.UTF8, "application/json");
using var httpClient = new HttpClient();
try
{
var request = new HttpRequestMessage(HttpMethod.Post, url)
{
Content = jsonContent
};
request.Headers.Add("Cache-Control", "no-cache");
var response = await httpClient.SendAsync(request);
response.EnsureSuccessStatusCode();
var result = await response.Content.ReadAsStringAsync();
Console.WriteLine("Response: " + result);
}
catch (Exception e)
{
Console.WriteLine("Error: " + e.Message);
}
}
}
Bot Detection Troubleshooting
If you're experiencing issues with the /content
API returning no data or partial data, this is typically due to bot detection mechanisms employed by the target website. Websites may use various techniques to detect and block automated browsers, which can result in empty responses or incomplete content.
Symptoms of Bot Detection
- Empty HTML response or minimal content
- Partial page content missing key elements
- Different content compared to what you see in a regular browser
- Blocked requests or access denied messages
Unblock API Alternative
When encountering bot detection issues, we recommend using the /unblock API as an alternative to the /content
API. The /unblock
endpoint is specifically designed to bypass bot detection mechanisms and can return HTML content directly in the response.
curl --request POST \
--url 'https://production-sfo.browserless.io/unblock?token=YOUR_API_TOKEN_HERE&proxy=residential' \
--header 'Content-Type: application/json' \
--data '{
"url": "https://www.example.com/",
"browserWSEndpoint": false,
"cookies": false,
"content": true,
"screenshot": false
}'
The /unblock
API offers several advantages for bot detection bypass:
- Specialized unblocking: Designed specifically to bypass bot detection mechanisms like Datadome and passive CAPTCHAs
- Direct content return: Returns HTML content directly in the response when
content: true
is set - Enhanced success rate: Works best when combined with residential proxies (
&proxy=residential
) - Simple integration: Provides the same content extraction functionality as the
/content
API
For more information about the /unblock
API and its capabilities, see the /unblock API documentation.