Skip to main content
Version: v2

Best Practices

This document outlines essential best practices for creating robust and reliable Puppeteer and Playwright scripts when using Browserless. These practices will help you avoid common pitfalls, improve performance, and create more maintainable automation code.

Optimize Your Page Navigation​

One of the most impactful optimizations you can make is changing how you handle page navigation. Instead of waiting for all network activity to cease, use domcontentloaded for faster execution when you don't need all resources loaded.

Use Appropriate waitUntil Options​

import puppeteer from "puppeteer-core";

const browser = await puppeteer.connect({
  browserWSEndpoint: `wss://production-sfo.browserless.io/?token=YOUR_API_TOKEN_HERE`,
});

const page = await browser.newPage();

// Fast navigation - use when you only need DOM content
await page.goto('https://example.com', {
  waitUntil: 'domcontentloaded',
  timeout: 30000
});

// Use networkidle2 only when you need all resources loaded
await page.goto('https://spa-app.com', {
  waitUntil: 'networkidle2',
  timeout: 60000
});

For detailed guidance on choosing the right waitUntil option, see our waitUntil blog post and timeout troubleshooting guide.

Monitor Network Requests and Failures​

Logging failed network requests is crucial for debugging issues, especially when dealing with dynamic content or API-dependent pages.

Implement Network Request Monitoring​

import puppeteer from "puppeteer-core";

const browser = await puppeteer.connect({
  browserWSEndpoint: `wss://production-sfo.browserless.io/?token=YOUR_API_TOKEN_HERE`,
});

const page = await browser.newPage();

// Monitor failed requests
page.on('requestfailed', request => {
  console.error(`Request failed: ${request.url()} - ${request.failure().errorText}`);
});

// Monitor response errors
page.on('response', response => {
  if (!response.ok()) {
    console.error(`Response error: ${response.url()} - ${response.status()}`);
  }
});

// Monitor page errors
page.on('pageerror', error => {
  console.error(`Page error: ${error.message}`);
});

await page.goto('https://example.com', { waitUntil: 'domcontentloaded' });

Handle Browser Disconnections​

When using remote browsers through Browserless, it's important to monitor for disconnection events and handle them gracefully.

Monitor Browser Connection Status​

import puppeteer from "puppeteer-core";

const browser = await puppeteer.connect({
  browserWSEndpoint: `wss://production-sfo.browserless.io/?token=YOUR_API_TOKEN_HERE`,
});

// Monitor browser disconnection
browser.on('disconnected', () => {
  console.error('Browser disconnected unexpectedly');
  // Implement reconnection logic or graceful shutdown
});

// Monitor target crashes
browser.on('targetdestroyed', target => {
  console.warn(`Target destroyed: ${target.url()}`);
});

const page = await browser.newPage();

try {
  await page.goto('https://example.com', { waitUntil: 'domcontentloaded' });
  // Your automation logic here
} catch (error) {
  console.error('Automation failed:', error.message);
} finally {
  if (browser.isConnected()) {
    await browser.close();
  }
}

Implement Separation of Concerns​

Create a helper class to abstract monitoring, logging, and common operations from your business logic. This makes your code more maintainable and reusable.

BrowserlessHelper Class Implementation​

import puppeteer from "puppeteer-core";

class BrowserlessHelper {
  constructor(token, region = 'production-sfo') {
    this.token = token;
    this.region = region;
    this.browser = null;
    this.page = null;
  }

  async connect() {
    this.browser = await puppeteer.connect({
      browserWSEndpoint: `wss://${this.region}.browserless.io/?token=${this.token}`,
    });

    this.setupBrowserMonitoring();
    return this.browser;
  }

  async createPage() {
    if (!this.browser) {
      throw new Error('Browser not connected. Call connect() first.');
    }

    this.page = await this.browser.newPage();
    this.setupPageMonitoring();
    return this.page;
  }

  setupBrowserMonitoring() {
    this.browser.on('disconnected', () => {
      console.error('Browser disconnected unexpectedly');
    });

    this.browser.on('targetdestroyed', target => {
      console.warn(`Target destroyed: ${target.url()}`);
    });
  }

  setupPageMonitoring() {
    this.page.on('requestfailed', request => {
      console.error(`Request failed: ${request.url()} - ${request.failure().errorText}`);
    });

    this.page.on('response', response => {
      if (!response.ok()) {
        console.error(`Response error: ${response.url()} - ${response.status()}`);
      }
    });

    this.page.on('pageerror', error => {
      console.error(`Page error: ${error.message}`);
    });
  }

  async navigateWithRetry(url, options = {}) {
    const defaultOptions = {
      waitUntil: 'domcontentloaded',
      timeout: 30000,
      ...options
    };

    const maxRetries = 3;
    let lastError;

    for (let i = 0; i < maxRetries; i++) {
      try {
        await this.page.goto(url, defaultOptions);
        return;
      } catch (error) {
        lastError = error;
        console.warn(`Navigation attempt ${i + 1} failed: ${error.message}`);
        
        if (i < maxRetries - 1) {
          await this.delay(1000 * (i + 1)); // Exponential backoff
        }
      }
    }

    throw lastError;
  }

  async screenshotOnError(error, filename = 'error-screenshot.png') {
    try {
      if (this.page) {
        await this.page.screenshot({ path: filename, fullPage: true });
        console.log(`Screenshot saved: ${filename}`);
      }
    } catch (screenshotError) {
      console.error('Failed to take screenshot:', screenshotError.message);
    }
  }

  async cleanup() {
    try {
      if (this.browser && this.browser.isConnected()) {
        await this.browser.close();
      }
    } catch (error) {
      console.error('Error during cleanup:', error.message);
    }
  }

  delay(ms) {
    return new Promise(resolve => setTimeout(resolve, ms));
  }
}

// Usage example
const helper = new BrowserlessHelper('YOUR_API_TOKEN_HERE');

try {
  await helper.connect();
  const page = await helper.createPage();
  
  await helper.navigateWithRetry('https://example.com');
  
  // Your automation logic here
  const title = await page.title();
  console.log('Page title:', title);
  
} catch (error) {
  console.error('Automation failed:', error.message);
  await helper.screenshotOnError(error);
} finally {
  await helper.cleanup();
}

Implement Security Best Practices​

Prevent your automation from accessing local files or sensitive network resources by implementing proper security measures.

Block Dangerous Requests​

import puppeteer from "puppeteer-core";

const browser = await puppeteer.connect({
  browserWSEndpoint: `wss://production-sfo.browserless.io/?token=YOUR_API_TOKEN_HERE`,
});

const page = await browser.newPage();

// Block dangerous requests
await page.setRequestInterception(true);

page.on('request', request => {
  const url = request.url();
  
  // Block file:// protocol requests
  if (url.startsWith('file://')) {
    console.warn(`Blocked file:// request: ${url}`);
    request.abort();
    return;
  }
  
  // Block localhost and private IP ranges
  const dangerousPatterns = [
    /^https?:\/\/localhost/,
    /^https?:\/\/127\./,
    /^https?:\/\/10\./,
    /^https?:\/\/172\.(1[6-9]|2[0-9]|3[01])\./,
    /^https?:\/\/192\.168\./,
    /^https?:\/\/169\.254\./ // Link-local addresses
  ];
  
  const isDangerous = dangerousPatterns.some(pattern => pattern.test(url));
  
  if (isDangerous) {
    console.warn(`Blocked potentially dangerous request: ${url}`);
    request.abort();
    return;
  }
  
  request.continue();
});

await page.goto('https://example.com', { waitUntil: 'domcontentloaded' });

Use Remote Browser Connections​

Browserless provides managed browser instances, eliminating the need to run browsers locally. This improves reliability, scalability, and resource management.

Regional Endpoint Optimization​

Choose the Browserless region closest to your application for optimal performance:

import puppeteer from "puppeteer-core";

// Choose the region closest to your application
const regions = {
  'us-west': 'production-sfo.browserless.io',
  'us-east': 'production-nyc.browserless.io',
  'europe': 'production-lon.browserless.io'
};

const browser = await puppeteer.connect({
  browserWSEndpoint: `wss://${regions['us-west']}/?token=YOUR_API_TOKEN_HERE&timeout=300000`,
});

// Use connection pooling for multiple pages
const pages = await Promise.all([
  browser.newPage(),
  browser.newPage(),
  browser.newPage()
]);

try {
  // Process multiple pages concurrently
  await Promise.all(pages.map(async (page, index) => {
    await page.goto(`https://example.com/page${index + 1}`, {
      waitUntil: 'domcontentloaded'
    });
    
    // Your page-specific logic here
    const title = await page.title();
    console.log(`Page ${index + 1} title:`, title);
  }));
} finally {
  await browser.close();
}

For more information about regional endpoints and performance optimization, see our slow response times troubleshooting guide.

Capture Screenshots on Failures​

Screenshots provide invaluable debugging context when automation fails. Implement automatic screenshot capture for all error scenarios.

Comprehensive Error Screenshot Strategy​

import puppeteer from "puppeteer-core";
import fs from 'fs';
import path from 'path';

class ScreenshotManager {
  constructor(screenshotDir = './screenshots') {
    this.screenshotDir = screenshotDir;
    this.ensureDirectoryExists();
  }

  ensureDirectoryExists() {
    if (!fs.existsSync(this.screenshotDir)) {
      fs.mkdirSync(this.screenshotDir, { recursive: true });
    }
  }

  generateFilename(prefix = 'error') {
    const timestamp = new Date().toISOString().replace(/[:.]/g, '-');
    return path.join(this.screenshotDir, `${prefix}-${timestamp}.png`);
  }

  async captureError(page, error, context = '') {
    const filename = this.generateFilename('error');
    
    try {
      await page.screenshot({
        path: filename,
        fullPage: true,
        type: 'png'
      });

      console.error(`Error occurred${context ? ` in ${context}` : ''}: ${error.message}`);
      console.log(`Screenshot saved: ${filename}`);
      
      return filename;
    } catch (screenshotError) {
      console.error('Failed to capture error screenshot:', screenshotError.message);
      return null;
    }
  }

  async captureStep(page, stepName) {
    const filename = this.generateFilename(`step-${stepName}`);
    
    try {
      await page.screenshot({
        path: filename,
        fullPage: true,
        type: 'png'
      });

      console.log(`Step screenshot saved: ${filename}`);
      return filename;
    } catch (error) {
      console.error('Failed to capture step screenshot:', error.message);
      return null;
    }
  }
}

// Usage example with comprehensive error handling
const browser = await puppeteer.connect({
  browserWSEndpoint: `wss://production-sfo.browserless.io/?token=YOUR_API_TOKEN_HERE`,
});

const page = await browser.newPage();
const screenshotManager = new ScreenshotManager();

try {
  // Navigation with screenshot on failure
  try {
    await page.goto('https://example.com', { 
      waitUntil: 'domcontentloaded',
      timeout: 30000 
    });
    await screenshotManager.captureStep(page, 'after-navigation');
  } catch (error) {
    await screenshotManager.captureError(page, error, 'navigation');
    throw error;
  }

  // Element interaction with screenshot on failure
  try {
    await page.waitForSelector('#submit-button', { timeout: 10000 });
    await page.click('#submit-button');
    await screenshotManager.captureStep(page, 'after-click');
  } catch (error) {
    await screenshotManager.captureError(page, error, 'button-click');
    throw error;
  }

  // Form submission with screenshot on failure
  try {
    await page.waitForSelector('#result', { timeout: 15000 });
    const result = await page.$eval('#result', el => el.textContent);
    console.log('Result:', result);
    await screenshotManager.captureStep(page, 'final-result');
  } catch (error) {
    await screenshotManager.captureError(page, error, 'result-extraction');
    throw error;
  }

} catch (error) {
  console.error('Automation failed:', error.message);
  // Final error screenshot if not already captured
  await screenshotManager.captureError(page, error, 'final-failure');
} finally {
  await browser.close();
}

Additional Browserless-Specific Best Practices​

Session Management​

Always close your browser sessions properly to avoid hitting concurrency limits:

import puppeteer from "puppeteer-core";

const browser = await puppeteer.connect({
  browserWSEndpoint: `wss://production-sfo.browserless.io/?token=YOUR_API_TOKEN_HERE`,
});

try {
  const page = await browser.newPage();
  await page.goto('https://example.com', { waitUntil: 'domcontentloaded' });
  
  // Your automation logic here
  
} catch (error) {
  console.error('Automation failed:', error.message);
} finally {
  // Always close the browser, even on errors
  if (browser.isConnected()) {
    await browser.close();
  }
}

Reduce Network Round-trips​

Minimize await calls by using page.evaluate() for multiple DOM operations:

// DON'T DO - Multiple round-trips
const button = await page.$('.buy-now');
const buttonText = await button.getProperty('innerText');
const isVisible = await button.isIntersectingViewport();
await button.click();

// DO - Single round-trip
const result = await page.evaluate(() => {
  const button = document.querySelector('.buy-now');
  const buttonText = button.innerText;
  const isVisible = button.offsetParent !== null;
  button.click();
  
  return { buttonText, isVisible, clicked: true };
});

Use Pre-session Checks​

Monitor your account's capacity before starting sessions to avoid rejections:

note

The pressure API is only available for dedicated machines on enterprise plans. For shared plans, sessions will automatically queue when capacity is reached.

// Check capacity using the GraphQL API
const response = await fetch('https://api.browserless.io/graphql', {
  method: 'POST',
  headers: {
    'Content-Type': 'application/json',
  },
  body: JSON.stringify({
    query: `
      query {
        pressure(apiToken: "YOUR_API_TOKEN_HERE") {
          running
          recentlyRejected
          queued
          isAvailable
          date
        }
      }
    `
  })
});

const data = await response.json();
const { isAvailable, running, queued } = data.data.pressure;

if (!isAvailable) {
  console.warn(`Browserless capacity full. Running: ${running}, Queued: ${queued}`);
  // Implement retry logic or queue management
}

Getting Help​

If you continue to experience issues after implementing these best practices:

  1. Check your account dashboard for usage metrics
  2. Review our troubleshooting guides for specific issues
  3. Contact Browserless support for assistance

These best practices will help you create more reliable, maintainable, and efficient browser automation scripts with Browserless.