Skip to main content

Calling the BrowserQL API

Once you have set up your query, you can export it to be called via an API.

  1. Click the Export as Code button.
  2. Choose your preferred language.

The export will include the endpoint, token, headers, and payload.

Of course, you might need multiple API calls for a scraping session, such as one to crawl the search results then a second to grab the details from each page.

const { JSDOM } = require("jsdom");

async function ExtractJobsHTML() {
    const url = "https://production-sfo.browserless.io/chromium/bql?token=YOUR_API_TOKEN_HERE";
    const options = {
      method: 'POST',
      body: '{"query":"mutation ExtractJobsHTML {\n  goto(url: \"https://news.ycombinator.com\", waitUntil: firstMeaningfulPaint) {\n    status\n  }\n  \n  click(selector: \"span.pagetop > a[href=\'jobs\']\") {\n    x\n    y\n  }\n  \n  html {\n    html\n  }\n}","variables":null,"operationName":"ExtractJobsHTML"}'
    };

    let response;

    try {
      response = await fetch(url, options);
    } catch (error) {
      console.error(error);
    }

    const json = await response.json();
    const html = json.data.html.html;

    const dom = new JSDOM(html);
    const document = dom.window.document;
    const rows = document.querySelectorAll("tr.athing");

    const titlesAndLinks = [];
    rows.forEach((row) => {
        const anchor = row.querySelector("a");
        if (anchor) {
            titlesAndLinks.push({
                title: anchor.textContent.trim(),
                link: anchor.href.trim(),
            });
        }
    });

    console.log(titlesAndLinks);
}

ExtractJobsHTML();