For AI agents: a documentation index is available at /llms.txt
Skip to main content

Anthropic Computer Use Integration

Anthropic Computer Use lets Claude drive a browser from screenshots. It takes screenshots, moves the cursor, clicks, and types. This guide connects that loop to a Browserless cloud browser over Playwright, so you get stealth mode, residential proxies, and regional endpoints without running Chrome yourself.

Prerequisites
  • Browserless API token (available in your account dashboard)
  • Anthropic API key (from your Claude Console)
  • Node.js 20.6+ (for TypeScript) or Python 3.10+ (for Python)

How it works​

This integration uses a screenshot + action loop with Playwright over CDP:

  1. Connect: Playwright connects to Browserless over a CDP WebSocket.
  2. Send the task: you send Claude the task, a screenshot of the page, and the computer use toolset.
  3. Execute actions: Claude replies with one or more tool_use blocks (click, type, scroll). You run them in order through Playwright.
  4. Return results: you answer every block with a tool_result, including a fresh screenshot when Claude asks for one.
  5. Repeat until Claude stops requesting actions.

Computer use is a client-side tool. Anthropic never touches the browser, so you write the loop that connects Claude to it. If you'd rather have Claude call Browserless tools server-side with no browser code at all, see the Anthropic SDK page.

Step-by-step setup​

  1. Get your API keys

    Go to your Browserless account dashboard and copy your API token.

    Then set your environment variables:

    BROWSERLESS_API_KEY=your-browserless-token
    ANTHROPIC_API_KEY=your-anthropic-key

  2. Install dependencies

    npm install playwright-core @anthropic-ai/sdk
    npm install -D tsx @types/node

    You don't need to install a local browser. Playwright only acts as the client for the Browserless browser.

  3. Create the agent

    Save the example below as agent.ts or agent.py. It opens Hacker News in a Browserless browser and asks Claude for the title of the top post.

    Model and tool version

    On the Claude API, computer use is generally available as the computer_toolset_20260801 toolset. It needs no beta header, so you call client.messages.create rather than the beta namespace. Claude 5.5 and later models accept only this toolset.

    • Model: claude-opus-5-5
    • Tool type: computer_toolset_20260801
    • Beta header: none

    Older models (Claude Sonnet 4.6, Opus 4.7, and earlier), and Amazon Bedrock and Microsoft Foundry, still use the beta computer_20251124 tool with the computer-use-2025-11-24 header. See Earlier tool versions for that request shape.

    import { chromium, Page } from "playwright-core";
    import Anthropic from "@anthropic-ai/sdk";

    const BROWSERLESS_URL = `wss://production-sfo.browserless.io?token=${process.env.BROWSERLESS_API_KEY}`;
    // Anthropic recommends 1280x800 for web apps. Claude returns coordinates in the
    // pixel space of the screenshots you send, so the viewport and screenshots must match.
    const VIEWPORT = { width: 1280, height: 800 };
    const MODEL = "claude-opus-5-5";
    const HALT_TEXT = "Not executed: an earlier computer action in this turn failed.";

    // Claude sends xdotool-style key names ("Return", "ctrl+s"). Playwright expects "Enter", "Control+s".
    const KEY_MAP: Record<string, string> = {
      return: "Enter", enter: "Enter", ctrl: "Control", control: "Control",
      alt: "Alt", shift: "Shift", super: "Meta", cmd: "Meta", meta: "Meta",
      escape: "Escape", esc: "Escape", tab: "Tab", space: "Space",
      backspace: "Backspace", delete: "Delete", home: "Home", end: "End",
      page_up: "PageUp", page_down: "PageDown",
      up: "ArrowUp", down: "ArrowDown", left: "ArrowLeft", right: "ArrowRight",
    };
    const toKeys = (combo: string) => combo.split("+").map((k) => KEY_MAP[k.toLowerCase()] ?? k);

    type MemberInput = {
      coordinate?: [number, number];
      start_coordinate?: [number, number];
      text?: string;
      scroll_direction?: "up" | "down" | "left" | "right";
      scroll_amount?: number;
      repeat?: number;
      duration?: number;
    };

    // Playwright can't read the mouse position back, so track it for cursor_position.
    let cursor = { x: 0, y: 0 };

    async function moveTo(page: Page, coordinate?: [number, number]) {
      if (!coordinate) return;
      cursor = { x: coordinate[0], y: coordinate[1] };
      await page.mouse.move(cursor.x, cursor.y);
    }

    // Click, drag, and scroll members can carry modifier keys ("shift", "ctrl+alt") in `text`.
    async function withModifiers(page: Page, text: string | undefined, action: () => Promise<void>) {
      const modifiers = text ? toKeys(text) : [];
      for (const key of modifiers) await page.keyboard.down(key);
      try {
        await action();
      } finally {
        for (const key of modifiers.reverse()) await page.keyboard.up(key);
      }
    }

    async function screenshotBlock(page: Page): Promise<Anthropic.ImageBlockParam> {
      const png = await page.screenshot({ type: "png" });
      return { type: "image", source: { type: "base64", media_type: "image/png", data: png.toString("base64") } };
    }

    async function runMember(page: Page, name: string, input: MemberInput): Promise<string | Anthropic.ImageBlockParam[]> {
      switch (name) {
        case "screenshot":
          return [await screenshotBlock(page)];
        case "mouse_move":
          await moveTo(page, input.coordinate);
          return "OK";
        case "left_click":
        case "right_click":
        case "middle_click":
        case "double_click":
        case "triple_click": {
          const button = name === "right_click" ? "right" : name === "middle_click" ? "middle" : "left";
          const clickCount = name === "double_click" ? 2 : name === "triple_click" ? 3 : 1;
          // Without a coordinate, Claude means "click where the cursor is".
          await moveTo(page, input.coordinate);
          await withModifiers(page, input.text, () => page.mouse.click(cursor.x, cursor.y, { button, clickCount }));
          return "OK";
        }
        case "left_click_drag":
          await moveTo(page, input.start_coordinate);
          await withModifiers(page, input.text, async () => {
            await page.mouse.down();
            await moveTo(page, input.coordinate);
            await page.mouse.up();
          });
          return "OK";
        case "left_mouse_down":
          await page.mouse.down();
          return "OK";
        case "left_mouse_up":
          await page.mouse.up();
          return "OK";
        case "cursor_position":
          return `X=${cursor.x}, Y=${cursor.y}`;
        case "scroll": {
          await moveTo(page, input.coordinate);
          // scroll_amount counts wheel clicks; ~100px per click matches a desktop mouse wheel.
          const px = (input.scroll_amount ?? 3) * 100;
          const dir = input.scroll_direction;
          const dx = dir === "left" ? -px : dir === "right" ? px : 0;
          const dy = dir === "up" ? -px : dir === "down" ? px : 0;
          await withModifiers(page, input.text, () => page.mouse.wheel(dx, dy));
          return "OK";
        }
        case "type":
          await page.keyboard.type(input.text ?? "");
          return "OK";
        case "key":
          for (let i = 0; i < (input.repeat ?? 1); i++) await page.keyboard.press(toKeys(input.text ?? "").join("+"));
          return "OK";
        case "hold_key": {
          const keys = toKeys(input.text ?? "");
          for (const key of keys) await page.keyboard.down(key);
          await page.waitForTimeout((input.duration ?? 1) * 1000);
          for (const key of keys.reverse()) await page.keyboard.up(key);
          return "OK";
        }
        case "wait":
          await page.waitForTimeout((input.duration ?? 1) * 1000);
          return "OK";
        default:
          throw new Error(`Unsupported member: ${name}`);
      }
    }

    function toolResult(id: string, content: string | Anthropic.ImageBlockParam[], isError = false): Anthropic.ToolResultBlockParam {
      // Every result for a toolset call must echo toolset_name.
      return { type: "tool_result", tool_use_id: id, toolset_name: "computer", content, is_error: isError };
    }

    async function runComputerUseLoop(client: Anthropic, page: Page, task: string, maxIterations = 15): Promise<string> {
      // Anthropic recommends putting the instruction text before the screenshot.
      const messages: Anthropic.MessageParam[] = [
        { role: "user", content: [{ type: "text", text: task }, await screenshotBlock(page)] },
      ];

      for (let i = 0; i < maxIterations; i++) {
        const response = await client.messages.create({
          model: MODEL,
          // Thinking is always on for claude-opus-5-5 and counts against max_tokens.
          max_tokens: 16000,
          // Zoom is enabled by default. This example doesn't implement it, so turn it off.
          tools: [{ type: "computer_toolset_20260801", configs: { zoom: { enabled: false } } }],
          messages,
        });
        messages.push({ role: "assistant", content: response.content });

        // Claude can batch several actions in one turn. Run them in order, stop at the
        // first failure, and still answer every block or the next request is rejected.
        const toolResults: Anthropic.ToolResultBlockParam[] = [];
        let failed = false;
        for (const block of response.content) {
          if (block.type !== "tool_use" || block.toolset_name !== "computer") continue;
          if (failed) {
            toolResults.push(toolResult(block.id, HALT_TEXT, true));
            continue;
          }
          try {
            toolResults.push(toolResult(block.id, await runMember(page, block.name, block.input as MemberInput)));
          } catch (err) {
            failed = true;
            toolResults.push(toolResult(block.id, `Error: ${(err as Error).message}`, true));
          }
        }

        if (toolResults.length === 0) {
          return response.content
            .filter((b): b is Anthropic.TextBlock => b.type === "text")
            .map((b) => b.text)
            .join("\n");
        }
        messages.push({ role: "user", content: toolResults });
      }

      return "Max iterations reached";
    }

    async function main() {
      const browser = await chromium.connectOverCDP(BROWSERLESS_URL);
      try {
        const page = await browser.newPage({ viewport: VIEWPORT });
        await page.goto("https://news.ycombinator.com", { waitUntil: "domcontentloaded" });

        // Reads ANTHROPIC_API_KEY from the environment.
        const client = new Anthropic();
        const result = await runComputerUseLoop(client, page, "What is the title of the top post on this page?");
        console.log("Result:", result);
      } finally {
        // Closing the browser ends the Browserless session so it stops counting against your plan.
        await browser.close();
      }
    }

    main();

  4. Run it

    # With a .env file
    npx tsx --env-file=.env agent.ts

    # With exported shell variables
    npx tsx agent.ts

    Claude reads the page from the screenshot and replies with the title:

    Result: The title of the top post is "Show HN: …".

Available actions​

The computer_toolset_20260801 toolset has 17 members. Claude names the member in the tool_use block's name and sends only that member's parameters in input. There's no action field like in the earlier tool versions.

MemberParametersDescription
screenshot—Return the current screen as an image
left_click / right_click / middle_click / double_click / triple_clickcoordinate (optional), text (optional modifiers)Click at [x, y], or at the cursor if coordinate is omitted
left_click_dragstart_coordinate, coordinate, text (optional)Drag from one point to another
mouse_movecoordinateMove the cursor
left_mouse_down / left_mouse_up—Press or release the left button for fine-grained drags
cursor_position—Return the cursor position as text, for example X=512, Y=384
scrollscroll_direction, scroll_amount, coordinate (optional), text (optional)Scroll up, down, left, or right
typetextType a string
keytext, repeat (optional, 1–100)Press a key or +-joined combo, such as Return or ctrl+s
hold_keytext, durationHold a key for up to 300 seconds
waitdurationPause for up to 300 seconds
zoomregion ([x0, y0, x1, y1])Return a closer view of part of the screen. Disabled in this example

Hybrid pattern: Playwright first, Claude as fallback​

Computer use is slower and costs more per step than a Playwright selector, because every action is a model round trip with a screenshot. For flows you already know, run the deterministic Playwright step first and hand the same goal to Claude only when that step fails, for example when a selector changes or a modal covers the target. Both share the same Browserless page, so Claude picks up exactly where Playwright stopped.

This reuses runComputerUseLoop / run_computer_use_loop from the example above. Call it inside main() after page.goto and after client is created:

async function stepWithFallback(client: Anthropic, page: Page, goal: string, step: () => Promise<unknown>) {
  try {
    await step();
  } catch (err) {
    console.warn(`Playwright step failed, falling back to Claude: ${(err as Error).message}`);
    await runComputerUseLoop(client, page, `${goal} Reply "done" when you've finished.`);
  }
}

// A short timeout makes a broken selector fail fast instead of waiting 30 seconds.
await stepWithFallback(client, page, "Click the 'new' link in the top navigation bar.", () =>
  page.click("a[href='newest']", { timeout: 5000 })
);

Verify the result after a fallback the same way you would after a Playwright step, for example with page.waitForURL("**/newest"). Claude's final reply tells you what it did, not whether it worked.

Advanced configuration​

Stealth mode​

For sites with bot detection, use the stealth route:

const browser = await chromium.connectOverCDP(
  `wss://production-sfo.browserless.io/stealth?token=${process.env.BROWSERLESS_API_KEY}`
);

Residential proxies​

Route traffic through residential IPs for geo-targeting and rate-limit avoidance:

const browser = await chromium.connectOverCDP(
  `wss://production-sfo.browserless.io?token=${process.env.BROWSERLESS_API_KEY}&proxy=residential&proxyCountry=us`
);

Regional endpoints​

Pick the region closest to you for lower latency:

RegionEndpoint
US West (San Francisco)wss://production-sfo.browserless.io
Europe (London)wss://production-lon.browserless.io
Europe (Amsterdam)wss://production-ams.browserless.io

Why use Browserless with Anthropic Computer Use​

  • Stealth mode: bypass bot detection by adding /stealth to the endpoint URL
  • Residential proxies: route traffic through real residential IPs to avoid blocks
  • Global regions: choose from US West (San Francisco), Europe (London), or Europe (Amsterdam) endpoints for lower latency
  • No infrastructure: skip managing Chrome installations, updates, or scaling
  • Parallel sessions: run multiple browser sessions simultaneously

Resources​

FAQ & Troubleshooting​

The API returns invalid_request_error for the tools entry

You're mixing the toolset with fields from an earlier tool version. computer_toolset_20260801 rejects name, display_width_px, display_height_px, display_number, and enable_zoom, and it doesn't need a beta header. If you're on a model or platform that only supports computer_20251124 (Claude Sonnet 4.6, Amazon Bedrock, Microsoft Foundry), keep that tool version, its display_* fields, and the computer-use-2025-11-24 beta header instead.

The loop changes too. computer_20251124 returns one tool named computer with the action in input.action, for example {"name": "computer", "input": {"action": "left_click", "coordinate": [500, 300]}}, and its blocks have no toolset_name, so the examples' toolset_name check skips them. Match block.name === "computer" instead, pass input.action to runMember (run_member in Python) as the member name along with the remaining input fields, which keep the same names, and leave toolset_name off each tool_result. Anthropic's migration guide lists every difference between the two versions. Check the computer use docs for which version your model accepts.

The next request fails after Claude returns several actions in one turn

Every tool_use block in a response needs its own tool_result, or the API rejects the next request with invalid_request_error. Loop over all blocks, not just the first, and answer the ones you skip after a failure with the halt text, as the example does.

Claude's clicks land in the wrong place

Claude returns coordinates in the pixel space of the screenshot you sent. Keep the screenshot the same size as the Playwright viewport. New Playwright contexts use a device scale factor of 1 by default, so a 1280x800 viewport produces a 1280x800 screenshot. If you set deviceScaleFactor above 1 or resize screenshots before sending them, scale Claude's coordinates back to viewport pixels before you click.

Playwright throws Unknown key: "Return"

Claude uses xdotool-style key names such as Return, Page_Down, and ctrl+s. Map them to Playwright names (Enter, PageDown, Control+s) before calling keyboard.press. The KEY_MAP in the example covers the common ones. The Playwright keyboard API docs list every valid name.

The loop stops with "Max iterations reached"

maxIterations / max_iterations caps how many model round trips the loop runs before giving up. Raise it for tasks that need more steps, or break the task into smaller prompts so each run completes within the cap.

A site blocks the browser mid-task

Add the /stealth route and residential proxies to the CDP connection URL. See Advanced configuration above for both examples.

Next steps​

Anthropic SDK

use the hosted Browserless MCP server from the Messages API

AI integrations overview

all supported AI frameworks and tools

BQL overview

use BrowserQL for advanced browser automation

Was this page helpful?