OpenAI Computer Use Integration

OpenAI's computer use tool looks at screenshots and returns structured actions (click, type, scroll) that Playwright executes in a Browserless cloud browser. This guide builds that loop with the Responses API, so you can run form filling, web research, and data extraction without managing browser infrastructure yourself.

Prerequisites

• Browserless API token (available in your account dashboard)
• OpenAI API key (get it from OpenAI API Keys) for a project with access to a model that supports the computer tool
• Node.js 22+ or Python 3.10+

computer-use-preview has been shut down

OpenAI shut down the computer-use-preview model on July 23, 2026. Requests that use it, or the computer_use_preview tool type, no longer work. This guide uses the generally available computer tool with gpt-6.1-sol. If you're updating an older integration, see OpenAI's migration guide.

How it works

1. Send the task: call the Responses API with the task and a computer tool.
2. Execute actions: the model returns a computer_call with an ordered actions array. Run each action through Playwright.
3. Return a screenshot: send a fresh screenshot back as a computer_call_output, chained with previous_response_id.
4. Loop until the model stops returning computer_call items.

The computer tool runs client-side, so you write the loop. If you'd rather have OpenAI call Browserless tools server-side with no browser code at all, see the OpenAI SDK page.

Step-by-step setup

In this guide you'll build an example that navigates to Bing, searches for "Browserless.io", and returns a summary of what the company does. We use stealth mode to avoid bot detection.

1. Set your API keys

   Grab your Browserless token from your account dashboard and your OpenAI key from OpenAI API Keys.

   .env file

   BROWSERLESS_API_KEY=your-browserless-token
   OPENAI_API_KEY=your-openai-key

   Command line

   export BROWSERLESS_API_KEY=your-browserless-token
   export OPENAI_API_KEY=your-openai-key

2. Install dependencies

   TypeScript

   npm install openai playwright-core
   npm install -D tsx @types/node

   Python

   pip install openai playwright python-dotenv

   You don't need to install a local browser. Playwright only acts as the client for the Browserless browser.

3. Connect to Browserless

   Put the snippets from this step onward in one file, agent.mts or agent.py. In Python, put them all inside one async def main() function, and end the file with import asyncio and asyncio.run(main()), since the snippets use await. Connect over CDP with stealth mode (recommended for avoiding bot detection):

   TypeScript

   import { chromium, Page } from "playwright-core";
   import OpenAI from "openai";
   
   // Reads OPENAI_API_KEY from the environment.
   const client = new OpenAI();
   const MODEL = "gpt-6.1-sol";
   
   const browser = await chromium.connectOverCDP(
     `wss://production-sfo.browserless.io/chromium/stealth?token=${process.env.BROWSERLESS_API_KEY}`,
     { timeout: 60000 }
   );
   // OpenAI reports strong results at 1440x900. The model returns coordinates in
   // screenshot pixels, so keep the viewport and screenshots the same size.
   const context = await browser.newContext({ viewport: { width: 1440, height: 900 } });
   const page = await context.newPage();

   Python

   import os
   import base64
   from dotenv import load_dotenv
   from openai import AsyncOpenAI
   from playwright.async_api import async_playwright
   
   # Load .env before reading the keys. Exported shell variables still win.
   load_dotenv()
   
   # Reads OPENAI_API_KEY from the environment.
   client = AsyncOpenAI()
   MODEL = "gpt-6.1-sol"
   
   p = await async_playwright().start()
   browser = await p.chromium.connect_over_cdp(
       f"wss://production-sfo.browserless.io/chromium/stealth?token={os.environ['BROWSERLESS_API_KEY']}",
       timeout=60000
   )
   # OpenAI reports strong results at 1440x900. The model returns coordinates in
   # screenshot pixels, so keep the viewport and screenshots the same size.
   context = await browser.new_context(viewport={"width": 1440, "height": 900})
   page = await context.new_page()

   tip

   This snippet and every later Python snippet in this guide run inside the same async def main() function, called with asyncio.run(main()). When you're done, call await browser.close() and await p.stop() to clean up.

4. Navigate and add a screenshot helper

   Open the start page and define the helper the loop uses to capture screenshots. Over remote WebSocket connections, standard Playwright screenshots can time out, so the helper falls back to CDP:

   TypeScript

   await page.goto("https://www.bing.com", { waitUntil: "domcontentloaded" });
   
   async function getScreenshot(page: Page): Promise<string> {
     try {
       const buffer = await page.screenshot({ type: "png", timeout: 10000 });
       return buffer.toString("base64");
     } catch {
       // Fall back to CDP directly when the Playwright screenshot times out.
       const cdp = await page.context().newCDPSession(page);
       // Raw CDP captures in device pixels, which the stealth route scales up from the
       // viewport size. Scale the capture back so it matches the viewport the model clicks in.
       const { cssLayoutViewport, cssVisualViewport, layoutViewport } = await cdp.send("Page.getLayoutMetrics");
       // Device pixels per CSS pixel, rounded so a scrollbar's width doesn't skew it.
       const deviceScale = Math.round((layoutViewport.clientWidth / cssLayoutViewport.clientWidth) * 100) / 100;
       const { width, height } = page.viewportSize()!;
       const result = await cdp.send("Page.captureScreenshot", {
         format: "png",
         clip: { x: cssVisualViewport.pageX, y: cssVisualViewport.pageY, width, height, scale: 1 / deviceScale },
       });
       await cdp.detach();
       return result.data;
     }
   }

   Python

   await page.goto("https://www.bing.com", wait_until="domcontentloaded")
   
   async def get_screenshot(page) -> str:
       try:
           screenshot_bytes = await page.screenshot(type="png", timeout=10000)
           return base64.b64encode(screenshot_bytes).decode("utf-8")
       except Exception:
           # Fall back to CDP directly when the Playwright screenshot times out.
           cdp = await page.context.new_cdp_session(page)
           # Raw CDP captures in device pixels, which the stealth route scales up from the
           # viewport size. Scale the capture back so it matches the viewport the model clicks in.
           metrics = await cdp.send("Page.getLayoutMetrics")
           # Device pixels per CSS pixel, rounded so a scrollbar's width doesn't skew it.
           device_scale = round(
               metrics["layoutViewport"]["clientWidth"] / metrics["cssLayoutViewport"]["clientWidth"], 2
           )
           visual = metrics["cssVisualViewport"]
           result = await cdp.send("Page.captureScreenshot", {
               "format": "png",
               "clip": {
                   "x": visual["pageX"],
                   "y": visual["pageY"],
                   "width": page.viewport_size["width"],
                   "height": page.viewport_size["height"],
                   "scale": 1 / device_scale,
               },
           })
           await cdp.detach()
           return result["data"]

5. Send the initial request to the model

   Send the task with the computer tool. You don't need to attach a screenshot: if the model needs to see the page first, its first computer_call contains a single screenshot action.

   TypeScript

   const task = "Search for 'Browserless.io' and tell me what the company does";
   
   let response = await client.responses.create({
     model: MODEL,
     tools: [{ type: "computer" }],
     input: task,
   });

   Python

   task = "Search for 'Browserless.io' and tell me what the company does"
   
   response = await client.responses.create(
       model=MODEL,
       tools=[{"type": "computer"}],
       input=task,
   )

6. Process actions and loop

   Each computer_call holds an ordered actions array. Run every action in order, capture one screenshot, and send it back with the matching call_id. Repeat until no computer_call appears (task complete).

   note

   The model sends key names like CTRL, ENTER, or CMD, and the mouse button wheel. Playwright expects Control, Enter, Meta, and middle. The examples below map them.

   TypeScript

   // Map model key names to Playwright key names.
   const keyMap: Record<string, string> = {
     enter: "Enter", return: "Enter", esc: "Escape", escape: "Escape",
     tab: "Tab", space: "Space", backspace: "Backspace", delete: "Delete",
     home: "Home", end: "End", pageup: "PageUp", pagedown: "PageDown",
     up: "ArrowUp", down: "ArrowDown", left: "ArrowLeft", right: "ArrowRight",
     arrowup: "ArrowUp", arrowdown: "ArrowDown", arrowleft: "ArrowLeft", arrowright: "ArrowRight",
     capslock: "CapsLock", insert: "Insert",
     ctrl: "Control", control: "Control", shift: "Shift", alt: "Alt", option: "Alt",
     cmd: "Meta", command: "Meta", meta: "Meta", super: "Meta", win: "Meta",
   };
   const toKey = (key: string) => keyMap[key.toLowerCase()] ?? key;
   // Playwright has no "back" or "forward" mouse button, so runAction maps those to history navigation.
   const buttonMap: Record<string, "left" | "right" | "middle"> = { left: "left", right: "right", wheel: "middle" };
   
   type Action = {
     type: string;
     x?: number; y?: number; button?: string; keys?: string[] | null;
     text?: string; scroll_x?: number; scroll_y?: number;
     path?: { x: number; y: number }[];
   };
   
   // Mouse actions can carry modifier keys (e.g. shift-click) in `keys`.
   async function withModifiers(keys: string[] | null | undefined, run: () => Promise<void>) {
     const modifiers = (keys ?? []).map(toKey);
     for (const key of modifiers) await page.keyboard.down(key);
     try {
       await run();
     } finally {
       for (const key of modifiers.reverse()) await page.keyboard.up(key);
     }
   }
   
   async function runAction(action: Action) {
     switch (action.type) {
       case "click": {
         // The browser's back and forward mouse buttons navigate history.
         // Pages restored from the back-forward cache don't fire load events again,
         // so wait for the document to be parsed instead.
         if (action.button === "back" || action.button === "forward") {
           await (action.button === "back" ? page.goBack({ waitUntil: "commit" }) : page.goForward({ waitUntil: "commit" }));
           await page.waitForFunction(() => document.readyState !== "loading");
           break;
         }
         const button = buttonMap[action.button ?? "left"];
         if (!button) throw new Error(`Unsupported mouse button: ${action.button}`);
         await withModifiers(action.keys, () => page.mouse.click(action.x!, action.y!, { button }));
         break;
       }
       case "double_click":
         await withModifiers(action.keys, () => page.mouse.dblclick(action.x!, action.y!));
         break;
       case "move":
         await withModifiers(action.keys, () => page.mouse.move(action.x!, action.y!));
         break;
       case "drag": {
         const [start, ...rest] = action.path ?? [];
         if (!start || rest.length === 0) throw new Error("drag needs at least two path points");
         await withModifiers(action.keys, async () => {
           await page.mouse.move(start.x, start.y);
           await page.mouse.down();
           for (const point of rest) await page.mouse.move(point.x, point.y);
           await page.mouse.up();
         });
         break;
       }
       case "scroll":
         // Wheel events scroll whatever is under the cursor, including inner scroll containers.
         await withModifiers(action.keys, async () => {
           await page.mouse.move(action.x!, action.y!);
           await page.mouse.wheel(action.scroll_x ?? 0, action.scroll_y ?? 0);
         });
         break;
       case "keypress":
         await page.keyboard.press((action.keys ?? []).map(toKey).join("+"));
         break;
       case "type":
         await page.keyboard.type(action.text ?? "");
         break;
       case "wait":
         await page.waitForTimeout(2000);
         break;
       case "screenshot":
         // The screenshot is captured after the batch, below.
         break;
       default:
         throw new Error(`Unsupported action: ${action.type}`);
     }
   }
   
   const MAX_TURNS = 20;
   // The extra pass reads the response to the last turn's actions before giving up.
   for (let turn = 0; turn <= MAX_TURNS; turn++) {
     const computerCall = response.output.find(
       (item): item is OpenAI.Responses.ResponseComputerToolCall => item.type === "computer_call"
     );
   
     if (!computerCall) {
       // Task complete. Print the model's answer.
       console.log(response.output_text);
       break;
     }
   
     // Safety checks need a human decision, so this example stops instead of continuing.
     if (computerCall.pending_safety_checks?.length) {
       console.log("Stopped on safety check:", computerCall.pending_safety_checks);
       break;
     }
   
     if (turn === MAX_TURNS) {
       console.log(`Stopped after ${MAX_TURNS} turns without a final answer; the task may be incomplete.`);
       break;
     }
   
     let failed = false;
     for (const action of computerCall.actions ?? []) {
       try {
         await runAction(action as Action);
       } catch (err) {
         // computer_call_output carries only a screenshot, so the model can't be told why an
         // action failed. Stop the run cleanly instead of letting it retry the action blind.
         console.error(`Action ${action.type} failed, stopping:`, err);
         failed = true;
         break;
       }
     }
     if (failed) break;
   
     const screenshot = await getScreenshot(page);
   
     response = await client.responses.create({
       model: MODEL,
       // Tools aren't stored with the response, so send them on every call.
       tools: [{ type: "computer" }],
       previous_response_id: response.id,
       input: [
         {
           type: "computer_call_output",
           call_id: computerCall.call_id,
           output: {
             type: "computer_screenshot",
             image_url: `data:image/png;base64,${screenshot}`,
             // "original" keeps full resolution, which OpenAI recommends for click accuracy.
             detail: "original",
           },
         },
       ],
     });
   }
   
   await browser.close();

   Python

   # Map model key names to Playwright key names.
   key_map = {
       "enter": "Enter", "return": "Enter", "esc": "Escape", "escape": "Escape",
       "tab": "Tab", "space": "Space", "backspace": "Backspace", "delete": "Delete",
       "home": "Home", "end": "End", "pageup": "PageUp", "pagedown": "PageDown",
       "up": "ArrowUp", "down": "ArrowDown", "left": "ArrowLeft", "right": "ArrowRight",
       "arrowup": "ArrowUp", "arrowdown": "ArrowDown", "arrowleft": "ArrowLeft", "arrowright": "ArrowRight",
       "capslock": "CapsLock", "insert": "Insert",
       "ctrl": "Control", "control": "Control", "shift": "Shift", "alt": "Alt", "option": "Alt",
       "cmd": "Meta", "command": "Meta", "meta": "Meta", "super": "Meta", "win": "Meta",
   }
   # Playwright has no "back" or "forward" mouse button, so run_action maps those to history navigation.
   button_map = {"left": "left", "right": "right", "wheel": "middle"}
   
   
   def to_key(key: str) -> str:
       return key_map.get(key.lower(), key)
   
   
   async def with_modifiers(keys, run) -> None:
       # Mouse actions can carry modifier keys (e.g. shift-click) in `keys`.
       modifiers = [to_key(k) for k in (keys or [])]
       for key in modifiers:
           await page.keyboard.down(key)
       try:
           await run()
       finally:
           for key in reversed(modifiers):
               await page.keyboard.up(key)
   
   
   async def run_action(action) -> None:
       keys = getattr(action, "keys", None)
       if action.type == "click" and action.button in ("back", "forward"):
           # The browser's back and forward mouse buttons navigate history.
           # Pages restored from the back-forward cache don't fire load events again,
           # so wait for the document to be parsed instead.
           await (page.go_back(wait_until="commit") if action.button == "back" else page.go_forward(wait_until="commit"))
           await page.wait_for_function("document.readyState !== 'loading'")
       elif action.type == "click":
           button = button_map.get(action.button or "left")
           if button is None:
               raise ValueError(f"Unsupported mouse button: {action.button}")
           await with_modifiers(keys, lambda: page.mouse.click(action.x, action.y, button=button))
       elif action.type == "double_click":
           await with_modifiers(keys, lambda: page.mouse.dblclick(action.x, action.y))
       elif action.type == "move":
           await with_modifiers(keys, lambda: page.mouse.move(action.x, action.y))
       elif action.type == "drag":
           path = [(point.x, point.y) for point in action.path]
           if len(path) < 2:
               raise ValueError("drag needs at least two path points")
   
           async def drag():
               await page.mouse.move(*path[0])
               await page.mouse.down()
               for x, y in path[1:]:
                   await page.mouse.move(x, y)
               await page.mouse.up()
   
           await with_modifiers(keys, drag)
       elif action.type == "scroll":
           # Wheel events scroll whatever is under the cursor, including inner scroll containers.
           async def scroll():
               await page.mouse.move(action.x, action.y)
               await page.mouse.wheel(action.scroll_x or 0, action.scroll_y or 0)
   
           await with_modifiers(keys, scroll)
       elif action.type == "keypress":
           await page.keyboard.press("+".join(to_key(k) for k in action.keys))
       elif action.type == "type":
           await page.keyboard.type(action.text)
       elif action.type == "wait":
           await page.wait_for_timeout(2000)
       elif action.type == "screenshot":
           pass  # The screenshot is captured after the batch, below.
       else:
           raise ValueError(f"Unsupported action: {action.type}")
   
   
   MAX_TURNS = 20
   # The extra pass reads the response to the last turn's actions before giving up.
   for turn in range(MAX_TURNS + 1):
       computer_call = next(
           (item for item in response.output if item.type == "computer_call"), None
       )
   
       if computer_call is None:
           # Task complete. Print the model's answer.
           print(response.output_text)
           break
   
       # Safety checks need a human decision, so this example stops instead of continuing.
       if computer_call.pending_safety_checks:
           print("Stopped on safety check:", computer_call.pending_safety_checks)
           break
   
       if turn == MAX_TURNS:
           print(f"Stopped after {MAX_TURNS} turns without a final answer; the task may be incomplete.")
           break
   
       failed = False
       for action in computer_call.actions or []:
           try:
               await run_action(action)
           except Exception as err:
               # computer_call_output carries only a screenshot, so the model can't be told why an
               # action failed. Stop the run cleanly instead of letting it retry the action blind.
               print(f"Action {action.type} failed, stopping: {err}")
               failed = True
               break
       if failed:
           break
   
       screenshot_base64 = await get_screenshot(page)
   
       response = await client.responses.create(
           model=MODEL,
           # Tools aren't stored with the response, so send them on every call.
           tools=[{"type": "computer"}],
           previous_response_id=response.id,
           input=[
               {
                   "type": "computer_call_output",
                   "call_id": computer_call.call_id,
                   "output": {
                       "type": "computer_screenshot",
                       "image_url": f"data:image/png;base64,{screenshot_base64}",
                       # "original" keeps full resolution, which OpenAI recommends for click accuracy.
                       "detail": "original",
                   },
               }
           ],
       )
   
   await browser.close()
   await p.stop()

7. Run it

   TypeScript

   npx tsx --env-file=.env agent.mts

   The .mts extension makes the file an ES module, which the top-level await calls need.

   Python

   python agent.py

   The model searches Bing, reads the results, and prints a summary:

   Browserless is a cloud browser automation platform that runs headless Chrome for …

Supported actions

Action	Properties	Description
click	x, y, button, keys (optional modifiers)	Click at coordinates. button is left, right, wheel, back, or forward
double_click	x, y, keys (optional)	Double-click at coordinates
move	x, y, keys (optional)	Move the cursor
drag	path (array of {x, y}), keys (optional)	Drag along a path of points
scroll	x, y, scroll_x, scroll_y, keys (optional)	Scroll by a pixel delta at a position
keypress	keys[]	Press a key or key combination
type	text	Type text
wait	—	Pause so the page can settle
screenshot	—	Request a fresh screenshot

Advanced configuration

Without stealth mode

If you don't need anti-detection and just want a managed cloud browser:

TypeScript

const browser = await chromium.connectOverCDP(
  `wss://production-sfo.browserless.io?token=${process.env.BROWSERLESS_API_KEY}`
);

Python

browser = await p.chromium.connect_over_cdp(
    f"wss://production-sfo.browserless.io?token={os.environ['BROWSERLESS_API_KEY']}"
)

Residential proxies

Route traffic through real residential IPs for additional anti-detection:

TypeScript

const browser = await chromium.connectOverCDP(
  `wss://production-sfo.browserless.io?token=${process.env.BROWSERLESS_API_KEY}&proxy=residential&proxyCountry=us`
);

Python

browser = await p.chromium.connect_over_cdp(
    f"wss://production-sfo.browserless.io?token={os.environ['BROWSERLESS_API_KEY']}&proxy=residential&proxyCountry=us"
)

Regional endpoints

Connect to the closest region for lower latency. See Connection URLs for all available endpoints.

Resources

• OpenAI computer use guide
• OpenAI computer use integration recipes
• OpenAI SDK page: the hosted Browserless MCP server with the Responses API, no browser code required
• Browserless connection options
• Stealth mode
• Proxy configuration

FAQ & Troubleshooting

The request fails with a model or tool type error

You're probably still sending the preview request shape. computer-use-preview was shut down on July 23, 2026. Use a model that supports the computer tool (this guide uses gpt-6.1-sol), set the tool to { type: "computer" } without display_width, display_height, or environment, and drop truncation: "auto". Read computer_call.actions (an array) instead of computer_call.action. See OpenAI's migration guide.

The loop stops with "Stopped on safety check"

The model attached pending_safety_checks to a computer_call, which means the next action needs a human decision. The example stops rather than continuing on its own. In production, show the check to a user and only continue if they approve. See OpenAI's confirmation and consent guidance.

The screenshot fallback still times out

Increase the Playwright connection timeout, or check your network connection to Browserless:

TypeScript

const browser = await chromium.connectOverCDP(url, { timeout: 120000 });

Python

browser = await p.chromium.connect_over_cdp(url, timeout=120000)

The model returns key names Playwright doesn't recognize

The keyMap / key_map in the action loop covers the most common mismatches (CTRL → Control, CMD → Meta, etc.). If you encounter new ones, add them to the map. The Playwright keyboard API docs list all valid key names.

Clicks land in the wrong place

The model returns coordinates in the pixel space of the screenshot you sent. Keep the screenshot the same size as the viewport, and send it with detail: "original" so it keeps full resolution. If you downscale screenshots to save tokens, scale the model's coordinates back up before you click.

A site blocks the browser mid-task

Add residential proxies to the CDP connection URL, and confirm you're still using the /stealth route from Connect to Browserless. See Advanced configuration above for both examples.

Next steps

OpenAI SDK

use the hosted Browserless MCP server from the Responses API

Hybrid Playwright fallback

run Playwright first and call a computer use model only when a step fails

AI integrations overview

all supported AI frameworks and tools