Gemini Computer Use Integration

Google's Gemini Computer Use model looks at a screenshot of the page and returns UI actions (click, type, scroll) that Playwright executes in a Browserless cloud browser. This enables form filling, web research, and multi-step automation without managing browser infrastructure yourself, backed by Browserless stealth mode and residential proxies.

Gemini returns normalized 0–1000 coordinates

Unlike OpenAI CUA and Anthropic Computer Use, which return absolute pixel coordinates, Gemini returns coordinates on a normalized 0–1000 grid. You must scale them to your actual viewport (pixel_x = x / 1000 * viewport_width) before passing them to Playwright. Feeding Gemini's raw coordinates directly into page.mouse.click() makes every click land in the top-left corner of the page — forms get mis-filled and submit/CAPTCHA elements are missed. This is the single most common mistake when porting an OpenAI/Anthropic loop to Gemini. See Denormalize coordinates.

Prerequisites

• Browserless API token (available in your account dashboard)
• Gemini API key (get it from Google AI Studio)
• Python 3.10+ and the google-genai SDK (2.7.0+)

How it works

Gemini Computer Use runs an agentic screenshot + action loop:

1. Send a request — call generate_content with the computer_use tool, the task prompt, and a screenshot of the current page
2. Receive an action — the model responds with a function_call (e.g. click, type) whose x/y are on a normalized 0–1000 grid, plus an intent explaining the step
3. Execute the action — denormalize the coordinates to viewport pixels and run the action via Playwright
4. Return the new state — capture a fresh screenshot and send it back in a function_response, then repeat until the model stops emitting actions

Model names

gemini-3.5-flash is Google's recommended Computer Use model and uses the streamlined action set below (click, type, ...). The legacy gemini-2.5-computer-use-preview-10-2025 uses a different set (click_at, type_text_at, ...). Both return normalized 0–1000 coordinates. Check the Gemini Computer Use docs for the latest model IDs.

Step-by-Step Setup

In this guide you'll build an example that navigates to Hacker News and reports the title of the top post. We use stealth mode to avoid bot detection. The example follows Google's official Computer Use reference implementation, adapted to connect to Browserless over CDP.

1. Set your API keys

   Grab your Browserless token from your account dashboard and your Gemini key from Google AI Studio.

   .env file

   BROWSERLESS_API_KEY=your-browserless-token
   GEMINI_API_KEY=your-gemini-key

   Command line

   export BROWSERLESS_API_KEY=your-browserless-token
   export GEMINI_API_KEY=your-gemini-key

2. Install dependencies

   pip install google-genai playwright
   playwright install chromium

   Language support

   Gemini Computer Use is Python-first during preview; this guide uses the google-genai Python SDK. A Node.js SDK (@google/genai) exposes the same computerUse tool if you prefer TypeScript — the Browserless connection and coordinate-denormalization steps are identical.

3. Connect to Browserless

   Connect Playwright to Browserless over CDP using the stealth route (recommended for avoiding bot detection). Set a fixed viewport and reuse those exact dimensions when you denormalize coordinates.

   import os
   from playwright.sync_api import sync_playwright
   
   SCREEN_WIDTH = 1440
   SCREEN_HEIGHT = 900
   
   BROWSERLESS_URL = (
       "wss://production-sfo.browserless.io/chromium/stealth"
       f"?token={os.environ['BROWSERLESS_API_KEY']}"
   )
   
   playwright = sync_playwright().start()
   browser = playwright.chromium.connect_over_cdp(BROWSERLESS_URL, timeout=60000)
   context = browser.new_context(
       viewport={"width": SCREEN_WIDTH, "height": SCREEN_HEIGHT}
   )
   page = context.new_page()
   page.goto("https://news.ycombinator.com", wait_until="load")

4. Denormalize coordinates

   Gemini's x/y are on a 0–1000 grid. Scale them to the viewport you configured above. This is the step that's missing when a ported OpenAI/Anthropic loop "clicks in the wrong place".

   def denormalize_x(x: int) -> int:
       return int(x / 1000 * SCREEN_WIDTH)
   
   def denormalize_y(y: int) -> int:
       return int(y / 1000 * SCREEN_HEIGHT)

5. Map Gemini actions to Playwright

   Translate each function_call into a Playwright action, denormalizing every coordinate first. This covers the gemini-3.5-flash action set; the legacy gemini-2.5-* names are noted in Supported actions.

   KEY_MAP = {
       "control": "ControlOrMeta",
       "cmd": "Meta",
       "command": "Meta",
       "enter": "Enter",
       "return": "Enter",
       "tab": "Tab",
       "escape": "Escape",
       "backspace": "Backspace",
       "delete": "Delete",
       "up": "ArrowUp",
       "down": "ArrowDown",
       "left": "ArrowLeft",
       "right": "ArrowRight",
   }
   
   
   def press_keys(keys: list[str]) -> None:
       mapped = [KEY_MAP.get(k.lower(), k) for k in keys]
       for key in mapped[:-1]:
           page.keyboard.down(key)
       page.keyboard.press(mapped[-1])
       for key in reversed(mapped[:-1]):
           page.keyboard.up(key)
   
   
   def execute_action(call) -> None:
       name = call.name
       args = dict(call.args or {})
   
       if name in ("open_web_browser", "take_screenshot"):
           pass
       elif name == "wait":
           page.wait_for_timeout(int(args.get("seconds", 1)) * 1000)
       elif name == "navigate":
           page.goto(args["url"], wait_until="load")
       elif name == "go_back":
           page.go_back()
       elif name == "go_forward":
           page.go_forward()
       elif name in ("click", "double_click", "right_click", "middle_click", "move"):
           x, y = denormalize_x(args["x"]), denormalize_y(args["y"])
           if name == "click":
               page.mouse.click(x, y)
           elif name == "double_click":
               page.mouse.dblclick(x, y)
           elif name == "right_click":
               page.mouse.click(x, y, button="right")
           elif name == "middle_click":
               page.mouse.click(x, y, button="middle")
           else:
               page.mouse.move(x, y)
       elif name == "type":
           page.keyboard.type(args["text"])
           if args.get("press_enter"):
               page.keyboard.press("Enter")
       elif name == "scroll":
           x, y = denormalize_x(args["x"]), denormalize_y(args["y"])
           direction = args["direction"]
           raw = args.get("magnitude", args.get("magnitude_in_pixels", 800))
           step = denormalize_y(raw) if direction in ("up", "down") else denormalize_x(raw)
           dx = -step if direction == "left" else step if direction == "right" else 0
           dy = -step if direction == "up" else step if direction == "down" else 0
           page.mouse.move(x, y)
           page.mouse.wheel(dx, dy)
       elif name in ("hotkey", "key_combination"):
           keys = args["keys"]
           press_keys(keys if isinstance(keys, list) else str(keys).split("+"))
       elif name == "press_key":
           press_keys([args["key"]])
       elif name == "drag_and_drop":
           x, y = denormalize_x(args["x"]), denormalize_y(args["y"])
           dx, dy = denormalize_x(args["destination_x"]), denormalize_y(args["destination_y"])
           page.mouse.move(x, y)
           page.mouse.down()
           page.mouse.move(dx, dy)
           page.mouse.up()
       else:
           raise ValueError(f"Unsupported action: {name}")
   
       page.wait_for_load_state()

6. Run the agent loop

   Send the task plus the initial screenshot, execute each returned action, and send a fresh screenshot back in a function_response. The loop ends when the model stops emitting function calls.

   generate_content vs. the Interactions API

   This example uses client.models.generate_content with a client-side contents history, matching Google's official Computer Use reference implementation. Google also offers a newer Interactions API (client.interactions.create) that manages conversation state server-side via previous_interaction_id and is recommended for new development generally; generate_content remains fully supported. The Browserless connection and the 0–1000 coordinate denormalization below are identical either way.

   from google import genai
   from google.genai import types
   
   
   def capture(page) -> bytes:
       page.wait_for_load_state()
       return page.screenshot(type="png")
   
   
   client = genai.Client()  # reads GEMINI_API_KEY from the environment
   
   config = types.GenerateContentConfig(
       tools=[
           types.Tool(
               computer_use=types.ComputerUse(
                   environment=types.Environment.ENVIRONMENT_BROWSER,
               )
           )
       ],
       thinking_config=types.ThinkingConfig(include_thoughts=True),
   )
   
   contents = [
       types.Content(
           role="user",
           parts=[
               types.Part(text="What is the title of the top post on this page?"),
               types.Part.from_bytes(data=capture(page), mime_type="image/png"),
           ],
       )
   ]
   
   for _ in range(30):  # cap iterations so the loop can't run forever
       response = client.models.generate_content(
           model="gemini-3.5-flash",
           contents=contents,
           config=config,
       )
       candidate = response.candidates[0]
       contents.append(candidate.content)
   
       parts = candidate.content.parts or []
       function_calls = [part.function_call for part in parts if part.function_call]
   
       if not function_calls:
           print("Done:", " ".join(part.text for part in parts if part.text))
           break
   
       function_responses = []
       for call in function_calls:
           # A `safety_decision` of "require_confirmation" means you should
           # confirm before executing — see the Safety section below.
           execute_action(call)
           function_responses.append(
               types.FunctionResponse(
                   name=call.name,
                   response={"url": page.url},
                   parts=[
                       types.FunctionResponsePart(
                           inline_data=types.FunctionResponseBlob(
                               mime_type="image/png", data=capture(page)
                           )
                       )
                   ],
               )
           )
   
       contents.append(
           types.Content(
               role="user",
               parts=[types.Part(function_response=fr) for fr in function_responses],
           )
       )
   
   browser.close()
   playwright.stop()

Supported actions

Gemini 3.5-flash (recommended) returns these browser actions. Every coordinate is on the 0–1000 grid and must be denormalized before use.

Action	Properties	Description
click / double_click / right_click / middle_click	x, y	Click at coordinates
move	x, y	Move the cursor
type	text, press_enter	Type text, optionally pressing Enter
press_key / key_down / key_up	key	Press / hold / release a key
hotkey	keys[]	Press a key combination
scroll	x, y, direction, magnitude	Scroll at a position
navigate	url	Go directly to a URL
go_back / go_forward	–	Browser history navigation
drag_and_drop	x, y, destination_x, destination_y	Drag from one point to another
wait	seconds	Pause for the page to settle
open_web_browser / take_screenshot	–	No-ops in this loop (state is captured each turn)

Legacy 2.5 action names

The legacy gemini-2.5-computer-use-preview-10-2025 model uses click_at, hover_at, type_text_at, scroll_document, scroll_at, wait_5_seconds, key_combination, search, navigate, go_back, go_forward, and drag_and_drop. The coordinate handling is identical.

Safety confirmations

For sensitive actions (e.g. clicking a CAPTCHA checkbox or completing a purchase), Gemini may attach a safety_decision to the function call with decision: "require_confirmation". In production you should pause and require explicit approval before executing those actions, then acknowledge the decision when returning the function_response. See Google's safety guidance for details.

Advanced configuration

Residential proxies

reCAPTCHA v3 and similar defenses score requests largely on IP reputation. If your automation completes the form but the site rejects the submission, route traffic through residential IPs:

BROWSERLESS_URL = (
    "wss://production-sfo.browserless.io/chromium/stealth"
    f"?token={os.environ['BROWSERLESS_API_KEY']}&proxy=residential&proxyCountry=us"
)

See Proxy configuration for country and session options.

Without stealth mode

If you don't need anti-detection and just want a managed cloud browser:

BROWSERLESS_URL = (
    f"wss://production-sfo.browserless.io?token={os.environ['BROWSERLESS_API_KEY']}"
)

Regional endpoints

Connect to the closest region for lower latency. See Connection URLs for all available endpoints.

When to use Computer Use vs. the MCP browser agent

Computer Use drives the browser from screenshots and pixel coordinates, so its reliability depends on the model clicking the right spot. For structured, multi-step flows (logins, form filling), the Browserless MCP server's browserless_agent tool is often more robust: it drives the browser from accessibility/DOM snapshots and element references rather than coordinates, so it doesn't depend on pixel-accurate clicks and works the same across models. If you're seeing coordinate-related flakiness with Computer Use, consider the MCP agent for the structured parts of your workflow.

Why use Browserless with Gemini Computer Use

• Stealth mode: bypass bot detection by adding /stealth to the endpoint URL
• Residential proxies: route traffic through real residential IPs to improve IP reputation for score-based defenses like reCAPTCHA v3
• Global regions: choose US West, London, or Amsterdam endpoints for lower latency
• No infrastructure: skip managing Chrome installations, updates, or scaling
• Parallel sessions: run multiple browser sessions simultaneously

Resources

• Gemini Computer Use Documentation
• Gemini Computer Use Reference Implementation
• Browserless Connection Options
• Stealth Mode
• Proxy Configuration

FAQ & Troubleshooting

The model's clicks land in the wrong place (usually the top-left)

You're passing Gemini's raw 0–1000 coordinates straight to Playwright. Denormalize every x/y to your viewport first (x / 1000 * viewport_width), and make sure the width/height you scale against match the viewport you set with new_context(viewport=...). See Denormalize coordinates.

The automation fills the form but the site rejects the submission

This is usually not a coordinate issue — score-based defenses like reCAPTCHA v3 rely heavily on IP reputation. Add residential proxies to the CDP URL and keep the /stealth route. See Advanced configuration.

The model returns key names Playwright doesn't recognize

The KEY_MAP in the action handler covers common mismatches (control → ControlOrMeta, cmd → Meta, etc.). Add new ones as needed — the Playwright keyboard API docs list all valid key names.

The loop never stops

The range(30) cap bounds how many screenshot-action rounds run before giving up. Raise it for longer tasks, or break the task into smaller prompts so each run completes within the cap.

A site blocks the browser mid-task

Add residential proxies to the CDP connection URL and confirm you're still using the /stealth route from Connect to Browserless.

Next steps

AI integrations overview

all supported AI frameworks and tools

BQL overview

use BrowserQL for advanced browser automation