Gemini Computer Use Integration

Google's Gemini Computer Use tool looks at a screenshot of the page and returns UI actions (click, type, scroll) that Playwright executes in a Browserless cloud browser. This guide builds that loop with the Gemini Interactions API, so you can run form filling, web research, and multi-step automation on Browserless stealth browsers and residential proxies without managing browser infrastructure yourself.

Gemini returns normalized 0–1000 coordinates

Unlike OpenAI and Anthropic computer use, which return pixel coordinates, Gemini returns coordinates on a normalized 0–1000 grid. You must scale them to your actual viewport (pixel_x = x / 1000 * viewport_width) before passing them to Playwright. If you feed Gemini's raw coordinates into page.mouse.click(), clicks land in the wrong place, forms get mis-filled, and submit buttons are missed. This is the most common mistake when porting an OpenAI or Anthropic loop to Gemini. See Denormalize coordinates.

Prerequisites

• Browserless API token (available in your account dashboard)
• Gemini API key (get it from Google AI Studio)
• Python 3.10+ and the google-genai SDK (2.34.0+, the first version that continues long interactions for you)

How it works

Gemini Computer Use runs an agentic screenshot + action loop:

1. Send a request: call client.interactions.create with the computer_use tool, the task prompt, and a screenshot of the current page.
2. Receive actions: the response's steps contain one or more function_call steps (for example click or type) whose x/y are on a normalized 0–1000 grid, plus an intent explaining the step.
3. Execute the actions: denormalize the coordinates to viewport pixels and run each action through Playwright.
4. Return the new state: send a function_result for every call, with the current URL and a fresh screenshot, chained with previous_interaction_id. Repeat until the model stops emitting function calls.

Model names

gemini-3.8-flash is Google's recommended Computer Use model. gemini-4-argon and gemini-3.5-flash-lite also support the tool, as does the preview model gemini-3-flash-preview, which can change or be removed without notice. The legacy gemini-2.5-computer-use-preview-10-2025 model was shut down on July 28, 2026, and gemini-3.5-flash now routes to a model that isn't on Google's Computer Use list. Check the Gemini Computer Use docs for the latest model IDs.

Step-by-step setup

In this guide you'll build an example that navigates to Hacker News and reports the title of the top post. We use stealth mode to avoid bot detection. The example follows the request shape in Google's Computer Use guide, adapted to connect to Browserless over CDP.

1. Set your API keys

   Grab your Browserless token from your account dashboard and your Gemini key from Google AI Studio.

   .env file

   BROWSERLESS_API_KEY=your-browserless-token
   GEMINI_API_KEY=your-gemini-key

   Command line

   export BROWSERLESS_API_KEY=your-browserless-token
   export GEMINI_API_KEY=your-gemini-key

   The script below reads both variables from the environment. If you use a .env file, export it into your shell first (for example set -a; source .env; set +a).

2. Install dependencies

   pip install "google-genai>=2.34.0" playwright

   You don't need playwright install. Playwright only acts as the client for the Browserless browser.

   Language support

   This guide uses the google-genai Python SDK. Google's docs also show the same interactions.create request in the Node.js SDK (@google/genai), Java, and Go. The Browserless connection and coordinate-denormalization steps are identical in every language.

3. Connect to Browserless

   Save the snippets from this step onward in one file, agent.py. Connect Playwright to Browserless over CDP using the stealth route (recommended for avoiding bot detection). Set a fixed viewport and reuse those exact dimensions when you denormalize coordinates.

   import base64
   import json
   import os
   from playwright.sync_api import sync_playwright
   
   SCREEN_WIDTH = 1440
   SCREEN_HEIGHT = 900
   
   BROWSERLESS_URL = (
       "wss://production-sfo.browserless.io/chromium/stealth"
       f"?token={os.environ['BROWSERLESS_API_KEY']}"
   )
   
   playwright = sync_playwright().start()
   browser = playwright.chromium.connect_over_cdp(BROWSERLESS_URL, timeout=60000)
   context = browser.new_context(
       viewport={"width": SCREEN_WIDTH, "height": SCREEN_HEIGHT}
   )
   page = context.new_page()
   page.goto("https://news.ycombinator.com", wait_until="load")

4. Denormalize coordinates

   Gemini's x/y are on a 0–1000 grid. Scale them to the viewport you configured above. This is the step that's missing when a ported OpenAI or Anthropic loop "clicks in the wrong place".

   def denormalize_x(x: int) -> int:
       return int(x / 1000 * SCREEN_WIDTH)
   
   def denormalize_y(y: int) -> int:
       return int(y / 1000 * SCREEN_HEIGHT)

5. Map Gemini actions to Playwright

   Translate each function_call into a Playwright action, denormalizing every coordinate first. This covers the browser action set listed in Supported actions.

   # The Browserless browser runs on Linux, so map "control" to Control. Playwright's
   # ControlOrMeta resolves on your machine and sends Meta when the script runs on a Mac.
   KEY_MAP = {
       "control": "Control",
       "ctrl": "Control",
       "cmd": "Meta",
       "command": "Meta",
       "shift": "Shift",
       "alt": "Alt",
       "enter": "Enter",
       "return": "Enter",
       "tab": "Tab",
       "escape": "Escape",
       "backspace": "Backspace",
       "delete": "Delete",
       "pageup": "PageUp",
       "pagedown": "PageDown",
       "up": "ArrowUp",
       "down": "ArrowDown",
       "left": "ArrowLeft",
       "right": "ArrowRight",
   }
   
   
   def to_key(key: str) -> str:
       return KEY_MAP.get(key.lower(), key)
   
   
   def press_keys(keys: list[str]) -> None:
       mapped = [to_key(k) for k in keys]
       for key in mapped[:-1]:
           page.keyboard.down(key)
       page.keyboard.press(mapped[-1])
       for key in reversed(mapped[:-1]):
           page.keyboard.up(key)
   
   
   def point(args: dict) -> tuple[int, int]:
       return denormalize_x(args["x"]), denormalize_y(args["y"])
   
   
   def execute_action(name: str, args: dict) -> None:
       if name == "take_screenshot":
           pass  # A fresh screenshot is returned after every action anyway.
       elif name == "wait":
           page.wait_for_timeout(int(args.get("seconds", 1)) * 1000)
       elif name == "navigate":
           page.goto(args["url"], wait_until="load")
       elif name == "go_back":
           # Pages restored from the back-forward cache don't fire load events again,
           # so wait for the document to be parsed instead.
           page.go_back(wait_until="commit")
           page.wait_for_function("document.readyState !== 'loading'")
       elif name == "go_forward":
           page.go_forward(wait_until="commit")
           page.wait_for_function("document.readyState !== 'loading'")
       elif name in ("click", "double_click", "triple_click", "right_click", "middle_click"):
           x, y = point(args)
           button = {"right_click": "right", "middle_click": "middle"}.get(name, "left")
           click_count = {"double_click": 2, "triple_click": 3}.get(name, 1)
           page.mouse.click(x, y, button=button, click_count=click_count)
       elif name == "move":
           page.mouse.move(*point(args))
       elif name in ("mouse_down", "mouse_up"):
           page.mouse.move(*point(args))
           page.mouse.down() if name == "mouse_down" else page.mouse.up()
       elif name == "type":
           # Some responses include a target coordinate. Click it first to focus the field.
           if "x" in args and "y" in args:
               page.mouse.click(*point(args))
           page.keyboard.type(args["text"])
           if args.get("press_enter"):
               page.keyboard.press("Enter")
       elif name == "scroll":
           page.mouse.move(*point(args))
           step = int(args.get("magnitude_in_pixels", 300))
           direction = args["direction"]
           dx = -step if direction == "left" else step if direction == "right" else 0
           dy = -step if direction == "up" else step if direction == "down" else 0
           page.mouse.wheel(dx, dy)
       elif name == "press_key":
           press_keys([args["key"]])
       elif name == "key_down":
           page.keyboard.down(to_key(args["key"]))
       elif name == "key_up":
           page.keyboard.up(to_key(args["key"]))
       elif name == "hotkey":
           press_keys(args["keys"])
       elif name == "drag_and_drop":
           page.mouse.move(denormalize_x(args["start_x"]), denormalize_y(args["start_y"]))
           page.mouse.down()
           page.mouse.move(denormalize_x(args["end_x"]), denormalize_y(args["end_y"]))
           page.mouse.up()
       else:
           raise ValueError(f"Unsupported action: {name}")
   
       # Wait for the page to finish loading. Check readyState rather than a load
       # event, which pages restored from the back-forward cache never fire.
       page.wait_for_function("document.readyState === 'complete'")

6. Run the agent loop

   Send the task plus the initial screenshot, execute each returned action, and answer every function_call with a function_result that carries the current URL and a fresh screenshot. previous_interaction_id keeps the conversation on Google's side, so you only send the new results each turn. The loop ends when the model stops emitting function calls.

   It also stops if an interaction ends as incomplete, failed, or cancelled. incomplete means the model used up max_output_tokens, and the example doesn't resume it. To resume, call interactions.create again with previous_interaction_id, the interaction's continuation_token, and a higher max_output_tokens. See Google's long decode guide.

   from google import genai
   
   MODEL = "gemini-3.8-flash"
   # Prompt injection detection scans screenshots for hidden instructions. It's opt-in.
   TOOLS = [{"type": "computer_use", "environment": "browser", "enable_prompt_injection_detection": True}]
   
   client = genai.Client()  # Reads GEMINI_API_KEY from the environment.
   
   
   def screenshot_b64() -> str:
       page.wait_for_function("document.readyState === 'complete'")
       return base64.b64encode(page.screenshot(type="png")).decode("utf-8")
   
   
   def confirm(safety_decision: dict) -> bool:
       # Gemini flags sensitive actions. A human decides; the model doesn't.
       print(f"Safety confirmation required: {safety_decision.get('explanation', '')}")
       return input("Allow this action? [y/N] ").strip().lower() == "y"
   
   
   interaction = client.interactions.create(
       model=MODEL,
       input=[
           {"type": "text", "text": "What is the title of the top post on this page?"},
           {"type": "image", "data": screenshot_b64(), "mime_type": "image/png"},
       ],
       tools=TOOLS,
   )
   
   MAX_TURNS = 30  # Cap action turns so the loop can't run forever.
   # The extra pass reads the response to the last turn's actions before giving up.
   for turn in range(MAX_TURNS + 1):
       steps = interaction.steps or []  # A response can come back without steps.
       # "incomplete" means the model ran out of max_output_tokens, so its answer may be cut off or empty.
       if interaction.status in ("incomplete", "failed", "cancelled"):
           print(f"Stopped: the interaction ended with status {interaction.status}.")
           break
       calls = [step for step in steps if step.type == "function_call"]
       if not calls:
           answer = " ".join(
               block.text
               for step in steps if step.type == "model_output"
               for block in step.content if block.type == "text"
           )
           print("Done:", answer)
           break
   
       if turn == MAX_TURNS:
           print(f"Stopped after {MAX_TURNS} turns without a final answer; the task may be incomplete.")
           break
   
       results = []
       declined = False
       for call in calls:
           args = dict(call.arguments or {})
           result = {}
           decision = args.get("safety_decision")
           if decision and decision.get("decision") == "blocked":
               # Gemini's safety system blocked this action, so it must not run.
               print(f"Blocked: {decision.get('explanation', '')}")
               declined = True
               break
           if decision and decision.get("decision") == "require_confirmation":
               if not confirm(decision):
                   declined = True
                   break
               result["safety_acknowledgement"] = True
           try:
               execute_action(call.name, args)
           except Exception as err:
               # Report the failure to the model instead of crashing, so it can try another approach.
               result["error"] = str(err)
           results.append((call, result))
   
       if declined:
           print("Stopped: action blocked or not approved.")
           break
   
       screenshot = screenshot_b64()
       interaction = client.interactions.create(
           model=MODEL,
           previous_interaction_id=interaction.id,
           input=[
               {
                   "type": "function_result",
                   "name": call.name,
                   "call_id": call.id,
                   "result": [
                       {"type": "text", "text": json.dumps({"url": page.url, **result})},
                       {"type": "image", "data": screenshot, "mime_type": "image/png"},
                   ],
               }
               for call, result in results
           ],
           tools=TOOLS,
       )
   
   browser.close()
   playwright.stop()

7. Run it

   python agent.py

   Gemini reads the page from the screenshot and replies with the title:

   Done: The top post on Hacker News is "Show HN: …".

Supported actions

Gemini returns these actions in the browser environment. Every coordinate is on the 0–1000 grid and must be denormalized before use. Every action also carries an intent string explaining why the model chose it.

Action	Properties	Description
click / double_click / triple_click / right_click / middle_click	x, y	Click at coordinates
mouse_down / mouse_up	x, y	Press or release the mouse button at coordinates
move	x, y	Move the cursor
type	text, press_enter	Type text, optionally pressing Enter
press_key / key_down / key_up	key	Press / hold / release a key
hotkey	keys[]	Press a key combination
scroll	x, y, direction, magnitude_in_pixels (default 300)	Scroll at a position
drag_and_drop	start_x, start_y, end_x, end_y	Drag from one point to another
navigate	url	Go directly to a URL
go_back / go_forward	–	Browser history navigation
wait	seconds (default 1)	Pause for the page to settle
take_screenshot	–	No-op in this loop (state is captured each turn)

Safety confirmations

Gemini may attach a safety_decision to a function call's arguments with decision: "require_confirmation", for example before a purchase, sending a message, or accepting terms. Pause and ask a human before you execute that action. If they approve, set safety_acknowledgement in that call's function_result, as the loop above does. If they decline, stop the run. Google's docs list the built-in safety policy categories and how to override them.

Advanced configuration

Residential proxies

reCAPTCHA v3 and similar defenses score requests largely on IP reputation. If your automation completes the form but the site rejects the submission, route traffic through residential IPs:

BROWSERLESS_URL = (
    "wss://production-sfo.browserless.io/chromium/stealth"
    f"?token={os.environ['BROWSERLESS_API_KEY']}&proxy=residential&proxyCountry=us"
)

See Proxy configuration for country and session options.

Without stealth mode

If you don't need anti-detection and just want a managed cloud browser:

BROWSERLESS_URL = (
    f"wss://production-sfo.browserless.io?token={os.environ['BROWSERLESS_API_KEY']}"
)

Regional endpoints

Connect to the closest region for lower latency. See Connection URLs for all available endpoints.

When to use Computer Use vs. the MCP browser agent

Computer Use drives the browser from screenshots and pixel coordinates, so its reliability depends on the model clicking the right spot. For structured, multi-step flows (logins, form filling), the Browserless MCP server's browserless_agent tool is often more reliable: it drives the browser from accessibility/DOM snapshots and element references rather than coordinates, so it doesn't depend on pixel-accurate clicks and works the same across models. If you're seeing coordinate-related flakiness with Computer Use, consider the MCP agent for the structured parts of your workflow.

You can also keep deterministic Playwright steps for the parts of the flow you know and call the model only when a step fails. The hybrid pattern on the Anthropic page shows the structure. Wrap the loop above in a function and call it from the except block.

Why use Browserless with Gemini Computer Use

• Stealth mode: bypass bot detection by adding /stealth to the endpoint URL
• Residential proxies: route traffic through real residential IPs to improve IP reputation for score-based defenses like reCAPTCHA v3
• Global regions: choose US West, London, or Amsterdam endpoints for lower latency
• No infrastructure: skip managing Chrome installations, updates, or scaling
• Parallel sessions: run multiple browser sessions simultaneously

Resources

• Gemini Computer Use documentation
• Gemini Computer Use reference implementation (legacy: uses generate_content and the older action arguments, not the Interactions API flow in this guide)
• Google ADK page: Gemini agents that call the hosted Browserless MCP server
• Browserless connection options
• Stealth mode
• Proxy configuration

FAQ & Troubleshooting

The model's clicks land in the wrong place

You're passing Gemini's raw 0–1000 coordinates straight to Playwright. Denormalize every x/y to your viewport first (x / 1000 * viewport_width), and make sure the width/height you scale against match the viewport you set with new_context(viewport=...). See Denormalize coordinates.

The request fails with a model or tool error after upgrading

Check the model ID first. gemini-2.5-computer-use-preview-10-2025 was shut down on July 28, 2026, and gemini-3.5-flash no longer resolves to a Computer Use model. Use gemini-3.8-flash or another model from Google's model versions list. If you're porting code written for the legacy model, its action names (click_at, type_text_at, scroll_document) and argument names (destination_x, magnitude) differ from the current set in Supported actions.

The automation fills the form but the site rejects the submission

This is usually not a coordinate issue. Score-based defenses like reCAPTCHA v3 rely heavily on IP reputation. Add residential proxies to the CDP URL and keep the /stealth route. See Advanced configuration.

The model returns key names Playwright doesn't recognize

The KEY_MAP in the action handler covers common mismatches (control → Control, cmd → Meta, etc.). Add new ones as needed. The Playwright keyboard API docs list all valid key names.

The loop never stops

MAX_TURNS caps how many screenshot-action rounds run before giving up. Raise it for longer tasks, or break the task into smaller prompts so each run completes within the cap.

A site blocks the browser mid-task

Add residential proxies to the CDP connection URL and confirm you're still using the /stealth route from Connect to Browserless.

Next steps

Google ADK

build Gemini agents on the hosted Browserless MCP server

AI integrations overview

all supported AI frameworks and tools

BQL overview

use BrowserQL for advanced browser automation