Benchmark

How many tokens does a browser agent spend to see a page, and what leaks to the model? We ran WhittleBrowser, Playwright MCP, browser-use and Stagehand on the same pages in the same browser and counted exactly what each sends to the LLM.

Including fixed per-call overhead, WhittleBrowser used 93% fewer tokens than Playwright MCP, 86% fewer than browser-use and 87% fewer than Stagehand to look at the same 11 pages, and it was the only one of the four that let none of the planted secrets or hidden injections through. In real agent runs on whole tasks the saving is smaller and varies by task: see With a real agent.

Read this first

Two different benchmarks, answering two different questions:

1. Page views (no model). For the same pages, how many tokens does each tool put in front of the model? This is deterministic and cheap to rerun. It is the first half of this page.
2. Real agent runs (Claude). With a real model driving each tool, does the task still get done, and what does it cost? This is the section With a real agent. Results are noisier and cover fewer tasks.

Terms used below:

• Token: the unit a language model reads and is billed in, roughly three-quarters of a word. We count with the o200k_base tokenizer as a model-neutral stand-in.
• Page view / snapshot: the text a tool returns so the model can see the page.
• Fixed overhead: text sent on every request that is not the page, such as tool definitions and system prompts. Prompt caching makes repeats cheaper to pay for, but they still fill the context window.
• Modeled: computed from measured page sizes and how a tool calls its model, instead of running a real agent. We say so wherever it applies.

The short version.

• On page views, WhittleBrowser is far smaller than Playwright MCP and Stagehand, and about half the size of browser-use. On two text-heavy pages (BBC News, Python docs) its page text is about the same size as browser-use's.
• On whole tasks with a real model, WhittleBrowser finished every task it was given. It used far fewer tokens on large or re-rendering pages, and about the same as the others on small pages. One live task, counting star ratings, needed extra engine support before it was cheap (described below).
• Stagehand's text view, as we drove it, was about as small as ours over whole tasks, even though its page views are larger in the table below.

Results at a glance

How it was tested

Same pages, same browser. Every tool loads the same 11 URLs in the same Chromium build (Playwright's Chrome for Testing) at a 1280×860 viewport:

• 5 local test pages: a shipping form, a React-style to-do app that re-renders, a catalog with 30 product cards and a 120-row table, a 40-paragraph article, and a page with prompt-injection text.
• 6 live sites: Wikipedia (Pancake), Hacker News, a GitHub repository, MDN <input>, BBC News, Python asyncio docs.

A note on Playwright MCP's numbers. After a tab's first page, Playwright numbers each element ref with a navigation prefix (f1e2, f2e2, ...), which makes its snapshots longer. We measure the pages in order in one tab, as an agent uses a tab, so the Playwright figures include that prefix. On a freshly opened page they are about 13% lower: the product catalog page is 10,106 tokens in the benchmark and 8,758 fresh, against 672 and 661 for WhittleBrowser. The gap on that page is about 13 to 15 times either way.

Each tool's own code, no re-implementation:

Exact text, counted. No LLM is called. We capture the exact text each tool would send and count it with the o200k_base tokenizer, as a model-neutral proxy. The only estimate is screenshot tokens for browser-use (vision is on by default), at Claude's rule of about width × height / 750 ≈ 1,468 tokens per 1280×860 screenshot.

Fixed overhead is what every call carries besides the page: tool definitions for MCP servers, the system prompt plus action schema for browser-use (5,242 + 5,765), and the act system prompt and instruction template for Stagehand (rebuilt from its own prompt builders). Prompt caching lowers the price of repeats, but they still occupy the context window.

Multi-step tasks. Playwright MCP and WhittleBrowser are measured by running each task end to end. browser-use and Stagehand are modeled from their measured page sizes and how they call the model:

• browser-use sends the full state on every agent step: up to 5 actions per step, but a step ends when the page changes.
• Stagehand sends the page tree on every act or extract call.

Safety. A page with 7 planted secrets (a test card number, three API keys, a JWT, an SSN, and a password inside a field) and a page with 5 hidden prompt injections (white-on-white, 1px font, off-screen, aria-hidden, opacity: 0). We count how many reach the text each tool sends.

Page by page

browser-use's page text is the leanest of the three competitors. It keeps mostly interactive elements and nearby text, then adds a screenshot and a large fixed prompt on every step. On two text-heavy pages (BBC News, Python docs) its page text is about the same size as ours. WhittleBrowser's advantage there comes from the fixed overhead and screenshots, not the page text.

Across a whole task

The article task asks for the full text of two sections, so WhittleBrowser returns the article's content (about 1.9k tokens). The browser-use model counts only one page state there, which under-counts it: its extract action sends the page markdown in a separate call.

All-in cost

Safety

WhittleBrowser also flags visible instruction-like text with ⚠ and auto-pauses on payment pages; none of the other tools do either.

With a real agent

The page-view numbers above leave one question open: does a model still finish the task with a smaller view, and what does a finished task cost? We ran Claude Code as the agent (claude -p, built-in tools off, a short system prompt) with each browser tool attached as an MCP server, on Haiku and Sonnet, 7 tasks × 4 tools × 3 repetitions = 168 runs. Every answer is scored automatically.

Tasks. Four local pages: find two prices in the 30-card catalog and 120-row table, count the 40 paragraphs of the article, fill the shipping form and place the order, and tick three boxes on the re-rendering to-do list then read its render counter. Three live pages on toscrape.com: the cheapest Travel book, the number of five-star Mystery books across two pages, and log in then read a quote on page 2.

What each arm is.

haiku (OpenAI estimate priced as gpt-6-luna)

sonnet (OpenAI estimate priced as gpt-6.1-sol)

Pass rate and mean tokens per task (all models)

Cost of one pass over the battery (7 tasks)

Reading the numbers.

• Success is close to level. WhittleBrowser finished all 42 runs; the misses were Playwright MCP on two spa runs and one table run, Stagehand on three books-five-star runs, and browser-use on one login run.
• Where pages are big or re-render, a small view helps a lot: on spa WhittleBrowser used about 23k tokens against 97k for Playwright MCP and 106k for browser-use, and on form about 20k against 46k and 147k.
• Where pages are small, the savings mostly disappear. Tool definitions and the model's own turns dominate, and on short tasks (table, books-cheapest) all four tools are within a few thousand tokens of each other.
• Stagehand was about level with WhittleBrowser (27k against 28k on Haiku, 25k against 20k on Sonnet), so its larger page views in the tables above do not turn into larger whole-task costs on pages this size. We have not isolated why. This arm leaves out Stagehand's own agent prompts and exposes 8 small tools, and the model usually needed only one snapshot, so it probably flatters Stagehand.
• books-five-star exposed a gap, which we then fixed. A star rating is a CSS class, so no tool's text view shows it. Playwright MCP has a JavaScript evaluate tool and counted the elements in 2 to 6 calls; WhittleBrowser had no equivalent and took 15 to 20 screenshots, about 128k tokens against 32k. We added two things: text-less rating elements now show as rating 5/5 in the page view, and find css:SELECTOR counts visible matches and lists their class and attributes without running any script. After that WhittleBrowser took 4 to 15 calls and about 39k tokens on this task, still above Playwright MCP's 32k. The "before fix" rows keep the old numbers for comparison.
• The other tasks also moved between the two WhittleBrowser runs (for example spa 27k to 23k), and the same code ran in both, so with 3 repetitions differences of that size are noise. Only the books-five-star change is large enough to be the fix.
• On Sonnet WhittleBrowser now uses fewer tokens than Playwright MCP (20k against 29k per run) but still costs more per pass ($0.145 against $0.119 on Claude list prices). We have not isolated why; the cache-read versus cache-write mix is a likely part of it.

What the OpenAI columns mean. They re-price the same measured tokens at OpenAI list prices (gpt-6-luna for the Haiku runs, gpt-6.1-sol for the Sonnet runs; $0.10 / $0.01 cached / $0.50 and $2 / $0.10 cached / $10 per 1M tokens, from developers.openai.com/api/docs/pricing). They are an estimate of what this battery would cost on OpenAI, not a measurement: a different model uses a different tokenizer and takes a different number of turns.

Limits of this benchmark

• The page-view tables measure input size and leaks with no model. The agent section adds success and cost, but only for Claude (Haiku and Sonnet), 7 tasks and 3 repetitions each, so one slow run moves an average. It does not cover other models, and the tasks are small.
• Token counts use one tokenizer; real counts vary a little by model.
• browser-use and Stagehand task totals are modeled; page sizes and safety numbers are measured.
• Live sites change daily, so numbers drift between runs. The JSON with every number is in docs/benchmark.json.

Reproduce it

packages/engine/bench/compare/setup.sh   # once: browser-use (uv venv) and Stagehand (npm)
pnpm bench -- --live --pw-mcp            # writes bench/results/latest.md and docs/benchmark.json
pnpm bench:agent --models haiku,sonnet --reps 3   # agent run, needs the claude CLI; then: pnpm tsx bench/agent/summarize.ts