What the Automated Browser Benchmark tests, how a run works, and how to read the results without over-interpreting them.
Latest published infrastructure run: Oct 21, 2026
Browser providers are difficult to compare from feature lists alone. A fast session may struggle on a protected site, a strong stealth product may slow down under load, and an agent that completes a simple lookup may fail on a form or cart flow.
The Automated Browser Benchmark turns those trade-offs into repeatable tests. It asks two separate questions: how well does the browser infrastructure perform, and how reliably can a hosted browser agent finish real web tasks? The tracks stay separate because speed and task completion are different product decisions.
Both tracks use versioned inputs and preserve the raw results. They differ in what they run and what counts as success.
Track 1 · Infrastructure
8 managed browser providers are tested on 34 open pages and 35 pages protected by real bot defenses. Each target is sampled 10 times at one session and five concurrent sessions.
Track 2 · Agents
Seven hosted agent products run 50 real web tasks, normally three times each. The set covers reading, forms, search and filtering, cart-style interactions, and generated artifacts such as screenshots and PDFs.
Provider-specific code connects to each product. Shared code controls the inputs, timing, classification, and output format.
A run uses checked-in target or task files, a declared region, and recorded provider settings.
Every provider receives the same applicable workload. One provider failing does not stop the others.
Timings, success state, errors, sample counts, and available costs are written before the site aggregates anything.
Schemas reject malformed output. Published files record the commit, runtime, input versions, region, and pricing snapshot.
The runner requests a fresh browser, connects, loads a page, and checks for content that should be on the real page. That last check matters: a challenge page can still return HTTP 200. A protected visit only counts as a bypass when the expected content appears.
Baseline pages isolate normal browser performance. Protected pages exercise full stealth. Concurrency 1 shows an isolated cold start; concurrency 5 includes queueing and contention.
The vendor's hosted agent receives a starting URL and a plain-language task. It reasons with its own model and tools, then returns an output or artifact. A five-minute limit prevents a stuck task from holding a run open forever.
Repeated trials expose flaky success, not just a lucky pass. Results are grouped by interaction shape so simple reading cannot hide weaker form or transactional work.
There is no single “best browser” score. Reliability, speed, and cost remain separate, and the two tracks never share a ranking. Start with the workload that resembles yours, then compare the metrics that matter.
| Result | What counts | What does not count |
|---|---|---|
| Track 1 bypass | The expected real-page content is present. | A request that only reached a challenge or block page. |
| Track 1 latency | Recorded session, navigation, and end-to-end timings from successful work. | Failed or blocked attempts treated as fast measurements. |
| Track 2 success | Verified ground truth, real artifact bytes, or an independent yes/no judge. | The agent saying it succeeded, or a run that ended with an error. |
| Missing data | A dash or “not benchmarked,” with the reason preserved. | A made-up zero that would unfairly change a ranking. |
Track 2 uses deterministic checks whenever possible: fact answers match versioned or live ground truth, and screenshots or PDFs must contain real file signatures. Open-ended tasks use a separate judge that sees the task and final output. Artifact tasks appear in their own breakdown and do not affect the headline rate.
A benchmark is useful only when its assumptions are visible. These rules are built into the runner and published result format.
Check out the commit recorded with a result, use the pinned Node and npm versions, add keys for the providers you want, and declare the same region. Live benchmark commands can create billable usage.
git clone https://github.com/browserless/browser-benchmark cd browser-benchmark nvm install && nvm use npm ci cp .env.example .env export BENCHMARK_REGION=eu-west npm run benchmark:infrastructure # Track 1 · billable npm run benchmark:agentic # Track 2 · billable
The detailed methodology explains measurement and scoring. The operator guide covers setup and commands, and the official results directory contains the files this site reads.
Found a stale target, unfair assumption, or parsing error? Open a correction. Methodology changes are recorded in the changelog and do not rewrite older results.