Skip to content
MiraiTech
AI Lab

Experiment · Automation

Browser automation agent

A Playwright agent that reads each page with a vision model and decides what to click or type next. It can follow multi-step instructions such as "find the cheapest USB-C hub with at least four stars and add it to the cart".

Type
Automation
Status
Live
Models
GPT-4o Vision, Claude Sonnet

Results

  • 78% task completion
  • Multi-step navigation
  • Reads pages visually

The agent completed 78% of the 50 benchmark tasks. Single-site tasks took 34 seconds on average and multi-site tasks 92 seconds. Error recovery saved 15% of failing tasks. The agent handled dynamic content, pop-ups and cookie banners without site-specific code.

What we tested

A browser agent that takes a goal in plain English, such as "Find the cheapest flight from SFO to JFK next Friday", then moves through websites, fills in forms, clicks buttons and extracts the results on its own. It uses Playwright for browser control, GPT-4o Vision to understand pages and LangGraph for multi-step planning. There is no site-specific code.

Method

We built a 50-task benchmark in 5 categories: e-commerce (search, filter, add to cart), travel (flight search, hotel booking), information retrieval (Wikipedia lookups, documentation search), form filling (contact forms, applications) and multi-site workflows (price comparison across 3 sites). Each task was run 3 times to measure consistency. The agent reads page layouts with GPT-4o Vision, finds interactive elements by parsing the accessibility tree and plans multi-step actions with LangGraph.

Findings

  1. 01

    78% of 50 tasks completed

    The agent completed 78% of the benchmark tasks end to end. It did best on e-commerce (88%) and worst on multi-site workflows (62%), where keeping context across site transitions was hard.

  2. 02

    Screenshots plus the accessibility tree beat screenshots alone

    Combining screenshots with accessibility tree parsing improved task completion by 22% over vision alone. The accessibility tree gives reliable element selectors, which screenshots cannot.

  3. 03

    Retrying a different way recovered failed actions

    When an action failed, the agent retried with a different strategy, for example keyboard navigation instead of a click. That recovered 15% of failed actions and raised overall completion from 65% to 78%.

  4. 04

    CAPTCHAs and bot detection set the limit

    Of the failures, 60% came from CAPTCHAs, bot detection or dynamic loading that changed the page after it first rendered. Every browser automation agent runs into these limits.

How it is built

The user gives a goal in natural language. LangGraph breaks it into a sequence of web actions (navigate, click, type, scroll, extract). For each action, Playwright captures a screenshot and the accessibility tree, GPT-4o Vision picks the target element and action type, and Playwright performs the interaction. After each step, the agent checks the resulting page for the expected outcome. If the check fails, an error recovery loop tries other ways to interact.

Playwright drives the browser with full Chrome DevTools Protocol (CDP) access. GPT-4o Vision reads page layouts and identifies interactive elements in screenshots. LangGraph manages the multi-step planning and execution graph. Redis caches page states for error recovery and rollback.

  • Playwright
  • GPT-4o Vision
  • Python
  • LangGraph
  • Redis

Problems we hit

Dynamic content and lazy loading

Elements that loaded after the first render were missing from the first screenshot. The agent now scrolls the page, waits for network idle and takes several screenshots before it decides an element is missing.

Many similar buttons

Pages with several similar buttons, such as a list of "Add to Cart" buttons, confused the vision model. We split selection into two steps: find all candidate elements, then use the surrounding context (product name, price) to pick the right one.