LLM-generated end-to-end tests needed edits to 10% of their lines

Building end-to-end tests is always a trade-off between speed and reliability. The scripts have to walk the whole user journey: UI, business logic, integrations. Writing them by hand takes weeks and demands expertise in frameworks, selectors and stable locators. Large language models can already generate unit tests, but integration scenarios are a harder problem. The authors of GenIA-E2ETest set out to close that gap: take requirements written in plain language and turn them automatically into executable scripts for Robot Framework.

The idea in short

GenIA-E2ETest pairs an LLM with a crawler: it reads a text scenario, locates the interface elements it needs on the pages, and emits a finished E2E script. The approach is open, modular and not tied to one stack — today it runs Robot Framework with Selenium, but the concept carries over to Playwright or Cypress.

How it works under the hood

The system is built on three levels of prompting:

• Parsing the scenario. The LLM turns free-form text into a structured list of steps, broken down by page, with the checks each step expects.
• Finding and refining UI elements. The crawler (Crawl4AI) pulls the HTML, the LLM extracts candidates with their types and XPaths, and a second pass strips duplicates and hardens the selectors.
• Generating executable code. From the validated JSON the LLM writes a readable Robot Framework script: keywords, navigation, data entry and assertions.

How they tested it

The evaluation ran on two web applications: the public AutomationExercise site (login, registration, forms, cart) and a teaching movie service built on React/Vite with dynamic routes. Twelve scenarios, three runs each — 36 runs in total. Generating all the scripts took about 26 minutes; the scripts were then executed under supervision, with every small fix recorded. The model was ChatGPT-4o (the gpt-4o-mini variant) at temperature zero.

How it performed

• On interface elements, precision averaged 77% and recall 77%. Nine of the twelve scenarios cleared the 70% mark, five came in above 91%. The main failure was the scenario with context-dependent navigation.
• On script execution, precision was 82% and recall 85%. Those figures already include small manual fixes: adjusting XPaths, adding waits, renaming keywords.
• Step coverage came out at 104%, thanks to sensible decomposition: where the user wrote “enter email and password”, the system produced two separate steps plus extra page-load checks.
• The average edit rate was 10% of lines per scenario (median 6%). In simple cases it was close to zero; in the hard context-dependent one it reached 49%.

Where it gets thin

The approach holds up on familiar navigation and structured interfaces. Trouble starts where:

• the next page depends on hidden conditions and state has to be carried explicitly;
• elements appear dynamically and change their attributes;
• the semantics of a step are ambiguous (a link styled as a button throws off the choice of locator);
• outside noise such as pop-ups blocks clicks;
• unstable IDs in modern frameworks break the XPaths.

The authors sketch out where to strengthen it: better context retention between steps, more robust locator strategies, semantic matching of elements, and prompts adapted to long scenarios.

What this means for teams

GenIA-E2ETest shortens the ramp-up to automation, especially where there are many routine flows: portals, admin panels, e-commerce. A tester states the scenario in natural language, the system generates working code, and an engineer cleans up the hard parts. That lowers the barrier to entry and frees time for the scenarios that are genuinely difficult. All it asks is that the pages be reachable during element extraction and that the steps be written unambiguously.

The solution differs from methods built on interface exploration and reinforcement learning: instead of autonomous navigation there is an explicit scenario, and the intelligence goes into interpreting the text and mapping it precisely onto the DOM. Next to proprietary record-and-replay platforms, the approach is open, transparent and fits Robot Framework cleanly.

This is not yet a magic “generate everything” button, but in its niche GenIA-E2ETest is already useful: it produces executable E2E scripts fast, keeps them readable and needs few edits. The next step is handling dynamic pages and context reliably. If that works, automation from text becomes routine rather than an experiment.

💾 Code