Porting 23 Playwright tests to an agentic framework

I ported the same automationexercise.com journeys to e2e, a framework that reads the page as an accessibility tree and locates by role and name. The suite had not run in a while, and the first honest run failed 15 of 23 tests.

Strict locators find loose assumptions

e2e refuses a locator that matches more than one node, so every loose regex showed up at once:

• /men/i also matched "Women", and /kids/i matched "Kookie Kids". Exact names fixed both.
• The login page has two "Email Address" boxes, one for login and one for signup. .first() and .last() say which one I mean.
• "Add to cart" on product cards is an anchor without an href, so its role is text, not link.
• The search and subscribe buttons have no accessible name. I located them by position and by their container, and that is worth a bug report to the site.

Green locally, red in CI

Three tests passed on my machine and failed in GitHub Actions with "locator matched 2 nodes". The CI log printed the second match: a link named "Polo T-Shirts" next to the real brand link "(6) Polo". My sandbox blocked Google ads; CI loaded them, and the ads injected links that collided with the site's own. Blocking ad networks before each test made every run see the same page.

const AD_HOSTS = /googlesyndication|doubleclick|googleadservices|adservice\.google/;

export async function blockAds(browser: Browser) {
  await browser.route(AD_HOSTS, (route) => route.abort());
}

The bug that wasn't

One or two random tests still failed per run, with the framework reading the page as a near-empty 2-node tree. It looked like an engine bug. Before filing it, I recorded the framework's view and the real DOM side by side every 400 ms.

They agreed every time. The site sometimes serves a "One moment, please..." holding page for about five seconds, and the default assertion timeout is five seconds. The framework was reading a nearly empty page correctly. Checking against ground truth first saved me from filing a false bug report with the maintainers.