Playwright Patterns

E2E testing patterns with Playwright for CI and production verification -- CI-safe test splitting, console error monitoring, image interception, production build verification, and sharded CI runs.

CI-Safe vs @interactive Test Split

Not all E2E tests can run in CI. Tests requiring keyboard shortcuts, clipboard access, or desktop-specific interactions should be tagged and split:

// e2e/basic-navigation.spec.ts -- runs in CI
import { test, expect } from "@playwright/test";

test("loads the home page", async ({ page }) => {
  await page.goto("/");
  await expect(page.locator("h1")).toBeVisible();
});

// e2e/keyboard-shortcuts.spec.ts -- only runs locally
import { test, expect } from "@playwright/test";

test("@interactive Ctrl+S saves document", async ({ page }) => {
  await page.goto("/editor");
  await page.keyboard.press("Control+KeyS");
  await expect(page.locator(".save-indicator")).toHaveText("Saved");
});

// playwright.config.ts
import { defineConfig } from "@playwright/test";

export default defineConfig({
  projects: [
    {
      name: "ci",
      grepInvert: /@interactive/,
    },
    {
      name: "interactive",
      grep: /@interactive/,
    },
  ],
});

Guard against specs that match no project

The CI-safe / interactive split above leaves testMatch at its default (no path restriction) on the ci project, so every spec file is collected regardless of title — that implicit catch-all is exactly what prevents the trap described here. The trap only appears once you move to a partitioned setup where each project maps to a disjoint filename prefix (e.g. one project per fixture or app):

// playwright.config.ts — partitioned by filename prefix (NOT a catch-all)
import { defineConfig } from "@playwright/test";

export default defineConfig({
  projects: [
    { name: "fixtureA", testMatch: /fixtureA[^/]*\.spec\.ts/ },
    { name: "fixtureB", testMatch: /fixtureB[^/]*\.spec\.ts/ },
    { name: "fixtureC", testMatch: /fixtureC[^/]*\.spec\.ts/ },
  ],
});

In this config, a spec whose filename starts with anything other than fixtureA, fixtureB, or fixtureC matches no project and is collected by zero projects. Playwright runs without error — it simply has nothing to run — and no test failure reveals it. The missing spec produces no result to fail.

The fix mirrors the single-source-of-truth meta-test pattern: a tiny script that asserts every e2e spec filename starts with a known project prefix, wired into both the local pre-push gate and CI. Enumerate specs recursively with find (not ls e2e/*.spec.ts, which only globs the top level and would silently miss a spec nested in a subdirectory — the very hole this guard exists to close):

# Every e2e spec must start with a known project prefix, or it runs nowhere.
# find recurses into subdirectories; `ls e2e/*.spec.ts` would miss e2e/<dir>/*.spec.ts.
known='fixtureA|fixtureB|fixtureC'
bad=$(find e2e -type f -name '*.spec.ts' | grep -Ev "/($known)[^/]*\.spec\.ts$" || true)
[ -z "$bad" ] || { echo "specs match no Playwright project:"; echo "$bad"; exit 1; }

A Bad workers Value Runs Zero Tests and Exits Green

The guard above catches a spec that no project collects. There is a second route to the same false green, and it survives that guard untouched: every spec matches its project perfectly, and the run still executes nothing — because of the workers value alone.

config.workers accepts a number (4) or a percentage string ("50%"). It does not accept a numeric string: workers: "3" errors with config.workers must be a number or percentage. That asymmetry is what makes the obvious env-override idiom a footgun:

// playwright.config.ts -- Anti-pattern. Number() is the whole bug.
export default defineConfig({
  workers: Number(process.env.PW_WORKERS),
});

Number("abc") is NaN — a typo'd or stale env var. Number("50%") is also NaN — Playwright's own native percentage syntax, destroyed by the coercion meant to accept it. And with NaN workers, Playwright does not error. It prints no Running N tests line, no pass/fail summary, and exits 0:

$ PW_WORKERS=abc npx playwright test
$ echo $?
0

That is a gate reporting success having executed nothing.

The failure is asymmetric, which is why it survives casual testing — the values a developer thinks to try are the ones that behave:

Validate the override at config load, so a bad value fails at boot instead of at nothing:

// playwright.config.ts
function resolveWorkers(raw: string | undefined): number | string | undefined {
  if (raw === undefined) return undefined; // let Playwright pick its default
  if (/^\d+%$/.test(raw)) return raw;      // pass percentages through as strings
  const n = Number(raw);
  if (!Number.isInteger(n) || n < 1) {
    throw new Error(`PW_WORKERS must be a positive integer or a percentage -- got "${raw}"`);
  }
  return n;
}

export default defineConfig({
  workers: resolveWorkers(process.env.PW_WORKERS),
});

Returning undefined for the unset case matters: it hands the decision back to Playwright's own default rather than pinning a number the config author guessed.

This is the same lesson as the filename-prefix guard, one layer down: a green run proves the tests that ran passed — it never proves any test ran. The cheapest independent check is to assert the count, not the exit code. A gate that reads its own reporter output for a nonzero test count catches every member of this class at once — the uncollected spec, the NaN worker count, and whatever the next mechanism turns out to be. (It is a collected-count guard: a suite whose specs all test.skip still clears it — that is the separate pass-by-skip trap below.)

# The gate ran green -- but did it collect anything? Fail if the suite collected nothing.
# Note the report must live OUTSIDE outputDir to survive; see the outputDir section below.
REPORT=b4push-reports/report.json
rm -f "$REPORT"  # a stale report from a prior run would satisfy both checks below
PLAYWRIGHT_JSON_OUTPUT_NAME="$REPORT" npx playwright test --reporter=json

[ -f "$REPORT" ] || { echo "FAIL: no report written -- the run did not reach the reporter"; exit 1; }
count="$(jq '[.. | objects | select(.tests? and .title?)] | length' "$REPORT")"
[ "$count" -gt 0 ] || { echo "FAIL: suite collected 0 specs -- the gate is vacuous"; exit 1; }

Quarantining Flakes: The Retries-Asymmetry Trap

Beyond the CI-safe vs @interactive split, there is a third tag worth knowing: @flaky. It exists because of a subtle trap — CI and your local pre-push gate often run with different retry budgets, so a test can be green in one and red in the other.

The trap starts here:

// playwright.config.ts
import { defineConfig } from "@playwright/test";

export default defineConfig({
  // CI retries twice; local runs get zero retries.
  retries: process.env.CI ? 2 : 0,
});

With retries: 2 in CI, a test that passes on its second or third attempt is reported green. Run the exact same test on a local b4push gate with retries: 0 and it goes red on the first failure. The test did not change — only the retry budget did. This is the insight to internalize: "flaky" is gate-relative. A test is only as flaky as the strictest gate it has to clear.

When you have a known-flaky test that already lives on main, deleting it loses coverage. Instead, tag it @flaky in the title and quarantine it from the strict local gate without removing it:

# scripts/run-b4push.sh -- exclude @flaky from the strict local gate
CHROMIUM_INVERT="@interactive|@flaky"
WEBKIT_INVERT="@flaky"

# Chromium step: skip both @interactive and @flaky
pnpm test:e2e --project=chromium --grep-invert="$CHROMIUM_INVERT"

# WebKit @interactive step: run @interactive but still drop @flaky
pnpm test:e2e --project=webkit --grep="@interactive" --grep-invert="$WEBKIT_INVERT"

The Chromium step adds @flaky to its --grep-invert (alongside @interactive), and the WebKit @interactive step also excludes @flaky. The tests stay in the suite — CI still runs them and tolerates the occasional retry — but they no longer trip the zero-retry local gate.

test.step Shares One Test Timeout

test.step() is a reporting construct. It groups actions under a named node in the HTML report and the trace viewer — it does not open a new timeout budget. Every step inside a test() draws down the same single deadline: the timeout your config sets (Playwright's own default is 30s; suites with heavy setup routinely raise it), or whatever test.setTimeout() last set for that test. Nesting two scenarios in two steps buys a tidier report and nothing else.

That distinction only bites when the steps are independent sequential scenarios rather than phases of one scenario:

// Anti-pattern: two independent scenarios sharing one 60s test budget.
test("commit density stays invariant", async ({ page }) => {
  await test.step("free mode", async () => {
    await page.goto("/");
    // fresh app boot + scenario -- budgeted ~20-30s by this file's own comment
  });

  await test.step("fixed mode", async () => {
    await page.goto("/");
    // fresh app boot + scenario -- budgeted ~20-30s by this file's own comment
  });
});

Each step is a complete scenario: its own navigation, its own app boot, its own bounded waits. The file's own comment budgeted each pass at roughly 20–30s, while the config default was 60s for the whole test. Two passes near the top of their stated budgets overrun that, and the failure surfaces in the second step — where nothing is individually hung. Every wait in the Fixed-mode phase is bounded and event-keyed; the phase simply started with too little of the shared budget left.

That is what makes this class hard to read off a failure message: the step that fails is not the step that is slow. The Free-mode phase spends the budget; the Fixed-mode phase pays for it. Widening the deadline on any wait inside the second step changes nothing, because the deadline that fired belongs to the parent test().

The Fix: One test() per Independent Scenario

If two steps could each run standalone — fresh navigation, no state carried across the boundary — they are two tests. Splitting them gives each its own full budget while preserving the fresh setup they were already performing:

test("commit density stays invariant -- free mode", async ({ page }) => {
  await page.goto("/");
  // ...
});

test("commit density stays invariant -- fixed mode", async ({ page }) => {
  await page.goto("/");
  // ...
});

Splitting the pgen case -- a private GPU pattern generator -- this way produced, across three bundled Chromium runs, Free timings of 22.3 / 17.1 / 21.9s and Fixed timings of 15.5 / 11.3 / 16.8s — every one of them under 45s, against the same unchanged 60s per-test budget. The scenarios were never slow. Only their sum was.

When the stages genuinely cannot be split — they share expensive state, or the later stage asserts on something the earlier one produced — keep them in one test() but write the arithmetic down: sum the worst case of every stage deadline plus setup and teardown, derive an explicit test.setTimeout() from that sum, and leave the sum in a comment so the next person editing a stage deadline can see what they are spending.

This is the Playwright test-level shape of a budget that sums past its deadline; the runner-level shape — sequential stage deadlines totalling past a nextest or CI per-test kill deadline, where the kill also preempts the test's own diagnostic flush — is the sequential-stage-budget entry in the flake root-cause catalog.

test.skip as a Precondition — the pass-by-skip Trap

test.skip is for genuine environment dependencies: a test that only makes sense on a specific OS, or when a particular service is reachable. Even then, audit that your gold-standard CI hosts actually run the spec — a skip that fires on every environment you own is a permanent pass-by-skip: the suite reports green because the test never executed, not because the behaviour is correct. The test is broken, not flaky.

Preconditions that must always hold belong in hard assertions:

// Anti-pattern: silently skips when user is null, hiding a broken setup
test.skip(!user, "no user");

// Correct: fails loudly if setup is broken
expect(user).toBeTruthy();

See Decision Guide — When to Write a Heavy Test for the Step 0 gate that determines whether the precondition belongs in the test at all.

Selector Actionability: When the Right-Looking Element Is the Wrong Target

A locator's contract is to resolve to a DOM node. Its contract is not to resolve to the node you can actually act on — and on rich UIs (canvas editors, multi-surface viewports, portalled overlays) those two nodes routinely differ. The element looks right in the inspector, toBeAttached() passes, and the click still lands on the wrong surface or times out. Three shapes of this trap recur, and all three share one fix: make the selector encode the intended active surface or state, then assert actionability on that exact target before acting — never select on element type or first match and hope the resolved node is the live one.

.first() on Duplicate Surfaces Grabs the Hidden One

Symptom: A viewport shows exactly one visible <canvas>, yet the test interacts with nothing — a click draws onto a surface that never appears on screen, or a drag lands in empty space. A mid-test screenshot shows the pointer action missing the rendered content entirely.

Diagnosis: DOM order is not render order. Frameworks keep more than one surface mounted all the time — a previous slide, an offscreen double-buffer, a display:none export canvas — and page.locator("canvas").first() returns whichever appears first in source, not whichever is painted. Element type (canvas) and position (first) are the two weakest signals you can select on: neither says anything about which surface is active.

Fix: Encode the active surface in the selector — scope to the visible/active container, filter by a state attribute ([data-active], [aria-hidden="false"]), or select canvas:visible — and assert toBeVisible() on the chosen target before touching it:

// Anti-pattern: DOM order != render order. `.first()` returns the first
// <canvas> in source order — which may be a hidden, offscreen, or torn-down
// surface the framework has left mounted.
await page.locator("canvas").first().click();

// Fix: scope to the active container and select the visible surface, then
// prove you picked the rendered target before acting on it.
const surface = page.locator('[data-active="true"] canvas:visible');
await expect(surface).toBeVisible();
await surface.click();

If the app has no state attribute to scope on, .filter({ visible: true }) narrows a multi-match locator to the one on screen — but adding a data-active / aria-hidden marker to the surface container is the durable fix, because it makes the intended surface explicit instead of inferring it from paint state.

A Label Shared by a Slider and a Text Input Trips Strict Mode

Symptom: The test throws before any action runs — the failure is at resolution, not interaction:

Error: strict mode violation: getByLabel('Amount') resolved to 2 elements:
    1) <input type="range" aria-label="Amount"> …
    2) <input type="text"  aria-label="Amount"> …

Diagnosis: A range input and a paired numeric/text input under the same panel both expose the accessible name Amount (a slider with a linked value box is the classic case). getByLabel matches on accessible name and is role-agnostic, so it selects both, and Playwright's strict mode refuses to guess which one you meant.

Fix: The two controls share a name but not a role — disambiguate by role. The range input is a slider; the value box is a textbox (type="text") or spinbutton (type="number"). Note a range input is not fillable — fill() throws Input of type "range" cannot be filled — so drive the slider with a role-appropriate action and fill() the value box:

// Anti-pattern: name-only, role-agnostic — resolves 2 elements, throws.
await page.getByLabel("Amount").fill("50");

// Fix: same name, different role — the ambiguity disappears.
await page.getByRole("textbox", { name: "Amount" }).fill("50"); // value box
await page.getByRole("slider", { name: "Amount" }).press("ArrowRight"); // nudge the range input

// If both controls genuinely share a role too, scope to the dialog/panel
// and reach for a test id rather than the shared label:
await page
  .getByRole("dialog", { name: "Transfer" })
  .getByTestId("amount-input")
  .fill("50");

An Overlay Intercepts the Click Even Though the Target Is Visible

Symptom: The target is visible and enabled, yet .click() times out. The call log names a different element than the one you selected:

locator.click: Timeout 30000ms exceeded.
Call log:
  - waiting for element to be visible, enabled and stable
  - element is visible, enabled and stable
  - scrolling into view if needed
  - <div class="modal-backdrop">…</div> from <div id="portal-root">…</div> subtree intercepts pointer events
  - retrying click action

Diagnosis: Playwright runs four actionability checks before dispatching a click — the element must be visible, stable (not mid-animation), enabled, and receiving pointer events (nothing else sits on top at the click point). A portalled overlay — a modal backdrop, a toast, a dropdown that just opened, a full-viewport loading veil — is layered above the target at those coordinates, so the fourth check never passes even though the first three do. The target is fine; something is on top of it.

Fix: Wait for the intercepting layer to leave, or scope to the real interactive layer, then click normally so the actionability checks keep protecting you:

// Fix: wait for the intercepting overlay to go away, then click normally.
await expect(page.locator(".modal-backdrop")).toBeHidden();
await page.getByRole("button", { name: "Save" }).click();

When Playwright reports a hidden or intercepted target and the selector looks correct, stop guessing and look: capture a screenshot or dump the computed geometry (bounding boxes, z-index, pointer-events) of both the target and whatever is on top of it. Seeing what actually occupies the click point turns "the click mysteriously times out" into "the toast at these coordinates has a 400ms exit animation." The deterministic computed-style dump plus the screenshot review in Level 5: Deterministic + Visual Verification is the tool for exactly this — it shows you the on-top element's geometry that the strict-mode error only hints at.

The through-line: select on the intended active surface or state, not on element type or source position; assert visibility/actionability on the actual target before acting; disambiguate shared labels by role or a panel-scoped test id; and gate readiness-dependent interactions on an explicit signal instead of a fatter timeout.

Editor Input in E2E

Driving a code editor (CodeMirror, Monaco, ProseMirror, or any contenteditable) from Playwright is harder than page.fill(). If the editor has a vim mode, page.keyboard.type("hello") is a disaster: the leading h moves the cursor left, i enters insert mode, and the rest is interpreted as commands rather than text.

The reliable approach is to select all existing content via the DOM Selection API, then push the new content with page.keyboard.insertText(). insertText dispatches a synthetic input event that the editor handles directly, bypassing vim-mode command interpretation entirely:

// e2e/helpers.ts
import type { Page } from "@playwright/test";
import { expect } from "@playwright/test";
import os from "os";

// Platform-aware modifier: Meta on macOS, Control on Linux/Windows
export const mod = os.platform() === "darwin" ? "Meta" : "Control";

export async function setEditorContent(page: Page, content: string) {
  const editor = page.locator(".cm-content");
  await editor.waitFor({ timeout: 5000 });
  await editor.click();

  // Select all content via the DOM Selection API (works regardless of vim mode)
  await page.evaluate(() => {
    const el = document.querySelector(".cm-content");
    if (!el) return;
    const range = document.createRange();
    range.selectNodeContents(el);
    const sel = window.getSelection();
    sel?.removeAllRanges();
    sel?.addRange(range);
  });

  // insertText dispatches an input event the editor handles directly,
  // bypassing vim-mode command interpretation entirely.
  await page.keyboard.insertText(content);

  // Wait for the Lezer parse + decoration updates to land before asserting.
  const firstLine = content.split("\n").find((l) => l.trim()) || content;
  await expect(page.locator(".cm-content")).toContainText(firstLine.slice(0, 20), {
    timeout: 5000,
  });

  // wait-ok: 500ms is the known auto-save debounce constant; split-pane reads
  // content back from the backend, so the test must wait >= the debounce or it races the persist.
  await page.waitForTimeout(500);
}

The platform-aware mod helper lets the same spec drive editor shortcuts on macOS (Meta) and Linux/Windows (Control) without branching in every test.

const POST_MOUNT_LOOP_SETTLE_MS = 2000;

test("no update-depth errors on startup", async ({ page }) => {
  const errors: string[] = [];
  page.on("console", (msg) => {
    if (msg.type() === "error") errors.push(msg.text());
  });

  await page.goto("/");

  // wait-ok: asserting ABSENCE of errors over a time window — no positive
  // event to poll for; converting to a condition wait would gut the assertion.
  await page.waitForTimeout(POST_MOUNT_LOOP_SETTLE_MS);

  expect(errors.filter((e) => e.includes("Maximum update depth"))).toEqual([]);
});

Ratcheting Down Wait Debt

Every waitForTimeout without a // wait-ok: <why> annotation is a debt item: it might be correct, but nobody can tell at a glance. The ratchet baseline turns that into a tracked, decreasing count rather than an invisible accumulation.

The check script

The script greps for unannotated waitForTimeout calls — those not preceded by a // wait-ok: comment within the two lines above — and compares the per-file count against a committed baseline:

#!/usr/bin/env bash
# scripts/check-wait-debt.sh
set -euo pipefail

BASELINE_FILE="e2e/wait-debt-baseline.txt"
SPEC_DIR="e2e"

# Nothing to check until the baseline has been introduced (existence guard).
[ -f "$BASELINE_FILE" ] || exit 0

# Count waitForTimeout calls that lack a // wait-ok: comment in the 2 lines above.
count_unannotated() {
  local path="$1" hits
  [ -f "$path" ] || { echo 0; return; }
  hits=$(grep -n "waitForTimeout" "$path" 2>/dev/null || true)
  [ -n "$hits" ] || { echo 0; return; }
  printf '%s\n' "$hits" | while IFS=":" read -r lineno _rest; do
    start=$(( lineno - 2 )); [ "$start" -lt 1 ] && start=1
    sed -n "${start},$((lineno - 1))p" "$path" | grep -q "wait-ok:" || echo found
  done | wc -l | tr -d ' '
}

# Expected count for a path: its baseline line, or 0 if absent (the implicit-zero rule).
expected_for() {
  awk -v p="$1" '$2 == p { print $1; found=1 } END { if (!found) print 0 }' "$BASELINE_FILE"
}

# Check EVERY spec file (so a file absent from the baseline is held to an implicit 0),
# unioned with the baseline's own paths (to catch a now-deleted file that still has an entry).
failed=0
checked=""
for path in $(find "$SPEC_DIR" -type f -name '*.spec.ts' 2>/dev/null) $(awk '{ print $2 }' "$BASELINE_FILE"); do
  case " $checked " in *" $path "*) continue ;; esac
  checked="$checked $path"
  expected=$(expected_for "$path")
  actual=$(count_unannotated "$path")
  if [ "$actual" -gt "$expected" ]; then
    echo "FAIL $path: $actual unannotated waits (baseline $expected) — annotate new waits with // wait-ok: <why>"
    failed=1
  elif [ "$actual" -lt "$expected" ]; then
    echo "FAIL $path: baseline is stale ($expected → $actual) — shrink the baseline to $actual"
    failed=1
  fi
done

exit "$failed"

Baseline format

The baseline file records the per-file count of unannotated waits — not line numbers, so unrelated edits don't churn it:

2 e2e/editor.spec.ts
1 e2e/startup.spec.ts

Rules:

• actual > baseline — new bare wait added; CI fails.

• actual < baseline — baseline is stale; CI fails with "shrink the baseline to N". The baseline may only decrease, never increase without a matching annotation.

• File absent from baseline — implicit count of 0; any unannotated wait fails immediately.

Wiring into pre-push and CI

# .github/workflows/e2e.yml (excerpt)
- name: Check wait debt
  run: bash scripts/check-wait-debt.sh

# scripts/run-b4push.sh (excerpt)
bash scripts/check-wait-debt.sh

The existence guard ([ -f "$BASELINE_FILE" ] || exit 0) means you can introduce the script before the baseline file exists — no breakage during rollout.

Known tradeoff

The add-one-remove-one case is invisible: if a single file gains one unannotated wait and loses another, the count stays the same and the ratchet does not catch it. This is acceptable for a debt ratchet — the goal is a monotonically shrinking total, not per-line enforcement. Pair with code review for the edge case.

Generalising to other debt classes

The same pattern applies to any greppable debt: any casts without a // any-ok: <why> comment, TODO comments without an issue reference, disabled lint rules without an expiry. Introduce one baseline file per debt class and wire them all into the same pre-push pass.

Console Error Monitoring

Extend Playwright's test fixture to automatically fail on console errors:

// e2e/fixtures.ts
import { test as base, expect } from "@playwright/test";

export const test = base.extend<{ consoleErrors: string[] }>({
  consoleErrors: async ({ page }, use) => {
    const errors: string[] = [];

    page.on("console", (msg) => {
      if (msg.type() === "error") {
        errors.push(msg.text());
      }
    });

    page.on("pageerror", (error) => {
      errors.push(error.message);
    });

    await use(errors);

    // Assert no console errors after each test
    expect(errors).toEqual([]);
  },
});

export { expect };

// e2e/app.spec.ts
import { test, expect } from "./fixtures";

const CONSOLE_SETTLE_MS = 1000;

test("home page has no console errors", async ({ page, consoleErrors }) => {
  await page.goto("/");
  await expect(page.locator("h1")).toBeVisible();

  // wait-ok: this test asserts the ABSENCE of console errors, so it must keep
  // observing past first paint — late console/pageerror events (a failed lazy
  // chunk, a post-hydration warning) fire after the heading is visible. There is
  // no positive event to poll for, so hold a bounded settle window before the
  // fixture teardown asserts. This is the documented absence-window exception.
  await page.waitForTimeout(CONSOLE_SETTLE_MS);
  // consoleErrors assertion happens automatically in fixture teardown
});

Filtering benign errors with a curated allowlist

The expect(errors).toEqual([]) assertion above works on a pristine app — but real suites quickly hit a wall. There are almost always benign errors: framework dev warnings, third-party SDK noise, adapters that fail gracefully outside their real runtime. A strict empty-array assertion turns every one of those into a red test, and the usual reaction — loosening the check until it stops complaining — throws away the regression-catching value entirely.

The fix is an assertNoConsoleErrors() that filters a curated allowlist. The discipline that keeps it honest: every allowlist entry carries a why-comment justifying why that specific message is safe to ignore.

// e2e/helpers.ts
import { expect } from "@playwright/test";

export function assertNoConsoleErrors(errors: string[]) {
  const unexpected = errors.filter((msg) => {
    // React DevTools install nag — dev-only, not an app error.
    if (msg.includes("Download the React DevTools")) return false;
    // Favicon 404 — the mock server has no favicon; harmless.
    if (msg.includes("Failed to load resource") && msg.includes("favicon")) return false;
    // Tauri listen() fails in browser/mock mode: @tauri-apps/api's transformCallback
    // is undefined outside the WebView runtime. The error is caught internally and
    // the mock adapter registers its own in-memory listeners instead.
    if (msg.includes("Failed to register Tauri event listener")) return false;
    // React warns on an iframe rendered with src="" — known v1 limitation of the
    // preview pane when no URL is seeded; the iframe renders harmlessly.
    if (msg.includes('An empty string ("") was passed to the %s attribute') && msg.includes("src")) {
      return false;
    }
    return true;
  });
  expect(
    unexpected,
    `Unexpected console errors:\n${unexpected.join("\n")}`,
  ).toHaveLength(0);
}

Cross-Context Terminal Evidence: Race the Browser Event, Keep Page Diagnostics in Node

The monitor above buffers page.on events in Node, and that is the part worth generalising. When the outcome under test is a browser-level terminal event — a download, a popup, a file chooser — Playwright observes it in Node, while the diagnostics that explain it live in the page, and the page's execution context is allowed to disappear at exactly that boundary. The durable shape is a page-side bridge whose records are copied into Node the moment they arrive, plus the browser's own terminal event registered before the trigger and raced against everything that means red.

Worked case: the export flow in pgen — a dialog that generates a file and hands it to the browser — produced two false negatives in a row out of that one seam.

Two false negatives from the same seam

The first installed the listener before the trigger, correctly, and lost the download anyway:

// Listener exists, but page-resident polling can still defeat it.
const download = page.waitForEvent("download");
await start.click();
await expect.poll(() => readDialogStatus(page)).toBe("done");
await download;

The download promise is registered before the click, exactly as Step 1 of the deflaking recipe requires. The bug is the line between the trigger and the consumption: readDialogStatus reads the page, and at the terminal boundary the dialog's execution context went away. The poll threw Execution context was destroyed, and the test went red while holding an already-observed download. The product worked; the instrument's own DOM read discarded the proof.

The correction kept the outcome in Node and moved the status onto a page-side done binding delivered through page.exposeFunction — then required that binding to arrive within one scheduler turn after the real download. Binding delivery crosses the same boundary that is being torn down, so it can lag by exactly enough to lose the run. The verdict now belonged to redundant observer telemetry rather than to the product: a different arrangement of the same mistake, a diagnostic outranking the outcome it exists to describe.

One seam, both times — an outcome observed in Node, evidence resident in a page that is permitted to die.

Install the bridge before the first navigation

Three calls, in this order, before the first navigation and before any trigger:

1. page.exposeFunction("recordDiagnostic", ...) — the Node-side sink, reachable from the page as window.recordDiagnostic. Functions installed this way survive navigations.

2. page.addInitScript(installDiagnosticListener) — the page-side listener that calls the binding. Init scripts are evaluated after the document is created but before any of the page's own scripts run, and they re-run on every navigation and on every child frame that attaches or navigates.

3. Only then page.goto(...) and the trigger.

exposeFunction comes first because the init script calls the binding at document start; installing the binding afterwards leaves the first document's records with nowhere to go. The re-run property cuts both ways: the bridge repairs itself after a navigation that would have wiped a listener installed through page.evaluate, and the page side must therefore be idempotent, with every record carrying the run id it belongs to. The one existing addInitScript use on this site — the A/B flag seeded before page.goto() in the connection-starvation entry — leans on the same before-the-document-runs guarantee.

The Node side is a bounded buffer keyed by run id:

• Bounded — a cap on records per run and a cap on retained runs, oldest dropped. A chatty page must not be able to grow the test process.

• Keyed — records from an abandoned earlier run can never satisfy a later run's contract.

• Copied on arrival — the callback validates and stores a plain snapshot at delivery time, never a page handle. What has already arrived cannot be erased by context loss.

// e2e/cross-context-evidence.ts -- observer-only; nothing here is on the product path.
// parseDiagnostic and validateRun are elided; "Validate, then apply the
// evidence-source hierarchy" below is their entire specification.
import type { Page } from "@playwright/test";

const MAX_RECORDS_PER_RUN = 64;
const MAX_RUNS = 4;

export type Diagnostic = {
  v: 1;
  runId: string;
  seq: number;
  stage: "start" | "progress" | "done" | "error" | "cancel";
  reason?: "quota" | "network" | "aborted";
};

export async function installEvidenceBridge(page: Page, runId: string) {
  const runs = new Map<string, Diagnostic[]>();
  const stats = { malformed: 0 };
  let signalTerminal!: (record: Diagnostic) => void;
  const terminal = new Promise<Diagnostic>((resolve) => {
    signalTerminal = resolve;
  });

  // exposeFunction BEFORE addInitScript: the init script calls this binding at
  // document start. It must never throw -- a throw rejects the page-side call and
  // resurfaces as a page error. Malformed input is counted, and counted is red.
  await page.exposeFunction("recordDiagnostic", (raw: unknown) => {
    const record = parseDiagnostic(raw); // version + closed enums, per record
    if (!record) {
      stats.malformed += 1;
      return;
    }
    // Copy on arrival: a snapshot held in Node cannot be erased by context loss.
    const list = runs.get(record.runId) ?? [];
    list.push(record);
    runs.set(record.runId, list.slice(-MAX_RECORDS_PER_RUN)); // bounded per run
    for (const stale of [...runs.keys()].slice(0, -MAX_RUNS)) runs.delete(stale);
    if (record.runId === runId && (record.stage === "error" || record.stage === "cancel")) {
      signalTerminal(record); // resolves the race's active-run error/cancel branch
    }
  });

  // Re-runs on every navigation and every attached frame, so it must be idempotent.
  // The observer stamps the run id: the product never learns it, so a record can
  // only belong to the run whose bridge delivered it.
  await page.addInitScript(({ runId }) => {
    if ((window as any).__evidenceBridgeInstalled) return;
    (window as any).__evidenceBridgeInstalled = true;
    addEventListener("app:export-diagnostic", (event) => {
      // Fire-and-forget: the product path never awaits the bridge.
      void (window as any).recordDiagnostic({ ...(event as CustomEvent).detail, runId });
    });
  }, { runId });

  return { runs, stats, terminal, attached: false };
}

export type Evidence = Awaited<ReturnType<typeof installEvidenceBridge>>;

Register the terminal event before the trigger, then race

page.waitForEvent("download") goes before the click — Playwright's own downloads guide starts the wait before clicking, and Step 1 already derives why. What this pattern adds is what happens next: not a page read, a race of four branches.

1. The terminal event itself — the download Playwright delivers to Node.

2. Active-run error or cancel evidence — the promise the buffer resolves when a record for this run id reports a failing stage.

3. Page crash or close.

4. One absolute deadline.

Branch 3 is not there to rescue branch 1: page.waitForEvent already rejects when the page crashes or closes. It is there to name the outcome, because a page that died is a different verdict from a product that failed, and the evidence has to say which.

Cleanup is deterministic and runs whichever branch wins — clearTimeout for the deadline, page.off for the crash and close handlers. A timer that outlives the race keeps the worker's event loop alive and fires into a torn-down fixture; handlers that outlive it accumulate across a page reused for several steps. And no branch of the race is ever wrapped in .catch(() => null): that is Step 3 turning a real timeout into a silent green. The absolute deadline is the test's own budget minus margin, not a fresh one — every step inside a test() draws down the same single timeout, so a deadline at or above it can never fire.

// e2e/export.spec.ts
import { test, expect } from "@playwright/test";
import { attachEvidenceOnce, installEvidenceBridge, validateRun } from "./cross-context-evidence";

const EXPORT_DEADLINE_MS = 20_000; // sits under this test's own timeout, with margin

test("export delivers a file", async ({ page }, testInfo) => {
  const runId = crypto.randomUUID();
  const evidence = await installEvidenceBridge(page, runId); // before the first navigation
  await page.goto("/export");

  let timer: ReturnType<typeof setTimeout> | undefined;
  let onDead!: () => void;
  const dead = new Promise<never>((_resolve, reject) => {
    onDead = () => reject(new Error("page crashed or closed during export"));
  });
  page.on("crash", onDead);
  page.on("close", onDead);

  try {
    const download = page.waitForEvent("download"); // registered BEFORE the trigger
    await page.getByRole("button", { name: "Export" }).click();

    // Nothing reads the page between the trigger and the race: the execution
    // context is allowed to disappear right here. No branch is ever swallowed.
    const outcome = await Promise.race([
      download.then((value) => ({ kind: "download" as const, value })),
      evidence.terminal.then((record): never => {
        throw new Error(`export reported ${record.stage}: ${record.reason ?? "unspecified"}`);
      }),
      dead,
      new Promise<never>((_resolve, reject) => {
        timer = setTimeout(() => reject(new Error("export deadline exceeded")), EXPORT_DEADLINE_MS);
      }),
    ]);

    validateRun(evidence, runId); // required minimum evidence; malformed is red
    expect(outcome.value.suggestedFilename()).toMatch(/\.zip$/);
    // The page-side "done" record is redundant here: the browser already delivered.
  } finally {
    clearTimeout(timer); // deterministic on every branch, winner or not
    page.off("crash", onDead);
    page.off("close", onDead);
    await attachEvidenceOnce(testInfo, evidence, runId);
  }
});

Validate, then apply the evidence-source hierarchy

Validation is two halves of one contract, and both are named in the example above. parseDiagnostic checks a single record as it arrives: the version field, and stage and reason drawn from closed sets. validateRun checks the run as a whole before the verdict: a start record exists for the active run id, seq is strictly increasing with nothing recorded after a terminal stage, and stats.malformed is zero. Neither re-specifies the discipline — evidence is a versioned contract and reason codes are a closed, non-leaking set already do. Malformed active-run evidence is red: an unrecognised stage string is exactly what a closed enum exists to catch, and reading it as "probably fine" restores the ambiguity the contract removed. Those two functions are the only bodies the example above elides, and the rules just listed are their entire content.

That leaves the question the two false negatives got wrong — which source decides:

Freeze and attach once, in finally

The evidence is frozen before it is attached. An init script that re-runs after a late navigation can still push records into the buffer while the attachment is being written, and an attachment that does not match the run it claims to describe is worse than none. Snapshot the active run's records, count the malformed ones, serialize, then attach — exactly once, guarded by a flag on the evidence object so that two exit paths cannot write two attachments under the same name.

// e2e/cross-context-evidence.ts (continued)
import type { TestInfo } from "@playwright/test";

export async function attachEvidenceOnce(testInfo: TestInfo, evidence: Evidence, runId: string) {
  if (evidence.attached) return; // two exit paths must not write two attachments
  evidence.attached = true;
  // Freeze first: a late init-script re-run must not mutate what this describes.
  const frozen = JSON.stringify({
    runId,
    malformed: evidence.stats.malformed,
    records: [...(evidence.runs.get(runId) ?? [])],
  });
  try {
    await testInfo.attach("export-evidence.json", { body: frozen, contentType: "application/json" });
  } catch (attachError) {
    // Never rethrow from finally: that would replace the failure under investigation.
    testInfo.annotations.push({ type: "evidence-attach-failed", description: String(attachError) });
  }
}

The try/catch inside the helper is the load-bearing part. A finally block that throws replaces the error the test was about to report with a storage complaint, and the original failure is gone. Attachment is its own concern with its own verdict, recorded as an annotation and never allowed to mask the run's — the same separation as the three layers of teardown, where cleanup reports a verdict of its own instead of being folded into the product's. And an attachment is still only written to the runner's disk: it needs the upload gate in Recording a Trace Is Not Keeping It to become evidence anyone can read after the job ends.

CI Image Interception for Speed

In CI, network requests for large images slow down tests. Intercept and replace them with tiny placeholders:

// e2e/fixtures.ts
export const test = base.extend({
  page: async ({ page }, use) => {
    // Intercept image requests in CI
    if (process.env.CI) {
      await page.route("**/*.{png,jpg,jpeg,webp,gif}", (route) => {
        route.fulfill({
          status: 200,
          contentType: "image/png",
          // 1x1 transparent PNG
          body: Buffer.from(
            "iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mNk+M9QDwADhgGAWjR9awAAAABJRU5ErkJggg==",
            "base64"
          ),
        });
      });
    }
    await use(page);
  },
});

Production Build Verification

Test against the production build, not the dev server. This catches build-specific issues:

// playwright.config.ts
import { defineConfig } from "@playwright/test";

export default defineConfig({
  webServer: {
    command: "npm run build && npm run preview",
    port: 4173,
    reuseExistingServer: !process.env.CI,
  },
  use: {
    baseURL: "http://localhost:4173",
  },
});

// e2e/production.spec.ts
import { test, expect } from "@playwright/test";

test("production build serves all pages", async ({ page }) => {
  const urls = ["/", "/docs", "/about", "/contact"];
  for (const url of urls) {
    const response = await page.goto(url);
    expect(response?.status()).toBe(200);
  }
});

test("production build has no broken links", async ({ page }) => {
  await page.goto("/");
  // Collect hrefs as plain strings before navigating -- .all() locators
  // re-resolve against whatever page is currently loaded, so navigating away
  // mid-loop would have iteration 2+ walk hrefs from a random later page.
  const hrefs = await page.$$eval("a[href^='/']", (els) =>
    els.map((a) => a.getAttribute("href")),
  );
  for (const href of hrefs) {
    if (href) {
      const response = await page.goto(href);
      expect(response?.status()).toBe(200);
    }
  }
});

The Bundled-Dep E2E Lane for Monorepo Dev Servers

The production-build lane above verifies build correctness — but many app suites cannot run their main e2e volume against the production build: MSW's service-worker setup, dev-only routes, and dev-only test bridges (state exposed on window behind an import.meta.env.DEV guard) all exist only under the dev server. The lanes are complementary, not competing: source-serving dev server for interactive work and HMR, this bundled-dep lane for the main CI e2e volume, production build for build-specific verification. What forces the middle lane into existence is a hidden per-test tax in pnpm-workspace monorepos.

The tax: a workspace package resolved through a development export condition —

"exports": {
  ".": {
    "types": "./dist/index.d.ts",
    "development": "./src/index.ts",
    "default": "./dist/index.js"
  }
}

— is served as raw source, one HTTP request per module. That is exactly what you want for interactive dev (HMR into the package's source). But every Playwright test gets a fresh browser context with an empty HTTP cache, so every test re-downloads the package's whole module graph. For a heavy package (in the worked case: a pattern-generator registry of ~3,500 modules) that was +10–28s on every test whose flow touched it — enough to push two 15-minute CI shards into timeout-minutes kills.

The middle path — bundle only the heavy, stable dep; keep everything else source-served:

// vite.config.ts
const BUNDLED_GENERATORS = process.env.E2E_BUNDLED_GENERATORS === "1";
const GENERATORS_DIST = fileURLToPath(
  new URL("../generators/dist/index.js", import.meta.url),
);
if (BUNDLED_GENERATORS && !existsSync(GENERATORS_DIST)) {
  throw new Error(
    "E2E_BUNDLED_GENERATORS=1 but the bundle is missing — build the package first.",
  );
}

export default defineConfig({
  resolve: {
    // However your setup enables the `development` condition, preserve
    // Vite's defaults — a bare `conditions: ["development"]` REPLACES
    // them (dropping `module`/`browser` and the mode-dependent
    // production condition) and can change dependency resolution in the
    // production build. On Vite 6: spread `defaultClientConditions`.
    conditions: [...defaultClientConditions, "development"],
    ...(BUNDLED_GENERATORS
      ? {
          alias: [
            // Exact-match so hypothetical subpath imports still resolve normally
            { find: /^@acme\/generators$/, replacement: GENERATORS_DIST },
          ],
        }
      : {}),
  },
});

# CI e2e job: build the heavy package (~15s), then run with the flag
- name: Build the bundled-dep e2e lane
  run: pnpm --filter @acme/core build && pnpm --filter @acme/generators build

- name: Run Playwright e2e tests
  env:
    E2E_BUNDLED_GENERATORS: '1'
  run: pnpm exec playwright test

Three properties are load-bearing:

• The existsSync guard throws on a missing bundle. Note what it cannot do: detect a stale one — freshness comes from the CI step building the package immediately before Playwright, not from the guard. Without the guard, a missing dist still fails (the alias points imports at a nonexistent file), but as a confusing mid-run module-load error; the guard converts it into an immediate, named boot-time failure.

• The flag is opt-in. Local pnpm test:e2e and interactive dev keep source-serving and HMR; only CI (and anyone reproducing CI) pays the ~15s build for the collapsed graph.

• Only the heavy, stable package is bundled. The app's own source — the code under test — is still served live. You are bundling a dependency, not the subject. The alias is exact-match by design: subpath imports (@acme/generators/foo) fall through to normal source resolution, so if the package exposes subpath entry points on hot paths, alias each one too.

Worked-case evidence: the two e2e shards went from repeated 15-minute timeout-minutes kills with 10–23 failed attempts per run to 6m35s and 8m14s with zero failed attempts — roughly 2× faster than the suite's best-ever green.

The Hidden Auxiliary-Socket Port Race

The port-race guidance above, and in Execution Tiers, covers the application port: give each fixture its own port and the EADDRINUSE collisions across N parallel webServer entries go away. There is a subtler variant that distinct app ports do not fix.

Symptom: N parallel webServer entries, each with a correctly distinct app port, still crash intermittently. The failure names a port that appears nowhere in playwright.config.ts:

Error: listen EADDRINUSE: address already in use :::9229

Diagnosis: the dev-server process each webServer entry boots doesn't only open the app port — it also opens an auxiliary socket: a devtools inspector, an HMR websocket, or a metrics endpoint, each with its own default port that is independent of the app port you configured. Node's inspector, for example, defaults to 9229 no matter which app port the process serves on. When N fixtures boot in parallel, every instance races to bind that same shared default, and whichever loses dies with EADDRINUSE — on a port nobody put in the config. The diagnostic tell: the crashing port is unfamiliar — it matches no port or baseURL value anywhere in the test setup.

Fix: configure the auxiliary socket to an OS-assigned port (0), the same way you would the app port itself:

# Anti-pattern: --inspect binds the fixed default (9229) regardless of app port
node --inspect ./server.js

# Fix: --inspect=0 binds an OS-assigned port -- no shared default left to race on
node --inspect=0 ./server.js

The same move applies to any dev-tool's auxiliary channel — an HMR websocket port, a metrics/telemetry port, a bundler's own devtools bridge: whatever config knob it exposes, point it at 0 instead of leaving the tool's built-in default in place.

See also the native-suite "Port races (EADDRINUSE)" entry in the Flake Root-Cause Catalog — same root cause, a fixed port shared across parallel processes, here specific to a dev-server's non-obvious secondary socket rather than its primary one.

fullyParallel Defaults to False — the Mega-File Tail

The workers value decides how many parallel lanes a run has. fullyParallel decides what is allowed into them, and its default is narrower than most people assume: it is false, which parallelizes across files while serializing every test within a file onto a single worker. The unit of concurrency is the file, not the test.

That makes the suite's wall clock a maximum, not a division:

critical path = max(slowest single file, total test time ÷ workers)

Only the second term responds to buying cores. The first does not move at all — so once total ÷ workers drops below the slowest file, every additional worker buys nothing, and the suite's floor is one file's runtime.

A real suite with 92 minutes of aggregate test time — the 46-minute two-worker step that Aggregate Suite Budget prices at 5,520 worker-seconds, converted back from wall clock to work — whose top two files ran 10:08 and 9:41:

 2 workers:  max(10:08, 92:00 ÷  2 = 46:00)  =  46:00   ← the division dominates
 4 workers:  max(10:08, 92:00 ÷  4 = 23:00)  =  23:00   ← still the division
 8 workers:  max(10:08, 92:00 ÷  8 = 11:30)  =  11:30   ← nearly a tie
16 workers:  max(10:08, 92:00 ÷ 16 =  5:45)  =  10:08   ← the file is the floor

Somewhere between eight and sixteen workers this suite stops being worker-bound and becomes file-bound. Paying for the bigger runner past that point buys the doubled per-minute rate and almost no wall clock — the failure CI Runner Sizing Rule 1 warns about, arriving here through a config default rather than an explicit workers cap.

The lever is fullyParallel, suite-wide or per file:

// playwright.config.ts -- every test becomes its own unit of concurrency
export default defineConfig({
  fullyParallel: true,
});

// e2e/checkout-flow.spec.ts -- opt one genuinely coupled file back OUT
import { test } from "@playwright/test";

// These tests build on a cart seeded by the first one and must run in order.
test.describe.configure({ mode: "serial" });

The inverse works too, while the config default stays false: test.describe.configure({ mode: "parallel" }) inside a single mega-file splits just that file across workers. That is the smaller, more targeted move when exactly one or two files are the tail.

Flip It as an Experiment, Not as a Recommendation

fullyParallel: true is not a settings improvement to apply on sight. It removes an ordering guarantee the suite has been getting for free — tests inside a file were running one at a time in source order, whether or not anyone intended that — and any test quietly depending on it starts failing intermittently rather than loudly.

So run it as a matched experiment, and hold the result to two accept criteria, both of which must pass:

1. Wall clock. The after-side maximum critical-path job time must sit under the ceiling you gate on. Compare maxima, not medians: the slowest attempt is what a PR author actually waits for.

2. Retries. Retry counts and retry markers must not rise. A flip that buys wall clock by spending retry budget has bought nothing — it has converted a measurable cost into an unmeasurable one.

Keep the experiment honest with the usual controls: change exactly one line, hold the lockfile, runner shape, shard matrix and selected test count fixed, take no serial opt-outs on the first pass, and run at least five attempts per side. One attempt per side measures noise.

The Counter-Case: Under the Ceiling and Still Rejected

A 178-test consolidated suite (Playwright 1.59.1, two CI workers on a four-vCPU runner, fixed three-shard matrix) ran exactly that experiment — one line changed, five attempts before and five after, same selected count, no serial opt-outs:

Wall-clock criterion: passed. The after-side maximum, 484s (8m04s), stayed under the 9m24s ceiling. Note what "passed" does and does not mean here — the median moved 365s → 364s and the maximum moved 441s → 484s, so the flip bought no measurable speed at all. It merely failed to break the budget.

Retry criterion: failed. Explicit retry markers rose from 2 total to 4 total, and median retries per attempt went 0 → 1.

The natural next move — exempt the offending files with mode: "serial" and re-run — had no target. The markers were generic command-palette-nested-pick.spec.ts noise that recurred on both sides, plus one composer-overlay-crisp-862.spec.ts occurrence. No repeated ordering or shared-state signature pointed at a specific file, so there was no principled set of files to exempt.

fullyParallel was restored to false. That is the mechanical rule working as designed: two criteria, one passed, one failed, so the change does not land. The temptation at that moment is to accept anyway — it is under the ceiling, and four retry markers across five attempts look like noise — which is precisely how a suite acquires a retry budget nobody chose. A retry that CI absorbs is still a failure the zero-retry local gate will surface; see Quarantining Flakes: The Retries-Asymmetry Trap.

Price the Per-Test Fixture Boot

A slow suite gets investigated as "which test is slow?" — a report sorted by per-test duration, worst offenders first. That question structurally cannot find the largest line item in many suites, because the largest line item is uniform. Setup that every test pays does not show up as an outlier; it raises the baseline everywhere by the same amount, which reads as "the app is just slow."

Price it directly instead:

fixture cost = per-invocation cost × invocation count

Two shapes of this are worth pricing separately.

An isolation helper in beforeEach — goto("/") → clear localStorage/sessionStorage → reload() — costs two full application boots per test that the test never asked for: roughly 1,132 avoidable boots across a single run. It stays invisible in a per-test profile precisely because every test pays it equally.

A setup helper gets priced by the same arithmetic, with numbers you can actually measure. One representative test ran 32.8s of Playwright wall time (34.33s outer); a DEBUG rerun attributed about 10.1s of that to the helper's boot portion — roughly 7.0s navigation plus 2.0s waiting for a step rail to become visible. A static audit found 77 call sites across 35 spec files:

10.1s per call  ×  77 call sites  =  777.7s  =  13.0 serial minutes

Check Whether the Clear-Helper Is Cargo Cult

Playwright's default page fixture already hands every test a fresh BrowserContext: its own cookie jar, its own localStorage and sessionStorage, both empty. Storage does not bleed between tests unless you deliberately arranged for it to — a worker-scoped context reused across tests, a context you created by hand, a storageState you loaded on purpose. A beforeEach that clears storage on the default fixture is clearing something that was already empty.

Such helpers arrive honestly. They get written against a same-context assumption carried in from another harness, copied wholesale from a project that genuinely did reuse contexts, or added while chasing a flake and kept afterwards because the flake stopped. None of those origins is legible in the code, and by the time anyone asks, the helper has 77 call sites.

The cheap experiment answers the question before you build anything:

# Delete the clear step, then run only the specs most likely to have depended on it.
npx playwright test e2e/seed-heavy-*.spec.ts --repeat-each=3

Target the seed-heavy specs — the ones that write storage — not the whole suite; you want a fast answer, not a nightly. Green across repeats means the clear was cargo cult, and the answer cost one command. Building an addInitScript fixture to do the clearing "properly" costs a day and answers a question you never confirmed you had.

The Trap: The Helper's goto Is Often the Spec's Only Navigation

"Just remove the helper" is where this goes wrong, because these helpers navigate:

// e2e/helpers/isolation.ts
import { Page } from "@playwright/test";

export async function resetApp(page: Page) {
  await page.goto("/"); // boot #1 -- and the spec's ONLY navigation
  await page.evaluate(() => localStorage.clear());
  await page.reload(); // boot #2
}

// e2e/settings.spec.ts -- note the absence of any goto of its own
import { test, expect } from "@playwright/test";
import { resetApp } from "./helpers/isolation";

test.beforeEach(async ({ page }) => {
  await resetApp(page);
});

test("saves the theme", async ({ page }) => {
  await page.getByRole("button", { name: "Dark" }).click();
  await expect(page.getByTestId("theme")).toHaveText("dark");
});

Delete resetApp and this spec never navigates anywhere. It fails on about:blank with a locator timeout that looks nothing like "you removed a storage clear," and the same failure lands in every spec that inherited the habit. The navigation side effect is the de facto contract, whatever the helper is named for.

So the removal is a decomposition, not a deletion: keep the navigation, drop only the step you priced.

One Canonical Setup Abstraction per Domain

The section above prices one setup helper. This one is about the suite that has two of them for the same domain, both defensible, neither obviously wrong.

Both halves arrive honestly. Function helpers come first because they are the cheapest thing to write, and consumers accumulate against them — 101 of them in the audited suite. A PageObject class arrives later because a class reads better for multi-step flows, and it collects 11. Crucially, the class does not reimplement boot; it delegates to the function helper. That delegation is what makes the arrangement look safe: one boot path, two spellings of it.

The delegation is exactly why it is not safe. The two surfaces are equivalent only across the part the class forwards, and drift enters through the part it does not.

// e2e/helpers/composer.ts -- surface A, 101 consumers
import { Page, expect } from "@playwright/test";

export async function openComposerWithTemplate(page: Page, template: string) {
  await page.goto(`/composer?template=${template}`);
  await expect(page.getByTestId("step-rail")).toBeVisible();
}

// e2e/fixtures/composer-page-object.ts -- surface B, 11 consumers, boot delegated to A
import { Page } from "@playwright/test";
import { openComposerWithTemplate } from "../helpers/composer";

export class ComposerPageObject {
  constructor(private readonly page: Page) {}

  async open(template: string) {
    await openComposerWithTemplate(this.page, template); // delegation -- the shared half
  }

  async selectVariant(name: string) {
    await this.page.getByRole("tab", { name }).click(); // behavior -- the unshared half
  }
}

Add a readiness step to surface A — wait for the canvas to paint, not just for the step rail — and every class consumer inherits it too. Add the same step to ComposerPageObject.open instead, because that is the file you happened to have open, and the 101 function consumers do not. Either way the review saw one correct-looking diff. Specs that import both styles are worse still: their boot contract depends on which call runs first, which is a property of the spec's line order rather than of anything anyone decided.

The cost of leaving both surfaces standing is a review cost, and it scales with the suite. The same audit that priced the boot helper counted 77 boot-helper calls across 35 spec files and 460 waitForLayoutFlush calls. At that size, "does this spec boot the way the other 76 do?" is already hard to answer by reading. Two setup surfaces double the question without doubling the coverage.

Folding the Smaller Surface into the Larger

1. Inventory the consumers of each surface. Count spec files, not call sites — the file is the unit you will have to run and review. Two grep -rl passes produce the numbers that decide everything after this, and they cost a minute.

2. Let consumer count choose the direction, function or class. 101 versus 11 is not a design debate; it is arithmetic about how much migration you are signing up for. Pick the surface most consumers already use, even when the other one is the nicer abstraction — taste is worth 11 migrations, not 101.

3. Fold the smaller surface's useful methods into the winner. The class was not pure overhead. Its delegating methods are noise, but the ones carrying real behavior — selectVariant, and whatever else the function module never grew — are the reason it existed. Move those across as functions and drop only the delegation.

4. Search for the dual importers before you touch anything. Specs using both styles are where the migration actually breaks, and they are invisible in either surface's own consumer list because each list counts them as a normal consumer.

5. Batch-run every migrated consumer green before deleting the old module. While the old module is still in the tree, a missed migration fails loudly on the new path. Once it is deleted, the same mistake arrives as a resolution error in an unrelated file — or worse, as a spec that quietly stops matching its project and is never run again.

# Steps 1 and 4: consumer counts per surface, then the specs importing BOTH.
grep -rl "helpers/composer" e2e --include="*.spec.ts" | wc -l
grep -rl "ComposerPageObject" e2e --include="*.spec.ts" | wc -l
grep -rl "ComposerPageObject" e2e --include="*.spec.ts" \
  | xargs grep -l "helpers/composer"

# Step 5: every migrated spec, in one batch, while the old module still exists.
npx playwright test $(git diff --name-only main...HEAD -- 'e2e/**/*.spec.ts')

One Readiness Ladder, Stated Once

What the fold protects is the boot/navigation/readiness contract, and that contract is worth stating as a single ordered ladder living in the canonical module:

// e2e/helpers/composer.ts -- one ladder, one file, after the fold
export async function openComposerWithTemplate(page: Page, template: string) {
  await page.goto(`/composer?template=${template}`); // 1. navigate
  await expect(page.getByTestId("app-shell")).toBeVisible(); // 2. shell mounted
  await expect(page.getByTestId("step-rail")).toBeVisible(); // 3. domain ready
}

Stated once, a readiness change is one diff that every consumer inherits, and the review question becomes "is this ladder right?" rather than "which of the two ladders did this spec get?" Keeping navigation on the ladder as an explicit rung matters for a second reason: it is the step Price the Per-Test Fixture Boot shows people deleting by accident, and a rung nobody wrote down is a rung nobody defends in review.

Sharded CI Runs

For large test suites, shard across multiple CI runners:

# .github/workflows/e2e.yml
jobs:
  e2e:
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        shard: [1/4, 2/4, 3/4, 4/4]
    steps:
      - uses: actions/checkout@v4
      - uses: pnpm/action-setup@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 20
      - run: pnpm install
      - run: npx playwright install --with-deps
      - run: npx playwright test --shard=${{ matrix.shard }} --reporter=blob
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: blob-report-${{ strategy.job-index }}
          path: blob-report/
          if-no-files-found: error   # default is warn: a shard that wrote no blob report must go red
          retention-days: 1

  merge-reports:
    if: always()
    needs: e2e
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: pnpm/action-setup@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 20
      - run: pnpm install
      - uses: actions/download-artifact@v4
        with:
          path: all-blob-reports
          pattern: blob-report-*
          merge-multiple: true
      - run: npx playwright merge-reports --reporter=html ./all-blob-reports
      - uses: actions/upload-artifact@v4
        with:
          name: playwright-report
          path: playwright-report/
          if-no-files-found: error   # default is warn: an empty merged report must go red, not green
          retention-days: 14

Composite Actions Own Setup, Not Test Selection

Near-identical e2e jobs — a light shard matrix and a separate @heavy lane — invite folding the whole job body into one composite action. Fold the setup only. The pnpm and Node setup, pnpm install, the browser install, and the build are identical across both callers and belong in a shared .github/actions/setup-e2e/action.yml. Test selection does not travel with them: a composite action can own setup, but it cannot safely supply the arguments of a command its caller runs afterwards. Routing the selection string through the composite gives one file two owners, and either lane changing what it runs then means editing a file the other lane also depends on.

Keep each lane's selection string in exactly one source that lane owns. In the worked case the heavy lane obtains NUL-delimited arguments from its own scripts/ci/e2e-heavy-args.sh and invokes Playwright itself, while the light matrix keeps its fixed three-shard command in the workflow beside it. Both callers use the same setup composite; neither can silently change what the other selects.

Artifacts You Want to Keep Must Live Outside outputDir

Playwright empties outputDir at the start of every run. The default is test-results/, and the wipe is unconditional: it is not scoped to files this run will overwrite, and it does not care what wrote them. Anything sitting in that directory when a run begins is gone.

This is correct behaviour — a trace directory full of last run's failures would be worse than useless — and it is invisible until you try to keep something.

# Run 1 -- writes its report into outputDir
PLAYWRIGHT_JSON_OUTPUT_NAME=test-results/report-1.json npx playwright test

# Run 2 -- a DIFFERENT filename, so surely both survive?
PLAYWRIGHT_JSON_OUTPUT_NAME=test-results/report-2.json npx playwright test

ls test-results/
# .last-run.json  report-2.json
#
# report-1.json is gone. So is anything else that was in there.

The fix is placement, not naming. Write anything you intend to keep or compare to a sibling directory outside outputDir:

# Reports accumulate -- run N leaves run N-1's report untouched
PLAYWRIGHT_JSON_OUTPUT_NAME=b4push-reports/report-1.json npx playwright test
PLAYWRIGHT_JSON_OUTPUT_NAME=b4push-reports/report-2.json npx playwright test

ls b4push-reports/
# report-1.json  report-2.json

PLAYWRIGHT_JSON_OUTPUT_NAME resolves relative to the config directory, and the JSON reporter creates missing parent directories itself — no mkdir -p step is required. Add the sibling directory to .gitignore.

Why this bites the cross-run workflows this site recommends

The hazard is local-first. In CI it is nearly unobservable — each run gets a fresh runner, so the start-of-run wipe finds an empty directory and the artifact-upload step ships the report regardless. It surfaces the moment two runs share a working tree, which is precisely the shape of the local triage this site prescribes: the per-test duration diff between two runs and any tooling that decides "did this spec fail in both runs?" (see lane over-subscription). A tool built on "run twice, compare the two JSON reports" is structurally impossible when both reports are written into test-results/ — and it fails silently: the second run finds one report, concludes there is nothing to compare, and reports success.

(Verified against Playwright 1.58.2 by running it: sentinel files placed in test-results/ were deleted by the next run; two reports written to a sibling directory coexisted.)

Recording a Trace Is Not Keeping It

trace: 'on-first-retry' and screenshot: 'only-on-failure' write evidence onto the runner's disk, and the runner is deleted when the job ends. Unless a step uploads that evidence first, the config spends time recording a trace for the sole purpose of discarding it — and nothing about the run looks wrong. The job goes red on the real failure exactly as intended; only the artifact list is empty, and nobody reads an empty artifact list until they need it.

Measured: a PR gate carried trace: 'retain-on-failure' and screenshot: 'only-on-failure' for months with no actions/upload-artifact step anywhere in the workflow. A later flake investigation paid a 92-execution reconstruction cost re-running the suite to reproduce a failure CI had already captured and thrown away.

Retention is part of the T1 definition rather than an operations detail bolted on afterwards — see T0 + T1: the defaults. Below are the three ways a correct-looking upload still yields nothing, each of which fails silently, and then the storage ceiling that decides what to upload.

The CI default reporter writes no playwright-report/

Playwright's default reporter is list locally and dot under CI. Neither writes an HTML report, so playwright-report/ does not exist unless a reporter is configured to create it. An upload step pointing at that path uploads nothing: actions/upload-artifact logs a "No files were found" warning and succeeds, so the workflow stays green and the evidence gap stays invisible.

The directory that does hold failure evidence by default is outputDir — test-results/ — and it resolves relative to the config file's directory, not the working directory of the step. A monorepo job running npx playwright test -c apps/web/playwright.config.ts from the repo root writes to apps/web/test-results/; an upload step pointing at a root-level test-results/ finds nothing and still succeeds.

So pick one deliberately: configure reporter: [['html', { open: 'never' }]] and upload playwright-report/, or upload test-results/ at its config-relative path. What never works is uploading a default-named directory no reporter was asked to write.

Gate the upload on failure() || cancelled()

- uses: actions/upload-artifact@v4
  if: failure() || cancelled()
  with:
    name: playwright-report-${{ strategy.job-index }}-${{ github.run_attempt }}
    path: playwright-report/
    if-no-files-found: error   # default is warn: an upload that finds nothing must go red, not green
    retention-days: 2

An upload step with no if: is skipped the moment an earlier step exits non-zero — which is every run the artifact exists for. if: failure() fixes that and still misses the other half: a job killed at its timeout-minutes cap, or cancelled by a concurrency group when a new push lands, is cancelled rather than failed, and cancellation is exactly the evidence-loss case. The run you most want a trace from is the run that quietly produces no artifact. if: always() covers both as well, at the cost of uploading a report on every green run. actions/upload-artifact@v4 defaults to warn, so the missing-playwright-report/ case from the previous section logs "No files were found" and exits 0 — if-no-files-found: error is what makes that step red instead.

A second Playwright run empties outputDir before the upload

Playwright empties outputDir at the start of every run — the section above. So a job that invokes Playwright twice, a CI-safe lane followed by a @flaky quarantine lane, has its second invocation delete the first invocation's trace zips and failure screenshots before the upload step ever runs. The upload can satisfy every rule above and still ship nothing but the quarantine lane's output.

Worked case: an e2e-full job ran pnpm test:e2e:ci:json, then npx playwright test --grep @flaky, then a single actions/upload-artifact covering both playwright-report/ and test-results/ under if: always(). Every retention rule was satisfied and the CI-safe lane's evidence was gone anyway. Its JSON reports survived only incidentally, because PLAYWRIGHT_JSON_OUTPUT_NAME redirected them into playwright-report/, which Playwright does not clear.

The fix is one --output per invocation, both uploaded, plus a one-line guard so a later edit cannot silently collapse the two paths back into one:

e2e-full:
  runs-on: ubuntu-latest
  env:
    CI_OUTPUT: test-results-ci
    FLAKY_OUTPUT: test-results-flaky
  steps:
    - name: Each Playwright invocation needs its own --output
      run: '[ "$CI_OUTPUT" != "$FLAKY_OUTPUT" ]'
    - run: pnpm test:e2e:ci:json --output="$CI_OUTPUT"
    - run: npx playwright test --grep @flaky --output="$FLAKY_OUTPUT"
    - uses: actions/upload-artifact@v4
      if: failure() || cancelled()
      with:
        name: e2e-evidence-${{ github.run_attempt }}
        path: |
          ${{ env.CI_OUTPUT }}/
          ${{ env.FLAKY_OUTPUT }}/
        if-no-files-found: error   # default is warn; fires only when every listed path is empty
        retention-days: 2

The guard runs first and costs nothing. It earns its line because the failure it prevents is invisible: pointing both lanes back at test-results/ produces a green workflow with a plausible-looking artifact attached.

if-no-files-found: error on that upload fires only when every listed path is empty. It catches the total miss — neither lane wrote anything — not the partial one where the quarantine lane's output uploads on its own and the run stays green.

The path: here ships the raw output directories to keep the two-lane mechanic visible — do not copy it verbatim. On the measured numbers in the storage-cap subsection below, raw directories blow the storage allowance on their first failed run; keep the per-invocation --output split, but upload the self-contained HTML report instead.

The latent form is worth naming. In the worked case @flaky matched zero specs, so nobody had ever lost evidence to it — which is the point rather than a reprieve. A quarantine lane that matches nothing today starts destroying traces on exactly the night a real nightly failure needs one. The T1 checklist line is one sentence: if the job invokes Playwright more than once, give each invocation its own --output.

The storage cap: upload the report, not the raw directory

Raw test-results/ is not free to keep. Measured on a three-shard e2e job: 100–615 MB per failed shard, up to 1,845 MB across three shards, against a 0.5 GB Actions-storage allowance. Retaining the raw directory for even two days was rejected outright on those numbers.

The middle path is to upload only the self-contained HTML report — it embeds the first-retry traces and the failure screenshots, which is what a triage session actually opens — with a short retention-days, gated on failure() || cancelled(), under unique shard and run names.

Mock Backend Adapter for Frontend-Only E2E

When testing frontend behavior independently from the real backend:

// e2e/mocks/backend-adapter.ts
import { Page } from "@playwright/test";

export async function mockBackend(page: Page) {
  await page.route("**/api/**", async (route) => {
    const url = new URL(route.request().url());

    const mocks: Record<string, unknown> = {
      "/api/user": { id: 1, name: "Test User", email: "test@example.com" },
      "/api/settings": { theme: "dark", language: "en" },
      "/api/documents": [
        { id: 1, title: "Doc 1" },
        { id: 2, title: "Doc 2" },
      ],
    };

    const mockData = mocks[url.pathname];
    if (mockData) {
      await route.fulfill({
        status: 200,
        contentType: "application/json",
        body: JSON.stringify(mockData),
      });
    } else {
      await route.continue();
    }
  });
}

// e2e/frontend.spec.ts
import { test, expect } from "@playwright/test";
import { mockBackend } from "./mocks/backend-adapter";

test.beforeEach(async ({ page }) => {
  await mockBackend(page);
});

test("displays user name from mock API", async ({ page }) => {
  await page.goto("/dashboard");
  await expect(page.locator(".user-name")).toHaveText("Test User");
});

See Also

Running these patterns inside a sandboxed container (Claude Code on the web, locked-down WSL) where the Playwright CDN is blocked? See Browser Verification in Limited Environments for the seeing-eye fallback to a pre-installed Chromium, 127.0.0.1 dev-server binding, and the PR-preview-URL verification path.