Execution Tiers

Where and when a test runs -- the second axis, orthogonal to testing levels.

The Second Axis

The rest of this guide is organized around the 6 testing levels, which answer one question: what can a test see? A unit test sees pure logic; an E2E test sees a real browser. But the levels say nothing about a different question: where and when does the test run?

That question has its own axis: the execution tier.

• Level = what the test can see (logic, DOM, build output, browser, pixels)

• Tier = where and when it runs (inner loop, PR CI, a schedule, a local heavy lane)

Every test has a position on both axes. A pixel-level L5 spec can run on every PR if the runners can render it, or on a nightly schedule if it needs a real GPU. A plain L1 unit test almost always lives in the inner loop and the PR gate.

Keeping the axes separate dissolves a common escalation mistake: when an E2E test feels "too heavy for PR CI", that is not a reason to rewrite it at a lower level. "Too heavy" is a tier question, not a level question. The level is fixed by what the assertion needs to see; the tier is chosen by cost, hardware, and timing. When a specific test genuinely is too heavy, apply the Heavy Test Decision Rule -- it first asks whether the test should exist at its level at all, and only then moves it between tiers.

Verification vs Regression

A test's tier follows from its job, and there are exactly two jobs. Conflating them is the root cause of most bloated CI pipelines.

• A verification artifact is a one-time "it was done" proof. It exists to show that a change worked when it landed. It may be manual, an L5 screenshot, even an L6 AI judgment -- determinism is not required, because it will never run again. It belongs to no gate.

• A regression gate is a repeatable, deterministic "everything still works" proof. Anyone must be able to run it, it must produce the same verdict every time, and it must live in a tier with a paper trail -- a required check, a CI run log, a scheduled job result.

Promotion of temporary verification is explicit. First establish an ongoing contract/failure and suitable level, then rerun conditions, frequency, determinism, and budget, and finally tags. This is distinct from writing regression coverage directly under established project policy.

The enforceable agent rules around graduation -- verification specs are tagged and excluded from all gates, and an agent never promotes its own verification test -- live in Required Testing Behavior.

Silent graduation happens below the tag system

The graduation rule above is written for specs that carry a @verification tag -- it fires when someone proposes removing that tag. A spec that never entered the tag system is invisible to it. No tag, no proposal, no decision: the spec simply runs, on every PR, forever.

That is where the leak actually happens, and it is upstream of tagging entirely. An agent is asked to fix a bug, writes a spec to prove the fix worked, and drops that spec into the gate's spec directory untagged -- not to evade the rule, but because at authoring time it looks like an ordinary test. In one measured case the spec was a one-day UI-review checklist, whose whole job was to confirm that a specific afternoon's tweaks looked right. It billed every PR from every author for three months, re-answering a question that had been settled on day one.

Names and tags are discovery hints

Names containing bugfix, issue numbers, dates, iteration counts, or checklists help find candidates; they do not prove temporary verification. An issue-numbered data-loss regression with no recent failures still belongs when it guards a live contract. Zero CI cost for tagged verification may simply reflect intended exclusion; it does not itself prove unclassified artifacts exist.

For candidates establish ongoing purpose, suitable level, rerun conditions/frequency, determinism, and budget. Do not exclude/delete by name, or treat a rename as classification. Test lifecycle owns admission and retirement. Review may be scheduled or triggered by budget overruns; use an AI suite audit to bound effort.

Change-scoped verification covers selection and reuse. Retention, this run's scope, scheduling, and reuse are different decisions.

The Five Tiers

T0 + T1: the defaults

T0 provides fast feedback; T1 provides authoritative integration evidence. Normally green is required before merge. An explicit transient-red development policy may separate CI execution from merge waiting while retaining green at the external-impact boundary. Non-required red still creates Rule 7 ownership. The ruleset example below describes the green-before-merge topology.

Make the gate enforced, not just declared

When declaring green-before-merge, verify an enforcement layer actually blocks merges and pushes. For asynchronous development, verify enforcement at the declared publish/deploy boundary instead. These commands inspect the merge boundary.

gh api repos/{owner}/{repo}/rulesets                      # [] means no rulesets
gh api repos/{owner}/{repo}/branches/main/protection      # 404 means no branch protection

If both come back empty, "the authoritative gate" is a gentleman's agreement. Fix it with a branch ruleset on the default branch (rulesets are the current mechanism; classic branch protection is the legacy one): require a pull request plus required status checks.

The 3-pitfall checklist when adding the ruleset:

1. Required status checks block direct pushes too. A commit pushed straight to main has no completed checks at push time, so it is rejected -- which breaks release automation that pushes version-bump commits directly to main. Configure an explicit bypass_actors entry, then verify the bypass with an actual test push (a no-op commit through the real release path).

2. Never mark a path-filtered check as required. A workflow with on.pull_request.paths: never runs on a non-matching PR, so that PR waits forever on "Expected -- waiting for status" and can never merge. Only always-running checks can be required; path-filtered checks stay non-required (Rule 7 still makes their reds owned; blocking follows the declared boundary policy).

3. Enumerate exact check-run names from a recent merged PR -- never guess. Required checks match by exact check-run name (the job name, or its name: override -- not the workflow name). A typo'd or guessed name hangs on "Expected" exactly like pitfall 2. Pull the real names instead of guessing:

gh pr view <recent-merged-PR> --json statusCheckRollup --jq '.statusCheckRollup[].name' | sort -u

Create the ruleset via gh api -X POST repos/{owner}/{repo}/rulesets --input ruleset.json:

{
  "name": "main gate",
  "target": "branch",
  "enforcement": "active",
  "conditions": { "ref_name": { "include": ["~DEFAULT_BRANCH"], "exclude": [] } },
  "rules": [
    { "type": "pull_request",
      "parameters": { "required_approving_review_count": 0,
        "dismiss_stale_reviews_on_push": false,
        "require_code_owner_review": false,
        "require_last_push_approval": false,
        "required_review_thread_resolution": false } },
    { "type": "required_status_checks",
      "parameters": {
        "required_status_checks": [ { "context": "health" } ],
        "strict_required_status_checks_policy": false } }
  ],
  "bypass_actors": [
    { "actor_id": 5, "actor_type": "RepositoryRole", "bypass_mode": "always" }
  ]
}

Exact context values come from pitfall 3; the bypass_actors entry is what pitfall 1's test push validates.

Rule 7 ("red checks block the author") and the ruleset are complementary layers, not substitutes for each other. The convention layer governs the agent that authored the PR; the enforcement layer governs everyone and everything else, including the merge button and direct pushes. A strategy with only the convention layer has no gate -- see Required Testing Behavior for Rule 7.

A recorded trace with no upload step is not evidence

The T1 row above says "e2e retries 1–2 with trace-on-first-retry". That is half a mechanism. trace: 'on-first-retry' writes the trace onto the CI runner's disk, and the runner is torn down when the job ends -- so with no upload step the tier pays to record evidence for the sole purpose of throwing it away. Nothing looks wrong while this is true: the gate still goes red on the real failure, and only the artifact list is empty.

Measured: a PR gate ran trace: 'retain-on-failure' and screenshot: 'only-on-failure' for months with no upload step anywhere in the workflow. A later flake investigation paid a 92-execution reconstruction cost re-running the suite to reproduce a failure CI had already captured and discarded.

Retention therefore belongs in the tier's definition, not in an operations checklist somewhere else. T1 is not complete until all three hold:

1. Failure evidence is uploaded, gated on if: failure() || cancelled(). A run killed at its timeout cap or cancelled by a concurrency group is exactly the case whose trace you want, and plain failure() skips it.

2. The uploaded path is one a reporter actually writes. Playwright's default reporter under CI is dot, which produces no playwright-report/ directory at all; uploading that path uploads nothing and the step still succeeds. Guard against it with if-no-files-found: error on the upload step -- see Gate the upload on failure() || cancelled().

3. If the job invokes Playwright more than once, give each invocation its own --output. Playwright empties outputDir at the start of every run, so a second lane deletes the first lane's traces before the upload step runs.

The mechanics, the measured storage numbers behind uploading a report rather than the raw directory, and the workflow guard for point 3 are in Recording a Trace Is Not Keeping It.

T2: full-e2e split -- only when T1 overflows

The trigger is concrete: T1 exceeds its ~10 minute budget. Even then, prefer keeping the split e2e suite required on the PR. Tune the runner's worker count before reaching for sharding -- shard only at roughly 100+ tests and 30+ minutes of runtime. The trade-offs between tuning workers, sharding, and sizing the runner itself are covered in CI Runner Sizing.

T3: scheduled re-exam -- for tests CI runners cannot judge on every PR

T3 has three triggers, not one: a test that is heavy, a test bound to an environment or platform that PR runners cannot provide, or a suite guarding against external dependencies that move independently of the repo -- published packages consumed by pin, where the PR gate's lockfile-pinned install can never see what has changed in the registry since the lockfile was generated. The runner follows the trigger, not a fixed default. Heavy or platform-bound tests need capable hardware: prefer hosted macOS on Apple silicon, with a self-hosted runner as an escalation, not a starting point -- see CI Runner Sizing for the shapes, pricing, and sizing rules behind that choice. An external-dependency drift net needs none of that -- it should run on the same plain-ubuntu runner as T1, because the trigger is registry drift, not a platform capability gap. A workflow_dispatch trigger makes the same job available on demand before merging a risky change. Failures file deduplicated issues instead of blocking PRs, and quarantined @flaky tests run here allowed-to-fail. The concrete implementation -- including the external-dependency drift-net pattern -- is described in Scheduled Re-exam and Night Exam. Gating T2/T3 lanes by base branch rather than running them on every PR is its own branch-topology decision, covered in Release Rounds: A develop→main Branch Strategy.

T4: local heavy lane -- convenience, not enforcement

T4 exists only for genuinely CI-impossible lanes. It splits into a bounded fast pre-push pass (b4push, 5–10 minutes) and an opt-in, platform-gated heavy run (exam). Because anything local can be bypassed -- by humans and by AI agents alike -- T4 is a convenience layer and never a substitute for T1 or T3.

If a wall-clock number is genuinely needed

Split the measurement by what it actually requires:

Four practices that make the remaining local measurement survivable:

• Read the log before believing an elapsed time. A sandboxed environment can deny reads of ordinary files (EPERM: operation not permitted on node_modules/..., on OS temp paths). The runner reports these as unhandled errors, the shard exits non-zero, a wrapper retries, and the retry's time is silently added to elapsed. One observed run logged 101 EPERM occurrences across 71 distinct paths in a single shard. It always resolves on a bare retry, so it is invisible unless the log is grepped. Note the direction of the bias: environmental retries inflate the current configuration's time, flattering whatever change is being proposed. A measurement error that agrees with the hypothesis is worse than no measurement.

• Salvage beats restart. A killed run's --reporter=json --outputFile= reports survive and carry trustworthy per-file durations even when wall-clock totals are contaminated. In the case above, salvaging the completed shards produced the session's most valuable findings and refuted two planning claims.

• Measure a constant directly rather than differencing two full runs. A plan wanted to attribute cost between per-pass startup overhead and worker count, and proposed three full-suite runs to do it. The startup constant takes ~2 minutes: time one trivial test file once, then six times sequentially -- the difference is 5x the per-pass cost, which is startup plus a negligible test body. It isolates the variable better than differencing two full runs, because no suite-execution time is mixed in.

• Kill the driver before the workers, and path-scope every kill. Killing workers alone leaves a backgrounded retry loop reparented to PID 1, which simply re-runs the shard; the symptom is a shard logging exit=143 (your own SIGTERM) with an incremented attempt count while the run continues after you believe it dead. And never pkill by a generic tool name -- pkill -9 -f "vitest" matches other repositories' runs on the same machine. Scope by path.

Execution Surfaces

The diagram shows the four surfaces a test can run on. The PR column is the only enforced gate; the implementing and pre-push surfaces are feedback loops, and the scheduled surfaces are a safety net for what CI runners cannot judge per-PR. Note the dotted edge: when a change touches code that is covered only by heavy-lane tests, a scoped heavy run on a capable host is mandatory before push -- the agent must not declare the work done on the strength of T0/T1 alone.

Making T0 Real for Multi-Fixture E2E

The T0 column in the diagram says "seconds." For a Playwright suite with a single site that is already built, that is achievable. For a suite whose fixtures are N self-contained sites -- each requiring its own build step and its own preview server -- the default setup makes it impossible.

The gap

When playwright.config.ts lists webServer as an array of N entries, every run before a single spec executes must:

1. Build all N fixture sites.

2. Boot all N preview servers and wait for each to become ready.

The inner loop becomes minutes, not seconds. Worse, booting N servers in parallel multiplies the port-race surface: EADDRINUSE collisions and inter-server start-stagger appear as intermittent test failures, which look like flaky tests but are actually a topology problem. The right fix is to design away the port-race surface, not to quarantine it with retries -- see also Heavy Test Decision Rule for when flakiness is a signal that topology needs attention.

Distinct app ports alone do not close this surface: each dev-server instance can also open an auxiliary socket (a devtools inspector, HMR websocket, or metrics endpoint) on a shared default port unrelated to the app port. See The Hidden Auxiliary-Socket Port Race for the symptom, diagnosis, and fix.

Mechanism 1: scope build, webServer, and specs to one fixture via env var

Introduce a single FIXTURE=<name> env var that narrows the build step, the webServer array, and the spec set to just the target fixture. Scoping the servers alone is not enough: if FIXTURE=site-a boots only site-a's server while Playwright still collects site-b and site-c's specs, those specs run against servers that were never started and fail. Narrow the projects array by the same env var so only the target fixture's specs are collected. The inner loop then builds one site, boots one server, and runs one fixture's specs -- no N-way stagger, no port-race. CI (no env var set) still builds, serves, and tests all N.

// playwright.config.ts
import { defineConfig } from "@playwright/test";

const target = process.env.FIXTURE; // e.g. "site-a"

const fixtures = [
  { name: "site-a", port: 4001 },
  { name: "site-b", port: 4002 },
  { name: "site-c", port: 4003 },
];

// One fixture when FIXTURE is set, all N otherwise (CI).
const selected = target ? fixtures.filter((f) => f.name === target) : fixtures;

export default defineConfig({
  // Boot only the selected fixtures' preview servers...
  webServer: selected.map((f) => ({
    command: `pnpm build:${f.name} && pnpm preview:${f.name}`,
    url: `http://localhost:${f.port}`,
    name: f.name,
  })),
  // ...and collect only their specs. Without this, FIXTURE=site-a still runs
  // site-b / site-c specs against servers that were never booted.
  projects: selected.map((f) => ({
    name: f.name,
    testMatch: `**/${f.name}/**/*.spec.ts`,
    use: { baseURL: `http://localhost:${f.port}` },
  })),
});

When developing: FIXTURE=site-a pnpm test — one build, one server, only site-a's specs, feedback in seconds.

Mechanism 2: skip the rebuild when nothing changed

Even with a single fixture scoped, the build step still runs on every inner-loop invocation. A freshness hash eliminates redundant rebuilds.

How to derive the hash: compute it over the fixture's own content files, plus all shared build inputs (config, tooling, shared source), plus the lockfile and the runner config (pnpm-lock.yaml, playwright.config.ts) -- the two files that make dependency and runner-behavior changes visible in the working tree. Derive the shared-input set from git-tracked files' working-tree content. Lockfile-visible dependency changes DO bust the hash: pnpm-lock.yaml changes whenever an installed dependency's version changes, and the lockfile is itself one of the hashed inputs.

The loop:

# scripts/build-if-changed.sh
FIXTURE_DIR="fixtures/$FIXTURE"
HASH_FILE=".build-cache/$FIXTURE.hash"

# hash fixture content + all shared inputs (git-tracked, working-tree content)
# -z / xargs -0 keep filenames with spaces intact and avoid deleted-file breakage
NEW_HASH=$(git ls-files -z -- "$FIXTURE_DIR" shared/ pnpm-lock.yaml playwright.config.ts | sort -z | xargs -0 sha256sum | sha256sum | cut -d' ' -f1)

mkdir -p "$(dirname "$HASH_FILE")"

if [ "$FORCE_REBUILD" = "1" ] || [ ! -f "$HASH_FILE" ] || [ "$(cat "$HASH_FILE")" != "$NEW_HASH" ]; then
  pnpm build:fixture "$FIXTURE"
  echo "$NEW_HASH" > "$HASH_FILE"
fi

FORCE_REBUILD=1 is the escape hatch for changes the hash cannot see: uncommitted tooling state, environment shifts, or external dependency updates.

One input the hash above deliberately omits is the build's own output, and that omission has a cost worth knowing: a key derived from sources alone still matches after a concurrent writer truncates or deletes the artifact the marker vouches for, so where more than one process can write the build directory the key must hash the artifact's own bytes too -- see A Built Artifact With More Than One Writer Is Shared Mutable State.

What this buys

The port-race surface that EADDRINUSE and start-stagger create is a cause of perceived flakiness, not a symptom to quarantine. Scope-and-hash removes it at the topology level. The concrete multi-server webServer patterns for build-and-serve setups are covered in the Production Build Verification section of Playwright Patterns.

Retry Budgets

Two rules keep retries honest:

• Pass-on-retry is a triage signal, not a success. The test is telling you it is nondeterministic; record it and schedule the fix.

• More than 2 retries is a smell. At that point you are paying compute for nondeterminism instead of fixing it.

These budgets govern the suite's own automatic per-test retries. Operator-initiated workflow reruns at a merge gate are a different mechanism with a separate, tighter bound — see the gate operator's playbook.

On plain cargo test this budget is unimplementable — the built-in harness has no retry mechanism at all; see the nextest migration recipe for how a Rust suite gets retries, pass-on-retry telemetry, and per-test timeouts.

The Migration Rule

The ordering matters: if you slim the local gate first and the CI destination then turns out to be incapable of running the test, the test now runs nowhere -- and its coverage silently disappears.

Which Tiers Does a Project Need?

The table below maps by archetype. But archetype is only half the picture -- project maturity is the other dimension.

A tool can be CLI-shaped on the generator side and web-shaped on the runtime it emits -- a Rust code generator (T0–T1, like any CLI) that also ships a browser runtime (client router, islands, view transitions). The test surface follows the emitted artifact, not just the CLI: the runtime's behavior is covered by L2 DOM tests today, and L4–L5 concerns stay latent until a browser-driven suite is actually added -- only then do the heavy tiers apply.

Pre-release / WIP: T3 is deferrable

For a project that has not yet shipped to users, T3 (scheduled rich CI) can be deferred. Standing up T3 early is not cost-justified: hosted-macOS Actions minutes and self-hosted GPU runners cost real money for a project nobody uses yet, and the cron exam.yml + file-exam-issue.sh infrastructure is non-trivial to maintain while the product surface is churning daily.

The interim heavy-lane safety net is T4 (local exam lane). Run the heavy specs locally on a capable machine; the nightly exam run surfaces regressions before they accumulate.

Adopt T3 at or after cutover (release), when the project has users whose regressions justify the standing infrastructure cost.

Three Cases

Case A: a CLI that never needed more than T1

A markdown formatter CLI keeps its entire regression surface in a shared fixture corpus that runs in seconds, so T0+T1 covers everything. That gate proved strong enough to enable a full TypeScript-to-Rust rewrite: the corpus, running on every PR, verified behavioral equivalence while the implementation was swapped underneath. No T2–T4 was ever scaffolded, because there was nothing heavy to put there.

Case B: pixel specs no PR runner can judge

A canvas/GPU-heavy pattern-generation web app has pixel-level specs that fail on software-rendering CI runners. This is environment-incapability, not slowness -- and demotion does not help, because a component test would run in the same GPU-less environment. The specs live in the local heavy lane (T4) and a scheduled re-exam on capable hardware (T3), while T1 keeps everything that does not need a real GPU.

Case C: keyboard specs only real macOS can be trusted with

A Tauri text-editor app has keyboard-shortcut e2e specs that are only trustworthy on real WebKit/macOS. The frontend keeps mocked-IPC tests in T1; the platform-bound specs run on a scheduled macOS job (T3) with on-demand dispatch, plus the local heavy lane (T4) on a capable machine.

Related Pages

• Heavy Test Decision Rule -- the per-test procedure: demote, delete, or assign a tier by asking why it is heavy

• Local Heavy-Lane Admission -- when concurrent sessions starve the T4 lane: admission gate, verdict classification, and the deferral rule

• Scheduled Re-exam and Night Exam -- the concrete T3/T4 implementation pattern

• CI Runner Sizing -- runner shapes, pricing, and the sizing rules behind the T2/T3 hardware choices this page assumes

• Release Rounds: A develop→main Branch Strategy -- gating T2/T3 heavy lanes by base branch instead of running them on every PR

• Required Testing Behavior -- the enforceable agent rules, including graduation and anti-gaming