Heavy Test Decision Rule

What to do when an E2E test is too heavy for PR CI — fix, demote, or delete, then classify why it is heavy and assign the right tier.

The Question This Page Answers

Sooner or later, every project with E2E tests hits the same moment: a test looks valuable, but it is too slow, too hardware-dependent, or too unreliable for the PR gate. The instinctive question is "where else can this test run?" — but that is the second question. Where and when a test runs is a tier question, answered by Execution Tiers. The first question is whether the test deserves a tier at all.

Every heavy test leaves this page through one of three exits:

• Fix — keep the test, classify why it is heavy, and assign it the matching tier (if the test is flaky, see Flake Root-Cause Catalog & Deflaking Recipe)

• Demote — rewrite the assertion at a lower testing level

• Delete — remove the test and watch what happens

Branch 0: Before Assigning Any Tier

Before classifying a heavy test by its heaviness, ask two questions in order.

(a) Is the assertion expressible at a lower level?

If what the test actually asserts can be expressed by a component test or a unit test, demote it. A 90-second browser flow that ends in "the computed total equals 42" is a unit test wearing an E2E costume. Demotion makes the assertion faster, more deterministic, and cheaper — with no loss of meaning.

(b) Has it never caught a real regression, and is it not a critical user journey?

If both are true, delete it — then watch the production bug rate for roughly three months. If nothing surfaces that the test would have caught, it was dead weight. If something does, you have learned exactly which assertion matters, and you can write a better, cheaper test for it.

Only survivors of Branch 0 get a tier.

The Decision Flow

Classify by Why It Is Heavy

"Too heavy" is not one condition. There are four distinct reasons a test can feel too heavy for PR CI, and each demands a different treatment. Conflating them is how projects end up with tests that are local-only (bypassable), deleted out of frustration, or scheduled when they could have stayed on the PR.

1. Slow but CI-capable

The test runs fine on CI runners — it just takes long. This category stays on CI:

• Tune test-runner workers first; parallelism is the cheapest win

• Shard across runners only past roughly 100 tests and 30 minutes of runtime

• Use --only-changed as a lossy PR prefilter, backstopped by a scheduled full run

Tag: @heavy. Never make this category local-only — slowness alone is not a reason to leave the enforced gate.

2. Environment-incapable

The test needs hardware the CI runner does not have: a real GPU, a hardware video encoder. Software rendering on a standard CI runner produces different pixels, so pixel-level assertions fail for environmental reasons, not product reasons. Case B — a canvas/GPU-heavy pattern-generation web app whose pixel-level specs fail on software-rendering CI runners — is the canonical example.

The treatment is the scheduled tier (T3) on capable hardware, plus the local heavy lane.

3. Platform-incapable

The test is only trustworthy on a specific platform: real OS keyboard delivery, a native webview, macOS-only behavior. Case C — a Tauri text-editor app whose keyboard-shortcut e2e specs are only trustworthy on real WebKit/macOS — is the canonical example.

The treatment is a layered split, not one monolithic heavy suite:

1. Mocked-IPC frontend tests — the webview UI logic against a mocked native bridge; CI-safe, stays in the PR gate

2. Native-side mock-runtime tests — the native layer's logic against a mock runtime; also CI-safe

3. A thin native-integration layer — the few specs that genuinely need the real platform; tagged @interactive / @macos-only, run as a T3 scheduled macOS job with on-demand workflow_dispatch for pre-merge escalation

Most of the suite's protection moves into the CI-safe layers; only the thin remainder is scheduled. See Scheduled Re-exam and Night Exam for how that scheduled job is built.

4. Flaky — Not a Heaviness Category at All

A test that fails randomly is not heavy — it is broken. Flakiness routinely gets misfiled as heaviness ("it needs retries, it slows the pipeline down") and shipped off to a scheduled tier, where it keeps flaking with even less visibility. Flaky tests go to the quarantine pipeline below — with a deadline, not a new home.

Tag Taxonomy

The classification result is recorded as a tag on the test itself, so the tier mapping stays mechanical and grep-able:

@verification marks one-time proof artifacts — specs written to prove a change worked once, not to guard it forever. They belong to no gate; see Required Testing Behavior for how promotion from verification to regression must happen.

Classifying a New Test: Mechanical Rules

For a new test, classification must not require judgment. Apply the first rule that matches:

1. Reads pixels or depends on GPU timing → @gpu

2. Needs the OS to deliver real keyboard events → @interactive

3. Runtime over ~60 seconds, or a multi-minute flow → @heavy

4. Otherwise → untagged

The default matters most: a new e2e test is untagged and CI-safe unless one of these rules forces a tag. An agent (or a developer) never starts from "which special tier does my test deserve?" — it starts from "my test runs in the PR gate" and tags only when a rule fires.

The Quarantine Pipeline for @flaky

Quarantine is a pipeline with an exit, not a parking lot:

Step 0 of quarantine: verify the test has ever passed (not pass-by-skip) on some host; if it cannot pass, it is broken, not flaky — fix or delete immediately, no quarantine.

Before entering the pipeline, confirm there is at least one verifiable green run of the exact assertion on any CI host. Pass-by-skip does not count — Playwright reports skips as - rather than ✓, so check the distinction carefully. A test that has never produced a genuine green run is not flaky; it is broken. Sending it to quarantine would suspend product coverage without any chance of recovery. Note that this step is distinct from Branch 0 (which asks whether the test deserves a tier at all): Step 0 asks whether the test can even pass anywhere. See Playwright Patterns for notes on test.skip preconditions — a precondition that "should always hold" must be a hard assertion, not a conditional skip that can silently degrade into a permanent vacuous green.

1. Tagging requires a paper trail. @flaky is only valid with an inline issue URL right next to it:

   // quarantined: https://github.com/your-org/your-app/issues/123
   test("drag preview follows cursor @flaky", async ({ page }) => {
     // ...
   });

2. Excluded from strict gates. A quarantined test must not block a PR — otherwise quarantine means nothing.

3. It still runs — allowed-to-fail — in the scheduled tier. A @flaky tag that runs nowhere is a graveyard: results stop being collected and the test silently rots. Running allowed-to-fail in T3 keeps fresh failure data flowing into the tracking issue.

4. Fix, demote, or delete — with a deadline. The exits are the same three as Branch 0. If nobody fixed it by the deadline, it was not worth fixing: demote it or delete it. For the Fix path, see Flake Root-Cause Catalog & Deflaking Recipe — it covers the five causes of E2E flakiness and the mechanical steps to eliminate each one.

// quarantined: https://github.com/your-org/your-app/issues/123 #[test] #[ignore = "flaky: https://github.com/your-org/your-app/issues/123"] fn drag_preview_follows_cursor() { // ... }

Where to Go Next

• Execution Tiers — the definitions of T0–T4 and when a project should adopt each one

• Scheduled Re-exam and Night Exam — how the scheduled tier (T3) is actually built: cron, macOS runners, deduped issue filing, on-demand dispatch

• Required Testing Behavior — the agent-facing rules: never tag @flaky without a linked open issue, promote verification specs explicitly, and more