Required Testing Behavior

Mandatory behaviors for AI agents when testing frontend code.

The Five Core Rules

Every AI agent working on frontend code must follow these five rules when testing:

Rule 1: Declare Your Test Plan First

Before writing any test or running any verification, state:

• What you are testing

• Which level you are using

• Why that level is appropriate

Test plan:
- Testing: notification banner visibility after CSS fix
- Level: 5 (verify-ui + screenshot)
- Reason: This is a visual bug — the element exists in DOM but
  user reports it's not visible. Need to check computed styles.

Rule 2: Match Test Level to Goal

The test level must match what you are actually verifying:

Do not use Level 1 for a Level 5 problem. The test will pass, but the bug will remain. Do not jump to Level 6 just because L4 is hard or L5 is awkward — L6 requires that both lower levels (L4 and L5) are genuinely intractable.

Rule 3: Escalate When Lower Levels Pass But Problem Persists

When a test passes but the user says the problem is not fixed:

1. Do not re-run the same test

2. Do not suggest the user clear their cache

3. Do escalate to the next testing level

4. If already at Level 4, escalate to Level 5

5. If at Level 5, investigate deeper (check ancestor elements, stacking context, etc.)

6. Only escalate to Level 6 (AI-based) when both L4 is intractable to write and L5 cannot reach the assertion. L6 is not the next step after L5 by default — it is reserved for canvas / photo-editor surfaces where no deterministic check is feasible.

Rule 4: Default to Level 5 for UI/CSS

Any task involving:

• CSS changes

• Layout modifications

• Visibility issues

• Visual appearance

• Responsive design

• Spacing, colors, fonts

should default to Level 5 verification. Lower levels are structurally unable to verify CSS correctness.

Rule 5: Report What Was NOT Tested

After testing, explicitly state the blind spots:

Verification complete:
- Tested: computed styles confirm banner has display:block,
  opacity:1, and parent has overflow:visible
- Screenshot: banner is visible at top of page
- NOT tested: responsive behavior at mobile breakpoints
- NOT tested: animation transition timing

This transparency helps the user decide if additional testing is needed.

The Four Execution Rules

Agents operating inside a tiered test suite must also follow these four rules. For background on verification artifacts, tier tags, and the graduation lifecycle, see Execution Tiers.

Rule 6 — Verification Tests Do Not Graduate Themselves

Classify ongoing purpose → level → rerun conditions/frequency/budget → tags under test lifecycle. Writing regression coverage directly under established policy is normal development; it does not need new individual human approval. One-time verification artifacts carry @verification and stay outside gates. Promotion requires an explicit proposal and reviewer decision; an agent does not self-promote its own artifact.

The proposal names the continuing contract and failure, suitable level, duplication/replacement evidence, relevant rerun conditions and detection timing, determinism, capable environment, tier, and budget. After admission remove @verification and assign the appropriate tier tag. Untagged means CI-safe after admission, not automatic lifetime approval.

Enforcement: catch the specs that never got tagged

Use names containing dates, issue numbers, bugfix, or iteration counters as discovery hints, not automatic gate failures or grounds for exclusion. Inspect the contract and role. An issue-numbered data-loss regression may be permanent coverage despite years of green. Renaming does not decide admission. Execution tiers covers the silent-admission failure mode; AI suite audit bounds investigation without requiring a ledger or human review of every test.

Rule 7 — Red Checks Block the Author

A red check creates an author obligation even when it is not required by repository settings. Under the default green-before-merge policy, resolve it before merging, except valid @flaky quarantine with a linked open issue and a reporting lane. A declared transient-red development policy can permit an authorized development merge while CI is pending or a localized red has tracked repair. The failure remains owned after merge; it is never reported as passed.

Continue independent work only when the failure is understood/localized, the next work does not depend on it, and a concrete owner/checkpoint tracks repair. Pause affected chains for unknown or cross-cutting failures, broken build/runtime foundations, dependent work, lost attribution, or security/data/publication hazards. Required release/deploy/publish/migration evidence remains blocking before external impact. Overall completion still requires integrated verification and alert reconciliation.

A merge is integration, not proof of success. Pending, cancelled, superseded, skipped, and deferred checks are not passes. Record tested SHA, run/job, evidence, owner, and next action. Neither a non-required label nor an old green run excuses ignoring red. Respect the actual protections and approvals; this rule grants no bypass.

The gate operator's playbook supplies bounded triage for red you did not author, including asynchronous integration alerts.

Rule 8 — Never Game the Gate

Never introduce any of the following without a linked open issue reference:

• test.skip on an existing test

• @flaky tag on a test that was previously passing

• Loosened screenshot tolerance (e.g., raised pixel threshold)

• Deleted assertions

Making a gate pass by editing existing assertions requires a fresh-context review of the test diff — do not apply such edits in the same session that authored the original change.

This requirement extends to gate-defining files, not just assertions. A gate can be neutralized without touching a single assertion: narrowing a testMatch pattern so failing specs stop running, adding a --grep-invert filter, raising a retry count, dropping or de-requiring a CI job, or editing a fixture/setup script so the failing scenario is never built. Because retry budgets, grep filters, testMatch patterns, and job wiring can neutralize a gate this way, any edit to a gate-defining file — test-runner config, CI workflow definitions, fixture/setup scripts, the pre-push gate script — requires the same fresh-context review: read the gate file from scratch rather than trust the author's diff summary, and do not apply such edits in the same session that authored the original change.

Projects should enumerate their concrete gate-file list in their testing strategy doc (for example: the runner config, the fixture setup script, both CI workflow files, and the pre-push script).

A runner per-test timeout or liveness deadline is a guardrail, not an assertion: it turns a hang into a diagnosable failure and asserts nothing about the product. Raising it is not a loosened assertion tolerance, provided you show that the new value is derived either from a measured latency distribution (baseline versus under load) or from the child deadline the budget has to contain, link an open issue, and obtain the same fresh-context review for any gate-defining file edit, including runner config. See the "A fixed deadline as a failure guardrail on an already event-keyed wait" bullet in the deflaking recipe's root-cause catalog and Per-Project Timeout Budgets.

# Wrong — no issue reference
test.skip('banner renders correctly', () => { ... })

# Wrong — no issue reference
expect(page).toHaveScreenshot({ maxDiffPixels: 500 }) // was 50

# Wrong — gate-defining file edited to make a gate pass, no fresh-context review
testMatch: ['**/*.spec.ts', '!**/banner-*.spec.ts'] // silently excludes failing spec

# Wrong — no derivation, no open issue, no fresh-context review
testTimeout: 30_000 // was 5000

# Right (a) — legitimate skip: a real, deliberate skip, not a flake
// skipped: https://github.com/your-org/your-app/issues/55 — pending backend flag rollout
test.skip('banner renders correctly', () => { ... })

# Right (b) — quarantine: NOT skipped; the test still runs, allowed-to-fail,
# with @flaky in the title, exactly matching the quarantine pipeline's convention
// quarantined: https://github.com/your-org/your-app/issues/55
test('banner renders correctly @flaky', () => { ... })

# Right (c) — project-scoped guardrail; runner config received fresh-context review
test: {
  projects: [
    { extends: true, test: { name: 'unit', include: ['src/**/*.test.ts'] } },
    {
      extends: true,
      test: {
        name: 'scripts',
        include: ['scripts/**/*.test.ts'],
        // 2x the largest 30s child deadline; open issue:
        // https://github.com/your-org/your-app/issues/56
        testTimeout: 60_000,
      },
    },
  ],
}

For projects that condense Rule 8 into their own agent-facing docs, use this one-sentence summary:

Never weaken a gate without a linked open issue; require fresh-context review for assertion changes made to pass the gate and all edits to gate-defining files; and treat runner timeouts and liveness deadlines as guardrails, not assertions, raising them only with a linked open issue and a value derived either from a measured latency distribution (baseline versus under load) or from the child deadline the budget must contain.

Rule 9 — Scoped Heavy Verification

When a change touches code that is covered only by heavy-lane tests (tagged @heavy, @gpu, @interactive, or @macos-only), run the related heavy specs on a capable host before declaring the work done. Alternatively, dispatch the exam workflow on the branch.

See Heavy Test Decision Rule for how to identify which tests count as heavy-lane and which host is required.

# Before declaring done — change touches GPU rendering path
$ gh workflow run exam.yml --ref my-feature-branch
# or, if running locally on a capable host:
$ pnpm exec playwright test --grep "@gpu" tests/gpu-render.spec.ts

Split Confirm Checks by Input Availability

In an agent workflow, a confirm task is a validation pass with named checks. It runs after sibling work has merged into a base branch, and its job is to exercise the merged result rather than add features.

Classify every named check by when its input exists, and write both lists and their owners in the task text:

• Local checks on the merged base — run by the child. These are available as soon as the child is spawned: fast gates, unit suites, self-tests, formatting, and typechecking.

• Checks on the PR or a CI run — run by the manager after the push. These include reading a job log, confirming that a required check is green, and observing a warning annotation. In the worktree/epic workflow shape, where the root PR opens only after all topics have merged and review has finished, those inputs cannot exist when the child is spawned. This is a property of that workflow shape, not a universal restriction on children or CI.

Stating both lists prevents the executor from silently skipping the CI half or stalling while it waits for a PR that cannot exist yet.

The zfb project -- Takazudo/zudo-front-builder, the Rust-based frontend build orchestrator this site runs on -- has a worked case in Takazudo/zudo-front-builder#2883. Its confirm sub-issue asked for five local commands and two CI-log reads. The child ran all five local commands, then reported the two log checks as deferred and explained that the root PR did not exist. After the root PR opened, the manager performed those checks and posted the observations on the sub-issue, including the run URL and whether the drift warning annotation appeared. This ad hoc split cost nothing only because the child reported the deferral explicitly instead of marking the whole task done.

Following Rule 9, assign expensive verification once to the executor that can run it: either a capable local host or the CI run. Input availability determines the split, not whether a check is cheap. A confirm task may still run a full local build on the merged base.

Summary Checklist

Before declaring any fix complete:

• Test plan was declared before testing

• Test level matches the nature of the change

• If test passes but user reports failure, escalated to next level

• For CSS/visual changes, used Level 5

• Reported what was and was not tested

• New verification specs are tagged @verification and not self-promoted

• Admission was decided by ongoing purpose, not names or renaming

• Red checks remain owned; the declared merge and external-impact gates are satisfied

• No test.skip, @flaky, or loosened tolerances introduced without a linked issue

• If change touches heavy-lane-only code, related heavy specs were run on a capable host

• Confirm tasks separate child-runnable local checks from manager-runnable PR/CI checks, and say which is which