Migrating a Rust Suite to cargo-nextest

Why the guide's retry-budget, per-test-timeout, and flake-telemetry policy is unimplementable on plain cargo test, and a mechanical checklist for migrating a Rust suite to cargo-nextest.

Why nextest Is the Gate for the Rust Flake Pipeline

The Heavy Test Decision Rule and Execution Tiers impose the same three obligations on every CI suite in this guide: a retry budget of 1–2, pass-on-retry recorded as a triage signal, and a per-test timeout so a hung test cannot hang the whole job. On a Rust project those are not merely awkward on plain cargo test — they are unimplementable. cargo test has no retry mechanism, no per-test timeout, and no stable machine-readable output. cargo-nextest supplies all three, which is why the Rust flake pipeline runs through nextest rather than the built-in harness.

The retry budget maps to retries, the pass-on-retry triage signal to nextest's FLAKY report and JUnit output, the per-test timeout to slow-timeout + terminate-after, and the machine-readable issue-filing feed to JUnit XML. Each policy line in this guide has a direct nextest mechanism and no cargo test equivalent.

Config Sketch

Two files carry the policy. The nextest profile encodes the retry budget, the per-test timeout, and the JUnit telemetry, and it re-establishes the serialization that plain cargo test gave the suite for free:

# .config/nextest.toml
[profile.ci]
retries = 2                                              # the CI retry budget (1–2)
slow-timeout = { period = "60s", terminate-after = 4 }   # per-test timeout

[profile.ci.junit]
path = "junit.xml"                                       # flake telemetry / deduped issue filing

# Preserve the serialization the suite silently relied on under cargo test:
[test-groups]
heavy-serial = { max-threads = 1 }

[[profile.default.overrides]]
# every integration-test binary that boots a dev server / binds ports / embeds V8
filter = 'binary(dev_serve_e2e) | binary(dev_build_static_parity) | binary(dev_serve_injected_routes_e2e)'
test-group = 'heavy-serial'

The CI step swaps cargo test for cargo nextest run — but nextest does not run doctests, so the doctest step has to be kept explicitly:

# CI
- uses: taiki-e/install-action@nextest   # cargo nextest is not part of the default toolchain — install it first
- run: cargo nextest run --workspace --profile ci
- run: cargo test --doc --workspace   # nextest does not run doctests — drop this step and doctest coverage silently disappears

Migration Checklist (Mechanical)

1. Measure the doctest surface first — classify fences (runnable rust vs text/ts/html/json that rustdoc never runs); record the real count in the migration PR. Keep cargo test --doc --workspace as a separate step. (Folklore counts are usually far higher than the measured count.)

2. Enumerate implicit serialization — cargo test runs test binaries sequentially; nextest parallelizes across binaries. List every integration-test binary that binds ports / boots servers / embeds a heavyweight runtime and put them in a max-threads = 1 test-group before the first parallel run, not after the first flake storm. Optionally add an advisory flock in the shared test-utils crate as defense-in-depth.

3. Audit concurrency claims in comments — grep for the claimed mechanism (flock, lock files, mutexes) and verify it actually exists before trusting it.

4. Reconcile inventories — diff cargo test -- --list vs cargo nextest list per binary; explain every delta in the PR. Silent omission (a test nextest filters differently) is the main failure mode — a green migration PR running fewer tests is worse than no migration.

5. Quarantine interop — the existing #[ignore] quarantine Note maps cleanly: the T3 allowed-to-fail job becomes cargo nextest run --run-ignored ignored-only (optionally name-filtered, same caveat as the current Note).

Executing the Taxonomy

The reason-string taxonomy on Heavy Test Decision Rule § Same Rules, Rust Syntax classifies why a test carries #[ignore]. This section covers how nextest turns that classification into an actual execution lane.

--run-ignored: the Quarantine/Exam Lane

cargo nextest run --workspace --profile ci                              # T1: the PR gate -- never runs an #[ignore]'d test, whatever its reason
cargo nextest run --workspace --profile exam --run-ignored ignored-only # T3: the quarantine/exam lane -- runs ONLY the ignored set
cargo nextest run --workspace --profile exam --run-ignored all          # full-inventory sanity run -- everything, ignored and not; not a gate

--run-ignored ignored-only is the nextest analogue of Playwright's --grep "@flaky" quarantine lane from Scheduled Re-exam — except it selects on the presence of #[ignore], not on the reason string's content. nextest never parses what is inside the quotes; the string is for humans and grep, not for nextest's own filtering. A workspace whose ignored set mixes heavy: and flaky: reasons runs both in the same ignored-only invocation, which erases a distinction the Playwright side keeps separate via independent --grep expressions. Narrow with a name filter when the two need to be judged differently:

# separate "expected to always pass, just slow" from "expected to sometimes fail"
cargo nextest run --profile exam --run-ignored ignored-only -E 'test(/^heavy_/)'
cargo nextest run --profile exam --run-ignored ignored-only -E 'test(/^flaky_/)'

This generalizes the caveat the quarantine Note above already raises for flaky: alone: #[ignore] is a broader marker than any single reason across the whole taxonomy, not just against @flaky.

Test-Groups: Budget and Serialization Control in the Exam Lane

The Config Sketch above uses a test-groups entry (heavy-serial) to re-serialize integration binaries that bind ports. The exam lane needs the same mechanism for a different reason: it is allowed to be slow, but not unbounded — a heavy:-ignored set that saturates every thread on the runner turns "a few slow tests" into the exam lane becoming its own heavy-suite problem.

Add the new group to the existing [test-groups] table from the Config Sketch above -- a second [test-groups] header in the same file is invalid TOML, since a table cannot be declared twice:

# .config/nextest.toml -- inside the [test-groups] table already opened by the Config Sketch
heavy-serial = { max-threads = 1 } # existing, from the Config Sketch above
exam-heavy = { max-threads = 2 }   # new: caps concurrency for `heavy:`-ignored tests specifically

The exam profile itself is new, so it gets its own tables:

# .config/nextest.toml (new tables, appended after the Config Sketch above)
[profile.exam]
retries = 0                        # exam is diagnostic, not gated -- no retry budget to hide a real fail

[profile.exam.junit]
path = "junit.xml"                 # same machine-readable feed the ci profile uses, for the filing script

[[profile.exam.overrides]]
filter = 'test(/^heavy_/)'         # name-convention filter narrowing to the `heavy:` reason
test-group = 'exam-heavy'

A dedicated [profile.exam] — not the ci profile with --run-ignored bolted on — matters because the exam lane's retry and threading defaults are legitimately different from the PR gate's: T1 wants a real retry budget (see Retry Budgets); the exam lane wants zero, so a pass is unambiguous and a fail is real.

The Allowed-to-Fail Scheduled Exam Job

The scheduled job mirrors the Playwright quarantine lane's exit-code-capture pattern from Scheduled Re-exam § Flake Telemetry Mechanics exactly: continue-on-error: true paired with if: failure() is a dead filing step there for the same reason it would be here — step-level continue-on-error rewrites the step's conclusion to success, so a downstream if: failure() never fires.

# .github/workflows/exam.yml
name: exam

on:
  schedule:
    - cron: "51 3 * * *"  # off-minute on purpose -- see Scheduled Re-exam
  workflow_dispatch:

permissions:
  contents: read
  issues: write

jobs:
  exam:
    runs-on: ubuntu-latest
    timeout-minutes: 60
    steps:
      - uses: actions/checkout@v4
      - uses: dtolnay/rust-toolchain@stable
      - uses: taiki-e/install-action@nextest

      - name: Run the quarantine/exam lane (allowed-to-fail BY DESIGN)
        id: exam
        run: |
          set +e
          cargo nextest run --workspace --profile exam --run-ignored ignored-only
          rc=$?
          set -e
          echo "rc=$rc" >> "$GITHUB_OUTPUT"
          # Deliberate: this lane is green-by-design. The captured rc -- not the
          # job status -- drives the filing step below.
          exit 0

      - name: File or update the tracking issue
        if: steps.exam.outputs.rc != '0'
        env:
          GH_TOKEN: ${{ github.token }}
        run: bash scripts/file-exam-issue.sh

file-exam-issue.sh is the same one-tracking-issue-per-workflow pattern from Scheduled Re-exam — the dedup-by-label-and-title, comment-vs-create, and close-on-green logic all port unchanged. Only the failure-extraction step differs: parse the junit.xml produced by [profile.exam.junit] above instead of Playwright's JSON reporter.

Where to Go Next

• Heavy Test Decision Rule — the full #[ignore] reason-string taxonomy (env-gate:, heavy:, flaky:, verification:, pending-feature:) this nextest config executes, and how a Rust flake is quarantined

• Execution Tiers — the retry budget this migration exists to make implementable on a Rust suite

• Flake Root-Cause Catalog & Deflaking Recipe — the native-suite flake causes (port races, implicit serialization, shared state) that surface the moment nextest parallelizes across binaries