CI Runner Sizing

Provider choice by repo visibility, Blacksmith runner shapes and pricing, the default-small stance, and the four rules for when a bigger or different runner actually helps.

The Gap This Page Fills

The rest of this guide talks about "CI runners" as if they were a single, fixed thing. Execution Tiers assumes a PR gate runs "under 10 minutes"; the Heavy Test Decision Rule sends environment-incapable tests to "capable hardware"; Scheduled Re-exam puts platform-bound specs on "hosted macOS." Every one of those pages quietly assumes a two-vCPU Linux runner exists, is cheap, and is the thing you reach for by default.

None of them documents the layer underneath: what runner shapes exist, how they are priced, and the decision rules for when a bigger or different runner actually helps. That knowledge usually lives in a comment at the top of one project's workflow file and dies there — never written down where the next project can find it. This page is that missing layer, written against Blacksmith, the runner provider these strategies are tuned for.

Provider Choice Comes Before Sizing

Every rule below assumes you are paying for CI minutes. Whether that assumption holds at all is decided by a question that comes before any label or sizing table: is the repository public or private?

• Public repository → standard GitHub-hosted runners, full stop. The biggest reason is the only one you need: they are free. GitHub's billing policy states it directly — "GitHub Actions usage is free for self-hosted runners and for public repositories that use standard GitHub-hosted runners" (verified 2026-07-05). No per-minute discount competes with a bill of zero, so for a public repo the provider question ends here. One caveat, and it previews this page's default stance: larger runners are always charged, even on public repositories — the moment a public repo's job leaves the standard shape, its bill starts existing.

• Private repository → Blacksmith, and cost is the #1 reason. A private repo draws down the plan's included minutes and then pays GitHub's list rate for every minute after. At agent-driven PR volume the included quota is gone within days, so the number that matters is the marginal minute — and Blacksmith's roughly half-price Linux minutes on faster machines win that comparison outright. That is why this guide's private-repo CI runs on Blacksmith, and why the rest of this page is written against it.

Visibility first, provider second, sizing last. The sections below only matter once this question has put you on a metered runner.

Blacksmith in One Minute

Blacksmith runners are a drop-in replacement for GitHub-hosted runners: you change runs-on and nothing else. The value proposition is faster machines at a lower per-minute price, billed pay-as-you-go instead of GitHub's included-minutes model.

Runner labels

The label encodes the shape directly, so sizing a job is a one-token edit:

jobs:
  test:
    runs-on: blacksmith-4vcpu-ubuntu-2204

So blacksmith-8vcpu-ubuntu-2404-arm is an eight-vCPU arm64 runner on Ubuntu 24.04. Prefer ubuntu-2204 as the default image — it is the closest match to GitHub's long-standing ubuntu-latest behavior, so migrating a workflow rarely trips over a toolchain-version difference. Move to ubuntu-2404 deliberately, when a job actually needs the newer base.

Pricing and the free tier

• 3,000 free minutes per month, denominated in x64 two-vCPU-equivalent minutes. Bigger, arm, Windows, and macOS shapes draw down that same budget at their own multipliers (an arm two-vCPU minute costs less than an x64 one; a Windows minute costs more; macOS costs much more — see below).

• **~0.004/min for a two-vCPU x64 runner** — roughly half GitHub-hosted's 0.008/min list rate for a standard two-core Linux runner. Blacksmith's own pricing page frames the combined per-minute-plus-speed advantage as roughly a two-thirds total cost saving, because its machines also run the same job faster.

• Price scales approximately linearly with vCPU count. The pricing page publishes the exact figure for the two-vCPU base; from there a four-vCPU runner bills at roughly twice the two-vCPU rate (~0.008/min), an eight-vCPU at roughly four times (~0.016/min), and so on. This linearity is the load-bearing fact behind Rule 1 below — hold onto it.

Transparent dependency caching — and when to skip it

Blacksmith co-locates a cache that the standard cache actions (actions/cache and the popular ecosystem wrappers) hit transparently at high throughput. You do not configure anything special to benefit.

The counterintuitive part is that you often should not add explicit caching at all:

macOS (Apple Silicon) Runners

macOS is a different cost universe, and the shape of that difference decides how you are allowed to use it.

• The smallest macOS shape is six vCPU. There is no small, cheap Mac runner — the entry price is a six-vCPU machine (blacksmith-6vcpu-macos-latest and pinned variants such as blacksmith-6vcpu-macos-15), with a twelve-vCPU option above it.

• Billing is roughly 20× a two-vCPU x64 minute. A six-vCPU macOS minute runs about **0.08/min** — twenty times the ~0.004/min x64 rate. That multiplier, not the absolute number, is what you have to design around.

The consequence is a hard architectural rule, not a preference:

There is an upside hiding in that math. The predictable ~$110/month figure is what finally makes a scheduled real-GPU heavy lane feasible for a project whose Metal/WebGL coverage currently runs only on a developer's local Mac. This is exactly the "environment-incapable" and "platform-incapable" case from the Heavy Test Decision Rule: tests that need a real GPU or real macOS move to a scheduled T3 tier on capable hardware. Hosted Apple-silicon runners are what make that tier a bounded monthly cost instead of "hope someone runs it locally before release." See Scheduled Re-exam and Night Exam for how the scheduled macOS job itself is wired.

The Four Runner-Sizing Rules

The labels and prices are the easy part. The hard part is knowing when a bigger or different runner is the right tool — because the instinct to "just throw more cores at it" is right about half the time and actively wrong the other half. These four rules are the core wisdom of this page.

The default: the smallest shape that works

Before the four rules earn their keep, hold the default: every lane runs on the smallest, cheapest shape that completes reliably — 2vcpu until something below names a reason otherwise.

The justification is an AI-dev-era observation about whose time CI latency actually spends. When implementation is agent-driven, a slower lane's wall-clock lands between agent turns, not in a human's foreground wait — the agents produce work faster than any human reviews it, so a PR gate that takes a few extra minutes still finishes comfortably inside human review cadence. Latency nobody is waiting on is not worth paying for.

So a bigger or more exotic runner needs an explicitly named reason, not a vibe. Legitimate ones look like: a preview deploy the team genuinely sits and waits on, an OS-specific build environment (macOS, Windows), or a failure mode diagnosed under Rule 2 or Rule 3. Rule 1 tells you what a bigger runner buys when you choose to buy it — the default is still not to buy it until the latency has a named owner.

Rule 1: A bigger runner is a latency lever, not a cost lever

For a parallelizable workload, going bigger buys you speed, not savings.

Playwright workers scale with cores by default, and — from the pricing section above — price scales linearly with cores. Those two linear relationships cancel: doubling from four to eight vCPU roughly halves the wall-clock while total cost stays about the same, because you pay twice the per-minute rate for about half as many minutes. The cancellation rests entirely on that first leg — it holds only while the job's effective worker count actually scales with the added cores.

4vcpu:  ~10 min  ×  ~$0.008/min  ≈  $0.08
8vcpu:  ~5 min   ×  ~$0.016/min  ≈  $0.08   ← half the wait, same bill

So the decision is purely about feedback-loop speed. Size a parallelizable job up when a faster PR gate is worth engineering attention; leave it small when the wait is already tolerable. Do not size up expecting the bill to shrink — for this class of work it will not.

For a Playwright suite specifically, run one check before buying any cores: fullyParallel defaults to false, which serializes the tests inside each file onto a single worker, so the suite's floor is its slowest single file rather than total ÷ workers — and cores added past that floor buy nothing at all. See fullyParallel Defaults to False — the Mega-File Tail.

Rule 2: A bigger runner does NOT fix memory-bounded failures

This is the most expensive mistake, because a bigger machine looks like it should help and sometimes appears to for a while.

A vitest/jsdom lane hitting a JavaScript heap OOM was not fixed by moving from two to four vCPU. A larger machine raises the heap ceiling, which only delays the same OOM — the process still climbs to the new ceiling and dies. The failure is bounded by memory accumulation, and a taller ceiling does not change the slope of the climb.

The actual fix is to bound the per-process file count so memory is reclaimed between shards:

vitest --shard=1/4

Sharding caps how many test files a single worker process loads before the run rolls to a fresh process, which keeps peak heap under the ceiling regardless of the machine's total RAM. Note that --shard=1/4 runs only the first of four shards — a CI lane must run all four (1/4 through 4/4, in a matrix or in sequence) or three-quarters of the suite silently goes unrun. The flag bounds per-process memory; it does not by itself execute the whole suite. (Evidence: zudolab/zudo-pattern-gen#1912 — a heap-OOM lane in pgen, Takazudo's private GPU pattern generator with a Cloudflare Workers backend, that survived the 2→4vCPU upgrade and only went green once --shard bounded the per-process file count.) See Vitest Patterns for the surrounding worker- and timeout-budget configuration.

The local-side counterpart -- a shared dev machine starved by concurrent heavy lanes, where the lever is admission rather than lane shape -- is Local Heavy-Lane Admission.

Rule 3: Diagnose the failure mode before resizing

Rule 2 says a bigger runner does not fix memory problems — but it genuinely does fix a different problem that looks almost identical from the outside.

CPU starvation: on a two-vCPU runner, e2e workers and the dev server contend for the same two cores. Under contention, specs miss their timing budgets and fail as timeouts — which reads, on the surface, exactly like a flaky or under-resourced test. Here, moving to four vCPU is the fix: giving the dev server and the test workers their own cores removes the contention. (Evidence: zudolab/zudo-pattern-gen#3321 — spec timeouts under two-core contention, resolved by four vCPU.) If the resize leads you to tune worker counts through an environment variable, validate the value at the boundary — a Playwright workers override that coerces to NaN runs zero tests and exits green, a false green worse than the timeouts you started with.

This fix has a ceiling that Rule 1's cancellation does not. The mechanism is relieving contention at a fixed worker count — not adding parallelism — and it saturates: roughly once the pinned workers and the dev server each have a core of their own, further vCPU buys nothing. With workers: 2 pinned, a 4→8 vCPU bump hands two workers and one dev server more idle cores than they can use — wall-clock ~unchanged, bill doubled, which is Rule 1's pinned-workers trap entered from the other side. Treat the core count as a heuristic, not a formula (browsers spawn subprocesses, and the OS takes its share); the real stopping condition is that the contention timeouts are gone and one more size step no longer moves wall-clock. Size up to that point, then stop.

So two failures that both present as "tests dying under load" have opposite remedies:

The rule that follows: diagnose the binding constraint before you resize, and size each job to its own failure mode. Do not uniform-size a whole workflow — a memory-bound vitest lane and a CPU-starved e2e lane in the same workflow want different treatments, and a blanket "everything to 8vcpu" masks the memory bug while overpaying for the jobs that never needed it.

Rule 4: Cost control comes from trigger design, not runner choice

The single largest lever on your CI bill is not which runner you pick — it is when the expensive lanes run at all.

A heavy lane on an eight-vCPU runner that fires on every topic PR costs far more than the same lane on a schedule, and the runner size is a rounding error next to that difference. The macOS math above is the extreme case — 20× per minute makes the trigger the whole story — but the principle holds for any expensive lane.

Gate expensive lanes by two things:

1. PR base branch. Run the heavy suite only on release/round PRs — those whose base is main — not on every topic-branch PR feeding into an integration branch. Most PRs in an agent-driven flow are topic PRs; keeping the heavy lane off them removes the bulk of the cost. See Release Rounds § Gating Heavy CI by Base Branch for the concrete branch topology and CI workflow this rule assumes.

2. A nightly schedule:. Back the base-branch gate with a scheduled full run so nothing that skipped the topic PRs goes unverified for long. This is the T3 scheduled re-exam tier doing its job.

Reach for a bigger runner to make a lane faster (Rule 1); reach for trigger design to make a lane cheaper (Rule 4). Conflating the two — trying to save money by downsizing a per-PR heavy lane instead of moving it off per-PR entirely — optimizes the rounding error and ignores the bill.

pnpm -r Runs in Topological Chunks

The four rules above all assume the job's shape is what you think it is. For a monorepo whose test lane is one pnpm -r run test, that assumption usually breaks in the same place, upstream of every sizing decision: pnpm -r does not fan every package out at once — it executes in topological chunks, and the next chunk cannot start until the whole previous one has drained. One slow package is therefore a barrier, not merely a slow package. Measured on a 55-package workspace, a seven-minute worker-test suite held 12 unrelated packages behind it, and the effective parallelism of the entire fanout was 1.27× on a twelve-core box — a machine bought for parallel work spent nearly the whole run doing one thing at a time.

That ratio is the diagnostic, and it is arithmetic on numbers the run already prints:

effective parallelism = Σ(per-package wall time) ÷ (run wall time)

55 packages, 12 vCPU available  →  1.27×   ← cores bought, not used

A ratio near 1× on a many-core runner means the lane is barrier-bound — and that is Rule 1's cancellation breaking from a direction Rule 1 does not cover. The workload is parallelizable and nothing pinned a worker count; cores still buy nothing, because the scheduler above the workers refuses to overlap the chunks. Check this ratio before sizing a monorepo test lane up, or a bigger runner becomes the pure-added-cost case wearing a parallel job's clothes.

Severity scales with the package count

Do not carry 1.27× into a workspace it does not describe. The same measurement on a four-package workspace with exactly one real dependency edge — so at most two chunks — found 4.41 s of idle in a 37.7 s run (11.7%), and 3.18 s of 36.8 s (8.6%) on a second run. One package there accounts for 80% of the wall time, so removing the barrier entirely would save about four seconds out of thirty-eight: a refactor nobody should schedule.

More packages means more chunks, and more packages stalled inside each one, so the pattern's severity scales with the package count. At 55 packages it is the dominant cost of the lane; at four it is noise. Measure your own ratio before spending anything on it.

Two levers, and the per-process startup tax

Once the ratio says the lane is barrier-bound, there are two ways out.

Exclude the barrier package. When the slow package already has its own CI job, running it inside the fanout as well pays for it twice and serializes everything behind it:

pnpm -r --filter '!@scope/worker-tests' run test

Or remove the chunking entirely, by moving to a single scheduler at the workspace root — a root vitest config whose test.projects covers every package. One scheduler distributes test files across workers, so there are no package chunks left to wait on. See Project-Level Vitest Configs for the config shape.

The second lever collects a bonus the first does not. pnpm -r run test boots one test-runner process per package, and that startup costs roughly 0.6–1 s per package before a single test executes. Across 55 packages that is about a minute of pure process startup on every run, which no runner size reduces. A single root scheduler pays it once.

Check the edge is real before removing a barrier

Where a real dependency edge exists, topological order is a correctness prerequisite, not only a throughput cost. Measured case: a package's TypeScript declaration pass imports a sibling package's types, so building the dependent first does not run slowly — it fails outright with TS2307. That is precisely why the workspace in question hard-codes a WORKSPACE_BUILD_ORDER constant as the single source of truth for the order, instead of trusting whatever order an invocation happens to produce.

Timing Retros Read Job-Level Timestamps

Every rule on this page sits downstream of a measurement, and the most convenient measurement available is the wrong one. gh run list reports workflow-level createdAt → updatedAt, and that interval includes the time the run spent waiting for a runner. Queue wait and execution are different signals with different owners, and a number that silently adds them together cannot support any decision above.

The gap is not marginal. Measured: two runs sat 72–73 minutes in the queue before they started executing. At workflow level, a perfectly normal 62-minute execution inside such a run reads as a 134-minute near-death run — a number that sends someone resizing a lane whose actual duration never moved.

Short jobs distort worse, proportionally. One sampled run had a seven-second pure-Node guard job wait 303 seconds for a runner. At workflow level that wait is indistinguishable from work: it reads as a five-minute regression in a gate whose actual work was 232 s.

Read the jobs endpoint, not the run list

The corrected measurement is job-level startedAt → completedAt, which the run's jobs endpoint reports per job:

gh api repos/{owner}/{repo}/actions/runs/<run-id>/jobs \
  --jq '.jobs[] | {name: .name, started: .started_at, completed: .completed_at}'

Re-measuring does not always overturn the claim. A documented "~3 min" e2e job, re-measured this way, came back at 212 s and 242 s — the claim survived. The re-measurement was still worth doing, because what was broken there was the protocol, not the number.

Everything that aggregates runs inherits the same defect: budget dashboards, cancellation-rate stats, and especially "X% of runs hit the timeout cap" claims need re-verification at job level before they drive a decision. A cap-hit rate computed from workflow timestamps counts queue-heavy runs as timeouts they never were, and then argues for exactly the wrong lever — a bigger runner for a lane that was never slow. The "under 10 minutes" PR-gate budget in Execution Tiers is an execution budget in the same sense: measure it at job level, or you are tuning your lane against a queue you do not control.

An unpinned timing claim silently regresses

A number with no stated source rots quietly. The "~3 min" claim above named no timestamp source, so it was right by luck — and the next person to refresh it would reach for gh run list, the convenient command, write a workflow-level number under the same sentence, and nothing would fail. Pin the source next to the number: which endpoint, which field pair, how many samples.

Keeping the signals apart is what makes them comparable over time. The matched experiment behind these figures recorded queue delay, setup, and job execution separately for exactly that reason, and those signals then moved independently: per-job queue-delay maxima went from 83/17/13/14/11 s before a change to 18/14/12/14/15 s after, while critical-path job maxima stayed inside the same 313–484 s band throughout. Blended into one workflow-level number, that queue improvement would have read as an execution change nobody made.

Rule 3 says diagnose the failure mode before resizing. These two sections are the sentence underneath it: diagnose with numbers that measure the thing you are about to change — the lane's own execution, not the scheduler above it and not the queue in front of it.

The full experiment protocol these measurements feed — which clock is the target, how many identical-SHA samples decide a keep or a revert, and how to prove that the faster gate still runs the same tests — is CI Gate Optimization.

Putting It Together

The through-line: a runner label is a two-token edit, but choosing it well means naming the binding constraint first. Latency, memory, CPU contention, and total cost are four different problems with four different levers, and a bigger runner is the correct answer to exactly two of them — latency (Rule 1) and CPU starvation (Rule 3) — with both answers bounded by the same config knob: the effective worker count.

References

• About billing for GitHub Actions — the free-for-public-repos policy on standard runners, and the larger-runners-are-always-charged caveat (verified 2026-07-05).

• Blacksmith runner overview — the full label matrix, OS images, and free-tier / consumption-multiplier details (verified 2026-07-05).

• Blacksmith pricing — per-minute rates by shape (verified 2026-07-05).

• Execution Tiers — the T0–T4 model these sizing rules serve.

• Heavy Test Decision Rule — how environment- and platform-incapable tests reach the scheduled macOS tier.

• Scheduled Re-exam and Night Exam — how the scheduled (T3) macOS job is actually built.

• CI Gate Optimization — the acceptance protocol these timing rules feed: naming the clock, the median-of-three sample rule, and the identity proof that a faster gate did not quietly lose coverage.

Scope of the optimization

Speeding up the same suite preserves canonical test identities and real-environment evidence, not merely test counts. Replacing/consolidating tests instead needs a guarantee mapping; reducing frequency or retiring detection needs the explicit timing/risk decision in test lifecycle. AI suite audit compares these alternatives and distinguishes billed execution, elapsed wait, and estimates. Change-scoped verification governs selection and reusable results. Removing merge waiting under transient-red development does not itself reduce runner cost.