Balancing test shards to reduce CI wall-clock time

This page is part of the Test Parallelization & Sharding in CI section, which covers splitting a slow test suite across runners so the pipeline finishes sooner. Here we fix a specific and expensive symptom: you already shard your visual tests, but one shard runs for eight minutes while the others finish in two — so the extra runners buy you almost nothing.

Problem statement

You split your Playwright visual suite across four shards expecting a roughly 4× speed-up. Instead the pipeline still takes eight minutes, because shard 3 always runs long while shards 1, 2, and 4 sit idle after two minutes. The total pipeline duration is the maximum shard time, not the average, so an unbalanced split throws away most of the parallelism you paid four runners for.

The imbalance is not random. The slow shard is the one that happened to receive several heavy visual specs — the ones that launch a browser, render a page, wait for fonts, and diff a full-page screenshot. You want each shard to carry roughly equal total time so the max and the mean converge and the pipeline is gated by the average shard, not the worst.

Root cause

Default sharding splits tests by count — an equal number of files or a hash of the file path per shard — which silently assumes every test costs the same wall-clock time. Visual specs cost an order of magnitude more than unit specs, so a count-balanced split routinely puts several slow specs on one shard and a pile of fast ones on another. Because sharding across CI runners makes the pipeline wait for the slowest shard to report, that one overloaded shard sets your entire wall-clock and gates the merge.

Count-based versus duration-balanced sharding Left: four shards split by file count, shard 3 is an 8-minute bar while the others are 2 minutes, and the pipeline is gated at 8 minutes. Right: the same tests split by measured duration, all four shards are about 3 minutes and the pipeline is gated at 3 minutes. Split by file count Split by measured duration s1 s2 s3 8 min s4 pipeline gated at 8 min s1 s2 s3 s4 pipeline gated at 3 min

Minimal reproduction

A count-based shard split looks balanced by file count and is wildly unbalanced by time. Here four shards each get a quarter of the files:

# .gitlab-ci.yml — count-based sharding, no timing awareness
visual-tests:
  image: mcr.microsoft.com/playwright:v1.48.0-jammy
  parallel: 4
  script:
    - npm ci
    # CI_NODE_INDEX / CI_NODE_TOTAL are 1-based in GitLab; Playwright wants shard i/N
    - npx playwright test --shard=$CI_NODE_INDEX/$CI_NODE_TOTAL

Measuring each shard reveals the spread that file count hides:

shard 1/4   file count: 22   wall-clock: 2m 04s
shard 2/4   file count: 22   wall-clock: 1m 58s
shard 3/4   file count: 21   wall-clock: 8m 12s   ← four heavy visual specs landed here
shard 4/4   file count: 21   wall-clock: 2m 11s

pipeline wall-clock: 8m 12s   (gated by shard 3)

Every shard has about the same number of files, yet shard 3 runs four times longer because the expensive specs clustered on it.

Step-by-step fix

1. Capture per-test timing from a representative run

You cannot balance what you have not measured. Emit a report that records each test’s duration so a splitter can weight files by real cost.

// playwright.config.ts
import { defineConfig } from '@playwright/test';

export default defineConfig({
  testDir: './tests/visual',
  fullyParallel: true,               // let Playwright parallelize within a shard too
  reporter: [
    ['list'],
    ['json', { outputFile: 'playwright-report/timings.json' }],  // per-test durations
    ['blob'],                        // blob report feeds --shard timing balance
  ],
});

What this does: the json reporter writes a timings.json with the wall-clock of every test, and the blob reporter produces the artifact Playwright’s own balancer can read on the next run. fullyParallel: true ensures each shard uses all its workers rather than serializing files.

2. Let Playwright balance shards by recorded timings

Recent Playwright balances --shard by duration when a previous run’s blob report is available. Persist that report as a CI artifact and restore it before the sharded run.

# .gitlab-ci.yml
stages:
  - baseline
  - test

# Nightly job records timings once and caches them for the sharded jobs
record-timings:
  stage: baseline
  image: mcr.microsoft.com/playwright:v1.48.0-jammy
  rules:
    - if: '$CI_PIPELINE_SOURCE == "schedule"'
  script:
    - npm ci
    - npx playwright test          # full run, no shard → complete timing data
  artifacts:
    paths:
      - playwright-report/timings.json
    expire_in: 1 week

visual-tests:
  stage: test
  image: mcr.microsoft.com/playwright:v1.48.0-jammy
  parallel: 4
  needs:
    - job: record-timings
      optional: true              # use cached timings when present
  script:
    - npm ci
    # Duration-aware: Playwright reads timings to make shards equal-time
    - npx playwright test --shard=$CI_NODE_INDEX/$CI_NODE_TOTAL

What this does: the scheduled record-timings job runs the whole suite once (unsharded) to produce complete per-test durations, and the sharded visual-tests jobs consume that report so each --shard slice targets equal total time instead of equal file count.

3. Fall back to a duration-aware partitioner when you cannot rely on Playwright’s balancer

If you run a mixed toolchain or an older version, compute the split yourself: greedily assign the longest tests to the currently-shortest shard (the classic longest-processing-time heuristic).

// scripts/balance-shards.mjs — greedy longest-processing-time bin packing
import fs from 'node:fs';

const shardTotal = Number(process.env.CI_NODE_TOTAL ?? 4);
const shardIndex = Number(process.env.CI_NODE_INDEX ?? 1);   // 1-based (GitLab)

// { "tests/visual/table.spec.ts": 41200, ... } durations in ms
const timings = JSON.parse(fs.readFileSync('playwright-report/timings.json', 'utf8')).durations;

const files = Object.entries(timings).sort((a, b) => b[1] - a[1]); // slowest first
const shards = Array.from({ length: shardTotal }, () => ({ total: 0, files: [] }));

for (const [file, ms] of files) {
  const lightest = shards.reduce((a, b) => (a.total <= b.total ? a : b));
  lightest.files.push(file);
  lightest.total += ms;
}

// Emit just this runner's files for `playwright test <files...>`
process.stdout.write(shards[shardIndex - 1].files.join(' '));
visual-tests:
  stage: test
  image: mcr.microsoft.com/playwright:v1.48.0-jammy
  parallel: 4
  script:
    - npm ci
    - FILES=$(node scripts/balance-shards.mjs)
    - npx playwright test $FILES     # this runner's duration-balanced slice

What this does: the script sorts specs slowest-first and repeatedly drops the next spec onto whichever shard currently has the least accumulated time, so no single shard absorbs all the heavy visual specs. Each runner then executes only the files assigned to its index.

4. Re-measure and enforce a wall-clock budget

Balancing is only real if you verify it. Re-run the pipeline, compare the slowest shard to the mean, and fail loudly if any shard blows a budget.

# Print each shard's duration and the imbalance ratio (max / mean)
$ node scripts/report-shard-times.mjs
shard 1  2m 55s
shard 2  3m 06s
shard 3  3m 12s
shard 4  2m 58s
max/mean imbalance: 1.05   (was 3.9 before balancing)

What this does: an imbalance ratio near 1.0 means the shards are even and the pipeline is gated by the average, not the worst. Tracking the ratio over time catches drift when someone adds a new heavy spec that timing data has not yet accounted for.

Verification

The pipeline wall-clock should now track the average shard, and the max/mean ratio should sit close to 1:

shard 1/4   wall-clock: 2m 55s
shard 2/4   wall-clock: 3m 06s
shard 3/4   wall-clock: 3m 12s
shard 4/4   wall-clock: 2m 58s

pipeline wall-clock: 3m 12s   (was 8m 12s — 2.6× faster on the same 4 runners)

The heavy visual specs are now spread one-per-shard instead of clustered, so no single shard runs long. Because the pipeline is gated by the slowest shard, driving that number down from 8m to 3m 12s is the entire win — and it came from redistributing existing work, not adding runners.

Edge cases and caveats

  • Timing data goes stale as the suite changes. A newly added heavy spec has no recorded duration, so a duration-aware splitter treats it as cheap and may pile it onto an already-full shard until the next timing run. Refresh timings on a schedule (nightly is usually enough) and treat a spike in the max/mean ratio as the signal to re-record.
  • Fixed per-shard overhead sets a floor. Every shard pays browser start-up, npm ci, and artifact upload regardless of how few tests it runs. Past a certain shard count the slowest shard stops shrinking because that fixed overhead dominates — adding runners then only adds queue contention and cost.
  • Balance by test, not just by file, when one file is the outlier. If a single spec file contains many slow cases, no file-level split can break it up; the file itself becomes the floor. Enable fullyParallel so Playwright parallelizes cases within the file across the shard’s workers, or split that file so the balancer has finer-grained units to distribute.

FAQ

Why is one CI shard so much slower than the others?

Default sharding splits by file count or by a hash of the file path, which assumes every file costs the same. Visual specs are far heavier than unit specs — they launch a browser, render, wait for fonts, and diff screenshots — so when several of them land on the same shard that shard runs for minutes while count-balanced siblings finish in seconds. The pipeline is gated by the slowest shard, so the imbalance sets your total wall-clock.

Does Playwright balance shards automatically?

Playwright’s --shard flag distributes tests across shards and, from recent versions, balances by recorded timings when a previous run’s report is available, aiming for equal duration rather than equal count. It still splits at the test level within the shard’s assigned set, so pairing --shard with a persisted timing report and fully-parallel mode gives the most even distribution without a third-party splitter.

How many shards should I use for visual tests?

Add shards until the slowest shard’s wall-clock stops dropping meaningfully — usually when per-shard time approaches the cost of your single most expensive spec plus fixed browser start-up overhead. Beyond that point extra shards only add runner start-up cost and queue contention. Measure the slowest shard, not the average, because the slowest one is what gates the merge.