Test Parallelization & Sharding in CI #

A component and visual test suite grows faster than any other part of a frontend codebase — every new variant adds stories, and every story adds screenshots. Left on a single runner, the suite eventually dominates pull-request wall-clock time, and slow feedback is the surest way to erode trust in a gate. This page sits under the CI/CD test gating guide and covers the fix: splitting the suite across many runners so total elapsed time falls even though total compute stays roughly constant.

The two levers are easy to conflate. Parallelization runs several tests at once inside one machine using worker processes. Sharding cuts the whole suite into disjoint slices and sends each to a separate machine. They multiply: four shards of four workers each run up to sixteen tests at once. The work of this page is choosing a shard count, fanning out over a CI matrix, recombining the per-shard results into one verdict, and keeping the split balanced so no single runner drags out the pipeline.

Prerequisites #

• Playwright ≥ 1.44 (the visual regression runner used throughout) with a working local playwright test run
• A CI pipeline running tests on pull/merge requests (see GitHub Actions workflows)
• Committed baselines captured in the CI environment, not locally
• Per-engine tolerance thresholds matching the cross-browser matrix you shard across
• A merged-result gate in place (see pipeline gating thresholds) so the combined report, not a single shard, decides the check

How sharding maps a suite onto runners #

Sharding is a partition problem. Playwright sorts every test deterministically, divides the sorted list into total contiguous slices, and each runner executes only the slice named by its index. Because the partition is disjoint and complete, running all shards is equivalent to running the whole suite — just spread across machines.

Two properties matter for correctness. First, the split is deterministic: the same suite and the same total always produce the same partition, so a rerun of shard 3/4 runs exactly the same tests. Second, the split is by test count, not by duration — Playwright does not know in advance how long each spec takes. That single fact is the source of most of the tuning work later: an uneven distribution of slow specs leaves one shard running long after the others have finished, and the pipeline is only as fast as its slowest shard.

Step-by-step implementation #

Step 1 — Split the suite with --shard #

Prove the partition works locally before involving CI. Running two complementary shards should, together, execute every test exactly once.

# Each command runs a disjoint quarter of the sorted suite.
npx playwright test --shard=1/4
npx playwright test --shard=2/4
npx playwright test --shard=3/4
npx playwright test --shard=4/4

# Sanity: shard 1/2 and shard 2/2 together must equal the full run's test count.
npx playwright test --shard=1/2 --reporter=line | tail -1
npx playwright test --shard=2/2 --reporter=line | tail -1

Verify: Add the reported test counts from --shard=1/2 and --shard=2/2. Their sum must equal the count from an unsharded npx playwright test run — if it does not, a testDir or grep filter is being applied inconsistently across shards.

Step 2 — Fan out over a GitHub Actions matrix #

Move the split into CI by mapping shard indices onto a matrix. GitHub launches one runner per matrix value, all in parallel, from a single job definition.

# .github/workflows/visual-sharded.yml
name: Visual Regression (sharded)
on: [pull_request]

jobs:
  test:
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false           # let every shard finish so you see all failures
      matrix:
        shard: [1, 2, 3, 4]      # four parallel runners
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: '20'
          cache: 'npm'
      - run: npm ci
      - run: npx playwright install --with-deps chromium
      - name: Run shard ${{ matrix.shard }}
        run: npx playwright test --shard=${{ matrix.shard }}/4

fail-fast: false is deliberate: the default would cancel the other shards the moment one fails, hiding failures that live in the cancelled slices and forcing a second run to see the full picture.

Verify: Open the workflow run and confirm four test jobs appear and start within seconds of each other. Total wall-clock time should drop toward one-quarter of the single-runner duration, minus fixed setup overhead.

Step 3 — Emit a blob report per shard #

An HTML report per shard is useless for a combined verdict. The blob reporter writes a machine-readable partial that can be merged later, so configure it and upload each shard’s blob as an artifact.

// playwright.config.ts — use blob on CI, human-readable locally
import { defineConfig } from '@playwright/test';

export default defineConfig({
  // 'blob' emits a zip that merge-reports can recombine; 'list' for local dev.
  reporter: process.env.CI ? 'blob' : 'list',
});

      # Append to the sharded job's steps
      - name: Upload blob report
        if: ${{ !cancelled() }}         # keep blobs even when the shard failed
        uses: actions/upload-artifact@v4
        with:
          name: blob-report-${{ matrix.shard }}
          path: blob-report/
          retention-days: 7

Verify: After the run, the workflow’s artifacts list should contain blob-report-1 through blob-report-4, each a non-empty zip. A failed shard must still upload its blob — if: ${{ !cancelled() }} guarantees it.

Step 4 — Merge shard reports into one verdict #

Add a dependent job that runs only after all shards finish. It downloads every blob and calls merge-reports to produce a single HTML report and, crucially, a single exit code the gate can read.

  merge:
    if: ${{ !cancelled() }}
    needs: [test]                # waits for every matrix shard
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: '20'
      - run: npm ci
      - name: Download all blob reports
        uses: actions/download-artifact@v4
        with:
          path: all-blobs
          pattern: blob-report-*
          merge-multiple: true
      - name: Merge into one HTML report
        run: npx playwright merge-reports --reporter=html ./all-blobs
      - uses: actions/upload-artifact@v4
        with:
          name: html-report
          path: playwright-report/

Register this merge job — not the individual shards — as the required status check so a failure in any shard propagates into the merged result and blocks the merge.

Verify: Force a failure in a test that lands in shard 3/4. The merge job must go red and its html-report artifact must show that failing test, proving the combined result reflects every shard.

Step 5 — Balance shards to cut wall-clock time #

Because Playwright splits by test count, one shard often ends up with the heavy specs and runs long. Measure per-shard duration, then rebalance so the slowest shard sits close to the average — that number is your true pipeline time.

# Read each shard's duration from its blob report's timing metadata.
# A quick approximation: sum test durations per shard from the merged JSON.
npx playwright merge-reports --reporter=json ./all-blobs > merged.json
node -e '
  const r = require("./merged.json");
  const byShard = {};
  for (const s of r.suites.flatMap(x => x.specs))
    for (const t of s.tests)
      byShard[t.annotations?.shard ?? "?"] =
        (byShard[t.annotations?.shard ?? "?"] ?? 0) +
        t.results.reduce((a, x) => a + x.duration, 0);
  console.table(byShard);
'

// playwright.config.ts — push the slowest suites into their own project/tag
// so you can size shards around them, and cap workers to avoid CPU contention.
export default defineConfig({
  workers: process.env.CI ? 2 : undefined,   // 2 workers per shard on a 2-vCPU runner
  fullyParallel: true,                        // parallelize within files, not just across them
});

When one spec dominates (a 200-screenshot storybook page), split it into smaller files or move it to its own shard. The detailed rebalancing workflow — including sizing shard count against runner cost — is covered in balancing test shards to reduce CI wall-clock.

Verify: Compare per-shard durations before and after rebalancing. The spread between the fastest and slowest shard should shrink; the slowest shard’s time — the number that gates the pipeline — should drop measurably.

Configuration reference #

Common pitfalls #

1. Gating on individual shards instead of the merged result. If each shard is registered as its own required check, a rerun that re-greens one shard can let a merge through while another shard is still red on a stale run. Register only the merge job as required, so the gate always reflects the whole suite. This is the parallel-testing corollary to the rules in pipeline gating thresholds.

2. Leaving fail-fast at its default. The default cancels sibling shards as soon as one fails. You then see a subset of failures, fix them, rerun, and discover more — a slow, demoralizing loop. Set fail-fast: false on the test matrix so every shard reports its own failures in one pass.

3. Cross-shard state collisions. Tests that assumed sequential execution break when spread across machines: a hard-coded port, a shared database row, or a snapshot written to a common path. Because shards run on isolated runners they must not depend on each other or on shared external state. Give each shard its own fixtures, its own database namespace, and its own output directory. This is the sharded form of the isolation discipline in component testing fundamentals.

4. Over-sharding until setup overhead dominates. Every shard re-runs npm ci and playwright install, which can be a minute or more of fixed cost. Split a 3-minute suite into 12 shards and you spend more time installing than testing, and you burn 12× the runner minutes for a marginal wall-clock gain. Size the shard count so per-shard test time stays comfortably above setup time.

5. Merging without keeping blobs from failed shards. If the artifact upload is guarded by if: success(), a failed shard uploads nothing and merge-reports silently produces a report missing that shard’s results — which can look green. Use if: ${{ !cancelled() }} so failing shards still contribute their blob to the merge.

Integration point #

Sharding is a throughput optimization layered on top of the same pipeline everything else in this guide describes. It runs the jobs defined in GitHub Actions workflows, splitting each across runners, and it feeds a single merged result into the pipeline gating thresholds that decide whether the merge is allowed — gate on the merged report, never on a lone shard. The per-engine tolerance thresholds and cross-browser matrix still apply within each shard, and the interaction tests and accessibility checks that make up the suite are the same tests, merely distributed.

Two focused walkthroughs continue from here: sharding Playwright visual tests across CI runners works a full GitHub Actions example end to end, and balancing test shards to reduce CI wall-clock covers measuring per-shard duration and redistributing slow specs.

FAQ #

What is the difference between parallelization and sharding? #

Parallelization runs multiple tests at once inside a single machine using worker processes. Sharding splits the whole test suite across multiple machines, each running a disjoint slice. They compose: each shard is one machine that itself runs several parallel workers, so a 4-shard by 4-worker setup executes up to 16 tests simultaneously.

How does Playwright’s --shard flag decide which tests run where? #

--shard=index/total sorts all tests deterministically and hands each shard a contiguous, disjoint slice. With --shard=1/4 the first runner takes roughly the first quarter of the sorted tests. The split is by test count, not by duration, which is why a slow spec can leave one shard running much longer than the others.

How do I combine results from multiple shards into one report? #

Have each shard emit a blob report, upload it as an artifact, then run a dependent job that downloads all blobs and calls npx playwright merge-reports to produce one HTML report. Gate the merge on the combined result so a failure in any shard fails the overall check.

Why do tests pass in a single runner but fail when sharded? #

Cross-shard flakiness usually comes from shared mutable state: a fixed port, a common database row, a snapshot output path, or a global fixture written to disk. Because shards run on separate machines they should be fully isolated, but tests that assumed sequential execution or a shared resource break when that assumption is removed.

Related #

• CI/CD Test Gating & Pipeline Integration — the parent guide covering how component and visual tests plug into a pipeline
• Sharding Playwright Visual Tests Across CI Runners — a full GitHub Actions sharding example worked end to end
• Balancing Test Shards to Reduce CI Wall-Clock — measuring per-shard duration and redistributing slow specs
• GitHub Actions Workflows — defining the jobs that sharding fans out across runners
• Pipeline Gating Thresholds & Required Checks — gating on the merged shard result rather than a single runner
• Cross-Browser Matrix — the engine dimension you shard alongside test slices