Setting failure thresholds for visual diffs in CI

This page is part of the Pipeline Gating Thresholds & Required Checks section of the CI/CD test gating guide, which covers how to turn a passing test suite into a merge gate that actually blocks regressions. Here the specific problem is calibration: a visual diff gate that is too strict fails on every anti-aliased pixel, and one that is too loose waves through a genuinely broken button.

Problem statement

A visual regression job has exactly one useful output: a boolean that says “this render changed in a way a human should look at.” In practice most freshly-added visual gates produce a different, useless output — a boolean that flips based on the GPU that happened to render the screenshot.

Two failure modes dominate:

  • Too strict. The default of “any pixel differs → fail” means a font hinting change, a sub-pixel anti-aliasing difference on a rounded corner, or a one-pixel shift in a box-shadow fails the build. Developers learn that the visual job is noise and start re-running it until it goes green, which trains the whole team to ignore it.
  • Too loose. After enough noise, someone sets a large tolerance — “allow 5% of pixels to differ” — to make the job stop crying wolf. Now a button that lost its background color, or a heading that dropped two font sizes, sails through because the changed region is under 5% of the screenshot.

Both failure modes end at the same place: the gate is not trusted, so it is not enforced, so real regressions merge.

Root cause

The default comparison is a strict pixel-for-pixel equality check, but two runs of the same UI are almost never pixel-identical. Anti-aliasing, font rasterization, and GPU compositing differ across machines, so an un-tuned gate measures the rendering environment rather than the component. The fix lives in tolerance thresholds: a per-pixel color tolerance to absorb sub-perceptual noise, plus a whole-image pixel budget so that a large enough real change still trips the gate. Without both numbers set deliberately — and tiered by how much noise each component actually produces — the CI job’s exit code is not a signal you can gate a merge on.

Untuned versus tiered visual diff thresholds Three horizontal bars representing accumulated pixel difference. The top bar is a strict gate whose fail line sits at zero, so anti-alias noise fails it. The middle bar is a loose gate whose fail line sits far right, so a real regression passes under it. The bottom bar is a tiered gate whose fail line sits just above the noise band and below the regression band. Strict (default): fails on noise anti-alias noise fail line = 0 Loose (5%): real regression passes lost button color fail line = 5% Tiered: noise absorbed, regression caught fail line just above noise

Minimal reproduction

The smallest way to see the noise problem is to compare the same UI against itself. Take one component test and run it twice on two different machines — a local Mac and a Linux CI runner is the classic pairing — with a strict, un-tuned config:

// playwright.config.ts — no threshold set → strict pixel equality
import { defineConfig } from '@playwright/test';

export default defineConfig({
  // expect block omitted entirely → maxDiffPixels defaults to 0
  testDir: './tests/visual',
});
// tests/visual/button.spec.ts
import { test, expect } from '@playwright/test';

test('primary button', async ({ page }) => {
  await page.goto('/iframe.html?id=button--primary');
  await expect(page.locator('#storybook-root')).toHaveScreenshot('button-primary.png');
});

The baseline was captured on macOS; CI runs on ubuntu-latest. The job fails with a diff of a few dozen pixels along the button’s rounded border — pixels that no human would notice, produced entirely by different font and edge rendering:

Error: Screenshot comparison failed:
  245 pixels (ratio 0.002 of all image pixels) are different.

Nothing about the button changed. The 245 differing pixels are the CI environment, not a regression. With the strict default, every push fails.

Step-by-step fix

1. Set a per-pixel threshold and a whole-image budget

Configure the two independent knobs. threshold (0–1) is the per-pixel color-distance tolerance in the YIQ color space — how different one pixel must be before it counts as changed at all. maxDiffPixelRatio is the fraction of the total image that is allowed to differ before the test fails.

// playwright.config.ts
import { defineConfig } from '@playwright/test';

export default defineConfig({
  testDir: './tests/visual',
  expect: {
    toHaveScreenshot: {
      threshold: 0.2,            // per-pixel: absorb anti-aliasing / font hinting
      maxDiffPixelRatio: 0.001,  // whole-image: fail if >0.1% of pixels differ
      animations: 'disabled',    // freeze CSS animations before capture
      caret: 'hide',             // hide the text caret so it never diffs
    },
  },
});

What this does: threshold: 0.2 means a pixel is only “different” if its color moved meaningfully, which silences sub-perceptual edge noise. maxDiffPixelRatio: 0.001 then fails the test only when more than 0.1% of the image is genuinely different — enough to catch a moved element or a color change, tight enough to reject the 245-pixel rounding noise from the reproduction (0.002 > 0.001 would still fail here, so measure and set the budget just above your observed noise floor — see step 3).

2. Tier the thresholds per component

One global number cannot serve both a monochrome icon and a component with an animated gradient. Override per test so strict components stay strict.

// tests/visual/icon.spec.ts — strict tier: a shifted pixel is a real bug
import { test, expect } from '@playwright/test';

test('close icon is pixel-exact', async ({ page }) => {
  await page.goto('/iframe.html?id=icon--close');
  await expect(page.locator('#storybook-root')).toHaveScreenshot('icon-close.png', {
    threshold: 0,
    maxDiffPixelRatio: 0,       // icons must be identical
  });
});
// tests/visual/hero.spec.ts — loose tier: gradient + shadow are inherently noisy
import { test, expect } from '@playwright/test';

test('marketing hero', async ({ page }) => {
  await page.goto('/iframe.html?id=hero--gradient');
  await expect(page.locator('#storybook-root')).toHaveScreenshot('hero.png', {
    threshold: 0.3,
    maxDiffPixelRatio: 0.01,    // 1% budget for gradient banding
  });
});

What this does: the icon test rejects any change at all, while the gradient hero tolerates the banding that GPU compositing produces. Each component is gated at the noise level it actually generates rather than one compromise number that is wrong for both.

3. Measure the noise floor instead of guessing

Do not pick maxDiffPixelRatio by feel. Run the suite twice against an unchanged commit on the CI runner and read the largest no-change diff out of the report, then set the budget just above it.

# Capture baselines, then immediately re-run against the same code
npx playwright test --update-snapshots tests/visual
npx playwright test tests/visual --reporter=json > diff-report.json

# Largest differing-pixel ratio across a clean re-run = your noise floor
node -e "const r=require('./diff-report.json'); \
  const ratios=JSON.stringify(r).match(/ratio [0-9.]+/g)||['ratio 0']; \
  console.log('max no-change ratio:', ratios.sort().pop())"

What this does: it turns threshold selection into a measurement. If the noisiest clean re-run differs by 0.0004 of the image, a maxDiffPixelRatio of 0.0008 gives you headroom over the noise while still failing on anything structural.

4. Wire the runner’s exit code straight to the gate

The threshold work is wasted if the job cannot fail. The test runner already exits non-zero when a comparison exceeds the budget — the CI step must let that exit code propagate untouched.

# .github/workflows/visual.yml
name: Visual regression
on: [pull_request]

jobs:
  visual:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with: { node-version: 20, cache: npm }
      - run: npm ci
      - run: npx playwright install --with-deps chromium
      # No continue-on-error, no `|| true`, no trailing echo:
      # a non-zero exit from Playwright fails the job.
      - run: npx playwright test tests/visual
      - uses: actions/upload-artifact@v4
        if: ${{ !cancelled() }}   # upload the diff report even on failure
        with:
          name: playwright-report
          path: playwright-report/

What this does: the playwright test step is the gate. There is no continue-on-error: true masking the result and no trailing command to reset $?, so the job’s status equals the suite’s result. The if: ${{ !cancelled() }} on the upload runs even when the tests fail, so reviewers get the diff images. To make this job actually block merges, add it as a required check — covered in making visual review a required status check.

Verification

Prove the gate discriminates by feeding it one non-regression and one regression.

First, a one-pixel nudge that should pass. Add padding-top: 1px to a noisy component and run:

npx playwright test tests/visual/hero.spec.ts
Running 1 test using 1 worker
  ✓  tests/visual/hero.spec.ts:4:1 › marketing hero (1.2s)

  1 passed (1.4s)

The shift stayed under the tiered budget, so the job is green — no false alarm. Now a real regression: change the primary button’s background to transparent and run the strict icon-adjacent test:

npx playwright test tests/visual/button.spec.ts
  ✘  tests/visual/button.spec.ts:4:1 › primary button (0.9s)

  Error: Screenshot comparison failed:
    18432 pixels (ratio 0.153 of all image pixels) are different.
    Expected: button-primary.png
    Received: button-primary-actual.png
      Diff:  button-primary-diff.png
  1 failed

The exit code is non-zero, the job fails, and the diff artifact shows the missing background. A gate that passes the 1px nudge and fails the missing background is correctly calibrated.

Edge cases and caveats

  • threshold and maxDiffPixelRatio interact — do not tune them independently. Raising threshold shrinks the count of “different” pixels, which effectively loosens maxDiffPixelRatio too. If you push threshold to 0.5 to kill noise, a genuine low-contrast change (grey text going slightly lighter) may never register a single differing pixel. Prefer a low threshold plus environment stabilization (disabled animations, hidden caret, pinned fonts) over a high threshold.
  • A percentage budget hides small-but-total regressions on large screenshots. maxDiffPixelRatio is relative to the whole image, so a 32×32 icon that turns completely wrong inside a 1920×1080 full-page screenshot is a rounding error against the budget. Screenshot the tightest element locator, not the full page, so the changed region is a large fraction of the captured area — or gate icons with maxDiffPixels (an absolute count) instead of a ratio.
  • Font rendering can exceed any sane threshold across OSes. If your baselines are captured on macOS and CI runs on Linux, glyph rasterization differs enough that no reasonable threshold fully stabilizes text. Capture baselines on the same platform CI uses — run the update inside the CI container — or handle it as described in handling font rendering differences across browsers.

FAQ

What is a good maxDiffPixelRatio for component visual tests?

Start at 0.001 (0.1% of the rendered area) for most components and tighten to 0 for icons, logos, and typography where a single shifted pixel is a real defect. Loosen to around 0.01 only for components with gradients, shadows, or sub-pixel animation that no clamping fully stabilizes. Tune from measured noise, not guesses: run the same commit twice and set the ratio just above the largest observed no-change diff.

Should I use threshold or maxDiffPixelRatio in Playwright?

Use both — they control different things. threshold (0 to 1) is the per-pixel color-distance tolerance that decides whether an individual pixel counts as different, which is what absorbs anti-aliasing. maxDiffPixelRatio is the whole-image budget that decides whether the accumulated differing pixels fail the test. A low threshold with a small maxDiffPixelRatio catches structural changes while ignoring rendering noise.

Why does my CI job pass when the visual test clearly failed?

The test’s non-zero exit code is being swallowed before it reaches the job’s status. The usual causes are continue-on-error: true on the step, a trailing command such as echo done that resets the exit code, or piping the runner through tee without pipefail. Let the runner exit code propagate unmodified so the gate reflects the real result.