Gating visual regression tests in GitHub Actions

This page is part of the GitHub Actions Workflows section of the CI/CD test gating guide, which covers wiring component and visual tests into a pipeline so they actually block bad merges instead of decorating them.

The specific problem: a pull request changed a button’s padding, the visual regression job noticed, and the PR merged anyway with a green tick. The job ran, the diff was real, and nobody was stopped. That is a gating failure, not a testing failure — the test worked; the pipeline ignored it.

Problem statement

You added Playwright visual comparisons (or Chromatic) to catch unintended UI changes. A screenshot diff fires on a PR, but the workflow reports success and the merge button stays enabled. Later, the regression ships and someone asks why the visual test “didn’t catch it” — when in fact it did, and the result was discarded.

There are only two places this breaks. Either the visual command’s non-zero exit code is being swallowed inside the job (continue-on-error, || true, a trailing exit 0, or Chromatic’s exitZeroOnChanges), or the job exits red correctly but its check was never added to branch protection, so GitHub treats it as advisory. Both produce the same symptom: a red-worthy diff that merges green.

Root cause

A GitHub Actions job blocks a merge only when two conditions hold together — the job must exit non-zero, and its check name must be listed under Require status checks to pass in branch protection. Most “it merged anyway” incidents come from an exit-zero mask on the visual step (so the job is green despite a real diff) combined with a check that was never marked required, which is why making visual review a required status check is the other half of the fix described in the pipeline gating thresholds section. Fix one without the other and the gate still leaks.

Leaking gate versus blocking gate in GitHub Actions Top row: a pixel diff detected, then a step with "|| true / continue-on-error", producing a green job that is not a required check, ending in a merged pull request. Bottom row: the same pixel diff, a step with no exit-zero mask, producing a red job that is a required check, ending in a blocked merge button. Leaking gate pixel diff detected || true / continue-on-error green job, not required PR merged regression ships Blocking gate pixel diff detected no exit-zero mask red job, required check merge blocked

Minimal reproduction

The shortest workflow that reproduces the leak: a Playwright visual job whose failure is masked with continue-on-error, so the job — and therefore the whole workflow — reports success even when a snapshot differs.

# .github/workflows/visual.yml — BROKEN: masks the failure
name: Visual
on: pull_request

jobs:
  visual:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 20
      - run: npm ci
      - run: npx playwright install --with-deps chromium
      - name: Visual regression
        run: npx playwright test --grep @visual
        continue-on-error: true          # ← swallows the non-zero exit
// tests/button.visual.spec.ts — a real diff that should block
import { test, expect } from '@playwright/test';

test('primary button @visual', async ({ page }) => {
  await page.goto('/iframe.html?id=button--primary');
  // Snapshot baseline has 12px padding; the PR shipped 16px.
  await expect(page.locator('#storybook-root')).toHaveScreenshot('button-primary.png', {
    maxDiffPixelRatio: 0.01,
  });
});

Playwright exits 1 and prints the diff, but continue-on-error: true converts the step to a success. The visual job is green, the Visual check is green, and if that check was never required, the PR merges with the regression baked in.

Step-by-step fix

1. Remove every exit-zero mask from the visual step

Delete continue-on-error, any || true, and any trailing exit 0. The visual command must be allowed to fail the step so its exit code propagates to the job conclusion.

      - name: Visual regression
        run: npx playwright test --grep @visual   # no continue-on-error, no || true

What this does: when npx playwright test exits 1, the step fails, the job fails, and the workflow’s Visual check turns red. That red check is the raw material branch protection needs in order to block anything.

2. Upload diff artifacts so the failure is reviewable

A red check with no evidence forces the author to reproduce the diff locally. Publish Playwright’s report and the raw test-results directory (which holds the actual, expected, and diff PNGs) with if: always() so the upload runs even though the previous step failed.

      - name: Visual regression
        run: npx playwright test --grep @visual

      - name: Upload diff report
        if: always()                     # runs even when the visual step failed
        uses: actions/upload-artifact@v4
        with:
          name: playwright-visual-report
          path: |
            playwright-report/
            test-results/
          retention-days: 14

What this does: if: always() decouples the upload from the job status, so the artifact attaches to failed runs — exactly when reviewers need it. The zip appears in the run summary’s Artifacts section; opening test-results/**/diff.png shows precisely which pixels moved.

3. Configure Playwright so the assertion is deterministic

A gate is only useful if it fails on real changes and not on rendering noise. Pin the snapshot comparison and disable animations so the diff reflects the component, not the environment. Threshold selection is covered in depth under tolerance thresholds.

// playwright.config.ts
import { defineConfig } from '@playwright/test';

export default defineConfig({
  expect: {
    toHaveScreenshot: {
      maxDiffPixelRatio: 0.01,   // ≤1% of pixels may differ before failing
      animations: 'disabled',    // freeze CSS animations for a stable capture
    },
  },
  use: { baseURL: 'http://127.0.0.1:6006' },   // served Storybook
});

What this does: a fixed maxDiffPixelRatio and disabled animations make the pass/fail boundary reproducible in CI, so the red check reflects an intentional signal rather than a flaky one that reviewers learn to ignore.

4. Make the visual job a required status check

A red check still does not block a merge until branch protection requires it. Add the exact check name to the rule for main. You can do this in the UI (Settings → Branches → Branch protection rules) or with the API:

# Require the "visual" job's check before merging into main
gh api -X PUT repos/acme/design-system/branches/main/protection \
  --input - <<'JSON'
{
  "required_status_checks": {
    "strict": true,
    "checks": [{ "context": "visual" }]
  },
  "enforce_admins": true,
  "required_pull_request_reviews": { "required_approving_review_count": 1 },
  "restrictions": null
}
JSON

What this does: checks[].context must match the job’s check name as GitHub displays it (the job id visual, or Visual / visual when the workflow has a display name). Once listed, GitHub disables the merge button while that check is failing or pending. This is the required-check half described in making visual review a required status check.

5. For Chromatic, drop exitZeroOnChanges and require its own check

If you gate with Chromatic instead of Playwright, the equivalent mask is the exitZeroOnChanges flag. Remove it so the CLI exits non-zero on unreviewed changes, and rely on Chromatic’s UI Tests check for human approval.

      - name: Chromatic
        uses: chromaui/action@latest
        with:
          projectToken: ${{ secrets.CHROMATIC_PROJECT_TOKEN }}
          # exitZeroOnChanges: true   ← removed; changes now fail the job
          onlyChanged: true           # TurboSnap: test only affected stories

What this does: without exitZeroOnChanges, a detected visual change fails the workflow step. Adding Chromatic’s UI Tests context to branch protection then blocks the merge until a reviewer accepts or rejects each change in the Chromatic UI.

Verification

Open a PR that deliberately shifts one pixel — for example the padding change in the reproduction — and watch the workflow fail:

gh pr checks 218

Expected output once the gate is live:

NAME                        STATUS      CONCLUSION
Visual / visual             completed   failure
Upload diff report          completed   success

X 1 check failed

The failing job’s log names the mismatch:

  1) button.visual.spec.ts:4:5 › primary button @visual ────────────────────

    Error: Screenshot comparison failed:
      2143 pixels (ratio 0.03 of all image pixels) are different.

    Expected: button-primary-expected.png
    Received: button-primary-actual.png
    Diff:     button-primary-diff.png

Confirm the gate holds end to end:

gh pr view 218 --json mergeable,mergeStateStatus
{
  "mergeable": "MERGEABLE",
  "mergeStateStatus": "BLOCKED"
}

mergeStateStatus: BLOCKED with the merge button greyed out in the UI is the signal that the required check is doing its job. Download playwright-visual-report from the run’s Artifacts panel to review diff.png before deciding whether to update the baseline or fix the component.

Edge cases and caveats

  • Renaming a job silently drops the requirement. Branch protection matches by check name string. If you rename the job from visual to visual-tests, the old required context no longer appears, GitHub sees the required check as never-reported, and — depending on the “strict” setting — PRs may either block forever or slip through. Update the protection rule in the same PR that renames the job.
  • if: always() only belongs on reporting steps. Putting if: always() on the visual test step itself would run it regardless of earlier setup failures, but it will not mask its own exit code. The mask people accidentally add is continue-on-error; keep that off the test step and reserve if: always() for the artifact upload alone.
  • Forked-PR runs cannot read secrets. A Chromatic projectToken stored as a secret is unavailable to workflows triggered from forks, so the job errors instead of diffing. Use pull_request_target with an explicit checkout of the PR head, or restrict visual gating to same-repo branches, so external contributors do not see a spurious required-check failure.

FAQ

Why did a failing visual test still let the pull request merge?

Two independent things have to be true to block a merge: the job must exit non-zero, and that job’s check must be listed as required in branch protection. A continue-on-error: true, a || true, or a check that was never marked required will each let a red visual diff merge green. Remove the exit-zero mask and add the check to the required list.

How do I let reviewers see the visual diff images from a failed CI run?

Add an actions/upload-artifact step with if: always() that publishes the playwright-report and test-results directories. Playwright writes actual, expected, and diff PNGs into test-results on failure; the uploaded artifact appears at the bottom of the workflow run summary as a downloadable zip.

Should the Chromatic step use exitZeroOnChanges in a gating pipeline?

No. exitZeroOnChanges makes Chromatic return 0 even when it detects visual changes, which is exactly the mask that lets regressions merge. Drop the flag so the CLI exits non-zero on unreviewed changes, and rely on Chromatic’s own UI Tests required check to gate the merge on human approval.