Posting visual diff results as a pull request comment

This page is part of the artifacts and pipeline reporting section of the CI/CD test gating guide, which covers what happens after a run finishes.

The specific problem: the visual check goes red, and the pull request shows only a failed check name. The reviewer has to open the run, find the log, identify the failing stories and download an artefact before they can judge whether anything is actually wrong.

Problem statement

Visual review is the one gate whose output a human must interpret. A pixel difference is not a defect on its own — it is a change, and only a person can say whether the change was intended.

A red check that does not say what changed therefore stalls the review. The path of least resistance for the author is to rerun the job; the path of least resistance for the reviewer is to approve on the strength of the code diff and ignore the visual one.

Root cause

The information exists — the runner knows the story ids and the diff percentages, and the artefact holds the images — and none of it is where the decision is being made. The pull request shows a check name and a colour.

Bridging that gap is a small amount of work with a large effect, and it has one important constraint: the bridge must be a single comment, updated in place. A bot commenting per push turns the conversation into a log.

Why the comment must be updated, not repeated Top lane: the bot comments on push one, two and three, and the human review conversation is buried under bot output. Bottom lane: the bot creates a comment on the first push and updates it on each subsequent one, so the pull request always shows one current result. One comment per push — the thread becomes unreadable push 1 comment push 2 comment push 3 comment review buried One comment, updated in place push 1 create push 2 update push 3 update always current

Minimal reproduction 

      - run: npx playwright test --grep @visual
      # red check, no explanation, artefact three clicks away

Step-by-step fix

1. Emit a machine-readable result 

// playwright.config.ts
export default defineConfig({
  reporter: [['list'], ['json', { outputFile: 'reports/visual.json' }]],
});

What this does: gives the comment step a structured input, so it can name the stories and their percentages rather than pasting a log.

2. Build the comment body from the result 

// scripts/visual-comment.mjs
import { readFileSync } from 'node:fs';

const report = JSON.parse(readFileSync('reports/visual.json', 'utf8'));
const failed = report.suites
  .flatMap((s) => s.specs ?? [])
  .filter((spec) => !spec.ok)
  .map((spec) => ({ title: spec.title, project: spec.tests?.[0]?.projectName ?? 'chromium' }));

const marker = '<!-- visual-diff-report -->';
const body = failed.length === 0
  ? `${marker}\nAll stories match their baselines.`
  : [
      marker,
      `**${failed.length} story/stories differ from their baselines.**`,
      '',
      ...failed.map((f) => `- \`${f.title}\` — ${f.project}`),
      '',
      `[Review the diff images](${process.env.REPORT_URL})`,
      '',
      'If the change is intended, update only those baselines:',
      '```bash',
      `npx playwright test --grep "${failed[0].title}" --update-snapshots`,
      '```',
      '',
      `_commit ${process.env.GITHUB_SHA?.slice(0, 7)}_`,
    ].join('\n');

console.log(body);

What this does: produces a comment that answers what changed, where to look, and what to do next.

3. Update one comment rather than adding another 

      - name: Comment visual results
        if: always() && github.event_name == 'pull_request'
        uses: actions/github-script@v7
        env:
          REPORT_URL: ${{ steps.publish.outputs.url }}
        with:
          script: |
            const { execSync } = require('node:child_process');
            const body = execSync('node scripts/visual-comment.mjs', { encoding: 'utf8' });
            const marker = '<!-- visual-diff-report -->';

            const { data: comments } = await github.rest.issues.listComments({
              ...context.repo, issue_number: context.issue.number, per_page: 100,
            });
            const mine = comments.find((c) => c.body.includes(marker));

            if (mine) await github.rest.issues.updateComment({ ...context.repo, comment_id: mine.id, body });
            else await github.rest.issues.createComment({ ...context.repo, issue_number: context.issue.number, body });

What this does: the hidden marker makes the bot’s own comment findable, so subsequent pushes edit it rather than appending.

4. Comment on success too, briefly

A comment that only appears on failure leaves the reviewer unsure whether the check ran. One line — “all stories match their baselines” — closes that gap and costs nothing.

5. Skip forks 

        if: github.event.pull_request.head.repo.full_name == github.repository

What this does: a fork’s token cannot write comments, so the step fails noisily for external contributors unless it is skipped.

Verification

What a useful visual comment contains A pull-request comment for number 418. Two stories differ from their baselines: Card WithAvatar at 1.84 percent on chromium, and Table Dense at 0.91 percent on chromium and firefox. The comment links to the report, gives the exact command to update those baselines if the change was intended, and states which commit it describes and when it was updated. Visual tests — PR #418 2 stories differ from their baselines Card/WithAvatar 1.84% chromium Table/Dense 0.91% chromium, firefox Review the diffs: <report url> If intended, run: npm run test:visual -- -u -g 'Card' updated 3 minutes ago for commit a3f19c2

Push twice to the same branch. The pull request should show exactly one visual comment, describing the second commit. Two comments means the marker is not matching — usually because the body was reformatted and the marker moved or was dropped.

Edge cases and caveats

Six things the comment has to say Six rows. The failing story ids matter because the same string is searchable in Storybook, the report and the baseline directory. The diff percentage distinguishes noise from a real change. Naming which engines differ matters because a single-engine difference points at a rendering bug rather than a code change. A link to the images is the only way to judge whether a change was intended. Including the update command removes the reviewer's next question. The commit sha proves the comment is not describing an older push. Comment contains Why it matters The failing story ids searchable in three tools The diff percentage distinguishes noise from change Which engines differ one engine means a rendering bug A link to the images the only way to judge intent The update command removes the next question The commit sha proves it is not stale

  • Comment length. A run where a token change moved two hundred baselines produces an unusable comment. Cap the list at ten and link to the report for the rest.

  • Permissions. The workflow needs permissions: pull-requests: write. The default token is read-only in many repositories, and the failure message is unhelpfully generic.

  • Notification noise. Editing a comment does not notify, which is exactly why updating in place is better than posting. Deleting and reposting reintroduces the notification and defeats the purpose.

Writing for the person who has to decide

The comment’s job is not to report a result — the check already did that. Its job is to move a reviewer from “something changed” to a decision, and the wording matters more than the plumbing.

Say what changed, not that something failed. “2 stories differ from their baselines” is a statement of fact a reviewer can act on. “Visual tests failed” invites the reading that the tooling is broken, which is how these checks acquire a reputation for flakiness they may not deserve.

Give the percentage next to the name. A story at 0.9% and one at 14% need different amounts of attention, and putting both numbers in front of the reviewer lets them triage before opening anything.

Name the engines. A story differing on one engine and not the others is a rendering difference, not a code change, and that distinction changes who should look at it. Making it visible in the comment saves the reviewer from deducing it from four separate check names.

Include the command, exactly. The next question after “yes, that was intended” is always “how do I update just that one”. Answering it pre-emptively, with a copyable command scoped to the affected stories, is what prevents someone reaching for an unscoped update and rewriting every baseline in the repository.

Timestamp it against a commit. A comment that does not say which commit it describes is untrustworthy after the next push, and a reviewer who has once been misled by a stale comment will stop reading them.

FAQ

Should the comment include the diff images inline?

Only if they are already hosted somewhere the pull request can reach, in which case a couple of thumbnails are genuinely useful. Do not upload images to the pull request itself: a comment carrying a dozen screenshots is unreadable, and a token change producing two hundred would be worse. A link to the report scales; embedded images do not.

What should the comment say when the baselines are missing entirely?

That a baseline does not exist, explicitly, rather than reporting a difference. A missing baseline and a changed baseline are different situations — the first needs someone to generate and review a new reference, the second needs someone to judge a change — and conflating them is how an unreviewed baseline gets created on a gated branch.

Is a comment better than a check annotation?

They complement each other. An annotation attaches to the file and line, which suits a lint error and not a visual diff, since the relevant artefact is an image rather than a line of code. A comment carries links and commands, which is what visual review actually needs. Where a platform supports both, the check summary is the better home for counts and the comment for the actionable detail.

How should the comment behave when the run is cancelled?

It should not update at all. A cancelled run has partial results, and overwriting a complete comment with them replaces good information with misleading information. Conditioning the step on success() || failure() rather than always() covers the two states that produced a real result and skips cancellations entirely.