Artifacts and pipeline reporting

This section sits under CI/CD test gating and covers what happens after a test run finishes. A gate that reports only pass or fail is only half a gate: when a visual regression check goes red, the developer needs the diff image, and when a story times out they need the trace. Publishing that evidence is what turns a red check from an obstacle into a diagnosis.

Prerequisites

What to publish, and for whom

Every file a run produces has exactly one audience, and confusing them is what produces pipelines that upload hundreds of megabytes nobody opens while the one useful image is missing.

Machines that consume immediately — a coverage merger, a threshold checker — need nothing uploaded. The file is read inside the job and discarded.

Machines that consume later — the merge step of a sharded run — need a short-lived artefact. A blob report is meaningless to a human and essential to the merge, so it is uploaded with a one-day retention and never linked anywhere.

Humans debugging a failure need the diff image, the HTML report and the trace, and they need them only when something failed. Conditioning the upload on failure keeps storage proportional to problems.

Humans reading the result need numbers and links, not files. A job summary showing “47 passed, 1 failed” with a link to the failing story is read by everyone; an artifact zip is downloaded by almost nobody.

Deciding what to publish A decision node asking who needs the file. Something a machine consumes immediately is used in the job and never uploaded. Something a machine needs later, such as a blob report, is uploaded with short retention. Something a human needs on failure, such as a diff or a trace, is uploaded conditionally on failure. Something a human always wants — counts and links — belongs in the job summary. Anything nobody needs yet should not be uploaded, because storage is not free. Who needs this file? Consume in-job no upload needed a machine, immediately Upload, short retention blob reports a machine, later Upload if: failure() diffs, traces a human, on failure Job summary counts and links a human, always Do not upload storage is not free nobody, yet What happens to a report after the run Five stages left to right. The test run emits files. The highlighted collect stage gathers the JUnit XML, diff images and coverage data into one directory. The upload stage publishes it with an always condition so a red run keeps its evidence. A summarise stage writes a job summary and a pull-request comment. Finally a retention policy bounds how long it is kept. A failure nobody can inspect is a failure nobody fixes test run emits files collect junit, diffs, coverage upload if: always() summarise job summary + PR comment retain 7–30 days

Step-by-step implementation

Step 1 — Make the runners emit machine-readable output 

// playwright.config.ts
export default defineConfig({
  reporter: [
    ['list'],                                     // human-readable in the log
    ['junit', { outputFile: 'reports/junit.xml' }],
    ['html', { outputFolder: 'reports/html', open: 'never' }],
  ],
  use: { trace: 'on-first-retry', screenshot: 'only-on-failure' },
});

Verify it works: after a local run, reports/ should contain the XML and the HTML folder. on-first-retry keeps traces small — a trace per run is large and almost always unnecessary.

Step 2 — Upload conditionally, and always on failure 

      - name: Upload diff artefacts
        if: failure()
        uses: actions/upload-artifact@v4
        with:
          name: visual-diffs-${{ matrix.browser }}
          path: |
            test-results/**/*-diff.png
            test-results/**/*-actual.png
            reports/html
          retention-days: 7

      - name: Upload test report
        if: always()                 # the XML feeds the summary even on a pass
        uses: actions/upload-artifact@v4
        with:
          name: junit-${{ matrix.browser }}
          path: reports/junit.xml
          retention-days: 30

Verify it works: a deliberately failing run should attach a diff artifact; a passing run should attach only the XML.

Step 3 — Write a job summary a human will actually read 

{
  echo "## Visual tests — ${BROWSER}"
  echo ""
  echo "| Result | Count |"
  echo "|---|---|"
  echo "| Passed | $(grep -c 'testcase' reports/junit.xml) |"
  echo "| Failed | $(grep -c '<failure' reports/junit.xml) |"
  echo ""
  echo "[Download the diff images](${RUN_URL}#artifacts)"
} >> "$GITHUB_STEP_SUMMARY"

Verify it works: the summary should appear on the run page without anyone downloading anything. This is the single highest-value item in this section, because it is the only one people read by default.

Step 4 — Name artefacts so a matrix does not collide 

          name: visual-diffs-${{ matrix.browser }}-${{ matrix.shard }}

Verify it works: a three-browser, four-shard matrix should produce twelve distinctly named artefacts. A single artefact after a matrix run means the name is not parameterised and eleven uploads were rejected or overwritten.

Step 5 — Set retention deliberately 

          retention-days: 7          # diffs are read within a day or not at all

Verify it works: check the storage figure after a fortnight. Default retention on a busy repository is one of the larger unnoticed costs in CI, and diffs specifically have a very short useful life.

Step 8 — Aggregate a matrix into one report

A three-browser matrix produces three sets of everything, and a reviewer wants one answer. An aggregation job that runs after the matrix turns twelve artefacts into a single summary.

  report:
    needs: [visual]
    if: always()                     # the matrix failing is exactly when this matters
    runs-on: ubuntu-latest
    steps:
      - uses: actions/download-artifact@v4
        with:
          pattern: junit-*
          path: reports/
          merge-multiple: true

      - name: Summarise the matrix
        run: |
          TOTAL=$(grep -ho 'tests="[0-9]*"' reports/*.xml | grep -o '[0-9]*' | paste -sd+ | bc)
          FAILED=$(grep -ho 'failures="[0-9]*"' reports/*.xml | grep -o '[0-9]*' | paste -sd+ | bc)
          {
            echo "## Visual matrix"
            echo ""
            echo "| Browser | Result |"
            echo "|---|---|"
            for f in reports/junit-*.xml; do
              name=$(basename "$f" .xml | sed 's/junit-//')
              fails=$(grep -o 'failures="[0-9]*"' "$f" | grep -o '[0-9]*')
              echo "| $name | $([ "$fails" = 0 ] && echo 'passed' || echo "$fails failed") |"
            done
            echo ""
            echo "**${FAILED} of ${TOTAL} across the matrix.**"
          } >> "$GITHUB_STEP_SUMMARY"

Verify it works: fail one browser deliberately. The summary should name that browser and leave the others marked as passing, so the reviewer knows immediately whether the failure is engine-specific — which is the first question anyone asks about a cross-browser failure.

Configuration reference

Six artefacts, and who each is for Six rows. JUnit XML is produced by any runner, read by the CI test summary, and worth keeping thirty days. Diff images come from the visual runner, are read by a human reviewer, and need about seven days. Playwright's HTML report is read by a human on failure and also needs seven days. A blob report from a sharded run is read only by the merge step and needs a single day. Coverage lcov is read by a coverage service over thirty days. A trace, captured on retry, is read by a human debugging and needs about three days. Artefact Produced by Read by Retain JUnit XML any runner the CI test summary 30 days Diff images the visual runner a human reviewer 7 days HTML report Playwright a human, on failure 7 days Blob report sharded Playwright the merge step 1 day Coverage lcov vitest / istanbul a coverage service 30 days Trace Playwright on retry a human, debugging 3 days
Key Platform Effect
if: always() GitHub Actions Upload even when a previous step failed
retention-days GitHub Actions Overrides the repository default
artifacts:when: on_failure GitLab CI Publish only for failed jobs
artifacts:expire_in GitLab CI Retention for that job’s artefacts
artifacts: Bitbucket Glob of paths kept after the step
GITHUB_STEP_SUMMARY GitHub Actions Markdown appended to the run page
reporter: blob Playwright Mergeable per-shard output

Step 6 — Comment on the pull request, in place

A summary lives on the run page; a comment lives where the review is happening. The rule that keeps comments useful is to update one rather than adding another.

      - name: Comment visual results
        if: always()
        uses: actions/github-script@v7
        with:
          script: |
            const marker = '<!-- visual-report -->';
            const body = `${marker}\n### Visual tests\n\n` +
              `${process.env.FAILED === '0' ? 'All stories matched their baselines.' :
                `**${process.env.FAILED} story/stories differ.** ` +
                `[Download the diffs](${process.env.RUN_URL}#artifacts)`}`;

            const { data: comments } = await github.rest.issues.listComments({
              ...context.repo, issue_number: context.issue.number,
            });
            const existing = comments.find((c) => c.body.includes(marker));

            if (existing) {
              await github.rest.issues.updateComment({ ...context.repo, comment_id: existing.id, body });
            } else {
              await github.rest.issues.createComment({ ...context.repo, issue_number: context.issue.number, body });
            }

Verify it works: push twice to the same branch. The pull request should carry exactly one visual-report comment, showing the latest result. Two comments means the marker is not being found — usually because the body was reformatted between runs.

Step 7 — Keep the collection step separate from the test step 

      - name: Run visual tests
        run: npx playwright test --grep @visual

      - name: Collect artefacts             # runs whatever the tests did
        if: always()
        run: |
          mkdir -p reports/collected
          cp -r test-results/**/*-diff.png reports/collected/ 2>/dev/null || true
          cp reports/junit.xml reports/collected/ || true

Verify it works: a failing test run should still reach the collect step, and the collected directory should contain the diff images. Doing the collection inside the test command means a non-zero exit skips it, which is precisely when it was needed.

Common pitfalls

Uploading without if: always(). The default is to skip a step when a previous one failed, which means the artefact is published on green runs and missing on exactly the red ones that needed it.

Uploading everything. path: . publishes node_modules and the build output. Glob the specific files, or storage cost and upload time both grow without bound.

Unparameterised artefact names in a matrix. Every job writes the same name, and you end up with one browser’s diffs and no indication the others were discarded.

No summary. An artifact zip requires a download, an unzip and a file browser. A three-line table on the run page is read by everyone who looks at the pull request.

Indefinite retention. Diff images are useful for about a day. Keeping them for ninety multiplies storage by ninety for no benefit.

Making a red check self-explanatory

The measure of a reporting setup is how much a developer has to do between seeing a red check and understanding it. Every step in that chain is a place people give up and rerun the job instead.

Zero steps is the target for the common case: the check’s name and the job summary together say what failed and why. visual (chromium) — 1 story differs: Card/WithAvatar is a complete diagnosis for a large fraction of failures, and it costs three lines of shell.

One step is acceptable for anything visual: the summary links to the artefact, and the reviewer downloads it to look at the image. Making that link explicit — rather than expecting someone to find the artifacts section of the run page — is the difference between one step and three.

Three or more steps is where the setup has failed. Opening the run, expanding the log, finding the failing test name, navigating to the artifacts, unzipping, and locating the right image is enough friction that the rational response is to rerun the job and hope.

Two habits close most of the gap. Put the failing identifiers in the summary rather than only in the log, because a log is a wall of text and a summary is a table. And name artefacts after the thing that failed rather than after the job, so a reviewer downloading visual-diffs-chromium knows what they are getting before it finishes.

The underlying point is that CI runs unattended, so everything a developer would have looked at had they been present must be captured at the moment of failure and presented where they will actually see it. The evidence exists either way; the question is only whether anyone can reach it.

Integration point

Reporting is the last stage of every pipeline this guide describes. The GitHub Actions, GitLab CI and Bitbucket Pipelines sections each define jobs whose output lands here, and a sharded run depends on artefacts to merge its results at all. The diff images come from visual regression captures, and reviewing them properly is what baseline management requires. The Storybook test-runner emits JUnit and coverage into the same collection step.

FAQ

Should diff images be uploaded on every run or only on failure?

Only on failure. A passing run’s diffs are, by definition, images that matched — nobody will open them, and on a busy repository they dominate storage within a week. The exception is a deliberate baseline-generation run, where the point is to review the new images before committing them.

How long should artefacts be kept?

Long enough for the person who needs them to look. In practice that is a day or two for diffs and traces, a week for HTML reports, and a month for the JUnit and coverage data that feeds trend reporting. Anything longer is paying to store files whose context — the branch, the reviewer’s attention, the conversation — has already gone.

Is a pull-request comment better than a job summary?

They serve different moments. A job summary is attached to the run and costs nothing to produce; a comment appears where the conversation is happening and is therefore harder to miss. The failure mode of comments is volume: one per push quickly becomes noise, so update a single comment in place rather than adding a new one each time.

What should happen to artefacts from a cancelled run?

Nothing — do not upload them. A cancelled run’s output is partial and misleading, and if: always() will happily publish it. Conditioning on success() || failure() rather than always() covers the two states worth keeping and skips cancellations.

Do artefacts need to be uploaded at all if the report is inline?

An inline summary covers the common case — how many passed, what failed, where to look — and that is enough for most red runs to be understood without a download. What a summary cannot carry is an image or a trace, so the diff and the trace still need uploading. The useful split is that the summary answers “what happened” and the artefact answers “show me”, and the summary should always link to the artefact so the second question is one click away.

How do I keep artefact storage from growing without bound?

Three levers, in order of effect: upload conditionally so passing runs publish almost nothing, set retention per artefact rather than accepting the repository default, and glob narrowly so a directory copy cannot sweep in the build output. Reviewing the storage figure once a quarter catches the fourth cause, which is invariably a job someone added that uploads its whole working directory.

Should the aggregation job be the required status check?

Yes — it is the only job that has seen every matrix result, which makes it the right place to express one verdict. A required check on an individual matrix job leaves the others ungated, and requiring all of them individually breaks the moment a browser is added or removed. One aggregation job declared with needs over the matrix is stable across changes to the matrix itself.