GitHub Actions test report

Every one of these actions reads a file, not your test runner. JUnit XML is the format they all understand, and most runners write it behind one flag: pytest --junitxml, Jest and Vitest through a junit reporter, Go through gotestsum, Maven and Gradle by default. In pytest it is this:

- name: Run tests
  run: pytest --junitxml=junit.xml

The examples below use a small pytest suite with four tests, one of which fails about 40% of the time. That flaky test is the point: it is the test a report is supposed to help you with.

The most used of the three. It creates a check run named after the name input and writes the same report to the job summary: a badge, a table per file, and the failed test with its assertion.

name: Tests

on:
  pull_request:

permissions:
  contents: read
  actions: read
  checks: write

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
      - uses: actions/setup-python@v7
        with:
          python-version: "3.13"
      - run: pip install -r requirements.txt
      - name: Run tests
        run: pytest --junitxml=junit.xml
      - name: Test report
        uses: dorny/test-reporter@v3
        if: ${{ !cancelled() }}
        with:
          name: pytest
          path: junit.xml
          reporter: java-junit

Two things trip people up. reporter is the input format, not a style: java-junit for generic JUnit XML (pytest included), jest-junit for jest-junit's flavour, and native parsers for Mocha JSON, .NET TRX, Dart, Flutter, Go test JSON and RSpec JSON, which skip the XML step entirely. And the step fails when the report contains a failed test (fail-on-error defaults to true), so a red report step next to a red test step is expected, not a second problem.

On pull requests from forks the token is read-only and the check run cannot be created. dorny's README documents the fix: upload junit.xml as an artifact in the test workflow and run the reporter from a second workflow on workflow_run, using its artifact input.

This one is built around the pull request. Besides the check run and the job summary, it posts a comment with the totals (tests, suites, files, passed, skipped, failed, duration) and updates that same comment on every push, so the history of the totals lives in the comment's edit history.

name: Tests

on:
  pull_request:

permissions:
  contents: read
  issues: read
  checks: write
  pull-requests: write

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
      - uses: actions/setup-python@v7
        with:
          python-version: "3.13"
      - run: pip install -r requirements.txt
      - name: Run tests
        run: pytest --junitxml=junit.xml
      - name: Publish test results
        uses: EnricoMi/publish-unit-test-result-action@v2
        if: ${{ !cancelled() }}
        with:
          files: junit.xml

In a private repository it needs contents: read and issues: read on top of checks: write and pull-requests: write; with comment_mode: off you can drop the last one. The default action runs in Docker on Linux. On macOS and Windows runners, or self-hosted Linux without Docker, use the /macos, /windows or /linux variants of the action.

Its comparison with earlier commits is the closest any of the three gets to history, and it is worth knowing what it compares: totals. "1 failed, was 0" tells you something changed. It does not tell you whether that test has failed four times this week on commits that passed it too.

The annotation specialist. It creates a check run whose annotations point at the failing line (tests/test_calc.py:20 in our run), writes a job summary, and with detailed_summary: true lists every test in it rather than only the totals.

name: Tests

on:
  pull_request:

permissions:
  contents: read
  checks: write

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
      - uses: actions/setup-python@v7
        with:
          python-version: "3.13"
      - run: pip install -r requirements.txt
      - name: Run tests
        run: pytest --junitxml=junit.xml
      - name: Test report
        uses: mikepenz/action-junit-report@v6
        if: ${{ !cancelled() }}
        with:
          report_paths: junit.xml
          detailed_summary: true
          flaky_summary: true

It has the cleanest answer for forks: annotate_only: true skips the check run and annotates the files directly, which works with a read-only token. It also outputs total, passed, failed and retried counts for later steps.

flaky_summary deserves a precise reading. It lists tests the report itself shows as retried, meaning your framework re-ran them inside that run (pytest-rerunfailures, Playwright's retries), so it shows retries that already happened. A test that fails on Monday's run and passes on Tuesday's never appears in it.

We ran all three on the same pull request. On the first push our flaky test failed: dorny and mikepenz both reported 3 passed, 1 failed with the assertion, in a nice table. On the next run it passed, and every report said 4 passed. Each report was right about its own run, and none of them could say the obvious thing, which is that this test passes and fails on the same code.

The actions are doing their job. A report action reads one file from one run and exits. To call a test flaky you need its results across runs, ideally on the same commit, stored somewhere that outlives the workflow. GitHub Actions has no such place: it keeps logs and artifacts for a retention window, per run, with no idea which test is which.

Without that history, the report tells everyone a test failed and leaves them to guess why. Someone clicks "Re-run failed jobs", it passes, and the next person to hit it does the same. The report made the failure easier to read and did nothing about the fact that it keeps happening.

Test Engine is the test layer GitHub Actions is missing. It records every test's result on every run, classifies each test as healthy, flaky or broken from that history, and can quarantine the flaky ones so they keep running without failing the job. You do not have to give up your report action to get it:

name: Tests

on:
  pull_request:

permissions:
  contents: read
  actions: read
  checks: write

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
      - uses: actions/setup-python@v7
        with:
          python-version: "3.13"
      # requirements.txt lists pytest and pytest-mergify
      - run: pip install -r requirements.txt
      - name: Run tests
        env:
          MERGIFY_TOKEN: ${{ secrets.MERGIFY_TOKEN }}
        run: pytest --junitxml=junit.xml
      # Optional: keep the job summary you already like.
      - name: Test report
        uses: dorny/test-reporter@v3
        if: ${{ !cancelled() }}
        with:
          name: pytest
          path: junit.xml
          reporter: java-junit

For pytest, the pytest-mergify plugin sends results as soon as MERGIFY_TOKEN is in the environment; the JUnit file is only there for the report action. The test output gains a Mergify CI section with the run's ID. Vitest, Playwright, Jest and RSpec have the same kind of native plugin, and any other runner uploads its JUnit XML with the Mergify CI action.

In the same sandbox repository, after a few runs, Test Engine had our test as flaky, because one commit had both failed and passed it. A report action on those runs would have said failed, then passed, and moved on. From there, the quarantine keeps it from blocking pull requests while someone fixes it, and the flaky tests guide covers the usual causes per framework.

GitHub Actions still sees jobs, not tests. Test Engine is the test layer, and the job layer is next.