Skip to content
GitHub Actions

GitHub Actions test report

GitHub Actions doesn't know what a test is. It knows a step exited with code 1, and it leaves you to scroll the log for the assertion. Four community actions fix the display: dorny/test-reporter, EnricoMi/publish-unit-test-result-action, mikepenz/action-junit-report and ctrf-io/github-test-reporter. Here is a working workflow for each, what each one is good at, and the part none of them do.

By Julien Danjou, Co-founder & CEO of MergifyUpdated

In one paragraph

Have your test runner write JUnit XML, then add one report step with if: ${{ !cancelled() }} so it runs when tests fail. Pick dorny/test-reporter for the widest format support and a clean job summary, EnricoMi's action for a pull request comment that compares totals with the previous commit, and mikepenz's for fork-friendly annotations. Those three render one run and then forget it. ctrf-io/github-test-reporter looks back: it reads the reports of earlier runs on the same branch or pull request and shows a fail rate and a flaky rate per test. None of them can tell you that a test passes and fails on the same code anywhere in the repository. That takes per-test history kept outside the workflow, which is what Mergify Test Engine adds on top.

First, write a JUnit XML report

Every one of these actions reads a file, not your test runner. JUnit XML is the format they all understand (ctrf-io's action prefers its own JSON format, see below), and most runners write it behind one flag: pytest --junitxml, Jest and Vitest through a junit reporter, Go through gotestsum, Maven and Gradle by default. In pytest it is this:

- name: Run tests
  run: pytest --junitxml=junit.xml

The examples below use a small pytest suite with four tests, one of which fails about 40% of the time. That flaky test is the point: it is the test a report is supposed to help you with.

dorny/test-reporter

The most used of the three. It creates a check run named after the name input and writes the same report to the job summary: a badge, a table per file, and the failed test with its assertion.

name: Tests

on:
  pull_request:

permissions:
  contents: read
  actions: read
  checks: write

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
      - uses: actions/setup-python@v7
        with:
          python-version: "3.13"
      - run: pip install -r requirements.txt
      - name: Run tests
        run: pytest --junitxml=junit.xml
      - name: Test report
        uses: dorny/test-reporter@v3
        if: ${{ !cancelled() }}
        with:
          name: pytest
          path: junit.xml
          reporter: java-junit

Two things trip people up. reporter is the input format, not a style: java-junit for generic JUnit XML (pytest included), jest-junit for jest-junit's flavour, and native parsers for Mocha JSON, .NET TRX, Dart, Flutter, Go test JSON and RSpec JSON, which skip the XML step entirely. And the step fails when the report contains a failed test (fail-on-error defaults to true), so a red report step next to a red test step is expected, not a second problem.

On pull requests from forks the token is read-only and the check run cannot be created. dorny's README documents the fix: upload junit.xml as an artifact in the test workflow and run the reporter from a second workflow on workflow_run, using its artifact input.

EnricoMi/publish-unit-test-result-action

This one is built around the pull request. Besides the check run and the job summary, it posts a comment with the totals (tests, suites, files, passed, skipped, failed, duration) and updates that same comment on every push, so the history of the totals lives in the comment's edit history.

name: Tests

on:
  pull_request:

permissions:
  contents: read
  issues: read
  checks: write
  pull-requests: write

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
      - uses: actions/setup-python@v7
        with:
          python-version: "3.13"
      - run: pip install -r requirements.txt
      - name: Run tests
        run: pytest --junitxml=junit.xml
      - name: Publish test results
        uses: EnricoMi/publish-unit-test-result-action@v2
        if: ${{ !cancelled() }}
        with:
          files: junit.xml

In a private repository it needs contents: read and issues: read on top of checks: write and pull-requests: write; with comment_mode: off you can drop the last one. The default action runs in Docker on Linux. On macOS and Windows runners, or self-hosted Linux without Docker, use the /macos, /windows or /linux variants of the action.

Its comparison with earlier commits is the closest any of the three gets to history, and it is worth knowing what it compares: totals. "1 failed, was 0" tells you something changed. It does not tell you whether that test has failed four times this week on commits that passed it too.

mikepenz/action-junit-report

The annotation specialist. It creates a check run whose annotations point at the failing line (tests/test_calc.py:20 in our run), writes a job summary, and with detailed_summary: true lists every test in it rather than only the totals.

name: Tests

on:
  pull_request:

permissions:
  contents: read
  checks: write

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
      - uses: actions/setup-python@v7
        with:
          python-version: "3.13"
      - run: pip install -r requirements.txt
      - name: Run tests
        run: pytest --junitxml=junit.xml
      - name: Test report
        uses: mikepenz/action-junit-report@v6
        if: ${{ !cancelled() }}
        with:
          report_paths: junit.xml
          detailed_summary: true
          flaky_summary: true

It has the cleanest answer for forks: annotate_only: true skips the check run and annotates the files directly, which works with a read-only token. It also outputs total, passed, failed and retried counts for later steps.

flaky_summary deserves a precise reading. It lists tests the report itself shows as retried, meaning your framework re-ran them inside that run (pytest-rerunfailures, Playwright's retries), so it shows retries that already happened. A test that fails on Monday's run and passes on Tuesday's never appears in it.

ctrf-io/github-test-reporter

The newest of the four, and the only one that keeps results per test across runs. It reads CTRF, a JSON test report format with a reporter for most frameworks (pytest-json-ctrf for pytest, behind --ctrf), or JUnit XML through its junit-to-ctrf integration. It writes to the job summary and annotates failures by default, and you pick the content from about two dozen built-in reports.

name: Tests

on:
  pull_request:

permissions:
  contents: read
  actions: read

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
      - uses: actions/setup-python@v7
        with:
          python-version: "3.13"
      # requirements.txt lists pytest and pytest-json-ctrf
      - run: pip install -r requirements.txt
      - name: Run tests
        run: pytest --ctrf ctrf/report.json
      - name: Test report
        uses: ctrf-io/github-test-reporter@v1
        if: ${{ !cancelled() }}
        with:
          report-path: ctrf/*.json
          summary-report: true
          failed-report: true
          fail-rate-report: true
          flaky-rate-report: true
          upload-artifact: true
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

upload-artifact: true is what turns on the history. Each run stores its report as an artifact, and the next run downloads the earlier ones (up to 100, from the same workflow, on the same branch for pushes or the same pull request for pull_request) to build the fail rate and flaky rate tables. That download is why it needs actions: read and the GITHUB_TOKEN in its env. Unlike dorny's, the step stays green when a test fails unless you set exit-on-fail: true; the test step already carries the failure.

Read "flaky" the way CTRF defines it: a test that failed, then passed on a retry inside the same run. We pushed five commits with identical code to a sandbox pull request. With the workflow above, our flaky test failed on four runs and passed on one, and the report put it at an 80% fail rate and skipped the flaky rate table: no flaky tests to display. Adding pytest-rerunfailures (pytest --reruns 2 --ctrf ctrf/report.json) turned every run green, and after five runs the flaky rate table listed the test at 28.57%: 2 flaky attempts out of 7. The flaky rate only counts retries, and the retry that makes a test count as flaky is also what hides it from the job's result.

Side by side

dorny/test-reporterEnricoMi/publish-unit-test-result-actionmikepenz/action-junit-reportctrf-io/github-test-reporter
Where the report shows upA check run and the job summaryA check run, the job summary and a pull request commentA check run with annotations and the job summary (a PR comment with comment: true)The job summary and annotations; a PR comment or a status check on request
Input formatsJUnit XML in several flavours (Java, jest-junit, PHPUnit, Python, Swift), plus Mocha JSON, .NET TRX and NUnit, Dart, Flutter, Go test JSON and RSpec JSONJUnit, NUnit and xUnit XML, TRX, and Dart or Mocha JSONJUnit XMLCTRF JSON (a reporter per framework), or JUnit XML through its junit-to-ctrf integration
Runs onAny runner (JavaScript action)Linux runners with Docker by default; separate macOS, Windows and no-Docker variantsAny runner (JavaScript action)Any runner (JavaScript action)
Pull requests from forksNo check run (read-only token); split into a workflow_run job that reads an artifactNeeds the same split into a second workflowannotate_only: true posts annotations without a check runThe job summary works; a PR comment needs the same workflow_run split
Compares with earlier runsNoTotals only, against the earlier commitNoPer test, over up to 100 earlier runs of the same branch or pull request, read from artifacts
Flaky testsNoNoLists tests retried inside the same report (flaky_summary)A flaky rate across those runs, counting tests that passed on a retry

Versions as of November 2026: dorny/test-reporter v3, EnricoMi/publish-unit-test-result-action v2, mikepenz/action-junit-report v6, ctrf-io/github-test-reporter v1.

What none of them can tell you

We ran the first three on the same pull request. On the first push our flaky test failed: dorny and mikepenz both reported 3 passed, 1 failed with the assertion, in a nice table. On the next run it passed, and every report said 4 passed. Each report was right about its own run, and none of them could say the obvious thing, which is that this test passes and fails on the same code.

ctrf-io's action goes further, and its limits show where that history has to live. It rebuilds the history from artifacts, so it sees one branch or one pull request at a time, for as long as the artifacts are kept (90 days by default: our sandbox reports expire in February 2027). A test that fails once on main, once on your pull request and once on a colleague's leaves three short histories that never meet. Without a retry it calls the test failing, which is what a person concludes from a red run anyway.

To call a test flaky you need its results across runs, ideally on the same commit, stored somewhere that outlives the workflow. GitHub Actions has no such place: it keeps logs and artifacts for a retention window, per run, with no idea which test is which.

Without that history, the report tells everyone a test failed and leaves them to guess why. Someone clicks "Re-run failed jobs", it passes, and the next person to hit it does the same. The report made the failure easier to read and did nothing about the fact that it keeps happening.

Add the history: Mergify Test Engine

Test Engine is the test layer GitHub Actions is missing. It records every test's result on every run, classifies each test as healthy, flaky or broken from that history, and can quarantine the flaky ones so they keep running without failing the job. You do not have to give up your report action to get it:

name: Tests

on:
  pull_request:

permissions:
  contents: read
  actions: read
  checks: write

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
      - uses: actions/setup-python@v7
        with:
          python-version: "3.13"
      # requirements.txt lists pytest and pytest-mergify
      - run: pip install -r requirements.txt
      - name: Run tests
        env:
          MERGIFY_TOKEN: ${{ secrets.MERGIFY_TOKEN }}
        run: pytest --junitxml=junit.xml
      # Optional: keep the job summary you already like.
      - name: Test report
        uses: dorny/test-reporter@v3
        if: ${{ !cancelled() }}
        with:
          name: pytest
          path: junit.xml
          reporter: java-junit

For pytest, the pytest-mergify plugin sends results as soon as MERGIFY_TOKEN is in the environment; the JUnit file is only there for the report action. The test output gains a Mergify CI section with the run's ID. Vitest, Playwright, Jest and RSpec have the same kind of native plugin, and any other runner uploads its JUnit XML with the Mergify CI action.

In the same sandbox repository, after a few runs, Test Engine had our test as flaky, because one commit had both failed and passed it. A report action on those runs would have said failed, then passed, and moved on. From there, the quarantine keeps it from blocking pull requests while someone fixes it, and the flaky tests guide covers the usual causes per framework.

GitHub Actions still sees jobs, not tests. Test Engine is the test layer, and the job layer is next.

FAQ

Does GitHub Actions have a built-in test report?

No. GitHub Actions sees steps and jobs, and a test step is a command that exits 0 or not. To see which tests failed you either read the log or add an action that parses the JUnit XML your test runner writes and publishes it as a check run or a job summary.

Which test report action should I use?

dorny/test-reporter if your runner writes a format it reads natively (Mocha JSON, .NET TRX, Dart, Go test JSON, RSpec JSON) or you want a clean job summary. EnricoMi/publish-unit-test-result-action if you want a pull request comment that compares totals with the previous commit. mikepenz/action-junit-report if you need annotations on pull requests from forks or want the flaky summary for tests your framework already retries. ctrf-io/github-test-reporter if you want per-test fail and flaky rates over earlier runs of the same branch or pull request, and you are fine adding a CTRF reporter to your test runner.

Why does the test report action fail my job?

Because a failed test is a failure. dorny/test-reporter fails its own step when the report has a failed test (fail-on-error defaults to true), and the test step itself already failed. ctrf-io/github-test-reporter does not fail its step unless you set exit-on-fail: true. Give the report step an if: condition on !cancelled() so it still runs after the test step fails, otherwise you get no report on exactly the runs you need one.

Why does the action say 'Resource not accessible by integration'?

The workflow token cannot create check runs. Add checks: write to the workflow permissions (and pull-requests: write for EnricoMi's comment). ctrf-io's action also needs actions: read and the GITHUB_TOKEN in its env to fetch earlier runs. On pull requests from forks the token is read-only whatever you set, so move the reporting into a second workflow triggered by workflow_run, or use mikepenz's annotate_only mode.

Can a test report action detect flaky tests?

Partly. dorny, EnricoMi and mikepenz read one report from one run; mikepenz can list tests that a framework retried inside that run. ctrf-io/github-test-reporter reads the reports of earlier runs from their artifacts and computes a flaky rate, but it counts a test as flaky only when it failed and then passed on a retry in the same run, and only over runs of the same branch or pull request. Without retries, a test that failed yesterday and passed today shows up in the fail rate, not as flaky. Flaky detection across the repository needs per-test history stored somewhere that outlives the run.

Do I have to remove my test report action to use Mergify Test Engine?

No. Test Engine reads the same test results and keeps their history; the action keeps rendering the job summary. Plenty of teams run both. With pytest-mergify, @mergifyio/vitest, @mergifyio/playwright, @mergifyio/jest or rspec-mergify you do not even need the JUnit file for Mergify: the plugin sends results on its own.

Last updated:

Your test report, with a memory

Keep the job summary. Add per-test history, flaky detection and quarantine to the same GitHub Actions workflow. Free for up to 5 users.