How to detect and quarantine flaky tests automatically

Detection needs the failures, so the upload has to happen whether the suite passed or not. In GitHub Actions that means two changes to the job that runs your tests:

- name: Run tests
  id: tests
  continue-on-error: true
  run: pytest --junit-xml=report.xml

- name: Mergify CI Upload
  if: success() || failure()
  uses: Mergifyio/gha-mergify-ci@v25
  with:
    token: ${{ secrets.MERGIFY_TOKEN }}
    report_path: report.xml
    test_step_outcome: ${{ steps.tests.outcome }}

continue-on-error: true on the test step is what makes quarantine possible later. It hands the pass or fail decision to the Mergify step, which is the only step that knows which failures came from quarantined tests. Passing test_step_outcome covers the nastier case where the runner crashes and leaves a report that looks clean.

Pytest users can skip the report plumbing entirely: the pytest-mergify plugin uploads results on its own once MERGIFY_TOKEN is in the environment. Buildkite and the language-specific setups are in the Test Engine docs.

Once reports are flowing, Mergify classifies each test from its results across runs, weighting recent ones more heavily. Only the tests behaving badly show up:

Two numbers sit next to each test. Confidence says how much data Mergify has: low confidence means the verdict could still move, so give a fresh setup a few days before you act on it. Impact counts the failed executions a test caused, which is the honest measure of what it costs you. Sorting by impact usually shows the same thing in every repository, that a handful of tests are responsible for most of the red builds.

If you would rather fix than quarantine, the framework-by-framework breakdowns in the flaky tests guide name the usual culprits: shared state across parallel workers, frozen clocks, mocks that outlive their test, ordering assumptions that only hold on your laptop.

Auto-quarantine is a per-repository switch on the Mitigation page of the dashboard. Enable it and Mergify quarantines flaky tests as detection finds them, without anyone filing a ticket first. Quarantining known-broken tests is a second option next to it, deliberately separate: a consistently failing test is usually reporting something real, and you should decide to ignore it rather than drift into it.

A quarantined test still runs and still shows its full failure history in the dashboard. What changes is the conclusion your CI reports. Quarantine applies on every branch, not only the default one, so a pull request author gets the benefit immediately.

When you want to handle one test by hand, the CLI does it from the terminal or from a script:

# Quarantine a test by its fully qualified name.
mergify tests quarantines add \
  --reason "flaky under xdist, tracked in MRGFY-1234" \
  "tests/auth/test_login.py::test_login_timeout"

# See everything currently quarantined, and why.
mergify tests quarantines list

# Take it back out once it is fixed.
mergify tests quarantines remove \
  "tests/auth/test_login.py::test_login_timeout"

The same commands are worth wiring into an AI coding agent. mergify tests show returns a test's health as an exit code, so an agent that hits a red test can ask whether it is flaky before it spends an hour rewriting code that was never broken.

Nothing in your queue config, which is the point. The queue keeps gating on the same CI check it always did:

queue_rules:
  - name: default
    merge_conditions:
      - check-success=ci

pull_request_rules:
  - name: Queue approved pull requests
    conditions:
      - "#approved-reviews-by>=1"
    actions:
      queue:
        name: default

That check is now reported by the Mergify upload step, so it goes green when the only failures came from quarantined tests. A flaky test that used to knock a pull request out of the queue, send it to the back, and burn a full CI run on the retry stops doing any of that. If you have not set the queue up yet, the merge queue guide walks through how it works and how to configure one. Quarantine also pairs with auto-retry: retry buys you a second opinion on a suspicious failure, quarantine is what you reach for once the test has proven it cannot give one.

The failure mode of any quarantine system is that it becomes a graveyard. Tests go in, nobody looks again, and a year later a quarter of the suite is decorative. Two habits prevent it.

Put a real reason on every manual quarantine, with a ticket reference. Six months later "flaky" tells nobody anything, while "flaky under xdist, tracked in MRGFY-1234" tells them where to look. Then review the list on a fixed cadence, monthly is enough for most teams. Mergify records whether each quarantined test has recovered, which turns the review into a short list of tests you can take back out rather than an audit of everything.

Watch the trend too. A quarantined test that goes from failing 5% of the time to 40% is telling you something changed, even though it is no longer blocking anyone. That is the part CI Insights is for: quarantine removes the interruption, the data stays.