---
title: "pytest-xdist makes the suite faster and the flakes weirder"
description: "Why a test that always passes alone fails on `pytest -n auto`, the fixture-scope rule that prevents most worker races, and the worker_id pattern for genuinely shared resources."
author: remy-duthu
date: 2026-05-09
category: testing
readTimeMinutes: 6
featured: false
product: test-engine
---

A pytest test that asserts on the contents of `/tmp/test-output/report.txt` passes locally. CI runs `pytest -n auto` across 4 workers and the test fails 30% of the time with `FileNotFoundError`. The file path is hardcoded. The test that creates the file is in a different module. The two tests share the same path string, and that string is the bug.

We see this pattern often enough on [Mergify Test Engine](/product/test-engine) that it earned its own slot in our [flaky pytest catalog](/learn/flaky-tests/pytest). The cause is shared external state across workers. The fix is per-worker namespacing, plus the right fixture scope.

## What you see

```python
WORKDIR = "/tmp/test-output"

def test_writes_report():
    with open(f"{WORKDIR}/report.txt", "w") as f:
        f.write("hello")

def test_reads_report():
    assert os.path.exists(f"{WORKDIR}/report.txt")
```

Sequential run, single worker: `test_writes_report` runs first, creates the file, `test_reads_report` runs next, finds it. Pass.

Under `pytest -n auto`, [pytest-xdist](https://pytest-xdist.readthedocs.io/en/stable/) distributes tests across worker processes. The two tests can land on different workers. The reader checks for the file before the writer creates it (or after the writer's worker cleaned up the directory). The failure looks like a missing file but the actual cause is two parallel processes racing on a shared path.

The frustrating part: the shared path is invisible from the test code. `WORKDIR` looks like a constant. It is — the constant just refers to a piece of state outside the test process.

## Why xdist's distribution algorithm matters

pytest-xdist's default scheduler is `loadscope`, which keeps tests from the same module on the same worker. That helps for module-level fixtures but does not help for shared external resources between modules.

The `loadfile` scheduler keeps tests from the same file on the same worker. Better for file-level coupling, still does not help for cross-file shared state.

The `worksteal` scheduler dynamically rebalances. It is the fastest for unbalanced suites but the most likely to expose cross-test races because it cares less about locality.

You can pick the scheduler with `--dist`:

```sh
pytest -n auto --dist worksteal
```

Pick the one that matches your suite's reality. None of them help if your tests share a `/tmp/test-output` path.

## The fix that holds

Use pytest's built-in `tmp_path` fixture. It gives every test its own directory, scoped to that test:

```python
def test_writes_report(tmp_path):
    (tmp_path / "report.txt").write_text("hello")

def test_reads_report(tmp_path):
    (tmp_path / "report.txt").write_text("hello")
    assert (tmp_path / "report.txt").exists()
```

Each test owns its directory. There is nothing shared, so there is no race. The directory cleans up automatically at the end of the test.

For test pairs that genuinely need to share a directory (a producer and a consumer), use `tmp_path_factory` at session scope:

```python
@pytest.fixture(scope="session")
def shared_dir(tmp_path_factory):
    return tmp_path_factory.mktemp("shared")

def test_writes_report(shared_dir):
    (shared_dir / "report.txt").write_text("hello")

def test_reads_report(shared_dir):
    assert (shared_dir / "report.txt").exists()
```

The session-scoped fixture is created once per worker, and `tmp_path_factory.mktemp` includes the worker ID in the path. Workers cannot collide. Tests within the same worker see each other's writes.

## When the resource is genuinely process-wide

Some resources are not file paths. A test database, a Redis instance, a Kafka topic — these live outside the worker. pytest-xdist exposes a `worker_id` fixture so you can namespace by worker:

```python
@pytest.fixture
def redis_namespace(worker_id):
    namespace = f"test-{worker_id}"
    yield namespace
    redis.delete_pattern(f"{namespace}:*")
```

Each worker gets its own Redis namespace. Cleanup is per-worker. The shared Redis instance no longer becomes a race.

For test databases under `pytest-django` or similar plugins, the standard pattern is one database per worker. The `--reuse-db --create-db` flags handle this once configured. Without per-worker databases, every test creates rows the others can see, and `--dist worksteal` will eventually surface a race.

## How Mergify catches this before you ship

xdist races are easy to dismiss as "weird, retry the suite." The retry usually wins because the next worker assignment is different. Teams learn to ignore the failure category, which is exactly when a real ordering bug ships to production.

[Test Engine](/product/test-engine) groups failures by their xdist worker ID. When a test only fails when assigned to worker 2 alongside a specific other test on the same worker, the dashboard surfaces the parallelism dimension: "test_reads_report fails when scheduled on the worker after test_writes_report's cleanup." You see the pair without manually bisecting or minimizing the failing set yourself.

Quarantine kicks in once the pattern is clear, so the merge queue keeps moving while you switch to `tmp_path` or `worker_id`-scoped resources.

Stop staring at xdist worker logs trying to find the shared resource. [Point Mergify at your suite](/discovery) — the native `pytest-mergify` plugin installs in one `pip install`.

## More patterns like this

Worker races are one of the eight patterns in [the flaky-tests-in-pytest guide](/learn/flaky-tests/pytest). The others are variants of the same theme: state that crosses tests in ways the test author did not anticipate. Fixture teardown order, autouse fixtures touching globals, monkeypatch leakage, async event-loop scope mismatches under pytest-asyncio. Different trigger, same root.

Most fixes are pytest's built-in fixtures (`tmp_path`, `monkeypatch`, `worker_id`) that already exist for exactly this purpose. The bug is usually the test reaching outside what those fixtures cover.