Run pytest tests in parallel with pytest-xdist

Install pytest-xdist in the test environment, then use python -m pytest -n auto. The plugin runs tests in separate worker processes and combines their results. The benefit depends on test independence and resource capacity, not simply on the number of tests.

python -m pip install pytest-xdist
python -m pytest -n auto
python -m pytest -n 4

auto selects a worker count based on detected physical CPU cores, with environment/hook overrides available. Container and CI resource limits may not match the machine's reported capacity. Set an explicit count when that makes resource use more predictable.

Choose a scheduling boundary

The default scheduler distributes tests to available workers. If tests should share a worker at a coarser boundary:

python -m pytest -n auto --dist loadfile
python -m pytest -n auto --dist loadscope

loadfile groups by file. loadscope groups functions by module and methods by class. These policies can help reuse fixtures, but they are not a global lock: different groups still run concurrently.

Isolate resources between workers

Use distinct databases or schemas, temporary directories, ports, and output files when concurrent access would conflict. The worker_id fixture provides identifiers such as gw0 and gw1:

def test_worker_resource_name(worker_id):
    resource_name = f"test_database_{worker_id}"
    assert resource_name.startswith("test_database_")

A session fixture can run once in each worker that needs it. For a resource that must be initialized only once across workers, use explicit coordination and a design that handles failures and cleanup. Sharing mutable state safely requires more than changing a fixture's scope.

Measure before increasing the worker count

Four workers do not guarantee one quarter of the runtime. Startup cost, test duration imbalance, database limits, memory pressure, and I/O contention can erase or reverse the gain. Compare total runtime and failure behavior with a serial baseline on the same CI class.

FAQ

Why do tests become flaky in parallel?

Resource collisions are a common cause, but also inspect collection consistency, timing assumptions, and external-service limits. Grouping tests may hide a dependency without fixing it. Prefer independent setup and cleanup.

Can I use -s to stream worker prints?

Not as in a serial pytest run. Xdist does not forward worker stdout/stderr for -s; captured failure reports remain useful. Reproduce with -n 0 or without xdist when debugging interactive output.

Can database tests run in parallel?

Yes, when the database strategy supports isolation and sufficient capacity. Per-worker schemas are one approach; a deliberately concurrency-safe shared fixture can also work. Verify transactions, migrations, unique names, and cleanup rather than assuming parallelism is always forbidden or always safe.

References: pytest-xdist scheduling, pytest-xdist usage, pytest-xdist limitations.