Moving Our Python Engine to httpx2: the Tracer and the Trust Store Were the Hard Part

encode/httpx hasn’t had a release since 0.28.1, in December 2024. I noticed it in June while keeping up with tech news. Mergify’s engine, which makes every outbound call we have (GitHub, Slack, Datadog, Linear, our subscription API), moved off it on September 22. Nothing was forcing the engine off it, but staying was a risk: if a CVE were discovered in httpx, we’d have to migrate in a hurry. On httpx2 we can apply the fix as soon as it’s released. Since there was no rush, we waited until our monitoring could see the new client, and we kept the engine on the certificates it already trusted.

The two candidates I looked at were httpx2 and httpxyz. httpx2 won because Pydantic took it over, and Pydantic is a known organization. It keeps httpx’s API, private modules included, so a port is mostly import httpx becoming import httpx2.

Waiting for the tracer

In June, Datadog’s Python tracer, ddtrace, patched httpx and nothing else. If we moved our clients, every outbound span would disappear from APM. Our ddtrace.config.httpx["split_by_domain"] = True setting, which gives each remote host its own service in Datadog, would have turned into dead config.

We could instrument the httpx2 transport by hand or wait for dd-trace-py#18780, the native support request. ddtrace 4.14 did trace httpx2, but only when it was aliased under the httpx module name.

We tried the alias: alias_httpx put httpx2 into sys.modules['httpx'] early enough for the tracer to patch it. The alias worked, but messing with sys.modules might not work everywhere, and the switch wasn’t urgent enough to take that risk. We dropped it and waited. The ddtrace issue closed on September 14 with a native httpx2 integration in 4.15.

The native integration reads config.httpx2, not config.httpx, so our split_by_domain setting didn’t reach it. With 4.15.1 installed next to both libraries, httpx2 spans landed on the engine’s own service instead of one per remote host. The ddtrace bump on September 18 also set ddtrace.config.httpx2["split_by_domain"] = True, with a test so nobody would delete the line as dead config before the clients moved. The migration PR went up seven minutes after that bump merged. Spans now show up with component:httpx2, and Sentry’s SDK already traced httpx2 natively.

If another Python team asked me when to do this migration, I’d say the minimum is to wait for the monitoring stack, then check that your spans still land where they used to.

The port itself

The engine PR touched 132 files, +1,013 −925, mostly import lines. Most of the test changes came from respx, our HTTP mocking library, which isn’t fully ready for httpx2.

We now build the transport through httpx2 instead of on a pool we assembled by hand, so httpx2 pins httpcore2 and we don’t. authlib 1.8 and later picks httpx2 by itself when it’s installed, which moved our OAuth clients and the exceptions our auth code catches along with it.

The old package stays installed, though, because other dependencies still import it. import httpx inside the engine would still work, and it would build a client that goes around our transport (retries included) and our TLS context without any error. A ruff ban on httpx in the engine turns that into a lint failure.

The trust store

httpx 0.28 verifies TLS against certifi’s CA bundle by default. httpx2 verifies against the operating system’s store, through truststore. The engine had never configured its public roots. It trusted certifi because that was httpx’s default.

My first reaction was to ask why we’d bother. If we never documented changing the trust store, customers weren’t doing it, and all our customers run our Docker image, so the OS store would be ours too.

The engine isn’t only HTTP clients, though. A helper called ca_bundle already hands certifi (plus an optional EXTRA_CA_BUNDLE) to Postgres, S3, Google Cloud Storage, Sentry and git, clients that take a single bundle file. If we’d left httpx2 on its default, the engine’s HTTP clients would have trusted the OS store while everything else in the same process trusted certifi.

We kept certifi to stay consistent with the other libs we use. The PR before the migration made that choice explicit. create_ssl_context stopped calling httpx.create_ssl_context and started resolving the store itself, in the same order httpx does:

def create_ssl_context() -> ssl.SSLContext:
    if cafile := os.environ.get("SSL_CERT_FILE"):
        context = ssl.create_default_context(cafile=cafile)
    elif capath := os.environ.get("SSL_CERT_DIR"):
        context = ssl.create_default_context(capath=capath)
    else:
        context = ssl.create_default_context(cafile=certifi.where())
    if settings.EXTRA_CA_BUNDLE is not None:
        context.load_verify_locations(cafile=str(settings.EXTRA_CA_BUNDLE))
    return context

EXTRA_CA_BUNDLE is added on top of the public roots, never instead of them. An installation whose GitHub Enterprise runs on a private PKI still has to reach our subscription API and every other public endpoint. Since that resolution order is now our code and not httpx’s, tests pin it, including SSL_CERT_FILE winning over SSL_CERT_DIR and EXTRA_CA_BUNDLE landing on top of each.

SSL_CERT_DIR points to a directory, which doesn’t fit in a single bundle file. When it’s set, the HTTP clients read the directory while ca_bundle’s clients fall back to certifi, and ca_bundle logs that it did.

The one place that built its own httpx client, the Datadog marketplace key request, now gets this context too. Without it, that request alone would have switched to the OS store.

flowchart LR
    subgraph default["httpx2 left on its default"]
        H1[GitHub / Slack / Linear clients] --> OS[(OS store via truststore)]
        P1[Postgres / S3 / GCS / Sentry / git] --> C1[(certifi + EXTRA_CA_BUNDLE)]
        V1[Vertex AI via aiohttp] --> C3[(certifi, built per request)]
    end
    subgraph shipped["What we shipped"]
        H2[GitHub / Slack / Linear clients] --> C2[(certifi + EXTRA_CA_BUNDLE)]
        P2[Postgres / S3 / GCS / Sentry / git] --> C2
        V2[Vertex AI via google-genai] --> C2
    end

google-genai

The ticket for google-genai, which the engine uses to call Gemini on Vertex AI, said we had to move its httpx usage to httpx2. When I checked it against google-genai 2.20.0, the description was wrong in three places.

The SDK built an httpx client at init and never used it for our traffic. _use_aiohttp() returns True whenever aiohttp is importable, and it was, pulled in by aioboto3. So Gemini calls streamed over aiohttp. The private-API risk the ticket worried about was already gone in 2.20.0. And the SDK has accepted injected httpx2 clients since 2.18.0.

On aiohttp, the SDK built its TLS context per request, after our service setup had cleared the environment variables that create_ssl_context reads at import. So Vertex always verified against certifi, even on a deployment that set SSL_CERT_FILE for every other client, and EXTRA_CA_BUNDLE only reached it because some code copied it across by hand. ddtrace also traced aiohttp without split_by_domain, so Vertex calls showed up under the engine’s own service name instead of Vertex’s.

The fix injects httpx2 clients built on the engine’s shared TLS context. They restate the SDK’s own timeout and follow_redirects so they aren’t stricter than the clients it would have built, and they’re closed in a finally, because httpx2 has no finalizer and we build one per request. Deployments with client certificates keep the SDK’s own transports, since its mTLS path only exists there.

The aiohttp path had a single connect-level retry, which we lost. With about five Vertex calls a month, it fired zero times in 30 days.

Where it stands

cassian, the AI code reviewer we run on our own pull requests, went last and was the easiest. openai and pydantic-ai-slim already required httpx2, so both stacks were installed and our own pin was the only thing keeping the unmaintained one alive. pydantic-ai was also warning that httpx.AsyncClient support for OpenAI-compatible providers “is deprecated and will be removed in v3”. The suite logged four of those warnings before and none after. encode/httpx and httpcore are gone from its lockfile.

They’re still in the engine’s lockfile. google-genai does a bare import httpx at module scope, and python-genai#2970, which would have made it conditional, was closed as stale without merging, and wouldn’t have removed httpx from the dependencies anyway. respx, which our tests use to mock HTTP, still depends on it: our tests reach httpx2 through pytest-httpx2’s mocker, and the respx PR adding httpx2 support (respx#317) is still open. Until both move, the ruff ban is what keeps our own code off the old package.

The one upstream change we’ve made so far went into aiobotocore: since aio-libs/aiobotocore#1669, merged on July 31, its optional httpx backend prefers httpx2 when it’s installed. The engine itself still talks to S3 over aiohttp, aiobotocore’s default. respx may be next.