A GitHub Actions Cache Key Is Not the Cache Key

On August 31, two of our twenty engine test shards were killed at their 20-minute timeout while the other eighteen finished between 10m36s and 14m14s, on the same commit, running the same suite. A slow suite makes every shard slow, so twenty jobs splitting one body of work should land close together. Two of them ran at least 40% longer than the slowest of the rest, so the work wasn’t being divided properly. And they had been doing it intermittently since early June, often enough to be annoying and rarely enough that we wrote it off as a slow day.

Dividing the work is pytest-split’s job. It reads a .test_durations file and packs each shard to roughly equal wall-clock time instead of equal test counts, which is one of the things that keeps us under a timeout we set as a budget, so nobody adds four minutes to CI without noticing. Most of our jobs run on a self-hosted fleet. A handful still run on GitHub-hosted runners, including, until recently, the two jobs that write the caches everything else reads.

Finding out the cache was never read

When I finally dug in, the durations file wasn’t there, so pytest-split had fallen back to splitting by test count and hoping the tests were the same size. The step that should have restored the file printed Cache not found for input keys and exited zero, which is the same thing it prints when a cache really is cold. A cache miss isn’t an error, so nothing failed and nothing went red, and nobody reads the logs of a job that passed. The timeouts were how we found out.

How a cache entry is actually addressed

A cache entry is addressed by two things: the key you wrote, and a version you didn’t. actions/cache computes that version as a SHA-256 over the path input exactly as written, the compression method the runner used, and a salt. Both halves have to match, and only one of them is visible to you.

The compression half is the part that surprised me, because it isn’t a setting. actions/cache looks for zstd on the machine and uses it if it finds it (spelled zstd-without-long internally), and gzip if it doesn’t. That choice goes straight into the hash, so the same key resolves to two different addresses depending on which runner ran the step:

sha256("engine/.test_durations|gzip|1.0")               # 1dd0e73a8f5e...
sha256("engine/.test_durations|zstd-without-long|1.0")  # 8eb9909db3f1...

There’s no input to pin it, either. None of the inputs that actions/cache/restore accepts control compression, so the cache address depends on whether a binary happens to exist on your runner image, not on anything in your workflow file.

The REST API exposes version per entry, which the web UI doesn’t, so you can see which runner wrote each one: 1dd0e73a… for a gzip runner, 8eb9909d… for a zstd one. The logs show it more directly: a GitHub-hosted job unpacks cache.tzst with --use-compress-program unzstd, while a self-hosted job minutes later on the same commit unpacks cache.tgz with -z. Both machines report runner.os as Linux and both are Ubuntu 24.04.

flowchart LR
    W["engine-update-test-durations<br/>runs-on: ubuntu-latest<br/>zstd present"]
    R["engine-test x20<br/>runs-on: self-hosted-ubuntu-24.04<br/>no zstd"]
    C[("Actions cache")]
    W -->|"key: test-durations-4e636b85...<br/>version: 8eb9909d... (zstd-without-long)"| C
    C -->|"looks up<br/>key: test-durations-4e636b85...<br/>version: 1dd0e73a... (gzip)"| R
    R --> M["'Cache not found for input keys'<br/>exit 0"]
    M --> S["no durations, every test weighted 1"]
    S --> T["shards packed by count"]

The mismatch started on April 20, when we moved our test and linter jobs onto the self-hosted fleet and left the two cache writers behind on GitHub-hosted runners. The pull request that made the move explains why, and the reasoning is sound on its own terms:

The engine-update-test-durations job keeps runs-on: ubuntu-latest on
purpose — it runs only on push to main and doesn't need self-hosted
capacity.

From April 20 to September 7 the restore step went green on every run and restored nothing.

What we changed

Once we understood the addressing, there were two ways to make both kinds of runners agree, and we did both. Installing zstd on our self-hosted image means both kinds of runner now pick the same compression, so every cache in the fleet is readable whichever runner writes it. It’s the safety net for whatever job somebody adds next. The fix in ci.yaml is moving the two writers onto the self-hosted fleet, which is two lines.

We didn’t try forcing GitHub-hosted runners onto gzip. Since there’s no compression input, that would mean deleting or shadowing zstd on a machine whose image ships it and relying on undocumented internal behavior. When that breaks, the failure mode is the same silent cache miss.

In hindsight, fail-on-cache-miss: true on the restore step would have turned four months of green into one red run.

Moving them was easy to justify, because those jobs are close to free. engine-mypy-cache-generator averages 35 seconds, between 21s and 62s, because it restores a warm cache and mypy has nothing left to do. Across 100 runs since September 4 that’s about one slot-hour per week, against roughly 40,300 available on our 240-slot pool. About 0.0025%. engine-update-test-durations merges some JSON and finishes in seconds.

Capacity was the entire argument for splitting them off in April. It saved 0.0025% of the pool and cost four months of unbalanced shards. The comment we left in ci.yaml sums it up: keeping writers and readers on the same runner image is worth the capacity it costs. Since the fix, no shard has been killed at the cap across the 59 matrices I sampled over four days. That’s a short window, and shard times still swing by several minutes with how loaded the pool is, so I’d call that early rather than settled.

I’d been treating a green check as proof the cache worked, and for four months it was only telling me that nothing had thrown an exception. A cold cache and a cache you can never read print the same log line and the same green check, and only one of them fixes itself on the next run.