hash(id) % 8 Pinned One Python Process at 98% While Its Neighbors Idled

Our Cloud Run dashboard said the container was at 54% CPU. The container had two vCPUs and ran a single asyncio event loop, so 54% meant one core pinned solid and the other one nearly idle.

The fix was a balancer that places orgs by measured load instead of a hash. It then refused to move a single org, and its “add capacity” alarm helped grow the fleet from 8 to 12 processes that, it turned out, was never at capacity. It was ranking processes by the peak of a metric capped at one core.

Eight processes, one of them pinned

At Mergify, some customers get dedicated workers: their GitHub events and their merge queue are processed in their own worker processes instead of the shared pool. Each dedicated container has 2 vCPUs and runs one Python process, which I’ll call a shard. The second vCPU is there because Cloud Run ties memory to vCPU count, and because the sidecar processes need a bit of CPU too. Every org on a shard is an asyncio task in the same event loop.

Which process gets which org was decided by one line:

process_index = blake2s(owner_id) % processes  # simplified

That line weighs nothing. A tiny org and our heaviest customer cost the same to place. With eight processes, the draw came out badly: process 1 got 33 orgs, including several of our heaviest. It sat at 98% of a core for days, while processes 2 and 5 idled at 17% and 27%. Nine of the ten most-starved orgs were on it, and every firing of our stuck-event monitor since late July was an org on that one process.

We didn’t see it for weeks because we were watching the wrong number. We don’t run the Datadog agent inside each container, so we had no per-process CPU, only Cloud Run’s container utilization. That’s a fraction of 2 vCPU, and a single-threaded process can only ever use one of them, so 50% is the wall. Process 1’s container read 54.5%: one pinned Python process plus a little container overhead.

What finally showed it was the APM runtime metric, runtime.python.cpu.percent, which measures the Python process itself. With that in front of us, we saw some processes at 100% a lot of the time, not in short spikes, inside containers that looked like they had room to spare. Across the fleet we paid for 16 vCPU and could only ever reach 8 of them. On average we used about 4.84 cores, and that number was capped too.

A second metric had been pointing us the wrong way. asyncio.scheduling.latency was sent in seconds but declared as milliseconds, so dashboards showed it 1000 times lower than it was. A saturated event loop looked calm, and the shards looked like they were waiting on I/O.

Adding processes doesn’t fix a modulo

Adding containers was the obvious lever, and over the following days the fleet went from 8 to 10 to 12 processes. It helped a little: after the new containers came up, the load was still badly spread and some of them had almost nothing to do.

That’s the modulo at work. The process count is part of the key, so changing it moves most orgs to a different process: about 80% on our 8 → 10 resize and 83% on 10 → 12. Every resize is a fresh random draw. And the draw doesn’t improve steadily as you add capacity. Here’s the load of the worst shard, in percent of one core, for every process count from 8 to 16, as our coding agent modeled it from each org’s estimated demand. These are modeled, and they go past 100% because demand isn’t capped the way a process is.

Processes	8	9	10	11	12	13	14	15	16
Worst shard	183%	127%	106%	118%	105%	94%	84%	99%	152%

Doubling the fleet from 8 to 16 processes still leaves the worst one at 152% of a core. Going from 10 to 11 makes it worse. You can try counts until one looks fine, but the next customer who onboards re-rolls it.

Consistent hashing would fix the reshuffle and leave the hotspot where it is. Even the bounded-load variant caps how many keys land on a node, not how much they cost. The hotspot comes from a few heavy orgs landing together, and no hash function knows which orgs are heavy. Any scheme that does know needs a per-org cost measurement first, and once we had one, the next step was to write placements into a table.

The other fix everyone suggests is to use the idle core by running two processes per container. We shipped the groundwork for that too, but never turned the second process on. Either way, twice the processes would still be placed by a hash that weighs nothing.

Why not pin the heavy ones by hand

The low-tech fix is to move the worst orgs by hand and write it down. I didn’t want that, because pins go stale: every new customer changes the load, and a hand-tuned map is only correct on the day you write it.

The other half of the decision was cost. A few years ago, a measured-load balancer for a fleet of eight processes would have been over-engineering, and I’d have written the pins. With a coding agent doing most of the typing, writing the complex version is cheap. Testing and adjusting it still costs what it always did, and here that meant two wrong load models and a false capacity alarm.

What we built

It shipped as a stack of PRs, each safe to deploy on its own, and the end result looks like this:

flowchart LR
    subgraph C["Dedicated container, 2 vCPU"]
        P["Python process<br/>one asyncio loop"] --> R["LoadRecorder<br/>getrusage + per-org wall time"]
    end
    R -- "report every 60s" --> H[("Redis hash<br/>dedicated-workers-load")]
    H --> B["Balancer<br/>singleton service"]
    B -- "at most 2 moves per run" --> T[("Postgres<br/>dedicated_worker_placement")]
    T -- "row wins, else hash" --> P

A table that falls back to the hash

Placement now resolves in two steps: a row in dedicated_worker_placement wins, and no row means the hash. The fallback is what made the first PR boring to deploy. An empty or unreadable table behaves exactly like before, and so does an org nobody has placed. A freshly onboarded org gets processed with no human step.

A row pointing at a process index outside the reader’s fleet also falls back to the hash. One of our pools always runs a single process and reads placements written for a much larger fleet, and any scale-down creates the same mismatch. Honoring the row would leave the org with no process at all.

There’s still a pinned column for the rare org an operator wants to hold in place. A pinned org counts toward its process’s load like any other; the balancer just won’t move it.

Measuring what one org costs

This was the hard part. Everything on a process shares one event loop, so “how much CPU does this org use” has no direct answer.

Our coding agent’s first design charged each org the time spent inside its worker tasks. My first review question was how that elapsed time was computed, and the answer was the problem: it was wall time. Wall time isn’t CPU. Concurrent tasks overlap, and an org waiting on GitHub looks busier than one actually computing. Add up wall time across orgs and you get more than one core on a process that can’t use more than one.

So the recorder measures two things differently on purpose:

def process_cpu_seconds() -> float:
    usage = resource.getrusage(resource.RUSAGE_SELF)
    return usage.ru_utime + usage.ru_stime

# at the end of each 60-second window (simplified: the real code
# also handles a window in which no org ran at all)
cpu_cores = cpu_seconds / window_seconds
total_busy = sum(owner_busy_seconds.values())
owner_cpu_cores = {
    owner_id: cpu_cores * busy / total_busy
    for owner_id, busy in owner_busy_seconds.items()
}

The process total comes from getrusage and is exact. It decides whether a shard is saturated, and getting that wrong is expensive. The per-org split uses wall time, but only as a share of that exact total, never as an absolute. Getting the split wrong costs one suboptimal move.

Reports go to a Redis hash, not through Datadog, so deciding where an org runs doesn’t depend on a metrics pipeline. Processes with no org on them report too. An empty shard is exactly where work should go, and a silent one would look the same as a dead one.

The balancer

A singleton service reads the reports on every run and plans moves. Nothing happens until a process passes a high-water mark, and once one does, orgs move until the hottest process is back under a lower mark, so it doesn’t flap around a single threshold.

It makes the fewest moves it can and doesn’t repack the fleet. Every move restarts that org’s workers. That costs the customer less than a second of delay, and in-flight work resumes after the restart, but it’s still churn. So there are at most two moves per run, a one-hour cooldown per org, and a move has to be worth it:

# Judge the move by the worst process, not the source
if max(source_after, destination_after) > load[hottest] - MIN_IMPROVEMENT_CORES:
    continue

A move that relieves the hottest process but leaves the destination nearly as hot as the source was just moves the problem to a different index. MIN_IMPROVEMENT_CORES is 0.02 of a core. Without that minimum, a fleet that’s already as level as it gets still finds a move that shaves a rounding error off the maximum, every run.

There’s also a floor no algorithm can beat. The agent’s per-org demand estimate, the same one behind the table above, put total demand around 566% of a core, which over eight processes is a fair share of about 71%, so no placement on eight processes gets the worst one under 71%. One org can’t be split across processes either, so the heaviest orgs push the real floor even higher. The balancer can’t lower that floor; it gets the fleet down to it and keeps it there as customers onboard. When it can’t find a legal move for a hot process, it increments placement.blocked, which is supposed to mean “this fleet needs capacity, not rebalancing.”

It shipped in dry-run, with dashboards and alerts, before it was allowed to move anything.

What didn’t work: the load model was wrong twice

The agent wrote the load model, got it wrong twice, and I approved both versions. The dry-run caught the first mistake, and the second only showed up after the balancer went live.

Peaks don’t add

Version one weighted each org by its daily peak, so the fleet wouldn’t be re-planned around a nightly or weekend lull. A process’s load was the sum of its orgs’ peaks.

Peaks don’t add. Two orgs that each reach 0.4 of a core, four hours apart, cost their process 0.4 at any instant, never 0.8. On the first day of real traffic, the model was off by 2.2x to 5.6x: all eight processes read between 1.2 and 2.6 cores against a ceiling of about 1.0. Every shard looked saturated, so nothing could ever get under the low-water mark and placement.blocked fired on every run.

Nothing moved, because nothing was allowed to. The fix also recorded a daily peak for each process, and scaled that process’s org weights so they added up to that peak. A process’s own peak is the peak of the sum, which is the number we actually wanted. Once the dry-run predictions matched measured shard CPU, we turned the balancer on.

The peak of a capped metric is always 1.0

This one reached production. The recorder can’t observe demand: a process is one event loop, so what it sees is min(demand, 1.0). Peak is the statistic that hits that ceiling first: one busy minute anywhere in the day pins it at 1.0.

By then the fleet had grown to twelve processes, and every one of them read a 24-hour peak between 0.987 and 1.008, while their means ranged from 0.27 to 0.95. Scaling each process to its own peak gave twelve modeled loads that were effectively identical. The balancer compared twelve numbers near 1.0, found no destination under the high-water mark, and gave up. Over two hours: 0 moves, 119 blocked runs, and the fleet sitting at a 4.8x spread between its hottest and coldest process, the exact shape the balancer existed to fix.

The worst part was what blocked was telling us. It stayed lit through the 8 → 10 → 12 expansion and helped drive those resizes, while real utilization sat near half. Once the model was fixed, the agent’s summary of the twelve-process fleet was blunt: “The fleet was never at capacity.” I don’t know whether eight processes would have been enough. We kept the twelve containers anyway, because headroom for the next customers isn’t wasted money. I’d still rather have bought it on purpose.

The fix was to weight orgs by an exponentially weighted mean over a one-hour horizon, folded once per 60-second window with smoothing at 1/60:

def fold_mean(baseline, observed, *, smoothing):
    for key in baseline.keys() | observed.keys():
        baseline[key] = (
            smoothing * observed.get(key, 0.0)
            + (1 - smoothing) * baseline.get(key, 0.0)
        )
    return baseline

An org missing from a window counts as zero, which is what makes this a mean: an org that ran one minute of the last hour costs its process at most a sixtieth of what it costs while running, less the longer ago that minute was. Means add, so the org weights sum back to the process’s own measurement for free, and the rescaling step from the first fix could go.

The mean is capped by the same ceiling, but only a process pinned continuously gets there, and then every org on it is understated by the same factor. That’s safe for the shard being drained and optimistic for the one receiving orgs, so we handle bursts with the water marks instead: a target mean utilization of 0.75 high and 0.65 low. Both marks have to sit above the fleet’s fair share, or a perfectly level fleet reports itself out of room.

Fair share moves with the time of day. It reached 0.61 during peak hours on August 31, so a mark set from the daily average of about 0.55 would have fired blocked every afternoon. blocked now fires only when a process sits above 0.75 and no legal move exists, which happens once peak-hour fair share gets close to 0.75. At that point it means what it says: add a process. A signal the ceiling can’t clip, like event loop lag or stream backlog, would make that call more reliable. We haven’t wired one in yet.

The trade is reaction time. After an hour, a growing org is charged about 63% of its growth, whereas the peak model charged all of it within a minute, and a new org eases in from zero. The balancer ends up late but still right, and an hour of no moves on a fleet that had been unbalanced for days costs less than a move made on one minute of data.

Results

After we deployed the mean-based model:

• The modeled maximum load finally moved off the 1.007 it had been stuck at.
• placement.blocked went from firing on every run to zero.
• The balancer made 12 moves in its first four hours.
• The hottest and coldest processes went from 0.953 and 0.197 cores to 0.694 and 0.261, so the spread dropped from 4.8x to 2.7x.
• The stuck-event monitor that had been firing since July stopped.

No customer reported a delay; the stuck-event monitor was the only thing that noticed.

2.7x sounds like it should bother me more than it does. The balancer relieves anything above the high-water mark and leaves the rest alone; it doesn’t try to make the fleet level. A cold process costs nothing but the idle core we were already paying for, and every move restarts an org’s workers.

The same load-blind hash was also placing a second fleet, the roughly 190 orgs whose webhooks are ingested by dedicated github-in-postgres workers. Placement is now keyed by (fleet, owner_id), and the same balancer is wired up to place that fleet from its own processes’ CPU. It isn’t moving anything yet: reports come first and moves later, same as the first time.

What I’d tell the next team

If your tenants differ in cost by orders of magnitude, hash % N is a lottery, and adding processes buys another ticket. And if a process can only ever use one core, look at the process’s CPU. For weeks, our container metric told us we had room.

The mistake I’d warn you about most: never rank things by the maximum of a capped metric. At the ceiling, everything busy looks the same, and your “needs more capacity” alert will happily tell you to buy hardware you don’t need.

Writing the balancer was the cheap part. The dry-run caught one wrong model. Production caught the other, after its alarm had already helped talk us into four more processes.