Our Frontend Was Guessing What Our Parser Accepts. It Was Wrong for 70 of 75 Attributes.

A few weeks ago I compared the operator menus in Mergify’s condition editor against the engine that actually parses conditions. The menus were wrong for 70 of our 75 attributes (offering operators the parser rejects, hiding ones it accepts), and nobody had complained.

The problem

Mergify automation runs on conditions. You write things like label ~= "^bug" or #approved-reviews-by >= 2; the engine parses them and decides whether your pull request gets queued or left alone. On the wire, a condition is an opaque string. The engine’s parser is the only authority on what’s valid.

The dashboard has a structured editor for these conditions: pick an attribute and an operator from a menu, then fill in a value. Which operators belong in that menu? The frontend guessed. It re-derived them from each attribute’s JSON Schema type and format:

function opsFor(type?: string, format?: string): string[] {
  if (type === 'boolean') {
    return ['='];
  }
  if (type === 'integer' || type === 'number' || format === 'date-time') {
    return COUNT_OPS;
  }
  if (type === 'array') {
    return ['=', '!='];
  }
  return ['=', '!=', '~='];
}

Twelve lines, and they look completely reasonable: booleans get equality, numbers and dates get ordering, arrays get plain equality, and everything else gets regex on top. The problem is that the engine’s parser agrees with almost none of it. When I audited every attribute, only 5 of 75 matched.

The damage came in two flavors.

Over-offering (19 attributes). The picker let you build conditions the engine rejects. The worst case was booleans: the engine’s grammar has no draft = false. A boolean condition is the bare attribute (draft) or its negation (-draft). The picker didn’t know that, so it fabricated = false values the parser would refuse. Timestamps had the same issue with = and !=, which the engine rejects on updated-at and friends.

Under-offering (54 attributes). This one is worse, and it’s the reason nobody filed a ticket. For text and list attributes, the picker silently hid *= (glob) and ~= (regex on lists). The ordering operators never showed up either. You couldn’t build label ~= "^bug" or files *= "*.py" through the picker at all, even though the engine happily accepts both. Nothing errored. The feature just looked like it didn’t exist.

On top of that, two entries in the picker (check and co-authors) aren’t condition operands in the first place. Any condition built on them was rejected outright.

If you’re doing the arithmetic: five attributes managed to be wrong in both directions at once. Four nullable timestamps have an anyOf schema shape that hid their type from the heuristic entirely, and sender-permission hid its type behind a $ref; all five fell through to the string default, over-offering regex (plus equality on the timestamps) while under-offering ordering. That’s 19 + 54 minus the 5 overlapping, plus the 2 non-operands: 70. And yes, I count a hidden operator as wrong. A menu that says a capability doesn’t exist is lying to the user just as much as one that offers garbage.

The asymmetry explains the silence. An over-offered condition fails at save time with a validation error, which the user at least gets to see and route around. An under-offered operator is invisible: the user concludes the product can’t do it and moves on. Full disclosure: the per-attribute menus were only three weeks old when I audited them, so the silence hadn’t had much time to become damning. But the editor they replaced had offered one flat operator list for every attribute since its first commit, equality on booleans included, and nobody had complained about that either. Silence was never evidence the menus were right.

I didn’t find this because of a bug report. I was wiring allowed template variables into our docs for an unrelated feature. Along the way I looked at how the picker decided its operator menus and realized the frontend had its own private copy of the engine’s grammar.

What we did

The tempting fix is to make opsFor smarter. It’s one TypeScript function; an afternoon of work would cover the known cases. We didn’t do that, because the function would be right for exactly as long as nobody touches the parser. Drift is the charitable reading, though: this heuristic was born wrong, written against a parser that already accepted glob and regex on lists, and nobody ever validated the guess. A patched heuristic dies the same death, just later.

There’s a more radical fix, and someone will suggest it: stop storing conditions as strings and make them structured data, so a plain JSON Schema enum could validate operators with no custom keys at all. It would work. It would also force a migration on every existing config, for a gain our users wouldn’t feel: a condition that reads like a query language is exactly what makes the YAML pleasant to write. The strings stay.

Instead, the engine now publishes its rules. Our config JSON schema is generated from the engine’s Pydantic models, so we added two custom extension keys to every condition attribute:

"draft":      { "x-allowed-operators": [],
                "x-modifiers": [{"type": "negate"}] },
"updated-at": { "x-allowed-operators": [">=", "<=", "<", ">"],
                "x-modifiers": [] },
"label":      { "x-allowed-operators": ["=", "!=", "~=", ">=", "<=", "<", ">", "*="],
                "x-modifiers": [{"type": "negate"}, {"type": "length"}] }

The keys carry meaning in their shape. An empty operator list means the attribute takes no operator at all, so the editor commits it as a unary leaf instead of inventing = false (booleans are the only attributes like that). A missing key means the attribute isn’t a condition operand, so the picker drops it. x-modifiers lists what the attribute accepts beyond plain comparison: whether it can be negated with -, and whether the # length form (#label > 3) is available.

Under the hood

We compute the annotations inside a __get_pydantic_json_schema__ hook on the attributes model, straight from the parser’s own operator and modifier tables:

json_schema = handler(core_schema)
resolved = handler.resolve_ref_schema(json_schema)
for attribute, prop in resolved.get("properties", {}).items():
    if attribute in parser.CONDITION_PARSERS:
        prop["x-allowed-operators"] = parser.get_allowed_operators_for_display(
            attribute,
        )
    modifiers = _condition_modifier_schema_extra(attribute)
    if modifiers is not None:
        prop.update(modifiers)
return json_schema

There’s no per-field duplication anywhere. When someone changes what the parser accepts, the generated schema changes with it. A parity test locks the two together, and it does more than compare generated output to itself: a parametrized test pins the expected operator set for each parser class as hardcoded data (yes, a second copy of the grammar, but one that fails loudly in CI instead of silently in a menu), and a companion assertion checks that our display vocabulary covers every operator token the parser defines. Add an operator to the parser and CI fails until the schema story accounts for it. Those tests are the actual fix; the x- keys are just plumbing.

On the frontend, opsFor is gone. The picker imports the generated schema JSON (a checked-in artifact the dashboard build bundles) and reads the keys directly. The way it reads the keys is unit-tested too, down to the unary booleans and the dropped non-operands. TypeScript was already covering the opposite direction, and we gladly kept it: the set of attribute names comes from the imported JSON itself, so when the engine grows a new attribute, the dashboard’s category map stops compiling until someone routes the newcomer into the UI. Engine-adds-something-and-frontend-forgets-it was a build error before this work; engine-and-frontend-disagreeing-on-operators is what we fixed.

The docs consume the same keys. They never had per-attribute operator tables before, and the hand-written prose was partly wrong too (it said the # length modifier applied only to lists). Now the tables render from the schema artifact. One parser feeds both the picker and the docs, and neither can disagree with it anymore.

flowchart LR
    subgraph engine [Engine]
        P["Condition parser<br/>(operator & modifier tables)"]
        H["pydantic schema hook"]
        S["mergify-configuration-schema.json<br/>x-allowed-operators / x-modifiers"]
        P --> H --> S
        T["Parity test in CI"] -.locks.-> P
        T -.locks.-> S
    end
    S --> D["Dashboard condition picker"]
    S --> DOC["Docs attribute tables"]

One thing we kept deliberately narrow: these keys are UX guidance. Conditions still travel as opaque strings and the engine parser remains the final authority. If a stale frontend build over-offers something, the worst outcome is the same validation error we had before, never a wrongly accepted config. A stale build could also under-offer, which is the silent failure mode all over again. But the schema ships inside the same build as the picker that reads it, and the engine and dashboard deploy together, so the window is one deploy at most.

Results

The work shipped as a stack of four pull requests; the first went up June 29 and the last merged July 6. For every attribute the schema lists, the picker can no longer offer an operator or a modifier the engine rejects, and the missing operators finally show up in the menus: glob matching (which neither editor had ever offered) and regex on lists. Ordering on text made it in as well. The engine still validates values at save time, so a hand-typed broken regex will bounce; the schema keys cover grammar, not content.

I won’t pretend to have adoption metrics after a week. The number I do have is the audit: the old heuristic was right for 5 attributes out of 75, and for the attributes the schema covers, that class of bug is gone. The other win is for us: no more operator logic to maintain on the frontend, and the docs tables update themselves. Two remnants survive on the frontend: a permissive fallback list for a handful of legacy attribute names, and a small numeric set for the counted # forms. Once CI enforces the extension keys on the schema, the fallback goes too.

Where this leaves us

If your frontend re-derives rules that your backend already knows, you have one implementation and one rumor.

The rule we settled on is simple: information moves into the schema when it makes the UX better, either by letting the UI validate a value without a round-trip to the API, or by removing a choice that was never valid to begin with. The schema was the obvious transport because every consumer already loads it; no new endpoint or codegen step, and no shipping the parser to the browser. Things like categories and icons stay in the frontend; grammar belongs to the parser. The generated artifact is public if you want to see the keys in the wild: mergify-configuration-schema.json. I expect more of these x- keys to appear as we find the next rumor.