We Stopped Our Config Builder From Destroying Your YAML, Then Found a Worse Bug Next to It

Our no-code builder for Mergify configs used to strip comments and anchors out of your .mergify.yml on every commit, and for years the workaround was simply to lock anchor-using configs out of the builder entirely. Five weeks ago we fixed the real problem: patch the original document instead of re-dumping it. This week, while we were extending that work with a raw-YAML editing tab, an audit of an adjacent part of the builder surfaced something worse: a parser bug that had been silently inverting a security condition in production.

The problem

Mergify configs are YAML: Merge Queue rules, Merge Protections, Workflow Automation, Priorities. We also ship a no-code builder for all of it, because reading raw YAML isn’t for everyone. For a long time, the two didn’t get along.

The builder worked by parsing your YAML into a plain JavaScript object, letting you click around, then dumping that object back into YAML on commit. Dump is the operative word: yaml.parse throws away comments, anchors, aliases (&x / *x), and the exact formatting you wrote. Round-trip a config with a <<: *defaults merge key through the builder and you’d get back a file where the anchor is gone and the shared block is duplicated inline everywhere it was referenced.

Our answer, for a long time, was to not let that happen: any config using anchors was simply excluded from the builder. It wasn’t framed as a workaround for a bug. The assumption was that the builder mostly served newcomers getting a tour of what Mergify could do, and anyone reaching for anchors had already graduated to hand-writing YAML.

flowchart LR
    subgraph before["Before: re-dump"]
        A1["YAML file<br/>(comments, anchors)"] --> A2["yaml.parse()<br/>plain JS object"]
        A2 --> A3["builder edits<br/>the object"]
        A3 --> A4["dumpConfig()<br/>re-serialize"]
        A4 --> A5["YAML file<br/>(comments and anchors gone)"]
    end
    subgraph after["After: patch in place"]
        B1["YAML file<br/>(comments, anchors)"] --> B2["yaml.parseDocument()<br/>CST-backed Document"]
        B2 --> B3["builder edits<br/>become DocumentPatches"]
        B3 --> B4["patch applied<br/>to the live Document"]
        B4 --> B5["YAML file<br/>(comments and anchors intact)"]
    end

What we did first: stop destroying the file

In June, we rebuilt the commit path on a different contract: the YAML document is the source of truth, the builder is a projection onto it. Every builder edit becomes a patch applied to the parsed document, never a fresh object dumped back to text.

The yaml npm package gives you this if you ask for it. yaml.parse() returns a plain object and forgets everything about formatting. yaml.parseDocument() returns a CST-backed Document that retains anchors, anchor names, comments, and merge keys. You just have to never call .toJS() on it, because that’s the method that throws all of that away.

export function parseConfigDocument(yamlString: string): yaml.Document.Parsed | null {
  if (!yamlString || !yamlString.trim()) return null;
  const doc = yaml.parseDocument(yamlString);
  if (doc.errors.length > 0) return null;
  return doc;
}

We shipped the primitives (parseConfigDocument, a generic DocumentPatch type, a shared list-diffing helper) as one PR with no consumer wired up yet, then rewired each surface onto them one at a time over the following week: Commands Restrictions, Workflow Automation, Merge Protections, then the queue page’s pending-changes bar. Once every surface patched the original document instead of re-dumping it, the anchor lockout came off. It had been a workaround for a problem that no longer existed.

What we did next: an actual YAML tab, and a way to say “I can’t do this safely”

Fixing the commit path made the no-code builder safe. It didn’t give you anywhere to edit YAML directly next to it. That’s what we shipped this week: a Builder/YAML tab pattern, first on Workflow Automation, then carried over to Merge Protections and Queue Configuration. Switching tabs round-trips losslessly. Committing from the YAML tab pushes the raw text verbatim, and invalid or schema-invalid YAML blocks the commit instead of being accepted anyway.

That surfaced a question the June work hadn’t needed to answer: what happens when the builder hits a rule it simply can’t represent? Some edits can’t be expressed as a patch. If a rule uses an anchor the builder’s data model doesn’t understand, the patch has to fall back to overwriting that node from scratch, and every comment or alias inside it is gone. DocumentPatch now carries a lossy flag for exactly this case:

/**
 * `true` when applying this patch cannot preserve the formatting of the nodes
 * it touches. It re-creates a node from plain data, so any comment, anchor or
 * alias that lived inside the node it replaces is destroyed. The patch is
 * still applied (the resulting config is correct), but the surface must warn
 * the user.
 */
lossy?: boolean;

A rule the builder can’t safely round-trip gets flagged read-only instead of quietly rewritten. You can still see it, but you edit it from the YAML tab. The same logic guards an unparseable committed file: if the config on disk doesn’t parse, the builder refuses to render a possibly-wrong picture of it and defers to the raw text. Priorities and Commands Restrictions, which are plain forms rather than drag-and-drop builders, got the lighter version of the same thing: a warning before any commit that would have to re-dump and reformat the file.

The part that actually decides whether a rule is safe to hand to the builder isn’t the lossy flag itself. It’s a representability check: re-serialize what the builder would write, compare it byte-for-byte against the original, and if they don’t match beyond one allowed cosmetic normalization (collapsing operator spacing), treat the rule as unrepresentable and leave it alone. Fail closed, not fail silent. That same idea, applied one level down, is what exposed the next bug.

The bug next door

Part of hardening this work meant looking hard at the condition editor, the part of the builder that turns strings like #approved-reviews-by>=1 into the visual condition tree it displays. For close to two hours I ran an adversarial audit against a live test repo, using Claude subagents to stress-test condition strings nobody would type by hand: double-spaced operators (title ~= ^feat), or a negation stacked with a length marker (-#approved-reviews-by>=1). It came back with several findings; the worst was this.

The parser silently dropped a leading negation on a condition like -#approved-reviews-by>=1, which inverted the rule: the condition started matching exactly the cases it was written to exclude. That had been live in production. Nobody had reported it to us, but we hadn’t gone looking either: no support-ticket search, no error-log query, just silence. That’s not the same as “verified safe,” and it’s the uncomfortable part of this story. The only reason we know about the bug at all is that we went looking for it directly, not because our existing signals would have told us.

The root cause was a regex that was never anchored to the start of the string:

// before
const extractorRegex =
  /(?<prefix>[-|#])?(?<attribute>...)(?:\s?(?<operator>...)...)?$/g;

Because it wasn’t anchored with ^, the match could slide past the negation. It also allowed only a single space around the operator, so title ~= ^feat (two spaces instead of one) failed to extract the attribute or operator at all and got silently rewritten. The fix anchors the match and captures negation and the length marker (#) as independent groups, since a condition can carry both at once and they aren’t interchangeable:

const extractorRegex =
  /^(?<negation>-)?(?<length>#)?(?<attribute>[\w-]+)(?:\s*(?<operator>...)\s*(?<value>.+))?$/s;

The regex fix alone isn’t enough to trust against the next edge case nobody thought of. What actually backstops it is the same representability check from the previous section, now applied to individual condition strings: isConditionStringRepresentable re-parses and re-serializes a condition, and if the result doesn’t match the original byte-for-byte, the condition is flagged and kept exactly as written rather than “cleaned up.” A parser that gets an edge case wrong is a bug you’ll eventually find. A parser that gets an edge case wrong and silently rewrites the file anyway is a bug that hides.

Staying with a regex here instead of writing a small tokenizer was a deliberate call, not inertia: the condition microsyntax is tiny and stable (an optional negation, an optional length marker, an attribute, an optional operator and value), and that’s exactly the kind of shape a regex handles fine as long as it’s anchored and its groups are independent. The representability check is what makes that an acceptable trade-off: it doesn’t matter whether the regex is theoretically airtight, because anything it gets wrong is caught and refused rather than silently shipped.

What didn’t work, and what we’d still like to know

The June rollout wasn’t a big-bang rewrite: the shared patch primitives shipped inert first, each surface got rewired in its own small PR after, and that’s on purpose, not an accident worth calling a mistake. The part I don’t have a clean answer for is the condition-parser bug itself: how long it had actually been live, or how many real configs used a negated, length-marker condition with irregular spacing. Given how much of the last few weeks of work leaned on AI-assisted iteration, I don’t have full personal recall of every early attempt at the patch layer either. That’s its own small lesson about keeping enough of the “why” written down as you go, not just the diffs.

Results

The commit path across every config surface (Commands Restrictions, Workflow Automation, Merge Protections, and Queue) now preserves anchors, aliases, and comments, and the restriction that used to lock anchor-using configs out of the builder is gone. On top of that, three surfaces now have an actual raw-YAML tab, rules the builder can’t safely represent are flagged read-only instead of quietly mangled, and the condition editor no longer inverts a negated condition it misreads.

We don’t have adoption numbers for the new YAML tab yet since it only shipped this week. The clearer result so far is the one we didn’t plan for: a real, in-production correctness bug got caught by deliberately auditing an adjacent part of the same system, instead of waiting for a customer to hit it.

Lessons

If your tool round-trips structured data by parsing it into an object and dumping it back out, you will eventually destroy something the user cared about: a comment, a formatting choice, an anchor. Dumping more carefully doesn’t fix that; patching the original document in place does, as long as you fall back loudly whenever you can’t preserve it, and refuse to guess when you can’t tell if you got it right. That same “refuse to guess” instinct is worth pointing at the parts of your editor nobody exercises by hand, because that’s exactly where a rule silently flips its meaning and nobody notices until someone goes looking.