---
title: "How do I test AI-generated pull requests before they merge?"
description: "Run the same required checks as any pull request, then test each one against the main it will land on with a merge queue. Batching keeps CI affordable at volume, and Prevention flags new flaky tests."
url: https://mergify.com/guides/test-ai-generated-pull-requests
---

Guide

# How do I test AI-generated pull requests before they merge?

Run your normal required checks on the pull request, then send it through a merge queue so it is tested against the main it will actually land on. Batching keeps that affordable at high volume, and Prevention flags new flaky tests before they merge.

By [Julien Danjou](https://mergify.com/about), Co-founder & CEO of MergifyUpdated November 4, 2026

## In one paragraph

An AI-generated pull request fails the same way a human one does, so it gets the same required checks. What changes is how many are open at once. Each one was tested against a main that has moved by the time it merges, and two green changes can break main together. A merge queue closes that gap: it builds each pull request on the latest main and everything ahead of it, runs your CI on that exact state, and merges only if it passes. Batching stops this from costing one full pipeline per pull request, and Prevention reruns new tests so the flaky ones get flagged while the pull request is still open.

## What changes when an agent wrote the pull request?

The traffic. A type error or a failing test looks the same whoever typed the code, so the checks can stay as they are. But an agent can open a dozen pull requests in the time a person opens one, work overnight, and touch the same files as another agent running next to it.

Every one of those pull requests is tested against the main it branched from. With that many in flight, main moves between the test and the merge far more often, and the classic break gets common: [two changes that each pass on their own and fail once both are on main](https://mergify.com/guides/merge-queue-if-ci-is-fast). A review checklist will not catch this, because neither diff is wrong.

So there are two questions to answer before a merge. Does this change work, and does it still work on the main it is about to land on? Most advice on testing AI-generated code stops at the first one.

## What does each check catch?

Each layer covers something the one before it cannot see, so none of them replaces another:

| Check | Runs | Catches | Misses |
| --- | --- | --- | --- |
| Lint, types, unit tests | Every push to the pull request | Mistakes inside the change itself | Anything that depends on what else merged |
| Human review | Before the pull request can enter the queue | The wrong fix, the wrong design, a test that asserts nothing | A clash with a change that merges after the review |
| Test reruns (Prevention) | While the pull request is open | A new test that passes and fails on the same commit | A test that is consistent and wrong |
| Merge queue run | Just before the merge | A clash with the latest main and the pull requests ahead of it | A bug your test suite does not cover |

The last row is the limit of the whole setup. If an agent writes a test that asserts nothing, every layer but review will wave it through, which is why the human approval stays in front of the queue.

## How do I test a pull request against the main it lands on?

Put a [merge queue](https://mergify.com/learn/merge-queue) between the approval and main. When a pull request reaches its turn, the queue builds a temporary branch holding the latest main, the pull requests ahead of it and this one, opens a draft pull request so your CI runs on that branch, and merges only if it passes. If it fails, the pull request leaves the queue with the reason, and main never sees it.

The approval and the required checks stay where they already are, in your branch's ruleset on GitHub. Mergify reads them, so a pull request enters the queue only once a person has approved it and its checks are green. The queue itself needs one setting:

```
queue_rules:
  - name: default
    batch_size: 5
```

`batch_size` lets five queued pull requests share one CI run. The next section prices it.

To queue a pull request, comment `@mergifyio queue` on it. If you have no queue yet, [setting up a merge queue on GitHub](https://mergify.com/guides/setup-github-merge-queue) takes a few minutes.

## How do I keep CI affordable at this volume?

Test several pull requests in one run. Forty pull requests a day on a 30-minute pipeline is twenty runner-hours a day if the queue tests them one at a time. With `batch_size: 5` that becomes eight runs and four hours, plus a few extra runs on the days a batch fails and the queue bisects it to find the culprit. The rest of the batch still merges.

Two more settings take runs off the bill. A pull request that already contains the tip of the base branch merges with no queue run at all, which [Direct Merge](https://docs.mergify.com/merge-queue/direct-merge) does without any configuration. In a monorepo, [scopes](https://mergify.com/learn/monorepo/merge-queue) keep a frontend change from waiting behind a database migration, which matters once agents work across the whole repository at the same time. [Cutting CI costs with batching and bisection](https://mergify.com/guides/cut-ci-costs-batching-bisection) has the math for picking a batch size.

## How do I catch flaky tests before they land?

A new test that passed once has proved very little, and a repository with agents in it gains a lot of new tests. [Prevention](https://docs.mergify.com/test-engine/prevention), part of [Test Engine](https://mergify.com/product/test-engine), reruns the tests a pull request adds on the same commit, within a CI time budget you set. A test that gives two different results is flagged as flaky, with its health status and a confidence level next to the pull request, so the reviewer can send it back before it reaches main.

To turn it on, install the [Mergify plugin for your test framework](https://docs.mergify.com/test-engine/test-frameworks) and set a Prevention Budget on the Test Engine configuration page. Tests that are already flaky on main are a different job, and [detecting and quarantining them](https://mergify.com/guides/detect-and-quarantine-flaky-tests) keeps them from knocking good pull requests out of the queue.

## What should an agent do with a red check?

Ask whether the test is flaky before rewriting anything. A red test looks the same whether the change broke it or it fails one run in twenty, and an agent that guesses wrong spends its run rewriting code that was fine. The Mergify CLI answers from the test's history:

```
mergify tests show 'tests/billing/test_invoice.py::test_rounding' --json \
  | jq '.tests[] | {test_name, health_status}'
```

`health_status` is `healthy`, `flaky` or `broken`. Read the JSON rather than the exit code, which only reports whether the lookup worked. The command needs an `admin` application key or a user credential from `mergify auth login`. [Merging pull requests from coding agents](https://mergify.com/guides/merge-pull-requests-from-coding-agents) covers the rest of the loop, including what to put in the agent's instructions.

## FAQ

### Do AI-generated pull requests need different CI checks from human ones?

No. Keep one set of required checks. A separate, softer path for agents gives them a way around the strict one, and it leaves you with two pipelines to maintain. What changes is the number of pull requests in flight, which is why the merge step needs a queue.

### Does a merge queue replace code review?

No. Review answers whether this is the right change. The queue answers whether the change still works on the main it is about to land on. Require an approval on the branch and you get both, in that order.

### Can an agent's approval count as the required review?

Only if you let it. A required approval on GitHub counts any reviewer with write access, and an agent running under its own account can have that. To require a person, turn on required review from code owners and list a team of people in CODEOWNERS.

### Will a merge queue slow the agents down?

The agent is not waiting on it. It opens the pull request, it gets queued, and the agent moves on to the next task. Batching is what keeps the queue itself from falling behind when pull requests arrive faster than one CI run per merge can clear them.

### Does this work with my CI?

Mergify works on GitHub, with any CI that reports its results there: GitHub Actions, Buildkite, Jenkins, CircleCI and others. The queue does not run your jobs. It opens a draft pull request on a temporary branch, and your CI runs on it the way it runs on any pull request.

Full reference: [merge queue](https://docs.mergify.com/merge-queue), [batches](https://docs.mergify.com/merge-queue/batches) and [Prevention](https://docs.mergify.com/test-engine/prevention).

Last updated: November 4, 2026

## Test every pull request against the main it lands on.

Put a merge queue in front of main, whoever wrote the code. Free for up to 5 users.

[Get started](https://dashboard.mergify.com/)[See how Merge Queue works](https://mergify.com/product/merge-queue)