How to let an AI touch your test suite without losing sleep

Part three of the series on moving QA from classic automation to AI-assisted. Part one was about feeding the model real context; part two was about the richest source of it. Both ended on the same promise: nothing an AI suggests reaches your systems without passing a gate. This post is that gate, in detail — because it's the part that decides whether AI-assisted QA is safe to run near a real team, or a very fast way to corrupt a test suite.

The fear everyone has is reasonable: "what if the AI is wrong and it edits 400 test cases?" The wrong answer is "we'll use a better model so it won't be wrong." Models are wrong sometimes; that's a fact you design around, not away. The right answer is that being wrong shouldn't be able to do anything on its own. So you never let a suggestion become a change without it surviving four steps.

1. Dry-run: simulate, change nothing

Every write starts as a dry-run. The agent (or the human) produces exactly what it would do, against the real current state, and changes nothing. No API writes, no file edits — a simulation that answers "if we did this, what would happen?"

This sounds trivial and it's the most important step. It moves the entire operation from "trust me" to "look for yourself." Most bad changes die here, the moment a person sees the simulated result and says "that's not what I meant."

2. Plan + hash: make it diffable and pin it

The dry-run produces a plan — a human-readable, reviewable diff of every change: this case's steps go from X to Y, this section gets created, these five cases move. And critically, that plan gets a hash — a fingerprint of its exact contents.

The hash is the part people skip, and it's the whole point. It pins the plan. When you later confirm, you confirm this exact plan by its hash — not "whatever the tool recomputes at apply time." Without it, there's a silent gap: you review a plan at 2:00, the world shifts, and at 2:05 the tool recomputes and applies something subtly different from what you approved. The hash closes that gap. You reviewed plan a1b2c3; you apply plan a1b2c3; if the plan would now be different, the hash won't match and it refuses.

3. Re-check against live, right before applying

Between reviewing a plan and applying it, time passes — sometimes seconds, sometimes a day. In that window, someone else may have edited the same case, or the remote state may have moved. So immediately before executing, the system re-checks the plan against the live system. If the ground it was built on has shifted, it stops and makes you look again rather than blindly overwriting a change that landed after your baseline.

This is the step that turns "I ran the thing I reviewed" into "I ran the thing I reviewed, and it's still safe against reality."

4. Human confirms — explicitly, by hash

Only now does anything happen, and only because a human said so: --confirm --plan-hash a1b2c3. Not a checkbox that's on by default. Not an agent deciding it's confident enough. An explicit, per-operation confirmation that names the exact plan being applied.

For an AI operator this is a hard rule, not a preference: the agent can draft, simulate, and plan all day, but it must not run the confirm step unless a human explicitly asks. The interesting design choice is where that rule lives — not in a prompt the model might forget, but in the executable procedure itself. The tool refuses to write without the hash and the flag, so "the AI got confident and skipped review" is not a failure mode that can occur, regardless of what the model believes.

A blast-radius guard, on top

One more layer worth adding: a soft limit on how much a single operation can change. If a plan would touch more than, say, a few dozen cases, that's not automatically wrong — but it's exactly when a slightly-too-loose match turns into a mess. So a large plan requires a deliberate override, not a routine confirm. It's a circuit breaker: the cost of being wrong is capped at "annoying" instead of "portfolio-wide."

Why this is the real product

Here's the thing I've learned building this a few times: the retrieval quality, the model choice, the clever prompting — those are the parts people demo. This chain is the part that determines whether any of it is usable. Great context with no guard is a liability. Mediocre context with this guard is safe to iterate on, because the worst case is "a bad suggestion you caught in the dry-run," not "an incident."

And note none of it is AI-specific. This is just good change-management — the same discipline you'd want for a human doing bulk edits. AI doesn't change the pattern; it just makes the gate non-negotiable, because now the thing proposing changes doesn't get tired, doesn't feel doubt, and will absolutely apply a confident mistake if you let it.

If you're anywhere near letting an agent touch your QA systems — drafting cases, updating a suite, provisioning test data — this chain is the first thing I'd build, before any of the clever parts. It's usually where I start.