The delete nobody noticed for three weeks

A short war story, because the lesson in it is one almost every syncing system eventually teaches the hard way.

We had a tool that kept a local, queryable copy of our test management system — pull the cases down, work with them fast offline, push reviewed changes back. It worked well for months. Then one afternoon a completely unrelated operation failed with a vague error referencing an ID that, as far as our local copy was concerned, existed and was perfectly fine.

The trace-back

The ID pointed at a case that lived in a section. The failing call insisted the section wasn't there. Our local database said it was — with all its cases, exactly as of the last sync. Both were "right." The section had been deleted directly in the UI, by a person, weeks earlier, entirely outside the tool. And nothing had noticed.

The unsettling part wasn't the deletion. People delete things; that's allowed. The unsettling part was the three weeks of silence. For three weeks, our source of truth and our copy of it disagreed about something as basic as "does this section exist," and every dashboard, every search, every plan we ran off the local data was quietly built on a section that was already gone.

The blind spot: incremental sync can't see a deletion

Here's the root cause, and it generalizes far beyond one tool. Our sync was incremental — the efficient, obvious design. Each pull asks the remote "what do you have?" and updates the local copy to match. It's fast because it only moves what exists.

But think about what that can and can't detect. A new case? Seen — it's in the response. An edited case? Seen — the content differs. A deleted case or section? Invisible. A deletion is the absence of something in the response, and "pull what exists" has no way to notice an absence. To catch a deletion you have to compare the full set of what you had against the full set of what's there now — which is exactly the expensive, full reconciliation that incremental sync was designed to avoid.

So the tool did precisely what it was built to do, and that was the bug. Not a crash — a category of change it was structurally blind to.

The fixes, in order of honesty

The immediate fix was boring: reconcile, notice the orphaned local rows, and clean them up. The real fix was a mental one — accepting that "pull the changes" is never a complete model of reality, and that any system syncing "what changed" needs a periodic full reconciliation to catch what quietly disappeared or was altered out of band. Not on every operation (too slow), but on a cadence, and definitely before you trust the local copy for anything that matters — a release readout, a coverage number, a bulk change.

There's also a deeper governance point. The deletion happened outside the tool, in the UI, with no trace in any workflow that would surface it. The more your team can change the source of truth through paths your tooling doesn't see, the more your tooling's view drifts from reality — silently, and in exactly the direction that bites you at the worst moment.

Why this is a QA story, not just a sync story

QA lives or dies on the gap between "our view of the system" and "the system." A test suite is a model of the product; a local mirror is a model of the test suite; a coverage number is a model of the model. Every layer of model is a place where reality can drift away unnoticed — and incidents love to hide in that drift. The specific bug here was an unnoticed delete. The general lesson is to stay suspicious of any number or view that assumes your model is still faithful to reality, and to build the reconciliation that checks it rather than hoping.

The teams that get burned aren't the ones with bad tools. They're the ones who forgot their tools show a model, and started trusting it like the territory.

If you want a second set of eyes on where your QA view might be quietly drifting from reality, that's the kind of thing I do.