Semantic search is only half a QA knowledge base

Six weeks ago I wrote about making a test portfolio searchable by meaning, and later about pulling the team chat into the same index. Both posts are true, and both stop one step short. Once the index works, a different problem shows up, and it looks like success.

An agent asks "what happens on a second withdrawal inside the cooling-off window?" and gets an answer in a second: a test case from two releases ago, a Slack thread where someone decided it, and a paragraph of the spec. All three are close in meaning. One of them is no longer true. Nothing in an embedding can tell you which.

That's the limit of retrieval, and it isn't a model problem. Similarity says this sounds like what you asked. It cannot say whether the statement still holds, who observed it, when, or what contradicts it. A knowledge base that can only rank by similarity will hand you a confident, cited, wrong answer, and the citation makes it worse, because a reviewer sees a source and stops checking.

So the semantic index is deliberately only half of my knowledge layer. Retrieval finds; the record asserts.

The half that finds

This half is what the earlier posts described, and it's the messy corpus: a few thousand test cases, tickets with their comments, chat threads stored as whole conversations rather than isolated messages, research notes from walking the live product. It gets indexed twice, by keyword and by meaning, on a local encoder. Every hit carries where it came from: a case id, an issue key, a permalink into the thread.

Its job is narrow. It makes a record reachable. It has no opinion about whether the record is right, and it shouldn't.

The half that asserts

The other half is plain files in git, and the format is boring on purpose: one statement per file, a YAML header, links as data instead of prose. I borrowed the shape from the Open Knowledge Format idea, a knowledge base as a folder of Markdown files with front matter, one idea per file, related by links, and applied it to the thing QA actually needs: a specification with provenance.

A rule looks like this:

---
id: PAY-22
feature: payments
source: live site, 2026-09-02
observed: 2026-09-02
status: observed          # observed | declared | blocked
cases: [PAYMENTS-016, PAYMENTS-017]
discrepancies: [DF-23]
---

# PAY-22

Top-up limits are enforced on the client; the API accepts amounts outside them.

**Source:** live site, 2026-09-02 — research/ui/payments/findings.md

## Notes

> ⚠️ **Discrepancy, 2026-09-03.** Backend code rejects amounts above the daily
> limit with 422, so the UI limit and the API limit differ. Tracked as DF-23.

Four things in that header do the work that similarity can't:

• source and observed say who saw this and when. A statement without a source line is not allowed in; the checker rejects it.
• status says what an agent may do with it. observed means the mechanism is visible in code or on the live product, so cases can be written against it. declared means it's a promise from the terms page or the marketing site with no observable mechanism, so a case can only state an expectation. blocked means the rule sits under an open dispute, and no cases get written until a human resolves it.
• cases is the back-link: which cases depend on this rule. It's computed from the cases themselves, not typed, so it can't drift.
• discrepancies ties the rule to numbered findings, the defects and divergences that carry their own ids.

And one rule of conduct that matters more than the format: a rule is never rewritten when reality disagrees with it. The disagreement goes underneath as a dated note, and a human decides which is true, using a fixed hierarchy of sources: backend code beats frontend code, which beats the live site, which beats the written spec. The agent that discovered the contradiction gets to record it. It does not get to decide it.

Why one file per statement, not a wiki page

Because an agent then reads one rule instead of a whole feature page with its defect list, its screenshots and its history. Because the links between a rule, the cases that cite it, the test that implements each case and the last run result become data, joinable by script, instead of sentences someone has to keep in sync. Because a diff of one file is reviewable in a pull request, and a wiki edit is not. And because the whole base is a folder of text: any assistant, any tool, no export.

That joinability is the payoff. One script checks the form: header fields present, id matches the filename, source line present, back-links match what the cases actually reference, the per-feature index regenerated rather than hand-edited. A second script walks rule → case → test marker → last result from the test-management tool and writes a coverage map in which an observed rule with no case is a named hole, not a feeling. No model is involved in either. Everything they need is in the headers.

The part I didn't expect: the curated half doesn't need RAG

On the personal project where I run this end to end, the specification is around 140 rules across seven features. That fits in an assistant's context whole. A vector index over it would be engineering for its own sake, so I didn't build one, and the roadmap says why: semantic search goes over the research notes once there are enough of them to be unreadable, not over the spec.

That's the rule I'd now start from. Index the evidence. Curate the assertions. Don't embed what you can simply read. Retrieval earns its place over the corpus that's too big and too messy to hold in your head: the legacy suite, the ticket history, the chat. The curated layer is small by construction, because every statement in it had to earn a source line.

What the mix buys you

Put the two halves together and an agent can take a fuzzy question, use the index to reach the candidates, and then answer from the records, citing a file with a status and a date for every sentence. Where the index and a record disagree, the record wins, and the disagreement is written down. That last part is how the base gets better instead of drifting: every contradiction becomes a dated note that a human closes.

The same discipline governs what the agent is allowed to write back, which is its own article. Reading with provenance and writing with a plan hash are two ends of one idea: an agent gets every capability except the one that decides what's true.