How it works
What actually happens when we verify your system
The mechanism, without the sales frame. If you build this kind of thing for a living, this is the page to disagree with.
It is a contract you implement over your own system, not a way for us to call your model.
We never reach inside your stack. You wrap a function you already have so that it takes a test case and runs it through the whole pipeline exactly as a real user request would - your system prompt, your retrieval, your model call, your guardrails, your post-processing - and returns the answer plus a description of what produced it.
It is not a test harness. It holds no test cases, it makes no judgements, and it stores nothing. It is a seam.
Why this shape
The two obvious alternatives, and what each loses
Both were tried on paper first. They are written down because a reader who has built eval tooling will reach for one of them immediately, and the answer is not that they are lazy - it is that each one measures something slightly different from the thing in production.
- Call your model directly, with your key
- It measures a model in a vacuum. Migration regressions do not live in the model - they live in the interaction between the model and your prompt, your retrieval and your output parser. The canonical one is a candidate that quietly drops a required qualifier on eight cases. Testing the bare model would never have surfaced it.
- Call your product API
- It loses the retrieved sources, which is the single most valuable field we ask for. Without knowing what your retrieval actually returned, “unsupported by the corpus” cannot tell a hallucination apart from a passage we simply do not have a copy of.
One run
What actually happens, in order
The direction of travel is the part worth reading twice. Every connection is opened by your side, outbound - which is why this needs no firewall rule, no allowlisted address and no new authenticated endpoint reaching production.
Presoja
A run is due, or something changed
Weekly on the full set, daily on a smoke set, or within the hour of a change we detected. The cases for that run go into a dispatch queue for your deployment.
Your network
Your process asks for work
The runner you deployed authenticates with its key, registers what it can do, and polls. Outbound, HTTPS, port 443. Nothing of ours holds a route into your network.
Your network
Your handler runs one case
The case arrives as an input and a tenant. Your own pipeline runs it - retrieval, prompt, model, guardrails, parsing - exactly as it would a real request from that customer.
Your network
The answer goes back, with its fingerprint
The output, the sources your retrieval actually returned, the tenant echoed back, and a description of what produced the answer. The wrapped model client fills most of that in without you writing it.
Presoja
We score it, and seal the run
Against the gold answer confirmed by your expert, with the scorer version pinned before the run started. The result is hash-chained into the record your customer reads.
Then it repeats - on the cadence, forever
The contract
One case, on the wire
Identical over both transports. This is the whole of what crosses the boundary - there is no second channel, and nothing here is a sample of your production traffic.
What we send
{
"execution_id": "ex_01JR8...", // ours to mint, so a retry is idempotent
"run_id": "run_01JR7...",
"case_id": "cv_8842", // the case version, not the case
"tenant": "insurer-a", // which of THEIR customers this case belongs to
"input": { "question": "Is subsidence covered under a standard policy?" },
"overrides": { "models": { "generation": "gpt-5.1-2026-04-11" } }
}What comes back
{
"execution_id": "ex_01JR8...", // exactly as received, so a retry collapses
"output": "Subsidence is covered where the policy schedule lists it ...",
"retrieved_sources": [
{ "id": "policy-wording-v7#s4.2", "uri": "corpus/policy-wording-v7.pdf",
"span": [1840, 2110], "score": 0.83 }
],
"tenant": "insurer-a", // echoed back; a mismatch fails the whole run
"fingerprint": {
"models": { "generation": { "model_id": "gpt-5.1-2026-04-11" } },
"system_prompt_sha256": "9f2c...",
"retrieval_index_version": null,
"config_sha256": null
},
"overrides_applied": { "models": { "generation": true } },
"overrides_unsupported": [] // returned always, not only on failure
}- The execution id is ours to mint
- So a retry is idempotent rather than a second row, and so a result that arrives twice is recognisably the same result.
- The tenant is echoed back, and a mismatch is fatal
- Not to the case - to the whole run. A system that answered one case under the wrong customer’s configuration may have answered others that way, and there is no way to tell which from the outside.
- Retrieved sources are what makes groundedness real
- With them, “unsupported by the corpus” is a finding. Without them it is a guess about a corpus we cannot see.
- Every field is better absent than fabricated
- A system that cannot report its retrieval index version returns null, not a constant. A constant is indistinguishable from a stable index and would let a confounded comparison through. An absence prints in the report as itself.
The control
One field differs, or the comparison is refused
Every answer arrives with a description of what produced it. Diff two runs' descriptions, and the rule is mechanical.
Exactly one field moved
The delta is attributable to that field.
The model changed and nothing else did, so what the numbers moved by is what the model change cost you. That is a sentence a validator can rely on.
Two or more moved
The comparison is confounded, and we report no verdict.
Enforced in the harness, not written in a runbook. If you deploy mid-run, your prompt hash moves underneath us and the run says so - rather than quietly reporting your own deploy as a regression.
Wanting to move two things at once is legitimate, so the harness decomposes it. A candidate that is a new model and a new prompt becomes three arms - model alone, prompt alone, both - and you get three attributable results instead of one that explains nothing.
And a black box can be fully change-detected without being opened. If your pipeline is commercially confidential, disclose none of it and return a single digest over your configuration. When something inside it changes, the digest moves, two fields differ, and the comparison is marked confounded rather than blaming the thing we were testing. Requiring you to open the box to get that property would have been a demand we had no business making.
The trigger
What makes a run happen
The full set weekly, a smoke set daily, and a confirm run within the hour of a change we detect. Five signals feed that last one, and corroboration is what makes the best finding in the product possible: a canary shift and a changepoint with no fingerprint change is your provider having moved under you without telling either of us.
| Signal | Reliability | What it catches |
|---|---|---|
| Your system's own fingerprint | Highest | Anything your system knows it changed - including a prompt edit, which is the most frequent change and the one invisible to every other method. |
| A webhook from your CI/CD | High | Deploys, prompt edits, index rebuilds, with a commit SHA attached. Three lines in your pipeline. |
| Provider model-list polling | Medium | New models, deprecations, aliases appearing and disappearing. It does not catch an alias silently repointing. |
| Behavioural canaries | Medium | The silent update behind a stable alias - the case nothing else catches. Around thirty high-entropy prompts at temperature zero, sent straight to the provider four times a day. |
| Changepoint detection on our own numbers | Medium | Any behaviour change, whatever caused it. Lags by a run or two, and it is the safety net rather than the alarm. |