AI engineering & governance
Most pilots don’t fail at the model. They stop at five layers.
A demo proves that something can work. Production proves that it can work safely, reliably and repeatedly. That means answering the right users from current data, knowing when not to guess, and leaving a trail that can be audited. This page walks through three examples and the governance decisions each one brings with it.
01 · Problem to architecture
Where a pilot actually stops.
Illustrative — not a client engagementThis is a worked example, not a case study. It shows the shape of a problem we often encounter and how we approach it, using a representative composite rather than a named client engagement.
The setup: an internal retrieval assistant that answers staff questions from company documentation. It works in the notebook. Eight months later, it still has not reached production.
A request meeting a layer that was never built
Access
Where it stops
The documents live in four systems with four permission models. The pilot ran on an export somebody made by hand, so it never had to answer the question of who is allowed to see what.
What gets decided
Whether authorisation resolves per query against the source systems, or the index is partitioned by entitlement. It is the first decision and it constrains every later one.
Ground truth
Where it stops
Nobody can say whether an answer was right. The demo was judged by the people who built it, reading answers they already knew the answer to.
What gets decided
A test set drawn from real questions, a metric that reflects the actual cost of being wrong, and a threshold agreed before anything is built. This is the number Phase 05 measures against.
Freshness
Where it stops
The export is a snapshot. The documentation changed the week after, and the assistant now confidently cites a policy that was withdrawn.
What gets decided
How change propagates: re-index on write, scheduled reconciliation, or explicit staleness shown to the reader. Each is defensible; the wrong one for the domain is how trust is lost in month three.
Failure behaviour
Where it stops
Asked something outside its documents, the pilot answers anyway. In a demo that reads as capability. In production it is the incident.
What gets decided
What the system does when it does not know — abstain, escalate to a person, or answer with the evidence attached and let the reader judge. Wired in as behaviour, not appended as a disclaimer.
Ownership
Where it stops
It runs on somebody’s account, deploys from somebody’s laptop, and has no owner in the on-call rota. This is the layer that ends most pilots, and it is not a technical problem.
What gets decided
Which existing team owns it, what their runbook says, and what the alert means at three in the morning. If no team will take it, that is a finding worth more than the system.
Note what is not on that list. Which model, which vector store, which framework. Those are real decisions and they take an afternoon. The five above take the engagement, and getting them wrong is why a working demo is still a demo eight months later.
This is also why Approach Phase 02 — the unglamorous layer — is six weeks and not one. Access, ground truth and freshness are the foundation; everything after them is comparatively cheap.
02 · The evaluation harness
The part most proposals leave out.
What a threshold does to a spread of answers
“The model is 92% accurate” is not a finding, it is a claim. It becomes a finding when someone else can re-run it next quarter, on new data, and get a number they can compare.
So a Quenora engagement produces a harness alongside the system: the test set, the metric, the thresholds and the script that runs them. It is what Approach Phase 05 measures against, and it is handed over with the code — because the question that matters is not whether the system worked on the day we left, but whether your team can tell six months later that it still does.
It is also the honest answer to a fair objection. A firm with no case studies is asking you to trust an assertion. A firm that hands you the means to check its own work is asking for something much smaller.
03 · Principles
Five principles, and what each one is tied to.
Principles are cheap to list. Each of these is attached to something specific in how an engagement runs, so you can hold us to it.
It has to end
Every phase carries an exit condition written into the statement of work. The last one is that your team runs the system without us.
Tied to · Approach Phase 06, handoverIt has to be measurable
A number is agreed in Phase 01. Phase 05 measures against that number in writing. If it did not move, we say so.
Tied to · Approach Phases 01 and 05You keep everything
The code, the documentation and the evaluation harness are yours regardless of how the engagement ends. There is no scenario where you finish with nothing to show for it.
Tied to · every engagementIt has to run on your estate
Production means the systems you already have, with the permissions they already enforce. A capability that only works in a clean room has not reached the work.
Tied to · Approach Phases 03 and 04Simple enough to hand over
The measure of a design is whether the team inheriting it can change it safely. A design only its author can maintain is a cost we would be handing you.
Tied to · Approach Phase 0604 · Governance
Governance is a set of decisions, not a document.
Most governance material describes a policy. What a production system needs is a set of answers wired into the build, because each one changes the architecture. These are the questions we settle before anything ships, and where in the six Governance Phases each one gets settled.
Who is allowed to see what
Whether authorisation resolves per query against the source systems, or the index is partitioned by entitlement. Settled in Foundation, because every later decision inherits it.
The version that goes wrong quietly: a pilot built on a manual export, where the permission question was never asked because there was only one copy and one reader.
What the system refuses to decide
The threshold above which an answer is returned, below which it is escalated, and the path a human sees when it is. Settled in Foundation with the evaluation harness, and measured in Prove.
An automation earns trust by knowing what it should not decide. Drawing that line is the work; a disclaimer appended afterwards is not the same thing.
What is written down when it runs
Which inputs, outputs, model version and decision path are retained, for how long, and who can read them. Settled in Integrate, because retro-fitting an audit trail means re-running history you no longer have.
This is the evidence procurement and risk ask for after go-live — the question that arrives when something is already in production and nobody captured the answer.
Where the data goes, and to whom
Which providers process what, in which jurisdiction, whether anything is retained by them, and what leaves your boundary at all. Settled in Frame, because it constrains the architecture rather than following from it.
For European clients this is the first works-council and procurement question, and it is a design constraint, not a compliance exercise to be run at the end.
What happens when it drifts
What is monitored, what threshold triggers a human, who is on the escalation path, and what the rollback actually is. Settled in Ship, rehearsed rather than theorised.
Models drift, upstream systems change and volumes grow. A system nobody is watching is not governed, however good the policy document is.
Who owns it after we leave
A named owner, the runbooks, the decision records and the evaluation harness — handed over as working artefacts, not as a folder. Settled in Handover.
Governance that depends on the firm that built the system is not governance. It is a dependency.
What this page does not claim. Quenora holds no ISO 27001 or SOC 2 certification today, and this page will not imply one. The six decisions above describe how the work is engineered. Attestations, a data processing agreement, subprocessor list and insurance evidence are supplied on request during procurement — ask for them in the first conversation rather than the third.
Recognise your own system in any of those five layers? That is the conversation — two paragraphs on which one is stuck.
Start a conversation