The challenge
Five models had been shipped by three different teams, each with its own deployment path, its own idea of what 'good' meant, and no shared record of which prompt version produced which output.
SOC 2 review was the forcing function. Auditors wanted to know what changed, when, who approved it and what evidence existed that the change was safe. None of that was answerable from the existing setup.
Approach
01
One registry, one deployment path
Consolidated every model behind a single registry and a single promotion flow. Divergent paths were the root cause of the lineage gap, so uniformity was the fix.
02
Evaluation as a deployment gate
Each model carries its own held-out evaluation set. Promotion to production is blocked automatically on regression rather than reviewed by hand.
03
Drift detection on inputs and outputs
Input distribution shift and output quality drift are monitored separately, because they fail in different ways and need different responses.
04
Lineage that satisfies an auditor
Every production output traces back to model version, prompt version, retrieval index snapshot and the approval record for that promotion.
Architecture
Register
MLflow holds model versions, prompt versions and evaluation artifacts as first-class records.
Evaluate
Per-model evaluation suites run automatically on every promotion candidate.
Orchestrate
ZenML pipelines standardise training, evaluation and promotion across all five models.
Observe
LangSmith traces production calls; drift monitors run continuously on inputs and outputs.
Audit
Immutable lineage records link every output to the exact configuration and approval that produced it.
Results
5Models governed
24/7Drift detection
SOC 2Audit-ready lineage
100%Deploys gated on evaluation
1Unified deployment path
What I took from it
Automating the evaluation gate removed the argument about whether a regression was acceptable. When the pipeline blocks the deploy, the conversation shifts from negotiation to fixing the regression.
Stack
Registry
Pipelines
Observability
Governance