Rollout, results, and what I'd change today
Phased deployment across three stages, an 85% reduction in underclassification, and what I'd rebuild differently with today's models.
Parts 1–3 covered the policy gap, the two-agent architecture, and the grounding and disclosure problems that almost broke the project. Part 4 is what happened when it shipped — and what I'd change with today's tooling.
Phased rollout, not a company-wide flip
This shipped in three stages. At each stage, feedback drove another iteration before expanding further. This wasn't a "ship and see" rollout — every stage was a checkpoint.
| Stage | Audience | Purpose |
|---|---|---|
| 1 — Pilot | Team of 10 direct collaborators | Day-to-day use, explicit accept/reject feedback, rejection reasons shape iterations |
| 2 — Engineering | Broader engineering department | Validate system holds up outside a small, invested pilot group |
| 3 — Company-wide | All document types and non-technical users | Full production load — every kind of content the system would need to handle |
High-context feedback loop before broader exposure
Checkpoint-driven expansion, not a single flip
Accepted upgrades vs. pre-system baseline
Classifier + Verifier with restrictive bias on disagreement
Measuring the 85% reduction
The headline number — an 85% reduction in misclassification — specifically reflects underclassification: documents originally labeled less restrictively than they should have been.
It was measured by tracking cases where the system suggested an upgrade and the document's creator accepted it, against the baseline rate of underclassified documents that existed before the system was in place.
| Signal | What it tells you |
|---|---|
| Accepted upgrade rate | Direct underclassification catch |
| Rejection reasons | Where policy interpretation diverges |
| Behavioral edits (no accept click) | Suggestion changed content, not just label |
| Self-label vs. suggestion gap (trend) | Whether users are internalizing the policy |
What I'd do differently today
Building this in early 2025 meant working around limitations that don't fully apply anymore. If I were rebuilding this now, a few things would change:
| Then (early 2025) | Now |
|---|---|
| Fuzzy-match grounding against extracted lines | Native structured output + citation-grounded responses |
| Parse rendered Confluence HTML → DOM map | Structured document representation where available |
| Teacher-student distillation for cost/latency | Re-run cost/latency analysis before assuming distillation |
| Three-signal RLHF dataset, equal weighting | Formal active learning — prioritize uncertain/disputed cases |
-
Native structured output over fuzzy-matched grounding. The fuzzy-matching-to-real-document-lines workaround was the right call given the hallucination and extraction reliability of models available at the time. Current models support much more reliable structured output and citation-grounded responses natively, which would likely simplify — though probably not eliminate — the grounding problem, especially for the harder cases like tables and embedded content.
-
Structured document access over rendered-HTML parsing. A meaningful chunk of the hardest engineering work was parsing rendered Confluence content and mapping back to the DOM. Where possible, working from a structured document representation (rather than reconstructing structure from rendered output) would likely remove a large share of that complexity today.
-
Rethinking whether distillation was solving the right problem. The teacher-student approach was a reasonable answer to cost and latency constraints in early 2025. Model costs and inference speed for capable models have shifted enough that I'd want to re-run that cost/latency analysis before assuming distillation is still the right trade-off, rather than treating it as a given.
-
More deliberate active learning from day one. The three-signal dataset (labeled examples, explicit feedback, behavioral change) worked well, but I'd want to more formally prioritize which uncertain or disputed cases got surfaced for review, rather than treating all feedback as equally valuable input to the next training round.
What I learned
None of the "what I'd change today" list changes the part of the project I'd stand behind most: the realization that a well-intentioned classification system can itself become the disclosure it's meant to prevent, and that fixing that meant limiting visibility, keeping the human as the final decision-maker, and treating that as a governance decision — not just an engineering one. The models got better; that lesson didn't age out.