← Blog

Governing self-classified documents at scale

Series

Governing self-classified documents at scale

How a two-agent LLM system caught underclassified, sensitive financial documents before they became a security incident — and the design decisions that mattered more than the models themselves.

A case study in document classification governance. If you want the transferable lessons without the ML detail, start with the disclosure problem in Part 3 and the rollout metrics in Part 4.

  • Applied ML
  • Agentic systems
  • Security/Governance
  • Model distillation
  • RLHF
Two-agent document classification flow — document save, classifier agent, verifier agent, and escalation to the safer label
  1. Part 1

    The self-classification problem

    A structurally unenforceable policy, systematic underclassification at thousands of documents a day, and why keyword matching was never going to close the gap.

  2. Part 2

    A two-agent system, and the trade-off that shaped everything

    Why one model wasn't enough, how classifier and verifier agents split the work, and the distillation and RLHF loops that kept cost and behavior in check.

  3. Part 3

    The two problems that almost broke this

    Grounding model claims in real document lines, and the paradox where the classification system itself became the disclosure it was meant to prevent.

  4. Part 4

    Rollout, results, and what I'd change today

    Phased deployment across three stages, an 85% reduction in underclassification, and what I'd rebuild differently with today's models.