The two problems that almost broke this
Grounding model claims in real document lines, and the paradox where the classification system itself became the disclosure it was meant to prevent.
Parts 1 and 2 covered the policy gap and the two-agent architecture. Two problems in this project had nothing to do with model accuracy, and both mattered more than any accuracy gain would have.
Problem 1: grounding a model's claim in the actual document
Early 2025 models — the ones available for this project — hallucinated, and text extraction from real-world documents (nested tables, embedded images, links, arbitrary formatting) was unreliable. That's a serious problem when the entire value of the system is "here's exactly why this document needs a stricter label." A classification suggestion nobody could verify against the source text wouldn't be trusted, and shouldn't be.
The fix was to not trust the model's own citation of "where" the sensitive content was. Instead:
Agent output
Classifier and Verifier produce classification + natural-language justification referencing sensitive passages.
Fuzzy match against source
System matches justification fragments to actual lines of the extracted document — not the model's self-reported locations.
Verified pointers → UI
API returns the suggested label plus grounded line references the user can verify and highlight in Confluence.
Parsing Confluence documents cleanly enough for this to work — readable by an LLM without bloating context with markup, while still mapping back to exact DOM locations after the model identified sensitive content — was, by a wide margin, the hardest engineering problem in the project.
| Content type | Grounding difficulty |
|---|---|
| Plain text | Straightforward — line-level match |
| Nested tables | Structure loss in extraction; cell-level mapping |
| Embedded images | Sensitive text in images often invisible to LLM |
| Nested links | Link text vs. target URL carry different sensitivity |
Problem 2: the system was creating the exact risk it was meant to prevent
This is the part of the project I'd point to first if someone asked what the real lesson was.
Consider the failure case the system was built to catch: a document labeled public that actually contains something highly confidential. The system correctly identifies this, suggests the upgrade, and — to justify that suggestion — highlights the exact lines containing the sensitive information.
If that suggestion and its justification were visible to anyone with access to the document — which, since it was still labeled public, was the entire company — the system itself became the exposure. Pointing at the sensitive lines to justify a classification upgrade is, until the upgrade is accepted, itself a disclosure.
- Suggestion + justification visible to all page viewers
- Sensitive lines highlighted on a still-public document
- System becomes the disclosure vector
- Flag visible next to label; details creator-only
- Accept/reject + highlight controls behind creator gate
- Escalation suggestions only — never downgrade
- Creator retains final label authority
The fix: a highlight-issues control visible only to the document's creator. The suggestion appeared as a flag next to the existing label. Only the creator could open it to see the reasoning, the accept/reject controls (rejection required a reason), and the button that would highlight the specific justifying lines. Nobody else saw any of it until the creator acted.
Governance decisions, not just engineering
We also aligned with legal and security on related points that shaped the product surface:
| Decision | Rationale |
|---|---|
| Never suggest downgrading | False "less sensitive" suggestions create compliance risk; escalation only |
| Creator owns the final label | Accountability stays with the person who classified the document |
| No auto-restrict on flag | Locking a team out on an unconfirmed suggestion creates operational incidents |
| Creator-only detail view | Prevents the justification UI from becoming a disclosure channel |
Auto-restricting a page the moment the system flagged it would also have created a real operational problem — locking a team out of a document they needed, based on a suggestion that hadn't been confirmed yet.
Part 4 covers how this rolled out, what the results actually showed, and what I'd do differently building this today.
What I learned
A well-intentioned classification system can become the disclosure it's meant to prevent. Fixing that meant limiting visibility, keeping the human as the final decision-maker, and treating that as a governance decision — not just an engineering one. The fuzzy-matching grounding layer was a workaround for model limitations; the creator-only gate was a workaround for trust limitations — and the second one mattered more.