What's hard next
The remaining risk isn't batch failure — it's operational ambiguity: backfill, validation, rollback, and who owns the run button.
Part 3 showed the compute layer holds. This post is what remains — and it's where most of the product risk lives.
The remaining failure mode isn't batch timeout. It's operational ambiguity: a config typo that silences a slot, a resolver regression that picks the wrong model without error, a frontend that shows latest while the orchestrator isn't finished. I designed guards for these; several policy choices are still open.
Layer B is the product
Layer C was the foundation. Layer B is the promise: when Econ flips promotion, HROAS recomputes from the right runs, and anyone can audit what happened.
That work is gated on a final alignment with data engineering and the domain team before implementation kicks off — deliberately. I'd rather delay orchestration than ship a resolver that economists don't trust or data engineering can't operate.
Meanwhile I unblocked Layer D: the read surface — chart data, which models fed the current view — doesn't need the run button or status banner on day one. I made that call in a planning review when the default was to block the frontend on orchestration. Consumers shouldn't wait months for a UI because Layer B is hard.
Decided vs not yet
- Four immutable taxonomy fields; DTC and commerce are first-class model types
- Brand-equity siblings resolve together by shared training timestamp
- No separate batch image — runtime clone of the existing Python stack
- Resolution at submit time; batch jobs are pure compute
- Triggers: promotion toggle + manual button — no cron
- Config UI built by engineering, populated by Econ — light front end, real validation
- Backfill depth — how far back to populate historical audit rows
- Validation strictness — does a taxonomy mismatch warn or block?
- Rollback story — pin to a known-good run when a new compute misbehaves (likely v2)
- Status propagation — event-driven updates vs polling for run state
- Regional edge cases — mixed-market clients need product input
The August-shaped target for a full loop — promote, resolve, run, frontend — is gated less by coding speed than by Layer B clearing and the backfill decision. Net-new orchestration has no proven vertical slice yet; I'm not extrapolating velocity from Layer C.
Failure modes and guards
MMM's classic pains — irreconcilable results, silent input errors, opacity about what the model actually used — don't disappear downstream. They concentrate in the handoff. This table is how I think about them as an engineer:
| Risk | Guard |
|---|---|
| Typo in model scope silently dormants a slot | Bidirectional coverage check before submit + UI validation on registry writes |
| HROAS computed from a feature-branch model | Resolver requires main; branch predicate is non-negotiable |
| Resolver regression breaks latest-wins or branch filter | Unit tests as first-class — sticky multi-row pick, branch exclusion, BE timestamp anchor, under-filled slot failure |
| Stale promotion flag picked over newer run | Latest-wins ordering — never trust a boolean alone as the unique selector |
| Brand-equity components from different training runs mixed | Timestamp-anchored sibling set |
| Package drift between runtimes | Pin at the shared image level; no parallel image to fall out of sync |
Hindsight
Gating Layer B on alignment beat shipping a resolver nobody would operate — I'd make that call again. What I'm still uncertain about: whether rollback (pin to a known-good run) belongs in v1 or v2. Economists want it; engineering cost is real, and we haven't had a bad compute run in production yet to justify the complexity. I'd rather ship audit and validation first and learn which failure actually happens.
One case study, a recurring pattern
This series documents one instance of a problem I keep running into: governance over distributed, promoted model outputs — deciding which versions are authoritative, freezing that decision before compute, and giving consumers a stable read surface with an audit trail.
The marketing-analytics detail is load-bearing here. The taxonomy fields, resolver logic, and risk table are concrete because the domain is concrete. I didn't abstract them further; that would weaken the strongest parts of the writing.
The same design questions show up elsewhere. At BNY I worked on multi-agent orchestration and classification pipelines in financial operations — document routing and compliance workflows, not ROAS rollups — but the shape rhymed: who owns the authoritative version, what gets frozen before batch work runs, and how you prove what actually executed. If you're evaluating how I think about systems work, the trade-offs in Part 2 and the risk table in Part 4 are the most domain-agnostic entry points.
More selected work — including non-marketing projects — lives on my about page.