When model outputs aren't enough
Promoted ML outputs without governance — no audit trail, no latest pointer, no repeatability. The systems problem behind holistic ROAS, told without assuming you care about marketing analytics.
Holistic ROAS (HROAS) is the concrete case study — a single rollup over multiple upstream model outputs (national MMM, geo MMM, brand-equity components) into one governed answer with a stable latest pointer and an audit trail. The marketing domain is real context, not the point.
The systems problem is generic: distributed ML jobs produce outputs; someone has to decide which versions are authoritative, freeze that decision, run compute, and let consumers read a stable surface without knowing run IDs. HROAS is how we solved that for one high-stakes, low-throughput pipeline. I'm the lead engineer on it. This is Part 1 of four — starting with the gap, not the architecture diagram.
The product goal is straightforward: give stakeholders one accurate, repeatable view so allocation decisions rest on governed outputs, not tribal knowledge. Upstream models produce answers. HROAS produces accountability for those answers — one rollup, one latest pointer, and a record of which promoted models fed it.
Upstream here is media mix modeling (MMM) — coarse portfolio inference over time series, often built with stacks like Google Meridian. HROAS does not retrain models. It productizes a rollup from whatever upstream runs are currently promoted.
The hard part was never "can we compute ROAS once." It was can we compute the right ROAS, from the right models, on demand, and prove it.
Outputs existed. Governance didn't.
Before HROAS, producing holistic ROAS was a manual, error-prone handoff. Someone had to know which model runs were current, grab the right object-store paths, wire them into compute scripts, and remember to re-run when models updated. Promotion logic lived in people's heads. There was no single latest surface — consumers needed run IDs, paths, and institutional memory about which model family was authoritative for which slot.
Upstream jobs worked. National MMM runs worked. Geo models worked. Brand-equity components worked. Individual routines worked. Demos worked. One-off analyses worked.
What didn't work was accountability at the point of use.
When someone asked for holistic ROAS, three questions came up every time — the same three questions finance asks about any line item, applied to model output:
We couldn't answer those cleanly. That gap is the blind spot. Not missing algorithms — missing governance.
Models refresh on a monthly or bi-monthly cadence, but between refreshes someone still had to manually stitch outputs. Low velocity upstream doesn't excuse opaque handoffs downstream.
When product asked for a dashboard first, I pushed back. A chart on stale object-store paths would ship faster and fail slower — the UI would look fine; trust wouldn't. I framed the project as pipeline first, interface second: automatic and auditable when Econ promotes a model, not another surface on top of ad-hoc paths.
Not a cron job. A judgment call.
HROAS runs are low-throughput, high-stakes. Each client runs on its own cadence — not high frequency, not real time. A bad run doesn't just waste compute; it can send planning down the wrong path for weeks.
MMM results don't change meaningfully day to day. Neither should HROAS churn on a schedule. That constraint shaped early decisions I made:
- Manual trigger + toggle, not a nightly cron — runs happen when the domain team and engineering agree models meaningfully changed
- Resolution before compute — the batch job receives a frozen, fully-specified input; it doesn't guess which model is live
- Audit by default — every submission snapshots what was resolved, so "what ran?" has an answer in the database, not in Slack
These aren't optimizations. They're failure-mode design. When throughput is low, operability matters more than automation theater.
One mental model for the whole system
I split the work into four layers so each piece has a single job. Part 2 goes deep; here's the map:
Layer A — Upstream model jobs
National MMM, geo MMM, and brand-equity jobs write results — plus a small set of immutable taxonomy fields driven by Econ config, stamped at run creation.
Layer B — Promotion, resolution, audit
A registry per client. A resolver picks the latest promoted runs on main. Brand-equity siblings travel together. Every submission is recorded.
Layer C — Batch compute
One orchestrated job runs the full routine chain — standardized JSON in, artifacts out. Pure compute; no resolution logic inside.
Layer D — Outputs and frontend
Durable object-store layout with a stable latest pointer. Readers never need a run ID. The business-facing surface reads from latest.
Notice the separation: trust (B), compute (C), consumption (D). Meridian-class tools train models; HROAS governs what gets rolled up from promoted outputs.
Prove compute. Design trust. Don't block the reader.
I owned the sequencing — deliberate, and not the obvious order:
- Prove Layer C first — full chain, all model types, real batch, real artifacts, success marker, latest mirror
- Design Layer B in parallel — but gate implementation on final alignment with data engineering and the domain team
- Unblock Layer D early — the read path doesn't require the orchestrator to be finished; showing latest outputs beats a blank frontend for months
My first instinct was a nightly cron. I walked that back after two design reviews: HROAS runs are judgment calls, not hygiene tasks. When throughput is low and stakes are high, manual trigger + frozen inputs beats automation theater.
I also rejected a dedicated batch container image early — we clone the existing Python runtime at job start instead. Fewer moving parts in production; package drift managed at the image level. Part 2 unpacks that trade-off.
Proving compute end-to-end doesn't prove the business is ready to self-serve runs. Layer C was the foundation, not the finish line. Runtime minutes don't equal ROI — Part 3 returns to that.
Five parts. One pipeline.
| Part | Focus |
|---|---|
| 1 (here) | The blind spot — why governance, not math, was the gap |
| 2 | The architecture — layers, taxonomy, resolver, trade-offs I locked in |
| 3 | What we proved — production evidence, runtimes, honest limits |
| 4 | What's hard next — open decisions, risks, and guards (June snapshot) |
| 5 | When the blocker moves upstream — Layer A/B shipped, promotion gap |
If you're building something similar — promoted model outputs that need a trusted product surface — I hope the pattern transfers even where the domain doesn't. I wrote this for engineers who've lived the handoff problem and for anyone evaluating how I think about trade-offs, not for MMM practitioners looking for methodology.
Hindsight
I wish I'd written the three-question framework into the RFC on day one. It took two stakeholder meetings before everyone agreed the gap was governance, not missing math — and one of those meetings almost became a dashboard spec. If I were starting over, I'd make "what ran / was it right / can we repeat it" the acceptance criteria before anyone sketched a UI.