Blog/
11 min read

When the blocker moves upstream

Layer A and B shipped — and the remaining gate turned out to be upstream model promotion, not orchestration code.

Part 4 was a June snapshot: Layer B gated, orchestration unproven, August-shaped target. By mid-July the story flipped. The trust layer shipped. The read-side was already in production. And the remaining blocker turned out to be something I had under-weighted: who promotes which upstream model run.

01 — SHIPPED

What landed

For one client configuration — Poppi revenue HROAS — all four layers are wired:

Layer A: four taxonomy columns on the upstream job tables, a one-time backfill for existing promoted rows, and a write-path that tags every new run from Econ config.

Layer B: registry and audit schema, a TypeScript orchestrator (resolver, inputs builder, batch submit, manual trigger), and API gateway cut-over. Not the Java-on-MMMJob path I sketched early — those jobs have no direct Postgres access; the orchestrator needed to live where the database and Batch client already do.

Layer C: unchanged since Part 3 — five routines, one job, full-chain E2E proven in June.

Layer D: read-side merged and in production since late June — consolidated results endpoint, frontend reading latest/ instead of legacy bundle paths. Write-side (run button, status banner, "which models?" panel) still deferred.

That is a vertical slice, not fleet-wide rollout. But Layer B is real code in prod, not a design doc waiting on alignment. Part 4's gating call paid off: economists and data engineering signed off before I merged nine PRs across four repos.

02 — SURPRISE

The blocker moved upstream

I expected the hard part after Layer B merged to be trigger wiring or frontend polish. Instead I hit a promotion gap.

The resolver picks the latest client-facing run on main for each registered slot. Poppi's national MMM has a Meridian run in the batch location — correct artifacts, correct branch — but it is client_facing = false. The only promoted national row is a legacy-R run from March on a feature branch, which the resolver correctly excludes. Brand equity had the same shape: Meridian rows exist after a fresh rerun, but they need promote-and-tag before resolution can pick them. Geo already resolves cleanly.

This is not a seed bug or a backfill miss. The one-time backfill only tagged rows that were already promoted on main at backfill time. The Meridian runs landed afterward — or landed unpromoted — and the write-path tags future runs automatically; these pre-write-path rows need a one-time Econ action.

The durable engineering fix is a resolver Meridian gate (output_file_version >= 2) so legacy-R rows are skipped rather than selected-then-failed. That does not remove the need to promote the right run; it prevents silent selection of a run that cannot produce batch artifacts.

03 — GUARDS

What keeps me up at night now

Part 4's risk table still applies. These rows are new or sharper post-ship:

RiskGuard
Legacy-R row selected because it is the only promoted main rowMeridian version gate in resolver query — skip ofv < 2, don't select-then-fail
model_scope typo dormants a slot while ≥2 others resolveBidirectional coverage warning before submit — expected vs resolved slot set
Cross-repo deploy in wrong order (columns before insert-lambda before gateway cut-over)Checked runbook with smoke gates between steps
Burst of promotion toggles fires multiple ~2.5h batch jobsDebounce/dedup before wiring auto-trigger from the app toggle
Resolver SQL untested against real schemaIntegration test: resolve + build against seeded DB; assert every S3 key in inputs.json exists
hroas_runs.status stuck at submittedEventBridge status updater — still deferred; don't trust status for succeeded/failed yet

The silent-degradation case is the one I worry about most. The orchestrator has a ≥2-slot floor — if two of three slots resolve, it proceeds. A dormant third slot produces a quietly wrong HROAS, not an error. The bidirectional coverage warning is designed but not built yet. Until it is, correctness depends on humans noticing a missing model in the output.

04 — STILL OPEN

Decided vs not yet — July edition

  • Auto-registration on Meridian promotion via client_facing toggle — supersedes the config UI from Part 4
  • Orchestrator = TypeScript Lambda in consolidated hroas service, not Java
  • Registry + audit schema, resolver, builder, manual trigger — deployed
  • Layer D read-side in production; legacy bundle reads removed
  • All prior Part 4 decisions on taxonomy, triggers, batch image — unchanged
  • Upstream promotion — Meridian rMMM + BE promote+tag (Econ / data-eng action, not code)
  • Trigger wiring — app toggle → auto-run; frontend run button + status banner
  • Backfill depth, validation strictness, rollback, regional edge cases — same as Part 4
  • EventBridge → hroas_runs.status updates
  • Coverage warning, Meridian gate, schema integration test — hardening backlog

Part 4 asked whether rollback belongs in v1. I still don't have a production incident that justifies pin-to-known-good complexity. I'd ship the Meridian gate and coverage warning first and learn which failure actually happens.

05 — ONE CALL

One API call away

"Done" for the Poppi vertical slice is a four-gate test plan, not a merge:

  1. Prerequisites — taxonomy backfill and registry seed live in prod (confirmed).
  2. Dry-run — invoke resolver + builder read-only; assert three slots resolve and every S3 key in the produced inputs.json exists. Catches resolution and path errors without burning 2.5 hours of compute.
  3. Full submit — trigger orchestrator; compare structurally to the June 17 hand-built baseline (Part 3), not byte-exact — orchestrator points at newer Meridian batch models.
  4. Frontend — results panels render off latest/.

Once Meridian rMMM and BE are promoted and tagged, gate 3 is a single manual POST to the trigger endpoint. The frontend button and app-toggle auto-run are not wired yet — deliberate; I want the coverage warning and Meridian gate landed before a burst of expensive batch jobs can fire unattended.

A capstone post will follow: that first orchestrator-driven run, structural diff against the June baseline, and an honest answer to "what did HROAS use on day X?" from the audit table — not from archaeology.

06 — HINDSIGHT

Hindsight

Gating Layer B in Part 4 was the right call — I'd make it again. What I mis-forecast was where cross-team risk would surface after the build. I assumed alignment would slow implementation; instead implementation finished and promotion became the gate. Separating solo-build velocity from cross-team schedule risk was correct instinct; I just attributed the cross-team half to the wrong layer.

The Part 3 comms lesson still applies: name what your metrics don't prove before the demo. "All four layers merged" reads as "push the button" to stakeholders who don't live in promotion flags and output_file_version. I'm saying it upfront this time — merged is not the same as first real run.