INDEX // ARCHITECTURE DECISION RECORDS

Engineering Governance

Architecture Under Constraints

Big architectural decisions aren't about what's theoretically best; they're about trading off what you're willing to pay for what you need. At scale, operational overhead becomes the real constraint.

ADRs LOGGED

3DOCUMENTED

PRINCIPLES CODIFIED

5APPLIED

SCALE PROVEN AT

200+MERCHANTS

YEARS IN PRODUCTION

13+

ADR-001•RATIFIED · PRODUCTION

Land Raw vs. Normalize at Ingest

01 // Problem & Context

Four e-commerce platforms (Shopify, BigCommerce, WooCommerce, Magento 1, Magento 2) each send customer data in different schemas. Need a unified model for downstream products — recommendations, segmentation, attribution.

02 // Considered vs. Chosen

Normalize at IngestREJECTED

One unified schema across all platforms at write time. Simpler downstream, but a wrong shape at ingest becomes a multi-service migration to fix.

Land RawCHOSEN

Preserve each source's schema, normalize downstream via views and query models. New consumers can ask for data shaped differently without touching the landing layer.

Ratified Decision

Preserve each source's schema, normalize downstream via views and query models. New consumers can ask for data shaped differently without touching the landing layer.

Accepted Tradeoff

Thousands of tables instead of one unified schema. Schema migrations run across all client tables — complex coordination and operational overhead (index management, backups).

Lesson

At scale, operational overhead is worth buying architectural flexibility. Normalize where mistakes are cheap (views, query models) — don't commit the whole system to a wrong ingest decision.

ADR-002•RATIFIED · PRODUCTION

Regional Replicas (5-10s Lag) vs. Strong Consistency

01 // Problem & Context

Recommendation system serves 5 regions (US, EU, APAC, India, Brazil). Need 200ms p99 latency for good UX.

02 // Considered vs. Chosen

Central Redis + Pub/SubREJECTED

One region guarantees consistency, but causes 300-500ms latency from distant regions — merchants in India waiting half a second for a recommendation.

Regional ReplicasCHOSEN

5-10 second lag per region. Each region serves from local Redis with no central bottleneck, hitting 200ms p99.

Ratified Decision

5-10 second lag per region. Each region serves from local Redis with no central bottleneck, hitting 200ms p99.

Accepted Tradeoff

5-10 second data lag — recommendations might be stale. Fallback complexity if a replica lags too much.

Lesson

Latency matters more than freshness for UX. 5-10s stale is acceptable; 500ms slow is not. Trade off the consistency you don't need for the performance you do.

ADR-003•RATIFIED · PRODUCTION

Table-per-Tenant vs. Shared Tables

01 // Problem & Context

200+ merchants, each with orders/customers/products data. How to structure MySQL — one shared table or separate tables per tenant?

02 // Considered vs. Chosen

Shared TablesREJECTED

Looks cheaper until production scale. Validated by a real incident: lock contention at 100+ concurrent customers with LOAD DATA parallelism proved shared tables break.

Table-per-TenantCHOSEN

Each merchant has isolated tables. Merchant A's slow query doesn't block Merchant B; per-tenant resync is faster; cascading failures prevented.

Ratified Decision

Each merchant has isolated tables. Merchant A's slow query doesn't block Merchant B; per-tenant resync is faster; cascading failures prevented.

Accepted Tradeoff

Thousands of tables, not manageable with simple SQL. Schema migrations run across all tables — complex coordination and operational overhead.

Lesson

Table-per-tenant costs more to operate but prevents silent cascading failures. Accept the complexity of thousands of tables if it buys isolation.

Common thread across all three: at scale, operational overhead beats architectural purity. Accept the complexity of thousands of tables if it buys isolation. Accept the complexity of regional replicas if it buys latency. Accept the complexity of landing raw if it buys reversibility.

Five Principles That Guide These Decisions

Reversibility over purity

Normalize data where mistakes are reversible; defer normalization where they're expensive to fix. A wrong materialized view is a redefinition away from correction. A wrong schema at ingest is a multi-service migration.

In practice, I landed raw data first, then fanned out shaped copies (MongoDB, ClickHouse, BigQuery), not the other way around. Mistakes in views are cheap; mistakes in the landing layer are catastrophic.

Match storage to question

Point lookups → MongoDB. Population filtering → ClickHouse or BigQuery. One database can't efficiently answer all three questions. Denormalization-in-consumers beats materialization-in-database.

Recommendations wanted fast point lookups (MongoDB), segmentation wanted population filtering (BigQuery then ClickHouse), reporting wanted columnar aggregation (BigQuery). Three storage systems, not one-size-fits-all.

Operational simplicity over sophistication

A legible system is easier to debug than a clever one. A Pub/Sub consumer publishing lag metrics and failing visibly beats a Dataflow job hiding execution details.

I chose explicit Pub/Sub consumers over Dataflow pipelines. Observable, debuggable, no hidden execution state — the backpressure pattern became adoptable company-wide because it was legible.

Ship in trust order, not sophistication order

Recommendations: manual blocks first (day 1), then automated (month 2), then smart (month 6). Clients who started on manual graduated to smart. Adoption requires trust; trust requires shipping something usable first.

I launched with manual recommendation blocks (literally "we pick these products for you"), proved value, then built ranking. By then merchants trusted the system enough to adopt sophistication.

Constraints matter more than code

A pod memory limit is real; a network timeout is real; data growth rate is real. Feedback loops (backpressure, pause/resume) beat throwing resources at symptoms.

Segmentation daily batch beats real-time CDC because the actual constraint was "CSMs run campaigns once daily," not theoretical freshness ideals. Pub/Sub backpressure beats heap allocation because the constraint is memory, not code elegance.

These principles emerged from 13+ years of building systems and 5+ years of production operations at scale.

Read ten production incidents →