Architecture under constraints
Big architectural decisions aren't about what's theoretically best; they're about trading off what you're willing to pay for what you need. At scale, operational overhead becomes the real constraint.
01. Land raw vs. normalize at ingest
Problem
Four e-commerce platforms (Shopify, BigCommerce, WooCommerce, Magento 1, Magento 2) each send customer data in different schemas. Need unified model for downstream products (recommendations, segmentation, attribution).
Our choice
Land raw. Preserve each source's schema, normalize downstream via views and query models.
Cost
Thousands of tables (not one unified schema). Schema migrations run across all client tables (complex coordination). Operational overhead (index management, backups).
Benefit
Reversibility. Wrong normalization at ingest means multi-service migration. Wrong view is a redefinition away. New downstream consumer can ask for data shaped differently without changing the landing layer.
Lesson
At scale, operational overhead is worth buying architectural flexibility. Normalize where mistakes are cheap (views, query models). Don't commit the whole system to a wrong ingest decision.
02. Eventual consistency (5-10s lag) vs. strong consistency
Problem
Recommendation system serves 5 regions (US, EU, APAC, India, Brazil). Need 200ms p99 latency for good UX.
Our choice
Regional replicas with 5-10s lag. Central Redis + Pub/Sub in one region would guarantee consistency but cause 300-500ms latency from distant regions.
Cost
5-10 second data lag. Recommendations might be stale. Fallback complexity if replica lags too much.
Benefit
200ms p99 latency achievable. Merchants in India don't wait 500ms for a recommendation. No central bottleneck—each region serves from local Redis.
Lesson
Latency matters more than freshness for UX. 5-10s stale is acceptable; 500ms slow is not. Trade off the consistency you don't need for the performance you do.
03. Table-per-tenant vs. shared tables
Problem
200+ merchants, each with orders/customers/products data. How to structure MySQL? One shared table or separate tables per tenant?
Our choice
Table-per-tenant. Each merchant has isolated tables. Validates via Incident 2: production lock contention at 100+ concurrent customers proved shared tables break at scale.
Cost
Thousands of tables (not manageable with simple SQL). Schema migrations run across all tables (complex coordination). Operational overhead (index management, backups).
Benefit
Isolation. Merchant A's slow query doesn't block Merchant B. Lock contention prevented. Per-tenant resync is faster (don't have to filter by customer_id). Cascading failures prevented.
Lesson
Shared tables look cheaper until you hit production scale. At 100+ concurrent customers with LOAD DATA parallelism, lock contention becomes unbearable. Table-per-tenant costs more to operate but prevents silent cascading failures.
Pattern across all three
Each decision: Problem → Alternatives → Choice → Cost → Benefit → Lesson. Common thread: At scale, operational overhead beats architectural purity. Accept the complexity of thousands of tables if it buys isolation. Accept the complexity of regional replicas if it buys latency. Accept the complexity of landing raw if it buys reversibility.
Five principles that guide these decisions
Reversibility over purity
Normalize data where mistakes are reversible; defer normalization where they're expensive to fix. A wrong materialized view is a redefinition away from correction. A wrong schema at ingest is a multi-service migration.
In practice, I landed raw data first, then fanned out shaped copies (MongoDB, ClickHouse, BigQuery), not the other way around. Mistakes in views are cheap; mistakes in the landing layer are catastrophic.
Match storage to question
Point lookups → MongoDB. Population filtering → ClickHouse or BigQuery. One database can't efficiently answer all three questions. Denormalization-in-consumers beats materialization-in-database.
In practice: recommendations wanted fast point lookups (MongoDB), segmentation wanted population filtering (BigQuery then ClickHouse), reporting wanted columnar aggregation (BigQuery). I used three storage systems, not one-size-fits-all database.
Operational simplicity over sophistication
A legible system is easier to debug than a clever one. A Pub/Sub consumer publishing lag metrics and failing visibly beats a Dataflow job hiding execution details. When a small team owns it, debuggability under pressure beats abstraction elegance.
In practice, I chose explicit Pub/Sub consumers over Dataflow pipelines. Observable, debuggable, no hidden execution state. The backpressure pattern became adoptable company-wide because it was legible.
Ship in trust order, not sophistication order
Recommendations: manual blocks first (day 1), then automated (month 2), then smart (month 6). Clients who started on manual graduated to smart. Adoption requires trust; trust requires shipping something usable first.
In practice, I launched with manual recommendation blocks (literally "we pick these products for you"), proved value, then built ranking. By then merchants trusted the system enough to adopt sophistication.
Constraints matter more than code
A pod memory limit is real; a network timeout is real; data growth rate is real. Feedback loops (backpressure, pause/resume) beat throwing resources at symptoms. Database-specific behavior (ClickHouse refresh frequency, BigQuery slot pricing) matters as much as architectural desire for real-time.
In practice: segmentation daily batch beats real-time CDC because the actual constraint was "CSMs run campaigns once daily," not theoretical freshness ideals. Pub/Sub backpressure beats heap allocation because the constraint is memory, not code elegance.
These principles emerged from 13+ years of building systems and 5+ years of production operations at scale.
Read ten production incidents →