Recommendation Blocks
Built the company's product recommendation system from zero. Solved the cold-start problem with three independent engines under CSM-driven tier selection — Curated (day 1, zero risk), Automated (pattern rules), Smart (live behavioral ranking) — served across five regions under a 200ms p99 latency SLA.
My Role
Designed the three-tier trust model (Manual/Automated/Smart) to reduce merchant risk and enable gradual adoption, architected multi-region serving with latency SLAs and the real-time ranking engine, and directed the Redis-backed live tier, MongoDB serving layer, and ranking algorithm implementation with a small team (2-3 engineers) while staying close to product tradeoffs.
The platform serves 200+ merchants across five regions — SMBs with low transaction volume on day 1. Recommendation systems need behavioral data to produce quality results, so the architecture had to ship value on day one with zero data, then scale to algorithmic sophistication as data accumulated.
The Bootstrap Problem
A new merchant has zero behavioral data on day 1. Live recommendation ranking needs statistical significance to produce quality results — ship live ranking immediately and recommendations are garbage, so merchants disable the feature. By day 30, when enough data exists, trust is already destroyed and can't be recovered. You need immediate value with zero data, but can't gather data unless merchants use recommendations first.
- Ship live ranking day 1
- Trains on zero data
- Produces noise → merchants distrust system
- Immediate value, zero data required
- Bootstrap into sophisticated algorithms
- Serve multi-region under 200ms latency
Three Engines, CSM-Driven Tier Selection
Three independent engines — Curated, Automated, Smart — with tier selection driven by CSM discretion, not data availability. CSMs choose which tier per merchant based on trust and use case. Data flows live to Redis, then migrates to BigQuery after 30 minutes of session inactivity or session close.
Decisions & Tradeoffs
Tier Selection Driven by CSM Judgment
CSMs choose which tier per merchant — no automatic progression based on transaction thresholds. Puts control in the hands of customer success, not an algorithm.
Computation Matched to Data Maturity
Curated is static (no computation). Automated is batch rules. Smart is live ranking (statistical model). Computing live ranking on day-7 data would overfit.
Redis-First Serving, Accept Eventual Consistency
In-flight ranking state lost on Redis failure is acceptable for recommendations — would not accept this for transactional orders. The 200ms p99 requirement demands in-memory serving.
Read Replicas Trade Freshness for Latency
Regional replicas incur 5-10s replication lag — acceptable cost for 200ms p99 serving. A central Redis for all regions would mean 300-500ms network latency instead.
Validation, Monitoring, Rollback
Smart ranking trained on historical data was validated against held-out test sets before serving live — checking for cardinality issues, signal stability, and tail behavior, rejecting models that failed validation. Built a canary deployment for the Smart tier: new models ran on 5% of traffic first, compared against the incumbent.
Tracked block click-through rate and latency per tier. Anomalies (CTR drop >10%, latency spike) paged on-call, and tiers could be disabled independently if quality degraded.
Impact
Cold-start was solved by CSM-driven tier selection across three independent engines, not algorithm-triggered progression. Multi-region serving with regional Redis and fallback caching hit 200ms p99 latency across five regions despite 5-10s replication lag, and the model validation gate prevented garbage output from ever reaching merchants and ruining trust.
What I'd change today: canary deployments from day one rather than added later; a feature flag for the Smart tier per merchant instead of an all-or-nothing rollout; and separating model retraining from serving with an async cache layer, since the current design retrains synchronously and risks latency spikes.