Udhayakumar

ConvertCart is an e-commerce personalisation platform. Everything it sells — segmentation, recommendations, on-site personalisation, triggered email — is downstream of one thing: holding a current, accurate picture of each merchant's shoppers, orders, products and catalogue. When I joined, that picture was days old. I owned this platform for nearly six years, across 100+ merchant stores on Shopify, BigCommerce, WooCommerce and Magento 1 and 2. I designed the architecture and led a team of up to 6 engineers through every step of the rebuild, handling all architecture decisions, code review, and team operations.

Platform sync ran as scheduled batch pulls against each merchant's commerce API, so data landed days late — worst during sale periods, when volume and API rate limiting peak together. Commerce platforms were never the whole picture either — merchants also run loyalty platforms (Zinrelo) and marketing/CRM systems (Microsoft Dynamics), none of which resemble a commerce API or each other. The deeper problem was on the consuming side: recommendations want point lookups, segmentation wants population filtering, reporting wants columnar aggregation — one store can't serve all three, so every new feature meant another bespoke path back to the data.

No merchant-side changes (no schema changes, no downtime, merchants absorb zero cost). Five platforms with genuinely different webhook reliability. An open-ended, never-final set of source types. Consumers wanting structurally incompatible read shapes. A live migration with no coverage gap allowed.

"Land raw, fan out shaped."

Every source lands in MySQL in its native API schema. Debezium tails the landing store into Pub/Sub. Three consumer services normalize for their destination: MongoDB (recommendations), ClickHouse (reporting and click-tracking), BigQuery (historical analytics).

Reporting: BigQuery first (managed, right choice early), later ClickHouse (cheaper for repeated queries against the same data).

Debezium's initial snapshots of large schemas (50M+ rows in some client cases) occasionally exceeded Pub/Sub processing windows, causing lag spikes. Debugged by instrumenting consumer lag metrics per-topic and correlating with Debezium's snapshot progress logs. Resolution: split large snapshots into parallel chunks, reducing snapshot phase from 2 hours to 15 minutes per client. Also caught a silent failure mode where merchants adding new fields to their store caused ClickHouse schema mismatches — added schema detection and auto-migration on the consumer side, triggered a backward-compatible alert to engineering, allowing human review before auto-correcting.

The module everything else targets through. Representative segment: shoppers who viewed a product that's since dropped in price, haven't ordered in 12 months, have LTV above $300, and hold loyalty points expiring this month. Built this with a team of 2 engineers and 1 analytics engineer, working closely with the Customer Success team to understand segment use cases and validate the UX.

That question spans four systems of origin (click-tracking, commerce catalogue, transactional orders, third-party loyalty balance) across three storage engines. Nothing joins — different query languages, performance characteristics, owners, cadences. And the person asking the question is a CSM who doesn't write SQL.

Authored by non-engineers in a UI (arbitrary predicate combinations, not a fixed report menu). Open-ended source set. No client-side schema changes. Read-heavy and repeated (segments run on a schedule, reused across campaigns).

Consolidate four source systems (click-tracking, commerce, loyalty, third-party) into BigQuery via Datastream. Materialized views normalize at analytics layer. CSM-authored predicates in UI; segments precompute daily.

ConvertCart sold personalisation but had no first-party recommendation product. I initiated it, built the first version solo, later rebuilt and expanded it with a team of 3 engineers (including myself) into the three-tier system below, owning architecture decisions, launch strategy, and client adoption.

The tempting build is one behavioural engine: collect signals, rank, render. That doesn't get adopted — merchandising is the part of a store owner's job they're least willing to hand to a black box, and blocks were configured by CSMs on the client's behalf, so the person deploying one had to be able to explain exactly what it would show and why. The real problem wasn't ranking quality — it was building something a non-technical CSM could confidently deploy.

Storefront latency budget in the tens of milliseconds when served from a regional replica; up to 200ms P99 for the smaller share of cross-continental requests (mitigated via five-region replication — see below). Renders inside page load; slow is worse than absent. Configured by CSMs, not engineers — every capability had to be UI-expressible and client-explainable. Price/stock accuracy (a wrong recommendation damages trust worse than an empty block). Cold start on both sides (new stores, new products). One system across five platforms and 100+ stores with very different traffic profiles.

"A ladder of control, not one engine." Manual (CSM hand-picked) → Automated (rule-based, daily) → Smart (live, behavioural). Built manual first for client trust, even though it required zero intelligence.