POST-MORTEM LEDGER // 10 PRODUCTION INCIDENTS

Hard Problems & Learnings

Ten incidents where production pressure exposed a deeper problem. Each follows failure → diagnosis → decision → recovery → lesson.

DOCUMENTED INCIDENTS

10RECORDS

BEST DIAGNOSIS WIN

360xFASTER

COST REDUCTION (INC-08)

PLATFORM PATTERN ADOPTED

Problem

Paginated queries timing out. Team had exhausted index tuning — every optimization lever was maxed. Root issue: join happened before pagination. Materializing millions of rows, then limiting to 100, is cardinality inversion.

Solution

Reverse operation order: LIMIT from primary table first, then join only for those 100 rows. Changed cardinality from millions-in-join to 100-in-join.

Result

Response time: 3+ minutes → under 500ms (360x faster). Key insight: separates query tuning from query architecture.

Ten Incidents as a Progression

01–03

Senior

Query shape, lock contention, customer insight — Foundation of technical judgment

04–05

Staff

Architecture under deadline, process improvement — Own systems, build process

06–10

Principal

Leadership under crisis, systemic patterns, algorithmic optimization, database-specific constraints, resilience patterns — Mentor, recognize incompatibility, solve elegantly, adopt platform patterns

☰ALL CASE STUDIES VIEW THE RESUME THESE SHAPED→

Hard Problems & Learnings

Query Shape, Not Tuning

Problem

Solution

Result

MySQL Lock Contention at Scale

Real-Time Product Changes in Recommendations

MySQL Views → Debezium CDC

Missing Index on Production Query

Real-Time Optimization Gone Wrong

Repeated Pattern: Segment Ordering

Silent Failure, Fixed Algorithmically

ClickHouse Materialized-View Scale Mismatch

Node.js Memory and Backpressure

Ten Incidents as a Progression