High-Load Sportsbook Data Platform: .NET, SQL Server & Kafka at Tens of Billions of Records
Re-engineering real-time odds ingestion, event settlement, and high-volume relational storage across C#/.NET microservices, Apache Kafka, RabbitMQ, Redis, and partitioned SQL Server.
Engineering Context & Scale Bottlenecks
Live sports trading generates extreme burst traffic during simultaneous weekend fixtures: thousands of odds updates per second, concurrent bet validations, and rapid settlement waves hitting tables that had grown into tens of billions of rows.
Lock contention, index fragmentation, and synchronous cross-service calls risked cascading tail latencies precisely during peak match-day windows.
System Architecture & Technical Decisions
Partitioned Kafka & RabbitMQ Event Backbone
Decoupled live feed ingestion from downstream settlement and reporting consumers using partitioned Apache Kafka topics ordered by fixture/market key, paired with RabbitMQ for targeted command workflows.
SQL Server Partitioning, Covering Indexes & Batch Writes
Eliminated hot-page lock escalation through aligned table partitioning, surgical non-clustered covering indexes, lock-free read patterns, and batched bulk persistence.
Multi-Tier Redis Hot State & Idempotency Keys
Served sub-100ms market state reads out of Redis with deterministic idempotency keys so duplicate broker deliveries never double-settled a ledger entry.
Low-Allocation C# / .NET Worker Pipelines
Tuned async consumer groups, connection pooling, backpressure bounds, and memory allocations in .NET to prevent GC pauses during major tournament spikes.
Data Integrity, Financial Accuracy & Resilience
- Transactional outbox and idempotent consumer deduplication guaranteeing exact financial settlement.
- Dead-letter queues with automated replay tooling and schema compatibility checks.
- Real-time consumer lag, SQL wait-stat, and p99 latency alerting across peak sporting events.
Measured Production Impact
- Sustained tens of billions of stored records with predictable query and write latency under peak load.
- Eliminated settlement bottlenecks during concurrent international match windows.
- Improved partner operator SLA stability across global B2B sportsbook feeds.