Production2023·Backend Engineer

DataStream API

High-throughput real-time data pipeline

GoKafkaRedis
View Code
10M+
Events / Day
<8ms
P99 Latency
120K/s
Throughput
99.95%
Availability
Overview

DataStream is a high-performance event ingestion and processing API built in Go, designed to handle bursty workloads with consistent low latency.

The system uses Kafka for durable event streaming, Redis for hot-path caching, and a custom fan-out router that delivers events to multiple downstream consumers simultaneously.

Key Features
  • Kafka-backed durable event streaming with at-least-once delivery
  • Redis hot-path caching for sub-millisecond lookups
  • Horizontal fan-out to multiple downstream consumers
  • Schema validation and dead-letter queue for malformed events
  • Prometheus metrics and Grafana dashboards out of the box
Engineering Challenges

Achieving consistent sub-10ms P99 latency under 10x traffic spikes

Designing a backpressure mechanism that prevents Kafka consumer lag from growing unbounded

© 2026  ·  Built with precision by Ibrahim Kenia