Building Real-Time Data Pipelines at Scale: Lessons from Processing 12 Billion Events/Day

At BizDone, we operate one of the largest real-time data streaming platforms in the world. Every day, our infrastructure ingests, processes, and delivers over 12 billion events — from financial transactions to IoT sensor readings — with sub-millisecond latency. This is the story of how we built it.

The Challenge

When we started in 2021, we were processing about 100 million events per day. Our initial architecture — a fairly standard Kafka + Flink setup — handled that load comfortably. But as we onboarded larger enterprise clients, the numbers started climbing rapidly: 500M, 2B, 5B, and eventually 12B events per day.

Each order-of-magnitude increase exposed new bottlenecks. We didn't just need to scale — we needed to scale gracefully, without downtime, and without compromising on our latency guarantees.

Architecture Decisions

1. Partitioning Strategy

The single biggest factor in stream processing performance is partitioning. After extensive testing, we moved to a two-level partitioning scheme: logical partitions (by tenant ID) and physical partitions (hash-based, within each tenant). This gave us isolation between customers and even load distribution within each customer's data.

2. Backpressure Management

In a system processing 12B events/day, backpressure isn't an edge case — it's Tuesday. We built a custom adaptive backpressure system that monitors consumer lag at every stage of the pipeline and dynamically adjusts producer rates. The key insight: backpressure should be applied as close to the source as possible, preventing the "buffer bloat" problem that plagues many streaming systems.

3. Serialization and Wire Protocol

We moved from JSON to a custom binary protocol based on Apache Arrow Flight. For high-throughput numeric data (the bulk of our clients' workloads), this reduced serialization overhead by 80% and cut CPU utilization by nearly half. The protocol supports zero-copy deserialization for numerical columns, which is critical when you're processing millions of vectors per second.

Monitoring and Observability

At this scale, you can't debug problems by looking at logs — there are too many. We instrumented every stage of the pipeline with Prometheus metrics, exported to our own dashboards. Every event carries a trace ID that flows through the entire system, from ingestion to delivery. This lets us answer "what happened to this specific event?" in real time, even at 12B/day.

Key Takeaways

• Partition early, partition often. Data locality is your best friend at scale.
• Backpressure is a feature, not a bug. Design it into your system from day one.
• Binary protocols win. JSON is great for debugging, terrible for production throughput.
• Distributed tracing is non-negotiable. You will have bugs that only manifest at 1B+ events/day.

Building real-time data infrastructure at this scale is hard — but with the right architectural decisions, it's achievable. If you're facing similar challenges, reach out. We'd love to share what we've learned.