Batch and stream processing
Enormous volume in, aggregates out, and nobody waiting for any single event.
What problem does stream processing solve?
Some systems are not request-response at all. Billions of events arrive, and what people want is a number: clicks per ad per hour, revenue per region per day, active users this week.
You can compute those continuously as events arrive, or in large scheduled runs over stored data. Those two answers have different costs, different failure modes and different accuracy, and choosing without saying why is the mistake.
What stream processing costs you
Every concept in here states a cost. A technique with no price is a technique you have not understood yet — and naming the cost out loud is most of what separates a senior answer from a junior one.
When to reach for stream processing, and when not to
Use it when
- Stream: dashboards, fraud signals, live counters, anything where minutes-old is too old.
- Batch: billing, reporting, model training, anything that must be exactly right and can wait.
- Both: when a fast approximate number is shown immediately and corrected later by a definitive run.
Do not, when
- Do not build a streaming pipeline for a number someone looks at once a day. A scheduled query is enormously simpler.
- Do not build this like a request path. It has hours of slack, and the design should use it.
How stream processing actually works
- —Runs on a schedule over data already stored. Reads everything it needs.
- —Simple to reason about, easy to test, easily re-runnable when logic changes.
- —Late-arriving events are just included in the next run. No special handling.
- —Results are hours old, and a run over a huge dataset is expensive each time.
- —Right for: billing, reporting, anything definitive.
- —Processes each event as it arrives, updating aggregates continuously.
- —Results are seconds old, which is the reason to do it.
- —You must handle events arriving late and out of order, explicitly.
- —The job holds state, so a restart has to resume rather than start over.
- —Right for: live dashboards, anomaly detection, anything acted on quickly.
Windows are the core idea in streaming, because an unbounded stream has no natural end to aggregate over. A *tumbling* window is fixed and non-overlapping: every hour, on the hour. A *sliding* window overlaps: the last hour, recomputed every minute. A *session* window groups events by activity with a gap between them, which is how you measure a user's visit. Say which one you are using; "count per hour" is a tumbling window and "trending in the last hour" is a sliding one, and they cost very different amounts.
Event time versus processing time is the distinction that everything else depends on. An event happened at 13:59 on someone's phone and reaches you at 14:03, because they were in a tunnel. Do you count it in the 13:00 hour or the 14:00 hour? Processing time is easy and wrong for anything users will check against their own records. Event time is right and forces you to answer the next question: how long do you wait for stragglers?
That answer is the watermark: a declaration that you believe you have seen everything up to a point in event time, so the window can be closed and emitted. Set it too tight and you drop real events; too loose and every result is delayed by the allowance. Then decide what to do with anything that arrives after the watermark: drop it and count how often that happens, send it to a side output for inspection, or re-emit a corrected aggregate. A concrete answer here shows you have thought past the diagram.
Exactly-once, and its limits. The stream framework can often give you exactly-once *within* the pipeline, by checkpointing its state and its input positions together. What it cannot do is make an external side effect happen exactly once. So the rule is the same as everywhere: at-least-once delivery plus writes that are safe to repeat. Aggregate into a store where writing the same window result twice is harmless (keyed by window and idempotent), and the problem disappears.
Reprocessing is a design requirement, not an incident response. Logic will be wrong at some point, and you will need to recompute the last month with the fix. That means the raw events have to be kept, cheaply, in object storage, and the pipeline must be able to run over the archive as well as the live stream. Designing this in from the start is the difference between a bug being a re-run and a bug being permanent.
On architecture names, be brief and useful. The lambda architecture runs a batch layer and a streaming layer in parallel and merges them, which works and means writing the logic twice. The kappa architecture runs only the stream and handles corrections by replaying it, which is simpler if your log retention lets you. Naming them is fine. Explaining the trade, duplicate logic against dependence on replay, is what scores.
Finally, pre-aggregate early. Counting a billion raw events at query time is hopeless. Roll up as the data flows — per minute, then per hour, then per day — so a dashboard reads a few hundred rows instead of a few billion. And decide where approximation is acceptable, because HyperLogLog for distinct counts and a sketch for top-N turn impossible queries into cheap ones.
- Building a streaming pipeline for a number nobody acts on quickly.
- Aggregating by processing time and calling it hourly counts.
- No answer for late events beyond the watermark.
- Not keeping the raw events, so a logic bug can never be corrected.
- Claiming exactly-once without idempotent writes at the output.
The stream processing follow-up an interviewer will ask
"Your hourly click counts are wrong for last Tuesday. What do you do?"
The first question is whether they are wrong because of late data or because of a logic bug, since the fixes are different. If it is late data, the numbers were computed before some events arrived, so the pipeline needs to have kept those events and re-emitted the corrected windows. If it dropped them past the watermark, I need the count of how many, because a small percentage may be acceptable and a large one means the watermark is set wrong. If it is a logic bug, the fix is a reprocessing run, and whether that is possible depends on a decision made long before this incident: are the raw events still there? That is why I would always land raw events in cheap object storage as well as feeding them to the stream. Then reprocessing is running the corrected job over Tuesday's archive, writing to a separate output, comparing the two, and swapping. Two details make that survivable. The aggregates have to be keyed by window and safe to overwrite, so re-running produces the same result rather than adding to it. And anything downstream that already consumed the wrong numbers (a billing run, an email) needs its own correction path, which is usually the most expensive part. Cost: keeping raw events costs storage forever, and I would put a retention and tiering policy on it rather than keeping everything hot.
Your turn
An event happens at 13:59 but arrives at 14:03. Which hour does it count in, and what does your answer cost you?
Learn it from the source
This page gets you interview-ready. These take you to where the idea is actually specified or measured — read them when you want the real depth.
- Apache Kafka — Design ↗The log abstraction that lets the same data be read as a batch or as a stream.