Real-time data tells you what is happening. Historical data tells you what normally happens. Neither is useful alone: a live number with no baseline is just a number, and a baseline with no live feed is a report about a world that has already moved on. The synergy is that history turns a real-time reading into a judgment. It supplies the expected value, the normal variance, and the shape of the curve, so the live stream can be scored as “on track”, “anomalous” or “act now”. Technically, you get this by treating both as one thing: a single log of timestamped events that you process as it arrives and keep for later analysis. Everything below is about doing that without fooling yourself.
What can each kind of data actually tell you?
| Real-time data | Historical data | |
|---|---|---|
| Answers | Is something happening right now? | Is this normal, and where does it lead? |
| Typical latency | Sub-second to a few minutes | Hours to years |
| Strength | Speed of reaction | Statistical confidence, seasonality, trend |
| Weakness | Noisy, incomplete, often revised later | Describes conditions that may no longer hold |
| Fails when | Used without a baseline | Used without checking the present |
Managed streaming services now deliver the “real-time” half cheaply. Amazon documents a put-to-get delay of under one second for Kinesis Data Streams, and Apache Kafka offers similar latency on your own hardware. Speed is no longer the hard part. Interpretation is.
How do the two combine in practice?
Three mechanisms cover nearly every real use case.
- Baselines for anomaly detection. History defines the expected range for this hour, this weekday, this season. The live value is compared against it, not against zero. A spike in traffic at 9 a.m. Monday may be normal; the same spike at 3 a.m. Sunday is an incident. Security operations formalize this: NIST’s SP 800-61 Rev. 3 treats incident detection as spotting deviations from documented normal behavior, which requires retained logs.
- Forecasts updated by live evidence. A model trained on past outcomes gives a prior; incoming events shift it. Early readings are weighted by how early readings have historically related to final results.
- Context for a single event. A customer complaint, a sensor alarm or a failed payment means something different depending on the account’s history. The live event triggers the lookup; the history decides the response.
We see the forecasting case every time we launch a crowdfunding video. Kickstarter publishes its own live project statistics, and roughly four in ten projects fund. The dashboard on launch day is pure real-time data: backer count, pledge total, referrer breakdown, refreshing by the minute. On its own it is almost useless, because every campaign starts at zero and the founder’s nerves do the interpreting.
What makes that number meaningful is the pledge curve of previous campaigns in the same category and price band. Across the campaigns we have produced video for, the ones that reached roughly a quarter to a third of their goal inside the first 48 to 72 hours almost always finished funded, and the ones below about ten percent at that point rarely recovered without a change in ad spend or creative. Those are our observations, not a law, and the thresholds shift with goal size and category. But they turn “we have $14,000” into “we are tracking ahead of the median funded campaign in this bracket”, which is a decision you can act on.
What goes wrong when you only watch the live feed?
The most expensive mistake we see in marketing is killing creative too early. A video ad gets launched, the first six hours show a cost per result twice the target, and someone pauses it. Paid social platforms need a volume of conversions before delivery stabilizes, and early cost readings are both noisy and systematically pessimistic because conversions keep arriving for days after the click. The historical fix is simple: look at how your own past winners performed in their first day, and set the kill threshold from that distribution rather than from the target.
The second failure is trusting real-time numbers that will later be revised. Ad platforms backfill conversions, analytics tools reprocess sessions, and payment processors reverse charges. Google’s standard GA4 reports can lag a day or two before they settle. If your dashboard treats today’s partial figures as final, every day looks like a dip.
The third is losing the history you thought you had. Google stopped processing Universal Analytics data on July 1, 2023 and began deleting it a year later. Clients who never exported now have no comparable baseline before mid-2023. GA4’s own event data retention defaults to two months for explorations, with 14 months as the maximum on standard properties. Year-over-year comparison is impossible under the default setting unless you export to a warehouse. Check that setting today; it is the single most common cause of “we have real-time data but no history” that we encounter.
How should the plumbing be built?
The old answer was the Lambda architecture: one batch pipeline for history, one speed layer for live data, and reconciliation logic to stitch the two. That meant maintaining two codebases that computed the same metric and disagreed. Jay Kreps argued in Questioning the Lambda Architecture that a single replayable log could serve both roles, and that is what most new systems do now.
Three developments made the single-log approach practical:
- Event time, not processing time. Every record carries the time it happened, and the engine reasons about when it is safe to close a window despite late arrivals. The Dataflow Model paper from Google laid out the watermark approach, and Apache Flink implements it. Without this, a user whose phone was offline for an hour corrupts your hourly totals.
- Cheap long retention on the log itself. Kafka’s tiered storage moves older segments to object storage, so the same topic you read live can be replayed from months back.
- Open table formats. Apache Iceberg and similar formats let streaming writers append to the same tables that analysts query, with time travel to any earlier snapshot. Streaming and batch read one copy of the data.
For a small or mid-sized company none of this needs to be self-hosted. The practical version is: send every event, with an event-time timestamp and a stable schema, to one place; stream it to a dashboard for the live view; land the same stream in a warehouse for the historical view; and compute every baseline from the warehouse, never from a spreadsheet someone exported in 2023.
How much real time do you really need?
Latency costs money and attention, and most decisions do not improve below a few minutes. Ask what action you would take on a one-second signal that you would not take on a five-minute one. Fraud blocking, live broadcast monitoring and industrial safety have an honest answer. Marketing performance, inventory planning and content strategy usually do not.
Our own split is telling. During a live-streamed event we watch encoder bitrate and dropped frames second by second against the rehearsal baseline, because a stall costs the client the show. For a launch campaign we check a dashboard a few times a day and make creative decisions weekly, because the data that matters takes days to settle. Same company, same clients, two very different clock speeds, each chosen by asking how fast the underlying thing can actually change.
Start by writing down, for each metric you watch live, where its baseline comes from and how old that baseline is. Any metric with no baseline gets one from your warehouse or gets removed from the dashboard. Then confirm your analytics retention is set to the maximum and that raw events are exported somewhere you control. Only after that is it worth spending on lower latency.
