20 months on a platform for real-time quality analysis of video streams

The pipeline could only process the present

· 7 min read

We ran a platform that watched live sports video and reported when the picture broke. In February 2026 an infrastructure failure left several days of video segments sitting in storage with nothing processing them. Every file was intact. Not one of them could be replayed.

This series has mostly measured how we worked. What 20 months of AI adoption measured is the delivery side of this platform. This post is about what we built, and the assumption every stage of it shared.

TL;DR:

  • Nothing was lost. The segments were in storage, complete. They had simply never triggered the detectors that read them.
  • The pipeline could only understand the present. Every stage stamped the moment it ran and treated that as when the thing happened.
  • So a replay would have lied. A segment reprocessed on 12 February would claim to have happened on 12 February.
  • Kafka then dropped it anyway. Our stream windows ran with no grace period, so anything arriving late was discarded with no error and no log line.
  • The fix took about a week of work and 2 pull requests. One threaded a real timestamp through 15 files. The other built the rewind tool, which got reused a few times within a month.

Four clocks, and the pipeline trusted the wrong one

A video segment passes four hands before it becomes a fact on a customer’s timeline. Each one knows a different time.

  • The downloader clock. When the segment was actually pulled off the stream. This is the only one that describes the video.
  • The stored clock. When the object landed in storage, available as the event time on the notification.
  • The detector clock. When the detector ran and looked at the frames.
  • The Kafka processor clock. When the producer built the message and put it on the topic.

In normal running these are seconds apart, so nothing looks wrong. During a replay they are days apart, and only the first one is still true.

The field has a name for this. The Dataflow Model calls the first one event time, “the time at which the event itself actually occurred”, and the rest processing time.

Its own summary of the difference is the sentence I wish we had read first. “Event time for a given event essentially never changes, but processing time changes constantly for each event as it flows through the pipeline.”

The decision record we wrote lists three separate places the wrong one won. The Kafka producer hardcoded a local now() for the segment start time. The detectors passed no timestamp at all, so there was nothing better for the producer to use. And the stream windows were built with ofInactivityGapWithNoGrace().

The word in the middle of that method name

ofInactivityGapWithNoGrace() is a real method, and it does exactly what it says. The javadoc spells it out in a block headed CAUTION: records arriving after the window ends “are considered late and will be dropped”.

A session window groups events that arrive close together. The grace period is how long the window stays open for stragglers. With no grace, an event that turns up after its window closed is not an error. It is not a warning. It is discarded.

So even with a correct timestamp, replayed data from three days ago would have been dropped on arrival, silently. Two independent defects, and either one alone was enough to make the backlog unprocessable.

One timestamp, carried instead of invented

The shape of the fix is three decisions, and the first is the one worth copying. It spanned all components in our pipeline.

Resolve the timestamp from the most trustworthy source available, in a fixed order:

Diagram: two pipelines. Above, every stage stamps now() and Kafka drops the late event. Below, one segment timestamp resolved at the detector and carried unchanged through the producer, the topic and the window, which now has a grace period.

The change is not a new timestamp. It is one timestamp that survives the journey, and now() demoted from the only answer to the last resort.

The storage metadata field is set once at upload and never changes. The event time on the notification is close but slightly late. A local now() is still there at the end of the chain, and that matters. Nothing breaks when the metadata is missing, and old files without it still replay from the fallback.

Stamp it at upload from then on, so the next incident has the good value waiting.

Widen the window to a 14-day grace period, so a replay of anything inside two weeks is accepted rather than discarded. Calculate the burden on your system with bigger grace periods.

The rewind tool is the actual deliverable

The replay itself was a script that lists the stored segments for a fixture in order, builds synthetic events, and calls the detectors directly. It does not touch the stored files at all. That was a deliberate rejection of the easier option, copying objects in S3 to re-trigger notifications.

The tool’s own description says what it is for: a reprocessing endpoint that “can stay for future reprocessing in case of retrospective fixes”.

That sentence is the whole lesson. It was written while the outage was fresh in my mind, and it turned a one-off recovery into a standing capability.

It paid back within a month.

The tradeoff

The 14-day grace period was not free, and the decision record’s guess that the impact would be negligible was about storage, not about time.

The bill arrived on restart. Our deploys wiped the local state store every time, so the consumer came back with no memory of what it had seen. It then did exactly what we had told it to, and worked through two weeks of backlog.

So the honest version is a trade, not a win. We bought the ability to go back in time, and we paid for it in how slowly the system came up. If you widen a window, budget for the restart before you find out the hard way.

The checklist

  • Never hardcode now() in a pipeline stage. It records when your code ran, which is a fact about your infrastructure and not about the event.
  • Carry the timestamp; do not re-derive it. Every stage that recomputes is a stage that can disagree.
  • Resolve from a field a copy cannot change. Storage metadata set at upload survives; a last-modified date does not.
  • Expect real-time data not to arrive in real time. Even in a live system, plan for the message that shows up late or has to be sent again.
  • Read the grace period on every window. No-grace is a common default and it discards late data without telling you.
  • Build the rewind tool while the incident is fresh. An endpoint that takes a range of IDs and reprocesses them costs a day now, and you will use it again.

Conclusion

A distributed system does not have a time problem. It has an ownership problem that reports itself as a time problem, once per component, until somebody writes the rule down.

What would change my mind: this claims the rewind tool was the durable win, not the timestamp fix. Suppose a team builds one and never reaches for it again. Then it was insurance nobody claimed on, and the week was better spent elsewhere. Count the reuses. Ours was reused a few times within a month.

Next in this series: the workflow engine we built without noticing we were building one.

Methodology & limitations (click to expand)

Data sources

  • The decision record on timestamp propagation and retrospective reprocessing, created 12 February 2026 and last edited 3 March, read directly.
  • Pull request catalogue: 963 merged pull requests across 16 repositories, September 2024 to May 2026. The file counts, commit counts and review-comment counts here come from it.
  • The April containment work is three pull requests from the same catalogue.

Prior art

  • The Dataflow Model (Akidau et al., Proceedings of the VLDB Endowment, Vol. 8, No. 12, 2015). Section 1.3 defines event time and processing time, and the “inherent and dynamically changing amount of skew between the two domains”. The four clocks in this post are one pipeline’s version of that skew, named locally because nobody on the team had read this paper.
  • Kafka Streams SessionWindows (javadoc, 4.0.2). The source for the grace-period behaviour described above, including the CAUTION quoted in the body.
  • Designing Data-Intensive Applications, Martin Kleppmann (O’Reilly). The standard reference for the stream-processing side of this, and where I would send someone who wants the general case rather than our incident.

Definitions and bounds

  • “Several days” of unprocessed segments is the decision record’s own wording. The exact span was not recorded anywhere I can still read.
  • “About a week of work” is my own recollection of effort. The two pull requests merged 14 days apart, on 12 and 26 February, which is calendar time and includes review.
  • The four-clock framing is mine. The code does not name them that way. The three failure points it describes are the record’s own list.

What this does not show

  • No before-and-after defect rate. I cannot show that this class of bug became rarer, only that the capability was reused.
  • Description text is truncated in the archive at roughly 300 characters, so the reasoning inside these pull requests beyond the opening summary is not recoverable.
  • The replay covered a bounded period. The grace period was 14 days, so anything older than that was outside what this fix could recover, and the record flags the hardcoded value as a limitation.