20 months on a platform for real-time quality analysis of video streams

Our CI caught the loud failures and missed the quiet ones

· 5 min read

Our nightly end-to-end run failed to start one night. Then it failed a few nights in a row, and nobody touched it.

In the same weeks, a red check on a pull request was fixed in about half an hour. Same team, same pipeline, same tools. The only difference was whether someone was waiting on the failure.

TL;DR:

  • A failure that blocks a merge gets fixed fast. A red pull-request check went green again in a median 34 minutes.
  • A failure that blocks nobody waits. The nightly run stayed broken, and a production deploy said “success” while skipping its change.
  • So give every job that blocks nobody an owner, and treat “did not run” as red.

Failure types

Three CI jobs and who waits on each. A pull-request check has the author waiting and was fixed in a median 34 minutes. The nightly end-to-end run and the production deploy have nobody waiting: one failed to start night after night, the other said success and skipped its change.

Loud: the pull-request check. When one of our pull-request checks failed, it was fixed in a median 34 minutes. That is not discipline. A red check blocks the merge, and the author is sitting there waiting for it.

When a failure blocks someone, it gets fixed in minutes.

Quiet: the nightly run that never started. The end-to-end suite ran on a timer, so no pull request waited on it. When it broke, it failed to start night after night, and nobody re-ran it or fixed it.

Quiet: the deploy that skipped its own work. A production deploy reported success while skipping its infrastructure change. One line of shell read the wrong exit code, so “there are changes” always read as “no changes”. The same bug had quietly skipped every staging deploy for five days.

A green run proves the script finished. It does not prove the work happened.

What we changed

Three fixes from those weeks share one idea: make a quiet failure loud.

  • The version check stopped warning and started failing. A warning in a CI log is read by nobody.
  • Production apply got a manual approval. The plan goes into the run summary, and a person approves it. It replaced a blind auto-approve.
  • The deploy reads the wrapper’s exit code, so it acts on what Terraform said.

None of this is new. Google’s SRE book says every page should be actionable. A CI signal is a page with a slower pager.

The checklist

1. Name who waits on each job's failure

For each workflow, write down the person blocked when it goes red. A pull-request check has an author. A nightly job or a post-merge deploy usually has nobody. Those are your quiet ones.

2. Route quiet failures to a person

A failed scheduled job should open an issue or post to the team channel, with a name on it. The Actions tab only works for someone who goes and looks.

3. Treat "did not run" as red

Check that the job produced today’s result, not only that nothing failed. A job that never starts produces no failed test to notice.

The danger zone

The tempting fix is to make everything block the merge: end-to-end on every pull request, every slow job as a required check.

That puts the cost on every author and every push, at a hundred runs a day. A slow, flaky gate gets overridden or switched off, and you have a quiet failure again. Keep slow jobs off the merge path, and give each one an owner.

Conclusion

A CI failure gets fixed when it blocks somebody. Everything else needs an owner, or it waits until someone happens to look.

Next in this series: what the AI reviewer actually said, across 5,833 of its comments.


Methodology & limitations (click to expand)

Prior art

Data sources

  • Runs: every GitHub Actions run across three repositories, 7 April to 22 May 2026. That is 46 days and 4,704 runs, one row per run, with the team using coding agents daily.
  • The CI changes: the merged pull-request catalogue. The skipped apply’s resource counts are self-reported in the fix’s description; its success conclusion is in the run list.

Definitions and bounds

  • The window starts 7 April because both large repositories return exactly 3,000 runs, which reads as an extraction cap. So there is no before-and-after claim here.
  • Red to green runs from a branch’s first failed check to its next successful one. It measures fix plus rerun, not effort.
  • The nightly run last succeeded on 14 April and failed to start every night after, to the last day of data. That makes its run a lower bound.

What this does not show

  • Whether anyone saw the nightly failures, or whether any check was required. Neither was captured.
  • That agents caused any of this. The window sits inside daily agent use, with no comparison period for the same repository.