You cannot backfill a metric you never took

At the end of a twenty-month platform project, I tried to answer some basic questions about how it had run. Most had answers. Five did not, and why they did not is the more useful part.

The answers feed the rest of this series, such as What 20 months of AI adoption measured and Audit, then optimize. This post is the other half.

TL;DR:

• Five metrics were missing when I looked. Two are permanently gone, one is bent, and two never existed.
• The permanent losses were all retention windows. Seven months of cost history, and every alarm older than a fortnight.
• The widest loss was one checkbox. A root-account setting nobody had ticked, worth seven months of spend history, free, and available from day one.
• The two that never existed share a cause. We had no incident tracker, so recovery time and customer impact have no source anywhere.
• Day-one instrumentation means deciding which windows to widen before the clock starts, because the clock starts whether you decide or not. Dashboards can wait.

What actually goes missing

The interesting thing about the five is that almost none of them were hard.

Nobody had to build a data warehouse. In most cases somebody had to make a small decision, early, about how long a thing would be kept. While chasing milestones, we postponed the instrumentation, first by weeks, then by months.

The same project, seen through four different vendors' retention windows.

The two that are gone, and one that is bent

Seven months of cost history. Our billing console, AWS Cost Explorer, keeps 14 months by default, and the project ran longer. Everything before April 2025 was unreadable by the time I looked.

A setting extends that window to 38 months. It can only be enabled from the root account, it costs nothing, and it was never switched on. Those seven months are now unreconstructable by anyone.

Every alarm older than two weeks. Our monitoring, CloudWatch, showed 14 days of alarm state changes when I looked. Its documentation today says 30. So I can tell you exactly which alarms were noisy in that fortnight, and nothing about the previous year and a half. That fortnight is the whole surviving alarm history.

The real cost numbers, distorted rather than lost. The account runs on credits, so the dollar columns show a fraction of list price. There is a usage-quantity column that credits cannot touch, and it is the only honest signal in the file. That one is recoverable, as long as you know to distrust the obvious column.

The two that never existed

These are different, and worse, because no retention setting would have saved them.

We had no incident tracker. Not a lightweight one, not a spreadsheet. So there is no true recovery time, the metric DORA tracked as MTTR, because there is no record of when anything was noticed. There is no per-incident severity and no customer impact, for the same reason.

A rollback took one to six minutes, but that measures a traffic swap, not the hours before someone noticed. Those hours could be dug out of the team chat, except our Slack keeps only twelve months.

That is a whole category of engineering evidence that simply is not available about twenty months of my own work. Not because it was deleted, but because nothing ever wrote it down.

Day one, concretely

If I were standing at the start of the next platform, this is the list, and it is short on purpose.

1. Widen the retention windows you will actually query

Find each vendor console’s retention setting and extend it where you can name a question it answers. An hour of clicking, never in anyone’s backlog.

2. Write incidents down somewhere, anywhere

A tracker is nice. A dated text file with what broke, when someone noticed, and when it was fixed is enough to produce every number I could not produce. The tool does not matter. The habit of recording the noticing is the whole thing, because that timestamp exists nowhere else in your infrastructure.

3. Copy the short-window data out on a schedule

Anything with a window under a month is telling you it will not be there when you look. A weekly job appending the last seven days of throughput to a file in your own storage turns a seven-day window into a permanent record, and it is an afternoon of work.

4. Ask the question once a year, not once at the end

Pick a question you would want answered about the last twelve months and try to answer it today. Every gap you hit is a gap that is already permanent for the period behind you, and still fixable for the period ahead. This is the only item that catches problems the other three missed.

The danger zone

Conclusion

Every metric has a start date, and it is the date you decided to keep it, not the date the system started running. Nothing about a good engineering culture moves that date backwards.

What would change my mind: the claim is that collection, not analysis, is the binding constraint early on. Suppose a team does all five from day one. If, two years in, it still cannot answer basic questions about its own delivery, the constraint was never the data. I would rather be shown that than keep repeating this.

Next in this series: what 5.7× more pull requests actually cost us.

---

Methodology & limitations (click to expand)

Prior art

• Multi-year data at monthly granularity, AWS Cost Management user guide. Confirms the default 14-month history and the setting that extends it to 38 months “for your entire organization”. The root-account-only detail is from the project notes, not this page.
• Using Amazon CloudWatch alarms, Amazon CloudWatch user guide. The page today says alarm history is kept for 30 days. The console showed 14 in May 2026, which is the figure the post uses. Either way, the point stands: it is weeks, not months.
• Configuring the retention period for GitHub Actions artifacts and logs, GitHub Docs. Logs are kept 90 days by default, the three months in the chart. Adjustable up to 400 days for private repositories, which is the same free checkbox in a different console.
• DORA’s software delivery performance metrics, DORA. The framework the “two that never existed” belong to. The recovery metric moved from MTTR to failed deployment recovery time; neither can be produced without a record of when the failure was noticed.

Data sources

• The five missing metrics are the unavailable, blocked and lost table from the project’s own data documentation, written in May 2026 rather than reconstructed afterwards.
• The retention figures are the vendor limits recorded in that documentation as the reason each metric was unavailable, checked against the vendors’ current pages on 14 September 2026 (see Prior art for the one that has since changed).

Definitions and bounds

• “Permanently gone” means gone to a retention window that had already passed when I looked. It does not mean the underlying events never happened, only that no source of truth for them survives. Two rows qualify. The distorted cost column does not, because its usage-quantity column survives intact.
• The two that never existed both trace to the absence of an incident log, so they are arguably one gap counted twice.
• The one-to-six-minute rollback figure measures a routing swap and is included only to show what it is not.

What this does not show

• No counterfactual. I cannot demonstrate that widening the windows on day one would have produced better decisions, only that it would have produced data. Those are different claims and this post only supports the second.
• The retention limits are as documented in May 2026 and vendors change them. CloudWatch already has, from 14 days to 30. Check your own consoles rather than trusting these numbers.
• One platform, one team, one look back. The five gaps are what a particular set of choices produced, not a survey.

---

Get new posts by email

One email per post, about two a month. The numbers and the caveats, same as here. No sequence, no pitch, unsubscribe in one click.

Email address Subscribe

No tracking pixels. I never pass the address on. How this is handled.