In the previous post I described the Observability Pain Cycle. Everything is stored up front, only a small part of it gets used, and the one thing that ever triggers a correction is the pain of a high bill. The correction is a one-time trimming exercise, the volume silently grows back, and the cycle starts over. The party that knows best what is valuable, you, has little say in that loop.
This post is about the opposite loop. It runs on the same architecture: the same workloads, the same agent, the same dashboards and alerts. Only two defaults are flipped. The result is a system where you decide what gets stored, and the cost follows that decision instead of preceding it. I call it the Observability Value Cycle.
Two defaults, flipped
The Pain Cycle rests on two defaults. The agent forwards everything unless somebody tells it otherwise. And whatever happens to land in the store dictates what you can ask of it later. Both favor the vendor, and neither is a law of nature.
The Value Cycle turns them around:
- The agent forwards nothing by default. Workloads emit telemetry as before, but nothing reaches the central store just because it was emitted.
- A demand opens the filter. A demand is a declared need for specific telemetry: these metric series, these log streams, the traces of these services. What is demanded gets stored. What is not demanded stays out.
This may sound like a lot of declaring, but most of the declarations already exist. A dashboard panel is a query, and the query says exactly which series it needs. The same goes for an alert rule. The things that consume telemetry already describe the demand, and the platform can read it straight out of them. For everything else (data you want to keep for future investigations, for an audit, or simply on a hunch) you write an explicit rule.
How it works
The boxes are the same as in the Pain Cycle diagram. What changed is shown in green.
This is how the story unfolds this time:
- You onboard the same way. Enable the integrations, turn on auto-instrumentation, scrape the whole cluster. The agent still sees every metric, log line, and span at the source. But instead of passing it all through, it indexes what it sees and keeps a short full-fidelity window locally: think of an hour of logs and traces and a day of metrics. Nothing is forwarded yet, so the central cost starts near zero.
- You explore at the source. The agent reports the index of what it sees, not the data. So the platform knows everything that could be stored without storing any of it. You browse what is available, run queries against the local window, and build a dashboard on real data. The punishment from the Pain Cycle is gone: a closed filter no longer means that you can’t see what you are filtering.
- Saving a dashboard declares the demand. The moment you save the dashboard or activate the alert, the platform extracts the selectors from its queries, recomputes the full set of what is demanded, and pushes it to the agent. The agent starts forwarding the new series within seconds. No redeploy, no pipeline configuration, no ticket.
- Every stored signal has a reason. The stored set is derived from its consumers, so the link between a signal and its use exists by construction. “Why are we storing this?” and “What does this dashboard cost us?” become questions with exact answers.
- When the need goes away, the cost goes with it. Delete the panel or pause the alert, and its series are released, unless another consumer still needs them. And when a new service launches or auto-instrumentation lands on a hot path, the new telemetry stays at the source. It can reach the store only through a demand, so the volume can no longer grow back silently.
The shape of the cost
In the Pain Cycle, the signal that triggers a correction is the invoice, and it is late and coarse. In the Value Cycle the signal is the demand, and it has the opposite properties. It is early: the decision is made before the telemetry is ingested, so the waste is never paid for. And it is fine-grained: it is made per panel, per alert, per rule, by the person who knows why the data is needed.
The shape of the curve follows from that. The cost starts near zero and grows only when the demand grows. When a team retires a dashboard or a service gets decommissioned, it goes down again. There is no spike at the start and no cleanup project in the middle, because a pile of unused telemetry never builds up.
How big the gap is depends on how much of your telemetry you actually use. It is biggest at the beginning, when the Pain Cycle already stores everything and uses nothing. But a gap remains even against a carefully pruned setup, because pruning is a snapshot that starts to drift the next day, while the demand is always current. And a smaller store brings a benefit that never shows on the invoice: queries run over the data you care about, not over the haystack around it.
The obvious objections
“What about the signal the next incident will need?” This is the honest core of the “you never know” argument, and the Value Cycle has to answer it. It does so in two ways. The incident that is happening right now is covered by the window at the source: everything the agent saw recently is there in full fidelity, demanded or not, and you can query it. And the data you already know you want for later investigations is a demand like any other. You declare it, it gets stored, and it comes with a visible price. What you give up is long history for signals nobody ever asked for. That is a real trade-off. But you make the same trade-off in the Pain Cycle every time you sample, drop, or shorten retention after a bill, only blindly and under pressure.
“So it is an allowlist.” It is not. A hand-maintained allowlist is the Pain Cycle’s drop filter with the sign reversed: somebody guesses what is needed, writes it down, and the list is outdated a month later. Here nobody maintains a list. It is derived from the dashboards, alerts, and rules that exist right now, and recomputed whenever one of them changes. And you never choose blindly, because everything that exists is visible at the source.
“So it is sampling.” Sampling and the other volume controls decide how much data to keep. A demand decides which data to keep. A demanded series is stored in full, and an undemanded one is not stored at all.
“It means re-instrumenting everything.” It does not. The workloads emit what they emit today, over the same protocols. Dashboards and alerts are written in the same query languages. The only thing that moves is the decision about what to store, and it moves from the vendor’s defaults to you.
Conclusion
The Observability Value Cycle is the Pain Cycle with two defaults flipped: forward nothing unless it is demanded, and let the demand, not the stored data, drive the loop. The architecture stays the same. What changes is who is in control. You decide which questions your observability should answer, the system stores what those questions need, and the bill becomes the sum of your own decisions.
None of this requires exotic technology. So why does it still look unusual? Because it is not a feature that can be added on top of a store-everything platform. It is a different way of thinking about telemetry. And our industry would not be the first one to go through such a change. Manufacturing went through it decades ago, when it learned that producing more than anybody asked for is not a safety margin but the most expensive kind of waste. That is the topic of the next post.
← All posts