Running any service in production requires observability. Telemetry — metrics, logs, traces — is the feedback channel through which a system is operated: it shows whether the service is healthy, surfaces problems before users report them, and carries the evidence for root-cause analysis. The need is settled, and a mature industry serves it — Datadog alone passed $2.5 billion in annual revenue, and Cisco paid $28 billion for Splunk. This post questions none of that; it questions the cost behavior of the standard model.
If you operate observability for cloud-native infrastructure, this sequence is probably familiar. Telemetry costs climb for a few months. An invoice crosses a threshold and triggers an audit. The audit finds waste, filters are applied, costs come down. A few quarters later, it runs again. The pattern is common enough, and mechanical enough, to deserve a name: the Observability Pain Cycle. The claim here is structural: the cycle is not bad budgeting or the wrong vendor. It is the default behavior of the store-everything model that mainstream platforms are built on.
How store-everything became the default
The model was reasonable when it was introduced. On physical servers running monoliths, telemetry was hand-instrumented and volumes were small. “Send us all your telemetry and query what you need” was sound: storing some unused data cost less than deciding up front what to keep.
Two things have changed. Containers, microservices, Kubernetes, and distributed tracing multiplied telemetry-emitting components by orders of magnitude. And producing telemetry stopped requiring effort: auto-instrumentation, built-in library metrics, exporters, and eBPF ship it without anyone writing code — the decision to emit one more metric is made by a default, not an engineer. The ingestion model was never revisited. “Send us everything” scaled right along with the everything.
The signal-to-noise ratio
A modest Kubernetes cluster running the standard monitoring stack — node_exporter, kube-state-metrics, cAdvisor — produces tens to hundreds of thousands of active time series out of the box. The consuming side is far smaller, and it is measurable: Grafana’s mimirtool analyze walks your dashboards and alert rules and lists the series nothing references, and such analyses consistently find most ingested series feed no dashboard and no alert.
That remainder is not just an invoice problem. Every query runs against a larger corpus, and analysis — human or model — must find relevant series in a larger search space, raising the rate of spurious correlations. And nothing marks noise: no flag says “never queried.” It just ships, continuously.
The pipeline that produces it
The cycle follows from the architecture that runs it. Labels differ between vendors; the structure doesn’t.
Three properties matter.
The filter defaults to forward-everything. The agent sees every metric, log line, and span at the source — the right place for per-signal decisions — but it ships with pass-through as the default and hand-maintained drop rules as the only alternative.
Storage is unconditional; consumption is selective. Everything lands in the backend; dashboards and alerts read a small, stable subset. Which stored series feed a panel, an alert, an SLO — and which have never been read — is not part of the pipeline’s data model, and no collection decision is driven by it.
Control flows backward, and by hand. The only arrows pointing against the data flow originate at cost analysis: what happened to be stored determines what analysis can ask, and cost determines what gets cut. In control terms, the pipeline is open-loop with respect to value and closed-loop only with respect to cost.
This is why “transparent billing” doesn’t help. Every line item is accurate; none answers the operative question: which of these signals is worth paying for? Without a value signal in the loop, every pruning decision trades over-pruning — dropping the series the next incident needs — against under-pruning, keeping noise that keeps costing. Teams rationally choose under-pruning: a missing series during an incident is visible and attributable; the cost of retained noise is diffuse and attributed to nobody.
The cycle, step by step
Run this pipeline against a growing workload and the sequence is predictable.
- Collection defaults to everything
Onboarding follows the vendor’s recommended defaults: enable the integrations, turn on auto-instrumentation, scrape the cluster. Nobody made a mistake — this is the documented path. Volume grows with every service that ships, and nothing pushes back until an invoice exceeds what anyone expected.
- The bill triggers an audit
An engineer is assigned to find out where the money went. The audit reliably finds the same things — metrics nobody reads, debug logs at full retention, namespaces onboarded and forgotten. The waste is real, but it is measured after it has been paid for, and the audit consumes engineer-days planned for other work.
- Coarse filters bring cost down
The available remedies are coarse: drop these log levels, sample these traces at 10%, exclude these metric prefixes. Cost falls below the threshold that triggered the exercise, and the effort stops there — the target was never the optimum, it was the end of the pain.
- The workload drifts; volume regrows
New services launch; auto-instrumentation lands on a hot path; traffic shifts. The filters were calibrated against a workload that no longer exists, and nothing recalibrates them. Volume regrows silently until the next invoice makes it visible.
The chart’s shape follows from the only closed loop in the pipeline. The signal that triggers correction is the invoice — or a usage alert derived from the same meter — and it is late (volume already ingested and paid for), coarse (per data type or host, never per signal), and one-dimensional (cost, with no information about value). A loop with that feedback converges on the one target it can express — spend just below the pain threshold. Pruning starts when the bill hurts and stops when it doesn’t, never when the waste is gone.
None of this is hypothetical. The best-known data point is the $65-million-a-year observability commitment disclosed on a Datadog earnings call in 2022, widely reported to be Coinbase. The broader evidence is the genre of engineering posts titled “how we cut our Datadog bill by 70%” — each a written record of steps two and three.
Why the model persists
The pricing model rewards volume. Observability revenue scales with ingestion — per gigabyte, host, span, or active timeseries. Under that model, the noise fraction of your telemetry is not a defect from the vendor’s perspective; it is revenue. Not bad faith — an incentive constraint, and it predicts what we observe: cost pressure is answered with cost-management features that moderate the symptom while leaving volume-based charging, and therefore the cycle, intact.
The retention argument protects the noise. “You can’t know which signal the next incident will need” is true — for a thin slice of your telemetry. For the large majority of automatically emitted series, the probability of ever being queried is negligible. The argument works because the platform offers no way to locate the boundary, so “keep everything” is the only decision that requires no analysis.
“Storage is cheap” understates the cost. Bytes at rest are a fraction of it: you also pay ingestion and indexing compute, query compute over a growing corpus, and network egress — and the latency and analysis costs are paid at read time regardless.
Self-hosting does not exempt you. Running your own Prometheus, VictoriaMetrics, or Loki removes the vendor margin, not the model. The invoice becomes infrastructure spend plus the engineers who operate the cluster, so the pain threshold sits higher and the feedback is weaker. The self-hosted bill shock is well documented: a deploy adds a high-cardinality label and Prometheus announces it by OOM-ing.
Why the fixes don’t break it
AI features sit downstream of the problem. Anomaly detection, incident correlation, natural-language querying — operationally useful, structurally applied on top of data already collected, with no influence on whether it should have been. The noise degrades them too: every irrelevant series widens the search space and raises the rate of spurious findings.
So do the classic cost controls: sampling, shaping pipelines, cardinality pruning, usage caps. They decide how much data to keep, not which data matters. The exception proves the rule: tail-based trace sampling — keep the slow and failed traces — is a genuine per-signal value judgment, possible because traces carry an intrinsic value marker the pipeline can read: latency and error status. Metrics and logs have no such marker; their value lies in whether anything downstream uses them, which is exactly the information these pipelines don’t consume.
The newest tooling gets closer, and deserves credit. Grafana’s Adaptive Metrics recommends aggregating away series your dashboards and rules don’t use; Datadog’s Metrics without Limits makes unqueried custom metrics cost less. Note the concession — the vendors’ own tooling now assumes a large share of what you send serves no query — and note where it runs: after ingestion, on the vendor’s side of the meter, as advice a human reviews and applies. All of this lowers the peaks. None of it removes the loop: cost comes in, filter adjustments go out, and value never reaches the collection layer.
Recognizing the cycle
A practical test: list the telemetry-reduction efforts your organization has undertaken, and what initiated each. If every entry traces back to a bill, the loop is operating — whichever vendor is on the other end. That is not a verdict on the teams involved: bill shock, reactive pruning, and silent regrowth are not incidents to be managed better. They are the equilibrium of a pipeline in which collection is driven by defaults and corrected only by cost.
One objection any fix must survive: observability exists to answer questions you didn’t know you would ask. A fix that keeps only a hand-picked allowlist trades bill shock for blind spots and reinvents static monitoring. So the requirements read like an engineering spec. Information about value — what is consumed, and by what — must reach the collection layer and drive it continuously, not surface once a quarter in an audit. The ability to answer unexpected questions must survive. And the change is unlikely to be led by whoever profits from volume. That is not a pruning feature; it is a different control loop. What it looks like in practice is the subject of the next post.
← All posts