Brandon Axtmann

Designing a Cost-Controlled, Compliant Observability Platform at Scale

The goal wasn’t just better dashboards — it was building an observability system that could scale technically, financially, and operationally.

Roughly two months ago, I led the design and rollout of a next-generation observability platform for a high-volume, multi-service environment. The goal wasn’t just better dashboards — it was building an observability system that could scale technically, financially, and operationally.

At this scale, observability is no longer a tooling decision. It’s a platform architecture problem.

The Core Constraints

Before touching any tooling, we defined a few non-negotiable requirements:

  • Extremely high volumes of metrics, logs, and traces
  • Predictable and controllable costs
  • Strong privacy and data-retention guarantees
  • Support for modern debugging practices (especially distributed tracing)
  • Operational simplicity for a small team

Many managed observability platforms struggle under these constraints because they assume long-term storage and querying will happen entirely within their ecosystem. That model becomes expensive quickly and offers limited control over retention and data movement.

High-Level Architecture

We landed on a hybrid observability architecture built around the Grafana ecosystem:

  • Metrics: Mimir
  • Logs: Loki
  • Traces: Tempo
  • Control Plane: Grafana Cloud (dashboards, alerting, auth, UX)

The key decision was to self-host the data-heavy components while relying on a managed control plane for everything user-facing.

Kubernetes as the Foundation

All observability backends run inside AWS EKS, giving us:

  • Consistent deployment patterns
  • Strong isolation between environments
  • Familiar operational tooling
  • Horizontal scalability as ingestion volume fluctuates

Each component (Mimir, Loki, Tempo) is deployed via Argo CD, allowing us to manage observability infrastructure declaratively and apply the same GitOps principles we use for application workloads.

This approach gives us:

  • Versioned, auditable configuration changes
  • Safe rollouts and rollbacks
  • Clear separation between configuration and runtime state

Short-Term Storage with EBS

For recent, high-performance access, we use EBS volumes attached to the observability workloads for short-term storage.

Key characteristics:

  • Approximately 24 hours of local retention
  • Optimized for fast writes and low-latency queries
  • Supports real-time debugging and incident response

This ensures engineers can investigate active or recent issues quickly without paying long-term storage costs for hot data.

Long-Term Storage via Object Storage

After the short-term window, data is pushed to object storage (S3) using the native long-term storage capabilities built into Mimir, Loki, and Tempo.

This gives us:

  • Cheap, durable storage for large telemetry volumes
  • Clear separation between hot and cold data
  • The ability to retain historical data without operational overhead

From there, S3 lifecycle rules tier data down over time:

  • Transitioning older data to lower-cost storage classes
  • Eventually expiring data entirely based on retention requirements

Cost Control by Design

This architecture puts cost controls directly into the system:

  • High-volume ingestion stays local and short-lived
  • Long-term data moves to low-cost storage automatically
  • Retention is enforced by infrastructure, not policy documents
  • Sampling and retention can be tuned per signal type

As observability volume grows — especially with AI-assisted development introducing more execution paths and traces — costs scale linearly and predictably, not exponentially.

Privacy and Compliance Considerations

Another major driver behind this design was data governance.

By controlling:

  • Where data is stored
  • How long it is retained
  • When it is deleted

We can align observability data handling with privacy and regulatory requirements without relying on vendor-specific guarantees.

Retention rules are enforced at the storage layer, which removes ambiguity and reduces compliance risk.

Unlocking Distributed Tracing

With Tempo fully integrated into this stack, distributed tracing becomes a first-class signal rather than an afterthought.

Teams can:

  • Trace requests across multiple services
  • Correlate traces with logs and metrics
  • Debug complex, non-obvious failures faster
  • Understand system behavior introduced by automation and AI-generated code

This level of visibility is increasingly critical as systems grow more dynamic and less manually authored.

A Platform Decision, Not a Tool Choice

This wasn’t about picking a trendy observability product. It was about designing a platform that balances:

  • Engineering productivity
  • Financial sustainability
  • Operational simplicity
  • Compliance and data governance

By combining self-hosted observability backends with a managed control plane, we ended up with a system that scales with us — technically and organizationally.

For me, this kind of work sits squarely at the intersection of architecture, operations, and leadership — and it’s where thoughtful design decisions deliver the most long-term value.