Observability that travels: portable in-cluster to centralised multi-region

The problem

Two clients, two different observability problems, both rooted in the same underlying question: where should visibility live relative to the workloads it watches?

The stealth-mode AI company (see our multi-cloud LLM platform and cloud-agnostic platform case studies) needed observability that was as portable as the rest of their stack. If the platform could move to any cloud without a rewrite, the observability layer could not be the exception that stayed behind, dependent on a central system that might not exist in a new environment.

The AI-native HR platform (see our multi-region EKS case study) had the opposite shape of problem. Each region ran multiple EKS clusters - split by environment, and sometimes multiple clusters per environment - each uniquely named. Operators needed one place to look per region, not a dozen separate Grafana instances with no shared context.

What we built: portable, in-cluster (AI infrastructure client)

Every cluster in this platform ships with its own complete observability stack, entirely in-cluster, with no dependency on anything centralised:

• Loki for logs, with S3-class object storage as the backend.
• Mimir for metrics, Prometheus-compatible via remote write, also backed by S3-class object storage.
• Tempo for distributed traces, backed by the same S3-class object storage pattern.
• Kiali, giving service-mesh visibility into the Istio layer already running in every cluster - request flow, error rates, and traffic shape between services, without a separate tracing setup.

Because logs, metrics, and traces all persist to S3-class storage rather than local disk, a cluster can be rebuilt or moved without losing its observability history, as long as the underlying bucket survives. Nothing about this stack depends on a central Grafana, a shared Mimir, or any service outside the cluster - the same self-contained stack deploys to any cloud as part of the platform's Helm releases, exactly like every other component in this platform.

What we built: centralised per region (HR platform client)

Here the shape of the problem called for the opposite answer. Each region runs multiple EKS clusters - environment-specific, sometimes several clusters per environment - and each cluster carries a unique name used consistently across every signal it produces.

Rather than give every cluster its own Grafana, each region has one centralised observability cluster. Every workload cluster in that region runs its own Grafana Alloy instance, which tags every log line and metric with metadata identifying its source - cluster name, environment, region - before remote-writing it to the region's central observability cluster. Multiple Alloy instances, one destination per region.

The central cluster runs Loki for logs and Mimir for metrics, both backed by S3-class object storage, with Grafana as the single visualization layer for the region. Dashboards use custom template fields for environment and cluster, so operators filter to exactly the cluster or environment they need from one Grafana instance, without losing the ability to see the whole region at once.

Centralisation happens within a region, not across regions - consistent with how the rest of this platform keeps regions operationally independent. There is no global observability cluster aggregating every region into one place.

Two shapes of the same discipline

Both answers are correct for what they serve. The AI infrastructure platform needed observability that could not become the thing left behind when the platform moved clouds, so it stays in-cluster and self-contained everywhere. The HR platform needed one coherent view across several clusters serving the same region, so observability centralises - but only as far as the region boundary, never past it.

Key outcomes

• Portable stack moves with the platform. Loki, Mimir, and Tempo, all backed by S3-class storage, ship as part of the same Helm releases as every other component - no cluster depends on anything external for its own observability.
• Kiali gives Istio visibility for free. Because the mesh is already there, service-level observability comes with it - no separate tracing integration needed.
• One Grafana per region, not per cluster. Operators get a single, coherent view across every cluster in a region, filterable by environment and cluster name via custom dashboard fields.
• Metadata makes multi-cluster legible. Alloy tags every signal with cluster name, environment, and region before it leaves the source cluster, so nothing arrives at the central cluster unlabelled.
• Regional isolation is preserved. Centralisation stops at the region boundary. No single observability cluster spans regions, matching the same isolation principle used throughout the platform's infrastructure.