Multi-Cloud LLM Platform on EKS, AKS and GKE

The problem

A stealth-mode US AI company had a working proof of concept on Amazon EKS that had grown without structure. Everything assumed a single cloud, AWS-managed services were reached for wherever convenient, and there was no clean boundary between serving, orchestration, and data.

Their customers were spread across the world, some on Azure or Google Cloud, and several needed inference in their own region. The team could not keep forking the codebase per cloud. They needed AWS, Azure, and Google Cloud - region by region - before their next wave of launches, with as much of the platform stack as possible portable by design rather than rebuilt each time.

What we built

We re-architected the proof of concept into a modular platform with hard boundaries between serving, orchestration, and data. Provider-specific dependencies were pushed to the edges - wherever a Kubernetes-native component could do the job, we chose it over a cloud-managed service.

The serving path runs vLLM with LoRA adapters over knowledge-distilled models. Ray and AIBrix handle inference distribution and autoscaling. Temporal drives long-running workflows. Retrieval uses a graph and vector index backed by Redis.

We drew a deliberate line between what is cloud-native and what is cloud-agnostic: managed PostgreSQL per cloud (Amazon RDS / Cloud SQL / Azure Database) for relational data, and in-cluster Kafka, Redis, and HashiCorp Vault for everything that needs to move with the platform. Istio with an Istio Gateway handles traffic, mTLS, and HTTPS identically on every cloud. Everything ships as versioned Helm releases via GitHub Actions, with Terraform managing infrastructure and remote state in S3 / GCS / Azure Blob per cloud.

Multi-cloud LLM platform architecture: one codebase deployed to Amazon EKS, Azure AKS and Google GKE

Because the platform is a set of Helm releases over a thin per-cloud config layer, standing up a new region-local cluster on whichever cloud a customer requires is a repeatable operation, not a project. The same source of truth serves every cloud and every region.

Portable platform layers: Kubernetes-native components stacked above a thin per-cloud configuration layer

Key outcomes

Cloud-agnostic by construction. The same stack runs on EKS, AKS, and GKE - provider detail is confined to a thin config layer.
Reusable platform, not just a portable app. In-cluster Kafka, Redis, Vault, and Istio mean the platform itself moves between clouds - a new provider is an onboarding step, not a re-architecture.
Region-local serving. Customers get clusters in their own region and on their required cloud.
Extensible on demand. New regions and capabilities slot in as additional Helm releases, no core changes needed.
Fast to production. Re-architected in place from the existing proof of concept - no ground-up rewrite.