Cloud-agnostic by design: configurable GPU and a portable platform layer

The problem

A stealth-mode US AI company made a deliberate decision early: the platform should not be shaped by whichever cloud it landed on first. Cloud-managed services accumulate quickly, integrations harden, and moving later becomes a project rather than a config change.

The harder challenge was not the cluster layer - it was the internals. GPU handling was the sharpest pressure point. Each cloud has its own GPU node integrations, device plugins, and scheduling primitives. Using them is convenient until you need the same workload on a different cloud and find the integration is entirely provider-specific. They needed GPU handling as a configuration flag, not a fork. The same principle had to apply to secrets, messaging, and storage.

What we built

The platform follows one rule: any component that needs to move must not depend on the cloud it runs on. Cloud-native services are used where they earn their place. Everything else runs in-cluster.

In-cluster stack. Every cluster carries Istio with an Istio Gateway - consistent mTLS between services and HTTPS at the edge, identical on EKS, AKS, and GKE. HashiCorp Vault handles secrets in-cluster so no application is tied to AWS Secrets Manager, Azure Key Vault, or Google Secret Manager. Apache Kafka runs in-cluster as the event backbone - consistent producer and consumer behaviour across every cloud, no provider-specific IAM or endpoint quirks.

Configurable GPU handling. The GPU layer is a feature flag in Helm values. Two modes, one codebase:

• Cloud-managed mode - use the cloud provider's GPU device plugin (EKS GPU node groups, AKS GPU pools, GKE Accelerator nodes). Lower operational overhead, tighter cloud billing integration.
• NVIDIA-native mode - deploy the NVIDIA GPU Operator and NVIDIA device plugin directly. Same driver stack on any cluster, any cloud, or bare-metal.

Switching mode is a single values change. Application code, workload scheduling rules, and observability config are untouched. Both modes expose GPUs via the same nvidia.com/gpu resource - workloads are unaware of which is active.

Named node groups and workload placement. The cluster runs four named node groups out of the box, each autoscalable and purpose-scoped:

• cpu-general - standard CPU, on-demand. Coordination, API serving, platform services. Scales on pod load.
• gpu-inference - on-demand A10G-class GPU for low-latency inference. Scales on latency and queue depth. Minimum one node to stay warm.
• gpu-training - spot A100 / H100 GPU for training and fine-tuning runs. Scales to zero when idle - no training jobs running means no GPU cost. Scales on job queue depth.
• spot-batch - preemptible CPU for interruption-tolerant batch work, embedding pipelines, and cron jobs. Also scales to zero.

Node groups are fully configurable via a nested input variable in terraform.tfvars. Each entry defines the group name, instance type, disk size, CPU/GPU type, min/max counts, and spot eligibility - so teams extend, replace, or add node groups without touching module code. A new GPU generation or a specialised inference instance type is a tfvars change, not a re-architecture.

# terraform.tfvars - extend or replace node groups as needed
node_groups = [
  {
    name          = "cpu-general"
    instance_type = "m6i.2xlarge"
    disk_size_gb  = 100
    min_size      = 1
    max_size      = 20
    spot          = false
    gpu           = false
  },
  {
    name          = "gpu-inference"
    instance_type = "g5.2xlarge"    # A10G
    disk_size_gb  = 200
    min_size      = 1
    max_size      = 10
    spot          = false
    gpu           = true
  },
  {
    name          = "gpu-training"
    instance_type = "p4d.24xlarge"  # A100 - swap to p5.48xlarge for H100
    disk_size_gb  = 500
    min_size      = 0               # scales to zero when idle
    max_size      = 4
    spot          = true
    gpu           = true
  },
  {
    name          = "spot-batch"
    instance_type = "m6i.xlarge"
    disk_size_gb  = 100
    min_size      = 0
    max_size      = 30
    spot          = true
    gpu           = false
  }
  # add further node groups here - no module changes needed
]

Every workload declares its target node group in Helm values and Kubernetes YAML using nodeSelector and tolerations. Placement is explicit, reviewable in a pull request, and version-controlled. No workload lands on a node by accident, and spot interruptions affect only workloads that tolerate them.

Storage abstraction via CSI. Block storage (EBS / Azure Disk / Persistent Disk) is provisioned via Kubernetes CSI drivers with a common StorageClass name across all clusters. Object storage (S3 / GCS / Azure Blob) is accessed via a consistent interface. Applications reference storage class names, not cloud-specific APIs. The cloud difference lives only in the CSI layer, configured once at provisioning time via Terraform.

IaC. Infrastructure is Terraform with feature-flagged modules. GPU node groups, managed databases, VPC config, and IAM roles are all optional - a customer with existing infrastructure disables the relevant module and references their own. Remote state lives in S3 on AWS, GCS on Google Cloud, and Azure Blob on Azure.

Key outcomes

• GPU mode is a Helm flag, not a fork. Switching between cloud-managed and NVIDIA-native requires no application change and no new release pipeline.
• GPU training costs zero when idle. The gpu-training node group scales to zero when no jobs are queued - A100 / H100 nodes are provisioned on demand and released when done.
• Node groups are fully extensible via tfvars. Adding a new GPU generation, a specialised instance type, or an entirely new node group is a terraform.tfvars change - no module code touched, no re-architecture needed.
• In-cluster stack moves with the platform. Istio, Vault, and Kafka are never swapped for cloud equivalents. A new cluster on any cloud gets the same security posture and event backbone from the first install.
• Workload placement is explicit and auditable. Named node groups with declared nodeSelector and tolerations mean every scheduling decision is visible in version control.
• Storage is portable. CSI drivers and common StorageClass names mean workloads reference no cloud-specific storage API.
• Every cloud-native choice is reversible. Feature-flagged Terraform modules mean a customer's existing VPC, database, or IAM setup can be reused without conflict.