GPU utilization is probably worse than your dashboard says
A fleet dashboard showing 80% GPU utilization looks healthy. I've learned to be suspicious of exactly that number. Here's what it's actually hiding, and what to measure instead.
Topic
AI
3 articles tagged AI
Agentic AI workloads on Kubernetes: when your user is an LLM
Agent workloads don't fit the request-response assumptions most Kubernetes infrastructure is built around. Here's what actually changes when the client calling your API is another model.
Choosing an LLM inference stack: what actually matters at scale
vLLM, TGI, and a half-dozen other inference servers all claim to be fast. Here's what actually differentiates them once you're serving real traffic, not a benchmark.