Agentic AI workloads on Kubernetes: when your user is an LLM

Most Kubernetes infrastructure - autoscaling policies, timeout defaults, rate limiting - is built on an assumption: a human is on the other end, and the request-response cycle is short. Agentic AI workloads break that assumption in ways that aren't obvious until you're already in production.

Sessions run long, and that's normal

An agent working through a multi-step task - researching, calling tools, revising its own plan - can hold a session open for minutes, not milliseconds. Infrastructure defaults tuned for a web request will time out or forcibly recycle connections mid-task, and the failure mode isn't a clean error - it's a partially completed action with no clear record of what happened.

Retries need to be idempotency-aware, not just present

A standard retry policy assumes the failed request had no side effects. An agent's tool call might have already sent an email, written a database row, or triggered a downstream workflow before the connection dropped. Retrying blindly means doing it twice. This pushes idempotency keys and action logs from a nice-to-have into a requirement for any tool-calling agent infrastructure.

# Every side-effecting tool call carries an idempotency key,
# checked before the action runs - not after it fails.

def execute_tool_call(action, idempotency_key, action_log):
    existing = action_log.get(idempotency_key)
    if existing is not None:
        # Already ran - return the recorded result, don't repeat the side effect
        return existing

    result = action.run()                      # send email, write row, call webhook...
    action_log.record(idempotency_key, result)  # durable, checked before every retry
    return result

Autoscaling on the wrong signal

Request count and CPU utilization, the usual autoscaling signals, correlate poorly with agent workload cost. A single agent session might make dozens of downstream LLM calls while barely touching CPU on the orchestrating service. Cost and load are better tracked through token throughput and concurrent active sessions than through traditional pod-level metrics.

Observability needs a new unit of work

A trace for a normal request ends when the response is returned. An agent's unit of work is the whole task - which might span multiple model calls, tool invocations, and even human-in-the-loop pauses. Tracing needs to model the task as the span, with individual calls as children, or debugging a failed agent run turns into reconstructing a timeline from scattered logs.

What we're recommending right now

Treat agent orchestration as a distinct workload class in your platform, not a variant of a normal API service. Give it its own node group with autoscaling tuned to concurrency rather than CPU, its own timeout defaults, and tracing that treats the task - not the request - as the unit that gets a trace ID. The infrastructure patterns for this are still settling industry-wide, but the gap between "treat it like a normal API" and "treat it like what it actually is" shows up fast once real usage starts.