Choosing an LLM inference stack: what actually matters at scale

Every LLM inference server publishes benchmark numbers that look great in isolation. Tokens per second, time to first token, throughput under synthetic load. None of that tells you how it behaves under your actual traffic pattern, which is usually the only comparison that matters.

Continuous batching is table stakes, not a differentiator

vLLM popularized continuous batching - packing new requests into GPU batches as older ones finish, instead of waiting for a full batch to complete. It's genuinely important for throughput. It's also now standard across most serious inference servers, including TGI. If a vendor is still selling continuous batching as their headline feature, that's a signal to look at what else they're not telling you.

Where they actually diverge

Memory management under mixed request lengths. Real traffic isn't uniform - short chat completions next to long document-analysis requests on the same endpoint. Servers differ meaningfully in how gracefully they handle that mix without fragmenting GPU memory or starving short requests behind long ones.

LoRA adapter support. If you're serving multiple fine-tuned variants of a base model, the ability to swap adapters per-request without a full model reload changes your GPU economics substantially. Not every server does this well, and some don't do it at all.

Quantization support and quality tradeoffs. Nearly everything supports some quantization now. Fewer handle the harder cases - quantized KV cache, or maintaining quality at aggressive compression - without a fragile setup that breaks on the next model release.

Operational maturity. Prometheus metrics out of the box, sane default logging, a health check that actually reflects readiness rather than just process liveness. This is the boring category and it's the one that determines how many 2am pages you get.

The question to ask instead of "which is fastest"

Benchmark your own request distribution, not a published one. Run your actual mix of short and long requests, your actual concurrency, your actual model and adapter count, and measure p99 latency and cost per thousand tokens under that specific load. The server that wins a published benchmark and the server that wins on your traffic are frequently not the same one.

# Benchmark your own traffic shape, not a published one
import time
import statistics

REQUEST_MIX = [
    {"kind": "chat", "tokens_in": 80, "tokens_out": 120, "weight": 0.7},
    {"kind": "doc_analysis", "tokens_in": 4000, "tokens_out": 400, "weight": 0.3},
]

def run_load(server_url, concurrency, duration_s):
    latencies = []
    total_tokens = 0
    start = time.monotonic()
    while time.monotonic() - start < duration_s:
        req = sample_request(REQUEST_MIX)          # sample per real traffic weights
        t0 = time.monotonic()
        resp = send_request(server_url, req)        # your actual client call
        latencies.append(time.monotonic() - t0)
        total_tokens += resp.tokens_generated

    p99 = statistics.quantiles(latencies, n=100)[98]
    cost_per_1k = compute_cost(total_tokens, duration_s)  # your GPU-hour rate
    return {"p99_latency_s": p99, "cost_per_1k_tokens": cost_per_1k}

We default to vLLM for most inference workloads because its batching and memory management have been the most consistent under mixed real-world load in what we've run. That's a starting point, not a rule - the right answer depends on what you're actually serving.