LLM Inference Optimization: A Developer's Guide to Faster, Cheaper Model Serving

Another one in the series where I break down the parts of building with LLMs that took me a while to actually understand. This time it is LLM inference optimization, the difference between a demo that feels magical and a production feature that feels broken.

I shipped a small feature that called a model on every request. It worked great with three testers. Then a few hundred users showed up and everything crawled. Two second responses now took fifteen, and the GPU bill doubled. Nothing about my prompt had changed. What changed was that I had never thought about how the model actually runs when you ask it for tokens, and that gap is exactly what inference optimization fills.

What I was missing about inference

Training gets all the attention, but most of us never train a model. We serve one. Serving is inference, and it has its own performance rules that have nothing to do with how the model was built.

The thing that finally made it click: generating text is not one operation, it is two very different phases. Prefill reads your whole prompt at once and builds the model's internal state. It is compute heavy and the GPU loves it, because it can process every input token in parallel. Decode then produces the answer one token at a time. Each new token depends on the previous one, so you cannot parallelize it. You run the entire model forward just to get a single token, then do it again, and again.

That second phase is where the pain lives. Decode is memory bound, not compute bound. The GPU spends most of its time waiting to move weights and cached values in and out of memory rather than doing math. Once I understood that, every technique stopped feeling like a random trick and started answering a specific question: how do I move less memory per token, or produce more tokens per pass.

Why LLM inference feels slow and expensive

Three things quietly wreck your latency and your budget.

The first is the KV cache. To avoid recomputing attention over the whole sequence at every step, the model caches the key and value tensors for every token it has seen. That cache grows with sequence length and with concurrent requests, and it eats GPU memory fast. When it does not fit, you drop batch size or start swapping, and both tank throughput.

The second is poor batching. A single request rarely uses a GPU fully, so processing one at a time leaves the hardware mostly idle. But naive batching forces every request to wait for the slowest one to finish, so long generations block short ones.

The third is running a bigger model than the task needs, in higher precision than the task needs. Many requests do not require your largest model at full 16-bit weights, and paying for that on every call adds up. Improving LLM inference speed is mostly about attacking these three at once.

The core inference optimization techniques

KV cache management and PagedAttention

The KV cache is usually the first bottleneck you hit. The classic fix is PagedAttention, introduced by the vLLM project. Instead of reserving one big contiguous block of memory per request, it splits the cache into small pages, the way an operating system pages virtual memory. That kills internal fragmentation, lets you pack far more concurrent requests onto the same GPU, and even lets requests share pages when they have a common prefix. You do not implement this yourself, you get it by choosing a serving engine that has it.

Continuous batching

This gave me the biggest single win. Traditional static batching groups requests, runs them together, and starts no new work until the whole batch is done. Continuous batching, sometimes called in-flight batching, works at the token level. The moment one request finishes generating, its slot is freed and a waiting request drops in. The GPU stays busy instead of idling while it waits for the longest generation. For a workload with mixed prompt and response lengths, this can multiply throughput several times over without touching the model.

Quantization

Quantization stores the model weights in fewer bits. Instead of 16-bit floats you use 8-bit or 4-bit representations. The model gets smaller, so there is less memory to move per token, which directly speeds up the memory-bound decode phase and lets you fit larger models or bigger batches on the same card.

The tradeoff is quality. Aggressive 4-bit quantization can degrade output on hard tasks, so you test on your own prompts rather than trusting a benchmark. In practice 8-bit is close to free for most workloads, and 4-bit is worth trying when you are memory constrained. LLM quantization is the highest-leverage change when your model barely fits on the GPU.

Speculative decoding

Speculative decoding attacks the one-token-at-a-time problem head on. A small, fast draft model proposes several tokens ahead. The big model checks all of them in a single forward pass and accepts the ones it agrees with. When the draft guesses well, you get multiple tokens for the cost of one big-model pass. The output is identical to what the big model would have produced alone, so you buy speed without paying in quality. It shines on predictable text like code and structured output.

Use a smaller or cheaper path when you can

Not every request needs your flagship model. Routing easy requests to a smaller model, caching responses, and reusing computed prefixes all cut work before inference even starts. This overlaps with cost work more than raw speed, and I wrote about the money side in the AI cost optimization guide and in the piece on prompt caching. Faster inference is usually cheaper inference.

Picking a serving stack for efficient LLM serving

You almost never build these optimizations yourself. Efficient LLM serving comes from picking an inference server that ships them and configuring it well. The common choices are vLLM, which popularized PagedAttention and continuous batching, Hugging Face Text Generation Inference, NVIDIA TensorRT-LLM for maximum performance on NVIDIA hardware, and SGLang for heavy structured generation and shared prefixes.

They differ in setup cost and peak performance, but any of them beats a hand-rolled loop that calls the model once per request. For the deeper systems view of serving infrastructure, GPU allocation, and graceful degradation under load, I went into that in the model serving deep dive.

How to measure inference, so you know it worked

You cannot optimize what you do not measure, and LLM serving has more than one number that matters. Time to first token (TTFT) is how long the user waits before anything appears, dominated by prefill and queueing. Inter-token latency is the gap between streamed tokens after the first, and it drives how fast the response feels once it starts. Throughput, in tokens per second across all concurrent requests, is what your cost per request really comes from.

The trap is optimizing one and ignoring the others. Bigger batches lift throughput but can raise TTFT because requests wait longer to start. Decide which one your product cares about before you tune, then watch all three so a win in one place does not quietly become a loss somewhere else.

# Measuring TTFT and inter-token latency with a streaming client
import time

def measure_stream(client, prompt):
    start = time.perf_counter()
    first_token_at = None
    tokens = 0
    for chunk in client.stream(prompt):
        if first_token_at is None:
            first_token_at = time.perf_counter()
        tokens += 1
    end = time.perf_counter()

    ttft = first_token_at - start          # user's perceived wait
    gen_time = end - first_token_at         # time spent streaming
    tps = tokens / gen_time if gen_time else 0
    return {"ttft_s": round(ttft, 3), "tokens_per_s": round(tps, 1)}

Run that against real prompts, not toy ones. A ten-token prompt hides the prefill cost that a two-thousand-token prompt exposes.

A practical optimization checklist

When a serving setup is slow, this is the order I work through now, and each step answers one of the three bottlenecks from earlier. First, move to an engine with continuous batching and PagedAttention if you are not on one, the single biggest structural win and it needs no model changes. Second, quantize the weights to 8-bit and measure quality on your own prompts, then try 4-bit if you are memory bound. Third, turn on speculative decoding if your outputs are predictable, like code or JSON. Fourth, right-size the model and route easy requests to something smaller. Fifth, measure TTFT, inter-token latency, and throughput before and after every change so you know which lever did the work.

Most teams get the majority of the improvement from the first two steps. The exotic techniques matter, but only after the batching and memory basics are in place.

Where this fits

Inference optimization sits right next to cost work, and once you have serving under control the natural next questions are about spend. The LLM cost optimization guide and the model routing walkthrough pick up where this leaves off, and more engineering breakdowns live on the Levelop blog. I have been building and breaking these serving setups while working through the system design track on Levelop, and inference optimization looks academic until the day your traffic triples.

References

1. vLLM, Efficient Memory Management for LLM Serving with PagedAttention, docs.vllm.ai.
2. Hugging Face, Text Generation Inference, huggingface.co/docs/text-generation-inference.
3. NVIDIA, TensorRT-LLM, github.com/NVIDIA/TensorRT-LLM.
4. SGLang, Structured Generation Language for LLMs, github.com/sgl-project/sglang.