High-Throughput Serving
Understand batching, concurrency, and hardware choices for serving many local model requests.
Throughput versus latency
Throughput measures completed work over time, while latency measures how long one request waits. A single interactive user usually values fast time to first token; a batch job may prefer maximum tokens per second. Serving design should state which metric matters and for what traffic shape. Optimizing one can harm the other, especially when batching makes individual requests wait for more work.
Continuous batching
A server can combine sequences that arrive at different times and schedule their next tokens together. Continuous batching keeps hardware busy as requests finish, unlike a simple batch that waits for every sequence. It requires careful scheduling, memory accounting, cancellation, and fairness. The benefit depends on prompt lengths, output lengths, concurrency, and whether the runtime has efficient attention kernels.
server:
max_concurrency: 8
max_queue_size: 32
max_context_tokens: 4096
reject_when_queue_full: trueKV-cache management
During generation, the model stores attention keys and values so it does not recompute the full history for every token. This cache can consume substantial memory, especially for long contexts and many simultaneous users. Paged or block-based allocation reduces fragmentation and makes scheduling more flexible. Set maximum context and output limits, then observe cache pressure under realistic traffic.
Hardware and parallelism
CPU serving is simple and may be economical for small models or low traffic. GPUs improve parallel matrix operations, while multiple devices can distribute weights or requests. Hardware support varies by runtime and precision. Benchmark the complete stack, including data transfer and startup time, rather than comparing theoretical compute alone. A modest device with a well-matched model can beat an expensive but poorly configured setup.
Queues and backpressure
An unbounded request queue turns overload into rising latency and eventual memory exhaustion. Use bounded queues, admission control, per-user limits, and clear rejection responses. Separate interactive traffic from offline jobs when possible. Backpressure should be visible in metrics and logs, and clients should receive retry guidance only when retrying is safe and likely to succeed.
Observability
Track request count, queue time, time to first token, generation time, tokens per second, input and output lengths, cancellations, errors, memory use, and model load time. Break metrics down by model and route. Avoid logging sensitive prompt content by default. These measurements reveal whether a problem comes from scheduling, model size, context growth, hardware saturation, or a slow client.
Capacity planning
Create a workload model with requests per minute, concurrency, token distributions, peak periods, and acceptable latency. Load-test with representative prompts and cancellations, then identify the point where quality or responsiveness degrades. Reserve headroom for failures and maintenance. Scale by reducing context, selecting a smaller model, adding devices, or distributing replicas only after measuring the actual bottleneck.
Worked example: serving a document classification queue
For a batch of 10,000 short documents, maximizing throughput matters more than streaming every token. A scheduler can group compatible requests, cap the queue, and write results with an idempotency key. For an interactive assistant on the same hardware, reserve capacity or use a separate pool so the batch cannot consume all memory. The service should expose queue depth, active sequences, rejection count, and completion latency so operators can see overload before users report it.
Code walkthrough
The YAML sets eight active requests, a queue of 32, a context ceiling, and explicit rejection when the queue is full. Those limits are safety controls, not universal best values. Load-test with the expected prompt and output length distribution, then change one limit at a time. Record p50, p95, and p99 latency, tokens per second, peak memory, and error rate. A queue that grows forever hides overload and produces worse user experience.
Trade-offs to measure
More concurrency can improve hardware utilization until memory pressure, scheduling overhead, or cache eviction dominates. Larger batches often improve throughput but increase wait time for the first request and can create fairness problems. Replicas improve isolation and availability but multiply model memory. Decide whether the product needs low tail latency, maximum batch throughput, or predictable cost before tuning a server around a single benchmark.
Practical exercise
Generate a load mix of short chat requests and long summarization requests. Run it at one, four, and eight concurrent clients with a bounded queue. Plot p95 latency against throughput and note when rejections begin. Add a graceful overload response and verify that cancelled requests free capacity. Use the measurements to choose limits, then repeat after changing the model or hardware.
Sources and further reading
These primary or specialist references informed the concepts in this guide. Product details can change, so verify current documentation before implementation.