API Cost and Latency Management
Control usage while keeping AI features responsive and useful.
Understand the cost equation
AI cost is driven by input tokens, output tokens, model price, retries, tool calls, concurrency, and sometimes image or audio size. Aggregate spend can hide an expensive user journey with multiple failed attempts. Measure cost per successful task and per active user, not only cost per API request. Include preprocessing, storage, and observability costs when comparing architectures.
Measure latency correctly
Users experience connection time, queue time, time to first token, generation time, tool delays, and final rendering. Capture these stages separately so an optimization targets the real bottleneck. Streaming can improve perceived responsiveness without reducing total work. Set deadlines for the complete workflow and communicate progress when a long operation cannot reasonably be made instant.
input_tokens, output_tokens = 1200, 300
input_rate, output_rate = 0.000003, 0.000015
cost = input_tokens * input_rate + output_tokens * output_rate
print(f"estimated_cost_usd={cost:.5f}")Use the right model
Route classification, extraction, rewriting, and simple summarization to a smaller model when evaluation shows it is sufficient. Reserve a larger model for tasks that benefit from its capability. Routing should be explicit and monitored because a cheap model that causes retries or human corrections may cost more overall. Keep a fallback path for provider or model unavailability.
Reduce unnecessary context
Long instructions, duplicated history, irrelevant retrieval chunks, and verbose tool results consume tokens and slow generation. Summarize older turns, retrieve selectively, compress structured data, and set clear output limits. Do not remove evidence merely to save money; measure whether context reduction harms groundedness or completeness. Token budgets should reflect the task contract rather than an arbitrary global maximum.
Caching and batching
Cache stable embeddings, repeated retrieval results, and deterministic transformations when their inputs and permissions match. Never share a response across users without including all relevant authorization context in the cache key. Batch offline jobs and independent requests where the provider supports it. Streaming and caching need careful invalidation, because a fast stale answer is still a product failure.
Guardrails for spend
Apply per-user and per-tenant quotas, request size limits, concurrency caps, and maximum workflow budgets. Use exponential backoff only for transient failures and avoid retrying invalid requests. Add alerts for spend rate, token growth, retry storms, and unexpected model routing. Give operators a kill switch or degraded mode before a runaway loop becomes a large invoice.
Optimize with evidence
Change one cost or latency variable at a time and replay a representative evaluation set. Compare quality, time to first token, completion time, peak concurrency, error rate, and successful-task cost. Document the chosen trade-off and its operating limits. Optimization is complete when the product meets its user promise sustainably, not when one synthetic benchmark becomes faster.
Worked example: setting a request budget
Suppose an assistant receives 1,200 input tokens and returns 300 output tokens. The displayed estimate gives a first approximation, but a product budget also includes retries, embeddings, image processing, cache misses, and failed requests. Set a maximum token budget per route, measure actual usage by model, and alert when daily spend or p95 latency moves beyond the expected range. Cost controls should degrade gracefully instead of silently dropping important work.
Code walkthrough
The Python calculation multiplies token counts by separate input and output rates. Production code should read rates from versioned configuration, use decimal arithmetic where billing precision matters, and attach the provider and model to the estimate. Record estimated and billed usage separately. Add a guard before the request, then reconcile against provider invoices or usage exports so the estimate is not mistaken for accounting truth.
Trade-offs to measure
Shorter prompts save tokens but may reduce grounding or require more retries. Smaller models reduce cost and latency but can increase human correction effort. Caching lowers spend and response time while raising freshness and privacy questions. Batching improves throughput but may not fit interactive requests. Optimize total successful task cost and user-perceived latency, not one API metric in isolation.
Practical exercise
Instrument a small workflow with input tokens, output tokens, retries, cache hits, time to first token, total latency, and outcome quality. Run it with a short and a verbose prompt. Calculate cost per successful task, not merely cost per call. Add a timeout, a token ceiling, and a fallback model, then verify that a provider slowdown cannot create an unbounded retry bill.
Sources and further reading
These primary or specialist references informed the concepts in this guide. Product details can change, so verify current documentation before implementation.