Deploying to Production
Prepare LLM applications for reliable releases, monitoring, and incident response.
Separate the moving parts
Keep provider clients, prompts, retrieval, business rules, validation, tools, and presentation logic behind clear interfaces. This allows a model or provider to change without rewriting the product. Define which components are stateless and which hold data. A modular boundary also makes it easier to test failures locally instead of discovering them only after deployment.
Configuration and secrets
Use environment-specific configuration for model IDs, endpoints, limits, feature flags, and retention settings. Store credentials in a secret manager or protected deployment configuration, never in source or prompts. Validate required settings at startup without logging secret values. Make the active configuration visible to operators through safe metadata so incidents do not depend on guessing which model is running.
set -eu
curl --fail --silent https://example.com/health
curl --fail --silent -X POST https://example.com/smoke-test \
-H 'Content-Type: application/json' \
-d '{"prompt":"Return the word ready","dry_run":true}'Release safety
Use automated tests, evaluation gates, staging, canary traffic, feature flags, and a rollback plan. Test provider errors, streaming, timeouts, retries, tool authorization, empty retrieval, and malformed model output. Database or index migrations need compatibility checks with both old and new application versions. A deployment is not safe merely because the process exits successfully.
Reliability patterns
Set connection and total-operation deadlines, bound retries, use circuit breakers, and return honest degraded responses when dependencies fail. Queue long jobs and make them idempotent so a retry does not duplicate an email, payment, or record. Separate interactive traffic from batch work. Define health checks that distinguish process availability from useful model and dependency readiness.
Monitoring and traces
Monitor request volume, latency stages, errors, token usage, cost, model versions, validation failures, safety events, and user feedback. Use correlation IDs across retrieval, tools, and provider calls. Redact prompts and outputs by default, with controlled access for investigations. Dashboards should support a question such as “what changed for this failing request?” rather than only display totals.
Incident response
Prepare runbooks for provider outage, leaked data, unsafe output, runaway spend, retrieval corruption, and model regression. Assign owners, escalation paths, communication templates, and evidence-retention rules. During an incident, preserve enough trace information to understand impact while limiting further exposure. Practice the fallback and rollback path before it is needed under pressure.
Post-release learning
Review quality, reliability, cost, and user feedback after deployment, then compare them with the release hypothesis. Record what improved, what regressed, and which follow-up test or control is needed. Update the evaluation set with representative failures. Production operation is a feedback loop; the release is only successful when the system remains understandable and maintainable over time.
Worked example: a safe canary deployment
Deploy an AI endpoint behind a stable route and send a small percentage of traffic to the new model or prompt. Compare error rate, latency, cost, schema validity, groundedness, and user feedback against the old version. Keep the old bundle warm enough for rollback and make the release identifier visible in logs. A successful health check proves the process is alive, not that the model is producing acceptable answers.
Code walkthrough
The shell smoke test fails on a non-successful health response and sends a dry-run request to the endpoint. Add authentication through the CI secret store, validate the response schema, and assert a release ID rather than matching a fragile full answer. Keep smoke-test prompts synthetic and non-sensitive. Run the checks after deployment and record the revision, region, and timestamp for incident correlation.
Trade-offs to measure
Managed model APIs shorten operations but create provider, quota, and data-processing dependencies. Self-hosting offers control and predictable network boundaries but requires capacity, patching, observability, and on-call expertise. Blue-green deployment improves rollback speed at the cost of duplicate capacity. A canary reduces blast radius but requires enough representative traffic to reveal regressions.
Practical exercise
Create a deployment checklist with build provenance, configuration validation, secret checks, health test, synthetic generation test, canary, dashboards, rollback, and post-release review. Simulate an invalid schema and a latency breach. Confirm the pipeline blocks promotion or restores the previous revision, and verify that the smoke test does not log the synthetic prompt or credentials.
Sources and further reading
These primary or specialist references informed the concepts in this guide. Product details can change, so verify current documentation before implementation.