API Integrations
Build reliable Python clients for model APIs and other AI services.
Use a provider adapter
Put authentication, base URLs, request construction, timeouts, streaming, retries, and response normalization behind a small adapter. The rest of the application should depend on an internal contract rather than a vendor-specific dictionary. This makes provider changes and local testing safer. Keep model selection in configuration and expose provider details in controlled metadata for debugging.
Timeouts and retries
Set a connection timeout and a total operation deadline so a stalled provider cannot consume a worker forever. Retry only transient failures and operations that are safe to repeat. Use bounded exponential backoff with jitter, respect rate-limit guidance, and stop when the deadline is reached. Never retry invalid input, authorization failures, or a tool action that may already have succeeded.
import time
for attempt in range(3):
try:
response = call_provider(timeout=10)
break
except TemporaryError:
if attempt == 2:
raise
time.sleep(2 ** attempt)Normalize errors
Convert provider-specific failures into categories such as invalid request, authentication, rate limit, timeout, unavailable, and malformed response. Return a useful user-facing message while keeping diagnostic details in protected logs. Include a correlation ID so support can find the trace. Do not expose raw provider payloads, stack traces, keys, or private prompts in an error response.
Streaming responses
Streaming reduces perceived wait time but requires a parser for partial events, terminal markers, disconnects, cancellation, and incomplete output. Accumulate text safely and validate the final result before treating it as complete. Close the response and release resources when a client disconnects. Test slow, empty, malformed, and interrupted streams rather than only a happy-path example.
Validate inputs and outputs
Enforce request size, file type, token, and concurrency limits before sending data. Parse responses into a schema and reject unexpected or incomplete values. Escape model text for its destination, especially when it becomes HTML, SQL, shell input, or a browser command. Validation belongs in application code because a prompt instruction cannot enforce a security or business rule.
Protect data and logs
Decide what data the provider receives, how long it retains it, and whether it may use it for training. Minimize personal and confidential content, redact logs before storage, and restrict trace access. Do not log complete request bodies by default. If a support workflow needs original content, make that access deliberate, audited, and temporary.
Testing the integration
Use mocked responses for deterministic unit tests and a small controlled integration test for authentication, schema, streaming, timeout, rate-limit, and provider-outage behavior. Record the model and API version used by live tests. Exercise retries and cancellation without causing duplicate side effects. A reliable adapter makes provider behavior boring for the rest of the application.
Worked example: a resilient summarization client
A summarization endpoint should enforce a request size, set a total deadline, retry only transient failures, and return a typed result. If the provider returns a rate limit, the application can queue or ask the user to retry later. If a timeout occurs after the provider may have completed work, do not blindly repeat a side-effecting tool action. Correlation IDs let support connect the UI error to a redacted server trace.
Code walkthrough
The retry loop catches a temporary error, stops after three attempts, and uses exponential delays. Improve it by adding jitter, a deadline, and an allow-list of retryable status categories. Define `call_provider` so it accepts a timeout and returns a response object that is validated before use. Test the loop with a fake provider that fails twice and then succeeds, plus one that never recovers.
Trade-offs to measure
Retries improve resilience to brief outages but increase latency, duplicate cost, and pressure on an already failing provider. Streaming improves perceived speed while making recovery and partial results harder. A fallback provider improves availability but may change quality, privacy, or output schema. Make the policy route-specific and expose whether a response came from a retry or fallback.
Practical exercise
Implement a fake API with success, 429, timeout, invalid JSON, and authorization responses. Add tests for retry count, backoff cap, deadline, normalized error category, and resource cleanup. Then add a bounded fallback model for a read-only request. Verify that an authorization error is never retried and that a cancelled request stops waiting without starting another attempt.
Sources and further reading
These primary or specialist references informed the concepts in this guide. Product details can change, so verify current documentation before implementation.