Topic 8: LLMOps interview questions

LLMOps is the engineering discipline for operating applications built with large language models. It covers prompt and model versioning, data and evaluation pipelines, deployment, observability, cost control, security, and incident response. Unlike ordinary software operations, outputs are probabilistic and can change when a provider updates a model. A strong LLMOps process makes behavior measurable, releases reversible, and quality, latency, safety, and spend visible together. Interview signal: Define LLMOps as the practices that make model-powered features reproducible, observable, safe, economical, and recoverable.

I would keep prompts, model identifiers, parameters, tools, retrieval settings, and output schemas in version-controlled configuration. Each production request should record a non-sensitive configuration version and model response metadata. Changes should pass automated evaluation and, for important flows, a canary or shadow rollout. Feature flags make rollback fast. Prompt text should be reviewed like code because a small wording change can alter safety, cost, and business behavior. Interview signal: Version prompt text, model ID, parameters, tools, retrieval settings, schemas, and evaluation fixtures as one release unit.

It should contain representative test cases, expected properties, automated graders, human review for ambiguous samples, and regression reporting. Tests should cover normal requests, edge cases, adversarial prompts, refusals, tool calls, retrieval grounding, and structured output. I would segment results by language, customer type, task, and difficulty. The pipeline should compare candidate and baseline versions, fail on agreed thresholds, and preserve failed examples for future regression tests. Interview signal: Build an evaluation pipeline with golden cases, adversarial cases, graders, human review, thresholds, and regression diffs.

Hallucination needs a task-specific definition. For grounded question answering, I would measure whether claims are entailed by retrieved or supplied evidence, using an automated judge plus sampled human verification. For extraction, compare fields with labeled truth. I would track unsupported-claim rate, citation correctness, abstention quality, and severity, not only one aggregate score. Evaluation data must include questions whose answer is absent so confident guessing is detected. Interview signal: Use claim-level evidence checks and calibrated human review; fluency or a single judge score is not hallucination measurement.

I would monitor request volume, success and timeout rates, latency by stage, token usage, cost, model and prompt versions, tool errors, retrieval quality, schema failures, and safety outcomes. Traces should connect a user request to retrieval, model calls, tool calls, and the final response. Raw prompts may contain sensitive data, so logging needs redaction, access controls, sampling, and configurable retention rather than indiscriminate capture. Interview signal: Trace request IDs, model and prompt versions, tokens, latency, tool calls, retrieval IDs, errors, spend, and quality feedback.

First measure cost by workflow, tenant, model, and token type. Then reduce unnecessary context, cap output length, summarize long histories, cache stable results, batch offline jobs, and route simple tasks to smaller models. Budgets and rate limits prevent runaway usage. I would monitor quality after every optimization because a cheaper answer that causes retries or human correction may have a higher total cost. Cost alerts should be actionable and tied to owners. Interview signal: Set budgets per tenant and feature, route by task complexity, cache safe work, and measure cost per successful business outcome.

Prompt caching reuses processing for a repeated prefix or repeated request context, depending on the provider or infrastructure. It is useful for large stable system instructions, policies, or reference material that appears across many calls. Benefits include lower latency and input cost, but cache keys must reflect all content that affects behavior. I would define invalidation rules, avoid caching user-private data across tenants, and verify that cached content cannot become stale or unauthorized. Interview signal: Use prompt caching only for stable prefixes or repeated context and verify invalidation, privacy scope, and provider semantics.

I would first run offline regression and safety evaluations, then deploy behind a feature flag. Shadow traffic can compare outputs without exposing them to users, followed by a small canary cohort with quality, latency, cost, and error monitoring. The release needs a known-good configuration, rollback procedure, and owner on call. Provider model changes should be treated as releases too, even when application code has not changed. Interview signal: Roll out with offline gates, shadow or canary traffic, health thresholds, user feedback, and an immediate rollback path.

The application should use bounded timeouts, retries with exponential backoff and jitter, and idempotency for operations that may be repeated. A fallback model or deterministic path can handle supported low-risk tasks. If no safe fallback exists, return a clear temporary-unavailability message and preserve the user's work. Circuit breakers prevent a failing provider from consuming all capacity. Alerts should distinguish provider failures from application, network, and quota errors. Interview signal: Define provider fallbacks, timeout budgets, degraded responses, queue behavior, and data-routing policies before an outage.

An LLM gateway is a shared service that manages access to one or more model providers. It can centralize authentication, routing, retries, rate limits, quotas, logging, redaction, caching, and policy enforcement. This reduces duplicated integration code and makes provider changes easier. The gateway must avoid becoming an opaque bottleneck: teams need clear ownership, tenant isolation, trace propagation, timeout budgets, and a documented path for provider-specific capabilities. Interview signal: Use a gateway for routing, policy, logging, budgets, retries, and provider abstraction, while avoiding a hidden single point of failure.

I would enforce limits at the tenant and workflow levels using token-aware or request-aware budgets. Queues smooth bursts, while concurrency limits protect downstream providers. Responses to quota errors should be classified separately from transient failures, because retries can worsen the problem. Usage dashboards and alerts should show remaining quota, projected exhaustion, and the responsible tenant. For critical workflows, reserve capacity or define priority rules with explicit fairness and degradation behavior. Interview signal: Apply limits across user, tenant, model, tokens, concurrency, tool calls, and spend, with alerts for unusual consumption.

Tools should have narrow schemas, least-privilege credentials, server-side authorization, and strict validation independent of the model. The model may propose an action, but application code decides whether it is allowed and whether confirmation is required. I would separate read and write tools, constrain destinations, sanitize outputs, and log arguments without leaking secrets. High-impact actions need idempotency, replay protection, and an auditable link between user intent and execution. Interview signal: Validate tool identity, schema, authorization, destination, side effect, idempotency, and approval outside model output.

Retrieved documents are data, not instructions. I would separate system policy from retrieved content, label untrusted sources, limit tool permissions, and require the application to validate actions. Documents should be scanned for malicious patterns, but filtering is only one layer. The model should cite evidence and refuse instructions that conflict with policy. Red-team tests should include poisoned documents, indirect instructions, cross-tenant content, and attempts to exfiltrate system prompts or secrets. Interview signal: Use retrieval boundaries, content labeling, tool scopes, injection tests, and a refusal or escalation path for hostile context.

Offline evaluation runs a fixed dataset before release, making candidate comparisons repeatable and inexpensive. Online evaluation measures behavior on real traffic through monitoring, user feedback, sampled review, or controlled experiments. Offline tests catch known regressions, while online signals reveal distribution shifts, new user intents, and operational issues. I would use both, with privacy-safe sampling and guardrails so an experiment cannot expose users to unacceptable safety or business risk. Interview signal: Contrast fixed offline regressions with controlled online metrics and explain why both are required for release confidence.

Human review should target uncertainty and impact rather than every response. I would define clear labeling guidelines, show the source evidence and model version, and use double review for high-severity disagreements. Reviewers need escalation paths and periodic calibration. Sampling should include both suspected failures and random production outputs to measure detector blind spots. Their labels should feed evaluation sets, prompt improvements, routing thresholds, and incident analysis—not just a dashboard. Interview signal: Route uncertain, sensitive, or high-impact outputs to reviewers with evidence, rubric, deadline, and audit trail.

I would use provider-supported structured output or tool schemas where available, then validate every response with a strict schema. A bounded repair attempt may ask the model to correct only formatting, but it must not bypass business validation. If repair fails, route to a fallback or return a recoverable error. Metrics should distinguish invalid syntax, missing fields, invalid values, and downstream rejection so the actual failure mode can be fixed. Interview signal: Handle invalid JSON with bounded repair or safe failure, then measure schema-validity regressions separately from quality.

The process starts with detection, severity classification, and containment. I would freeze risky releases, disable a feature flag, reduce traffic, or switch models while preserving evidence. Then identify whether the cause is a prompt, model, data, retrieval, tool, provider, or infrastructure change. After recovery, run targeted regression tests, communicate impact, document the timeline, and add a durable prevention such as a test case, monitor, permission check, or rollback automation. Interview signal: Prepare runbooks for provider outage, leaked data, runaway spend, poisoned content, unsafe action, and rollback.

I would first establish whether the problem is missing knowledge, poor instructions, formatting, or a repeated behavior pattern. Retrieval is appropriate for changing factual content, while prompting and schemas often solve instruction and output issues. Fine-tuning may help with stable style, classification, or tool-selection behavior when sufficient high-quality examples exist. I would compare quality, data preparation, latency, cost, governance, and rollback complexity using the same evaluation set. Interview signal: Use a decision matrix: retrieval for changing facts, prompting for instruction changes, fine-tuning for stable behavior.

Governance should define what data may enter prompts, how it is classified, where it is stored, who can access it, and how long it is retained. Training and evaluation datasets need provenance, consent or licensing checks, redaction, deduplication, and access controls. Production logs should have separate retention policies from application records. I would document provider data-use terms, deletion procedures, audit ownership, and restrictions for regulated or confidential information. Interview signal: Define data ownership, purpose, retention, deletion, access, redaction, residency, provider terms, and audit responsibilities.

Production readiness means the system is useful, measurable, safe, and recoverable. It needs representative evaluations, versioned prompts and models, schema validation, timeouts, bounded retries, rate limits, observability, privacy controls, cost budgets, and an incident runbook. There should be a tested rollback or fallback path and clear ownership. I would also verify user-facing failure messages, data deletion behavior, accessibility, and performance under realistic concurrency before launch. Interview signal: Call a system production-ready only when it is useful, observable, secure, cost-bounded, tested, and safely recoverable.