Topic 7: Multimodal AI interview questions

Multimodal AI works with more than one type of data, such as text, images, audio, video, or sensor readings. A multimodal system may align inputs into a shared representation, reason across them, and produce an output in one or more modalities. For example, it can inspect a product photo and answer a text question about defects. The main engineering challenge is preserving useful information and relationships between modalities. Interview signal: Use an invoice or image-search example to show why combining pixels, text, audio, or metadata changes the system boundary.

A common design uses a separate encoder for each modality, followed by projection layers that map their outputs into a shared embedding space. A language model or fusion transformer then attends to the projected tokens. Early fusion combines signals near the input, while late fusion combines modality-specific decisions. The right choice depends on latency, data volume, alignment quality, and whether fine-grained cross-modal reasoning is required. Interview signal: Explain early fusion, late fusion, and shared representation choices with their latency, alignment, and debugging trade-offs.

A vision-language model connects visual understanding with language generation. An image encoder converts pixels into visual features, and a connector maps those features into tokens or embeddings that a language model can use. Training usually includes image-text alignment and instruction tuning. Such models can caption images, answer questions, extract fields, compare images with text, and reason about documents, but their answers still require validation for high-impact use cases. Interview signal: Contrast visual question answering with OCR-plus-text retrieval and state when each approach is easier to validate.

Multimodal alignment means representing corresponding concepts from different modalities so the model recognizes their relationship. An image of a dog and the phrase describing that dog should be close in a shared embedding space. Contrastive learning is one common method: matching pairs are pulled together and non-matching pairs are separated. Good alignment supports retrieval and grounding, but alignment alone does not guarantee accurate multi-step reasoning or factual answers. Interview signal: Define alignment as linking representations and meanings across modalities, then mention contrastive or paired-data evaluation.

I would first define supported image types, question scope, latency, and accuracy requirements. The request path would validate size and format, remove unsafe metadata when appropriate, and send the image with a structured prompt to a vision-language model. I would constrain the response schema, store the source and model version for auditability, and add confidence or escalation rules. Evaluation should include low quality images, missing context, adversarial text, and unanswerable questions. Interview signal: Describe upload validation, resizing, model call, grounding, structured extraction, confidence checks, and human fallback.

There are two main approaches. A document pipeline can perform OCR, detect layout, and pass structured text and coordinates to a language model. Alternatively, a vision-language model can inspect rendered pages directly. OCR is often cheaper and easier to cite, while direct visual reasoning can preserve tables, handwriting, and spatial relationships. A robust system combines both where useful, checks extracted fields against the image, and retains page-level evidence. Interview signal: Mention layout, tables, reading order, OCR errors, image resolution, and page-level citations when discussing documents.

Data quality is usually harder than model selection. Pairs may be weakly aligned, duplicated, mislabeled, low resolution, or missing important context. Captions can also contain copyright, privacy, or demographic bias. I would deduplicate by perceptual and semantic similarity, filter unsafe or sensitive content, verify samples manually, and track provenance. Evaluation sets should represent real workflows rather than only clean web examples, with separate slices for each modality and failure type. Interview signal: Call out missing modalities, noisy labels, cultural bias, licensing, temporal alignment, and train-test leakage in datasets.

I would combine task metrics with human review and operational measures. Captioning can use similarity metrics, but visual question answering needs exact or semantic correctness checks. For extraction, field-level precision, recall, and citation accuracy matter. I would test hallucinations, refusal behavior, image quality, multilingual inputs, OCR-heavy documents, and prompt injection inside images. Production evaluation should also track latency, cost, failure rates, and disagreement between model output and human adjudication. Interview signal: Evaluate modality-specific accuracy plus grounding, calibration, robustness, latency, cost, and accessibility.

Visual grounding connects a textual claim to a specific region or object in an image. Instead of saying that a document contains a signature, a grounded system can identify the page and bounding box containing it. Grounding improves trust, debugging, and downstream actions such as cropping or redaction. It can be implemented with object detectors, region encoders, coordinate-aware prompts, or post-processing that maps model references back to image coordinates. Interview signal: Use bounding boxes, regions, timestamps, or quoted text to distinguish a grounded answer from a plausible description.

I would make uncertainty and evidence part of the design. The prompt should require the model to say when an answer is not visible, cite a region or page, and return structured fields. Retrieval or OCR can provide additional context, but I would not treat retrieved text as proof without checking the source image. Threshold-based routing to human review, constrained decoding, diverse evaluation cases, and monitoring unsupported claims are also important. Interview signal: Combine retrieval, crop or region checks, OCR, constrained output, abstention, and human review for high-risk visual claims.

Audio can be handled through a speech encoder, an automatic speech recognition stage, or both. For a meeting assistant, I might transcribe speech with timestamps, preserve speaker segments, and pass selected audio features or transcript chunks to a language model. Direct audio models can capture tone and nonverbal signals, but transcripts are easier to search and audit. The design must address consent, noisy recordings, accents, overlapping speakers, and retention. Interview signal: Describe audio ingestion, diarization, transcription, embeddings, timestamp preservation, and a text or audio response path.

Tokenization converts modality data into units a transformer can process. Text uses subword tokens, while images may use patches or learned visual tokens and audio may use time-frequency frames. More tokens can preserve detail but increase memory, latency, and cost. A practical system resizes or compresses inputs based on the task, uses adaptive detail where available, and measures whether extra tokens improve accuracy enough to justify their operational cost. Interview signal: Explain patches, visual tokens, frames, and detail settings as a context and cost budget problem.

I would avoid blindly sending the original resolution. First, create a thumbnail for global context, then identify likely regions using layout detection, tiling, or a first-pass model. Relevant tiles can be sent at higher resolution and merged with the global result. The pipeline should preserve coordinates, cap total tokens, and define behavior when regions overlap. For documents, page-level processing and targeted crops often provide better cost and latency than one huge request. Interview signal: Use tiling, downsampling, adaptive crops, and a coarse-to-fine pass while preserving the evidence needed for the task.

I would minimize collection, define retention periods, and restrict access to raw media. Images and audio should be encrypted in transit and at rest, with sensitive metadata removed where possible. Face, identity, health, and location data may require redaction or explicit consent. Logs should store references and structured outcomes rather than raw content by default. I would also verify provider data-use policies, define deletion workflows, and test access controls with realistic media. Interview signal: Mention EXIF, faces, voices, hidden metadata, retention, consent, access controls, and redacted observability.

Multimodal retrieval finds relevant items when the query and stored data may use different modalities. A text query can retrieve images, a screenshot can retrieve related documents, and an audio transcript can retrieve video segments. A shared embedding model enables cross-modal search, while reranking can use richer joint reasoning. I would evaluate recall at useful cutoffs, metadata filters, duplicate handling, and whether retrieved items actually support the user task. Interview signal: Contrast text-to-image, image-to-text, and cross-modal retrieval indexes with their different embedding and citation needs.

I would treat visible text in an image as untrusted data, not as system instructions. OCR and the vision model can identify suspicious instructions, but detection should not be the only defense. The application should separate user policy from image content, constrain tools and permissions, validate tool arguments, and require confirmation for consequential actions. Test cases should include screenshots, handwritten instructions, hidden small text, QR codes, and instructions embedded in charts. Interview signal: Treat text in images as untrusted instructions and isolate OCR results from policy, secrets, and tool authorization.

Specialist models are preferable when a modality has strict accuracy, latency, or compliance requirements. A dedicated OCR, speech recognition, object detection, or moderation model may be cheaper and more predictable than a general model. A single multimodal model is attractive when cross-modal reasoning is central and integration simplicity matters. I would compare end-to-end task accuracy, error recovery, operating cost, observability, and the consequences of each model's mistakes before choosing. Interview signal: Choose specialist OCR, ASR, vision, or text models when observability, cost, or modality-specific accuracy beats one large model.

The fallback should be based on an observable failure condition, not a vague feeling. Examples include unsupported format, timeout, low image quality, schema validation failure, or a confidence threshold. A text-only path may use OCR, metadata, or a user request for clarification. I would preserve the original request, avoid silently changing its meaning, communicate limitations clearly, and record which path answered it so fallback quality and cost can be measured. Interview signal: Design graceful degradation: preserve the user task with OCR, captions, text-only retrieval, or a clear unsupported-modality message.

A multimodal agent can perceive and act on visual, audio, or video state, so it must reason about changing observations as well as text. It needs modality-specific preprocessing, grounding, temporal context, and stronger action confirmation. For example, a browser agent may inspect a screenshot but should verify target coordinates before clicking. I would isolate tools, limit permissions, validate observations, and maintain an event trace linking each action to its evidence. Interview signal: Add visual and audio tools to agents with explicit permissions, evidence references, and confirmation for side effects.

I would measure cost per successful task, not just cost per request. Techniques include resizing inputs, using thumbnails and selective crops, batching compatible requests, caching embeddings, routing simple cases to smaller models, and limiting output tokens. Preprocessing can remove irrelevant frames or silence. I would set budgets and timeouts, monitor quality by input slice, and only adopt compression or quantization after confirming that savings do not create expensive downstream review. Interview signal: Control cost through preprocessing, resolution, caching, batching, model routing, and measured quality thresholds.