Building Multimodal Chatbots
Combine uploads, vision, speech, and text into a useful conversational experience.
Define the conversation contract
Tell users what the assistant accepts, what it can inspect, how long processing may take, and which results require review. A multimodal chatbot should make the current assets visible and let users remove or replace them. Clear boundaries prevent the interface from implying that every file, language, or task is supported equally well.
Upload handling
Validate file type, size, dimensions, duration, page count, and decompression cost before invoking a model. Scan files for malware and store them with opaque identifiers rather than user-controlled paths. Generate safe previews and preserve originals only as long as necessary. Errors should identify the corrective action without exposing internal storage or provider details.
message = {
"role": "user",
"content": [
{"type": "text", "text": "Describe this receipt."},
{"type": "image_url", "image_url": {"url": "receipt.jpg"}},
],
}
print(message)Routing by modality
Route images, documents, audio, and text to the capability that fits the task. A document may need OCR plus layout extraction, while a voice message may need transcription before summarization. Keep a shared conversation ID but preserve each asset’s metadata and processing state. Routing rules should be observable so a wrong answer can be traced to the selected path.
Conversation state
Store user messages, asset references, model outputs, and validation results separately. Do not place full files or unbounded history into every prompt. Summarize older turns, retrieve relevant assets, and enforce a context budget. Support retries without duplicating side effects, and make it clear when a response is based on a previous upload that may no longer be available.
Safety and prompt injection
Text embedded in an image or document is data, not automatically an instruction. Keep system policy and tool permissions outside untrusted content, and validate any action proposed by the model. Add moderation for inputs and outputs, rate limits, abuse monitoring, and human escalation for sensitive categories. A chatbot’s friendly tone must not weaken server-side authorization.
Human-centered feedback
Show progress for slow operations, partial transcripts when useful, and uncertainty when extraction may be wrong. Let users correct fields, retry a failed modality, report harmful output, and inspect sources. Feedback should be tied to the model, prompt, asset, and pipeline version so it improves the system. Avoid collecting more conversation data than the feedback purpose requires.
Release and evaluation
Test complete journeys: upload, preview, processing, response, correction, retry, deletion, and failure recovery. Include large files, unsupported formats, network interruption, ambiguous images, noisy audio, and adversarial documents. Measure task completion, correction effort, latency, cost, privacy incidents, and unsafe outputs. Launch with a narrow promise and expand only when the evidence supports it.
Worked example: a receipt assistant
A receipt chatbot can accept a photo and a question such as “What was the total and which items were taxed?” The flow validates the image, extracts a structured record, shows evidence or asks for a clearer photo, and lets the user correct a field. Conversation history should retain only what the task needs. An image message is not equivalent to text: the product needs upload progress, orientation handling, privacy messaging, and a safe deletion path.
Code walkthrough
The message array places text and image parts in an explicit order so the model receives the user’s request and visual context. A real client should use a supported image URL or upload reference, not expose a local filesystem path to a browser. Normalize provider responses into text, structured fields, citations, and usage. Validate each image part and reject unexpected content types before sending it downstream.
Trade-offs to measure
A single multimodal model gives a simple user experience, but a staged OCR-plus-text pipeline can be cheaper, faster, and easier to validate for predictable documents. Keeping images in every conversation improves context while increasing privacy exposure and token cost. Tool use can add calculations or search, but also expands the attack surface. Compare complete task success, correction rate, latency, cost, and data retention.
Practical exercise
Prototype a chatbot that handles a receipt, a blurry receipt, and a non-receipt image. Add states for upload, processing, answer, clarification, and failure. Require the model to return total, currency, and evidence text, then verify the total with a simple calculation when line items are available. Test refresh, cancellation, duplicate submission, and deletion so the image does not remain accidentally in client or server storage.
Sources and further reading
These primary or specialist references informed the concepts in this guide. Product details can change, so verify current documentation before implementation.