Multimodal AI Explained (Teaching Models to See and Hear, Not Just Read)

Written by Syed Muhammad Ali Raza

Alright, eight articles into this series and every single thing we've built so far has been talking to a model that only understands text. Type something, get text back. That's genuinely most of what people mean when they say "AI" in casual conversation, but it's honestly only half the story of where this stuff has gone.

Think about literally any AI feature you've actually used recently that impressed you. Google Lens pointing your phone camera at a plant and it just tells you what it is. Those "describe this photo for a caption" tools. Apps that transcribe a voice memo into text notes automatically. Snapchat filters that track your face in real time. TikTok generating auto captions for a video. Every single one of those is multimodal AI doing the heavy lifting, models that don't just read, they see, and increasingly, they hear too.

This article is your actual hands on introduction to that world, real examples, real code, and honestly, once you get this working the first time, it's one of the more genuinely fun things you can build in this entire series. Let's get into it.

Okay but what does "multimodal" actually mean

A "mode" here just means a type of input, text, images, audio, video. A multimodal model is one that's been trained to understand more than one of these at the same time, in the same conversation, often mixed together in a single request.

So instead of only ever typing "describe a sunset over mountains" and getting text back describing an imagined sunset, you can literally hand the model an actual photo you took and ask "what time of day do you think this was taken, and does the lighting look natural or edited." The model isn't guessing based on the word "sunset," it's actually looking at the pixels in your specific photo and reasoning about what it sees.

A real life example before any code, promise

Picture two different friends helping you plan a trip.

Friend one only communicates through written notes slid under a door. You describe your travel photos to them in text, "there's a big mountain, some snow, kind of cloudy," and they respond based purely on your description. If your description is bad or you forget an important detail, their advice is only as good as what you managed to put into words.

Friend two is standing right next to you, actually looking at your phone screen with you. You don't have to describe anything, they just see the photo directly, notice the trail marker sign in the corner you didn't even mention, notice the weather looks worse than you described, and gives advice based on what's actually there, not just your secondhand description of it.

A text only LLM is friend one. A multimodal model is friend two. And once you've used friend two, going back to typing out clumsy descriptions of images feels genuinely limiting, the same way texting someone a play by play of a video call would feel limiting compared to just being on the video call.

How this actually works under the hood, briefly

You don't need a computer vision PhD for this, but a rough mental model helps a lot.

Remember back in article two of this series, we talked about how text gets broken into tokens, small chunks that get converted into number lists the model can actually work with. Images and audio go through a conceptually similar process. An image gets broken into patches, small square regions of pixels, and each patch gets converted into its own number list, its own embedding, sitting in the exact same kind of mathematical space that word tokens live in. Audio gets sliced into small time chunks and converted the same way.

The genuinely clever part, and the reason this whole thing works at all, is that during training, the model learns to place related concepts from different modalities close together in that shared number space. A photo of a golden retriever and the actual word "dog" end up landing near each other, even though one came from pixels and one came from text. That shared space is what lets the model reason across text and images in the same conversation, seamlessly, like they were never different types of data to begin with.

Let's actually build something, starting with vision

I'm going to walk through a genuinely useful example, analyzing a receipt image and extracting structured expense data from it, something that's an actual real headache for a lot of people, freelancers, small business owners, anyone who's ever tried to do expense reports manually.

Step 1, sending an image to the model

import anthropic
import base64

client = anthropic.Anthropic(api_key="your-api-key-here")

def encode_image(image_path):
    with open(image_path, "rb") as image_file:
        return base64.standard_b64encode(image_file.read()).decode("utf-8")

def analyze_receipt(image_path):
    image_data = encode_image(image_path)

    response = client.messages.create(
        model="claude-sonnet-4-6",
        max_tokens=500,
        messages=[
            {
                "role": "user",
                "content": [
                    {
                        "type": "image",
                        "source": {
                            "type": "base64",
                            "media_type": "image/jpeg",
                            "data": image_data
                        }
                    },
                    {
                        "type": "text",
                        "text": "Extract the merchant name, date, total amount, and each line item with its price from this receipt. Respond as clean JSON only."
                    }
                ]
            }
        ]
    )

    return response.content[0].text

result = analyze_receipt("my_receipt.jpg")
print(result)

Notice the structure of the content list, an image block first, then a text block asking a specific question about it, both in the same single message. This is genuinely the core pattern for every multimodal request you'll ever build, you're not sending the image and the question separately, they're part of the exact same conversational turn, exactly like showing your friend the photo while you're asking them the question, not describing the photo first and asking the question after.

Step 2, actually parsing that response into something useful

import json
import re

def parse_receipt_json(raw_response):
    # models sometimes wrap json in markdown code fences, strip that out first
    cleaned = re.sub(r"```

json|

```", "", raw_response).strip()
    try:
        return json.loads(cleaned)
    except json.JSONDecodeError:
        print("Could not parse response as JSON, raw output was:")
        print(raw_response)
        return None

receipt_data = parse_receipt_json(result)
if receipt_data:
    print(f"Merchant: {receipt_data.get('merchant_name')}")
    print(f"Total: {receipt_data.get('total_amount')}")

Once you've got this working, you've genuinely built the core of an actual expense tracking tool, snap a photo, get structured data back automatically, no manual typing. This exact pattern, image in, structured data out, is behind a huge chunk of real document processing products people pay real money for.

Analyzing charts and graphs, not just photos

Here's a genuinely underrated use case, feeding the model a screenshot of a chart or graph and asking it real analytical questions about the data, instead of you squinting at it and doing mental math yourself.

def analyze_chart(image_path, question):
    image_data = encode_image(image_path)

    response = client.messages.create(
        model="claude-sonnet-4-6",
        max_tokens=500,
        messages=[
            {
                "role": "user",
                "content": [
                    {
                        "type": "image",
                        "source": {
                            "type": "base64",
                            "media_type": "image/png",
                            "data": image_data
                        }
                    },
                    {"type": "text", "text": question}
                ]
            }
        ]
    )
    return response.content[0].text

answer = analyze_chart(
    "quarterly_revenue_chart.png",
    "Which quarter had the sharpest decline, and roughly what percentage did revenue drop by?"
)
print(answer)

This genuinely works well because these models were trained on a huge amount of visual data that includes charts, diagrams, and infographics, they're not just recognizing "this is a bar chart," they're actually reading the relative bar heights, the axis labels, and reasoning about the actual data being shown, similar to how a human would glance at a chart and pull out the trend.

Now let's add audio into the mix

Vision gets a lot of the spotlight, but audio understanding, mainly transcription, is honestly just as practically useful, maybe even more so for a huge number of real apps, meeting notes, voice memos, podcast summaries, accessibility tools.

# using OpenAI's Whisper model for transcription, since it's
# widely used and specifically built for this exact job
from openai import OpenAI

openai_client = OpenAI(api_key="your-openai-api-key-here")

def transcribe_audio(audio_file_path):
    with open(audio_file_path, "rb") as audio_file:
        transcript = openai_client.audio.transcriptions.create(
            model="whisper-1",
            file=audio_file
        )
    return transcript.text

transcript_text = transcribe_audio("voice_memo.mp3")
print(transcript_text)

Chaining audio and text together, transcribe then actually do something with it

The real power move here is combining tools, transcribing audio and then handing that transcript straight to your text model for analysis, summarization, action item extraction, whatever you actually need.

def summarize_voice_memo(audio_file_path):
    transcript = transcribe_audio(audio_file_path)

    response = client.messages.create(
        model="claude-sonnet-4-6",
        max_tokens=300,
        messages=[
            {
                "role": "user",
                "content": f"Here's a transcript of a voice memo:\n\n{transcript}\n\nSummarize the key points and list any action items mentioned."
            }
        ]
    )
    return response.content[0].text

summary = summarize_voice_memo("meeting_notes.mp3")
print(summary)

Record a rambling five minute voice memo while walking to class or driving home from work, and this turns it into a clean, organized summary with actual action items pulled out automatically. This exact pipeline, audio in, structured summary out, is genuinely the backbone of most of those trendy "AI note taking app" products you've probably seen advertised.

Generating images from text, the reverse direction

Everything above was understanding existing images and audio. The other direction, generating new images from a text description, uses a different kind of model entirely, usually a diffusion model, but it's worth knowing how to actually call one since it's such a common thing people want to add to a project.

def generate_image(prompt, output_path="generated_image.png"):
    response = openai_client.images.generate(
        model="dall-e-3",
        prompt=prompt,
        size="1024x1024",
        n=1
    )

    image_url = response.data[0].url

    # download and save it locally
    import requests
    image_response = requests.get(image_url)
    with open(output_path, "wb") as f:
        f.write(image_response.content)

    print(f"Saved to {output_path}")
    return output_path

generate_image("a cozy coffee shop illustration, warm lighting, minimalist flat design style")

A quick, genuinely important note here since it trips people up, image generation models are architecturally a completely different thing from the vision understanding we did earlier. Understanding an image, and generating a new one, are handled by different types of models under the hood, even though from a user's perspective it can feel like "the AI" doing both. Don't expect the same model handling your receipt scanning to also be the one drawing pictures for you, that's typically two separate API calls to two genuinely different systems.

Putting it all together, a genuinely fun mini project

Let's combine everything into one pipeline, take a voice memo describing an idea for a piece of art, transcribe it, and generate an actual image from that description. This is honestly one of the more satisfying things to watch actually run.

def voice_memo_to_image(audio_file_path):
    print("Step 1, transcribing your voice memo...")
    transcript = transcribe_audio(audio_file_path)
    print(f"Transcript: {transcript}\n")

    print("Step 2, turning that into a clean image generation prompt...")
    response = client.messages.create(
        model="claude-sonnet-4-6",
        max_tokens=150,
        messages=[
            {
                "role": "user",
                "content": f"Turn this rambling voice memo into a single, clear, vivid image generation prompt, one sentence: {transcript}"
            }
        ]
    )
    image_prompt = response.content[0].text
    print(f"Image prompt: {image_prompt}\n")

    print("Step 3, generating the actual image...")
    output_path = generate_image(image_prompt)

    return output_path

voice_memo_to_image("my_art_idea.mp3")

Say something out loud, and thirty seconds later there's an actual generated image sitting on your disk based on what you described. Genuinely, if you want a fun weekend project to really cement everything from this article, build a tiny app around this exact pipeline. It touches vision adjacent generation, audio transcription, and text reasoning to tie the whole thing together, and it's the kind of demo that gets people excited when you show it off.

The real world use cases worth knowing about

Once you've built the pieces above, a few genuinely valuable directions this opens up.

Accessibility tools are honestly one of the most meaningful uses of vision models, automatically describing images for visually impaired users, far beyond basic alt text, actual rich descriptions of what's happening in a photo or a scene.

Document processing at scale, receipts like we built, but also invoices, forms, ID verification, medical documents, insurance claims, basically anything that used to require a human manually reading and typing data from a scanned page.

Video understanding is a natural extension once you can handle images, since video is really just a sequence of image frames plus audio, sampling frames at intervals and feeding them through the same vision pipeline lets you build things like automatic scene descriptions or content moderation for video content.

Real time accessibility and translation, live captioning combining audio transcription with instant translation, or camera based tools that describe your surroundings out loud for someone who's visually impaired, genuinely powerful, genuinely available to build today with the exact pieces covered in this article.

The honest problems you'll actually run into

I want to be straight about the rough edges here too, because multimodal work has its own specific headaches beyond what we covered in the text only articles earlier in this series.

Images are expensive in tokens. A single decent resolution image can easily cost as many tokens as several paragraphs of text, which matters a lot for the cost tracking habits we covered in the last article. If you're processing a lot of images regularly, resizing them down to the minimum resolution that still lets the model see what it needs to see is a genuinely easy, meaningful cost saver, don't send a twelve megapixel photo when the model can extract the same information from something dramatically smaller.

Vision models can still get details wrong, misreading small or blurry text, miscounting objects in a busy scene, missing something in a cluttered image. Treat vision output with the exact same healthy skepticism from the evaluation article, especially for anything genuinely important like financial data, don't blindly trust extracted numbers without some kind of validation step, even a simple sanity check like "does this total match the sum of the line items."

Privacy deserves real thought here too. Images and audio often carry way more sensitive information than the user consciously intends, a receipt photo might have a home address on it, a voice memo might have someone else's voice or private information in the background. Think carefully about what you're actually sending to a third party API and what you're storing afterward, the exact same care from the security article in this series applies here, just with a new category of sensitive content to think about.

Bringing this back to the whole series

We started this entire series with a model that could only read plain text typed into a chat box. Now, across eight articles, that same core idea, an LLM predicting the next token, has grown into something that can ground itself in your own documents, take real actions in the world, defend itself against manipulation, coordinate across a team of specialized agents, get properly evaluated instead of just vibed on, run reliably at real production scale, and now, genuinely see and hear the world around it instead of only reading text describing that world secondhand. That's honestly the direction the entire field keeps moving, less "type words at a chatbot," more "an assistant that actually perceives and acts in the world the way people naturally do."

If you build even one small multimodal project this week, snap a photo of something and ask a real question about it, that's genuinely enough to feel the difference for yourself, and it's a good excuse to actually get your hands on the whole shared token space idea instead of just reading about it.

---

If you build something with vision or audio off the back of this article, genuinely send me what you built, this is the part of the series I'm most excited to see people run with.