What we're building: a multimodal customer-service agent
We'll build a customer-service agent that helps a customer troubleshoot an app error. The customer first describes the problem in text. The agent asks for a screenshot and a short voice explanation, then uses both files to suggest a next step. The customer doesn't have to start the conversation over.
The agent offers guidance; it does not actually repair the customer's device or claim that it has. In this example, we'll focus on how the conversation flows, how to send image and audio inputs, and how the agent keeps earlier turns in context.
For the second part of the example, prepare two sample files in the directory where you run the code:
-
error.png— a screenshot showing the app's error. -
issue_details.wav— a short voice recording with details about what happened.
The example uses a Gemini model with image and audio input support. Install its LangChain integration and configure your API key before running the code:
pip install -U "langchain[google-genai]"
export GOOGLE_API_KEY="your-google-api-key"
Build the customer-service agent
import base64
from pathlib import Path
from langchain.agents import create_agent
from langchain.chat_models import init_chat_model
from langchain.messages import HumanMessage
from langgraph.checkpoint.memory import InMemorySaver
# Initialize a model that can accept both image and audio inputs.
model = init_chat_model("google_genai:gemini-3.7-flash")
# Tell the agent how to handle support requests and when to ask for evidence.
agent = create_agent(
model=model,
tools=[],
system_prompt=(
"You are a friendly customer-service assistant helping customers "
"troubleshoot app problems. If the customer describes an error but "
"has not provided enough detail, ask them to upload a screenshot and "
"send a short voice recording explaining what they were doing. Use "
"provided screenshots and audio to understand the issue. Ask for "
"clarification if needed, and do not claim to have fixed anything."
),
# Keep conversation state between calls that use the same thread ID.
checkpointer=InMemorySaver(),
)
# Reuse this ID for every message in this support case.
thread_config = {"configurable": {"thread_id": "support-case-1042"}}
# First, the customer describes the problem using text only.
initial_result = agent.invoke(
{
"messages": [
{
"role": "user",
"content": (
"My budgeting app shows an error when I try to export "
"a monthly report. Can you help?"
),
}
]
},
config=thread_config,
)
print("Agent:", initial_result["messages"][-1].content)
# After the agent asks for more information, read the customer's screenshot
# and voice recording and encode their file contents for the message.
image_base64 = base64.b64encode(Path("error.png").read_bytes()).decode("utf-8")
audio_base64 = base64.b64encode(Path("issue_details.wav").read_bytes()).decode("utf-8")
# Send the requested screenshot and voice details as the next user turn.
evidence_result = agent.invoke(
{
"messages": [
HumanMessage(
content=[
{
"type": "text",
"text": (
"Here are the screenshot and voice details you asked for. "
"Please use them to help troubleshoot the export error."
),
},
{
# The MIME type should match the actual image format.
"type": "image",
"base64": image_base64,
"mime_type": "image/png",
},
{
# Audio is sent in its own content block.
"type": "audio",
"base64": audio_base64,
"mime_type": "audio/wav",
},
]
)
]
},
config=thread_config,
)
print("Agent:", evidence_result["messages"][-1].content)
How the tech works
Multimodal messages
A multimodal message contains more than text: it can include content such as images or audio. In LangChain, a HumanMessage can contain a list of content blocks, combining text with media. The model receives that message and can use the media to respond.
The first customer turn is text-only. In the second turn, a HumanMessage combines text, an image block, and an audio block. Each block's type identifies its content; base64 carries the encoded file data and mime_type identifies its format.
Base64 encoding
base64.b64encode(Path(...).read_bytes()).decode("utf-8") reads a local file, encodes its bytes as Base64, and converts the result to a string for the content block. The MIME type must match the file: this example uses image/png for the screenshot and audio/wav for the recording.
Conversation memory
The shared thread_id tells the agent that the screenshot and recording belong to the same support conversation as the customer's initial question. The checkpointer saves the conversation state between calls, allowing the agent to use earlier messages as context. Use a different thread ID for a different customer or case.
InMemorySaver is suitable for a small demo, but its checkpoints live only in the running process and disappear when it stops. For a production application, use a persistent checkpointer and an appropriate retention policy.
Model and media support
The exact formats and limits depend on the model provider. The selected Gemini model supports image and audio input, but not every model accepts every media type or content-block format. Check the provider's current documentation for supported formats and size limits. Sending a file as Base64 does not make a text-only model capable of understanding it.
Depending on the provider and model, audio may be transcribed, interpreted directly, or require a provider-specific format. If the standard audio block is unsupported, use the provider's documented format or transcribe the recording first and send the resulting text.
A quick privacy reminder
Screenshots and recordings can contain personal or confidential information. Send only files you are authorized to process, avoid logging sensitive media unnecessarily, and remember that a checkpointer may retain message content.
Top comments (0)