How to Build a Multimodal AI Chatbot with Claude 3.5 and React 19
Multimodal AI chatbots can process text, images, and other media inputs, delivering richer user experiences than text-only tools. Anthropic’s Claude 3.5 Sonnet leads the pack for multimodal reasoning, while React 19’s new features like Server Components and Actions simplify full-stack integration. This guide walks through building a production-ready multimodal chatbot from scratch.
Prerequisites
- Node.js 20+ installed locally
- Anthropic API key (sign up at console.anthropic.com)
- Basic knowledge of React and JavaScript
- Familiarity with REST APIs
Step 1: Set Up the React 19 Project
Start by creating a new React 19 project using Vite, which supports React 19’s latest features out of the box:
npm create vite@latest claude-multimodal-chatbot -- --template react
cd claude-multimodal-chatbot
npm install
Install required dependencies: the Anthropic SDK for API calls, and react-dropzone for handling file uploads (for image inputs):
npm install @anthropic-ai/sdk react-dropzone
Step 2: Configure the Anthropic Client
Create a .env file in the project root to store your Anthropic API key. Never commit this file to version control:
VITE_ANTHROPIC_API_KEY=your_api_key_here
Next, create a src/lib/anthropic.js file to initialize the Anthropic client. React 19’s environment variable handling automatically exposes variables prefixed with VITE_ to the browser:
import Anthropic from '@anthropic-ai/sdk';
const anthropic = new Anthropic({
apiKey: import.meta.env.VITE_ANTHROPIC_API_KEY,
dangerouslyAllowBrowser: true, // Only for demo purposes; use a backend proxy in production
});
export default anthropic;
Note: For production use, route API calls through a backend service to keep your API key secure. This guide uses browser-side calls for simplicity.
Step 3: Build the Chatbot UI with React 19
React 19’s Server Components and Actions make it easy to handle form submissions and state updates without complex client-side state management. First, create a src/components/Chatbot.jsx component:
import { useState } from 'react';
import { useDropzone } from 'react-dropzone';
import anthropic from '../lib/anthropic';
export default function Chatbot() {
const [messages, setMessages] = useState([]);
const [inputText, setInputText] = useState('');
const [uploadedFile, setUploadedFile] = useState(null);
const [isLoading, setIsLoading] = useState(false);
const { getRootProps, getInputProps } = useDropzone({
accept: { 'image/*': [] },
maxFiles: 1,
onDrop: (acceptedFiles) => {
setUploadedFile(acceptedFiles[0]);
},
});
const handleSubmit = async (e) => {
e.preventDefault();
if (!inputText.trim() && !uploadedFile) return;
setIsLoading(true);
const userMessage = {
role: 'user',
content: [],
};
// Add text content if present
if (inputText.trim()) {
userMessage.content.push({ type: 'text', text: inputText.trim() });
}
// Add image content if uploaded
if (uploadedFile) {
const reader = new FileReader();
reader.onload = async (e) => {
const base64Image = e.target.result.split(',')[1];
userMessage.content.push({
type: 'image',
source: {
type: 'base64',
media_type: uploadedFile.type,
data: base64Image,
},
});
// Send message to Claude
try {
const response = await anthropic.messages.create({
model: 'claude-3-5-sonnet-20241022',
max_tokens: 1024,
messages: [...messages, userMessage],
});
const assistantMessage = {
role: 'assistant',
content: response.content[0].text,
};
setMessages((prev) => [...prev, userMessage, assistantMessage]);
setInputText('');
setUploadedFile(null);
} catch (error) {
console.error('Error calling Claude API:', error);
alert('Failed to get response from Claude. Check your API key and try again.');
} finally {
setIsLoading(false);
}
};
reader.readAsDataURL(uploadedFile);
} else {
// Send text-only message
try {
const response = await anthropic.messages.create({
model: 'claude-3-5-sonnet-20241022',
max_tokens: 1024,
messages: [...messages, userMessage],
});
const assistantMessage = {
role: 'assistant',
content: response.content[0].text,
};
setMessages((prev) => [...prev, userMessage, assistantMessage]);
setInputText('');
} catch (error) {
console.error('Error calling Claude API:', error);
alert('Failed to get response from Claude. Check your API key and try again.');
} finally {
setIsLoading(false);
}
}
};
return (
{messages.map((msg, index) => (
{msg.role === 'user' ? (
{msg.content.map((item, i) => (
{item.type === 'text' ? {item.text} : }
))}
) : (
{msg.content}
)}
))}
{isLoading && Loading...}
{uploadedFile ? (
Uploaded: {uploadedFile.name}
) : (
Drag & drop an image here, or click to select
)}
setInputText(e.target.value)}
placeholder="Type your message..."
className="text-input"
/>
Send
);
}
Step 4: Style the Chatbot
Add basic CSS to src/index.css to make the chatbot usable:
.chatbot-container {
max-width: 800px;
margin: 2rem auto;
border: 1px solid #e5e7eb;
border-radius: 8px;
overflow: hidden;
height: 80vh;
display: flex;
flex-direction: column;
}
.messages-container {
flex: 1;
padding: 1rem;
overflow-y: auto;
background-color: #f9fafb;
}
.message {
margin-bottom: 1rem;
padding: 0.75rem 1rem;
border-radius: 8px;
max-width: 70%;
}
.message.user {
background-color: #dbeafe;
margin-left: auto;
}
.message.assistant {
background-color: #ffffff;
border: 1px solid #e5e7eb;
margin-right: auto;
}
.uploaded-image {
max-width: 200px;
border-radius: 4px;
margin-top: 0.5rem;
}
.input-form {
padding: 1rem;
border-top: 1px solid #e5e7eb;
background-color: #ffffff;
}
.dropzone {
border: 2px dashed #d1d5db;
border-radius: 8px;
padding: 1rem;
text-align: center;
margin-bottom: 1rem;
cursor: pointer;
}
.dropzone:hover {
border-color: #3b82f6;
}
.text-input-container {
display: flex;
gap: 0.5rem;
}
.text-input {
flex: 1;
padding: 0.75rem;
border: 1px solid #d1d5db;
border-radius: 8px;
font-size: 1rem;
}
.send-button {
padding: 0.75rem 1.5rem;
background-color: #3b82f6;
color: white;
border: none;
border-radius: 8px;
cursor: pointer;
font-size: 1rem;
}
.send-button:disabled {
background-color: #93c5fd;
cursor: not-allowed;
}
Step 5: Run and Test the Chatbot
Start the development server:
npm run dev
Open the provided local URL in your browser. Test the chatbot by sending text messages, uploading images, or both. Claude 3.5 will process multimodal inputs and return contextually relevant responses.
Production Considerations
- Never expose your Anthropic API key in client-side code. Use a backend proxy (e.g., Node.js, Express) to handle API calls securely.
- Add rate limiting to prevent API abuse.
- Implement error handling for network failures and invalid inputs.
- Use React 19’s Server Components to pre-render chat history for faster initial loads.
- Add support for more media types (e.g., PDFs, audio) by extending the dropzone accept rules and Claude’s supported media types.
Conclusion
Building a multimodal AI chatbot with Claude 3.5 and React 19 is straightforward thanks to Claude’s robust multimodal API and React 19’s streamlined development features. This setup provides a foundation you can extend with custom features like chat history persistence, user authentication, or integration with other tools. Start experimenting today to build AI experiences that go beyond text-only interactions.
Top comments (0)