DEV Community

Cover image for Build Your Own Jarvis AI Assistant in Python 🚀
Bilal Aslam
Bilal Aslam

Posted on Edited on

Build Your Own Jarvis AI Assistant in Python 🚀

#ai #aitool #automation #beginners 
# Build Your Own Jarvis AI Assistant in Python 🚀

![Python](https://img.shields.io/badge/Python-3.10%2B-3776AB?style=flat-square&logo=python&logoColor=white)

![Gemini](https://img.shields.io/badge/Google%20Gemini-API-8B5CF6?style=flat-square&logo=googlegemini&logoColor=white)

![Voice Assistant](https://img.shields.io/badge/Voice-Assistant-0EA5E9?style=flat-square)

![License](https://img.shields.io/badge/License-MIT-22C55E?style=flat-square)

> A beginner-friendly Python voice assistant that listens for the wake word **"Jarvis"**, understands voice commands, performs useful desktop and web actions, and uses **Google Gemini** for AI-powered responses.

---

## Introduction

Ever wanted to build your own AI assistant like **Jarvis from Iron Man**?

I started this project as a way to learn Python, APIs, speech recognition, automation, and AI integration by actually building something instead of only following tutorials.

The project has evolved from a simple Python voice assistant into a more capable assistant using **Google Gemini** for AI-powered responses.

Jarvis can:

- 🎤 Listen for the wake word `Jarvis`
- 🗣️ Convert speech into text
- 🤖 Generate AI responses using Google Gemini
- 🔊 Speak responses using text-to-speech
- 🌐 Open websites such as YouTube, Google, and Spotify
- 🎵 Play songs from a custom music library
- 📰 Fetch and read news headlines
- 📅 Tell the current date
- ⏰ Tell the current time
- 💬 Open WhatsApp on Windows
- ⚙️ Perform different local automation tasks

The goal is not to create a perfect commercial AI assistant.

The goal is to **learn by building**.

---

# ✨ Features

## 🎤 Wake Word Detection

Jarvis continuously listens for short audio recordings and waits until it detects:

```text
Jarvis
Enter fullscreen mode Exit fullscreen mode

Once detected, it responds:

Yes Commander?
Enter fullscreen mode Exit fullscreen mode

and starts listening for the actual command.

The user's title is configurable through:

USER_TITLE = "Commander"
Enter fullscreen mode Exit fullscreen mode

🗣️ Speech Recognition

The microphone input is recorded using sounddevice.

The recorded audio is then converted into an AudioData object and passed to SpeechRecognition.

The basic flow is:

Microphone
    ↓
sounddevice
    ↓
AudioData
    ↓
SpeechRecognition
    ↓
Text command
Enter fullscreen mode Exit fullscreen mode

This gives Jarvis the ability to understand spoken commands instead of requiring keyboard input.

🤖 Gemini AI Integration

One of the biggest changes in the newer version of Jarvis is the move to Google Gemini for AI-powered responses.

The project uses Google's modern GenAI SDK:

from google import genai
Enter fullscreen mode Exit fullscreen mode

The Gemini client is initialized using the API key stored in config.json:

client = genai.Client(api_key=gemini_api_key)
Enter fullscreen mode Exit fullscreen mode

The Gemini model can also be configured without changing the Python code:

{
  "GEMINI_API_KEY": "YOUR_GEMINI_API_KEY",
  "GEMINI_MODEL": "gemini-3.6-flash",
  "NEWS_API_KEY": ""
}
Enter fullscreen mode Exit fullscreen mode

If Jarvis does not recognize a command as one of its built-in actions, it sends the command to Gemini.

For example:

Jarvis
Enter fullscreen mode Exit fullscreen mode 
Explain object oriented programming in Python
Enter fullscreen mode Exit fullscreen mode

Jarvis sends the request to Gemini and then speaks the generated response.

👨‍✈️ Personalized Responses

Jarvis has a configurable user title:

USER_TITLE = "Commander"
Enter fullscreen mode Exit fullscreen mode

This allows the assistant to address the user naturally.

For example:

Yes Commander?
Enter fullscreen mode Exit fullscreen mode

or:

Sorry Commander, this song is not in your library.
Enter fullscreen mode Exit fullscreen mode

You can change this value to:

USER_TITLE = "Sir"
Enter fullscreen mode Exit fullscreen mode

or:

USER_TITLE = "Bilal"
Enter fullscreen mode Exit fullscreen mode

🌐 Web Automation

Jarvis can open commonly used websites directly through Google Chrome.

Currently supported:

Open YouTube
Open Google
Open Spotify
Enter fullscreen mode Exit fullscreen mode

For example:

Jarvis
Enter fullscreen mode Exit fullscreen mode 
Open YouTube
Enter fullscreen mode Exit fullscreen mode

Jarvis opens YouTube automatically in Chrome.

The current Chrome path is configured for Windows:

chrome_path = "C:/Program Files/Google/Chrome/Application/chrome.exe %s"
Enter fullscreen mode Exit fullscreen mode

If Chrome is installed somewhere else on your system, this path may need to be changed.

🎵 Music Library

Jarvis can play songs from a custom music library.

Songs are stored separately in:

music_library.py
Enter fullscreen mode Exit fullscreen mode

The command format is:

Play <song name>
Enter fullscreen mode Exit fullscreen mode

For example:

Jarvis
Enter fullscreen mode Exit fullscreen mode 
Play Never Gonna Give You Up
Enter fullscreen mode Exit fullscreen mode

Jarvis checks whether the requested song exists in:

music_library.library
Enter fullscreen mode Exit fullscreen mode

If the song exists, Jarvis opens the saved link.

If it doesn't exist, Jarvis responds:

Sorry Commander, this song is not in your library.
Enter fullscreen mode Exit fullscreen mode

This makes it easy to customize the assistant with your own songs.

📰 News Headlines

Jarvis can fetch the latest headlines using the News API.

Command:

Global news
Enter fullscreen mode Exit fullscreen mode

When a valid NEWS_API_KEY is configured, Jarvis requests the top headlines and reads up to five of them.

The current implementation uses the Pakistan news endpoint:

https://newsapi.org/v2/top-headlines?country=pk
Enter fullscreen mode Exit fullscreen mode

Example:

Headline 1: ...
Headline 2: ...
Headline 3: ...
Enter fullscreen mode Exit fullscreen mode

The News API key is optional.

If you don't configure it, Jarvis can still be used for its other features.

💬 WhatsApp Automation

The Windows version can launch the installed WhatsApp application using PowerShell.

Command:

Open WhatsApp
Enter fullscreen mode Exit fullscreen mode

Jarvis uses Windows' application launcher to find and start the WhatsApp application.

This feature is currently designed for Windows.

📅 Date and Time

Jarvis can provide basic date and time information.

Current Date

Command:

Today date
Enter fullscreen mode Exit fullscreen mode

Example response:

Today is Friday, 21 August 2026
Enter fullscreen mode Exit fullscreen mode

The date is generated dynamically using Python's datetime module.

Current Time

Command:

Current time
Enter fullscreen mode Exit fullscreen mode

Jarvis reads the current time aloud.

It also handles the special case where the minute is exactly 00, responding with an "o'clock" style response.

🧠 AI Fallback

This is one of the most useful parts of the project.

Jarvis first checks whether a command matches one of its built-in commands.

For example:

Open YouTube
Enter fullscreen mode Exit fullscreen mode

is handled locally.

But if the command doesn't match a predefined action:

Explain recursion in Python
Enter fullscreen mode Exit fullscreen mode

Jarvis sends it to Gemini.

The simplified architecture looks like this:

                 ┌──────────────────┐
                 │  Voice Command   │
                 └────────┬─────────┘
                          ↓
                 ┌──────────────────┐
                 │ Speech-to-Text   │
                 └────────┬─────────┘
                          ↓
                 ┌──────────────────┐
                 │  Wake Word Check │
                 └────────┬─────────┘
                          ↓
                 ┌──────────────────┐
                 │  Command Router  │
                 └───────┬───┬──────┘
                         │   │
              Known      │   │      Unknown
              command    │   │      command
                         ↓   ↓
                  ┌──────┐  ┌─────────────┐
                  │Local │  │ Gemini AI   │
                  │Action│  │ Response    │
                  └───┬──┘  └──────┬──────┘
                      │             │
                      └──────┬──────┘
                             ↓
                    ┌────────────────┐
                    │ Text-to-Speech │
                    └────────────────┘
Enter fullscreen mode Exit fullscreen mode

This gives Jarvis two different capabilities:

  1. Deterministic local commands
  2. AI-powered responses

🛠️ Tech Stack

Technology Purpose
Python Main programming language
Google Gemini AI-powered responses
google-genai Gemini API integration
SpeechRecognition Speech-to-text recognition
sounddevice Microphone recording
gTTS Text-to-speech generation
pygame-ce Audio playback
requests News API requests
webbrowser Opening websites
subprocess Windows application automation
datetime Date and time
JSON Configuration and API keys

📁 Project Structure

The project currently follows a simple structure:

Jarvis-AI-Virtual-Assistant/
│
├── main.py
├── music_library.py
├── config.json.example
├── requirements.txt
├── .gitignore
├── LICENSE
└── README.md
Enter fullscreen mode Exit fullscreen mode

The main application logic lives inside:

main.py
Enter fullscreen mode Exit fullscreen mode

The music collection is managed separately in:

music_library.py
Enter fullscreen mode Exit fullscreen mode

The example configuration is stored in:

config.json.example
Enter fullscreen mode Exit fullscreen mode

Your real config.json should remain local and should never be uploaded to GitHub.

🚀 Getting Started

1. Prerequisites

Before starting, make sure you have:

  • Python 3.10 or newer
  • A working microphone
  • Internet connection
  • Google Gemini API key
  • Windows if you want to use the WhatsApp launcher

2. Clone the Repository

Clone the project from GitHub:

git clone https://github.com/bilal-dev-0x/Jarvis-AI-Virtual-Assistant.git
Enter fullscreen mode Exit fullscreen mode

Then enter the project directory:

cd Jarvis-AI-Virtual-Assistant
Enter fullscreen mode Exit fullscreen mode

3. Create a Virtual Environment

Create a virtual environment:

python -m venv .venv
Enter fullscreen mode Exit fullscreen mode

Windows PowerShell 

.\.venv\Scripts\Activate.ps1
Enter fullscreen mode Exit fullscreen mode

Linux/macOS 

source .venv/bin/activate
Enter fullscreen mode Exit fullscreen mode

📦 4. Install Dependencies

Install the required Python packages:

pip install -r requirements.txt
Enter fullscreen mode Exit fullscreen mode

This installs the libraries required for:

  • Voice input
  • Speech recognition
  • Gemini
  • Text-to-speech
  • Audio playback
  • News API requests

🔐 5. Configure API Keys

The project uses a local config.json file for API configuration.

First create it from the example file.

Windows 

copy config.json.example config.json
Enter fullscreen mode Exit fullscreen mode

Linux/macOS 

cp config.json.example config.json
Enter fullscreen mode Exit fullscreen mode

Then open:

config.json
Enter fullscreen mode Exit fullscreen mode

and add your API keys.

Example:

{
  "GEMINI_API_KEY": "YOUR_GEMINI_API_KEY",
  "GEMINI_MODEL": "gemini-3.6-flash",
  "NEWS_API_KEY": ""
}
Enter fullscreen mode Exit fullscreen mode

Gemini API Key

The Gemini API key is required because Gemini is used for AI fallback responses.

News API Key

The News API key is optional.

It is only required for:

Global news
Enter fullscreen mode Exit fullscreen mode

⚠️ Never upload your real API keys to GitHub.

▶️ 6. Run Jarvis

Start the assistant with:

python main.py
Enter fullscreen mode Exit fullscreen mode

Jarvis will initialize and begin listening.

Say:

Jarvis
Enter fullscreen mode Exit fullscreen mode

Then give it a command.

🎯 Example Commands

Here are some commands you can try:

Voice Command Action
Hello Greets the user
Open YouTube Opens YouTube
Open Google Opens Google
Open Spotify Opens Spotify
Play <song> Plays a song from the music library
Global news Reads up to five top Pakistan headlines
Today date Tells the current date
Current time Tells the current time
Open WhatsApp Opens WhatsApp on Windows
Any other question Sends the request to Gemini

For example:

Jarvis
Enter fullscreen mode Exit fullscreen mode 
Explain Python functions
Enter fullscreen mode Exit fullscreen mode

or:

Jarvis
Enter fullscreen mode Exit fullscreen mode 
What is machine learning?
Enter fullscreen mode Exit fullscreen mode

Jarvis can pass those requests to Gemini and speak the response.

⚙️ How The Code Works

The project is divided into several simple pieces.

1. Recording Audio

The assistant records microphone input using sounddevice.

A simplified version looks like:

recording = sd.rec(
    int(seconds * sample_rate),
    samplerate=sample_rate,
    channels=1,
    dtype="int16"
)

sd.wait()
Enter fullscreen mode Exit fullscreen mode

The recording is then converted into AudioData:

audio_data = sr.AudioData(
    recording.tobytes(),
    sample_rate,
    2
)
Enter fullscreen mode Exit fullscreen mode

This allows SpeechRecognition to process the recorded audio.

2. Converting Speech to Text

The recorded audio is sent to Google's speech recognition service:

command = recognizer.recognize_google(audio)
Enter fullscreen mode Exit fullscreen mode

This gives Jarvis a text representation of what the user said.

3. Detecting the Wake Word

Jarvis checks whether the recognized text contains:

if "jarvis" in command.lower():
Enter fullscreen mode Exit fullscreen mode

If it does, Jarvis enters command mode.

It then responds:

Yes Commander?
Enter fullscreen mode Exit fullscreen mode

and records another audio clip for the actual command.

4. Processing Commands

After the wake word is detected, Jarvis checks several predefined commands.

For example:

elif "open youtube" in text.lower():
    webpath.open("https://www.youtube.com/")
Enter fullscreen mode Exit fullscreen mode

This connects a spoken command directly to a Python automation task.

Other commands follow the same basic structure.

5. Sending Unknown Commands to Gemini

If none of the predefined commands match, Jarvis sends the command to Gemini.

The project creates a Gemini chat using the configured model:

chat = client.chats.create(model=gemini_model)
Enter fullscreen mode Exit fullscreen mode

A prompt is then created:

prompt = f"""
You are Jarvis, a helpful AI virtual assistant.
Give short responses.
Address the user naturally as '{USER_TITLE}'.

User command: {command}
"""
Enter fullscreen mode Exit fullscreen mode

The command is sent to Gemini:

response = chat.send_message(prompt)
Enter fullscreen mode Exit fullscreen mode

If Gemini returns text, Jarvis speaks the response.

6. Text-to-Speech

Jarvis uses gTTS to generate speech:

tts = gTTS(text=text, lang="en")
Enter fullscreen mode Exit fullscreen mode

The generated audio is saved temporarily:

tts.save(filename)
Enter fullscreen mode Exit fullscreen mode

Then pygame plays the generated MP3.

After playback finishes, the temporary file is removed.

The file used for temporary speech output is:

jarvis_voice.mp3
Enter fullscreen mode Exit fullscreen mode

🧩 Customization

One of the best things about this project is that you can modify it yourself.

Change Jarvis' Title

In main.py:

USER_TITLE = "Commander"
Enter fullscreen mode Exit fullscreen mode

You can change it to:

USER_TITLE = "Sir"
Enter fullscreen mode Exit fullscreen mode

or:

USER_TITLE = "Bilal"
Enter fullscreen mode Exit fullscreen mode

Add More Songs

Open:

music_library.py
Enter fullscreen mode Exit fullscreen mode

and add your own songs and links.

This allows you to create your own personalized music library.

Change the Gemini Model

The Gemini model is controlled through:

config.json
Enter fullscreen mode Exit fullscreen mode

For example:

{
  "GEMINI_API_KEY": "YOUR_API_KEY",
  "GEMINI_MODEL": "gemini-3.6-flash",
  "NEWS_API_KEY": ""
}
Enter fullscreen mode Exit fullscreen mode

This means you can change the configured model without directly editing the Gemini code in main.py.

Add New Commands

You can extend the command section in main.py.

For example:

elif "open github" in text.lower():
    webpath.open("https://github.com/")
Enter fullscreen mode Exit fullscreen mode

Now Jarvis can understand:

Jarvis
Enter fullscreen mode Exit fullscreen mode 
Open GitHub
Enter fullscreen mode Exit fullscreen mode

and automatically open GitHub.

🔒 Security

The project uses a local configuration file:

config.json
Enter fullscreen mode Exit fullscreen mode

Your real API keys should remain there.

The repository contains:

config.json.example
Enter fullscreen mode Exit fullscreen mode

instead of your actual secrets.

Make sure .gitignore excludes:

config.json
.venv/
__pycache__/
*.pyc
jarvis_voice.mp3
Enter fullscreen mode Exit fullscreen mode

Never publish API keys in your source code.

🖥️ Platform Notes

The current project is primarily designed around a Windows environment.

Some functionality is platform-specific.

For example:

  • Google Chrome path is configured for Windows.
  • WhatsApp launching uses PowerShell.
  • Some desktop automation may need modification on Linux/macOS.
  • Microphone recording requires a working input device.
  • Speech recognition requires internet access.
  • Gemini responses require a valid Gemini API key.
  • News fetching requires a News API key.

The core Python concepts can still be adapted for other platforms.

🧠 What I Learned From This Project

This project has been a practical way for me to learn several concepts that are difficult to understand when studying them separately.

While building Jarvis, I practiced:

  • Python functions
  • Loops and conditionals
  • Error handling
  • Working with external libraries
  • APIs
  • JSON configuration
  • Speech recognition
  • Text-to-speech
  • Audio processing
  • Web automation
  • Windows automation
  • AI API integration
  • Modular Python projects
  • Git and GitHub

More importantly, I learned that a project doesn't have to be perfect before you start building it.

You can start small and keep improving it.

📈 From a Simple Python Project to an AI Assistant

The original idea behind this project was relatively simple:

Listen → Recognize → Execute
Enter fullscreen mode Exit fullscreen mode

As I learned more Python and started working with APIs, the project gradually became:

Listen
   ↓
Speech Recognition
   ↓
Wake Word Detection
   ↓
Command Processing
   ↓
Local Automation OR Gemini AI
   ↓
Text-to-Speech
   ↓
Spoken Response
Enter fullscreen mode Exit fullscreen mode

This progression is probably the most valuable part of the project for me.

It shows how a small beginner project can become a playground for learning new technologies.

🚧 Current Limitations

Jarvis is still a learning project, so there are several things I want to improve.

Some current limitations include:

  • Wake-word detection is based on speech recognition rather than a dedicated wake-word engine.
  • Some commands are hard-coded.
  • Browser paths are currently Windows-specific.
  • WhatsApp automation is Windows-specific.
  • Internet connectivity is required for speech recognition and Gemini responses.
  • The command router can be improved.
  • There are currently no automated tests.
  • The project does not yet have a graphical interface.
  • Pakistani news has a placeholder response in the current release.

These limitations are part of the reason I continue improving the project.

🔮 Future Improvements

There are several improvements I would like to make in future versions:

  • 🧠 Better command routing
  • 🎙️ Dedicated offline wake-word detection
  • 📴 More offline commands
  • 🖥️ GUI interface
  • 🛠️ Better error handling
  • 🧪 Automated testing
  • 🌍 Cross-platform support
  • ⚡ Faster voice interaction
  • 🗂️ Better project architecture
  • 🔌 More automation integrations
  • 🤖 More advanced AI capabilities

The project is still evolving, so more features will likely be added as I continue learning.

📂 GitHub Repository

The complete source code is available on GitHub:

👉 Jarvis AI Virtual Assistant

Feel free to explore the code, fork the project, experiment with it, and build your own version.

💡 Final Thoughts

This project started as a way to practice Python.

It eventually became something much more interesting: a small personal AI assistant that combines voice recognition, automation, APIs, text-to-speech, and Gemini AI.

If you're learning Python, I highly recommend building projects like this.

You don't need to understand everything before starting.

Build something.

Break it.

Fix it.

Add another feature.

Break it again. 😂

That's how you learn.

I'm still improving Jarvis as I continue my journey through Python, AI, and Machine Learning.

Connect With Me

👨‍💻 GitHub:
https://github.com/bilal-dev-0x

💼 LinkedIn:
https://www.linkedin.com/in/bilal-aslam-04a456383/

📝 DEV.to:
https://dev.to/bilal-dev-0x

Thanks for reading! 🚀

If you build your own version of Jarvis, I'd love to see what you add to it.

python #ai #gemini #automation #voiceassistant #beginners #programming #pythonprojects #learninginpublic #opensource

Top comments (2)

Collapse
 
waddimba_abel_5e15678db40 profile image
Waddimba Abel •

Thee GitHub requests the username how can I go over it

Collapse
 
bilal-dev-0x profile image
Bilal Aslam •

The only way to "get past" GitHub when it asks for your username is to provide your actual GitHub username. It's not optional.

If you've forgotten your username:

Check your email — GitHub sends a welcome email when you sign up that includes your username.
Look at your Git configuration locally: run git config --global user.name in your terminal, or git config --global user.email to find your associated email.
Use the "Forgot password" flow at github.com/password_reset — enter your email, and GitHub will reset your account and show your username.
There's no legitimate way to bypass the username field itself, because it's required for authentication. If your issue is that you're entering the wrong username or getting rejected, make sure you're using the exact username (case-sensitive) you registered with, not your email or display name.
OR
You have to create your Github account to access code published on Github.