Talkie Mail: Building a Local AI Email Companion with Gemma 4

This is a submission for the MLH x DEV Writing Challenge.

Email is one of those tools we use almost every day, but managing an inbox can become surprisingly overwhelming. Important information gets buried between newsletters, notifications, conversations, and messages that require action.

We started thinking about a simple question:

What if understanding your inbox felt more like having a conversation with a companion than reading through an endless list of emails?

That question led us to build Talkie Mail, a local-first AI email companion, during Hacktoberfest Hack Day Bhaktapur x Gurkha Technology.

We wanted to explore something beyond simply adding an AI summarization button to an email application. We wanted to understand how an open-weight language model could become part of a practical application, how email content could be handled carefully, and how the experience could feel approachable rather than complicated.

For that, we chose Gemma 4 E2B IT, running locally through llama.cpp.

Here's what we built, how the architecture works, and some of the engineering challenges we encountered along the way.

What I Built

Talkie Mail is an AI-powered email companion that helps users understand their Gmail messages through concise summaries, key points, and actionable items.

The idea is to reduce the effort involved in going through emails without taking away the user's control over their inbox.

At the center of the experience is Buddy, a friendly companion character that guides users through onboarding, confirms Gmail connections, and presents summaries when requested.

We wanted the interface to feel approachable and straightforward. Instead of immediately presenting users with a complicated dashboard, Buddy helps guide them through the experience.

How does it work?

The current application brings together three main parts:

1. Gmail integration: Users authorize access to their inbox through Google's OAuth 2.0 authorization flow. We request read-only permissions, allowing the application to retrieve and display messages without granting permission to send, modify, or delete emails.

2. Local AI processing: When a user requests a summary, the backend retrieves the selected email and sends its content to a locally running Gemma 4 model through llama.cpp.

3. Buddy's interface: The frontend presents the generated result in a structured format, separating the email's main gist, key points, and action items.

This keeps the user in control of which email is summarized while making the information easier to understand.

The architecture

Talkie Mail follows a browser-based frontend and backend architecture. The frontend communicates with a FastAPI application, which coordinates Gmail authorization, message retrieval, summarization, and communication with the local inference server.

Here's a high-level view of how the components work together:

At a high level, the flow is:

1. The user interacts with the Talkie Mail web interface.
2. The FastAPI backend handles requests and manages the application session.
3. Google OAuth 2.0 authorizes read-only access to Gmail.
4. The Gmail service retrieves the selected message and extracts its content.
5. The summarization service prepares the prompt and checks the available token budget.
6. An optional Redis cache is checked for a previously generated summary.
7. If there is no usable cached summary, the backend sends the request to the local llama.cpp server, where Gemma 4 generates the response.
8. The backend validates the result and returns the structured summary to the frontend.

One important security decision was to keep Gmail credentials on the backend. Access and refresh tokens are not exposed to the browser.

The current implementation uses an in-memory store for OAuth credentials and application sessions. This is suitable for our current development scope, but persistent credential storage and production-ready multi-user account isolation remain future work.

Why local AI?

One of our main technical decisions was to use Gemma 4 E2B IT in GGUF format through llama.cpp.

We wanted to explore what a useful AI-powered application could look like without depending on a hosted inference API for summarization.

Running inference locally gave us more control over the model execution environment and allowed us to experiment with an open-weight model on accessible hardware.

There is an important distinction in our local-first approach: Gmail messages are still retrieved through Google's HTTPS API. Once retrieved, the selected email content is sent to the configured llama.cpp endpoint for summarization. When that endpoint is running locally, inference happens on the local machine. So, we do not claim that the entire email retrieval process happens offline.

Making model responses reliable

One of the interesting engineering challenges was that integrating a language model into an application involves much more than sending a prompt and displaying the response.

We needed the model to produce predictable, structured output that the frontend could use.

Our expected summary format contains three fields:

• gist: A concise explanation of the email.
• key_points: The important information from the message.
• action_items: Tasks or actions the user may need to consider.

However, model output is not guaranteed to follow the expected structure every time. A response can be incomplete, malformed, or exceed the available context budget.

We addressed these issues through:

• Token-aware prompt fitting: We use llama.cpp's tokenization endpoint to estimate the size of the input before submitting the inference request.
• Context budgeting: We account for the available context and reserve room for the model's response instead of simply passing the entire email into the prompt.
• Structured output validation: We validate the generated JSON against the expected summary structure before returning it to the frontend.
• Bounded retries: We retry incomplete model output within a limited number of attempts rather than retrying indefinitely.

If the model produces invalid output, the backend reports a failure instead of presenting fabricated information as a successful summary.

This was an important lesson for us: a useful AI feature needs reliable input handling, output validation, and failure handling—not just a capable model.

Privacy and security considerations

Since email content can contain sensitive information, we wanted authorization and data handling to be part of the engineering design rather than an afterthought.

We implemented Gmail OAuth 2.0 with PKCE, server-side state validation, and a read-only Gmail scope:

https://www.googleapis.com/auth/gmail.readonly

This limits the permissions requested from Google.

Some of the security considerations in the current implementation include:

• OAuth authorization state is validated on the backend.
• PKCE is used as part of the authorization flow.
• Gmail credentials are held by the backend rather than exposed to the browser.
• The application uses a session cookie to maintain its application session.
• Email retrieval is restricted to messages carrying Gmail's INBOX label.

We also made sure that optional caching does not bypass Gmail access checks. The backend retrieves and validates the selected message before looking up its summary in Redis.

The current project is still a development-stage application. Its in-memory credential and session storage is not a substitute for a persistent, production-ready multi-user credential system.

Caching with Redis

We also explored using Redis as an optional summary cache.

Generating a summary repeatedly for the same email can involve unnecessary inference work. Redis gives us a way to reuse previously generated results when the relevant inputs have not changed.

The cache is designed around validated summaries. Its keys account for the email content, model and prompt settings, and session scope. Only successful, validated summaries are cached.

Redis is optional: if it is unavailable, the application can fall back to generating a summary through llama.cpp.

This is an optimization rather than a requirement for the core application to work.

What is next?

Talkie Mail is still evolving. The current version provides the core Gmail connection, inbox browsing, email selection, and on-demand summarization experience.

There are several directions we would like to explore next:

• Persistent credential and session storage.
• Stronger account isolation and multi-user support.
• Background email processing and proactive notifications.
• Conversational follow-up questions across multiple emails.
• Native-language interaction.
• Voice interaction with Buddy.

These are future directions, not features we claim are already implemented.

Our longer-term goal is to make email interaction feel more natural, accessible, and less overwhelming.

Demo

GitHub Repository:

Talkie Mail on GitHub

The repository includes the application source code, frontend, backend, configuration examples, and project documentation.

The current application demonstration flow is:

1. Open Talkie Mail and go through Buddy's onboarding experience.
2. Connect a Gmail account using OAuth authorization.
3. Browse the inbox and select a message.
4. Request an AI-generated summary.
5. View the email gist, key points, and action items.

To run the application, you need Google OAuth credentials and a configured llama.cpp server with the Gemma 4 model available. Redis can be configured for optional caching.

Demo video:

Here's a walkthrough of Talkie Mail, demonstrating Gmail integration,
inbox browsing, and AI-powered email summarization.

Partner Technologies

One of the main technologies we explored was Gemma 4, Google's open-weight model family. It was also the technology behind our submission to the Best Use of Gemma 4 prize category.

Gemma 4 E2B IT

We used Gemma 4 E2B IT for email summarization.

Our choice was influenced by our goal of running inference locally rather than depending on a hosted model API.

We used the GGUF version of the model, which allowed us to run it through llama.cpp.

llama.cpp

We used llama.cpp as our local inference runtime.

It exposes an OpenAI-compatible chat-completions endpoint, which our FastAPI backend communicates with.

This separation allowed us to keep model inference independent of the main application logic. We could configure the model server separately and control its context size, thread count, and other inference settings.

One particularly interesting part was working with limited context windows.

We implemented token-aware prompt fitting by using llama.cpp's tokenization endpoint before submitting requests. This helped us keep email content within the configured context budget while reserving space for the model's response.

We also used structured JSON responses and validated the generated output before returning it to the frontend.

This experience taught us that integrating a language model into an application involves much more than sending a prompt and displaying the response. Reliability, input sizing, output validation, and failure handling are equally important.

Other technologies

Alongside Gemma 4 and llama.cpp, we used:

• FastAPI: Backend application and API endpoints.
• Google Gmail API: Read-only inbox access and message retrieval.
• Google OAuth 2.0: Account authorization.
• Vanilla JavaScript, HTML, and CSS: Browser interface and Buddy's interactions.
• Redis: Optional caching of validated summaries to reduce repeated inference work.
• Python: Backend application logic and integration with the inference server.

We have not implemented SQLite or another persistent application database in the current version. Persistent storage remains part of our future development plans.

Hackathon Experience

We built Talkie Mail at Hacktoberfest Hack Day Bhaktapur x Gurkha Technology, an MLH hackathon held on October 15, 2026.

Our team consisted of:

• Sansar Chhetri
• Samartha Magar

The hackathon gave us an opportunity to take an idea and turn it into an actual application within a limited development window.

One of the most interesting parts of working on Talkie Mail was bringing together different areas of development: backend engineering, OAuth integration, local AI inference, and frontend experience design.

We also had to think beyond simply getting a language model to generate text. We needed to consider how users would connect their inbox, how email content would reach the model, how much context the model could process, and how the results would be presented in a useful way.

Working with Gemma 4 also gave us an opportunity to explore the practical side of running an open-weight model locally. We learned that the surrounding application architecture is just as important as the model itself.

We submitted Talkie Mail under the Best Use of Gemma 4 prize category.

The project is still evolving, but the hackathon gave us a starting point to explore local AI applications and build something we would like to continue improving beyond the event itself.

A big thank you to Major League Hacking, Gurkha Technology, and everyone involved in organizing the event.

We would also like to acknowledge Aziza Guluzade and Aziz Haziyev for their Talkie Language Learning App UI/UX case study, which served as a visual reference during our design exploration.

---

Explore the project: Talkie Mail on GitHub

If you're interested in local AI, open-weight models, or building applications with Gemma, we'd love to hear your thoughts!

#mlhacks #devchallenge #hackathon