NextCueAI: Local AI That Explains a Confusing Screen and Tells You What to Do Next

Ishara Lakshan — Mon, 05 Oct 2026 04:24:04 +0000

This is a submission for the Hacktoberfest Weekend Challenge: Build for a Friend

NextCueAI: See the problem. Know what to do next.

What I built

I built NextCueAI, a local-first AI assistant for people who get stuck on confusing screens.

You can drop, paste, or choose a screenshot from an app, website, settings page, error dialog, or form. NextCueAI looks at the screenshot and explains:

what is on the screen
what may be wrong
what to do next
clear step-by-step actions
any uncertainty or caution that matters

It supports both English and Sinhala, with Simple and Detailed explanation modes.

The project is open source and available here:

GitHub:

https://github.com/ish4ra/NextCueAI

Windows MSI:

https://github.com/ish4ra/NextCueAI/releases/tag/v1.0.1

Who I built it for

I built NextCueAI for my friend Iruka Akash, who is 26 and works at Galadari Hotel in Colombo.

Like many people who are comfortable using computers but do not want to spend time decoding every unfamiliar screen, he can occasionally run into confusing settings pages, error dialogs, installers, or forms where the difficult part is simply understanding what the screen is asking and what the safest next step is.

That gave me the idea for NextCueAI: a tool that can look at a screenshot and explain the situation in plain language, almost like having a technically minded friend looking over your shoulder.

I also wanted it to support Sinhala, because sometimes a clear explanation in your own language is much more useful than a technical answer in English.

Demo

Here is the demo:

https://youtu.be/rEi1WpOVcB8

In the demo, I open NextCueAI, add a screenshot, run local screenshot analysis, and get an explanation with practical next steps.

The current Windows release is packaged as an MSI, so Node.js is not required after installation.

For v1.0.1, Ollama and the local Gemma model are still installed separately.

Why I wanted the AI to run locally

A screenshot can contain much more private information than people realize.

It may include:

account names
email addresses
personal messages
file names
system information
browser tabs
settings
error logs
parts of forms or documents

That made local inference an important part of the design rather than just a technical preference.

By default, NextCueAI sends the screenshot only to a local bridge running on 127.0.0.1, which passes it to a locally running Ollama model.

There is:

no hosted AI fallback
no account system
no analytics
no database
no screenshot history
no cloud storage

The app uses an open-weight Gemma vision model through Ollama.

Open AI is what makes the core privacy model of this project possible.

How it works

The frontend is built with:

React
TypeScript
Vite

A lightweight local Express bridge handles:

image validation
image normalization
MIME verification
metadata removal
local Ollama communication
structured AI response validation

The Windows desktop build uses Electron and is distributed as an MSI installer.

The AI response is structured into fields such as:

what you are seeing
what may be happening
the next action
step-by-step guidance
caution
confidence

I used structured output rather than displaying arbitrary raw model responses.

The app also checks the selected Ollama model before analysis to make sure it supports image input.

Screenshot handling and safety

NextCueAI currently accepts:

PNG
JPEG
WebP

It limits screenshots to 10 MB and rejects unsuitable inputs such as animated images.

The local bridge checks the real image format, validates dimensions, removes metadata, handles image orientation, and normalizes screenshots before inference.

Model output is rendered as text rather than executable HTML.

Instructions visible inside screenshots are also treated as untrusted content by the analysis prompt.

English and Sinhala

One of the features I cared about most was Sinhala support.

It was not enough to simply add a language selector.

During testing, I found cases where Sinhala requests could still return English, and some systems did not render Sinhala reliably.

I tightened the language guidance and bundled a local Sinhala font so the UI does not depend on a remote font service.

That also fits the local-first design.

Making it usable

I did not want NextCueAI to look like another generic chatbot.

The interface is centered around the screenshot itself.

Users can:

drag and drop an image
paste from the clipboard
use a file picker
preview the screenshot
replace or remove it
select English or Sinhala
select Simple or Detailed guidance
cancel an analysis
retry
copy the response
start over

The application also handles cases such as:

Ollama offline
model missing
unsupported model
timeout
malformed model response
invalid image input

Testing and release

The repository includes:

unit tests
API tests
browser end-to-end tests
responsive layout checks
accessibility checks
GitHub Actions CI

The current release is:

NextCueAI v1.0.1

Windows users can install:

NextCueAI-1.0.1-Windows-x64.msi

Release:

https://github.com/ish4ra/NextCueAI/releases/tag/v1.0.1

What I learned

The hardest part was not simply making a vision model describe a screenshot.

The real challenge was turning that description into useful action.

A response such as “this is a settings screen” does not help much when someone is stuck.

The response needs to answer:

What am I seeing?

What is probably happening?

What should I do next?

I also learned that local AI changes the product design.

You have to think about:

model availability
first-load delay
memory usage
model capability detection
timeouts
malformed output
offline behavior
setup experience

Those constraints made the project feel much more like a real application than a thin AI API wrapper.

What my friend thought

I built the project around a real problem I wanted to make easier for Iruka.

The current release is complete and usable, and my next step is to have him test it with more of the kinds of confusing screens he actually encounters and use that feedback to improve the onboarding and explanations.

What's next

The current v1.0.1 release works, but Windows users still need Ollama and the Gemma model installed separately.

The next release is focused on making setup easier by:

detecting Ollama automatically
guiding installation when it is missing
detecting the local model
downloading the model from inside the app
showing setup progress
recovering cleanly from interrupted setup

I chose not to delay the working challenge release for that improvement.

Why open innovation mattered here

For NextCueAI, open AI is not just a challenge requirement.

It is what makes the main privacy goal possible.

The user can keep inference local, choose a compatible model, inspect the application code, and avoid sending screenshots to a remote AI provider.

That makes the project more useful for exactly the kind of person I built it for: someone who needs help understanding a screen, but should not have to give up control of that screen in order to get help.

DEV Community: Ishara Lakshan