How I Used an LLM to Explain Problems From Photos

How I Used an LLM to Explain Problems From Photos

The first version of the idea sounded almost too simple: take a photo of a problem, send it to an LLM, and return an explanation.

After working on it for a bit, I realized the useful part is not the model call itself. The useful part is everything that happens around the model call: turning a photo into a structured problem, choosing the right reasoning style, checking the answer, and making the explanation readable.

That is the part I wanted to explore in a small photo-based study app.

👉 Download Now from the App Store: https://apps.apple.com/us/app/ai-snapsolve-homework-solver/id6763911277

App Store Search: AI SnapSolve

Why A Photo Is A Harder Prompt

With a normal chat interface, the user gives the model a clean prompt.

With a camera-first workflow, the prompt has to be built. The input might include printed text, handwriting, a diagram, a table, multiple questions, or irrelevant content around the edges.

If the image is converted into rough text and sent directly to an LLM, the output can look confident while missing important context. That is not good enough for a learning tool.

So the first step is not solving. The first step is understanding the photo.

The system needs to preserve:

• the actual question
• math notation
• labels and units
• diagram context
• multi-part ordering
• uncertainty around unclear text

That structured representation becomes the real prompt.

Extract First, Explain Later

One lesson I kept running into was that extraction and explanation should be treated as different stages.

The extraction step asks: "What is on the page?"

The explanation step asks: "How should this be solved or understood?"

Combining those too early can cause problems. If the model starts explaining before the problem is clearly extracted, it may fill in gaps with assumptions. Sometimes that works. Sometimes it gives a polished answer to the wrong question.

Separating the stages makes the workflow easier to reason about:

• read the image
• identify the relevant problem
• check whether the input is clear enough
• route the problem by subject
• ask the LLM for a targeted explanation

That is a small architecture choice, but it makes the product more reliable.

Routing Makes The Explanation Better

Not every problem should be explained the same way.

An algebra equation needs a different explanation style than a geometry diagram. A physics word problem may need a list of known values before any equation appears. A chemistry question may depend on balancing, units, or reaction structure.

So I added a lightweight routing step. Before asking for a full solution, the system tries to identify:

• the subject
• the likely concept
• whether a diagram matters
• whether the problem spans multiple photos
• whether the answer should be numeric, symbolic, or verbal

This gives the LLM a better frame. Instead of asking for a generic answer, the system can ask for the kind of explanation that fits the problem.

The Output Should Teach The Next Step

The most tempting thing to generate is a final answer.

But for studying, the final answer is not enough. If a student is stuck, they often need to see the first move and why that move is allowed.

The explanation format I found most useful was:

• identify the concept
• show the first step
• explain why that step makes sense
• walk through the intermediate reasoning
• give the final result
• mention a common mistake when relevant

This keeps the LLM focused on teaching the path, not just producing the destination.

Multiple Solvers As A Check

Another useful pattern was comparing more than one solution path.

This does not mean showing every raw model response to the user. That can become noisy quickly. The better approach is to use multiple paths as a check.

If two explanations reach the same result, that increases confidence. If they disagree, the system can inspect the extraction, assumptions, or reasoning before presenting a polished answer.

For students, alternate paths can also be useful. One algebra solution might use factoring. Another might use a formula. Seeing both can make the concept feel less like a memorized trick.

Multi-Photo Context Matters

Real homework often does not fit into one image.

A student might capture a diagram, then a question, then a second page with follow-up parts. If each image is solved separately, the LLM loses context.

The workflow works better when multiple photos are merged into a single problem context before explanation. That way variables stay consistent, earlier givens remain available, and later parts are not treated as isolated tasks.

This is one of those features that sounds like convenience but is actually part of reasoning quality.

Where LLMs Still Need Guardrails

The main risk is confidence.

LLMs can produce fluent explanations even when the input is incomplete or ambiguous. In a study app, that means the system needs to be honest about uncertainty.

Some guardrails that help:

• flag unclear OCR
• ask for a better photo when needed
• show assumptions before solving
• compare independent solution paths
• avoid over-explaining simple problems

The goal is not to make mistakes impossible. The goal is to make mistakes easier to notice and recover from.

Closing Thought

Using an LLM to explain problems from photos taught me that the model is not the whole workflow.

The hard part is designing the path from messy visual input to a clear explanation. That path includes extraction, routing, reasoning, checking, and presentation.

When those pieces work together, the app feels less like "AI gives an answer" and more like "AI helps me understand what is happening."