Turning a Problem Photo Into a Step-by-Step Agent Workflow

Turning a Problem Photo Into a Step-by-Step Agent Workflow

I have been thinking about problem-solving apps less as a single model call and more as a small agent workflow.

The user action is simple: take a photo of a problem.

But the useful system behavior is not simple. The app has to read the image, decide what kind of problem it is, choose a reasoning strategy, check the answer, and explain the result in a way the user can follow.

That is the part I wanted to explore while building a photo-based AI study tool.

👉 Download Now from the App Store: https://apps.apple.com/us/app/ai-snapsolve-homework-solver/id6763911277
App Store Search: AI SnapSolve

Why Use An Agent Workflow?

For a small study app, "agent" does not need to mean a complicated autonomous system.

It can mean a series of focused steps where each step has a job, a boundary, and an output that the next step can use.

The Workflow Shape

The rough flow looks like this:

photo input
  -> extraction agent
  -> task planner
  -> solver
  -> checker
  -> explainer
  -> user-facing answer

This is still a compact workflow. The difference is that the system does not pretend the model can jump directly from pixels to a trustworthy explanation every time.

Each stage reduces a different kind of uncertainty.

Step 1: Extraction Agent

The first stage is about observation.

A problem photo may contain printed text, handwriting, answer choices, diagrams, labels, notes in the margin, or multiple questions on the same page. The extraction agent should not immediately solve anything. Its job is to answer:

• what problem is visible?
• what text and notation were detected?
• are answer choices present?
• is there a diagram or table?
• does any part look cropped or unclear?

For a clean algebra question, the extracted state might be:

subject_hint: algebra
problem_text: 3(x - 4) = 18
answer_choices: none
visual_context: none
uncertain_parts: []

For a geometry question, the state may need diagram labels. For a multi-image upload, it may need page order and references between parts.

This stage matters because a wrong extraction can produce a confident but irrelevant solution.

Step 2: Task Planner

After extraction, the workflow needs a planner.

The planner decides what kind of reasoning is appropriate. A linear equation does not need the same output as a geometry proof. A physics word problem needs units and known values. A reading question needs evidence tracking rather than algebraic steps.

The planner can stay lightweight:

task_type: solve_equation
expected_output: value
reasoning_style: step_by_step
verification_needed: simple substitution

The point is not to make planning elaborate. The point is to stop treating every input as the same generic prompt.

Step 3: Solver

The solver is where the visible reasoning happens.

For the algebra example:

3(x - 4) = 18

One path expands:

3x - 12 = 18
3x = 30
x = 10

Another path divides first:

x - 4 = 6
x = 10

Both are correct. The second path is shorter. A solver can return both when comparison is useful, or choose the simpler path when the goal is a concise explanation.

This is where multiple solving engines can help, but only if their output is organized. Three unstructured answers are not better than one clear one.

Step 4: Checker

The checker is the part I think many AI workflows need more often.

For a math problem, the checker might substitute the result back into the original equation:

3(10 - 4) = 18
3(6) = 18
18 = 18

For a multiple-choice question, it might verify that the final answer maps to the correct label. For a word problem, it might check whether units match. For a photo input, it might check whether the solution depends on text that was uncertain or cropped.

The checker does not have to be perfect. It just needs to create a moment of resistance before the app presents the answer.

Step 5: Explainer

The explainer turns the internal reasoning into something readable.

This is not just formatting. The same solution can be presented in ways that either help or overwhelm the user.

A useful explanation should include:

• what the app understood from the photo
• the selected method
• the steps that matter
• the final answer
• a short note about the reusable idea

For the example above, the reusable note is:

When a grouped expression is multiplied by a number, check whether dividing first makes the equation simpler.

That one sentence can be more valuable than the answer itself.

Handling Multi-Image Context

Some inputs do not fit in one photo.

A student might upload a problem statement, a diagram, and a follow-up question. If each image is solved independently, the workflow loses context.

In an agent workflow, multi-image upload changes the extraction stage. The system needs to merge images into one task state before planning:

image_1: passage or problem statement
image_2: diagram or answer choices
image_3: follow-up question
combined_context: required

Only after that merge should the planner decide what to do. Otherwise the solver may answer only the most recent image and ignore the information that makes the question solvable.

Where The Workflow Can Fail

The failure modes are practical:

• the extraction agent misses a minus sign
• the planner picks the wrong subject
• the solver uses an unnecessarily long method
• the checker verifies the wrong expression
• the explainer hides an important assumption

This is why each stage should leave traces that can be inspected. If the final answer seems wrong, it should be possible to see whether the problem came from input reading, planning, solving, checking, or presentation.

That kind of observability is useful even in a small consumer app.

Keeping The Product Grounded

The temptation with agent workflows is to make them sound more magical than they are.

For this use case, I prefer a grounded goal:

• reduce input friction
• preserve enough context
• choose a reasonable method
• check obvious mistakes
• explain the result clearly

That is not a grand promise. It is a useful loop.

The app should not pretend every photo is perfect. It should not treat every answer as certain. It should make the path from problem photo to explanation easier to inspect.

What I Would Improve Next

The next useful improvements are not flashy:

• better cropped-image detection
• clearer display of extracted problem text
• tighter checker logic for answer choices and units
• more compact explanations for easy problems
• cleaner comparison when multiple solution paths disagree

These details matter because agent workflows are only helpful when their intermediate steps improve the final user experience.

Final Thought

Turning a problem photo into an answer is the visible feature.

Turning that photo into a step-by-step agent workflow is the product work underneath it.

The agent does not need to be dramatic. It needs to observe carefully, plan just enough, solve with context, check itself, and explain the result in a way the user can reuse.