A product image contains a surprising amount of information.
A human can look at an image and often recognize the brand, product category, model, and several important characteristics within a few seconds. For an e-commerce system, however, that same image is just an image file.
Turning that visual information into structured product data requires multiple steps: extracting text, understanding what the product is, finding additional context, and finally producing information that can actually be stored and used by another system.
That was the problem I wanted to solve with an AI-powered product information agent.
The Problem
E-commerce and inventory systems depend heavily on structured product information.
A typical product record might contain:
Brand
Product name
Model
Category
Product descriptors
Product summary
Additional contextual information
But when the input is a product image, most of that information isn't directly available as structured data.
OCR can extract text, but text extraction alone doesn't tell you what the product is.
Search can find relevant information, but search results can also contain unrelated pages, duplicate information, advertisements, and conflicting descriptions.
An LLM can generate a description, but asking an LLM to work directly from an image without sufficient context can result in incomplete or unsupported information.
So instead of trying to make one model do everything, I approached the problem as a pipeline.
The Goal
The goal was to build a system that could take a product image as input and produce structured, useful product information as output.
The basic flow was:
Product Image
↓
Google Vision
↓
Extracted Text / Visual Signals
↓
Gemini
↓
Product Understanding
↓
Google Search
↓
Additional Context
↓
Crawl4AI
↓
Relevant Page Content
↓
GPT-4
↓
Structured Product Information
The important part of this architecture is that every component has a specific responsibility.
Why Multiple APIs?
One of the biggest design decisions was not trying to solve the entire problem with a single AI model.
Google Vision — extracting visible information
The first step was extracting information directly from the image.
Google Vision was useful for OCR and identifying visible signals from the product image.
For example, an image might contain:
SONY
WH-1000XM5
Wireless Headphones
The OCR layer gives the downstream system something concrete to work with.
However, OCR output isn't necessarily clean.
Text can be duplicated, characters can be misread, and information can appear in an unexpected order.
That meant OCR output had to be treated as an input signal rather than as the final answer.
Gemini — understanding the product
The next step was understanding what those extracted signals represented.
Gemini helped interpret the information and identify the likely product, category, and relevant descriptors.
This distinction is important.
There is a difference between:
"WH-1000XM5"
and:
Sony WH-1000XM5 wireless noise-cancelling headphones
The first is extracted text.
The second contains meaning.
*Google Search — enriching the context
*
Once the system had a likely product identity, search could be used to find additional information.
This step was necessary because the original image rarely contains everything required for a useful product record.
Search could help locate product pages, manufacturer information, reviews, specifications, and other relevant context.
But search introduces another problem: not every result is useful.
A search query can return:
Duplicate pages
Unrelated products
Marketplace listings
SEO-generated pages
Partial information
Conflicting descriptions
So search results needed another layer of processing.
Crawl4AI — getting the actual content
Instead of relying only on search snippets, Crawl4AI was used to extract relevant content from pages.
This allowed the pipeline to work with the actual page content rather than treating a search-result snippet as authoritative information.
The crawler therefore acted as the bridge between:
Search result
↓
Relevant webpage
↓
Usable content
GPT-4 — producing the final structured output
The final stage was turning all of these inputs into a clean product record.
The model could combine:
OCR output
Product identification
Search context
Crawled content
and produce a structured result.
For example:
{
"brand": "Sony",
"model": "WH-1000XM5",
"category": "Wireless Headphones",
"descriptors": [
"over-ear",
"noise cancelling",
"wireless"
],
"summary": "Sony WH-1000XM5 wireless over-ear headphones..."
}
The important point here is that the LLM wasn't expected to discover everything from scratch.
It was the final reasoning and formatting layer of a larger information pipeline.
Challenges
The interesting part of the project wasn't making the happy path work.
It was handling everything that didn't.
OCR isn't perfect
Images don't always contain clean, readable text.
Different fonts, angles, lighting, reflections, and image quality can all affect OCR results.
The downstream system therefore needed to tolerate imperfect input.
Search results are noisy
A product name doesn't guarantee that every search result refers to the same product.
This became a problem when multiple versions or similar products existed.
Search therefore had to be treated as an enrichment mechanism rather than unquestioned ground truth.
Duplicate information
The same product information can appear across multiple pages.
Feeding all of that content into the final model increases noise without necessarily improving the answer.
This made content selection and processing important.
Prompt design
The final model needs to understand exactly what information should be returned.
If the output format isn't constrained, the result can vary significantly between requests.
Structured prompting therefore became an important part of the pipeline.
Maintaining reliable output
An AI system is different from a traditional deterministic API.
You can't assume that every response will have exactly the same wording or structure.
For an application that eventually needs to store the result in a database, that is a serious engineering concern.
The system therefore needs validation around the model output instead of blindly trusting it.
Why the Architecture Matters
The biggest lesson from this project was the value of separating responsibilities.
Instead of creating one large process:
Image → AI → Answer
the architecture became:
Image
↓
Vision
↓
Product understanding
↓
Search
↓
Web extraction
↓
LLM processing
↓
Structured output
That makes debugging significantly easier.
If something goes wrong, you can ask:
Was the OCR incorrect?
Was the product identified incorrectly?
Did search return poor results?
Did the crawler extract the wrong content?
Did the final prompt interpret the information incorrectly?
Each stage becomes independently inspectable.
What I Would Improve Next
There are several directions I would take this system next.
First, I would add stronger validation between pipeline stages.
Second, I would improve product matching so that similar products and different product variants are handled more reliably.
Third, I would introduce better deduplication and source ranking for crawled content.
Finally, I would make the output schema stricter so that the generated information can be consumed directly by downstream inventory or catalog systems.
Final Thoughts
This project changed the way I think about AI applications.
The interesting part isn't simply connecting an LLM to an application.
The real engineering challenge is building the system around the model.
Vision, search, crawling, retrieval, prompting, validation, and structured output all have different responsibilities.
That architecture can be useful beyond this particular project.
The same approach could support:
Product catalog automation
Inventory onboarding
E-commerce data enrichment
Internal product databases
Automated product listing workflows
For me, the biggest takeaway was simple:
A useful AI system is usually a pipeline of smaller, well-defined components rather than one giant AI call.
Top comments (0)