DEV Community: Meet Dalsania

How I Built OmniScrape: A Modular RAG Engine with FastAPI and Qdrant

Meet Dalsania — Fri, 21 Aug 2026 09:06:22 +0000

Large websites contain an enormous amount of information.

The problem isn't always finding information.

The problem is asking the right question against that information efficiently.

If you crawl a website and simply store the raw HTML, you haven't really built a useful knowledge system.

You still need to:

Clean the content
Remove unnecessary information
Break the content into meaningful chunks
Generate embeddings
Store those embeddings
Retrieve relevant information
Generate an answer from the retrieved context
Show users where that information came from

That was the motivation behind OmniScrape, a modular RAG engine I built using FastAPI and Qdrant.

The Problem

Imagine a website containing hundreds or thousands of pages.

A user might ask:

"What does this documentation say about authentication?"

Traditional keyword search can work well for simple queries.

But what happens when the user doesn't use the exact terminology from the website?

For example, the documentation might say:

"Requests must include a valid bearer token."

while the user asks:

"How do I authenticate my API requests?"

The words are different, but the meaning is related.

This is where Retrieval-Augmented Generation (RAG) becomes useful.

Instead of asking an LLM to answer entirely from its general knowledge, the system first retrieves relevant information from the target knowledge base and then gives that information to the model.

The basic idea is:

Question
   ↓
Retrieve relevant content
   ↓
Give retrieved content to LLM
   ↓
Generate answer

OmniScrape was designed around this architecture.

The Goal

The goal was to build a system that could take website content and turn it into a searchable knowledge base.

The pipeline needed to:

Crawl websites
Extract useful content
Clean the content
Split it into chunks
Generate embeddings
Store embeddings and metadata
Retrieve relevant information
Generate an answer
Return citations to the original sources

The overall architecture looked roughly like this:

                 ┌──────────────┐
                 │    Website   │
                 └──────┬───────┘
                        ↓
                 ┌──────────────┐
                 │   Crawler    │
                 └──────┬───────┘
                        ↓
                 ┌──────────────┐
                 │Content Clean │
                 └──────┬───────┘
                        ↓
                 ┌──────────────┐
                 │   Chunking   │
                 └──────┬───────┘
                        ↓
                 ┌──────────────┐
                 │  Embeddings  │
                 └──────┬───────┘
                        ↓
                 ┌──────────────┐
                 │    Qdrant    │
                 └──────┬───────┘
                        ↓
                    Retrieval
                        ↓
                 ┌──────────────┐
                 │ Answer Model │
                 └──────┬───────┘
                        ↓
                    Answer + Sources

The important part is that this isn't one large AI operation.

Each stage has a specific responsibility.

Why Qdrant?

A vector database is one of the central components of a RAG system.

I chose Qdrant because it provides a dedicated vector search layer and allows embeddings to be stored together with useful metadata.

That metadata becomes particularly important when building source-aware retrieval.

A chunk shouldn't just be:

Some paragraph from a website...

It should also carry information about where that paragraph came from.

Conceptually, a stored record could look like:

{
  "text": "Authentication can be configured...",
  "source": "https://example.com/docs/authentication",
  "title": "Authentication",
  "chunk_id": "auth-004"
}

Now retrieval can return both the content and its provenance.

That becomes important later when generating citations.

Why FastAPI?

FastAPI was a natural fit for the backend because I wanted a clean API layer between the application and the RAG pipeline.

Instead of coupling the frontend directly to crawling, embedding, retrieval, and generation logic, the backend exposes clear endpoints.

Conceptually:

POST /crawl
POST /ingest
POST /query
GET  /sources

The frontend doesn't need to know how the RAG engine works internally.

It only needs to communicate with the API.

This separation also makes the backend easier to test and extend.

The Ingestion Pipeline

Before a user can ask questions, the website needs to be converted into useful information.

This is where most of the RAG pipeline actually happens.

1. Crawl

The first step is retrieving pages from the target website.

But raw HTML isn't suitable for embedding.

A webpage can contain:

Navigation menus
Scripts
Footers
Advertisements
Sidebars
Repeated content
Other elements unrelated to the actual information

So crawling is only the beginning.

2. Clean

The extracted content needs to be converted into meaningful text.

The objective is to preserve useful information while removing unnecessary page noise.

For example, instead of storing a page like:

Home
Products
Pricing
Documentation
Login

Authentication

Authentication allows applications...

Privacy
Terms
Copyright

the system should focus primarily on the useful content:

Authentication

Authentication allows applications...

This matters because irrelevant content can eventually affect retrieval quality.

3. Chunk

Long documents cannot simply be treated as one giant piece of information.

The content needs to be divided into smaller sections.

For example:

Documentation
├── Introduction
├── Authentication
├── Configuration
├── API Reference
└── Troubleshooting

Chunking looks simple, but it has a major effect on retrieval.

If chunks are too large, retrieval can return a lot of irrelevant information.

If chunks are too small, important context can be separated across multiple chunks.

So chunking becomes a tradeoff rather than a fixed rule.

4. Embed

Each chunk is converted into a vector representation.

The embedding captures semantic information about the content.

This allows the system to search based on meaning rather than requiring an exact keyword match.

For example:

User query:
"How do I authenticate API requests?"

        ↓

Semantic representation

        ↓

Vector search

        ↓

Relevant documentation

The user's wording doesn't need to exactly match the wording in the original documentation.

5. Store

The vectors and associated metadata are stored in Qdrant.

At this point, the website has effectively been transformed into a searchable semantic knowledge base.

Instead of searching thousands of raw pages every time a user asks a question, the system can search the vector representation of the processed content.

Retrieval

When a user asks a question, the query goes through a similar embedding process.

The system searches Qdrant for chunks that are semantically close to the query.

Conceptually:

User:
"How do I authenticate API requests?"

        ↓

Query embedding

        ↓

Qdrant similarity search

        ↓

Relevant documentation chunks

        ↓

LLM

The retrieved chunks become the context for answer generation.

This is the core of RAG.

But it also introduces one of the most important problems in RAG systems:

What happens if the system retrieves the wrong information?

Why Retrieval Quality Matters

A RAG system can generate a fluent answer and still be wrong.

The problem may not be the LLM.

The retrieval layer may simply have returned poor context.

For example:

Question
   ↓
Retriever
   ↓
┌──────────────────────────┐
│ Unrelated chunk          │
│ Navigation content       │
│ Old documentation        │
│ Relevant chunk           │
└──────────────────────────┘
   ↓
LLM

The model now has to generate an answer from a noisy context.

This is one of the most important lessons I learned while building OmniScrape:

Better generation cannot compensate for consistently bad retrieval.

This is why ingestion, chunking, metadata, and retrieval quality deserve as much attention as the final LLM.

Why Citations Matter

One of the features I considered particularly important was source awareness.

A generated answer without a source can be difficult to trust.

For example:

"API authentication requires a bearer token."

That might sound reasonable.

But where did it come from?

A better system can point the user back to the relevant source.

Instead of:

AI says something.

the experience becomes:

AI says something.

Source:
Authentication Documentation

The user can verify the information.

This is particularly useful for:

Technical documentation
Internal knowledge bases
Research
Support systems
Company documentation

The goal isn't just to generate an answer.

The goal is to make the answer traceable.

Engineering Challenges

Building the pipeline also exposed several engineering challenges.

Chunk Size

There is no universal perfect chunk size.

Documentation, blog posts, technical specifications, and long-form articles can have very different structures.

The chunking strategy therefore needs to match the type of information being processed.

Metadata

Metadata initially looks like an implementation detail.

It isn't.

Without useful metadata, it becomes much harder to understand where retrieved content came from.

Useful metadata can include:

Source URL
Page title
Section
Chunk ID
Other document-level information

This information becomes valuable when debugging retrieval and generating citations.

Retrieval Quality

Retrieval is one of the most important parts of the entire system.

If the wrong chunks are returned, the final answer can be wrong even if the LLM is working exactly as expected.

This means that evaluating a RAG system shouldn't stop at:

"Does the final answer sound good?"

A better question is:

"Did the system retrieve the right information to answer the question?"

Response Formatting

Even when the correct chunks are retrieved, the final response needs to be presented clearly.

The system needs to balance:

Answer quality
Source visibility
Context length
Response speed
Readability

Too little context can produce weak answers.

Too much context can introduce noise.

Too many citations can make the response difficult to read.

The final response therefore needs its own design considerations.

Why Modular Design?

I intentionally separated the system into components.

Conceptually:

crawler/
├── crawler.py

ingestion/
├── cleaner.py
├── chunker.py

embeddings/
├── embedding_service.py

retrieval/
├── retriever.py

generation/
├── generator.py

api/
├── routes.py

frontend/

The exact implementation can change, but the principle stays the same:

Each component should have a clear responsibility.

This makes the system easier to evolve.

For example:

Changing the embedding model shouldn't require rewriting the crawler.

Changing the frontend shouldn't require changing vector storage.

Improving retrieval shouldn't require rebuilding the entire API.

This is one of the biggest advantages of modular AI engineering.

What OmniScrape Can Become

The architecture isn't limited to one specific website.

A system like OmniScrape can provide the foundation for:

Internal knowledge search
Documentation assistants
Customer support tools
Research assistants
Technical knowledge bases
Website question-answering systems

The core idea remains the same:

Unstructured Information
        ↓
Structured Knowledge
        ↓
Semantic Retrieval
        ↓
Source-Aware Answer

That makes the architecture reusable across different applications.

What I Learned From Building It

The biggest lesson from OmniScrape was that RAG isn't simply:

Vector Database + LLM

There is considerably more engineering involved.

Crawling quality affects ingestion.

Ingestion affects chunking.

Chunking affects embeddings.

Embeddings affect retrieval.

Retrieval affects the context given to the model.

And the context affects the final answer.

In other words:

A RAG system is a chain. If one part of the chain is weak, the final answer suffers.

That is what made this project particularly valuable to me.

I wasn't just building a chatbot.

I was learning how to build an information pipeline around an AI model.

Final Thoughts

Building OmniScrape changed the way I think about RAG applications.

The LLM is only one component.

The real system includes:

Crawling
   ↓
Content Processing
   ↓
Chunking
   ↓
Embeddings
   ↓
Vector Storage
   ↓
Retrieval
   ↓
Generation
   ↓
Sources

Every stage contributes to the final result.

And when something goes wrong, the solution isn't always to change the model.

Sometimes the problem is the data.

Sometimes it's the chunking.

Sometimes it's retrieval.

Sometimes it's the way the final context is constructed.

That's the part of RAG engineering I found most interesting:

Building a system where the model doesn't have to know everything—it just needs to receive the right information at the right time.

Lessons I Learned While Building Real-World AI and Automation Projects

Meet Dalsania — Fri, 21 Aug 2026 09:04:25 +0000

When I started building AI projects, I was mainly focused on one question:

Can I make it work?

After working on projects involving AI APIs, web scraping, RAG pipelines, backend integrations, and automation, I realized that this is only the first question.

A demo can work once.

A real system needs to keep working when the input changes, an API fails, a website changes its structure, or an AI model returns something unexpected.

That shift in thinking has probably been one of the most valuable things I've learned while building real-world AI and automation projects.

In this post, I want to share some of the lessons that changed how I approach projects.

1. Building a Demo Is Not the Same as Building a System

A simple AI demo often looks like this:

Input
  ↓
AI Model
  ↓
Output

And when the output looks correct, it feels like the project is finished.

But real applications are rarely that simple.

A more realistic AI pipeline looks something like:

User Input
    ↓
Validation
    ↓
Processing
    ↓
External API
    ↓
Response Validation
    ↓
Additional Processing
    ↓
Final Output

There are multiple places where something can go wrong.

For example:

The input may be incomplete.
An API may return an error.
OCR may extract incorrect text.
A website may not contain the expected information.
Search results may be irrelevant.
An LLM may return an unexpected format.
A downstream component may fail because the previous component returned bad data.

This made me realize that the happy path is only one part of the application.

2. External APIs Can Fail

Many AI applications depend on several external services.

In one of my projects, the pipeline involved multiple components for processing product information:

Product Image
     ↓
Google Vision
     ↓
Gemini
     ↓
Google Search
     ↓
Crawl4AI
     ↓
GPT-4
     ↓
Structured Output

Each component has a specific responsibility.

But every external dependency also creates another potential failure point.

An API can:

Return an error
Return incomplete information
Take longer than expected
Change its response
Hit a rate limit
Return something different from what the application expects

This taught me not to treat APIs as if they were guaranteed to work.

Instead, every external service should be considered a component that can fail.

That means thinking about:

Request
  ↓
Did it succeed?
  ↓
Is the response valid?
  ↓
Can the next component use it?

This sounds simple, but it becomes extremely important when multiple APIs are connected together.

3. AI Output Needs Validation

One of the biggest differences between traditional software and AI-powered software is that AI output isn't always perfectly predictable.

Suppose an application expects:

{
  "brand": "Nike",
  "model": "Air Max",
  "category": "Shoes"
}

It is tempting to simply send the model's response to the next stage.

But what happens if the response looks like this?

{
  "brand": "Nike",
  "product": "Air Max"
}

Or even:

The product appears to be a Nike Air Max shoe.

For a human, both responses may be understandable.

For another program, they may cause problems.

This is why I started thinking about AI responses as data that needs validation, not as guaranteed application output.

A useful pipeline is:

LLM Response
     ↓
Parse
     ↓
Validate
     ↓
Normalize
     ↓
Use in Application

This becomes especially important when AI output is being stored in a database or passed to another API.

4. Modular Architecture Makes Debugging Easier

One of the most important lessons I've learned is to avoid putting an entire workflow into one large script.

For example, a RAG system can contain:

Crawler
   ↓
Content Cleaning
   ↓
Chunking
   ↓
Embeddings
   ↓
Vector Database
   ↓
Retrieval
   ↓
LLM
   ↓
Final Response

If all of this is implemented inside one large function, debugging quickly becomes painful.

Instead, separating the components makes it easier to understand where a problem is happening.

For example:

crawler/
    crawler.py

processing/
    cleaner.py
    chunker.py

retrieval/
    embeddings.py
    retriever.py

generation/
    generator.py

api/
    routes.py

Now, if retrieval quality is poor, I can investigate the retrieval layer.

If the content being retrieved is wrong, I can investigate ingestion.

If the retrieved information is correct but the final response is poor, I can investigate the generation layer.

That separation saves time.

5. Better Models Don't Fix Bad Data

This was one of the most important lessons I learned while working with RAG and AI pipelines.

It is easy to think:

"If I use a better LLM, the answer will become better."

Sometimes it will.

But sometimes the real problem is the information being provided to the model.

Consider a RAG system.

If the retriever returns:

Chunk 1 → Navigation menu
Chunk 2 → Cookie policy
Chunk 3 → Unrelated documentation
Chunk 4 → Relevant documentation

the LLM now has mostly irrelevant context.

Even a very capable model has limited ability to produce a reliable answer from poor context.

This changed the way I think about AI systems.

Instead of only asking:

Which model should I use?

I also ask:

What information am I giving the model?

That question often leads to a better solution.

6. Web Scraping Is Messier Than It Looks

Working with web data taught me another important lesson.

A webpage is not a clean document.

A typical webpage can contain:

Navigation
Header
Sidebar
Main Content
Advertisements
Related Articles
Footer
Scripts
Repeated Content

If you're building a RAG system, embedding all of that information can introduce unnecessary noise.

That's why crawling is only the first step.

The actual pipeline needs to be something like:

Website
   ↓
Crawl
   ↓
Extract
   ↓
Clean
   ↓
Remove Noise
   ↓
Chunk
   ↓
Embed

This made me realize that data preparation is often more important than it initially appears.

AI systems are still affected by the classic principle:

Garbage in, garbage out.

A powerful model doesn't eliminate bad input.

7. Small Validation Checks Can Prevent Big Problems

Not every reliability improvement needs to be complicated.

Simple checks can prevent entire parts of a pipeline from failing.

For example:

if not response:
    return None

if "brand" not in response:
    raise ValueError("Missing brand")

if not content:
    skip_document()

These checks may look small.

But imagine a pipeline where one stage returns empty content and that result is passed through five more stages.

The actual error may happen much later.

Now debugging becomes much harder.

Validating data as it moves through the pipeline makes the source of the problem much easier to identify.

8. Logging Is More Important Than I Expected

When everything works, logs don't seem particularly important.

When something breaks, they become extremely useful.

For a multi-stage AI workflow, I want to know:

Request received
      ↓
Vision API called
      ↓
OCR response received
      ↓
Product identification completed
      ↓
Search completed
      ↓
Crawler extracted content
      ↓
GPT processing started
      ↓
Structured output generated

If the final result is incorrect, this gives me a way to trace the process.

Without logs, debugging becomes:

"Something went wrong somewhere."

With logs, it becomes:

"The search stage returned irrelevant results, so the problem started here."

That's a much more actionable problem.

9. Don't Put Everything Into One AI Call

When building AI applications, there can be a temptation to give one model the entire problem.

For example:

Image → LLM → Final Answer

Sometimes that works.

But breaking a complex problem into smaller stages can make the system easier to understand and improve.

For example, my product information workflow separates:

Visual Information
        ↓
Product Understanding
        ↓
Web Search
        ↓
Web Content Extraction
        ↓
Final Generation

Now each component has a defined purpose.

This also makes it easier to replace one component later.

For example, I could improve the crawling stage without completely redesigning the product-generation stage.

10. Reusable Components Save Time

Another lesson I've learned is that not everything should be rebuilt from scratch.

If I have already created a useful API integration, utility function, crawler component, or retrieval module, I should think about whether it can be reused.

Instead of:

Project A → Custom crawler
Project B → Another custom crawler
Project C → Another custom crawler

I would rather move toward:

             ┌── Project A
Reusable ────┼── Project B
Components   └── Project C

This doesn't mean building an enormous framework before it's needed.

It means recognizing useful patterns and turning them into reusable pieces when there is a genuine reason to do so.

11. Maintainability Matters

A project can work perfectly and still be difficult to maintain.

For me, maintainability means that I should be able to come back to the project later and understand:

Where the API is
Where external services are called
Where configuration is stored
Where data is processed
Where errors are handled
Where the database is accessed
Where the AI model is called

This is why I increasingly prefer:

Clear naming
+
Small modules
+
Reusable functions
+
Predictable APIs
+
Version control
+
Documentation

over putting everything into one large file simply because it is faster initially.

12. Version Control Is Part of the Engineering Process

Git isn't just something to use when a project is finished.

It is part of how I understand the evolution of a project.

A meaningful commit history can show progression such as:

Initial project setup
        ↓
Add API integration
        ↓
Implement crawler
        ↓
Add vector storage
        ↓
Implement retrieval
        ↓
Add source metadata
        ↓
Connect frontend

This becomes useful for debugging, experimenting, and going back to a previous working version.

It also becomes increasingly important when working with other developers.

13. I Started Thinking More About Failure Modes

One of the biggest changes in my development process has been asking:

What happens if this doesn't work?

For every important component, I try to think about the failure case.

What if the API fails?

Have an error path.

What if the crawler returns nothing?

Don't send empty content to the next stage.

What if retrieval returns no useful chunks?

Don't generate an answer as if reliable context existed.

What if the LLM response isn't structured correctly?

Validate and handle it.

What if the user provides unexpected input?

Validate it before processing.

This way of thinking has helped me move from simply building features toward building systems.

14. What I Would Do Differently Now

If I rebuilt some of my earlier projects today, I would spend more time on the engineering around the AI.

I would prioritize:

Clear input and output contracts
Better validation
More structured error handling
Logging
Automated testing
Better configuration management
Reusable modules
Cleaner API boundaries
Documentation

These aren't necessarily the most exciting parts of an AI project.

But they are what make the project easier to maintain.

15. The Biggest Mindset Shift

The biggest change in my thinking can be summarized in two questions.

Earlier:

Can I make this work?

Now:

Can I make this continue to work?

That difference is important.

Building real-world AI applications has taught me that the AI model is only one part of the system.

The surrounding engineering matters just as much.

You need:

Good Input
   ↓
Reliable Processing
   ↓
Good Data
   ↓
Useful Retrieval
   ↓
Controlled AI Generation
   ↓
Validated Output

The goal isn't just to create something that produces an impressive response.

The goal is to create something that people can actually use.

Final Takeaway

Working on AI and automation projects has changed how I approach software development.

I still care about experimenting with new models and APIs.

But I now pay much more attention to everything around them:

architecture, validation, error handling, data quality, retrieval, logging, testing, and maintainability.

The most useful lesson I've learned is this:

Don't focus only on making AI produce a good answer. Focus on building a system that can reliably produce useful answers.

That's where an AI demo starts becoming an actual application.

What I'm Exploring Next

I'm continuing to explore how AI can be combined with traditional software engineering to build systems that are not only intelligent, but also reliable and maintainable.

The next projects I build will focus less on simply demonstrating what an AI model can do and more on answering a different question:

How can I turn AI capabilities into useful software systems?

AI #ArtificialIntelligence #Python #Automation #SoftwareDevelopment #RAG #WebDevelopment #MachineLearning

Building an AI-Powered Product Information Agent for E-Commerce

Meet Dalsania — Fri, 21 Aug 2026 08:51:10 +0000

A product image contains a surprising amount of information.

A human can look at an image and often recognize the brand, product category, model, and several important characteristics within a few seconds. For an e-commerce system, however, that same image is just an image file.

Turning that visual information into structured product data requires multiple steps: extracting text, understanding what the product is, finding additional context, and finally producing information that can actually be stored and used by another system.

That was the problem I wanted to solve with an AI-powered product information agent.

The Problem
E-commerce and inventory systems depend heavily on structured product information.

A typical product record might contain:

Brand
Product name
Model
Category
Product descriptors
Product summary
Additional contextual information

But when the input is a product image, most of that information isn't directly available as structured data.

OCR can extract text, but text extraction alone doesn't tell you what the product is.

Search can find relevant information, but search results can also contain unrelated pages, duplicate information, advertisements, and conflicting descriptions.

An LLM can generate a description, but asking an LLM to work directly from an image without sufficient context can result in incomplete or unsupported information.

So instead of trying to make one model do everything, I approached the problem as a pipeline.

The Goal
The goal was to build a system that could take a product image as input and produce structured, useful product information as output.

The basic flow was:

Product Image
↓
Google Vision
↓
Extracted Text / Visual Signals
↓
Gemini
↓
Product Understanding
↓
Google Search
↓
Additional Context
↓
Crawl4AI
↓
Relevant Page Content
↓
GPT-4
↓
Structured Product Information

The important part of this architecture is that every component has a specific responsibility.

Why Multiple APIs?
One of the biggest design decisions was not trying to solve the entire problem with a single AI model.

Google Vision — extracting visible information
The first step was extracting information directly from the image.

Google Vision was useful for OCR and identifying visible signals from the product image.

For example, an image might contain:

SONY
WH-1000XM5
Wireless Headphones

The OCR layer gives the downstream system something concrete to work with.

However, OCR output isn't necessarily clean.

Text can be duplicated, characters can be misread, and information can appear in an unexpected order.

That meant OCR output had to be treated as an input signal rather than as the final answer.

Gemini — understanding the product
The next step was understanding what those extracted signals represented.

Gemini helped interpret the information and identify the likely product, category, and relevant descriptors.

This distinction is important.

There is a difference between:

"WH-1000XM5"

and:

Sony WH-1000XM5 wireless noise-cancelling headphones

The first is extracted text.

The second contains meaning.

*Google Search — enriching the context
*
Once the system had a likely product identity, search could be used to find additional information.

This step was necessary because the original image rarely contains everything required for a useful product record.

Search could help locate product pages, manufacturer information, reviews, specifications, and other relevant context.

But search introduces another problem: not every result is useful.

A search query can return:

Duplicate pages
Unrelated products
Marketplace listings
SEO-generated pages
Partial information
Conflicting descriptions

So search results needed another layer of processing.

Crawl4AI — getting the actual content
Instead of relying only on search snippets, Crawl4AI was used to extract relevant content from pages.

This allowed the pipeline to work with the actual page content rather than treating a search-result snippet as authoritative information.

The crawler therefore acted as the bridge between:

Search result
↓
Relevant webpage
↓
Usable content
GPT-4 — producing the final structured output

The final stage was turning all of these inputs into a clean product record.

The model could combine:

OCR output
Product identification
Search context
Crawled content

and produce a structured result.

For example:

{
"brand": "Sony",
"model": "WH-1000XM5",
"category": "Wireless Headphones",
"descriptors": [
"over-ear",
"noise cancelling",
"wireless"
],
"summary": "Sony WH-1000XM5 wireless over-ear headphones..."
}

The important point here is that the LLM wasn't expected to discover everything from scratch.

It was the final reasoning and formatting layer of a larger information pipeline.

Challenges

The interesting part of the project wasn't making the happy path work.

It was handling everything that didn't.

OCR isn't perfect

Images don't always contain clean, readable text.

Different fonts, angles, lighting, reflections, and image quality can all affect OCR results.

The downstream system therefore needed to tolerate imperfect input.

Search results are noisy

A product name doesn't guarantee that every search result refers to the same product.

This became a problem when multiple versions or similar products existed.

Search therefore had to be treated as an enrichment mechanism rather than unquestioned ground truth.

Duplicate information

The same product information can appear across multiple pages.

Feeding all of that content into the final model increases noise without necessarily improving the answer.

This made content selection and processing important.

Prompt design

The final model needs to understand exactly what information should be returned.

If the output format isn't constrained, the result can vary significantly between requests.

Structured prompting therefore became an important part of the pipeline.

Maintaining reliable output

An AI system is different from a traditional deterministic API.

You can't assume that every response will have exactly the same wording or structure.

For an application that eventually needs to store the result in a database, that is a serious engineering concern.

The system therefore needs validation around the model output instead of blindly trusting it.

Why the Architecture Matters

The biggest lesson from this project was the value of separating responsibilities.

Instead of creating one large process:

Image → AI → Answer

the architecture became:

Image
↓
Vision
↓
Product understanding
↓
Search
↓
Web extraction
↓
LLM processing
↓
Structured output

That makes debugging significantly easier.

If something goes wrong, you can ask:

Was the OCR incorrect?
Was the product identified incorrectly?
Did search return poor results?
Did the crawler extract the wrong content?
Did the final prompt interpret the information incorrectly?

Each stage becomes independently inspectable.

What I Would Improve Next

There are several directions I would take this system next.

First, I would add stronger validation between pipeline stages.

Second, I would improve product matching so that similar products and different product variants are handled more reliably.

Third, I would introduce better deduplication and source ranking for crawled content.

Finally, I would make the output schema stricter so that the generated information can be consumed directly by downstream inventory or catalog systems.

Final Thoughts
This project changed the way I think about AI applications.

The interesting part isn't simply connecting an LLM to an application.

The real engineering challenge is building the system around the model.

Vision, search, crawling, retrieval, prompting, validation, and structured output all have different responsibilities.

That architecture can be useful beyond this particular project.

The same approach could support:

Product catalog automation
Inventory onboarding
E-commerce data enrichment
Internal product databases
Automated product listing workflows

For me, the biggest takeaway was simple:

A useful AI system is usually a pipeline of smaller, well-defined components rather than one giant AI call.

DEV Community: Meet Dalsania

How I Built OmniScrape: A Modular RAG Engine with FastAPI and Qdrant

The Problem

The Goal

Why Qdrant?

Why FastAPI?

The Ingestion Pipeline

1. Crawl

2. Clean

3. Chunk

4. Embed

5. Store

Retrieval

Why Retrieval Quality Matters

Why Citations Matter

Engineering Challenges

Chunk Size

Metadata

Retrieval Quality

Response Formatting

Why Modular Design?

What OmniScrape Can Become

What I Learned From Building It

Final Thoughts

Tags

Lessons I Learned While Building Real-World AI and Automation Projects

1. Building a Demo Is Not the Same as Building a System

2. External APIs Can Fail

3. AI Output Needs Validation

4. Modular Architecture Makes Debugging Easier

5. Better Models Don't Fix Bad Data

6. Web Scraping Is Messier Than It Looks

7. Small Validation Checks Can Prevent Big Problems

8. Logging Is More Important Than I Expected

9. Don't Put Everything Into One AI Call

10. Reusable Components Save Time

11. Maintainability Matters

12. Version Control Is Part of the Engineering Process

13. I Started Thinking More About Failure Modes

What if the API fails?

What if the crawler returns nothing?

What if retrieval returns no useful chunks?

What if the LLM response isn't structured correctly?

What if the user provides unexpected input?

14. What I Would Do Differently Now

15. The Biggest Mindset Shift

Final Takeaway

What I'm Exploring Next

AI #ArtificialIntelligence #Python #Automation #SoftwareDevelopment #RAG #WebDevelopment #MachineLearning

Building an AI-Powered Product Information Agent for E-Commerce