Building the AI Was Easy. Building Everything Around It Wasn't

Rohit Pathak — Fri, 07 Aug 2026 15:25:13 +0000

Like many engineers, I started with a simple assumption:

"AI can read a website and extract the information we need."

Technically, that's true.

But when your goal is to process thousands of websites instead of one, AI quickly becomes the smallest part of the problem.

We were building a platform that needed structured information from company websites at scale. Things like company descriptions, products, industries, certifications, contact details, and more.

At first, we focused on prompts.

Maybe a better prompt would improve the results.

Maybe another model would perform better.

Maybe we needed more examples.

After a while, we realized we were solving the wrong problem.

The real challenge wasn't extraction.

It was everything that happened before and after it.

Every website looked different.

Some were clean.

Some were outdated.

Some loaded content dynamically.

Others spread useful information across dozens of pages.

Instead of writing custom scrapers that would constantly break, we built a pipeline that could handle the mess for us.

Firecrawl handled crawling and converted websites into clean Markdown.

n8n orchestrated the entire workflow, from scheduling crawls and retrying failures to monitoring long-running jobs and triggering AI extraction.

Once that foundation was in place, AI simply became another step in the pipeline.

One lesson has stayed with me throughout this project.

Reliable systems beat clever prompts.

A perfect prompt can't fix incomplete data.

It can't recover failed jobs.

It can't monitor thousands of running tasks.

It can't tell you why a workflow silently stopped overnight.

Those are engineering problems, not AI problems.

Today, the pipeline runs with minimal manual intervention and continuously converts messy websites into structured, usable data.

Looking back, we didn't spend most of our time building AI.

We spent it building a system that allowed AI to work reliably.

And that's probably the biggest misconception about AI products today.

The model is important.

But the infrastructure around it is what makes it useful.

Scalability Isn’t About Traffic - It’s About Change

Rohit Pathak — Mon, 16 Feb 2026 13:36:43 +0000

Most people think scalability is about handling more users.

In practice, it’s about handling change.

From what I’ve seen across multiple web platforms, systems usually don’t break when traffic increases. They break when requirements evolve — new features, new workflows, or small changes that suddenly feel risky.

This often happens when early decisions are rushed:

business logic gets tightly coupled with UI
shortcuts become permanent
every new feature touches too many parts of the system

At Floatinity, we’ve learned that scalable platforms are built by keeping things simple early on. Clear separation of responsibilities, stable foundations, and small, well-defined iterations make systems easier to extend later.

Instead of optimizing for every future scenario, we design assuming things will change — because they always do.

If you’re building a platform today, a few principles help:

optimize for clarity, not cleverness
expect requirements to evolve
structure code so parts can change independently

Traffic problems are usually solvable.
Structural problems are expensive.

Scalability isn’t about predicting the future.
It’s about not blocking it.

DEV Community: Rohit Pathak

Building the AI Was Easy. Building Everything Around It Wasn't

Scalability Isn’t About Traffic - It’s About Change