DEV Community: Ellie

We built our 20-second launch video entirely in code (React, CSS and no 3D library)

Ellie — Thu, 22 Oct 2026 11:47:16 +0000

We built our 20-second launch video entirely in code (React, CSS and no 3D library)

We needed a launch video for Spicrawl, our web scraping API for AI agents. We didn't have a motion designer or After Effects, so we built the whole thing in code. It's React, CSS and a bit of SVG, rendered to MP4 with Remotion. Even the music is generated in JavaScript.

Here's the result 👇

Below is how each part works, in case you want to try it.

The setup: Remotion

Remotion lets you write a video as React components. Each frame is a render of your components at a given frame number. You animate by reading the current frame and computing styles from it:

const frame = useCurrentFrame()
const opacity = interpolate(frame, [0, 18], [0, 1], { extrapolateRight: 'clamp' })
return <h1 style={{ opacity }}>Scrape.</h1>

If you're building apps or AI agents that need live web data, check out Spicrawl. It provides a web scraping API and MCP to help you extract data from websites and use it in your projects.

Spicrawl is currently free to try during beta.

Do We Actually Need AI Agents for Everything? 🤔

Ellie — Tue, 13 Oct 2026 12:43:42 +0000

Do We Actually Need AI Agents for Everything? 🤔

AI agents are getting better at doing things on their own. They can write code, browse websites, use tools, and automate entire workflows.

But sometimes I wonder… are we making simple tasks more complicated by turning everything into an AI agent?

Think about it!!

If you need to process a CSV file, a simple script might be enough. You don't necessarily need an AI agent making decisions at every step.

But what if your project needs to collect data from hundreds of URLs, handle JavaScript-heavy websites, extract specific information, and keep that data updated?

Now things get more interesting.

You might need an AI agent to decide what data to collect, but a dedicated tool or API to handle the actual data extraction reliably.

I've been thinking about this while working on Spicrawl, especially when it comes to giving AI agents access to web data. Sometimes, letting an agent handle everything isn't the best approach. Giving it the right tools might be much more effective.

So I'm curious:

When do you think a simple script is enough, and when does a task actually need an AI agent?
Would you let an agent handle every step, or give it dedicated tools for specific jobs?
Are we making automation smarter, or just adding more complexity?

I'd love to hear how you're approaching this in your projects.

Are We Becoming Better Developers, or Just Better at Prompting? 🤔

Ellie — Sat, 10 Oct 2026 12:55:30 +0000

Are We Becoming Better Developers, or Just Better at Prompting? 🤔

Something I've been thinking about lately.

AI tools can write code, fix bugs, explain errors, and even build entire features in minutes.

Things that used to take hours can now take much less time.

And that's amazing.

But I sometimes wonder…

Are we actually becoming better developers, or are we just getting better at asking AI to do the work for us?

Think about it.

When AI fixes a bug, do we take the time to understand why it happened? When it generates a feature, do we understand how the code works, or are we just happy that it runs?

I'm not saying using AI is a bad thing. I use AI tools too, and they've helped me a lot.

But I think there's a difference between using AI to learn something and using AI to skip learning it.

And maybe the real skill now isn't just writing code. It's knowing what to ask, understanding the answer, and recognizing when the answer is wrong.

I'm curious about your experience:

Have AI tools helped you understand coding better?
Do you still try to solve problems yourself before asking AI?
If AI disappeared tomorrow, how confident would you feel building something on your own? 👀

No right or wrong answers. I'd genuinely love to hear different perspectives.

AI Can Write the Code. But Can You Debug It? 😅

Ellie — Fri, 09 Oct 2026 11:24:47 +0000

AI Can Write the Code. But Can You Debug It? 😅

AI makes writing code so much faster.

You describe what you want, get some code, run it, and sometimes it works on the first try. Feels amazing, right?

But then you need to change one small thing.

Suddenly, you're reading code you barely understand, fixing one bug creates another, and you're asking AI to fix the code it wrote in the first place. 😂

I've been thinking about this a lot lately.

AI is a great assistant, but I think there's still a big difference between getting code that works and understanding why it works.

And I'm curious about something:

Do you always understand AI-generated code before using it?
Have you ever fixed an AI-generated bug only to create two more?
Or has AI actually made debugging easier for you?

No judgment here. I'm genuinely curious about your experience. 👀

What's your most memorable experience with AI-generated code?

If Claude and Codex Can Scrape the Web, Why Do We Still Need Web Scraping Tools?

Ellie — Thu, 08 Oct 2026 16:26:45 +0000

If Claude and Codex Can Scrape the Web, Why Do We Still Need Web Scraping Tools?

This question has been on my mind lately.

Tools like Claude and Codex can already browse websites, read pages, extract information, and work with web content.

So if AI agents can do all of that...

Why would we still need a separate web scraping tool? 🤔

Maybe the difference starts showing when you need more than a few pages.

Hundreds of URLs. Structured output. JavaScript-heavy pages. Sessions. Repeated jobs. Reliable data that another part of your application can actually use.

I’m currently working in this space on Spicrawl, so I’m genuinely curious how other people see it.

Would you let your AI agent handle the whole web-data process itself, or would you still give it a dedicated scraping tool/API?

Would love to hear how you’re handling this in your projects.

What’s the Hardest Part of Building for the Web Right Now?

Ellie — Wed, 07 Oct 2026 07:43:59 +0000

I’m curious.. what’s the hardest part of building for the web right now?

For me, it’s making AI and web data work reliably together.

I’m currently working on a web scraping project and exploring how to make web data more accessible and useful for AI agents, especially when dealing with dynamic pages, blocked requests, and messy data 😅

What’s something you’ve been struggling with lately while building?
Would love to hear from other developers.

Cloudflare Is Changing How AI Agents Access the Web in 2026

Ellie — Tue, 06 Oct 2026 10:24:06 +0000

AI agents are getting better at using the web.

They can search for information, read documentation, compare products, research topics, and collect data for users.

But there is a problem:

More websites are deciding which kinds of AI traffic they want to allow.

Cloudflare is one of the biggest pieces of infrastructure between websites and their visitors. So when Cloudflare changes how AI traffic is handled, it matters to anyone building software that fetches web pages.

In July 2026, Cloudflare introduced separate controls for three types of AI-related traffic:

Search
Agent
Training

Then, in September 2026, Cloudflare changed the default settings for new domains.
Here is what developers need to know.

What changed?
Cloudflare's AI traffic controls were introduced in stages.

July 1, 2026
Cloudflare introduced three separate AI traffic categories:

Search
Crawlers that collect or index content so it can be used to answer questions later.

Agent
Automated software acting on behalf of a person, usually in real time.

Training
Crawlers collecting content to train or fine-tune AI models.

Website owners can decide how each type of traffic should be handled.

They can allow it, block it, or block it only on pages that display ads.
Existing Cloudflare customers keep their existing settings.

September 15, 2026
Cloudflare changed the default settings for new domains.
For new domains:

Traffic type New default
Search Allowed
Agent Blocked on pages with ads
Training Blocked on pages with ads

Website owners can change these settings.
The important part for AI developers is Agent traffic.

Why does this matter for AI agents?
Imagine a user asks an AI agent:
"Find the latest information about this product"

The agent may need to open several web pages, read them, and return the relevant information.
That is different from a traditional search crawler.
Cloudflare's definition of Agent is focused on automated software acting for a person in real time.
That is very close to how many AI agents work today.
So if more website owners use Cloudflare's new defaults, some AI agents may encounter more blocked pages.
But this won't happen to every website.
The new defaults apply to new domains onboarding to Cloudflare. Existing sites keep their current configuration unless the owner changes it.

Not every Cloudflare site will block AI agents

This is important.
Cloudflare isn't simply saying:
"AI crawlers are blocked"

The website owner decides what happens.
A site can:

Allow Search traffic
Allow Agent traffic
Allow Training traffic
Block one of them
Block traffic only on pages with ads
Block traffic everywhere

So a Cloudflare-protected website does not automatically mean an AI agent cannot access it.
You need to look at what the site actually allows.

What about Googlebot and other mixed crawlers?
Some crawlers do more than one job.
Cloudflare's documentation describes these as mixed-purpose crawlers.
For example, a crawler may be used for search indexing while also being associated with other AI-related activities.

Cloudflare has also added a Disallow AI Training option.
This allows a website owner to express a preference against AI training while still allowing accountable mixed-use crawlers to index the website for search.
So the exact setting chosen by the website owner matters.

What happens when your agent gets blocked?
A blocked request doesn't always mean the same thing.
You might receive:

403 Forbidden
A Cloudflare challenge
A rate-limit response
Another error from the website And you cannot always tell from the response whether the website intentionally blocked AI traffic or whether something else went wrong.

For example, in Spicrawl, an upstream bot challenge can appear as:
ERR::UPSTREAM::CHALLENGE
The target website's HTTP status can also be returned through the X-Target-Status header.
This distinction matters.
Your scraper's HTTP response is not always the same as the target site's actual response.

How should developers handle this?
The answer isn't to keep trying until the block disappears.
If a website has clearly decided not to allow your automated traffic, treat that as a decision.
Here are some better practices.

1. Check the site's rules

Look at:

robots.txt
Terms of service
API documentation
Other access policies These can tell you what the website allows.

2. Use an official source when available

If the website provides:

An API
A feed
A sitemap
An official data source use it instead of repeatedly fetching the website.

3. Treat a deliberate block as a "no"

If a website intentionally blocks automated access, don't try to work around the decision.
Ask for permission or use another source.

4. Fetch only what you need

Don't download an entire website when you only need one piece of information.
For example, request the specific content you need and avoid unnecessary requests.
This is better for both your application and the website you're accessing.

5. Avoid repeated requests

Caching can help prevent fetching the same page again and again.
For example, Spicrawl's cache is enabled by default and can keep results for up to 48 hours.

6. Slow down when necessary

If a website starts returning errors or rate limits, don't immediately send more requests.
Reduce the request rate and back off.

7. Check the real target status

If you're using a scraping API, don't assume its HTTP status tells the whole story.
Check whether the response contains the target site's actual status.
What this means for AI developers
The web is becoming a more controlled environment for AI agents.
Search crawlers, AI training crawlers, and real-time AI agents are increasingly being treated as different types of traffic.
That means an AI agent can't simply assume:
"If a browser can open the page, my agent can open it too."

Access can depend on the website's policies, infrastructure, bot controls, rate limits, and Cloudflare configuration.
For developers, this means your web-access layer needs to handle failures properly and respect the decisions made by website owners.

What website owners can do

If you run a website behind Cloudflare, you now have more control over AI traffic.
You can separately manage:

Search
Agent
Training You can also use Cloudflare's Disallow AI Training option to express that preference while allowing certain mixed-use crawlers to continue indexing for search. Cloudflare has also introduced Pay Per Use, currently in beta, for publishers who want to receive payment when AI products use their content.

The bigger picture

AI agents need access to the web to be useful.
At the same time, websites need control over how their content is accessed and used.
Cloudflare's new AI traffic controls are part of that shift.
For developers, the practical lesson is simple:
Don't treat every blocked page as a technical problem to bypass. First understand why access was denied and whether you're allowed to access the content.
As more websites define separate rules for search, AI agents, and AI training, responsible web access will become an important part of building reliable AI systems.

Originally published on Spicrawl
This article is adapted from the original: Cloudflare AI crawler blocking in 2026

About Spicrawl
Spicrawl is a web data platform for developers and AI agents, providing APIs for fetching web pages and returning clean data such as Markdown, JSON, HTML, and text.