DEV Community: David

Building Instagram Downloader: Turning a Simple Downloader Into a Real Product

David — Thu, 17 Sep 2026 18:05:57 +0000

At first, Instagram Downloader sounded like a small project.

A user pastes a public Instagram link, the service finds the media, and the browser downloads it. Simple.

But “simple” products are often the ones that require the most care. The user sees one input field and one button. Behind that interaction are dozens of decisions about usability, different media types, mobile behavior, accessibility, localization, reliability, and privacy.

This is the story of how FromInsta.net grew from a basic downloader into a focused web product.

Starting with one clear promise

The first version was built around a single workflow:

Copy a link to a public Instagram post.
Paste it into the service.
Choose the media you want to save.

There was no reason to make the interface more complicated than that. Users arrive with a very specific goal, so the page needs to help them complete it without registration, unnecessary settings, or a long explanation.

That principle remained important as the product expanded: new features should support the original workflow instead of competing with it.

A download button was not enough

The earliest product question was not simply whether a file could be downloaded. It was whether users could understand what they were about to download.

Instagram posts can contain a single photo, a video, a Reel, or a carousel with multiple items. A generic button does not communicate much when a post contains several files.

Media previews became an essential part of the experience. Users could inspect photos, play videos, move between carousel items, and then save exactly what they needed.

This changed the service from a link-processing utility into a visual workflow:

Paste, preview, choose, download.

Small interaction details also mattered. The interface needed clear loading states, useful error messages, predictable mobile controls, and an obvious way to recover when a request failed.

None of these features sounds remarkable on its own. Together, they determine whether a service feels dependable.

Audio became a product of its own

Downloading photos and videos was only part of the use case. Sometimes the valuable part of a Reel or video is its audio.

Adding audio support required a different user experience. Audio may need time to be prepared, so an immediate download is not always possible. Users need to know whether their request is waiting, being processed, ready, or no longer available.

The final experience lets users preview the audio and choose between the original available format and MP3. For carousel posts, the service processes the video items that contain audio and skips photos.

The important lesson was that audio could not simply be added as another button. It had its own states, expectations, and failure cases, so it deserved a dedicated flow.

Carousels exposed another kind of friction

A carousel may contain photos, videos, or a mixture of both. Downloading every slide individually works, but it becomes repetitive for larger posts.

That led to an option to save a complete carousel as a single ZIP archive.

Again, the main challenge was not presenting another feature. It was making the feature understandable:

Individual files should still be available.
Users should be able to preview each item.
The complete archive should be optional.
Temporary files should not appear permanent.

The result gives users both levels of control: save one specific item or download the full collection.

Localization changed more than the text

FromInsta.net is now available in 24 languages.

Localization initially looked like a content task, but it quickly became a design task. Labels become longer. Headings wrap differently. Some scripts require more space. Arabic and Persian introduce right-to-left layouts. Technical Instagram terminology does not always have a natural word-for-word translation.

Each language also needs more than a translated homepage. Navigation, instructions, errors, download states, accessibility labels, legal pages, and metadata must all remain consistent.

This taught us to treat localization as part of the product rather than as a final layer added after development.

A multilingual interface is not complete when every string has been translated. It is complete when every supported language can go through the entire workflow comfortably.

Mobile usability had to be treated as the default

Copying an Instagram link and opening a downloader is a naturally mobile workflow. That means mobile behavior cannot be a reduced version of the desktop experience.

Several details became especially important:

The link field must remain easy to use on a narrow screen.
Paste and clear actions must be easy to reach.
Security checks must fit without breaking the layout.
Preview controls must not cover the media.
Download buttons need clear pressed and disabled states.
Results should come into view when they are ready.

These improvements are easy to overlook when testing on a large screen. On a phone, they define the entire experience.

Reliability is a user-facing feature

Services that depend on another platform must expect change.

A link that works today may behave differently tomorrow. A post may be private, removed, restricted, or temporarily unavailable. A video may not contain an audio track. A request may take longer than expected.

Trying to hide this uncertainty would only create confusing failures. The better approach was to communicate it clearly.

FromInsta.net distinguishes between invalid links, unavailable posts, unsupported media, timeouts, processing problems, and temporary service errors. Users should know whether they can correct the problem, retry later, or whether the source itself is unavailable.

Monitoring and routine health checks also became part of the product’s development. They are invisible during normal use, but they reduce the time between a platform change and a fix.

Reliability is not only about preventing errors. It is also about making errors understandable.

Privacy boundaries should be visible

From the beginning, the service was designed for publicly available Instagram content.

It does not ask for an Instagram username, password, verification code, or access to a private account. Private, removed, and login-only posts are outside the supported use case.

Making these limits visible is important. A product should not imply that it can bypass access controls, and users should understand that downloading publicly accessible media does not transfer copyright or permission to reuse it.

FromInsta.net is an independent service and is not affiliated with Instagram or Meta.

Becoming an installable web app

As the interface matured, FromInsta.net also became installable as a Progressive Web App.

This was a natural extension of the mobile-first workflow. People who use the service regularly can open it from their home screen without treating it like a traditional application download.

The goal was not to imitate a native app for its own sake. It was to make a frequently repeated web task quicker to access.

What we learned

The biggest lesson from building FromInsta.net is that a narrow product can still have considerable depth.

A service with one main input field may need to handle multiple content types, asynchronous operations, multilingual layouts, mobile constraints, accessibility, external platform changes, and clear privacy boundaries.

A few principles guided the project:

Keep the primary workflow obvious.
Let users preview results before downloading.
Treat loading and error states as part of the product.
Design localization into the interface.
Be explicit about limitations.
Add features only when they reduce real user friction.
Remember that reliability work is product work.

FromInsta.net still begins with the same simple action: paste a public Instagram link.

Everything added since then has been built around making the next step clearer, safer, and more useful.

Try FromInsta.net

How I Built a Broken Link Scanner for Large Websites

David — Thu, 17 Sep 2026 17:25:54 +0000

A broken link checker sounds like a weekend project.

Fetch a page, collect every <a href>, request each URL, and report anything that returns 404.

That approach works surprisingly well—until you point it at a real website.

Large websites introduce duplicate URLs, redirects, external domains, missing assets, rate limits, malformed HTML, CSS imports, network failures, and thousands of pages competing for the same resources. At that point, a link checker stops being a script and starts becoming a distributed crawling system.

This is how I built one using Laravel as the control plane and Go as the scanning engine.

The architecture

I wanted Laravel to handle the application side of the product:

Users and authentication
Projects and scan settings
Usage limits
Scheduled scans
Result storage
Progress updates
Reports and exports

Go was a better fit for the network-heavy part: fetching many URLs concurrently while keeping memory usage and concurrency under control.

The final flow looks roughly like this:

User starts a scan
        ↓
Laravel creates a queued scan
        ↓
A Go worker leases the scan
        ↓
The worker crawls and checks URLs concurrently
        ↓
Results and progress are sent back in batches
        ↓
Laravel stores the data and builds the report

The Go workers are independent processes. They register with the application, send heartbeats, and ask Laravel for available jobs.

Laravel does not need to start a new operating-system process for every scan. A worker leases a queued scan through an API, processes it, and periodically reports its progress.

This made it possible to add more scanner servers without changing the web application.

A queue is more than a list of URLs

At first, my crawler had a basic queue:

Add the starting URL.
Download it.
extract its links.
Add new internal URLs to the queue.
Repeat until the queue is empty.

The difficult part was defining what “new” meant.

These URLs might all point to the same resource:

https://example.com/about
https://example.com/about#team
https://EXAMPLE.com/about
https://example.com:443/about
/about

Without normalization, the crawler may request the same page several times or even enter a loop.

Every discovered URL therefore passes through a normalization step. Relative URLs are resolved against the current document, fragments are removed, hosts are normalized, and default ports are handled consistently.

I still keep the original value found in the document. The normalized URL is useful for deduplication, but the original value is much more useful when someone needs to fix the link.

The crawler also separates discovery from traversal.

Internal pages can be added to the crawl frontier. External links are checked, but their pages are not recursively crawled. Otherwise, scanning one website could slowly turn into scanning the entire web.

Why parallel scanning needs limits

Checking URLs sequentially is polite but painfully slow. On a large website, network latency dominates the scan time.

Go makes it easy to create a goroutine for every URL, but unlimited concurrency is not a real solution. It can overwhelm both the scanner and the website being scanned.

I ended up using bounded concurrency at several levels:

A limit on simultaneous scans per worker
A worker pool inside each scan
A global limit on active HTTP requests
Per-host concurrency limits
Optional delays between requests to the same host

Per-host limits are especially important.

A page may contain resources from the main domain, a CDN, an analytics provider, and several external websites. The scanner can process different hosts in parallel without sending an unreasonable number of requests to any single server.

The crawler also reacts to temporary failures. Responses such as 429 Too Many Requests and some 5xx errors should not immediately become permanent broken-link reports. They may require a retry, a delay, or temporary backoff for that host.

Concurrency is not only a performance setting. It is part of the crawler’s behavior toward other websites.

HTTP status codes are not binary

One of the earliest mistakes was treating every response outside the 2xx range as broken.

In practice, the result is more nuanced:

2xx usually means the resource is available.
404 and 410 are strong broken-link signals.
3xx requires inspecting the complete redirect chain.
401 and 403 may mean the resource exists but rejects the scanner.
429 often means the scanner should slow down.
5xx may be a temporary server failure.
DNS, TLS, connection, and timeout errors need separate classifications.

A useful report should distinguish a confirmed broken URL from an inconclusive check.

For example, an external website may return 403 to automated clients while loading normally in a browser. Marking that URL as definitely broken would create a false positive and reduce trust in the entire report.

Instead of storing only is_broken, I found it more useful to preserve context:

HTTP status
Error category
Content type
Request duration
Whether the link is internal or external
Verification state
Final URL
Redirect history

The report can then prioritize clear failures while still showing ambiguous cases separately.

Redirects deserve first-class treatment

Redirects are not just an implementation detail.

A URL may return 301, lead to another redirect, cross to a different hostname, and finally return 200. Technically it works, but it may still be worth updating the original link.

There are also redirect loops, excessively long chains, malformed Location headers, and redirects to unsupported URL schemes.

For every redirect, the scanner records the status and destination. That makes it possible to show a chain such as:

/old-page [301]
→ /new-page [302]
→ https://www.example.com/new-page [200]

This is much more actionable than reporting only the final status.

Redirect targets also have to go through the same safety and scope checks as directly discovered URLs. Validating only the first URL is not enough because a public URL can redirect somewhere the scanner should never access.

Links are not limited to `<a href>`

A website can have no broken navigation links and still load incorrectly because an image, stylesheet, font, or script is missing.

The scanner therefore looks at several kinds of references in HTML, including:

Navigation links
Images and responsive srcset values
Stylesheets
Script sources
Iframes
Audio and video resources
Inline style URLs

CSS needs its own extraction step. A stylesheet may contain more resources through:

@import url("/theme.css");

.hero {
    background-image: url("/images/hero.webp");
}

@font-face {
    src: url("/fonts/site.woff2");
}

Those resources are easy to miss if the scanner only parses HTML.

JavaScript requires a careful boundary. The scanner checks referenced JavaScript files as resources, but it does not try to execute arbitrary application code or statically understand every URL constructed at runtime. Doing that reliably would require a browser-based rendering layer and would significantly change the cost and behavior of a scan.

Being explicit about that limitation is better than pretending a traditional crawler can see everything a browser can.

Keeping track of where a link came from

Knowing that a URL is broken is only half of the answer.

The next question is always:

Where do I need to fix it?

The same missing image or outdated URL may appear on hundreds of pages. Storing a separate complete result for every occurrence creates a lot of repetitive data, while keeping only the target URL loses the source information.

My solution was to treat the checked URL and its occurrences separately.

A result represents the target that was checked. It can also contain source information such as:

The page where it was discovered
The original URL value
The HTML element and attribute
The link text
The CSS discovery method
The number of occurrences

This allows the report to say that one missing asset affects 150 pages instead of displaying 150 nearly identical errors.

It also helps prioritize fixes. Repairing a broken global stylesheet is more important than fixing an unused link on an old article.

Sending results in batches

Large scans can produce a lot of data. Sending one API request per checked URL creates unnecessary overhead, while waiting until the entire crawl finishes risks losing everything if the worker stops.

The scanner sends progress and results in batches.

Each progress update can include:

Newly completed results
Current URL
Number of checked and queued URLs
Skipped URLs
Crawl metrics
A checkpoint of the remaining frontier

Laravel validates and stores the batch, updates the scan counters, and broadcasts the latest progress to the interface.

Batching reduced API overhead and made realtime progress inexpensive enough to be useful.

Making scans resumable

Long-running scans will eventually be interrupted.

A worker can restart, a deployment can happen, a network connection can fail, or a scan can exceed its allowed runtime. Starting again from the first page wastes time and generates duplicate traffic.

The crawler periodically saves enough state to continue:

URLs waiting to be processed
URLs already visited
Pending tasks
Crawl counters
Per-host backoff state

Because the job is leased rather than permanently assigned, Laravel can recover an interrupted scan and make it available to a worker again.

Checkpointing turned out not to be an optional reliability feature. For large websites, it is part of the normal execution model.

Turning crawl data into a useful report

Raw crawler output is useful for debugging, but it is not yet a product.

The report needs to answer practical questions:

Which links are definitely broken?
Which checks were inconclusive?
Which problems affect the most pages?
Which URLs redirect?
Where was each link found?
Is the target a page, image, stylesheet, script, or another resource?
Is the problem new, persistent, or resolved since the previous scan?

Laravel handles this part well. It stores the normalized results, calculates summaries, applies filters, generates CSV exports, and provides realtime progress while the Go scanner remains focused on crawling.

This separation also keeps presentation decisions out of the scanner. I can change how an issue is grouped or displayed without modifying the networking engine.

What I learned

The hardest part of building a broken link scanner was not making HTTP requests. It was deciding what every response meant and preserving enough context to make the result actionable.

A few lessons stood out:

Normalize URLs early, but keep the original values.
Bound concurrency at both the scan and host levels.
Treat redirects as data, not just transport behavior.
Separate confirmed failures from uncertain responses.
Record where every link was discovered.
Send incremental batches instead of waiting for completion.
Design for interrupted and resumed scans from the beginning.
Keep crawling and reporting as separate responsibilities.

A small link-checking script can be written in an afternoon. A scanner that works reliably across large, unpredictable websites needs queueing, backpressure, recovery, careful classification, and a reporting layer.

That engineering journey eventually became more than an internal experiment.

I ended up turning the scanner into BrokenLinks.pro — a tool for scanning websites, following redirects, checking assets, and turning crawl results into a practical report.