Why Instagram Downloader API Splitting Carousels Saves Storage Costs

Architectural Separation of Social Metadata and Binary Media Assets

When building production pipelines that ingest media from Instagram, developers frequently run into bottleneck and storage issues. Many social scrapers merely extract volatile, signed CDN URLs directly from Instagram's servers. These links expire quickly, often within hours, rendering your saved database references useless and forcing you to scrape the same profile multiple times.

The instagram-downloader-api solves this problem by using a dual-storage architectural model. It separates structured metadata from actual binary files. While metadata records are written to a default dataset, the heavy media assets (images, videos, reels, and carousels) are downloaded directly and stored in Apify's key-value store.

This separation introduces unique considerations for your data ingestion architecture. Every media file is assigned a unique storage key and a public download URL pointing directly to Apify's key-value store. If you scrape a multi-image carousel post, the Actor splits the post, creating separate records in the dataset and saving separate files in the key-value store. This design keeps your metadata search fast and ensures that your downstream application has access to reliable, non-expired media download links.

Understanding how to access these two distinct storage subsystems is critical for designing downstream ingestion workers. Rather than reading a single flat JSON response, your pipeline must parse the dataset records, extract the specific storage keys, and fetch the binary streams directly from the key-value store endpoints.

How does the media output schema differ from typical web scraper datasets?

The media output schema differs from typical web scraper datasets because it splits multi-asset carousel posts into separate, independent metadata records while pointing to permanent key-value storage keys instead of unstable, expiring external CDN URLs. Each individual image or video file produces its own distinct record in the dataset, meaning a carousel post with five images generates five separate metadata items and five key-value store files.

{
  "filename": "3918833148741120108.jpg",
  "post_url": "https://www.instagram.com/leomessi/p/DZifppZj_cB/",
  "username": "leomessi",
  "type": "image",
  "download_status": "finished",
  "downloaded_at": "2026-07-02T18:46:36.768000",
  "storage_key": "3918833148741120108.jpg",
  "download_url": "https://api.apify.com/v2/key-value-stores/abc123/records/3918833148741120108.jpg",
  "media_meta_data": {
    "width": 1080,
    "height": 1350,
    "ext": "jpg",
    "filesize_bytes": 284672,
    "aspect_ratio": 0.8
  }
}

By extracting the download_url, downstream applications can reliably stream the binary data without worrying about signed URL expiration. For video files, the schema expands to include advanced properties inside the media_meta_data object, such as fps, duration, video_codec, and total_bitrate_kbps.

How do you prevent duplicate media downloads when resuming a failed workflow?

To prevent duplicate media downloads when resuming a workflow, you must track processed files in an external database or local state file and verify whether an incoming dataset item's storage_key has already been recorded. If the storage key is present in your local index, your pipeline code should bypass downloading the corresponding binary stream from the key-value store.

Social media pipelines are subject to frequent infrastructure restarts and API execution changes. Running a workflow a second time without deduplication means downloading the same large files repeatedly, incurring unnecessary network transfer and processing overhead.

You can implement an incremental state sync in Python. By reading the dataset items and matching their storage_key values against a local state store, you only stream the files that have not been previously downloaded.

import os
import pandas as pd
import requests

STATE_FILE = "instagram_pipeline_state.parquet"
APIFY_TOKEN = os.environ.get("APIFY_TOKEN")

def load_state():
    if os.path.exists(STATE_FILE):
        return pd.read_parquet(STATE_FILE)
    return pd.DataFrame(columns=["storage_key", "post_url", "downloaded_at"])

def save_state(state_df):
    state_df.to_parquet(STATE_FILE, index=False)

def process_pipeline_run(dataset_id):
    state_df = load_state()
    existing_keys = set(state_df["storage_key"].tolist())

    dataset_url = f"https://api.apify.com/v2/datasets/{dataset_id}/items?token={APIFY_TOKEN}"
    response = requests.get(dataset_url)
    if response.status_code != 200:
        raise RuntimeError(f"Failed to fetch dataset {dataset_id}")

    items = response.json()
    new_records = []

    for item in items:
        if item.get("download_status") != "finished":
            continue

        key = item.get("storage_key")

        if key in existing_keys:
            print(f"Skipping already processed media asset: {key}")
            continue

        media_url = item.get("download_url")
        media_response = requests.get(media_url)
        if media_response.status_code == 200:
            local_filename = f"media_vault/{item['filename']}"
            os.makedirs("media_vault", exist_ok=True)
            with open(local_filename, "wb") as f:
                f.write(media_response.content)

            print(f"Successfully processed and stored: {local_filename}")

            new_records.append({
                "storage_key": key,
                "post_url": item.get("post_url"),
                "downloaded_at": item.get("downloaded_at")
            })

    if new_records:
        new_records_df = pd.DataFrame(new_records)
        updated_state = pd.concat([state_df, new_records_df], ignore_index=True)
        save_state(updated_state)
        print(f"Pipeline complete. Added {len(new_records)} new assets to state store.")
    else:
        print("No new assets to process.")

This pattern ensures that overlapping runs do not result in double processing of binary data, keeping your storage environment highly efficient.

Orchestrating Background Executions with Webhooks

To decouple your client connections from long-running media extraction tasks, you should execute the instagram-downloader-api asynchronously. This prevents your server from holding open idle HTTP connections while the Actor downloads several large files.

When you trigger a run asynchronously, the platform immediately returns a run object with an HTTP 201 status code. To get notified when the data is ready, you can attach a webhook that executes a POST request to your target server once the run succeeds.

Here is a Python script that sets up the input payload, triggers the run asynchronously, and registers an HTTP webhook for notification:

import requests

APIFY_TOKEN = "your_apify_api_token_here"
ACTOR_ID = "crawlerbros/instagram-downloader-api"

run_input = {
    "usernames": ["natgeo"],
    "maxPosts_per_username": 5
}

run_url = f"https://api.apify.com/v2/acts/{ACTOR_ID}/runs?token={APIFY_TOKEN}"

webhook_config = [
    {
        "eventTypes": ["ACTOR.RUN.SUCCEEDED"],
        "requestUrl": "https://your-api-endpoint.com/webhooks/instagram-ingest",
        "payloadTemplate": "{\n  \"runId\": {{eventData.actorRunId}},\n  \"datasetId\": {{eventData.defaultDatasetId}}\n}"
    }
]

response = requests.post(
    run_url,
    json=run_input,
    headers={"Content-Type": "application/json"},
    params={"webhooks": str(webhook_config)}
)

if response.status_code == 201:
    run_data = response.json()
    print(f"Run started successfully. Run ID: {run_data['data']['id']}")
else:
    print(f"Failed to trigger run: {response.status_code} - {response.text}")

Using this decoupled pattern, your systems trigger runs within milliseconds. The platform processes the media downloads in the background and sends the final execution metadata to your webhook endpoint when complete.

How to execute the Actor using an explicit JSON configuration?

To run the Actor using an explicit JSON configuration, you must construct a valid payload adhering directly to the input schema and execute a POST request to the run API. This ensures that you avoid relying on Console UI behaviors like prefill values, which are completely ignored by direct API calls and do not populate your payload configuration.

Here is the explicit configuration required for targeting usernames:

{
  "usernames": [
    "natgeo",
    "leomessi"
  ],
  "maxPosts_per_username": 5
}

By explicitly specifying the variables in your payload, you guarantee consistent behavior across direct API calls, CLI commands, and automated tasks.

Why does calling synchronous runs directly lead to HTTP 408 gateway errors?

Calling synchronous runs directly leads to HTTP 408 gateway errors because the platform enforces a strict 300-second synchronization cap on all API calls executed synchronously. If the target Instagram accounts contain large reels or carousels that require several minutes to process and download, the synchronous HTTP connection is forced to terminate by the server.

To avoid this, your integration code must post to the asynchronous execution endpoint /v2/acts/crawlerbros/instagram-downloader-api/runs and poll the execution state, or rely on webhooks. This asynchronous pattern is essential for workloads where the combined duration of downloading, metadata generation, and key-value storage pushes exceeds 5 minutes.

Here is an example of an asynchronous execution pattern using Python to safely handle runs that exceed the 300-second limit:

import time
import requests

APIFY_TOKEN = "your_apify_api_token_here"
ACTOR_ID = "crawlerbros/instagram-downloader-api"

run_input = {
    "usernames": ["natgeo"],
    "maxPosts_per_username": 50
}

# Run asynchronously to avoid the 300-second synchronous HTTP limit
async_run_url = f"https://api.apify.com/v2/acts/{ACTOR_ID}/runs?token={APIFY_TOKEN}"
response = requests.post(async_run_url, json=run_input)
run_info = response.json()
run_id = run_info["data"]["id"]

print(f"Asynchronous run started. Run ID: {run_id}")

while True:
    status_url = f"https://api.apify.com/v2/actor-runs/{run_id}?token={APIFY_TOKEN}"
    status_response = requests.get(status_url).json()
    status = status_response["data"]["status"]

    print(f"Current execution status: {status}")
    if status in ["SUCCEEDED", "FAILED", "ABORTED"]:
        break
    time.sleep(15)

Integration Architectures with n8n and Webhook Triggers

For developers using low-code or visual orchestration tools, n8n provides a natural way to consume data from this Actor. Instead of manually polling the Apify API for execution updates, you can use n8n to react when a run finishes.

By setting up an integration flow, you can limit the processing of dataset items, retrieve the public key-value store URLs, and download the binary assets using standard command-line tools like curl. Below is an example of an n8n workflow configuration in YAML format that shows how to configure this pipeline:

meta:
  instanceId: dd813c988b48590c12da954363ff9d8c973a903bb2296711ee0f2dbfbf873c52
nodes:
  - parameters:
      pollTimes:
        item:
          - mode: everyMinute
      actorId: crawlerbros/instagram-downloader-api
      outputStatus: SUCCEEDED
    id: b61001df-b9fb-4556-91e8-782a9db38be3
    name: Apify Trigger
    type: n8n-nodes-base.apifyTrigger
    typeVersion: 1
    position:
      - 250
      - 300
  - parameters:
      operation: limit
      limit: 100
    id: e90b0e52-1cd7-4eb2-bc32-723b7bca0012
    name: Limit Dataset Processing
    type: n8n-nodes-base.limit
    typeVersion: 1
    position:
      - 450
      - 300
  - parameters:
      operation: executeCommand
      command: =curl -L "{{ $json.download_url }}" --output "/tmp/{{ $json.filename }}"
    id: f1a2b3c4-5d6e-7f8a-9b0c-1d2e3f4a5b6c
    name: Download Binary Asset
    type: n8n-nodes-base.executeCommandLine
    typeVersion: 1
    position:
      - 650
      - 300

This model ensures that you do not waste resources polling. When n8n receives the successful run event, it processes the dataset records sequentially and downloads the media to your local system or staging server.

Operational Limitations and Caveats to Keep in Mind

When planning a high-throughput pipeline using this Actor, you must account for several structural limitations and platform behaviors.

First, unnamed storages on Apify are temporary. If you run your workflows without naming your target key-value stores or datasets, those files are subject to deletion. On the free plan, the platform retains only the 10 most recent runs, which are kept for 4 months before being automatically purged. If you need historical records, you must move the downloaded files to your own cloud storage bucket immediately upon run completion, or use named storages which are exempt from standard deletion policies.

Second, the platform enforces strict storage rate limits. Storage requests are capped at 60 requests per second per storage object and 400 requests per second for dataset item pushes and request queue CRUD operations. If you attempt to download hundreds of media files simultaneously using aggressive parallel workers, your downstream pipeline will receive HTTP 429 rate-limiting errors. You must implement exponential backoff and limit concurrency when querying key-value store URLs.

Third, a request queue can only be processed by one Actor or task run at a time. If you try to run multiple concurrent scraper runs using a single shared request queue, the runs will not execute in parallel. You must isolate your request queues for each concurrent execution to avoid conflicts.

Fourth, the Actor relies on session rotation to access Instagram post media. Residential proxy sessions persist for around 30 minutes, whereas datacenter sessions persist for 26 hours. If a single run takes exceptionally long because you are downloading hundreds of videos, the proxy sessions may rotate out mid-run, which can result in error records instead of successful downloads. Your downstream parser must verify the download_status field on each record.

Fifth, private profiles are skipped. The Actor cannot download stories, and profile post discovery is limited to a maximum of 100 posts based on what Instagram surfaces in its dynamic profile scroll.

Understanding Event-Based Run Costs

The pricing model for the Instagram Downloader API is PAY_PER_EVENT. This means your total run cost is calculated solely by multiplying the volume of specific events triggered during the run by their published per-event prices. There is no separate platform-usage charge, subscription-plan rate on top, or compute-time fee.

The events charged during a run are:

• Actor Start (apify-actor-start): This is still a flat per-event price, charged at $0.005 per GB of memory allocated to the run.
• Result (apify-default-dataset-item): Charged at $0.005 per event for a single result in the default dataset. This event uses volume-tier prices (which are Apify VOLUME tiers; do not rename them to Apify subscription plans): FREE $0.005, BRONZE $0.00367, SILVER $0.00233, GOLD $0.001, PLATINUM $0.001, and DIAMOND $0.001.
• Downloading Image (download-image): Charged at $0.001 per event when a post has an image to be downloaded.
• Downloading Video Cost per 10 seconds (download-video-10s): Charged at $0.005 per event when a video downloads 10 seconds for each input. This event uses volume-tier prices: FREE $0.005, BRONZE $0.00433, SILVER $0.00367, GOLD $0.003, PLATINUM $0.003, and DIAMOND $0.003.

Your input parameters directly dictate your billing. Adjusting maxPosts_per_username to fetch larger amounts of content increases both the dataset record events and the physical media download events. A post containing an image will trigger one "result" event, one "Actor Start" event, and one "Downloading Image" event. A post containing a 10-second video will trigger one "result" event, one "Actor Start" event, and one "Downloading Video Cost per 10 seconds" event.

Input Schema Verification and Contracts

Before deploying this architecture, you must verify that your API calls align perfectly with the inputs supported by the Actor. The parameters you pass via HTTP POST must target these fields exactly:

{
  "postUrls": [
    "https://www.instagram.com/leomessi/p/DZifppZj_cB/"
  ],
  "usernames": [
    "natgeo"
  ],
  "maxPosts_per_username": 5
}

Passing parameters outside this schema will result in validation errors before the run starts. Note that the input schema prefill is shown in the Console UI but is not applied to API calls; only default is. You must always pass an explicit input JSON payload when triggering runs through the API.

Checked against the Actor's input schema and Apify docs on 2026-09-20. Ensure your workflow validates that inputs are properly formatted before initiating API requests.

The Actor's README is the source of truth for its inputs, outputs and limits. Need a hand wiring this into your stack? Email [e-mail]