Automating video transcription for public Instagram reels
Extracting spoken audio from social media videos is a common requirement for content archiving, compliance tracking, and automated analysis pipelines. The instagram-transcript-scraper provides a structured way to fetch these transcripts directly from public Instagram URLs such as reels, posts, and IGTV. It reads pre-existing native captions when available or automatically falls back to transcribing the video audio locally using Whisper AI.
Because Instagram does not expose video data to logged-out users, the tool uses a managed pool of shared sessions. It handles the underlying authentication, media downloading, and speech-to-text processing, returning a clean dataset containing both the text and the metadata of the target post.
How do I run the scraper with Python?
You can run the scraper in Python by executing an HTTP POST request directly against the Apify API run endpoint using standard library tools like urllib3. This bypasses the higher-level SDK wrappers to afford you complete control over HTTP headers, raw payloads, connection timeouts, and customized retry policies.
Here is a production-ready implementation using Python's urllib3 library. It includes a custom backoff-retry loop that specifically intercepts transient network errors, HTTP 429 rate limits, and HTTP 504 gateway timeouts.
import json
import time
import urllib3
from urllib3.util import Retry
def run_instagram_scraper_raw(api_token, video_urls, transcription_method="auto", whisper_model="base"):
url = f"https://api.apify.com/v2/acts/crawlerbros~instagram-transcript-scraper/runs?token={api_token}"
payload = {
"videoUrls": video_urls,
"transcriptionMethod": transcription_method,
"whisperModel": whisper_model,
"includeSegments": True
}
encoded_data = json.dumps(payload).encode("utf-8")
# Configure pool manager with explicit timeouts and custom retry handling
# to handle raw API requests, backoff-retry for 429/504 errors, and timeouts
retries = Retry(
total=5,
backoff_factor=2,
status_forcelist=[429, 502, 503, 504],
raise_on_status=False
)
http = urllib3.PoolManager(retries=retries, timeout=urllib3.Timeout(connect=10.0, read=120.0))
try:
response = http.request(
"POST",
url,
headers={"Content-Type": "application/json"},
body=encoded_data
)
if response.status not in [200, 201]:
raise Exception(f"Failed to trigger Actor. Status: {response.status}, Response: {response.data.decode('utf-8')}")
run_data = json.loads(response.data.decode("utf-8"))
return run_data["data"]
except urllib3.exceptions.HTTPError as e:
print(f"Network exception encountered during execution: {e}")
raise e
# Example execution call
if __name__ == "__main__":
token = "apify_api_YOUR_TOKEN_HERE"
urls = ["https://www.instagram.com/reel/DV29mBcMQwp/"]
try:
run_info = run_instagram_scraper_raw(token, urls)
print(f"Run started successfully. Run ID: {run_info['id']}")
except Exception as err:
print(f"Execution failed: {err}")
This method avoids high-level SDK behavior. By configuring the Retry configuration manually, you dictate how your application reacts to rate limiting or gateway dropped packages.
How do I call the run endpoint asynchronously with Node.js?
To run this Actor asynchronously in Node.js, make a direct HTTPS request to trigger the run without blocking execution. This starts the transcription process on the platform immediately and returns a run object containing status and storage identifiers, allowing your application to monitor progress or retrieve the datasets out-of-band.
The following Node.js script avoids high-level SDK client packages. It uses the native https module to perform the API request, implementing manual backoff timing when hitting rate limits or gateway issues.
const https = require('https');
function startScraperAsync(apiToken, videoUrls, options = {}) {
return new Promise((resolve, reject) => {
const payload = JSON.stringify({
videoUrls: videoUrls,
transcriptionMethod: options.transcriptionMethod || 'auto',
whisperModel: options.whisperModel || 'base',
includeSegments: options.includeSegments || false
});
const requestOptions = {
hostname: 'api.apify.com',
port: 443,
path: `/v2/acts/crawlerbros~instagram-transcript-scraper/runs?token=${apiToken}`,
method: 'POST',
headers: {
'Content-Type': 'application/json',
'Content-Length': Buffer.byteLength(payload)
},
timeout: 15000 // 15-second socket timeout
};
const executeRequest = (attempt = 1) => {
const req = https.request(requestOptions, (res) => {
let responseData = '';
res.on('data', (chunk) => {
responseData += chunk;
});
res.on('end', () => {
const statusCode = res.statusCode;
// Check for transient rate limit (429) or gateway timeout (504)
if ((statusCode === 429 || statusCode === 504) && attempt <= 3) {
const delay = Math.pow(2, attempt) * 1000;
console.warn(`Encountered status ${statusCode}. Retrying in ${delay}ms (Attempt ${attempt}/3)...`);
return setTimeout(() => executeRequest(attempt + 1), delay);
}
if (statusCode !== 200 && statusCode !== 201) {
return reject(new Error(`API Error: ${statusCode} - ${responseData}`));
}
try {
const parsed = JSON.parse(responseData);
resolve(parsed.data);
} catch (err) {
reject(new Error(`Failed to parse JSON response: ${err.message}`));
}
});
});
req.on('error', (err) => {
if (attempt <= 3) {
const delay = Math.pow(2, attempt) * 1000;
console.warn(`Connection error: ${err.message}. Retrying in ${delay}ms...`);
return setTimeout(() => executeRequest(attempt + 1), delay);
}
reject(err);
});
req.on('timeout', () => {
req.destroy();
if (attempt <= 3) {
const delay = Math.pow(2, attempt) * 1000;
console.warn(`Request timed out. Retrying in ${delay}ms...`);
return setTimeout(() => executeRequest(attempt + 1), delay);
}
reject(new Error('Request timed out after maximum retry limit.'));
});
req.write(payload);
req.end();
};
executeRequest();
});
}
// Triggering the asynchronous function
const token = 'apify_api_YOUR_TOKEN_HERE';
const targets = [
'https://www.instagram.com/reel/DV29mBcMQwp/',
'https://www.instagram.com/p/DULBkEngpxg/'
];
startScraperAsync(token, targets, { transcriptionMethod: 'whisper', whisperModel: 'small', includeSegments: true })
.then(runInfo => {
console.log(`Successfully started Run ID: ${runInfo.id}`);
console.log(`Default Dataset ID is: ${runInfo.defaultDatasetId}`);
})
.catch(err => console.error(`Error: ${err.message}`));
By decoupling your application logic from high-level client wrappers, you gain fine-grained control over payload parsing, backoff delays, and network-level configurations.
How to trigger the transcription via curl?
You can trigger a transcription run via curl by sending an HTTP POST request to the Apify API run endpoint, passing your API token as a query parameter and specifying the input arguments inside a JSON payload. This enables integration with shell scripts, CI/CD tools, or systems written in languages without native Apify clients.
curl --request POST \
--url "https://api.apify.com/v2/acts/crawlerbros~instagram-transcript-scraper/runs?token=apify_api_YOUR_TOKEN_HERE" \
--header "Content-Type: application/json" \
--data '{
"videoUrls": ["https://www.instagram.com/reel/DV29mBcMQwp/"],
"transcriptionMethod": "auto"
}'
This request returns metadata containing the run status, the run ID, and the default dataset ID. You can use these values in subsequent GET requests to retrieve the output once processing finishes.
Why does my synchronous execution time out?
Synchronous execution times out because the Apify synchronous run endpoint has a hard-cap limit of 300 seconds, returning an HTTP 408 error if the run exceeds 5 minutes. This platform-wide limitation applies to any client calling the run directly instead of polling or utilizing asynchronous tasks.
When this timeout occurs, the container is not killed: the execution continues in the background, but the client connection is severed. This issue frequently affects runs processing multiple URLs or using the slower Whisper AI models. For tasks likely to exceed 5 minutes, you must trigger the run asynchronously via the /v2/acts/crawlerbros~instagram-transcript-scraper/runs endpoint and poll for completion, or set up a webhook to receive a notification once the status transitions to completed.
To configure an event-driven flow, webhooks on the Apify platform support exactly one action: posting a payload to a target URL. This allows you to handle completion events directly inside your web application without running continuous polling loops.
What is the structured schema of the scraper output?
The output returned for each successfully processed video includes full metadata, the generated transcript text, and optional timestamps if requested. The output JSON format contains the following keys:
{
"postUrl": "https://www.instagram.com/p/DV29mBcMQwp/",
"shortCode": "DV29mBcMQwp",
"pk": "3852537424986049577",
"id": "3852537424986049577_16278726",
"postDescription": "On Friday, US President Donald Trump claimed Iran's air force is \"no longer\"...",
"thumbnailUrl": "https://scontent-iad6-1.cdninstagram.com/v/thumbnail.jpg",
"videoUrl": "https://scontent-iad6-1.cdninstagram.com/v/video.mp4",
"pubDate": "2025-05-11T05:12:03Z",
"likeCount": 16799,
"commentCount": 1159,
"userId": "16278726",
"userName": "bbcnews",
"userFullName": "BBC News",
"avatarUri": "https://scontent-iad3-1.cdninstagram.com/v/avatar.jpg",
"fullText": "On Friday, U.S. President Donald Trump claimed Iran's Air Force is no longer, as a result of military action...",
"transcriptionMethod": "whisper",
"createdAt": "2026-06-08T06:09:05.841Z"
}
This output payload contains both the extracted transcript text in the fullText field and metadata about the post. This structured representation allows direct insertion into relational databases, search indexes, or vector stores for retrieval-augmented generation pipelines.
How to parse timestamped audio segments in Python?
When the includeSegments parameter is set to true, the scraper output includes a nested segments array. This array contains individual timestamped records for each transcribed phrase, which allows you to match the spoken text back to the exact time in the video.
The following Python script reads the default dataset, checks for successful items, and iterates over the timestamped segments to output the start, end, and text of each spoken sentence:
def print_timestamped_segments(dataset_items):
for item in dataset_items:
if "errMsg" in item:
continue
print(f"--- Transcript for {item['userName']} ({item['shortCode']}) ---")
# Verify if segments are present in the output
segments = item.get("segments")
if not segments:
print("No timestamped segments were included in this run.")
print(f"Full Text: {item['fullText']}\n")
continue
for segment in segments:
index = segment["index"]
start_time = segment["start"]
end_time = segment["end"]
text = segment["text"]
# Format output for subtitle generation or log tracking
print(f"[{index}] {start_time:05.2f}s -> {end_time:05.2f}s: {text.strip()}")
print("\n")
This script allows you to reconstruct the timeline of the video, which is helpful when building searchable media databases or matching subtitles back to original video timelines.
How do you handle failed transcription items?
To handle failed transcription items, parse the dataset returned by the scraper and check for the presence of the errMsg field in each record. Successful extractions populate fullText, whereas failures include errMsg detailing the reason for omission, such as private content or deleted posts.
For automated production pipelines, checking for this key allows you to handle issues gracefully. You can route invalid posts to an alternative tracking table while allowing valid metadata and transcriptions to flow into your main database.
import time
def process_and_route_item(item, dlq_list, database, retry_queue):
# Differentiate between scraping failure modes
# Detect if the video record is missing essential content or contains an explicit error
if "errMsg" in item:
error_message = item["errMsg"].lower()
post_url = item.get("postUrl", "Unknown URL")
# Determine if the error is permanent (private profile, deleted post)
if "private" in error_message or "deleted" in error_message or "not found" in error_message:
print(f"PERMANENT ERROR: Routing {post_url} to Dead Letter Queue (DLQ). Reason: {item['errMsg']}")
dlq_list.append({
"url": post_url,
"error": item["errMsg"],
"failed_at": time.time(),
"type": "PERMANENT_LIMITATION"
})
else:
# Differentiate transient proxy bans, empty payloads, or rate limits for retry
print(f"TRANSIENT ERROR: Routing {post_url} back to retry queue. Reason: {item['errMsg']}")
retry_queue.append({
"url": post_url,
"last_error": item["errMsg"],
"attempts": item.get("retry_attempts", 0) + 1
})
return
# Handle cases where the item was successfully scraped but returned empty text
full_text = item.get("fullText", "").strip()
if not full_text:
# Check if native transcription was used without speech
if item.get("transcriptionMethod") == "native":
print(f"EMPTY CONTENT: {item['postUrl']} returned no transcript text. Retrying with Whisper fallback.")
retry_queue.append({
"url": item["postUrl"],
"force_whisper": True,
"attempts": 1
})
else:
print(f"INFO: {item['postUrl']} processed but contains no audible speech.")
database.save(item["id"], item)
return
# Successful path
database.save(item["id"], item)
print(f"SUCCESS: Successfully saved transcribed item: {item['id']}")
This structural division ensures that your downstream services do not stall due to login-wall blockages while guaranteeing that scraping jobs interrupted by temporary network issues are safely retried.
Integrating the transcription workflow with AI agents via MCP
Integrating social media video intelligence directly into AI agent workflows requires bridging the gap between LLM execution environments and the Apify platform. By using the Model Context Protocol (MCP), you can expose the Apify platform as an execution engine that LLM agents can call dynamically. This approach turns the instagram-transcript-scraper into a structured function call that an agent can invoke when it needs to analyze video audio.
To make the scraper available to an LLM, you use the Apify MCP server at https://mcp.apify.com. By scoping your MCP server configuration with the specific query parameter ?tools=crawlerbros/instagram-transcript-scraper, you restrict the agent's toolset so it only sees this specific scraping capability rather than the thousands of other public Actors. This prevents token bloat and keeps the agent's system prompt focused on the transcription task.
This integration requires an Apify API token because running Actors always requires a token, even though the MCP server allows unauthenticated discovery tools like search-actors or fetch-apify-docs. Checked against the Actor's input schema and Apify docs on 2026-09-09.
When an AI agent discovers this scraper via MCP, the JSON schema of the Actor's input is translated directly into the tool's parameter signature. The LLM relies on these property descriptions to decide how to construct its function call arguments. When constructing tool calls, the LLM must generate an object matching this schema. Note that the prefill values configured in the Apify Console UI are ignored entirely when making API or MCP calls. Only the default properties defined in the actual input schema are applied automatically. Therefore, if the agent leaves out optional parameters, the platform will fall back to "auto" for transcriptionMethod, "base" for whisperModel, and false for includeSegments.
What are the technical limitations and caveats of this Actor?
This Actor operates within several technical boundaries defined by both the target platform and the execution environment, which can cause runs to fail or data to expire if not planned for. First, the tool can only scrape public videos. Private accounts, restricted posts, and login-gated content cannot be transcribed, and attempting to do so will result in an errMsg payload.
Second, the CDN URLs returned in the dataset are temporary. Fields like videoUrl, audioUrl, thumbnailUrl, and avatarUri point directly to Instagram's content delivery network. These URLs are signed and expire within hours or days. You must download the media files immediately during your run if you plan to archive them.
Third, if you decide to supply your own authentication cookies using the cookies input field, remember that proxy sessions play a role in cookie life cycles. Residential proxy sessions target around 30 minutes before rotation, while datacenter proxies persist for 26 hours. If your sessions rotate or your cookie data expires mid-run, the Actor will lose authentication.
Fourth, scheduling has specific constraints. Apify schedules use a 6-field cron pattern (where seconds are optional) with a minimum execution interval of 10 seconds. You cannot schedule this Actor unless it has been run at least once manually, and newly created schedules are disabled by default.
Fifth, storage limits can impact large-scale operations. Apify limits key-value store and request queue requests to 60 requests/second per storage object, and dataset pushes to 400 requests/second. Note that a request queue can only be processed by one run at a time. Trying to fan out a workload by running multiple instances of this Actor against a single shared request queue is not supported.
Finally, on the Free plan tier, unnamed storages are subject to strict retention policies: only the 10 most recent runs are kept, and they expire after 4 months. To prevent your data from being automatically deleted, you must name your datasets or key-value stores during the run creation.
How do you calculate the platform run costs?
Apify platform usage is calculated using compute units (CUs), which scale based on the memory allocated to the container and the total duration of the execution. One compute unit is defined as 1024 MB of memory running for exactly one hour (1024MB x 1 hour = 1 CU).
The mathematical formula for calculating the consumed CUs is:
Compute Units (CU) = (allocated_memory_mb / 1024) * duration_hours
If you double the memory allocated to a run, the run must execute in half the time to remain CU-neutral. This CU neutrality only applies to autoscaling runs, and autoscaling is only active on solutions that process multiple tasks or URLs for at least 30 seconds each.
The cost of a compute unit depends entirely on your subscription tier:
- Free Tier: $0.20 per CU
- Starter Tier: $0.20 per CU
- Scale Tier: $0.16 per CU
- Business Tier: $0.13 per CU
If your run uses residential proxies (which may be required if you disable custom cookies and rely on the managed pool of shared sessions), proxy data is billed separately. The residential proxy rates are:
- Free and Starter Tiers: $8 per GB
- Scale Tier: $7.50 per GB
- Business Tier: $7 per GB
Because Whisper AI transcribes speech by downloading the audio file and running local inference inside the Actor's container, it scales with both data transfer and processing time. Setting the whisperModel parameter to small requires more memory (the model file is 244 MB compared to the base model's 74 MB) and takes more processing time, which directly increases the compute duration. In contrast, the native transcription method does not download the audio or run inference; it reads pre-existing text captions, resulting in low memory usage, low run duration, and minimal data consumption.
To protect against unexpected bills from large runs, you can append the maxTotalChargeUsd query parameter to your run API calls. This parameter is exposed inside the Actor container as the environment variable ACTOR_MAX_TOTAL_CHARGE_USD. When the calculated cost of the run hits this limit, the run terminates, though it may consume resources briefly during the shutdown sequence before fully terminating.
Written with AI assistance and checked against the Actor's published input schema, README and Apify's platform documentation before publishing. Figures quoted here come from those sources, not from a benchmark we ran.
Top comments (1)
This implementation of the Instagram transcript scraper is impressive, especially the way it handles transient network errors with a custom backoff-retry loop. It’s a great example of how to ensure robustness in API interactions. One suggestion could be to implement logging for the retries; it might help track how often these are triggered and provide insights into potential systemic issues with API limits. If you’re considering further enhancements or support for additional transcription methods, I’d be glad to explore a paid collaboration to contribute to that aspect. How do you envision scaling this solution as demand increases?