Saving a download URL doesn't finish your archive.
Your overnight job finishes, and you save download_url for next week's processing queue. It feels settled. Then you notice expires_at.
Signed S3/R2 links last 1–7 days, independently of the object's storage lifecycle. An expired link doesn't prove the archived file is gone. It proves your saved permission to fetch it has expired.
Choose storage for the worker that arrives later
A signed URL carries temporary authorization. An object key identifies what you want. Your archive needs both concepts, even when the first download works.
This recipe uses a private S3 bucket and a narrow input schema for that route. Example input, with storage credentials supplied through the environment:
import { ApifyClient } from 'apify-client';
import { z } from 'zod';
import pg from 'pg';
import { createHash } from 'node:crypto';
import { Transform, type Writable } from 'node:stream';
import { pipeline } from 'node:stream/promises';
const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const pool = new pg.Pool();
const run_input = z.object({
mode: z.literal('download'),
videoUrls: z.array(z.string().url()).min(1),
maxVideos: z.number().int().min(1).max(50),
storage: z.literal('s3'),
storageRetentionDays: z.number().int().min(1).max(7),
s3Bucket: z.string().min(1),
s3Prefix: z.string(),
s3Region: z.string(),
s3AccessKeyId: z.string().optional(),
s3SecretAccessKey: z.string().optional(),
}).strict().parse({
mode: 'download',
videoUrls: ['https://www.youtube.com/watch?v=jNQXAC9IVRw'],
maxVideos: 5,
storage: 's3',
storageRetentionDays: 7,
s3Bucket: process.env.ARCHIVE_BUCKET ?? 'archive-example',
s3Prefix: 'youtube-downloader',
s3Region: process.env.ARCHIVE_REGION ?? 'us-east-1',
...(process.env.S3_ACCESS_KEY_ID && process.env.S3_SECRET_ACCESS_KEY
? {
s3AccessKeyId: process.env.S3_ACCESS_KEY_ID,
s3SecretAccessKey: process.env.S3_SECRET_ACCESS_KEY,
}
: {}),
});
Replace the placeholder bucket and video URL with your destination and authorized content. Supply both storage credentials together, or configure credentials available to the Actor.
The downstream worker needs its own storage access. Saving the Actor's credentials in your manifest would turn a useful table into an unfortunate secrets manager.
Separate object identity from access
The handoff follows this split:
Actor -> dataset -> validated manifest -> delayed worker
| |
bucket + storage_key ------+
download_url + expires_at |
checksum + size <--- verified transfer
Persist the exact returned storage_key. Don't reconstruct it from the video ID or prefix.
For this route, the destination also includes bucket and region. If you introduce a custom endpoint, persist that alongside them. Otherwise, the same key can point your worker at the wrong storage service.
Storage selection also changes transfer behavior. KVS file URLs don't support HTTP Range requests, so resumable or partial downloads need S3/R2. Ephemeral KVS follows your Apify account's retention; named KVS persists while its store exists.
A longer-lived store can't add a missing HTTP capability. Annoying, but useful to discover before writing the resume button.
Save the run ID before waiting
Starting asynchronously lets you persist the run ID before waiting. Restarting ingestion can then reuse that run instead of launching another archive.
The database keeps destination configuration with the run. A later environment change mustn't relabel an existing object's address.
const Destination = z.object({
bucket: z.string().min(1),
region: z.string().min(1),
});
async function getRun() {
await pool.query(`
CREATE TABLE IF NOT EXISTS archive_runs (
run_id text PRIMARY KEY,
destination jsonb NOT NULL
);
CREATE TABLE IF NOT EXISTS archive_manifest (
video_id text PRIMARY KEY,
storage_key text NOT NULL,
storage_mode text NOT NULL,
download_url text NOT NULL,
expires_at timestamptz NOT NULL,
checksum_sha256 text NOT NULL,
filesize_bytes bigint NOT NULL,
destination jsonb NOT NULL,
source_run text NOT NULL,
raw jsonb NOT NULL
);
CREATE TABLE IF NOT EXISTS archive_review (
run_id text NOT NULL,
item_offset bigint NOT NULL,
reason text NOT NULL,
raw jsonb NOT NULL,
PRIMARY KEY (run_id, item_offset)
);
`);
let runId = process.env.ARCHIVE_RUN_ID;
let destination: z.infer<typeof Destination>;
if (runId) {
const saved = await pool.query(
`SELECT destination FROM archive_runs WHERE run_id = $1`,
[runId],
);
if (saved.rows.length !== 1) {
throw new Error(`Missing destination configuration for run ${runId}`);
}
destination = Destination.parse(saved.rows[0].destination);
} else {
destination = Destination.parse({
bucket: run_input.s3Bucket,
region: run_input.s3Region,
});
const started = await client.actor(
'nimbryxo-studio/youtube-video-playlist-and-channel-downloader',
).start(run_input);
runId = started.id;
await pool.query(
`INSERT INTO archive_runs (run_id, destination) VALUES ($1, $2)`,
[runId, JSON.stringify(destination)],
);
}
const run = await client.run(runId).waitForFinish();
if (!run?.defaultDatasetId ||
!['SUCCEEDED', 'FAILED', 'TIMED-OUT', 'ABORTED'].includes(run.status)) {
throw new Error(`Run ${runId} has no terminal dataset to ingest`);
}
return { run, destination };
}
A fresh run uses the destination it just saved. A replay loads the original destination from the database, rather than trusting today's environment.
Set ARCHIVE_RUN_ID to replay a saved run. This implementation rereads its dataset from the beginning.
There's still a crash window between starting the Actor and saving its ID. Recover that ID before launching again. An idempotent database write doesn't make external starts exactly once.
Accept new metadata, reject broken handoffs
Empty output fields are omitted. Failure rows therefore need a different contract from downloadable assets.
Validate the row envelope first, then enforce the fields your worker requires:
const Row = z.object({
record_type: z.enum(['video', 'failure', 'input_error']),
video_id: z.string().min(1).optional(),
download_status: z.enum([
'success', 'metadata_only', 'subtitles_only',
'failed', 'skipped_unsupported', 'upload_failed',
]).optional(),
failure_code: z.string().optional(),
error: z.string().optional(),
}).passthrough();
const Asset = Row.extend({
record_type: z.literal('video'),
video_id: z.string().min(1),
download_status: z.literal('success'),
storage_mode: z.literal('s3'),
storage_key: z.string().min(1),
download_url: z.string().url(),
expires_at: z.string().datetime({ offset: true }),
checksum_sha256: z.string().regex(/^[a-fA-F0-9]{64}$/),
filesize_bytes: z.number().int().nonnegative()
.max(Number.MAX_SAFE_INTEGER),
}).passthrough();
The asset schema is deliberately stricter than the general row contract. A signed-link handoff without an expiry or checksum can't satisfy this worker, even if the record remains useful elsewhere.
Passthrough validation preserves additional metadata. Strict validation would reject harmless additions; passthrough trades that sensitivity for fewer false alarms.
Here, drift means a consumed field disappears, changes type, or gains an unsupported value. Additional fields remain available in raw. They don't trigger a page.
Upsert the current asset without erasing a good copy
Use video_id as the key for a current manifest. It identifies the video, rather than the temporary URL used to retrieve it.
That key isn't enough for representation history. Audio, different qualities, and successive downloads can all belong to the same video. If you need those together, introduce a representation identity and retain separate asset versions.
The storage split earns its extra columns:
| Storage choice | What it buys you | What you maintain |
|---|---|---|
| Typed handoff columns | Expiry queries, constraints, direct worker access | Explicit schema changes |
| Raw JSONB | Original metadata and rejected-record diagnosis | Access restrictions and retention policy |
The ingestion loop batches writes, deduplicates each page, and commits its review records with its accepted assets:
async function ingest() {
const { run, destination } = await getRun();
const pageSize = 100;
let reviewCount = 0;
for (let offset = 0; ; ) {
const { items } = await client.dataset(run.defaultDatasetId)
.listItems({ offset, limit: pageSize });
if (items.length === 0) break;
const accepted = new Map<string, Record<string, unknown>>();
const review: Array<{
run_id: string;
item_offset: number;
reason: string;
raw: unknown;
}> = [];
for (const [index, item] of items.entries()) {
const envelope = Row.safeParse(item);
let reason: string | undefined;
if (!envelope.success) {
reason = `schema: ${envelope.error.message}`;
} else if (envelope.data.record_type !== 'video' ||
envelope.data.download_status !== 'success') {
reason = envelope.data.failure_code
?? envelope.data.download_status
?? envelope.data.record_type;
} else {
const asset = Asset.safeParse(item);
if (!asset.success) {
reason = `handoff: ${asset.error.message}`;
} else {
accepted.set(asset.data.video_id, {
...asset.data,
destination,
source_run: run.id,
raw: item,
});
}
}
if (reason !== undefined) {
review.push({
run_id: run.id,
item_offset: offset + index,
reason,
raw: item,
});
}
}
const db = await pool.connect();
try {
await db.query('BEGIN');
await db.query(`
INSERT INTO archive_manifest
SELECT * FROM jsonb_populate_recordset(
NULL::archive_manifest, $1::jsonb
)
ON CONFLICT (video_id) DO UPDATE SET
storage_key = EXCLUDED.storage_key,
storage_mode = EXCLUDED.storage_mode,
download_url = EXCLUDED.download_url,
expires_at = EXCLUDED.expires_at,
checksum_sha256 = EXCLUDED.checksum_sha256,
filesize_bytes = EXCLUDED.filesize_bytes,
destination = EXCLUDED.destination,
source_run = EXCLUDED.source_run,
raw = EXCLUDED.raw
`, [JSON.stringify([...accepted.values()])]);
await db.query(`
DELETE FROM archive_review
WHERE run_id = $1 AND item_offset >= $2 AND item_offset < $3
`, [run.id, offset, offset + items.length]);
await db.query(`
INSERT INTO archive_review
SELECT * FROM jsonb_populate_recordset(
NULL::archive_review, $1::jsonb
)
`, [JSON.stringify(review)]);
await db.query('COMMIT');
} catch (error) {
await db.query('ROLLBACK');
throw error;
} finally {
db.release();
}
reviewCount += review.length;
offset += items.length;
}
if (reviewCount > 0 || run.status !== 'SUCCEEDED') {
process.exitCode = 1;
}
}
A rejected record never replaces an accepted asset. Replaying a page updates the same manifest and refreshes its review entries.
Serialize manifest-writing runs for this implementation. It uses the last accepted write, with no ordering guard against an older run replayed after a newer one.
Also remember that database preservation can't protect an object overwritten in storage. Retaining historical bytes requires a storage lifecycle or versioning policy of your own.
Can the delayed worker still fetch the object?
At execution time, compare expiry with the transfer deadline. Checking only whether the link is valid now leaves a download that may lose its access window halfway through.
When that window is insufficient, obtain fresh authorization using the saved destination and key. That authorization function belongs to your storage integration, using the worker's credentials.
This helper accepts that function and verifies a full transfer into a temporary destination:
async function transferVerified(
asset: z.infer<typeof Asset>,
destination: z.infer<typeof Destination>,
target: Writable,
transferDeadline: Date,
authorize: (
destination: z.infer<typeof Destination>,
storageKey: string,
) => Promise<string>,
) {
const url = Date.parse(asset.expires_at) > transferDeadline.getTime()
? asset.download_url
: await authorize(destination, asset.storage_key);
const response = await fetch(url);
if (response.status !== 200 || !response.body) {
await response.body?.cancel();
throw new Error(`Archive fetch failed: HTTP ${response.status}`);
}
let bytes = 0;
const hash = createHash('sha256');
const meter = new Transform({
transform(chunk, _encoding, callback) {
bytes += chunk.length;
if (bytes > asset.filesize_bytes) {
callback(new Error('Archive exceeded expected size'));
return;
}
hash.update(chunk);
callback(null, chunk);
},
});
await pipeline(response.body, meter, target);
if (bytes !== asset.filesize_bytes ||
hash.digest('hex') !== asset.checksum_sha256.toLowerCase()) {
throw new Error('Archive integrity check failed');
}
}
try {
await ingest();
} finally {
await pool.end();
}
Publish the temporary file only after verification resolves. Remove it on failure. A completed HTTP response is a transport result, not an integrity check.
The helper doesn't implement resuming. A resumable worker must verify the partial response and ensure it's continuing the same object, then check the checksum across the complete file.
A refreshed signature also can't resurrect a deleted object. Your bucket lifecycle must cover the queue's actual delay.
What can fail while the run continues?
The worked source is YouTube Video, Playlist & Channel Downloader with Metadata.
Individual video failures don't stop its run. They produce failure rows with failure_code, error, and retryable; failed rows don't claim download links. That's why terminal run status alone can't acknowledge the archive.
Drain accessible datasets from unsuccessful terminal runs too, then report the run failure. Accepted assets remain useful; reviewing failures shouldn't require throwing them away.
Route schema and handoff rejections to an engineering alert. Route source failures to review, where you can decide whether retrying makes sense. Don't repeatedly retry an unsupported asset merely because your queue knows how.
For the worker, alert on authorization failure, missing objects, integrity mismatches, and an access window it can't renew. Send the run ID and reason, rather than signed URLs or raw records.
Before you leave it overnight
- Confirm the worker can authorize the saved bucket and key.
- Keep storage lifecycle longer than the processing delay.
- Replay the saved run before launching another.
- Review rejected records without replacing accepted assets.
- Publish files only after size and checksum verification.
Your queue should carry an object it can retrieve, not a promise that happened to work when you saved it.
Drafted with AI assistance and reviewed for accuracy.
Top comments (0)