Archiving a Telegram channel with Telethon, and re-running it without downloading twice

Downloading a Telegram channel's media the first time is mostly plumbing. The design work is in the second run, a week or a month later, when the goal is to download what was posted since and leave every file already on disk alone.

TGLoader is a free desktop app I built for that job (tgloader.preetam.org). It saves a channel's photos, videos, audio, voice messages, stickers and files to disk, on 64-bit Windows 10 or later and on Apple Silicon Macs. Telethon signs it in as you, using your own api_id and api_hash registered at my.telegram.org, so the app can read a channel only when your account can. Version 0.3.1 handles the second run in two layers, and only one of them is fully wired.

The message cursor

Telethon has the standard tool for fetching only what is new. Message ids in a channel increase over time, and iter_messages (like get_messages) takes a min_id that excludes every message at or below it:

newest = last_seen_id
async for msg in client.iter_messages(channel, min_id=last_seen_id):
    newest = max(newest, msg.id)
    # queue msg.media for download
# save `newest` only after this run's downloads finish

In TGLoader this lives in src/api/channel.py. Each configured channel gets its own Channel object, and its get_new_messages() reads last_message_id from that channel's metadata (0 if absent) and passes it on as min_id. Underneath, the client pages through the channel 100 messages at a time with offset_id, pausing half a second between pages.

In 0.3.1 only the read side is wired. channel.py also has an update_last_message_id() method, and nothing calls it, so the stored id stays at 0 and every run lists the channel again from the newest message back. A second run still doesn't re-download the channel, because the check that does the real work sits one level down, on each file.

The per-file tracker

Each channel folder holds a channel_info.json. When a file finishes, the app records it there under an id built from Telegram's own media id, photo_<id> or document_<id>. Before a file joins the queue, three checks run in order:

1. If the id is in channel_info.json, skip it.
2. If a file already sits at the target path with exactly the size Telegram reports, skip it and add it to the tracker as verified_existing.
3. If a file sits there at a different size, treat it as a partial from an interrupted run, delete it and download again.

The second check means a lost or deleted channel_info.json doesn't cost you a full re-download. It depends on the target path being the same every time, which is why the filename carries the media id: <original name>-document_<id>.mp4 for a named document, photo_<id>.jpg for a photo. Two different files both called video.mp4 land at two paths, and the same file always lands at one.

The tracker can be switched off per channel ("Remember downloaded files for this channel"). The disk check in step 2 runs either way.

Sorting by type

The folder comes from the media attributes Telegram sends with each message. A MessageMediaPhoto goes to media/Pictures/. A document with a video attribute goes to media/Videos/, one with an audio attribute to media/Audio/ (or media/Voice Messages/ when the voice flag is set), and an image/webp or application/x-tgsticker MIME type to media/Stickers/. Everything else is a plain document and goes to documents/, next to media/ rather than inside it.

<channel folder>/
├── channel_info.json
├── documents/
└── media/
    ├── Pictures/
    ├── Videos/
    ├── Audio/
    ├── Voice Messages/
    └── Stickers/

Each channel can point at its own download folder, so this tree lands wherever you chose for that channel.

Big files, and a 2 GB cap that shipped by accident

download.max_file_size is in MB. The app multiplies it by 1024 × 1024 and skips a file only when the limit is above zero and the file is bigger. The default is 0, which applies no cap at all, so a 4 GB video takes the same path as a 40 KB sticker.

Version 0.3.0 got this wrong. A config.yaml with max_file_size: 2000 ended up inside the package, and a packaged build runs with the extracted payload as its working directory, so that file became each new user's config on first launch. Anything over 2 GB was skipped with a line in the log and nothing in the window. 0.3.1 fixed the package, but a config saved by 0.3.0 keeps its value, so if you upgraded, open Settings and set the Max File Size field back to 0.

Three at a time, 1800 seconds each

The defaults are 3 concurrent downloads and a 1800-second timeout per file. Concurrency comes from an asyncio.Semaphore sized to the setting, and the queue is also cut into batches of that size, each awaited before the next one starts. One slow video therefore holds the other two slots idle until it finishes or gives up.

Each attempt gets the full timeout. A failed file gets 3 attempts, 10 seconds apart; when Telegram answers with a flood wait, the app sleeps the seconds Telegram asks for plus 2 instead. The batch as a whole gets three times the per-file timeout before stragglers are cancelled.

The 1800 seconds are wall-clock time. A 4 GB file has to average about 2.3 MB/s to finish inside it, so on a slower connection raise download.timeout_seconds before you queue a channel of long videos.

download:
  max_file_size: 0          # MB, 0 = no limit
  concurrent_downloads: 3
  timeout_seconds: 1800

Limits on a large channel

Listing messages has a 120-second budget. When it runs out, the app logs a warning and carries on with what it fetched, which is the newest part of the channel. Since the cursor isn't saved, the next run lists the same newest stretch again, and the messages older than that window never come in, however many times you run it.

The per-channel date range gets you past it. The To date goes to Telegram as offset_date, so listing starts at the end of that day and walks back. The From date is applied after the fetch, so it trims what gets downloaded but doesn't shorten the listing. For a channel with a long back catalogue, set the To date to an older day, run it, and move the date back again for the next stretch.