A scraper isn’t a monitor: detecting new Telegram posts with Apify and n8n

Bhagya Rathnasekara — Tue, 01 Sep 2026 05:09:38 +0000

My Actor run succeeded. It returned five recent Telegram posts, each with an ID, timestamp, and permalink.

Then I asked the question the automation actually needed to answer:

Which posts are new?

The honest answer was: the current run cannot know by itself.

Nothing had failed. The Actor had done its job. It had retrieved what was visible now. But “new” is not a property contained in one snapshot; it is the difference between the current snapshot and a previous one.

That distinction is the difference between a scraper that runs on a schedule and a monitor you can trust.

In this tutorial, I’ll show the state layer I added around an Apify Actor:

a stable identity for every record;
state partitioned by Telegram channel;
a deliberate first-run policy;
duplicate prevention across executions;
a warning when the polling window may be too small; and
fail-closed handling for missing or malformed state.

The same design works for price trackers, job monitors, lead feeds, inventory checks, and any other workflow that repeatedly asks, “What changed?”

Implementation note: the code and evidence for this article are in the corrected companion package. That workflow deliberately stops at notification candidates; I have not treated untested delivery as finished.

Retrieval and change detection are different jobs

The Actor reads Telegram’s public t.me/s/ preview and returns recent messages in newest-first order. Each item includes fields such as:

{
  "channel": "telegram",
  "id": 454,
  "url": "https://t.me/telegram/454",
  "date": "2026-07-19T17:58:20+00:00",
  "text": "For all the details on these new features...",
  "views": 1180000,
  "scraped_at": "2026-08-05T14:13:17.444080+00:00"
}

That is enough to answer:

What is visible now?

It is not enough to answer:

What appeared since the previous successful check?

The second question requires historical information. That history can live inside an Actor, in a database, in a caller-supplied cursor, or in the workflow that calls the Actor. For this implementation, I kept the Actor stateless and made the downstream n8n workflow own the comparison state.

The boundary looks like this:

Schedule
   │
   ▼
Run Apify Actor ──► current message window
                         │
                         ▼
              validate and partition records
                         │
                         ▼
              compare with durable seen state
                         │
              ┌──────────┴──────────┐
              ▼                     ▼
      notification candidates   status and warnings
              │                     │
              └──────────┬──────────┘
                         ▼
                  persist new state

The experiment that exposed the missing layer

I first called the Actor from an AI client through the Apify MCP server. I retained four one-call conditions:

Condition	What the caller knew before the run	Could it determine new IDs?
Baseline	No previous result	Not asked; the result became the baseline
Same conversation	Previous IDs were still in context	Yes
Fresh session	No reachable previous result	No; newness was undetermined
Fresh session with prior IDs	A small state file plus comparison rules	Yes

All four Actor runs succeeded and returned the same five IDs, 454 through 450. In the fresh session with no baseline, retrieval still worked, but comparison did not.

That result matters because “no new records” and “I cannot determine whether records are new” are not the same answer. A reliable monitor must preserve that distinction.

The runs also exposed a tempting identity bug: every message’s views value changed between two observations, while the IDs stayed stable. Comparing whole records would have classified all five messages as different. Comparing the stable identity classified them correctly as the same messages.

Choose identity before choosing storage

Telegram’s Bot API defines message_id as unique inside a chat. That means id alone is not a reusable global key. The monitor uses this partitioned identity instead:

(normalized channel, message ID)

For example:

function normalizeChannel(value) {
  return String(value ?? '')
    .trim()
    .replace(/^@/, '')
    .toLowerCase();
}

function messageKey(channel, id) {
  const normalizedChannel = normalizeChannel(channel);
  if (!normalizedChannel) {
    throw new Error('Channel is required.');
  }

  if (!Number.isSafeInteger(id) || id <= 0) {
    throw new Error('Message ID must be a positive safe integer.');
  }

  return JSON.stringify([normalizedChannel, String(id)]);
}

views and scraped_at are deliberately excluded. They can change without creating a new message. Text can also be edited, so an ID-only state detects new messages but does not detect edits; edit detection is a separate contract.

This is the first design rule:

Use immutable source identity for new-item detection. Do not let mutable attributes redefine the record.

Store a baseline, not a vague memory

A minimal prior-state file for one channel can be as small as:

{
  "schemaVersion": 1,
  "channel": "telegram",
  "identityField": "id",
  "sourceRunId": "example-prior-run-id",
  "seenMessageIds": [454, 453, 452, 451, 450]
}

The n8n workflow uses the same contract in a more compact form. Its global workflow state is partitioned by normalized channel, and positive integer IDs are stored as merged ranges:

{
  "schemaVersion": 1,
  "channels": {
    "telegram": {
      "initializedAt": "<timestamp from the first successful run>",
      "baselineFloorId": 450,
      "seenRanges": [[450, 454]],
      "highestObservedId": 454
    }
  }
}

Range compression is an implementation optimization, not part of the general monitoring rule. Set comparison requires stable identifiers; range compression additionally assumes suitable integer IDs.

The workflow reads this state with n8n’s $getWorkflowStaticData('global'). n8n documents three important constraints: static data is intended to be small, it is saved after successful triggered production executions, and it can be unreliable under high-frequency execution. That makes it acceptable for this small serialized demonstration, not a universal production database.

For growing state, overlapping executions, or operational querying, move the same identity and comparison contract to a Data Table or external database.

Make the first run a policy decision

On the first successful execution, every returned ID is unseen. Calling all of them “new” may be mathematically consistent, but it is usually terrible monitoring behavior: the workflow sends a burst of alerts for posts that existed before monitoring started.

This implementation defaults to baseline mode:

Fetch the first non-empty window.
Validate it.
Store the IDs.
Send no notification candidates.
Begin new-item detection on the next successful run.

An explicit alert mode remains available for workflows that really do want the first window delivered.

An empty result does not initialize the baseline. Otherwise a transient empty response could create an empty history, and the next normal result could replay the entire window as new.

Compare, classify, then persist

The core comparison is set difference:

new IDs = current IDs − previously seen IDs

But the order around that expression matters:

Validate the response shape and channel partition.
Reject unusable IDs.
Deduplicate the current window by the partitioned identity.
Compare against previously seen identities.
Apply the first-run and historical-backfill policies.
Generate notification candidates.
Persist every valid observed ID, including records filtered out by keywords.

That final choice gives keyword filters at-most-once behavior. Changing a keyword later does not replay posts the workflow already processed.

State corruption also fails closed. If a non-empty state object has no supported schema or contains malformed channel state, the workflow stops instead of silently treating history as empty. Resetting broken state would turn old records into a fresh alert storm.

The complete comparator and the n8n Code-node implementation are in the public companion repository. The comparator is dependency-free and accepts either a bare dataset array or an MCP-style { "items": [...] } envelope.

Treat a full polling window as a warning

Suppose the workflow requests five messages and receives five. It does not know whether exactly five were available or whether older unseen messages fell outside the returned window.

The workflow therefore sets windowSaturated using the raw Actor response count:

const windowSaturated = rows.length >= Number(monitor.limit);

The check happens before deduplication. Duplicate or invalid rows should not hide the fact that the raw window reached its cap.

Saturation means:

The configured window may be too small for the polling interval.

It does not prove that a message was missed. It tells the operator to consider a larger limit, a shorter interval, or a proper catch-up strategy.

The same reasoning applies in the other direction: an ID that was previously seen but is absent from a capped current window is not automatically deleted. It may simply have fallen beyond the window.

MCP proved the information boundary; n8n owns the recurring state

The two caller paths in this project are intentionally separate:

The experiment used an AI client and the Apify MCP server to test what could be determined with and without prior state.
The recurring implementation uses n8n and Apify’s synchronous REST endpoint to fetch dataset items, compare them with n8n workflow state, and emit candidates.

They call the same Actor, but they are not one combined MCP-to-n8n pipeline.

In n8n, the HTTP Request node calls:

POST https://api.apify.com/v2/acts/{ACTOR_ID}/run-sync-get-dataset-items

The Apify API documents this endpoint as running an Actor synchronously and returning its dataset items. Store the Apify API token in an n8n Bearer Auth credential; do not paste it into the workflow JSON.

What I verified

The corrected companion package includes:

a dependency-free comparator with 23 deterministic tests;
an importable n8n workflow whose embedded state engine has 11 deterministic tests;
a fail-closed state schema;
first-run baseline suppression;
per-channel state partitions;
duplicate prevention across triggered runs; and
saturation reporting based on the raw response count.

I also retained two successful triggered n8n executions:

Execution	Result
First triggered run	Created independent baselines for two channels; zero notification candidates
Second triggered run	Reused the persisted state; all ten returned IDs were already seen

The second execution reading the first execution’s state is the important result. It demonstrates persistence across triggered production runs rather than only inside one editor session.

What I did not verify

The evidence has limits:

No genuinely new live Telegram ID appeared during the retained runs. The positive new-ID path is covered by deterministic constructed tests.
The corrected public workflow stops at notification candidates; Telegram delivery is intentionally disabled and untested.
Every observed result window reached the configured limit of five, so the experiment does not establish catch-up completeness.
n8n workflow static data was tested only with small state and serialized executions.
The MCP experiment and n8n workflow ran at different times and are not a one-variable performance benchmark.

Those boundaries do not weaken the state contract. They define exactly what the implementation proves—and what the next production test must cover.

The checklist I now use for every scheduled Actor

Before turning any recurring Actor call into a monitor, I answer these questions:

What is the stable identity? Partition it by source, tenant, channel, or account where necessary.
Which fields are mutable? Keep them out of identity unless change detection is explicitly the goal.
Where does the baseline live? Give one layer clear ownership.
What happens on the first non-empty run? Baseline or alert—never accidental behavior.
What happens when state is missing or malformed? Fail closed instead of replaying history.
Can the polling window saturate? Surface that condition before deduplication.
What does absence mean? In a capped window, it does not prove deletion.
Can executions overlap? If yes, use storage with explicit concurrency guarantees.

A scheduled scraper repeatedly tells you what exists. A monitor tells you what changed without inventing certainty, replaying old records, or hiding gaps.

The difference is not another API call. It is a state contract.

Resources

AI assistance disclosure: I used AI tools during drafting, editing, and QA. I reviewed the final article and supporting evidence and take responsibility for its claims and conclusions.

DEV Community: Bhagya Rathnasekara