DEV Community

Subhash Adhikari
Subhash Adhikari

Posted on

How We Designed a Production-Grade Communication Layer for an Investment Platform

#systemdesign #database #backend #architecture

Most teams building "groups and channels" underestimate how complex it gets when you add paid subscriptions, archive cutoffs, re-subscribe backfill logic, secure media, realtime delivery, and audit/compliance needs.

We recently designed a standalone Communication Layer for an investment platform that needed:

  • 📢 Admin-only broadcast channels
  • 💬 Structured discussion forums
  • 💰 Subscription-aware access control
  • 🔒 Secure media delivery
  • ⚡ Horizontal scalability
  • 🧱 Full decoupling from payments/courses

Here's how we approached it.

📢 Broadcast Channels — Store Once, Read Many

We avoided fanout-on-write (writing per subscriber). Instead:

  • Each post is stored once per channel
  • Each channel has a monotonic sequence number
  • Clients fetch using after_seq
  • Realtime pushes IDs only — (channel_id, seq, post_id)
  • Full content is fetched via REST (with access checks)

Why this matters

Fanout-on-write breaks at scale. With store-once:

  • 1M users → still 1 DB row per post
  • Catch-up is deterministic
  • No duplication bugs
  • Easier pagination

Example feed fetch 

GET /api/mobile/channels/{id}/posts?after_seq=120&limit=20
Enter fullscreen mode Exit fullscreen mode 
SELECT *
FROM posts
WHERE channel_id = $1
  AND seq > $after_seq
ORDER BY seq ASC
LIMIT $limit;
Enter fullscreen mode Exit fullscreen mode

Simple. Predictable. Scalable.

💬 Forums — Clean Separation from Broadcast

We intentionally separated broadcast from discussion.

Structure:

Forum
  └── Topic
        └── Message
Enter fullscreen mode Exit fullscreen mode
  • forums → discussion room
  • topics → thread
  • messages → replies

This prevents:

  • Announcement feeds turning into chat spam
  • Mixing system-critical updates with noise
  • Moderation chaos

Think: Telegram Channel + Linked Group or Reddit Subreddit → Post → Comment

💰 Subscription-Aware Access (Without Coupling Payments)

The most important design decision:

The Communication Layer knows nothing about subscriptions.

Instead, it calls an Access Provider:

canReadChannel(userId, channelId) -> AccessDecision

AccessDecision:
  mode: FULL | ARCHIVE_UNTIL | DENY
  read_windows?: [{start?, end?}]
Enter fullscreen mode Exit fullscreen mode

Why this is powerful

  • Courses can map to channels
  • Subscription passes can expire
  • Premium users can retain archive-only access
  • Re-subscribe can restore with partial backfill
  • All without polluting channel logic

The Communication Layer enforces only: FULL, ARCHIVE_UNTIL, or DENY — and fails closed if the provider is unavailable.

🗂 Handling Archive Cutoffs (Harder Than It Looks)

Example:

  • User subscribes Jan 1
  • Expires Jan 31
  • Re-subscribes Feb 20
  • Admin policy: 7-day backfill

Readable windows become:

(-∞, Jan 31]
[Feb 13, now]
Enter fullscreen mode Exit fullscreen mode

That middle 13-day gap? Hidden.

This required time-window enforcement at the query layer — applied consistently to REST, Realtime, and Signed Media URL issuance. Security invariants must be consistent everywhere.

🔐 Media Security — Best-Effort No-Download

For premium investment content:

  • All files stored privately
  • Signed URLs with TTL ≤ 5 minutes
  • Scope-bound to user + channel + post
  • Issued only after access check 
signed_url(user_id, channel_id, post_id, exp)
Enter fullscreen mode Exit fullscreen mode

We don't pretend screenshots can be stopped — but we prevent casual link sharing.

⚙️ Concurrency — Why MAX(seq)+1 Is Forbidden

Never allocate sequence numbers in application code.

❌ Wrong:

SELECT MAX(seq) FROM posts WHERE channel_id=1;
-- then +1 in code
Enter fullscreen mode Exit fullscreen mode

✅ Correct — atomic counter table:

INSERT INTO channel_seq_counter (channel_id, next_seq)
VALUES ($1, 2)
ON CONFLICT (channel_id)
DO UPDATE SET next_seq = channel_seq_counter.next_seq + 1
RETURNING next_seq - 1;
Enter fullscreen mode Exit fullscreen mode

Sequence allocation and post insert happen in the same transaction. No duplicates. No race conditions.

🔁 Idempotent Publishing — 24h Replay Protection

Admins double-click publish. Networks retry. We implemented:

  • Idempotency-Key header
  • Unique scope: actor + endpoint + channel
  • 24h TTL
  • Same key + same payload → replay original response
  • Same key + different payload → 409 Conflict

Atomic rule: post row + outbox row + idempotency row must commit in the same transaction. No partial success allowed.

📡 Realtime — IDs Only, Never Content

WebSocket model:

  • JWT handshake
  • Authorization on every subscription
  • On publish → send delta: 
{
  "channel_id": 1,
  "seq": 145,
  "post_id": 9001
}
Enter fullscreen mode Exit fullscreen mode

Client must call REST for full post content.

Why? Access can change between delta and fetch. REST re-checks entitlement. This prevents data leaks.

🧱 Reliability Layer

We added:

  • Transactional outbox pattern
  • At-least-once event publishing
  • Deduplication via event_dedupe_key
  • Tombstone deletes (cursor-safe)
  • RPO 15m / RTO 2h
  • Baseline SLO: 99.9% API availability

Deleted posts remain in sequence:

{
  "id": 123,
  "seq": 456,
  "is_deleted": true,
  "body_text": null
}
Enter fullscreen mode Exit fullscreen mode

This preserves pagination integrity.

🧠 Product Decisions That Shaped Architecture

  • Broadcast must stay clean → no replies allowed
  • Premium users may retain archive-only access
  • Re-subscribe should not expose full historical content
  • Access must fail closed
  • Communication must remain independent from monetization logic

Clean boundaries reduce long-term complexity.

🧩 Core Tables (Simplified)

Table Purpose
channels Broadcast channel definitions
posts Channel messages (store-once)
post_attachments Secure media references
forums Discussion rooms
topics Forum threads
messages Thread replies
visibility_rules Access window definitions
idempotency_keys Replay protection
outbox_events Reliable event publishing
*_seq_counter Atomic sequence allocation

Everything supports deterministic ordering, access-window enforcement, and production safety.

🏁 Final Thoughts

Building a Telegram-style system is easy.

Building one that supports paid archive cutoffs, re-subscribe backfill logic, secure premium media, concurrency-safe publish, audit logging, and horizontal scalability — is not trivial.

Key lessons:

  1. Separate broadcast from discussion
  2. Separate communication from monetization
  3. Enforce access at every boundary
  4. Make ordering deterministic
  5. Treat reliability as first-class

If you're building paid communities, investment signal platforms, course-based broadcast systems, or subscription media platforms — I'd love to exchange notes. Drop a comment below 👇

Top comments (0)