DEV Community

Rizwan Saleem
Rizwan Saleem

Posted on

Building a Robust Offline-First Data Sync Layer for Web Apps

#frontend #typescript #webdev #react

Building a Robust Offline-First Data Sync Layer for Web Apps

Building a Robust Offline-First Data Sync Layer for Web Apps

In this tutorial, you’ll learn how to design and implement an offline-first data synchronization layer for a web application. You’ll understand the common pitfalls, pick practical patterns, and see concrete code examples in a small, runnable example. The approach focuses on correctness, conflict resolution, and smooth user experience when network connectivity is flaky.

Illustration: Imagine a note-taking app that must work flawlessly offline, then reliably sync changes when the device goes back online. Users expect no data loss, deterministic conflict handling, and predictable merge behavior.

What you’ll build

  • An offline-first storage layer using IndexedDB for the browser and a small in-memory cache for fast reads.
  • A sync engine that reconciles local mutations with a remote backend, handling conflicts deterministically.
  • A simple REST-like API surface for mutations and queries, with optimistic UI updates.
  • Conflict resolution strategies that you can extend (Last-Writer-Wins, Operational Transformation, and a simplified CRDT approach).
  • Observability hooks to understand sync status, conflicts, and data freshness.

Prerequisites

  • Basic familiarity with JavaScript/TypeScript
  • Understanding of browser storage options (IndexedDB, localStorage)
  • A lightweight backend endpoint to simulate remote mutations (you can mock this in tests)

Architecture overview

  • LocalStore: wraps IndexedDB for durable offline storage and an in-memory LRU cache for fast reads.
  • SyncEngine: coordinates pushing local mutations to the server and pulling remote changes. Tracks a mutation log with unique IDs and timestamps.
  • ConflictResolver: pluggable strategy for resolving conflicts between local and remote versions.
  • API surface: store.query, store.mutate, store.sync, store.observe.

Step 1: Define data model and storage schema

  • Entities: we’ll model a simple “Note” with id, title, content, lastModified, and _version for optimistic concurrency.
  • Local mutation log: records mutations with a unique localId, type (create/update/delete), payload, timestamp, and remoteVersion when synchronized.

Code sketch (TypeScript)

  • Directory layout:
    • src/
    • storage/
      • indexedDbStore.ts
      • cache.ts
    • sync/
      • types.ts
      • syncEngine.ts
      • conflictResolver.ts
    • api/
      • httpClient.ts
      • noteApi.ts
    • models/
      • note.ts
    • app.ts

Key concepts implemented

  • IndexedDB wrapper with basic CRUD
  • In-memory cache
  • Mutation log and versioning
  • Simple conflict resolution strategy

Example: IndexedDB helper
import { openDB, DBSchema, IDBPDatabase } from 'idb';

export interface Note {
id: string;
title: string;
content: string;
lastModified: number;
_version?: number;
}

interface MyDB extends DBSchema {
notes: {
key: string;
value: Note;
indexes: { 'by-lastModified': number };
};
mutations: {
key: string;
value: MutationLogEntry;
indexes: { 'by-synced': boolean };
};
}

export type MutationType = 'create' | 'update' | 'delete';

export interface MutationLogEntry {
localId: string;
type: MutationType;
note: Note;
timestamp: number;
remoteVersion?: number;
synced: boolean;
}

export class IndexedDbStore {
private dbPromise?: Promise>;

async init() {
this.dbPromise = openDB('offline-notes', 1, {
upgrade(db) {
const notesStore = db.createObjectStore('notes', { keyPath: 'id' });
notesStore.createIndex('by-lastModified', 'lastModified');
const mutStore = db.createObjectStore('mutations', { keyPath: 'localId' });
mutStore.createIndex('by-synced', 'synced');
},
});
}

private async getDB(): Promise> {
if (!this.dbPromise) {
await this.init();
}
return this.dbPromise!;
}

async putNote(note: Note) {
const db = await this.getDB();
await db.put('notes', note);
}

async getNote(id: string): Promise {
const db = await this.getDB();
return db.get('notes', id);
}

async deleteNote(id: string) {
const db = await this.getDB();
await db.delete('notes', id);
}

async getAllNotes(): Promise {
const db = await this.getDB();
return db.getAll('notes');
}

// Mutation log helpers
async logMutation(entry: MutationLogEntry) {
const db = await this.getDB();
await db.put('mutations', entry);
}

async getUnSyncedMutations(): Promise {
const db = await this.getDB();
return db.getAllFromIndex('mutations', 'by-synced', false);
}

async markMutationSynced(localId: string, remoteVersion?: number) {
const db = await this.getDB();
const entry = await db.get('mutations', localId);
if (entry) {
entry.synced = true;
entry.remoteVersion = remoteVersion;
await db.put('mutations', entry);
}
}
}

Notes: The idb library is a popular wrapper for IndexedDB. This code sketch focuses on the interface rather than a production-ready implementation.

Step 2: Implement local mutation flow

  • When a user edits a note, we:
    • Update the local store immediately (optimistic UI).
    • Create a mutation log entry with type update and emit to the remote on sync.
    • Update lastModified locally.

Key idea: separate read path from write path. Reads hit the fast cache and then the durable store; writes mutate both.

Sample mutation flow
async function updateNoteLocally(note: Note) {
// Update in-memory cache (if present)
// Persist to IndexedDB
await indexedDbStore.putNote(note);

// Create a mutation entry
const entry: MutationLogEntry = {
localId: generateLocalId(),
type: 'update',
note,
timestamp: Date.now(),
synced: false,
};
await indexedDbStore.logMutation(entry);
}

Step 3: Sync engine design

  • Sync strategy:

    • Push: push unsynced mutations to server in order of timestamp.
    • Pull: fetch latest remote notes and versions; merge with local state.
    • Conflict handling: if remote version conflicts with local, invoke ConflictResolver.

  • Simple API surface:

    • POST /notes/mutations: accepts a batch of local mutations.
    • GET /notes/updates?since=TIMESTAMP: returns remote changes since timestamp.

ConflictResolver interface (simplified)
export interface ConflictResolver {
resolve(local: Note, remote: Note): Note;
}

export class LastWriteWinsResolver implements ConflictResolver {
resolve(local, remote) {
return (local.lastModified >= remote.lastModified) ? local : remote;
}
}

Sample SyncEngine outline
class SyncEngine {
constructor(private store: IndexedDbStore, private api: NoteApi, private resolver: ConflictResolver) {}

async sync() {
// Push local mutations
const mu = await this.store.getUnSyncedMutations();
if (mu.length > 0) {
const payload = mu.map(m => ({
localId: m.localId,
type: m.type,
note: m.note,
timestamp: m.timestamp,
}));
const res = await this.api.syncMutations(payload);
// Mark as synced on success
for (const entry of mu) {
await this.store.markMutationSynced(entry.localId, res?.map(r => r.remoteVersion).find(v => v && v.localId === entry.localId) as number | undefined);
}
}

// Pull remote changes
const latest = Date.now(); // or keep a lastSynced timestamp
const remoteChanges = await this.api.getUpdates({ since: latest - 24h });
for (const remoteNote of remoteChanges) {
  const localNote = await this.store.getNote(remoteNote.id);
  if (!localNote) {
    await this.store.putNote(remoteNote);
  } else {
    const merged = this.resolver.resolve(localNote, remoteNote);
    await this.store.putNote(merged);
  }
}
Enter fullscreen mode Exit fullscreen mode

}
}

Step 4: Conflict resolution strategies

  • Last-Writer-Wins (LWW): compare lastModified timestamps. Simple but can lose user edits if both edited simultaneously.
  • Operational Transformation (OT) or CRDT: more complex; for this tutorial, we outline a simplified LWW plus a timestamp-augmented strategy.
  • Extendable: allow server-generated version vectors to help resolve conflicts with more nuance.

Step 5: Observability and UX considerations

  • Show sync status: idle, syncing, synced, conflicts detected.
  • Non-blocking UI: allow edits to continue while syncing in background.
  • Conflict indicators: highlight notes with potential conflicts and offer user-first resolution in the UI.

Step 6: End-to-end example: a runnable minimal app

  • Create a small React or vanilla JS app that:
    • Renders a list of notes from local store.
    • Lets you add/edit/delete notes.
    • Has a “Sync” button and automatic background sync.
    • Mock remote API to demonstrate push/pull and conflict scenarios.

Pseudocode for a minimal React-like flow
function App() {
const [notes, setNotes] = useState([]);
const [status, setStatus] = useState<'idle'|'syncing'|'synced'|'conflicts'>('idle');

useEffect(() => {
async function load() {
const localNotes = await store.getAllNotes();
setNotes(localNotes);
}
load();
}, []);

async function onSave(note) {
await store.updateNoteLocally(note);
// Optimistic UI already updated by local store read
setNotes(await store.getAllNotes());
}

async function doSync() {
setStatus('syncing');
await syncEngine.sync();
setStatus('synced');
setNotes(await store.getAllNotes());
}

// UI rendering omitted for brevity
}

Step 7: Testing strategies

  • Unit tests for IndexedDbStore: mock IndexedDB via in-memory DB (idb has in-memory testing options or use a mock library).
  • Integration tests for SyncEngine with a mocked API:
    • Push a local mutation, ensure server receives it.
    • Pull remote changes, ensure local store merges correctly.
    • Conflict scenario: local and remote both change same note; ensure resolver applies LWW policy.
  • End-to-end tests: simulate offline then online transitions.

Step 8: Practical tips and pitfalls

  • Be deliberate about mutation ordering. If your server applies mutations with strict ordering, preserve that order locally and push in sequence.
  • Use stable IDs for notes so that local and remote are consistently identifiable.
  • Avoid leaking internal _version or mutation bookkeeping to the UI. Encapsulate in store layers.
  • Prefer a robust serialization format for mutations to simplify retries and debugging.

Step 9: Extending the system

  • Replace LWW with a CRDT for stronger convergence guarantees.
  • Add a reconciliation UI to show conflicts and allow user-driven resolution.
  • Introduce server-side version vectors to improve conflict detection.
  • Add telemetry: sync duration, bytes transferred, conflict count, success rate.

Concrete runnable snippet: simple local mutation and push logic (TypeScript)
type Note = { id: string; title: string; content: string; lastModified: number; _version?: number; };
type MutationType = 'create'|'update'|'delete';
type MutationLogEntry = { localId: string; type: MutationType; note: Note; timestamp: number; remoteVersion?: number; synced: boolean; };

class SimpleStore {
private db: IndexedDbStore;
constructor(db: IndexedDbStore) { this.db = db; }
async saveNote(note: Note) {
note.lastModified = Date.now();
await this.db.putNote(note);
const entry: MutationLogEntry = { localId: crypto.randomUUID(), type: 'update', note, timestamp: Date.now(), synced: false };
await this.db.logMutation(entry);
}
async getAllNotes(): Promise { return this.db.getAllNotes(); }
}

class SimpleSync {
constructor(private store: SimpleStore, private api: { syncMutations: (m: any[]) => Promise<{ remoteVersion?: number }[]>; getUpdates: () => Promise; }, private resolver: ConflictResolver) {}
async sync() {
const unsynced = await this.store['db'].getUnSyncedMutations();
if (unsynced.length) {
const payload = unsynced.map(m => ({ localId: m.localId, type: m.type, note: m.note, timestamp: m.timestamp }));
const res = await this.api.syncMutations(payload);
for (const local of unsynced) {
await this.store['db'].markMutationSynced(local.localId, res.find(r => r.localId === local.localId)?.remoteVersion);
}
}
const remote = await this.api.getUpdates();
for (const r of remote) {
const localNote = await this.store['db'].getNote(r.id);
if (!localNote) {
await this.store['db'].putNote(r);
} else {
const merged = this.resolver.resolve(localNote, r);
await this.store['db'].putNote(merged);
}
}
}
}

Notes on testing locally

  • You can mock the API layer to simulate network failures and latency.
  • Build small unit tests around IndexedDbStore to ensure data integrity.
  • Use an in-memory mock for IndexedDB if you want fast tests without a real browser.

Follow-up questions

  • Would you like this tutorial expanded into a complete, runnable project with a minimal UI (React + TypeScript) and a mock backend?
  • Do you prefer a more sophisticated conflict resolution approach (CRDT or OT) implemented in code, or stay with Last-Writer-Wins for simplicity?

-

Rizwan Saleem | https://rizwansaleem.co

Top comments (0)