DEV Community: Suyash Thakur

How We Built an Expression Engine Inside Image Templates

Suyash Thakur — Thu, 16 Apr 2026 05:22:33 +0000

I'm Suyash, solo founder of Pictify. What started as a side project for GIF templating has slowly turned into a full infrastructure layer for dynamic image generation. I build the whole thing alone: backend, frontend, editor, renderer, and apparently now, a custom expression engine.

I want to tell you the story of the engineering problem that almost made me rethink the entire product, and the ~1,000-line solution that turned it from a rendering utility into something genuinely useful.

The Ticket That Started Everything

Last year, a customer on our Slack channel sent this message:

"Hey, love the API. Quick question: can I show a 'PRO' badge on the card only when the user is on a paid plan? Right now I'm maintaining two separate templates and picking one in my code."

I said no. At the time, Pictify's templates were exactly what every image generation API offers: a canvas with variable placeholders. You mark a text element as {{ name }}, send { "name": "Sarah" } in the API call, and the renderer swaps it in. Find and replace. Done.

That customer was maintaining two templates that were identical except for a tiny badge. But here's the thing. Their next question was worse:

"Also, can I change the background color based on a score? Green for 80+, yellow for 50-79, red for below 50?"

Now they needed six templates. Two badge states × three colors. And they could see where this was going. Every new condition doubled the template count. Their "simple image API" was turning into a combinatorial nightmare managed entirely in application code.

We heard this same pattern from five other customers that month. The template system was the bottleneck, and the bottleneck was that templates had no logic.

The Template Explosion Problem

Let me make this concrete, because it's the core of why we built the expression engine.

On the left: the world every image API lives in. Your backend code picks a template based on conditions, passes variables, and gets an image. Two conditions × three colors = six templates. Add "show testimonial only if available" and you're at twelve. Add "different CTA for mobile vs desktop" and you're at twenty-four.

On the right: what happens when the template itself can evaluate conditions. One template. Any number of rules. Zero code changes. The API caller sends data; the template decides what to show.

This is the product insight that drove the whole project: the people managing templates (marketers, product managers, designers) shouldn't need to file engineering tickets to add personalization logic. The logic should live inside the template.

Why We Couldn't Use Existing Tools

Before building anything, we spent a week evaluating existing solutions. We needed something that could:

Evaluate boolean expressions safely (user-supplied input, no code execution)
Work on canvas object properties (not HTML strings)
Be readable by non-engineers (no JSON blobs)

Here's what we found:

eval() / new Function() was dead on arrival. Users supply expressions. A malicious user could access process.env, require('child_process'), or worse. Even with sandboxing attempts, the attack surface is enormous.

Handlebars, Nunjucks, Liquid are string template languages. They're designed to interpolate variables into HTML. Our templates aren't strings. They're FabricJS canvas objects: a JSON tree of rectangles, text nodes, images, and groups. We needed to evaluate expressions on object properties (should this rectangle be visible? what color should it be?), not render HTML.

JSON Logic / json-rules-engine had the right concept, but the syntax is hostile. {"and": [{">=": [{"var":"score"}, 80]}, {"var": "isPremium"}]} versus score >= 80 && isPremium. The people building templates in our visual editor aren't writing JSON DSLs. They're typing conditions into a text field.

So we built our own. A tokenizer + recursive descent parser that runs in a sandboxed context. No eval, no prototype access, no arbitrary code execution. About 1,000 lines including all 45 built-in functions.

The Architecture: Three Layers

Before diving into the expression engine itself, here's where it fits in the rendering pipeline:

Four steps, running sequentially:

API request arrives with a template ID and variables ({ name: "Sarah", plan: "pro", score: 92 })
Data Mapper extracts and transforms data using JSONPath ($.user.name → "Sarah") with fallback defaults
Expression Engine processes the template: evaluates conditionals, expands loops, interpolates text, applies filters
Canvas Renderer takes the processed FabricJS JSON, renders it with node-canvas, and uploads to S3/CDN

The expression engine adds ~2-5ms to total render time. The bottleneck is always canvas rendering (50-150ms). So we had room to add significant logic without hurting performance.

Building the Tokenizer

The first piece: turning a string like score >= 80 && isPremium into a stream of typed tokens.

The tokenizer does a single character-by-character pass over the expression. No regex. It recognizes:

Identifiers: variable names like score, isPremium, user.name
Operators: ==, !=, >, <, >=, <=
Logical operators: &&, ||, and, or, not
Literals: numbers, strings (single/double quoted), true, false, null
Special characters: (, ), ., ,, |, [, ]

The critical design choice: identifiers resolve against a variable context object, not the JavaScript scope. When the evaluator sees score, it doesn't look for a JavaScript variable called score. It does context["score"]. This is what makes the system safe. The expression can only read values you explicitly pass in.

// This is all the tokenizer does for identifiers:
if (/[a-zA-Z_$]/.test(expr[i])) {
  let value = '';
  while (i < expr.length && /[a-zA-Z0-9_$]/.test(expr[i])) {
    value += expr[i];
    i++;
  }
  // Check for keywords (true, false, null, and, or, not)
  // Otherwise: it's an IDENTIFIER token
  tokens.push({ type: TOKEN_TYPES.IDENTIFIER, value });
}

If someone tries require('fs'), the tokenizer sees require as an IDENTIFIER. The evaluator looks it up in the context, finds undefined, and returns false. No code execution happens.

The Recursive Descent Parser

The evaluator implements a standard recursive descent parser with operator precedence:

parseLogicalOr    →  ||, or     (lowest precedence)
  parseLogicalAnd →  &&, and
    parseNot      →  !, not
      parseComparison → ==, !=, >, <, >=, <=
        parsePrimary  → literals, identifiers, function calls, parens
                        (highest precedence)

Each level calls the next one down, which gives us correct evaluation order automatically. score > 80 && isPremium || isAdmin parses as (score > 80 && isPremium) || isAdmin. No special precedence-climbing code needed.

Here's the heart of the comparison evaluator:

parseComparison() {
  let left = this.parsePrimary();

  if (this.peek()?.type === TOKEN_TYPES.OPERATOR) {
    const operator = this.consume().value;
    const right = this.parsePrimary();

    switch (operator) {
      case '==': return left === right || left == right;
      case '!=': return left !== right && left != right;
      case '>':  return left > right;
      case '<':  return left < right;
      case '>=': return left >= right;
      case '<=': return left <= right;
    }
  }
  return left;
}

We intentionally use both strict and loose equality for ==, so "80" == 80 is true. Template builders don't think about type coercion, and API payloads sometimes send numbers as strings. This "do what I mean" behavior prevents a whole class of support tickets.

Text Interpolation and Filters

Inside text elements, we support double-brace syntax with pipe filters:

Hello, {{ name }}!
Price: {{ amount | currency }}
{{ title | uppercase | truncate: 40 }}

The interpolation engine finds all {{ ... }} blocks, splits on | to separate the expression from filters, evaluates the expression, then pipes the result through each filter.

We ship ~45 built-in filter functions:

// String
uppercase("sarah")         → "SARAH"
titleCase("new york")      → "New York"
truncate(longText, 50)     → "The quick brown fox..."

// Numbers
currency(29.99, "USD")     → "$29.99"
percent(0.85)              → "85%"
round(4.678, 1)            → 4.7

// Dates
date("2026-04-16", "short") → "Apr 16, 2026"

// Arrays
join(["a","b","c"], ", ")  → "a, b, c"
first(items)               → items[0]

// Logic
default(value, "N/A")      → "N/A" if value is null
isEmpty(text)              → true if "", null, or []

We also support inline conditionals inside text. Useful for things like "You have {{ if count == 1 }}1 item{{ else }}{{ count }} items{{ endif }}":

const conditionalRegex = 
  /\{\{\s*if\s+(.+?)\s*\}\}(.*?)(?:\{\{\s*else\s*\}\}(.*?))?\{\{\s*endif\s*\}\}/gs;

result = result.replace(conditionalRegex, (match, condition, truePart, falsePart) => {
  const conditionResult = evaluateExpression(condition.trim(), context);
  return conditionResult ? (truePart || '') : (falsePart || '');
});

This is intentionally not a full template language. No loops in text, no partials, no extends. Text interpolation handles the "put data in a string" case. Structural logic (showing/hiding whole elements, repeating them) happens at the object level. Clean separation.

Conditional Visibility: `showWhen` / `hideWhen`

This is where the expression engine becomes genuinely powerful for image generation. Every canvas object (text, rectangle, image, group) can have a showWhen or hideWhen expression:

{
  "type": "i-text",
  "text": "PRO",
  "showWhen": "plan == 'pro' || plan == 'enterprise'",
  "fill": "#FFD700",
  "left": 400,
  "top": 20
}

At render time, the engine evaluates the expression against the provided variables. If the condition is falsy, the entire object is removed from the canvas before rendering. Not hidden with CSS, not set to opacity 0. Removed from the FabricJS object tree entirely. The rendered image has no trace of it.

function processObjectWithContext(obj, context) {
  if (obj.showWhen) {
    const shouldShow = evaluateExpression(obj.showWhen, context);
    if (!shouldShow) return null; // Object is gone
  }

  if (obj.hideWhen) {
    const shouldHide = evaluateExpression(obj.hideWhen, context);
    if (shouldHide) return null;
  }

  // ... process text interpolation, variable bindings, etc.
  return obj;
}

Remember that customer with six templates? Now they have one. The PRO badge has showWhen: "plan == 'pro'". The green background has showWhen: "score >= 80". The yellow one has showWhen: "score >= 50 && score < 80". Three overlapping colored rectangles, each with a condition. The right one survives; the rest are removed before rendering.

It works recursively too. Groups with showWhen hide all their children. This means you can build complex conditional sections: a "premium features" panel that's a group containing a badge, a description, and a border, all controlled by one showWhen: "isPremium" on the group.

Loops: Repeating Elements

For elements that need to repeat (leaderboard rows, testimonial cards, product grids), we support loop expansion directly on canvas objects:

You define one template object with loopVariable pointing at an array. The engine clones it for each item, adjusts positions based on direction and spacing, and gives each clone its own context:

loopItems.forEach((item, index) => {
  const clonedObj = structuredClone(obj);

  const loopContext = {
    ...variables,
    [itemName]: item,        // current item
    [indexName]: index,       // 0, 1, 2...
    __loopFirst: index === 0,
    __loopLast: index === loopItems.length - 1,
    __loopLength: loopItems.length,
  };

  const processedObj = processObjectWithContext(clonedObj, loopContext);

  // Position based on layout direction
  if (obj.loopDirection === 'vertical') {
    processedObj.top = (obj.top || 0) + (index * (obj.loopSpacing || 50));
  } else if (obj.loopDirection === 'grid') {
    const cols = obj.loopColumns || 3;
    processedObj.left = (obj.left || 0) + ((index % cols) * obj.loopSpacingX);
    processedObj.top = (obj.top || 0) + (Math.floor(index / cols) * obj.loopSpacingY);
  }

  processedObj.id = `${obj.id}_loop_${index}`;
  processedObjects.push(processedObj);
});

Three layout modes: vertical (stack down), horizontal (lay out right), and grid (wrap into rows with configurable columns). Inside each iteration, {{ item.name }} and {{ if __loopLast }} just work because the loop context is merged with the parent variables.

If the array is empty, the object disappears entirely. No "No items to display" placeholder unless you explicitly add one with showWhen: "length(items) == 0".

Expression-Driven Properties

Beyond show/hide, expressions can drive visual properties directly:

{
  "type": "rect",
  "fillExpression": "score >= 80 ? '#22c55e' : '#ef4444'",
  "opacityExpression": "confidence / 100",
  "backgroundColorExpression": "isHighlighted ? '#ffc480' : 'transparent'"
}

The engine evaluates each *Expression property and applies the result:

if (obj.fillExpression) {
  const fillResult = evaluateExpression(obj.fillExpression, context);
  if (fillResult) obj.fill = String(fillResult);
}

if (obj.opacityExpression) {
  const opacityResult = evaluateExpression(obj.opacityExpression, context);
  if (typeof opacityResult === 'number') {
    obj.opacity = Math.max(0, Math.min(1, opacityResult));
  }
}

This means a single rectangle can change color, opacity, and visibility based on data, without duplicating objects on the canvas.

Context Variables: Personalization Without Data Sources

Here's my favorite part. For our Smart Links feature, we needed personalization that doesn't come from the API caller. It comes from the viewer.

When someone clicks a smart link, we resolve variables from the HTTP request itself:

The context variables system extracts geo-IP data, user-agent info, and timezone calculations from the viewer's request:

function buildContextVariables(requestContext) {
  const { device, geo, time, browser, referrer } = requestContext;

  return {
    'ctx.country': geo.country || '',
    'ctx.country_name': COUNTRY_NAMES[geo.country] || '',
    'ctx.city': geo.city || '',
    'ctx.device': device.type || 'desktop',
    'ctx.time_of_day': getTimeOfDay(time.hour),
    'ctx.greeting': getGreeting(time.hour),
    'ctx.referrer': referrer.domain || '',
  };
}

Combined with showWhen, one template becomes infinitely personalized:

"Good morning, San Francisco" on a share card, without the sender knowing the viewer's location
Mobile-specific CTAs with showWhen: "ctx.device == 'mobile'"
Localized content with showWhen: "ctx.country == 'DE'" showing German text
Referrer-aware messaging: different CTA when clicked from Twitter vs. email

The viewer has no idea they're looking at a dynamically generated image. It looks like a static PNG. But it was rendered on-the-fly with their context, in ~200ms, and cached on the CDN.

The Data Mapper: JSONPath to Variables

Before expressions even run, the data mapper transforms messy API responses into clean template variables. We built a custom JSONPath implementation (no external dependency) that handles the shapes we see in practice:

// Simple paths
$.product.name            → "Pictify Pro"

// Array access
$.items[0].price          → 29
$.items[*]                → [all items]

// Fallback syntax
$.stats.revenue || 0      → 0 if field is missing
$.user.bio || "No bio"    → "No bio" if null/undefined/empty

The || fallback is built into the mapper, not the expression engine. This means data source failures degrade gracefully. If an external API returns null for a field, the template gets the default value instead of rendering undefined as text.

const applyFallback = (value, expression, defaults = {}) => {
  if (value !== undefined && value !== null && value !== '') {
    return value;
  }

  if (expression.includes('||')) {
    const [, fallbackPart] = expression.split('||').map(s => s.trim());
    try { return JSON.parse(fallbackPart); }
    catch { return fallbackPart.replace(/^['"]|['"]$/g, ''); }
  }

  return value;
};

This one function eliminates an entire category of broken renders. Before we added it, every time an external data source had a missing field, the image rendered with "undefined" as literal text. Now it fails gracefully.

Performance

The expression engine adds 2-5ms to a typical template with 10-20 objects and a few conditions. Here's where time actually goes:

Phase	Time	What it does
Data mapping	~1ms	JSONPath extraction + fallbacks
Expression engine	~2-5ms	Conditionals, loops, text interpolation
Font registration	~10-30ms	Load custom fonts from S3
Canvas render	~50-150ms	FabricJS → node-canvas → PNG
S3 upload	~20-50ms	Fire-and-forget (non-blocking)
Total	~80-200ms

Design choices that keep the expression engine fast:

No regex for parsing. The tokenizer is a single character-by-character while loop. No match() calls, no backtracking.
No AST construction. The recursive descent parser evaluates directly. No intermediate tree allocation. The result of parseLogicalOr is the final boolean, not a node.
structuredClone() for loops. Faster than custom deep-clone for our object shapes.
Short-circuit evaluation. && and || skip the right side when unnecessary.

What We'd Do Differently

Ternary expressions. We don't support condition ? valueA : valueB in the expression language. Users work around it with three overlapping objects, each with a showWhen. It works but creates canvas clutter. If we were starting over, we'd add ternaries from day one.

A match function. Five overlapping objects for five states is ugly. Something like match(status, "active", "#22c55e", "pending", "#eab308", "error", "#ef4444") would cut object count significantly. This is on our roadmap.

Custom functions per template. Power users want formatters like formatPhoneNumber or calculateDiscount. We currently ship a fixed function set. User-defined functions with sandboxing is the next big expression engine feature.

The Takeaway

An expression engine for image templates is a surprisingly small project. Ours is ~1,000 lines including all 45 built-in functions. But it completely changes what a "template" can do.

Without it, every conditional personalization is an API-side if/else that selects a different template. The template count grows exponentially with the number of conditions.

With it, one template handles every variant. The person managing the template adds a condition in a text field. No code changes. No deploys. No engineering tickets.

If you're building any kind of template-to-image system (OG cards, certificates, share cards, reports), the expression layer is the thing that turns it from a rendering utility into a product. It's the difference between "we have an image API" and "we have a programmable image engine."

Don't skip it.

I'm building Pictify, an image generation API where templates have logic, not just variables. If your team is drowning in template variants or fighting Puppeteer servers, take a look.

I Built a Canva-Like Editor With FabricJS and SvelteKit.

Suyash Thakur — Mon, 23 Feb 2026 21:17:21 +0000

I Built a Canva-Like Editor With FabricJS and SvelteKit. Here's What the Tutorials Don't Tell You.

Every FabricJS tutorial shows you how to add a rectangle to a canvas. Cool. Now build undo/redo. Now add layers. Now make multi-select work without breaking your layer order. Now serialize custom properties that FabricJS silently throws away. Now make it responsive without destroying your export resolution.

That's the gap. The "hello world" is easy. The actual editor is months of "why is this broken" at 1am.

I'm building Pictify — a programmable image engine where you design templates visually and render them via API. Think Canva's editor, but the output isn't a PNG you download. It's an API endpoint that generates images from data. It's launching soon, and the editor is the piece I'm most proud of (and most scarred by).

FabricJS v6. SvelteKit. XState. Here's everything I learned the hard way.

1. FabricJS + SvelteKit: The SSR Trap

First day of the project. I import FabricJS. SvelteKit immediately crashes because FabricJS touches the DOM on import and SvelteKit renders on the server. document is not defined. Classic.

The fix: dynamic import inside onMount.

import { onMount } from 'svelte';

let fabricCanvas;

onMount(async () => {
  const { Canvas: FabricCanvas } = await import('fabric');

  const canvasElement = document.createElement('canvas');
  canvasContainer.appendChild(canvasElement);

  fabricCanvas = new FabricCanvas(canvasElement, {
    width: 1080,
    height: 1080,
    backgroundColor: '#ffffff',
    preserveObjectStacking: true,
    selectionColor: 'rgba(59, 130, 246, 0.05)',
    selectionBorderColor: '#3b82f6',
    cornerSize: 8,
    cornerStyle: 'circle',
    transparentCorners: false
  });
});

Two things to note:

preserveObjectStacking: true — Without this, FabricJS reorders objects when you select them (selected objects jump to the top). That destroys your layer order. Always enable this.

Create the canvas element in JS, don't put it in the template. If you put <canvas bind:this={el}> in your Svelte template, you'll fight lifecycle timing issues. Creating it in onMount guarantees the DOM is ready.

2. Custom Properties Vanish on Serialization

This one cost me three days and most of my sanity.

Pictify templates aren't just shapes on a canvas. Objects carry variable bindings, conditional logic, chart configurations, loop definitions. All custom properties. And FabricJS's toJSON() silently drops every single one of them.

// You add a custom property
rect.isVariable = true;
rect.variableBindings = [{ variableName: 'title', property: 'text' }];

// You serialize
const json = canvas.toJSON();
// json.objects[0].isVariable → undefined
// Gone.

The fix: pass an array of custom property names to toJSON().

const json = canvas.toJSON([
  'id',
  'isVariable', 'variableBindings',
  'isChart', 'chartType', 'chartData', 'chartConfig',
  'isTable', 'tableHeaders', 'tableRows', 'tableData',
  'isQRCode', 'qrData', 'qrConfig',
  'showWhen', 'hideWhen',
  'loopVariable', 'loopDirection', 'loopSpacing'
]);

Every custom property you ever want to survive a save/load cycle needs to be in that array. Miss one and you'll get a bug that only shows up after the user saves and reloads their template. The kind of bug where everything works perfectly in development because you never reload, and then your first real user loses all their variable bindings. Ask me how I know.

I keep this list in a constant and use it everywhere — toJSON(), undo/redo state snapshots, page switching, export. When I add a new custom property to objects, step one is adding it to this array. Before writing any other code. I have been burned enough.

3. The Event Loop From Hell: Why You Need Guard Flags

This is where FabricJS goes from "this is a great library" to "I am in a toxic relationship with this library."

FabricJS fires events in cascades. And if you're not careful, those cascades create infinite loops.

Here's the scenario that made me question my career choices: You listen to object:modified to save undo history. User moves a rectangle. Event fires, you save state. Great.

Now the user hits undo. You load the previous state with loadFromJSON(). FabricJS loads the objects... and fires object:added for each one. Your history listener sees object:added, saves state. Now your undo stack has a new entry that's identical to the one you just loaded. Hit undo again — nothing happens. Your history is corrupted.

The solution is guard flags. Not pretty. Not elegant. But I dare you to find a canvas editor that doesn't have them:

let isPerformingUndoRedo = false;
let isLoadingCanvas = false;
let isPageSwitching = false;

function saveState() {
  if (!fabricCanvas) return;
  if (isPerformingUndoRedo) return;
  if (isLoadingCanvas) return;
  if ($isBatching) return;
  if (isPageSwitching) return;

  historyStack = historyStack.slice(0, historyIndex + 1);
  const state = fabricCanvas.toJSON(CUSTOM_PROPERTIES);
  historyStack.push(state);

  if (historyStack.length > MAX_HISTORY) {
    historyStack.shift();
  } else {
    historyIndex++;
  }
}

function undo() {
  if (historyIndex <= 0) return;

  isPerformingUndoRedo = true;
  historyIndex--;
  const state = historyStack[historyIndex];

  fabricCanvas.loadFromJSON(state).then(() => {
    fabricCanvas.renderAll();
    setTimeout(() => {
      isPerformingUndoRedo = false;
    }, 50);
  });
}

That setTimeout of 50ms at the end? I know what it looks like. I know. But FabricJS fires events asynchronously after loadFromJSON resolves. Without the delay, you clear the flag before the last object:added event fires, and the guard fails. I tried 10ms. Broke on slow machines. 50ms is the magic number. Don't ask me why. I don't know. It works.

Here's every event I listen to and why:

// Selection tracking — sync with properties panel
fabricCanvas.on('selection:created', handleSelection);
fabricCanvas.on('selection:updated', handleSelection);
fabricCanvas.on('selection:cleared', handleSelectionCleared);

// Save to undo history when user finishes a transform
// (fires ONCE after drag/resize ends, not during)
fabricCanvas.on('object:modified', (e) => {
  if (isPerformingUndoRedo || isLoadingCanvas) return;
  saveState();
});

// Track additions/removals for history
fabricCanvas.on('object:added', (e) => {
  if (e.target?.guideline) return;  // Skip alignment guides
  if (e.target?.selectable === false) return;  // Skip helper objects
  if (isDrawing || isPerformingUndoRedo || isLoadingCanvas) return;
  saveState();
});

fabricCanvas.on('object:removed', (e) => {
  if (isDrawing || isPerformingUndoRedo || isLoadingCanvas) return;
  saveState();
});

// Free drawing needs special handling —
// don't save on each stroke, save when path is complete
fabricCanvas.on('mouse:down', () => {
  if (fabricCanvas.isDrawingMode) isDrawing = true;
});

fabricCanvas.on('path:created', () => {
  isDrawing = false;
  saveState();
});

Key insight: object:modified fires once when a transform ends. object:added fires for every object, including during loadFromJSON. You need different guards for different events.

4. Undo/Redo That Doesn't Corrupt Itself

Users don't appreciate undo/redo. Until it's broken. Then it's the only thing they notice.

The basic approach — snapshot the entire canvas as JSON — works fine. Until the edge cases show up. And they always show up.

The Branch Problem

When a user undoes 3 steps and then makes a new change, you need to discard the "future" states:

historyStack = historyStack.slice(0, historyIndex + 1);
historyStack.push(newState);
historyIndex++;

If you forget the slice, the user can redo into a state that has nothing to do with their current canvas.

Batching: AI Makes 15 Changes, User Wants 1 Undo

Pictify has an AI copilot that modifies the canvas programmatically. "Make the background darker and add a headline" turns into 15 canvas operations. Without batching, that's 15 undo entries. The user hits Cmd+Z expecting to reverse "what the AI just did" and instead the headline loses 2px of font size. They hit undo 14 more times before they're back to where they started. Horrible UX.

// In the store
export const batchState = writable({
  isActive: false,
  description: null,
  source: 'user' // 'user' | 'copilot' | 'system'
});

export const historyActions = {
  startBatch: (description, source = 'user') => {
    batchState.set({ isActive: true, description, source });
  },
  endBatch: () => {
    batchState.set({ isActive: false, description: null, source: 'user' });
  }
};

// In the canvas component — subscribe to batch state
batchState.subscribe(($batch) => {
  // When batch transitions from active → inactive, save one state
  if (!$batch.isActive && fabricCanvas) {
    setTimeout(() => saveState(), 10);
  }
});

Usage:

historyActions.startBatch('Copilot changes', 'copilot');
canvas.add(rect1);
canvas.add(rect2);
canvas.add(text1);
// ... 12 more operations
historyActions.endBatch(); // One undo entry for all of it

The saveState() function checks $isBatching and skips if true. When the batch ends, a single save captures the final state.

Per-Page History

If your editor supports multi-page templates (I use this for PDFs), each page needs its own history stack. Otherwise undo on page 3 reverts a change you made on page 1.

let pageHistories = new Map(); // pageIndex → { stack, index }

function savePageHistory(pageIndex) {
  pageHistories.set(pageIndex, {
    stack: [...historyStack],
    index: historyIndex
  });
}

function restorePageHistory(pageIndex) {
  const saved = pageHistories.get(pageIndex);
  if (saved) {
    historyStack = [...saved.stack];
    historyIndex = saved.index;
  } else {
    historyStack = [];
    historyIndex = -1;
    saveState(); // Save initial state for new page
  }
}

5. Grouping and Ungrouping: Coordinate Hell

Grouping objects sounds simple. Right-click, "Group", done. I estimated two hours for this feature. It took two days.

When FabricJS creates a Group, it converts each object's position from canvas coordinates to group-local coordinates. The group's center becomes the origin, and all objects are offset relative to that.

When you ungroup, you need to reverse that transformation. FabricJS has a private method for this:

function ungroupSelected(canvas) {
  const group = canvas.getActiveObject();
  if (group?.type?.toLowerCase() !== 'group') return;

  // Don't ungroup protected elements
  if (group.isQRCode || group.isChart || group.isTable) return;

  const objects = group._objects.concat();

  // This private method restores canvas coordinates
  if (typeof group._restoreObjectsState === 'function') {
    group._restoreObjectsState();
  }

  canvas.remove(group);
  objects.forEach(obj => canvas.add(obj));

  // Re-select all ungrouped objects
  const selection = new ActiveSelection(objects, { canvas });
  canvas.setActiveObject(selection);
  canvas.requestRenderAll();
}

_restoreObjectsState() is undocumented. It's a private FabricJS method that starts with an underscore, which is the JavaScript equivalent of a sign that says "you're on your own, pal." If it gets renamed in a future version, this breaks. But there's no public API for this. I checked. Twice. Asked in the FabricJS GitHub discussions. This is the way.

The same logic applies to double-click-to-ungroup:

fabricCanvas.on('mouse:dblclick', (e) => {
  if (e.target?.type === 'group') {
    const items = e.target._objects.concat();

    if (typeof e.target._restoreObjectsState === 'function') {
      e.target._restoreObjectsState();
    }

    fabricCanvas.remove(e.target);
    items.forEach(obj => fabricCanvas.add(obj));

    const selection = new ActiveSelection(items, { canvas: fabricCanvas });
    fabricCanvas.setActiveObject(selection);
    fabricCanvas.requestRenderAll();
    saveState();
  }
});

6. The Layers Panel: Everything Is Backwards

This one is a rite of passage. Every person who builds a layers panel with FabricJS goes through the same 45 minutes of confusion.

FabricJS stores objects in draw order: index 0 is the bottom-most layer, index N is the top. But every design tool shows layers top-to-bottom — the top layer is first in the list.

This means your UI displays the array reversed. And when you implement drag-to-reorder, "dragging up" in the UI means "moving to a higher index" in FabricJS.

function updateLayers() {
  const objects = canvas.getObjects();
  let result = [];

  // Iterate in REVERSE — top of canvas = top of layer list
  for (let i = objects.length - 1; i >= 0; i--) {
    const obj = objects[i];
    if (obj.guideline || obj.grid) continue; // Skip helper objects

    result.push({
      id: obj.id || `layer_${i}`,
      name: obj.name || `${obj.type} ${i + 1}`,
      type: obj.type,
      visible: obj.visible !== false,
      locked: obj.selectable === false,
      object: obj,
      index: i,
      isGroup: obj.type === 'group' && obj._objects?.length > 0
    });
  }

  layers = result;
}

The reorder logic inverts the "insert before/after" direction:

function reorderLayer(draggedObj, targetObj, insertAfter) {
  const objects = canvas.getObjects();
  const draggedIndex = objects.indexOf(draggedObj);
  const targetIndex = objects.indexOf(targetObj);

  // UI is reversed, so invert the insertion direction
  const effectiveInsertAfter = !insertAfter;

  let newPosition = targetIndex;
  if (draggedIndex < targetIndex) {
    newPosition = effectiveInsertAfter ? targetIndex : targetIndex - 1;
  } else {
    newPosition = effectiveInsertAfter ? targetIndex + 1 : targetIndex;
  }

  newPosition = Math.max(0, Math.min(newPosition, objects.length - 1));
  draggedObj.moveTo(newPosition);
  canvas.renderAll();
}

That !insertAfter inversion is one line of code. It took me 45 minutes of dragging layers up and watching them go down to figure out I needed it. Simple in hindsight. Maddening in the moment.

7. Responsive Canvas: Don't Use FabricJS Zoom

My first attempt at responsive scaling used FabricJS's built-in setZoom(). Seemed logical. It was a disaster. Zoom affects everything — object coordinates, mouse hit detection, exported image dimensions. If you zoom to 50% to fit the canvas on screen, your exported images come out at half resolution. For Pictify, where the entire point is generating pixel-perfect images via API, that's a dealbreaker.

Instead, use CSS transform: scale() on the canvas container:

function updateScale() {
  const wrapper = canvasContainer.parentElement.parentElement;
  const availableWidth = wrapper.clientWidth - 64;
  const availableHeight = wrapper.clientHeight - 160;

  const scaleX = availableWidth / fabricCanvas.width;
  const scaleY = availableHeight / fabricCanvas.height;

  // Fit entirely, with breathing room
  scale = Math.min(scaleX, scaleY, 1) * 0.95;
}

<div style="transform: scale({scale});">
  <div bind:this={canvasContainer}></div>
</div>

CSS scale doesn't affect FabricJS internals. Mouse coordinates still work correctly. Exported images are full resolution. And ResizeObserver keeps it responsive:

onMount(() => {
  const resizeObserver = new ResizeObserver(() => updateScale());
  resizeObserver.observe(canvasContainer.parentElement.parentElement);
  return () => resizeObserver.disconnect();
});

8. State Management: XState as a Thin Router

I know what you're thinking. "XState for a canvas editor? Isn't that overkill?"

Probably. I use it for the editor state, but not in the way any XState tutorial would suggest. There's one state: ready. No complex nested states. No guards. No parallel regions. It's a state machine that never transitions.

const editorMachine = createMachine({
  id: 'editor',
  initial: 'ready',
  context: {
    canvas: null,
    selectedComponent: null,
    activeSidebarTab: 'elements',
    activeRightSidebarTab: 'properties',
    canvasZoom: 100
  },
  states: {
    ready: {
      on: {
        SET_CANVAS: { actions: 'setCanvas' },
        SELECT_COMPONENT: { actions: 'selectComponent' },
        CLEAR_SELECTION: { actions: 'clearSelection' },
        SET_CANVAS_ZOOM: { actions: 'setCanvasZoom' },
        TOGGLE_LEFT_SIDEBAR_TAB: { actions: 'toggleLeftSidebar' },
        TOGGLE_RIGHT_SIDEBAR_TAB: { actions: 'toggleRightSidebar' }
      }
    }
  }
});

So why not plain Svelte stores? Honestly, it would work. But I'm a solo founder building this thing alone. When I come back to the editor code at 2am after a week of working on the API layer, I can look at the machine definition and immediately see every possible thing that can happen. "Which events update selectedComponent?" One file. One glance. That's worth the dependency for me.

The bridge to Svelte's reactivity:

const selectContext = (selector) =>
  derived(editorState, ($state) =>
    $state ? selector($state.context) : null
  );

export const editor = selectContext((ctx) => ctx.canvas);
export const selectedComponent = selectContext((ctx) => ctx.selectedComponent);

Components subscribe to exactly what they need. The properties panel watches selectedComponent. The canvas component watches editor. No prop drilling.

9. Properties Panel: The 200KB Monster

Every canvas editor has that one component. The one file that keeps growing. The one you're slightly afraid to open. For me, it's the properties panel. ~200KB. Handles 8+ different object types. Reactively updates on every selection change. It's the cockpit of the whole editor.

The core pattern:

$: if ($selectedComponent) {
  updateLocalState();
}

function updateLocalState() {
  const obj = $selectedComponent;
  const type = obj.type?.toLowerCase();

  // Common properties
  styles = {
    fill: obj.fill,
    stroke: obj.stroke,
    strokeWidth: obj.strokeWidth,
    width: Math.round(obj.getScaledWidth()),
    height: Math.round(obj.getScaledHeight()),
  };

  // Type-specific properties
  if (type === 'i-text' || type === 'text') {
    content = obj.text;
    // Load font, alignment, spacing...
  }

  if (type === 'rect') {
    cornerRadii = obj.isRoundedRect
      ? obj.cornerRadii
      : { tl: obj.rx || 0, tr: obj.rx || 0, br: obj.rx || 0, bl: obj.rx || 0 };
  }

  if (type === 'image') {
    loadImageFilters();
    currentClipShape = getImageClipShape();
  }

  // Custom element types
  if (obj.isChart) loadChartConfig();
  if (obj.isTable) loadTableConfig();
  if (obj.isQRCode) loadQRConfig();
}

The gotcha: FabricJS's fill can be a string ('#ff0000') OR a Gradient object. If you bind it directly to a color picker, the color picker explodes when it receives an object. You need to handle both cases.

Also, getScaledWidth() vs width. FabricJS objects have a width property and a scaleX property. The actual rendered width is width * scaleX. Use getScaledWidth() in the UI, or your dimension inputs will show wrong numbers when the user scales an object.

10. Complex Objects: QR Codes, Charts, Tables

This is where Pictify gets interesting and where FabricJS gets creative.

Pictify templates can contain QR codes, bar charts, data tables — things FabricJS has no concept of. These are all FabricJS Groups with custom properties glued on top. A QR code is literally a group of hundreds of tiny rectangles (or circles, depending on the style). A bar chart is a group of bars, labels, grid lines. FabricJS doesn't know they're special. My code does.

function createQRCode(config) {
  const { modules: matrix, size: moduleCount } = generateQRMatrix(data);
  const objects = [];

  // Background
  objects.push(new Rect({ width, height, fill: bgColor }));

  // Modules
  for (let row = 0; row < moduleCount; row++) {
    for (let col = 0; col < moduleCount; col++) {
      if (matrix[row * moduleCount + col] === 1) {
        objects.push(new Rect({
          left: offset + col * moduleSize,
          top: offset + row * moduleSize,
          width: moduleSize,
          height: moduleSize,
          fill: fgColor
        }));
      }
    }
  }

  return new Group(objects, {
    id: `qr_${Date.now()}`,
    isQRCode: true,       // Custom flag
    qrData: data,          // Preserved via toJSON([...])
    qrConfig: { fgColor, bgColor, patternStyle }
  });
}

The isQRCode flag does double duty. It prevents ungrouping (imagine a user accidentally exploding a QR code into 500 individual rectangles, then hitting Cmd+Z and praying) and tells the properties panel to show QR-specific controls instead of generic group controls.

And yes — these custom properties only survive serialization if you include them in the toJSON([...]) array from section 2. I learned that lesson once. I don't need to learn it again.

The Cheat Sheet

I'll save you some time. Here's everything from this post in a list you can bookmark:

Always pass custom properties to toJSON([]) — or they vanish on save
Use guard flags on every event listener — isPerformingUndoRedo, isLoadingCanvas, isBatching
Don't use FabricJS zoom for responsive scaling — use CSS transform: scale()
Enable preserveObjectStacking: true — or selections break your layer order
Iterate layers in reverse — and invert insertion direction for reordering
Use _restoreObjectsState() for ungrouping — no public API exists
Batch undo entries for programmatic changes — or your AI copilot creates 50 undo steps
Add setTimeout(fn, 50) after loadFromJSON() — FabricJS fires events asynchronously after the Promise resolves

Building a canvas editor with FabricJS is 20% "put shapes on a canvas" and 80% fighting the stuff on this list. The tutorials get you through the first 20%. Hopefully this post gets you through the rest without losing a week to each one like I did.

What I'm Building With All This

This editor is the front half of Pictify — a programmable image engine I'm launching soon. You design templates in this editor (with variable bindings, conditional logic, loops, charts, QR codes). Then you hit a REST API with JSON data and get rendered images back in ~200ms. Your marketing team designs the template. Your engineering team calls the API. Your AI agent generates 10,000 variations at 2am. Nobody opens a browser.

If that sounds useful to you — or if you're deep in the FabricJS trenches yourself and want to compare battle scars — find me on LinkedIn. I'm always down to talk canvas editors with people who understand the pain.

Drop a comment with the worst FabricJS bug you've encountered. I'll go first: mine was the serialization one. Three days. Three entire days.

Creating In-Video Search

Suyash Thakur — Sat, 01 Jun 2024 17:33:52 +0000

In-video search is ability to search for a specific content within a video. This can include searching for particular words spoken, objects shown or description of a scene.

With the current advancement in transformers the process of in-video search have become more accurate and fairly simple. Although most of the transformers doesn’t have a joint embedding space for multiple modalities but there are few models like Meta’s ImageBind that a joint embedding space between text, image, audio, depth, thermal and IMU, or OpenAI’s CLiP model have joint embedding space between text and image. We can use these models to create a relatively quick and accurate in-video service.

What data does a video consist of ?

A typical video would have the following data with them -:

Frames -: A video is basically a series of the frames. Some of these frames are complete frames called i-frames and others will be partial frames, also known as p-frames and b-frames, which contain only the changes from the previous frame.
Audio -: A video also might have a audio channels. These channels can contain different types of audio tracks, such as dialogue, music, and sound effects.
Subtitles/Transcript -: Many videos include subtitle or closed caption tracks or the transcript could be derived from the audio. This data is important for us to query a dialogue in the video.

We will process these data independently using different techniques.

Processing Frames

Typically frames in a video is of three types -:

i-frame -: They are complete, self-contained frames that do not rely on other frames for decoding.
p-frame -: These frames rely on previous frames for decoding. They are compressed by only storing the differences between the current frame and the previous frame.
b-frame -: These frames rely on both previous and future frames for decoding. They are compressed by storing the differences between the current frame and the previous frame, as well as the differences between the current frame and the next frame.

For our use-case we will rely on i-frame since they serve as the reference point for other type of frames. We will create vector representation of p-frames and store it in our vector DB with metadata such as timestamp of the frame using the following steps -:

Extract i-frames -: We can use ffmpeg to extract i-frames of a video.
Discard similar i-frames -: Often, two i-frames can be very similar to each other. Indexing these similar frames in our vector DB would waste resources. We will discard similar frames by calculating the Structural Similarity Index (SSIM) between two frames. SSIM measures the similarity between two images using luminance, contrast, and structure. For more information, use this link.
Generate Embeddings: Once we have discarded the similar i-frames, we will pass the remaining images to the ImageBind model to generate the associated embeddings and store them in our vector DB.

Processing Audio

Processing audio is optional and often does not significantly enhance results, as audio typically doesn't provide unique information beyond what is already available in video frames or transcripts. However, if your use case benefits from audio data, this method can be utilized.

To process audio, the first step is to divide it into smaller clips. Instead of clipping the audio at constant intervals, which might not yield the best results, we will smartly segment the audio so that each chunk retains complete contextual information.

Before that we will need to get familiar with two terms -:

Spectral Contrast -: The audio is divided into different sections based on pitch. Imagine dividing the music into high-pitch, middle-pitch, and low-pitch parts. In each section, we look for the loudest parts (peaks) and the quietest parts (valleys). We measure how big the difference is between the loudest and quietest parts in each section. This difference is the "contrast.” We will then calculate the mean of spectral contrast of different sections.
RMS Energy -: RMS (Root Mean Square) Energy is a measure of the power or loudness of the audio signal. This gives us an idea of how much energy the audio signal has, which correlates to its perceived loudness. It is calculated using the following formulae -:

\text{RMS} = \sqrt{\frac{1}{N} \sum_{i=1}^{N} x_i^2}

We will process each frame (a continuous segment of the audio signal), calculating and normalising its spectral contrast and RMS energy. By comparing these features with the previous frame, we will determine whether it belongs to the same clip or a new one. A new clip will be created if the following condition is met.

SC_2 - SC_1) +(RMS_2 - RMS_1) > Threshold

Once we have the clips we will create the embeds associated with it using ImageBind and store it in out vector database along with the metadata like timestamp of the clip.

Processing Text Data

Video data may include a subtitle file, or a transcript can be generated from the video. Working with text data is typically simpler. To preserve context and minimize information loss, we'll segment the text data with some overlap. Then, we'll generate the embeddings using ImageBind and store them in VectorDB, marking each with its corresponding timestamp.

Querying For Video Clip

The process of querying a video clip begins with a prompt provided by the user. Upon receiving the query, we follow these steps:

We generate the embed for the prompt and query our vector database to find the closest match. This could be a text, frame, or audio embed.
If the embed is text-based, we calculate the cosine similarity of the next and previous text embeddings. We continue to iterate in both directions until the cosine similarity falls below a certain threshold. This threshold can be calculated dynamically (I created an npm package for this purpose. You can check it out here). The start time of the first chunk becomes the start time of the video clip, while the end time of the last chunk becomes the end time.
If the embed is audio-based, the process is more straightforward. We simply use the start time of the audio clip as the start of the video, and the end time of the audio clip as the end of the video.
If the embed is of a frame, then the timestamp of the previous frame in vectorDB is the start time of the video clip, and the timestamp of the next frame is the end time of the video clip.
Once we have the start and end times for the video clips, we will generate the clip from the original video and provide the output.

In conclusion, with the help of advanced transformer models like ImageBind, creating a reliable and efficient in-video search service is more attainable than ever. By processing and embedding different data modalities from videos - frames, audio and text - we can create a comprehensive search system that caters to a wide array of user queries. Remember, the process highlighted in this article is a guide - the thresholds, processing techniques, and models can be tweaked and optimized as per your specific use case. Happy coding!

Converting HTML to Image. and GIF

Suyash Thakur — Sat, 16 Mar 2024 22:55:39 +0000

Introduction

Recently I created pictify.io a platform to convert HTML to Image and Gif. In this post I will try to give an overview of the working of it.

There are 4 main component of the system.

The HTML code
The server with puppeteer instance
FFMPEG
AWS S3

The Process

The user sends the request to server with HTML code. The CSS and Javascript must be included in the HTML or should be a publicly accessible link. Once the server receives the request it renders the the page using puppeteer a famous headless browser. The puppeteer waits till there are no more than 0 network connections for at least 500 milliseconds.

Using screenshot() method provided by the API the image is being captured. The captured image is piped to sharp a nodeJS image processing library to compress it.

Then the image is uploaded to a S3 bucket which is then served by cloudfront CDN. The server responds with the cloudfront URL to user.

This process becomes a little challenging when converting an HTML file with animation to GIF.

Creating GIF

The gif creation process is same as image till the html is rendered. Once the HTML is rendered the we identify the node from the DOM tree which has animation attached to it. Once we receive the animating node we start capturing image with the same screenshot() method but we also seek animation to get the next frame. We repeat this process till the animation is completed.

At the end of the process we have images as frames for the GIF. We then use FFMPEG to convert these images to GIF. Since GIFs are larger in size we use sharp to resize and compress the individual frame.

The process after this is same as for image. We upload the image to S3 bucket and send the cloudfront URL to the user.

Thanks for taking the time to read this. I hope this was helpful. If you have any doubt or what to know any process in details do let me know about it in comments.