DEV Community: Ejeh Daniel

How I Built a Tech Event Discovery Platform with Real-Time Scraping

Ejeh Daniel — Sun, 23 Nov 2025 13:00:33 +0000

I'm a software developer, and I've been attending tech events for over three years now. I've used platforms like Luma and Eventbrite to find events, but there's always been one problem that frustrated me. The noise.

Most event listing sites list cool tech events, but they also mix in so many non-tech events that it becomes overwhelming. When I'm looking for a React workshop or an AI conference, I don't want to scroll through cooking classes and yoga sessions. I remember searching for "JavaScript meetups" and getting results for wine tasting events and fitness bootcamps mixed in. The problem was clear. I wanted a clean, focused experience that only showed tech events.

At first, I just thought about it, but I didn't know how to approach building it. Then recently, I had to automate a dataset ops workflow at work. I needed to pull product details, categorize products, process and clean data, and save results. That's when I realized scraping could be useful here too. I've always loved building software solutions I wish I had. It's personal. I looked into it, decided it could be done, and I got started.

The goal was straightforward. I wanted a platform that caters exclusively to tech events with a clean interface and smooth experience. Success would be searching for tech events and getting relevant results without the noise, delivered quickly. I built it in a week, focusing on real-time scraping as users search rather than pre-scraping everything. I was curious about making scraping fast and reliable on demand.

Building the Architecture

The core idea is simple. Scrape tech events from platforms like Luma and Eventbrite, save them to the database, and then list them. Once events are in the database, I can filter, search, and display them without hitting the source platforms every time.

From there, I built the architecture around database-first search. When someone searches for events, it checks the database first. If results exist, they're served instantly. Only when nothing's in the database does it trigger a background scraping job. Most searches are fast this way, no waiting around for scraping when the data already exists.

Here's how the database-first lookup works in practice:

// Check database first
try {
  const dbResults = await searchDatabase(searchQuery, filters, DEFAULT_DB_SEARCH_LIMIT)

  if (dbResults.events.length > 0) {
    return NextResponse.json({
      success: true,
      source: 'database',
      events: dbResults.events,
      total: dbResults.total,
    })
  }
} catch (dbError) {
  console.error('Database search failed:', dbError)
}

For async processing, I went with BullMQ and the job queue pattern. When a search has no database results, the API creates a job and returns immediately with a job reference. A separate worker process handles the scraping in the background while the frontend maintains a connection to track job completion. This decouples the search request from the scraping operation. The user gets an immediate response, and scraping happens independently without blocking the request cycle. It's tempting to scrape synchronously, but that defeats the purpose of having a database layer in the first place. This way, you get responsiveness without sacrifice.

When there are no database results, the API queues a scraping job and returns immediately:

// Create unique job ID
const jobId = `search-${Date.now()}-${crypto.randomBytes(4).toString('hex')}`

await scrapingQueue.add('scrape-events', {
  jobId,
  query: searchQuery,
  platforms: searchPlatforms,
  city: searchCity,
})

return NextResponse.json({
  success: true,
  jobId,
  status: 'running',
})

To keep the database fresh, I set up daily scraping runs at 6 AM UTC using Vercel cron jobs. These runs hit all the event platforms systematically, ensuring the database stays current with new events. The cron jobs use Next.js after() to process scraping operations asynchronously, so the endpoint responds immediately while the work happens in the background. This keeps the database continuously updated without manual intervention or blocking request handlers.

For scraping, I started with Apify because it's battle-tested and handles most edge cases out of the box. It works well, but it introduces recurring costs and adds a dependency I can't control directly. If the platform changes and Apify's selectors break, I'm waiting on their updates.

That's when I added Puppeteer as a fallback. It's lightweight, gives me full control over selectors and timing, and with the stealth plugin, it handles anti-bot detection just fine. So now I run with both: Apify handles the heavy lifting most of the time, but if selectors fail or I need to adapt quickly, Puppeteer takes over. That dual approach gives me reliability plus flexibility. The trade-off is managing two tools instead of one, but for something as brittle as web scraping, having a fallback mechanism actually reduces risk.

Here's how I configure Puppeteer with stealth mode and handle browser initialization:

import puppeteer from "puppeteer-extra";
import StealthPlugin from "puppeteer-extra-plugin-stealth";
import { z } from "zod";
import { prisma } from "./prisma";

puppeteer.use(StealthPlugin());

// user agents to rotate
const USER_AGENTS = [
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36",
    "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36",
    "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36",
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:121.0) Gecko/20100101 Firefox/121.0",
];

// Get random user agent
function getRandomUserAgent(): string {
    return USER_AGENTS[Math.floor(Math.random() * USER_AGENTS.length)];
}

// Create browser with stealth configuration
async function createBrowser() {
    // Try multiple possible paths for Chromium
    let executablePath = process.env.PUPPETEER_EXECUTABLE_PATH;

    if (!executablePath) {
        const fs = require('fs');
        const possiblePaths = ['/usr/bin/chromium', '/usr/bin/chromium-browser', '/usr/bin/google-chrome'];
        for (const path of possiblePaths) {
            try {
                if (fs.existsSync(path)) {
                    executablePath = path;
                    break;
                }
            } catch {
                continue;
            }
        }
    }

    return await puppeteer.launch({
        headless: true,
        executablePath, // Use system Chromium in Docker
        args: [
            "--no-sandbox",
            "--disable-setuid-sandbox",
            "--disable-blink-features=AutomationControlled",
            "--disable-features=IsolateOrigins,site-per-process",
            "--disable-web-security",
            "--disable-dev-shm-usage",
            "--disable-gpu",
            "--disable-software-rasterizer",
            "--disable-extensions",
            "--no-first-run",
            "--disable-default-apps",
            "--disable-background-networking",
            "--single-process",
            "--disable-zygote",
            "--disable-crash-reporter",
            "--disable-breakpad",
            "--disable-background-timer-throttling",
            "--disable-backgrounding-occluded-windows",
            "--disable-renderer-backgrounding",
        ],
        ignoreDefaultArgs: ["--disable-extensions"],
    });
}

The batch scraping function navigates to the page, waits for content to load, and scrolls to trigger lazy loading. This approach handles dynamic content that loads as you scroll. It's important for platforms like Eventbrite where most events are loaded on demand rather than served upfront. The scraper tries multiple selectors to find event cards, accounts for network errors with exponential backoff, and detects when it's being blocked before wasting resources.

Here's how I handle navigation with retry logic for network errors:

let retries = 3;
let lastError: Error | null = null;

while (retries > 0) {
  try {
    await page.goto(searchUrl, {
      waitUntil: "domcontentloaded",
      timeout: 60000,
    });
    break;
  } catch (error: any) {
    lastError = error;
    retries--;
    if (
      error.message?.includes("ERR_NETWORK_CHANGED") ||
      error.message?.includes("net::ERR") ||
      error.message?.includes("Navigation timeout")
    ) {
      if (retries > 0) {
        await delay(2000 * (4 - retries));
        continue;
      }
    }
    throw error;
  }
}

Once the page loads, I scroll to trigger lazy loading and extract events with multiple selector fallbacks:

await page.evaluate(async () => {
  await new Promise<void>((resolve) => {
    let totalHeight = 0;
    const distance = 100;
const timer = setInterval(() => {
      const scrollHeight = document.body.scrollHeight;
      window.scrollBy(0, distance);
      totalHeight += distance;

      if (totalHeight >= scrollHeight) {
        clearInterval(timer);
        resolve();
      }
    }, 100);
  });
});

const selectors = [
  'article[class*="event-card"]',
  'div[class*="event-card"]',
  '[data-testid="event-card"]',
  "article.eds-event-card-content",
];

for (const selector of selectors) {
  const elements = document.querySelectorAll(selector);
  if (elements.length > 0) {
    // Extract events...
    break;
  }
}

// Detect if we're being blocked
const bodyText = document.body?.textContent || "";
if (bodyText.includes("blocked") || bodyText.includes("captcha")) {
  console.error("[PUPPETEER] Possible blocking detected");
}

The Redis setup ended up being one of those decisions where two tools actually work better than one. I'm using ioredis for the BullMQ connection because job queues need persistent, reliable connections. For caching though, I switched to Upstash Redis. It's HTTP-based and built for serverless, so it plays nice with Vercel. Two clients, two purposes, and together they give me reliable job processing plus fast caching that scales.

What It Actually Does

Here's how it works in practice. When you search for "React workshops in Seattle," the system checks the database first. If matching events are already saved, you get them instantly. No waiting, no scraping. Results show up immediately with all the details. Title, date, venue, price, and a link to register.

Image: Event listing page with search results from database

But what if you're searching for something not yet in the database. That's when the job queue kicks in. The API creates a scraping job and returns immediately with a job ID. A worker process starts scraping Luma and Eventbrite in the background while the frontend tracks the job status. Once the worker finds events, they get saved to the database and the frontend automatically updates with the results. From your perspective, you search, see a loading state, and then results appear. The page stays responsive throughout.

GIF: Live scraping in progress with results appearing as they're found

For daily updates, Vercel cron jobs run at 6 AM UTC and systematically scrape all the event platforms. Instead of a single broad search, I run multiple targeted queries per platform. "ai," "data science," "python," "reactjs," "javascript," "machine learning." This multi-query approach gives much better coverage than casting a wide net. For each city and platform combination, I deduplicate by URL to avoid saving duplicate events. When you wake up, fresh events are already there without any manual intervention.

What I Learned and What Went Wrong

Anti-bot detection was the first real hurdle. Eventbrite and Luma both have systems that detect automated browsing, and my initial Puppeteer setup got blocked almost immediately. I thought the stealth plugin would be enough, but it wasn't. I had to rotate user agents, override the webdriver property, set realistic viewports, and add random delays between actions. Even then, I still hit rate limits occasionally. The bigger lesson is that anti-bot systems are constantly evolving. What works today might not work next month when they update their detection. This is why having Apify as a fallback matters. If my Puppeteer setup breaks, I can switch strategies without rewriting the whole system.

Docker and Chromium compatibility turned into its own problem. When I first tried running Puppeteer in a Docker container, Chromium would crash with cryptic errors about crashpad handlers and zygote processes. I spent hours debugging before realizing I needed specific flags like --single-process and --disable-zygote for Docker environments. The executable path detection was also tricky. Different systems have Chromium in different locations, so I built fallback logic to find it automatically. This taught me that serverless deployment has its own constraints. You can't just run browser automation anywhere. You need to know your environment and adapt to it.

Data quality was messier than I expected. Event titles are inconsistent, dates come in different formats, and some events have missing fields. I use Zod schemas for validation, but incomplete data still slips through. Deduplication helps, but I've seen duplicate events when the URLs are slightly different or when the same event appears on multiple platforms with different identifiers. This is the reality of aggregating data from multiple sources. There's no perfect deduplication strategy. For a personal project, it's acceptable. For production, I'd need more sophisticated data cleaning, probably a dedicated validation pipeline. The irony is that the scraping is the easy part. Making the data consistent is where the real work is.

Conclusion

Tech event discovery should be simpler than it is. These Platforms have APIs, but they're not free to work with. Scraping fills the gap, but it's fragile. For now, Tech Event Vista solves my problem. I can find tech events without the noise.

Building this project revealed something important. The real challenge isn't technology. We have powerful tools like Next.js, Puppeteer, BullMQ, and Redis that make something like this possible in a week. The hard part is everything else. Anti-bot systems that constantly evolve, data quality across multiple sources, and the constant maintenance that scraping demands.

If you find it useful, great. The code lives on GitHub. Fork it, run it, break it, fix it. 🚀

You Don't Always Need useCallback and useMemo

Ejeh Daniel — Tue, 04 Nov 2025 01:35:46 +0000

React Hooks have revolutionized how we manage logic and side effects in functional components, but two hooks—useCallback and useMemo—are frequently misunderstood and overused in the process of obsessively trying to memoise.

This article explains when memoisation actually helps, when it's counterproductive, and which modern alternatives provide better solutions.

Why do we memoise in the first place? In React, more often than not, we decide to create a memoised version of a value with useMemo or a memoised function with useCallback in order to skip re-rendering a sub-tree. Now re-rendering a sub-tree is usually slow in React, so we'll prefer to skip any unnecessary re-renders. In order to optimise performance, we mostly reach out for React.memo to skip re-rendering a component when its props are unchanged.

Here's the catch: when we pass a function or a non-primitive value as props to this memoised component, we need to make sure they have stable references. Before React skips re-rendering a sub-tree, it first compares the props of the memoised component to ascertain if they have changed. So if our props always have unstable references, the value of optimisation is lost because then our memoised component will not get memoised at the end of the day.

function Nah() {
  return (
    <MemoizedComponent
      value={{ hello: 'world' }} // New object reference each render
      onChange={(result) => console.log('result')} // New function reference each render
    />
  )
}

function Okay() {
  const value = useMemo(() => ({ hello: 'world' }), [])
  const onChange = useCallback((result) => console.log(result), [])

  return <MemoizedComponent value={value} onChange={onChange} />
}

This is just one of the reasons we try to memoise; another reason is to prevent effects from firing too often. When you pass a prop as a dependency to an effect, React does the same thing it does: it compares the dependency to ascertain if the effect needs to be re-run. In all of these, React is trying to do the same thing, which is to keep the references stable by caching them. So it's normal that useCallback and useMemo always come through for this reason.

When Memoisation Fails

When memoisation isn't solving one of the two problems mentioned above, it becomes useless noise. So there are cases where striving for stability in references is important, and there are others where it's pointless. Let's see some cases where the use of useCallback and useMemo becomes redundant:

Case 1: Memoising Unmemoized Components (Zero Performance Gain)

The common mistake is using useCallback or useMemo on a prop being passed to an unmemoized functional component or a React built-in component like a button.

function Okay() {
  const value = useMemo(() => ({ hello: 'world' }), [])
  const onChange = useCallback((result) => console.log(result), [])

  return <Component value={value} onChange={onChange} />
}

Now it’s important to note that the custom component and the button don’t care if the props have stable references. So if your custom component is not memoised using React.memo, it really doesn't care about your referential stability. You gain no performance improvement while introducing unnecessary boilerplate.

Case 2: The Broken Dependency Chain

It's rarely a good idea to add non-primitive props like objects or functions to your dependency arrays. Why? Because your component has no control over whether the parent keeps those references stable

function NotOkay({ onChange }) {
  const handleChange = useCallback((e: React.ChangeEvent) => {
    trackAnalytics('changeEvent', e)
    onChange?.(e)
  }, [onChange])

  return <SomeMemoizedComponent onChange={handleChange} />
}

This useCallback is not useful in this context, or at best, it depends on how consumers will use this component. In all likelihood, there is a call-side that just invokes an inline function:

<NotOkay onChange={() => props.doSomething()} />

The parent's unstable prop reference forces the child's useCallback to invalidate its cache on every render. The only way a developer who writes this code could know that not wrapping the prop in useCallback themselves breaks some internal memoisation is if they drill down into the component to see how the props are being used.

That's a horrible developer experience.😰 The only other popular option will be to memoise everything, always reach out for useCallback and useMemo, but these aren't great practices, but rather create overhead under the hood.

Now, let's examine a real-world example that demonstrates why excessive memoisation becomes problematic. Even in well-architected codebases, I've seen cascading memoisation failures like this.

A Real Life Example

The WordPress Gutenberg project is the block editor powering millions of WordPress sites. It went through a major cleanup of unnecessary memoisation. They removed useCallback from multiple components because the function wasn't passed to any hook or memoised component that might require a stable reference.

Here's a pattern similar to what they found - a block settings component:

function BlockSettings({ onUpdate, settings }) {
  // Unnecessary useCallback - passed to regular component
  const handleChange = useCallback((key, value) => {
    onUpdate({ ...settings, [key]: value });
  }, [settings, onUpdate]);

  return (
    <SettingsPanel>
      <TextControl
        onChange={(val) => handleChange('title', val)}
      />
      <ToggleControl
        onChange={(val) => handleChange('visible', val)}
      />
    </SettingsPanel>
  );
}

Spot the problem? 🔍 The TextControl and ToggleControl aren't memoized components. They're going to re-render whenever BlockSettings re-renders anyway. The useCallback achieves absolutely nothing here.

function BlockEditor({ block }) {
  // Another useCallback
  const handleUpdate = useCallback((newSettings) => {
    updateBlock(block.id, newSettings);
  }, [block.id]);

  // This creates a new object every render!
  const settings = {
    title: block.title,
    visible: block.visible,
    layout: block.layout
  };

  return <BlockSettings settings={settings} onUpdate={handleUpdate} />;
}

Let's trace the cascade:

settings object - Created fresh every render (new object reference)
handleChange in BlockSettings - Depends on settings, so recreated every render despite useCallback
handleUpdate in BlockEditor - Only stable if block.id doesn't change, but...
settings breaks the chain - The moment settings is a new object, everything downstream fails

Even though we have two useCallbacks, both are completely useless at this point because settings isn't memoised.

The WordPress Gutenberg team's solution? They removed the unnecessary useCallbacks entirely. The code became simpler and more maintainable, with zero performance impact because the memoisation was never working anyway.

This same pattern appears everywhere in React codebases - well-intentioned useCallbacks that achieve nothing because somewhere in the dependency chain, a new object or array is created. Let's take a look at better ways to handle this.

Escaping the Dependency Trap

The immediate goal is to stabilise an effect dependency and avoid breaking memoizations in our code; there are currently far better ways than always reaching for manual useCallback and useMemo. These methods will allow you to access the latest state/props inside an effect without forcing the effect to re-run.

The Ref Pattern 🎯

This pattern is pretty straightforward; it aims to solve our problem of using unstable references and causing the effect to re-run unnecessarily. What we do here is store the value we want to access in a ref, and update it on every render, that’s it:

function useDragHandlers(draggableId: string, callbacks: DragCallbacks) {
  // Store callbacks in a ref
  const callbacksRef = useRef(callbacks);

  // Update ref every render (cheap operation)
  useEffect(() => {
    callbacksRef.current = callbacks;
  });

  // Handlers never change, but always use the latest callbacks
  const onDragStart = useCallback((event) => {
    callbacksRef.current.onStart?.(draggableId, event);
  }, [draggableId]); // Only draggableId in dependencies!

  const onDragEnd = useCallback((event) => {
    callbacksRef.current.onEnd?.(draggableId, event);
  }, [draggableId]);

  useEffect(() => {
    const element = document.getElementById(draggableId);
    element?.addEventListener('mousedown', onDragStart);
    element?.addEventListener('mouseup', onDragEnd);

    return () => {
      element?.removeEventListener('mousedown', onDragStart);
      element?.removeEventListener('mouseup', onDragEnd);
    };
  }, [draggableId, onDragStart, onDragEnd]); // These never change now

  return { onDragStart, onDragEnd };
}

// Now consumers don't need ANY memoisation
function DraggableCard({ id, onDragStart, onDragEnd }) {
  // Just pass callbacks directly - no useMemo needed!
  const handlers = useDragHandlers(id, {
    onStart: onDragStart,
    onEnd: onDragEnd
  });

  return <div {...handlers}>Drag me</div>;
}

The ref always points to the latest callbacks, the handlers never change, you've got stable dependencies!, and we've eliminated the entire fragile memoisation chain.

Many popular component libraries use this pattern to avoid forcing consumers to memoise their callbacks. For example, Headless UI (by Tailwind Labs) and Radix UI both store callback refs internally to ensure components work correctly regardless of whether users memoise their props. Imagine if these libraries required consumers to memoise their options manually, it would be a terrible developer experience

useEffectEvent🆕

React 19.2 recently introduced useEffectEvent, a hook that helps you separate non-reactive logic from effects, avoiding stale closures and unnecessary effect re-runs. In short, it's used when you need imperative access to the latest value of something during a reactive effect without explicitly forcing the effect to re-run. This is now the recommended solution for the pattern described above.

Here's how you can refactor with useEffectEvent:

function useDragHandlers(draggableId: string, callbacks: DragCallbacks) {
  // useEffectEvent handles callbacks that "aren't reactive"
  const handleDragStart = useEffectEvent((event) => {
    callbacks.onStart?.(draggableId, event);
  });

  const handleDragEnd = useEffectEvent((event) => {
    callbacks.onEnd?.(draggableId, event);
  });

  useEffect(() => {
    const element = document.getElementById(draggableId);
    element?.addEventListener('mousedown', handleDragStart);
    element?.addEventListener('mouseup', handleDragEnd);

    return () => {
      element?.removeEventListener('mousedown', handleDragStart);
      element?.removeEventListener('mouseup', handleDragEnd);
    };
  }, [draggableId]); // Only draggableId needed!

  return { onDragStart: handleDragStart, onDragEnd: handleDragEnd };
}

This makes handleChange non-reactive; it always "sees" the latest values of onUpdate and settings, and it's referentially stable between renders. The best of all worlds, without having to write a single useless useCallback or useMemo.

For new code on React 19.2+: Use useEffectEvent instead of the Latest Ref pattern.

For existing codebases or older React versions: The Latest Ref pattern remains a solid solution.

When Should You Use useCallback and useMemo?🤔

Now that you've seen the problems and solutions, here's a simple guide for evaluating the use of useCallback and useMemo in your code:

Is the function passed to React.memo() component?
- If NO — ❌ Don't use useCallback
- If YES — Continue to #2
Have you measured that the component is slow (>16ms render)?
- If NO — ❌ Don't use useCallback
- If YES — Continue to #3
Are all dependencies stable (not props or changing frequently)?
- If NO — ⚠️ Use useEffectEvent or Latest Ref pattern instead
- If YES — ✅ useCallback is appropriate
Is this for a useEffect dependency?
- — ⚠️ Use useEffectEvent (React 19.2+) or Latest Ref pattern instead

The key principle: Don't memoise unless you can answer "yes" to: "Will this actually prevent something expensive from happening?"

If you're unsure, don't memoise. It's easier to add optimisation later than to debug broken memoisation chains. Remember, the best code is simple code. Three unnecessary useCallbacks or useMemo are harder to maintain than zero.

Understanding How Computers Actually Work

Ejeh Daniel — Sat, 01 Nov 2025 12:55:30 +0000

I use computers every day. You probably do too. But if someone asked me to explain how they really work, the high-level understanding I have would not suffice. I've been writing code for years now. But the gap between typing a command and pixels lighting up? Total mystery.🤔

So I went down the rabbit hole. I spent more time researching, watching videos, and piecing together explanations from a dozen sources. It turned out to be surprisingly fun and more rewarding than I expected. Once you have the right frame, it's not hard to understand.

At the core, every computer is simply an electronic machine that takes in input data (keyboard presses, mouse clicks, voice commands, etc.), stores it (short-term or long-term in registers, hard drives, RAM, etc.), processes it (using the CPU), and outputs results in some form (displaying text and images on the monitor, sound via speakers, network packets, etc.).

Here's the thing: computers are built on layers of abstraction. Each layer, from hardware, firmware, operating system, programming language, and application, hides the complexity beneath it. When you write code or use a website, you don't think about transistors or voltage levels or how the CPU decodes instructions. You just call functions. The layers beneath handle the rest.

In this article, I'll focus on Storage and Processing, the parts that happen inside the computer and remain mysterious to most of us. The computer turns your input actions into signals and instructions, and the result is something you can interact with. The real question is where those signals and instructions live when they're inside, and how the computer transforms them? We'll unravel all of that.

Image: A concentric ring or pyramid diagram showing the layer of abstraction progression from Hardware → OS → Applications.

The Language: Why Binary and How Bits Encode Everything

Computers don't see the world the way we do. At the most basic level, they only speak binary: a language of 1s and 0s.💻 Each 1 or 0 is called a bit (short for binary digit), and it's the fundamental unit of information. This means that every input or output, including every photo, song, video, email, and program you've ever interacted with, gets translated into or from binary data.

Modern computers are digital. They don't just relay information. They give electrical signals meaning. A signal under a certain voltage threshold is interpreted as "off" (0), and over a certain threshold is "on" (1). The continuous voltage range is reduced to just two discrete states. This is where the term "digital" originates: from the digits 0 and 1.

You might wonder why computers use only two values (0 and 1) instead of ten, like our decimal number system (0 to 9). The answer is that two states are much easier to tell apart reliably. When you're working with electrical signals, noise and variations are inevitable. With just two distinct states, one representing 0 and the other representing 1, there's no ambiguity, even when conditions aren't perfect. More states would mean more confusion, more errors, more complexity.

Binary works just like decimal, except with powers of two. For example, 01001000 in binary equals 72 in decimal. This value represents the letter 'H' in ASCII, a standard whose first 128 characters are directly adopted by Unicode.

With 8 bits (a byte), you can represent 256 possible values (2^8 = 256, ranging from 0 to 255). Combine enough bits, and you can represent anything: numbers, letters, images, sound, video. What makes that possible are standards. The computer doesn't "know" what 'H' is. It just follows agreed-upon rules like Unicode for text, RGB for images, or WAV for sound.

Now, since everything boils down to numbers, how do we store something like a song or a video? Through encoding standards like MP4 for video or JPEG for images, these get converted to numbers that the computer can store and manipulate.

Everything inside the computer, everything Storage holds, and everything Processing touches, is built on this foundation of 1s and 0s. But those bits are not abstract ideas. They're physical voltages controlled by billions of switches working together. Let's see how

The Hardware: From Silicon to Logic

Now let's descend to the hardware layer and talk about how the computer physically creates those 1s and 0s using electricity, and how it uses them to make decisions and perform operations. Remember those binary digits we just discussed? Here's how they physically exist inside the machine.

The Motherboard

Inside your computer case, you'll find many things: a power supply, cooling fans, storage drives, and various cables connecting everything together. But most of the action takes place on the motherboard, the main circuit board of the computer, so that's where we'll focus.

The motherboard is a printed circuit board (PCB), a flat board with copper wires etched into it, connecting various components. The board can have multiple layers for efficient signal routing.

The key innovation for modern computers is the integrated circuit (IC), commonly called a chip or microchip. An IC is essentially a complete electronic circuit etched onto a single piece of silicon. This allows billions of tiny components called transistors to be packed onto a chip smaller than your fingernail.

Key components on the motherboard:

The CPU (Central Processing Unit) - A single chip that plugs into a socket on the motherboard. This is where all the processing happens. Inside the CPU are several specialized components working together: a control unit that orchestrates everything, an ALU that performs calculations, tiny fast memory locations called registers, and small pools of ultra-fast memory called cache.
RAM (Random Access Memory) - Memory sticks that slot into the motherboard. This is the fast, temporary storage for data that the computer is actively working on.
Chipset - A set of chips that manage data flow between the CPU, memory, and peripheral devices. Think of it as the motherboard's traffic controller, coordinating communication between all the major components.
BIOS/UEFI chip - Stores firmware (the low-level software layer between hardware and the operating system) that runs when you first turn on the computer. This firmware performs hardware checks and loads the operating system from storage.
Ports - USB ports, network ports, etc., that connect external devices.

Image:Motherboard layout showing the main circuit board with key components like the CPU socket, RAM slots, chipset, and ports.

We'll focus on what happens inside the CPU chip, because that's where processing happens.

Inside the CPU: Transistors and Gates

Under the hood, bits are voltages. In modern circuits, a "high" voltage encodes 1, a "low" voltage encodes 0. Numbers are literally voltage levels flowing through wires. The component responsible for controlling these voltages is called a transistor, an electrically controlled switch that's the fundamental building block of all digital electronics.

The type used in modern CPUs is called a MOSFET (Metal-Oxide-Semiconductor Field-Effect Transistor). Apply voltage to the control? Current flows. Switch ON. Remove voltage? Current stops. Switch OFF.

A modern processor can pack tens of billions of transistors into a chip the size of your fingernail. Apple's M3 chips, for example, contain over 90 billion transistors 🤯, each one flipping on and off billions of times per second.

Wiring these transistors in specific patterns creates logic gates. Gates are tiny circuits that implement rules. They take voltage inputs (1s and 0s) and produce voltage outputs based on logical operations, like AND gates that only output 1 when both inputs are 1, or NOT gates that flip their input.

Combine enough of these gates, and you can build something far greater: the ALU (Arithmetic Logic Unit)—the part of your CPU that adds, subtracts, compares numbers, and performs logical operations. All computer math happens here. Want to add two numbers? The ALU chains together XOR gates and AND gates in a specific pattern called an adder circuit. But there's a subtle problem lurking here.

The Clock: Keeping Everything in Sync

Here's something interesting about voltage. It takes time to ripple through gates. Send a signal through one gate, and it might take a nanosecond to stabilize. Chain thousands of gates together and you've got signals arriving at different times, some still changing while others have already settled. Without coordination, it's chaos. Results would be corrupted mid-flight. Outputs would be garbage.

That's where the clock comes in. Think of the clock like an orchestra conductor 🎵 keeping all the musicians in time. A CPU's clock ticks billions of times per second (measured in GHz, a 3.5 GHz CPU ticks 3.5 billion times per second). The clock generates consistent electrical pulses sent down the wires. It does this the same way a digital watch keeps time: using a quartz crystal that vibrates at a precise frequency when electricity is applied.

Between ticks: Voltages propagate through gates. Computation is happening, signals are flowing, and math is being done.

On each tick: Results get locked into place (stored in small memory circuits) so the next stage can start fresh with stable values.

The clock remains the fundamental heartbeat that synchronizes everything and keeps electrons marching in formation.

So now we have transistors that can switch on and off to represent 1s and 0s. We have gates that use those switches to make decisions. We have an ALU that chains gates together to do math. And we have a clock that keeps everything synchronized. That's processing. But processing is useless without somewhere to put the data.

Memory: Where Data Lives

Moving up a layer, we need to understand where data lives. Processing is only half the story. The CPU needs somewhere to store the data it's working on, and not all storage is created equal. Modern computers use a memory hierarchy, balancing trade-offs between speed, size, and cost. Think of memory like a pyramid:

Image: A pyramid diagram illustrating the memory hierarchy, showing the trade-off in speed, size, and cost. It progresses from the fastest layer Registers at the top, through Cache L1/L2/L3 and RAM, to Storage the largest layer at the bottom.

At the top (fastest, smallest, most expensive):

Registers - Tiny storage locations inside the CPU itself. A CPU might have only a few dozen registers, each holding a single number (typically 64 bits). These are where the CPU keeps the data it's using right now. There are general-purpose registers for holding any data, and special-purpose registers like the program counter which tracks which instruction to execute next and the instruction register which holds the current instruction being processed.
Cache - Small, fast memory also very close to the CPU. Split into levels: L1 (smallest, fastest, typically per-core), L2 (bigger, slower, also per-core in modern CPUs), L3 (bigger still, slower still, usually shared across all cores). Modern CPUs might have 256 KB of L1 per core, 1-8 MB of L2 per core, and 8-64 MB of shared L3.

In the middle:

RAM (Random Access Memory) - The main working memory. Your computer might have 8, 16, or 32 GB of RAM. Much larger than cache, but slower to access.

At the bottom (slowest, largest, cheapest):

Storage - SSDs and hard drives. Terabytes of space, but thousands of times slower than RAM.

This hierarchy exists because of physics and economics. You can't make infinite amounts of super-fast memory, it's expensive and generates too much heat. So computers keep a little bit of ultra-fast memory close to the CPU and larger pools of slower memory further away. But how do these different types of memory actually work, and how does data move between them?

Connecting CPU and RAM: Buses

Before I show you how memory works, remember that data doesn't magically move; it travels along wires grouped into buses (they "transport" information, like a bus transports passengers).

There are two critical buses connecting the CPU and RAM:

Address bus: The CPU sends memory addresses along this bus to tell RAM which location it wants to access. Think of it like telling a librarian which book you want by giving them the call number.
Data bus: The actual data travels back and forth along this bus. This is typically 64 bits wide (64 wires running in parallel), allowing 64 bits to transfer simultaneously.

When the CPU needs data from RAM, it sends the address on the address bus, and RAM responds by sending the data back on the data bus. This communication takes time, hundreds of clock cycles. This is why the memory hierarchy exists: to keep frequently used data closer.

RAM: The Active Workspace

The most common type of RAM is DRAM (Dynamic RAM). Each bit is stored as a tiny electric charge in a capacitor. If the capacitor is charged, that's a 1. If it's empty, that's a 0.

Reading the charge tells you the bit's value, but reading is destructive, it discharges the capacitor, which must be recharged. Additionally, capacitors naturally leak charge over time, so RAM constantly refreshes itself thousands of times per second. This is why it's called Dynamic RAM.

Here's the catch: DRAM is volatile memory because it requires constant power to maintain data. When you turn off the power, the capacitors lose their charge, and everything in RAM disappears. That's why you have to save your work to disk which uses non-volatile memory that retains data without power, like SSDs using flash memory or traditional hard drives using magnetic storage.

Cache and registers use a different technology: SRAM (Static RAM). Instead of capacitors, SRAM uses circuits made from transistors wired in a configuration that holds its state as long as power flows, no constant refreshing is needed. The same transistors that make logic gates can also be configured to store values. This makes SRAM much faster than DRAM, but also much more expensive and physically larger for the same amount of storage. That's why you only get megabytes of cache versus gigabytes of RAM.

Persistent Storage: Data That Lasts

RAM is fast but forgets everything when you power off. For data that needs to survive, computers use persistent storage like hard drives and SSDs.

Hard Disk Drives (HDDs) store data using magnetism. Inside are spinning metal platters with a tiny head floating above them like a record player. The head magnetizes microscopic spots on the disk, one direction is 1, the opposite is 0. Reading just detects those magnetic fields.

The problem? Moving parts. The head has to physically swing to the right location, and the platter has to spin to the right spot. That takes time. Millions of clock cycles. But HDDs are cheap and can store terabytes.

SSDs use flash memory, a special type of transistor with an extra "floating gate" that can trap electrons. Force electrons onto that gate, and they stay there even without power. That's your 1. No electrons? That's 0. Reading just checks if charge is trapped.

No moving parts means SSDs are significantly faster than HDDs. They're also more expensive per gigabyte, but for speed, nothing beats them.

Modern computers often use both: SSD for the OS and programs, HDD for bulk storage.

Why Location Matters

Here's a quick example: Let's say the CPU needs to add two numbers.

If the numbers are in registers: 1-2 clock cycles. Nearly instant.⚡
If they're in L1 cache: 4 cycles.
If they're in L3 cache: 40 cycles.
If they're in RAM: 200 cycles.
If they're on disk: Millions of cycles.🐌

This is why good software always tries to keep frequently-used data "close" to the CPU. This principle is called locality of reference. If the CPU just used a piece of data, it'll probably need it again soon, so keep it in cache. Modern CPUs are smart about this. They automatically predict what data you'll need next and pre-load it into cache. When they guess right, your program is faster. When they guess wrong, you wait.

Now here's where it all comes together. We've got transistors and gates that can compute. We've got memory that can store data at different speeds. But how does the CPU orchestrate all of these to actually run a program?

Processing: How the CPU Executes Instructions

We've seen the building blocks, transistors, gates, and memory. Now, let's explore how the CPU actually runs a program.

The Instruction Cycle

Every program is a list of instructions stored in RAM. Each instruction is just a number; some bits describe the operation (add, subtract, load, store), and other bits point to where the data lives in memory. The CPU runs these instructions one at a time in a loop called the instruction cycle also called the fetch-decode-execute cycle.

Image: A flow visualization of the fetch-decode-execute cycle, showing how instructions flow from RAM to the instruction register, through the control unit and ALU/registers, and back to memory.

It's worth noting that different CPUs use different instruction sets (called ISAs)—x86-64, ARM, RISC-V, etc. Each has its own binary encoding, which is why software compiled for one processor won't run on another without translation.

Putting It All Together

Here's a simple example. Imagine a program that adds two numbers and stores the result. In assembly language (a human-readable version of machine code), it’ll look like this:👇

LOAD R1, [1000] ; Load the value at memory address 1000 into register R1LOAD R2, [1004] ; Load the value at memory address 1004 into register 
R2ADD R3, R1, R2 ; Add R1 and R2, store result in R3STORE 
R3, [1008] ; Store R3's value back to memory address 1008

What the CPU does:

Fetch the first instruction from RAM
Decode it: "LOAD means get data from memory"
Execute: Send address 1000 to RAM, get back the value, and put it in register R1
Move to the next instruction, repeat for R2
Fetch the ADD instruction
Decode: "ADD means use the ALU"
Execute: The ALU takes values from R1 and R2, adds them using XOR and AND gate circuits, and puts the result in R3
Fetch the STORE instruction
Execute: Send R3's value to RAM at address 1008

Each step is just voltages changing, gates opening and closing, electrons flowing through silicon. But layer enough of these steps together, and you get everything a computer can do.

Modern CPUs do clever tricks to go faster: they fetch the next instruction while decoding the current one (called pipelining), they execute multiple instructions at once on different parts of the chip (called superscalar execution), and they run multiple programs simultaneously on different cores (multiple CPUs on one chip). But the fundamental cycle—fetch, decode, execute, is always there, happening millions of times in the blink of an eye.

The Conductor: How the OS Brings It All Together

We've covered most of storage and processing, the core of how computers work. But there's one more piece needed to make this all useful: something has to coordinate everything we've discussed.

That's where the operating system comes in. Think of it as the conductor of the orchestra we've been building.

You're probably running dozens of programs right now. Your browser, music player, and text editor, they're all loaded in RAM, all waiting for CPU time. The OS manages this by giving each program a tiny slice of CPU time, maybe 10 milliseconds, then switching to the next program. It happens so fast you don't notice. This is multitasking in play.🔄

The OS also protects programs from each other. Each program thinks it has full access to RAM, but the OS carves up memory into protected sections. If one program crashes, it can't corrupt another's data. A piece of hardware called the MMU (Memory Management Unit) works with the OS to implement virtual memory, where each program gets its own isolated address space. The MMU translates these virtual addresses to actual physical RAM addresses. When RAM fills up, the OS can even move unused data to disk (called paging or swapping), making programs think there's more RAM than physically exists, though accessing this disk-backed memory is much slower than real RAM as we now know.

When your program needs to talk to hardware,display something on screen, save a file, or send data over the network, it doesn't do it directly. It asks the OS through a system call, and the OS handles the details using device drivers that know how to communicate with specific hardware.

This layering means you can write code that says "save this file" without knowing anything about how your SSD controller works. The OS translates your high-level request into the low-level commands that storage hardware understands.

Conclusion

So, how do computers actually work?

At the bottom, billions of transistors switch on and off, representing 1s and 0s as voltage levels. Move one layer up, and logic gates combine those 1s and 0s into decisions, building up to ALUs that perform math using adder circuits and flags. Another layer up, the memory hierarchy balances speed and size—registers, cache, RAM, and storage each serve data at different speeds, connected by buses. Another layer up, the CPU executes its cycle: fetch, decode, execute, all synchronized by a clock ticking thousands of times per millisecond. One more layer, and the OS orchestrates it all, managing programs, protecting memory, and coordinating hardware. At the top sits your code, blissfully unaware of the intricate dance playing out below.

Remember when I said the gap between typing a command and pixels lighting up was a total mystery? Now you know: your keypress becomes a voltage signal, travels through buses, gets stored as charges in RAM, triggers CPU instructions fetched from memory, executed by billions of transistors arranged in logic gates, all synchronized by a clock ticking billions of times per second, with the result written back to memory and sent to your graphics card to light up specific pixels. What once seemed like magic is now just... Well, really pretty clever engineering.✨

Does understanding all this make you a better programmer? Maybe not directly. But there's something deeply satisfying about knowing how these various components make up the computer. And that, I think, is pretty remarkable.

Here’s a list of amazing resources I’ve found while doing some research for this post:🙂.📚

Crash Course Computer Science: Episodes 5, 7 and 17 - Good resource on CPU operations, memory hierarchy, and instruction cycles

Intro to Truth Tables & Boolean Algebra | by Brett Berry | Math Hacks | Medium - An introduction to Logic and Truth Tables:

See How a CPU Works - In One Lesson - A Visual demonstration of fetch-decode-execute cycle

How Motherboards Work - HowStuffWorks - Motherboard components and PCB structure

Why You Can't Use CPU Clock Speed to Compare Computer Performance - How-To Geek - An intro to Clock speed and CPU performance factors

How do hard drives work? - TED-Ed - A deep dive into HDD magnetic storage mechanism

Computer memory - Wikipedia - Comprehensive coverage of RAM, cache, registers, SRAM vs DRAM

Stack Exchange - Flash Memory - SSD flash memory technology and floating gates

How I Built an n8n Community Node for Neon Database

Ejeh Daniel — Sun, 07 Sep 2025 22:13:47 +0000

n8n is an open-source automation platform that connects APIs, databases, and tools through visual workflows. You build workflows using nodes—triggers, actions, and transforms—and run them self-hosted or on n8n Cloud.

Recently, I had to automate a product ops workflow: pull a product catalog, categorise items, surface close alternatives, and sync the results downstream. I used n8n as my automation engine and Neon as the database to handle deduping, versioning, and categorizing so the workflow could make smarter decisions over time.

n8n provides official nodes for popular services like Slack, Google Sheets, and GitHub, but the real strength shows up in the community. Developers step in to cover the integrations the core team might never build. For example, Minhlucvan’s Apify node brings web scraping workflows into n8n, Yatolstoy’s Kommo node connects to a well-known CRM, and Mohsen Hadianfard’s ApplyBoard node helps education agents sync student applications. These are polished, ready-to-use integrations built by third-party developers and published on the npm registry.

Here’s where things got interesting. At first, I used n8n’s official Postgres node to connect to my Neon database, but it quickly felt limiting. Neon is a modern, serverless Postgres built for speed, scale, and branching, yet the generic Postgres node couldn’t access features like branch switching. If I wanted to get the most out of Neon inside my workflows, I needed something else. That’s when I decided to build a custom Neon node for n8n.

The goal was to make something that works, and also build it in a way that followed n8n’s conventions, handled database operations securely (avoiding SQL injection), and matched the quality bar of community nodes. The node needed to feel native to both n8n and Neon.

The scope became clear:

Execute custom queries
Standard CRUD (INSERT, SELECT, UPDATE, DELETE)
Support Neon’s branch switching via the n8n UI
Follow n8n resources on building custom nodes

With this scope in mind, I also had to decide which type of node architecture to use. n8n supports two approaches: programmatic nodes (trigger nodes) and declarative nodes (JSON-based syntax nodes). Given my goals, declarative was the obvious choice. It would keep the node maintainable and aligned with community standards.

Setting up My Custom Neon Node

Before setting up my environment, the first thing I had to do was get familiar with how n8n nodes are actually built.

👉 Before diving in

If you’re thinking, maybe “I’ll try building a node myself” here’s what I had to get in place to get my environment running:

Node.js + npm → the bread and butter of building custom nodes
TypeScript → because n8n speaks TS, not plain JS
Docker + Docker Compose → easiest way to spin up n8n locally without headaches
Some JS/TS + Docker knowledge → nothing advanced, but enough to not get stuck on the basics.

Once I had this stack ready, it was smooth sailing to clone n8n’s starter template and strip it down for my own Neon node. 🚀

Nothing fancy yet, but at this point, only three essentials are really needed to get a custom node running:

The node JSON file
The node TypeScript file
The credentials file

That’s the bare minimum foundation.

Structuring the Node

The n8n’s official resource on building custom nodes already lays down the patterns people expect from database integrations, so sticking close to that gave my Neon node the same “native” feel. I also made one early call: go modular. It’s tempting to cram everything into a single file, but Neon workflows can get hairy fast, and n8n’s mental model is all about consistency. Every database node should feel familiar, so I leaned into that.

In the n8n ecosystem, the recommended file structure is this:

actions/ → sub-directories for each resource. Each contains:
- a resource description file (<resourceName>.resource.ts or index.ts)
- one file per operation (<operationName>.operation.ts), each exporting both the operation’s description and its execute function
methods/ → optional, for dynamic parameter functions
transport/ → handles the actual communication layer (API calls, DB connections, etc.)

With that mental model in mind, here’s how my Neon node structure shaped up:

nodes/Neon/
├── actions/operations/
│   ├── executeQuery.operation.ts
│   ├── insert.operation.ts
│   ├── select.operation.ts
│   ├── update.operation.ts
│   └── delete.operation.ts
├── helpers/
│   ├── utils.ts
│   └── interface.ts
└── methods/
    ├── credentialTest.ts
    ├── listSearch.ts
    └── resourceMapping.ts

This separation of concerns made the whole thing more maintainable, and way easier for me to reason about.

So basically, an n8n node is a single class in the main TypeScript file. That class acts as the entry poin, the hub where credentials, operations, and helper methods all plug in. Mine looked something like this:

export class Neon implements INodeType {
    description: INodeTypeDescription = {
        displayName: 'Neon',
        name: 'neon',
        icon: 'file:neon.svg',
        group: ['input'],
        version: 1,
        credentials: [
            {
                name: 'neonApi',
                required: true,
                testedBy: 'neonApiCredentialTest',
            },
        ],
             properties: [
              //fields the users interact with for each operation
             ]
    };

    methods = {
        credentialTest: { neonApiCredentialTest },
    };
      async execute(this: IExecuteFunctions):                                   Promise<INodeExecutionData[][]> {
}
// execution logic for each operation goes here
}

Going forward, having that mental model upfront, made it much easier to see how each piece fits together as I went deeper.

The next big decision was choosing a database engine. My first choice was to use the raw pg library. It works, but it leaves you with a lot of heavy lifting: managing connection pools, handling transactions, formatting queries, basically more boilerplate code for me.

Instead, I went with pg-promise. It’s lightweight, wraps pg under the hood, and takes care of the boring stuff for you. That made it a no-brainer for my custom node: less boilerplate.

Managing Credentials and Authentication

One of the first hurdles was handling credentials properly. In n8n, credentials abstract away sensitive details like connection URLs, usernames, and passwords. For my Neon node, that meant figuring out how to handle Neon’s connection string.

At first, I thought: why not just drop a single field for the whole connection string? Easy enough. But in practice, it got messy fast. One typo, a missing parameter, or a poorly formatted copy-paste, and suddenly nothing works. Not the best option after all.

So I decided to break things into explicit fields: host, port, database name, username, password, and an SSL toggle. That way, everything is clear, validated, and less error-prone.

Those fields live inside a NeonApi credential class:

export class NeonApi implements ICredentialType {
    name = 'neonApi';
    displayName = 'Neon Database API';

    properties: INodeProperties[] = [
        { displayName: 'Host', name: 'host', type: 'string', default: '' },
        { displayName: 'Database', name: 'database', type: 'string', default: '' },
        { displayName: 'Username', name: 'user', type: 'string', default: '' },
        { displayName: 'Password', name: 'password', type: 'string', typeOptions: { password: true }, default: '' },
        { displayName: 'Port', name: 'port', type: 'number', default: 5432 },
        { displayName: 'SSL', name: 'ssl', type: 'options', options: [{ name: 'Require', value: 'require' }, { name: 'Allow', value: 'allow' }], default: 'require' },
    ];
}

Notice: having explicit fields makes it cleaner — This saved me from a lot of silly mistakes.

Building the Core Operations

Once credentials were sorted out, the next challenge was to make the core operations work: INSERT, SELECT, UPDATE, DELETE, and EXECUTE. These are the bread and butter of any database node, and my Neon node needed to handle all of them to feel complete.

At the core, an operation in n8n is a collection of configuration objects. It describes what fields show up in the interface, how users fill them out. Practically, each operation had very different interface needs, so each operation gets its own description so the interface feels consistent and predictable.

INSERT Operation: Auto-Map vs Manual Mapping

The first operation I tackled was INSERT. The goal was simple: let users drop new rows into their database tables without needing to write custom queries.

But sometimes the input data already matched column names perfectly, other times it didn’t. So I provided two data map modes:

Auto-Map Input Data → n8n tries to map item keys directly to table columns. Perfect if the JSON keys from the previous node match the column names in the Neon database
Manual Map (Define Values Below) → if keys don’t match user will pick each column and set its value themselves. More tedious, but gives full control.

Here is an example of the INSERT operation config:

const properties: INodeProperties[] = [
 // Data to send for insert operations
 {
   displayName: 'Map Column Mode',
   name: 'mappingMode',
   type: 'options',
   options: [
     {
       name: 'Auto-Map Input Data to Columns',
       value: 'autoMapInputData',
       description: 'Use when node input properties names exactly match the neon column names',
     },
     {
       name: 'Map Each Column Manually',
       value: 'defineBelow',
       description: 'Set the value for each destination column manually',
     },
   ],
   default: 'autoMapInputData',
   description:
     'Whether to map node input properties and the table data automatically or manually',
   displayOptions: {
     show: {
       resource: ['row'],
       operation: ['insert'],
     },
   },
 },
 {
   displayName: 'Values to Send',
   name: 'valuesToSend',
   placeholder: 'Add Value',
   type: 'fixedCollection',
   typeOptions: {
     multipleValueButtonText: 'Add Value',
     multipleValues: true,
   },
   displayOptions: {
     show: {
       mappingMode: ['defineBelow'],
     },
   },
   default: {},
   options: [
     {
       displayName: 'Values',
       name: 'values',
       values: [
         {
           // eslint-disable-next-line n8n-nodes-base/node-param-display-name-wrong-for-dynamic-options
           displayName: 'Column',
           name: 'column',
           type: 'options',
           // eslint-disable-next-line n8n-nodes-base/node-param-description-wrong-for-dynamic-options
           description:
             'Choose from the list, or specify an ID using an <a href="https://docs.n8n.io/code/expressions/" target="_blank">expression</a>',
           typeOptions: {
             loadOptionsMethod: 'getTableColumns',
             loadOptionsDependsOn: ['schema', 'table'],
           },
           default: '',
         },
         {
           displayName: 'Value',
           name: 'value',
           type: 'string',
           default: '',
         },
       ],
     },
   ],
 },
 optionsCollection
];

const displayOptions = {
   show: {
     resource: ['row'],
     operation: ['insert'],
   },
   hide: {
     table: [''],
   },
};

If the user selects manual mode, they see a “Values to Send” field where they can pick columns and provide values one by one. Behind the scenes, I had to normalise data from both modes into the same object structure before building the SQL query. Otherwise, I had to deal with unstable data formats.

The catch? Not every input object matched the actual table schema. That led to confusing errors where inserts would silently fail. So before building the query, I pulled the schema directly from the database and cross-checked it with the input object. If any mismatch is found, I stop execution early.

Execute Operation: The Pain Hole

The INSERT operation was straightforward; EXECUTE was the tricky one. On the surface, the interface need is simple: a SQL editor for the raw query, plus a field for query parameters. A user could type something like:

SELECT * FROM users WHERE id = {{ $json.userId }}

Looks fine, but here’s the catch—n8n expressions ({{ $json.userId }}) aren’t plain strings. They’re dynamic placeholders. I passed them straight into my query, and Neon kept throwing errors.

The fix was small but critical. I looped through the query, found all expressions, and resolved them before execution:

for (const resolvable of getExpressions(query)) {
  query = query.replace(
    resolvable,
    this.evaluateExpression(resolvable, index) as string
  );
}

Worked!!!`, as expected, I just had to play nicely with the queries.

SELECT Operation: Condition and Sorting

The SELECT operation felt trickier, but needed more thinking around it. The goal was to let users fetch rows with filters, conditions, and sort results.

So the interface need was clear:

Select Rows → A fixed collection where users pick a column, an operator (=, >, <, etc.), and a value. Example: Column = age, Operator = >, Value = 18.
Combine Conditions → A simple dropdown: AND or OR. This lets users build compound filters.
Sort → A collection field where users choose a column and direction (ASC or DESC).

Behind the scenes, I build a WHERE clause with these input values before proceeding to add it to the final query to be executed. For example:

User config →{ column: 'age', operator: '>', value: 18 }
Values → ['age', 18]

Here’s the built WHERE clause:
WHERE $0:name > $1 AND $2:name = $3

This pattern made SELECT operations safe. Users could stack conditions, combine them, and sort results without writing queries themselves.

UPDATE and DELETE Operations

Once INSERT and SELECT were working, UPDATE and DELETE felt much easier. The patterns were already in place:

Users will specify conditions the same way as SELECT.
For UPDATE, they will choose which columns to modify and what the new values should be.
For DELETE, they will choose to drop the whole table, truncate all the table data, or just delete rows that match the filter conditions.

By this point, the heavy lifting was done. Most of the work was making sure inputs were validated against the Neon schema and queries were parameterised for safety.

Challenges and Moving Forward

Looking back, one of the earliest decisions that saved me countless headaches was sticking with parameterised queries. Raw SQL feels more straightforward, especially when you’re building complex conditions, but parameterisation gave me two wins at once: protection against injection and a consistent way to construct queries programmatically. pg-promise made this possible, to use syntax like $1:name or $2:name, with this everything became smoother.

Another key choice for me was leaning on n8n’s conventions. Their documentation and official node patterns became my guardrails. Any time I drifted too far from those patterns, things broke. In n8n, conventions aren’t optional.

The irony? I spent more time researching, planning, and translating Neon requirements into n8n’s mental model than actually writing code. The hardest part of building a custom node isn’t SQL or TypeScript, t’s learning how to translate the platform’s needs into something that fits seamlessly into n8n’s conventions.

The fact that almost any custom node you can imagine is buildable in just a few weeks is wild — the custom node came together in such a short time, and it blew my mind.

The code lives on GitHub. Fork it, run it, break it, fix it. 🚀