DEV Community: Alexander Voll

Building an AI powered search in Node.js using Qdrant Cloud

Alexander Voll — Tue, 12 Dec 2023 17:59:38 +0000

Introduction

Since the launch of ChatGPT, we've been seeing more and more use cases for LLMs reaching from helping with ideation to complex support agents connected to custom data bases.

In this article, I will talk about how I set up an AI powered search in Node.js. This can be used for any use case. In my case i connected it to the Codesphere blog, enabling an AI powered search.

What is Qdrant

In a nutshell, Qdrant provides vector databases, featuring a user-friendly API for managing collections of data points, querying, and searching.

My experience with vector databases began with Qdrant, which simplified an otherwise complex environment. It offers a JavaScript SDK, integrating seamlessly with existing Node.js projects to leverage the capabilities of large language models (LLMs).

All of this is tied together through Qdrant Cloud which essentially provides managed Qdrant databases for you to use and includes a free tier. It has a convenient graphical interface which can be used to look at the collections created.

Understanding Vector Databases

In more sophisticated applications involving large language models (LLMs) in some way, vector databases are often used to handle custom data. Unlike traditional databases that store data in tables and rows, vector databases use embeddings. These embeddings are lists of numbers representing data in a high-dimensional space, much more complex than a simple three-dimensional space.

The transformation of regular data into vector embeddings is carried out by trained LLMs. These models understand the context and semantics of the data, converting it into a vector that accurately represents its meaning in this multi-dimensional space. This allows for efficient processing and comparison of complex data, such as text or images, enabling more sophisticated and context-aware search capabilities.

The Workflow for my Blog Search

Having understood the basics about how a vector database works and how data can be converted into embeddings, the workflow itself is pretty simple:

Create a collection
Check which articles are already in the Qdrant database
Create embeddings for missing articles
Upload to Qdrant

With that, it is possible to create embeddings for search queries and search the collection in the database.

Setting Up Qdrant in JavaScript

Before diving into the specifics of building the AI-powered search, it’s crucial to understand how to set up Qdrant in a JavaScript (Node.js) environment. This setup forms the foundation upon which all the subsequent functionalities rely.

Qdrant Client Initialization

First, you’ll need to initialize the Qdrant client. This client acts as the bridge between your Node.js application and the Qdrant database, enabling you to interact with the database programmatically.

import { QdrantClient } from "@qdrant/js-client-rest";

const client = new QdrantClient({ 
    url: 'https://[your-qdrant-instance-url]',
    apiKey: "[your-api-key]"
});

Replace '[your-qdrant-instance-url]' and '[your-api-key]' with your Qdrant instance URL and API key. This setup assumes you have an account with Qdrant Cloud and have set up a database instance.

One thing to note is, that at least for me, I couldn't get the SDK to work with a local in memory setup of the database without using Docker. So keep in mind that you will either need Qdrant Cloud or a custom Docker image running locally with your database in it. Running a managed Docker image on Digital Ocean unfortunately didn't work for me either.

Installing Required Libraries

To fully leverage the capabilities of Qdrant and LLMs, you’ll need to install certain libraries. The @qdrant/js-client-rest library is used for interacting with the Qdrant API, and @xenova/transformers for processing text into vector embeddings.

npm install @qdrant/js-client-rest @xenova/transformers

This command installs the necessary Node.js packages into your project, allowing you to use the Qdrant client and the Xenova pipeline for text encoding.

Text Encoding with Xenova

The Xenova pipeline is a crucial part of transforming text data into vector embeddings. This is done through the encodeText function, which leverages a pre-trained model to process text.

import { pipeline } from "@xenova/transformers";

async function encodeText(text) {
    const extractor = await pipeline("feature-extraction", "Xenova/all-MiniLM-L6-v2");
    return await extractor(text, {pooling: "mean", normalize: true});
}

With these initial setup steps, you're now ready to dive into creating and managing your AI-powered search functionality using Qdrant.

Following this setup guide, let’s delve into the four key steps of building the AI-powered search.

1. Creating a Collection with Qdrant

To kickstart the AI-powered search, the first task is setting up a collection in the Qdrant database. Think of a collection as a flexible and dynamic container for your data, specifically tailored for vector-based operations.

Function: `recreateQdrantCollection`

The recreateQdrantCollection function is our gateway to initializing this collection. It's here that we define the crucial parameters of our vector space, like its size and the distance metric, which are key to how our data will be interpreted and queried. The vector size will be determined by the LLM you're using for transforming your data into embeddings. I simply just ran an example conversion to get the right size.

async function recreateQdrantCollection(collectionName, vectorSize) {
    console.log("Creating collection");
    try {
        await client.recreateCollection(collectionName, {
            vectors: {
                size: vectorSize,
                distance: "Dot"
            }
        });
        console.log(`Collection ${collectionName} created successfully`);
    } catch (error) {
        console.error(`Error creating collection ${collectionName}:`, error);
        throw error;
    }
}

As soon as the collection has been created, you will be able to see, access and manage it from the Qdrant dashboard:

2. Checking Existing Articles in the Database

Once the collection is set, it's important to ensure we're only adding new or updated content. This step is crucial for maintaining efficiency and avoiding redundancy.

Function: `scrollThroughPoints`

The scrollThroughPoints function comes into play here. It retrieves the IDs of all articles currently stored in the collection, giving us a clear picture of what’s already in there.

async function scrollThroughPoints(collectionName) {
    let existingIds = [];
    try {
        let hasMore = true;
        let scrollId = null;

        while (hasMore) {
            const response = await client.scroll(collectionName, {
                limit: 1000,
                with_payload: false,
                with_vector: false,
                scroll_id: scrollId,
            });

            response.points.forEach(point => existingIds.push(point.id));
            scrollId = response.scroll_id;
            hasMore = response.points.length > 0;
        }
    } catch (error) {
        console.error(`Error scrolling through points in collection '${collectionName}':`, error);
    }

    return existingIds;
}

Keep in mind that there is a limit to set here which you should take a look at before running it. You can then use the data retreived from the database to check if your articles are already in the collection or not by comparing the ids.

3. Creating Embeddings for Missing Articles

For the articles that are not in the collection, we need to transform their textual content into a format that Qdrant can understand and work with – vector embeddings.

Function: `encodeText`

Here's where encodeText shines. It uses an LLM to convert text into vector embeddings, encapsulating the nuances and context of the content into a numerical form.

async function encodeText(text) {
    const extractor = await pipeline("feature-extraction", "Xenova/all-MiniLM-L6-v2");
    return await extractor(text, {pooling: "mean", normalize: true});
}

4. Uploading to Qdrant

The final step in the setup is populating our Qdrant collection with these freshly minted embeddings.

Function: `uploadDocuments`

The uploadDocuments function takes these embeddings and carefully places them into our collection, making sure each piece of content is correctly represented and stored.

async function uploadDocuments(collectionName, documents) {
    console.log("Uploading documents");
    const points = documents.map(doc => ({
        id: doc.id,
        vector: doc.vector,
        payload: doc.payload
    }));

    try {
        await client.upsert(collectionName, { points });
        console.log(`Documents uploaded successfully to collection '${collectionName}'`);
    } catch (error) {
        console.error(`Error uploading documents to collection '${collectionName}':`, error);
    }
}

Keep in mind, that Qdrant does not support any kind of ids. The official documentation states that Qdrant supports using both 64-bit unsigned integers and UUID as identifiers for points.

In my case this was perfect, as our blog CMS (Ghost.io) provides a UUID for each article which I could just use to identify my points.

Implementing AI-Powered Search

After setting up our collection and populating it with blog articles, the next crucial step is to implement the search functionality. This involves querying the Qdrant database with a search vector and retrieving the most relevant articles.

Function: `searchArticles`

The searchArticles function is designed to perform this task. It takes a search query, converts it into a vector embedding (similar to how we processed our blog articles), and then uses this vector to search our Qdrant collection for the most relevant matches.

Here's an example of how this function could be structured:

async function searchArticles(query) {
    const vector = await encodeText(query); // Convert query text to vector

    try {
        const searchResults = await client.search("codesphere_blog", {
            params: {
                hnsw_ef: 128,
                exact: false,
            },
            vector: vector,
            limit: 3, // Adjust limit as needed
        });

        console.log("Search results:", searchResults);
        // Further processing of searchResults as needed
    } catch (error) {
        console.error("Error during search:", error);
    }
}

// Example usage of searchArticles
searchArticles("AI powered search").catch(console.error);

In this function, encodeText is used again to convert the search query into a vector. This vector is then passed to the client.search method, along with the name of your collection and other search parameters. The limit parameter determines how many results to return, and the hnsw_ef parameter is a configuration for the search algorithm (feel free to adjust these values based on your requirements).

With this setup, you can perform searches against your Qdrant collection, harnessing the power of vector similarity to find the most relevant articles for any given query.

The only thing that is left now is to put all of those pieces together in our production setup and connect it to the frontend. Since we're currently working on a more sophisticated version, I won't be going into detail on this but I think this should already give you a good overview about how you can use Qdrant to set up an AI powered search for your blog.

Conclusion

Qdrant made it very easy for me to get started in the vector database world. The JS SDK includes all necessary methods to interact with the database.

This in turn made is very easy to incorporate an AI-powered search into our blog. Now this could also be implemented into any other web application for wildly different use cases.

Whether for a blog, a knowledge base, or any other content-rich platform, this approach opens up new horizons in information retrieval and user engagement.

OpenAIs GPTs with a third party API: Does it work?

Alexander Voll — Mon, 13 Nov 2023 10:44:33 +0000

Introduction

OpenAIs most recent release of the new GPTs sparked interest in the AI space as it promises a very easy setup of custom GPT instances that are catered towards specific use cases and, according to OpenAI, should be able to communicate with external APIs.

I wanted to see if this is true, checking out if it would be possible to use a GPT in combination with our internal API to create a workspace on Codesphere.

My goal was to create a custom GPT that takes in a GitHub public Repo URL and creates a workspace on Codesphere that automatically pulls the files from the Repo.

In this article I will go into why this was a pretty unpleasant experience for me and why I think OpenAI will be able to fix this regardless.

To set up my GPT with an external API, I took information from the Zapier AI Actions article as it provided better information than the official documentation.

Setting up a new GPT

To set up a new GPT, you need GPT Plus or Enterprise. Other than that, setting up a new GPT is fairly easy. Just navigate to the explore tab in your navigation and you will find a few GPTs to play around with as well as the opportunity to create a new, custom one. This feature is prompted with a Beta flag which now seems like foreshadowing for what I was about to throw myself into.

The My GPTs View

As soon as you create a new GPT, you will be greeted by the GPT Builder, which is essentially a GPT in itself, made to help you create your custom GPT (I will be typing “GPT” a lot in this article). Here you can specify a specific purpose for you instance, which it will then cater towards.

The GPT Builder

While this is cool and very easy to set up, this is mainly an improvement for use cases that were already possible with regular ChatGPT, essentially stripping away the boilerplate prompts.

I seems like this is the way of creating a GPT that OpenAI is currently focussing on, targeting individual, more causal users.

While I enjoyed playing around with this, I wanted to find out, if it really was that easy to have a custom GPT that would communicate with an API. This would make for a ton of use cases where companies and developers would be able to integrate GPT with their own APIs, creating a super focussed experience.

GPT Actions

You can configure your instance of GPT under the “Configure” tab,

In terms of user facing configuration possibilities, you can name your GPT, Give it a description and conversation starters, which are essentially example prompts displayed in the GPT window.

For configuring your GPT iteself, you can write it an instruction, upload knowledge files and define Actions.

To integrate your GPT with an API, you would use a combination of Instructions and Actions.

OpenAPI Spec as the foundation

The foundation of implementing a custom GPT with an API is undoubtedly the OpenAPI specification of the API. This lays the basis for how the AI will interact with the provided API.

Unfortunately, this is also where problems start to come up. The OpenAPI spec is a standardized specification of an API endpoint that is either provided in YAML or JSON. Sounds easy enough. Unfortunately, GPTs Parser doesn't really know how to work with multiple server URLs or variables. If it finds only one specific error in the specs, it will give a fairly descriptive warning message. Warnings in this case don't solely need to be syntax errors. It might also be a error that the parser thinks the instance of GPT won't be able to work with even though in reality, it actually just might. Just like it was the case with this error message:

GPT OpenAPI Schema

If there are multiple errors/warnings, the GPT builder will simply let you know that it can't parse the specification. Keep in mind: Those errors don't need to be syntax errors so no need to go back and forth validating your YAML/JSON. It's best to set up your file step by step to be able to identify the source of the warning. And even if you do, there might not be a fix or it might not even be relevant after all.

Still, as soon as I got this to work, the GPT builder recognized the specified API endpoints (actions) regardless of the warning. This led me to writing the instructions for my custom GPT to tell it which steps to take to create a workspace on Codesphere.

GPT recognized Actions/API Endpoints

Instructions

The instructions are where you tell your GPT who it is and what it should do. As explained in the better prompts section of OpenAIs documentation, I used Markdown to format my instructions in a way where they're more structured.

As mentioned above, my goal was to have the GPT take in a GitHub URL and create a workspace on Codesphere that contains the contents of the repo. All needed data besides the Repo URL and a validation token should be pulled by making another API call.

After many iterations and tries for debugging (more on that in a sec), I came up with the following instructions

### Rules:
You are a deployment assistant for the cloud provider "Codesphere". Users will provide you with a github repo url which you should use to create a workspace on Codesphere by following the instructions below.
Before running any requests, ask user for a github repo url. If any errors happen upon calling the api, please let the user know which they are.
After Each step, please log the answer from the server and also the request 1:1 without changing it to the user for debugging including the json object received from the server.
If uncertain about the request, refer to the OpenAPI spec or ask the user.

### Instructions for starting a workspace on Codesphere using the provided API: 
- Step 1: Request the XYZ endpoint using the main url (codesphere.com) and get property 1 and property 2. Those should be used for requesting the XXXX endpoint.
- Step 2: Request the XXX endpoint using the second url ("https://XXXX.codesphere.com") where you replace XXXX with the XXXX from the previous step. Fill in the other properties as follows. Everything that is defined as "null" here should explicitly be set. All properties are needed for the request to succeed and are mandatory. Don't let any of the following properties out of the request!
    - Property 1: XXX,
    - Property 2: XXX,
    - Property 3: XXX,
    - Property 4: XXX,
    - gitUrl: git url provided be the user (if missing, add a .git in the end)

As you can see, I was pretty adamant about GPT keeping everything in the request because this is where the real fun (or lack thereof) started to begin.

API Calls

Even though my instance of GPT seemingly had everything it needed which included a detailled instruction, API Keys and OpenAPI spec, this is where it started to fall apart. Specifically at the second API call.

The initial API call to receive the data needed for the 2nd call worked fine and I was initially amazed at how cool it was to have had set up essentially a talking API client.

I also was initially very surprised, that my GPT was able to work with the variable that threw a warning in the configuration tab.

Unfortunately, GPT was not able to get the requests right that were needed for the 2nd API call. There seems to have been an error in how GPT interacts with the interface for the API calls. I could not get it to make the right request. It often seemed to leave out properties from the request body or specifying them in the wrong way. This is why you can see me get more and more strict and unpleasant as the Instruction goes on. I wasn't having it.

Limited Debugging Capabilities

Even though the Rules clearly state that every Request and Response should be logged to the user 1:1, this didn't work. I asked it multiple times why that was and after a while I learned, that the LLM itself only has access to a limited amount of the data and seems to have to interpret part of it. It felt to me like there was some amount of hallucinating going on.

Unfortunately, the GPT Builder comes equipped with next to no debugging features. The only thing you can really see, is the request body. The response, including the errors are only given to you by GPT through the chat, sometimes being incomplete or not containing any useful information.

This is what made the process very tedious as you had to wait for GPT to go through the calls, give you the limited information about your call and then essentially having to guess what you might have to do to fix the issue.

Known bugs without known fixes

After having more or less fixed missing properties (at least according to the limited logging capabilities of my GPT), the issue that gave my GPT adventure the final blow was an error called UnrecognizedKwargsError. This is an error that is known in the OpenAI developer community, coming up here and there. I was able to only find one proposed solution which unfortunately didn't work for me.

After playing around with it for a little longer and reaching the usage cap for the 2nd time, I finally decided to give up and leave it be.

While frustrating and not satisfying at all, I still think this was a worthwhile experiment and I am sure that OpenAI will be making this better in the future.

Conclusion

GPTs and especially the GPT Builder seem unfinished. The lack of debugging features and proper documentation make it a tedious task to create a GPT that goes beyond playing a character. I'm not saying it is impossible but it surely is not a pleasant experience to set it up at all. It is important to note that the Builder is currently a Beta though. It seems to me like OpenAI took the whole Fail Fast and Iterate approach a bit to seriously as they released a product that (at least for my use case) was riddled with bugs and hurdles.

Working at a tech startup myself, I understand however, that innovation moves fast, sometimes faster than one would like and it is important to push out features to validate. Comparing the 1st version of what we have today is a night and day difference. This is why I am confident that OpenAI will fix this. The question is: When?

For now, I will set the GPT builder and specifically GPT Actions aside for another time and stay with it playing my cooking buddy:)

Migrating Node.js to Bun: Really a drop in replacement?

Alexander Voll — Fri, 13 Oct 2023 16:32:50 +0000

Intro

Over the last few months, Bun has sparked interest in the web development community, claiming for it to be a drop in replacement for Node.js that is much faster at the same time.

We work with Node.js all the time here in the Codesphere Marketing team and wanted to put that to the test.

We tried out, if Bun would really be able to be dropped into the existing repo for our main page and if we could finally ditch Node.

Buns Bundler

Watch mode

Building the code with bun is actually really easy. After installing bun via npm (What the bun team claims will be our last npm command ever), executing bun run index.js instead of node index.js worked fairly well, except for the websockets we use for tracking. More on that later.

What we really liked about it was the —watch flag provided by Bun, which essentially eliminated the need for a watcher like nodemon. To run and then watch a specific file, simply run bun —watch run ./your-file.js. For more complex setups, a watcher might still be necessary, however for our use case, this worked just fine as we use Tailwind for our CSS and compile them that way and our EJS templates get built on the fly when a user accesses one of our pages.

Entrypoints

To be able to build more than one file at the same time, you need to configure entrypoints for the bundler to go through and bundle your code into usable pieces. Since we use EJS and JS, we only need to bundle our JS code. We created a bundling script called bundle.js in the root of our project which we then simply run using bun run ./build.js.

This bundles the files according to the configuration.

Seems easy enough, right? Well… yes and no.

While this works really well, Bun currently doesn’t allow for naming entrypoints like Webpack does. This meant for us to actually be able to use Buns Bundler, we needed to write some extra code to be able to give the right names to our bundles so everything would work without us having to touch all of our EJS templates.

We ended up with this for the configuration:

const path = require('path');

async function buildProject() {
    const entrypoints = [
        "./src/js/features/index.js",
        "./src/js/global/index.js",
        "./src/js/home/index.js",
        "./src/js/main/index.js",
        "./src/js/pricing/index.js",
        "./src/js/tracking/index.js",
    ];

    for (let entry of entrypoints) {
        const dirName = path.basename(path.dirname(entry));

        await Bun.build({
            entrypoints: [entry],
            outdir: './public/js',
            minify: true,
            naming: `${dirName}.[ext]`
        });
    }
}

buildProject();

Installing Dependencies

Using the Bun package manager to install packages worked just as expected. Simply run bun install to install any npm package. No issues here!

The Websocket issue

We work with Websockets for our UX tracking. Our current server setup features an Express server that uses the wslib package to handle our Websocket operations.

This unfortunately didn’t work together with bun. This left us to handling the Websockets with Bun and leaving the rest to Express which also isn’t a very clean solution.

If we were to implement Bun into our production system, this would essentially mean we would need to refactor our server. For a smaller website project like ours this might be sufficient but depending on the size of the project, this might become a daunting task.

However, Bun doesn't promise to be an Express and wslib replacement, rather a Node replacement. Depending on your setup, you will need some time for configuration though.

Conclusion

In our little migration experiment, we noticed that especially build times were A LOT faster than on our current setup using Node and Webpack in combination with nodemon.

Bun is a great and very fast tool and while most tasks worked fine, not all did. Depending on your use case, you might be able to just drop in and get going with Bun but depending on your installed packages, you might need to reconfigure some things to get going.

Now, is Bun a drop in replacement for Node?

Well, this is hard to answer. We would still say yes, as most things worked just fine and you can't expect Bun to replace each and every package you have installed.

We will definitely be considering Bun for our next project. This time not as a drop in replacement though but rather to start out with it from the beginning!

Improving Code Quality using Embold

Alexander Voll — Mon, 28 Aug 2023 17:05:18 +0000

Introduction

In the world of software development, code quality isn't just a buzzword – it's a necessity. The quality of code directly impacts the functionality, maintainability, and overall success of software projects. As projects grow and evolve, managing and ensuring code quality can become challenging. This is where tools like Embold come into play, offering a lifeline to developers and teams aiming for excellence.

What is Embold?

Embold is a software analytics platform designed to assist developers in crafting high-quality code. It scans code bases, identifies potential issues, and even suggests ways to enhance the code.

Key Features of Embold

Code Analysis : At the heart of Embold lies its potent code analysis capability. With support for multiple programming languages, it meticulously scans for anti-patterns, bugs, and code smells, ensuring that developers are always aware of potential issues.
Metrics : Metrics are the yardstick by which we measure code quality. Embold offers insights into various crucial metrics such as cyclomatic complexity, depth of inheritance, and coupling. These metrics give developers a clear picture of their codebase's health and areas that might require attention.
Repository Insights : Beyond just code, Embold offers a bird's eye view of the entire project. Visual representations of the codebase structure, dependencies, and hotspots of change equip teams to make informed decisions.
Integrations and CI/CD : In today's agile development environments, continuous integration and continuous delivery (CI/CD) are the norms. Embold seamlessly integrates into these pipelines, ensuring that code quality checks are an integral part of the development lifecycle.
Recommendations : One of the standout features of Embold is its ability to not just identify problems, but also suggest solutions. With its actionable recommendations, developers have a clear path towards improving their code.

Getting Started with Embold

As a part of the BrowserStack suite, starting with Embold requires a BrowserStack account. You can create your account directly from embold.io.

Creating Your Account :

Navigate to embold.io to begin the registration process.
Since it's under the BrowserStack umbrella, you might find options to explore other BrowserStack services. However, if your primary interest is in Embold, follow the registration prompts specific to it.

Linking Your Git Provider :

Once logged in, you'll be prompted to connect your preferred git provider. Embold supports a range of popular providers, ensuring a seamless integration process.
By linking your git provider, you're allowing Embold to access your repositories for scanning. Rest assured, Embold maintains strict security protocols, ensuring your code's privacy and safety.

Selecting Repositories for Analysis :

After connecting your preferred git provider, you'll be presented with a list of your repositories. Here, you can select which ones you'd like Embold to analyze.
Upon selection, Embold initiates its scanning process, diving deep into your code to identify potential issues, anti-patterns, and areas of improvement.

Navigating Embold's Dashboard

Embold's user interface is pretty self explanatory, after you selected one of your repos and Embold is finished analyzing, you will find a comprehensive overview of your codebase.

You will find deeper analysis over the results of your repo scan from the navigation menu which will give you information on Pull Requests, Commits, Found Issues, Duplication, Files as well as a blueprint of your repo in the form of a heat map or module dependency.

This way, you will be able to go into great detail on your code, even looking at individual files.

Conclusion

Embold makes it easy to scan your code for issues and receive in depth analysis. While we tried it on a small scale, this can especially come in handy for teams with codebases inceasing in complexity.

We recommend you to just connect your repo to see, if Embold might have found some issues you would otherwise have overlooked.

Building a Chrome Extension from Scratch using GitWit

Alexander Voll — Mon, 21 Aug 2023 16:10:47 +0000

Introduction

With the recent developments in AI and Machine Learning, there are a ton of AI tools popping up seemingly every day.

One that stands out is GitWit. Instead of trying to focus on creating full stack production-ready apps, it is better suited to create boilerplate code or add features to existing full-stack apps.

It is currently in Beta and not expected to yield perfect results on every iteration. However, we were able to build a small Chrome extension using GitWit with only minimal manual code work.

Starting out in GitWit

GitWit doesn't lose any time. As soon as you're logged in to the app, you're greeted with the project screen which allows you to select from a list of project base stacks.

GitWit automatically creates a repository for your project. You can configure the name of your repository in this step as well.

Once you click the create button, GitWit creates your boilerplate repository.

This alone is a really helpful step as it can help you save time on setting up your project.

One thing we noticed is, that for the Chrome extension, it created a manifest.json file with the manifest_version 2. This is not the newest version of the Chrome manifest and it is generally advised to use manifest 3. This probably has to do with the data accuracy and cut off date in GPT, however it is important to note that while GitWit will help you save time setting up your boilerplate, you should still check the set up to make sure everything works as intended.

Writing a prompt

In GitWit you write prompts in the form of revisions. The repo is initially set up and you can create a revision to modify the existing repo by clicking on the "Make a revision" button.

Upon clicking the button, you will be led to the revision screen, where you can type in your prompt. In this case, we're telling GitWit to add a color picker and a history of picked colors to our extension.

GitWit will then work on applying your changes and present them to you just like you're used from GitHub or other Git Providers.

You can then decide if you want to apply the proposed changes, skip entirely or try again.

Our Workflow

Since every GitWit project comes with a GitHub repo, it couldn't be simpler to just clone the repo and run it locally.

This way, you can iterate on and correct the changes made by GitWit if needed and then feed it back into GitWit to start your next iteration/revision.

This way, we set up this simple Chrome extension which features a color picker, color picker history and the possibility to clear the history in about 30 minutes - As someone who had never touched a Chrome extension before, ever.

Wrapping up

We really liked how it helped us set up a Chrome application in a breeze without any prior knowledge and how the workflow using GitHub goes seamlessly from repo to local to GitWit and back.

Anyone who has ever used GPT for coding purposes though will know, that it comes with its strengths and weaknesses. Those certainly also apply for GitWit.

In its current state, it probably isn't the right fit for a team working on huge applications but especially for full stack developers who like to try new technologies, it is a fun tool which can save you some time on the tasks most of you might dread.

It's free - give it a try and see for yourself!

How to create a custom Static Site Generator for Blogs in 2023

Alexander Voll — Mon, 14 Aug 2023 14:49:10 +0000

Introduction

Ever wondered how to set up a blazingly fast Blog for your personal project or company site without having to deal with complex security issues?

If so, you might have come around Static Site Generators. There are countless options out there that all cover specific use cases.

This however is also the issue with out-of-the-box solutions. They were built with a specific use case in mind.

At Codesphere, we are currently ramping up the decoupling of our website and blog from our main monorepo to become more lean in our marketing engineering efforts. We looked at different options and none fit our use case perfectly. Since we have a DIY-mindset here and like to build custom solutions for ourselves, we set out to build a custom static site generator for our blog. Turns out this isn't as hard of a task as it might initially sound.

In this article, I will explain, how we realized this. Feel free to code along or just clone and host the repo yourself for free with only a few clicks.

Let's get to it!

Project Setup

We will be using a fairly basic setup to not bloat our project.

Our static site generator will be running on a Node.js server using Express. The pages will be implemented using EJS and styled using TailwindCSS. The Posts data itself will be provided to our EJS template through a JSON file on our server.

You can also do the styling without Tailwind. You will find the copyStylesheets function in the generatePosts.js file of our repo. If you uncomment it, this will copy the stylesheets from your src to your repo directly. You can then simply uninstall tailwind using npm uninstall.

Static Site Generator

Our generator essentially consists of 3 functions (4 if you do it without Tailwind as described above).

generateIndexContent : Dynamically renders the index page template using EJS, including featured posts and a list of recent posts.
generatePostConten t: generates individual post pages, utilizing the EJS template for each post's content.
generateStaticSite : consolidates generateIndexContent and generatePostContent and uses the fs module to write the done sites to the public directory.

// const GhostContentAPI = require('@tryghost/content-api');
const path = require('path');
const fs = require('fs');
const ejs = require('ejs');
require('dotenv').config();

const outputDirectory = path.join(__dirname, '../public/posts/');
const indexDirectory = path.join(__dirname, '../public/');
const srcDirectory = path.join(__dirname, '../src/');
const postsData = path.join(__dirname, 'data/', 'data.json');

function generateIndexContent(posts) {
  // Use EJS to render the index template with the posts data
  const template = fs.readFileSync(path.join(__dirname, 'templates', 'indexTemplate.ejs'), 'utf-8');

  return ejs.render(template, {
      posts: posts,
      featuredPost: posts[0]
  }, {
      root: path.join(__dirname, 'templates')
  });
}

function ensureDirectoryExistence(directory) {
  if (!fs.existsSync(directory)) {
    fs.mkdirSync(directory, { recursive: true });
  }
}

function generatePostContent(post) {
  ensureDirectoryExistence(path.join(__dirname, '../public/posts/'));

  // Use EJS to render the post template with the post data
  return ejs.render(
    fs.readFileSync(path.join(__dirname, 'templates', 'postTemplate.ejs'), 'utf-8'), 
    {
      image: post.feature_image,
      title: post.title,
      content: post.html, // Content
      excerpt: post.excerpt
    }, {
      root: path.join(__dirname, 'templates') //Root directory for views
    }
  );
}

function saveToFile(filename, content) {
  fs.writeFile(filename, content, (err) => {
      if (err) {
          console.error(`Error saving ${filename}:`, err);
      } else {
          // console.log(`${filename} generated successfully!`);
      }
  });
}

async function generateStaticSite() {
  const posts = JSON.parse(fs.readFileSync(postsData, 'utf-8'));

  // Generate the main index page
  const indexContent = generateIndexContent(posts);
  saveToFile(path.join(indexDirectory, 'index.html'), indexContent);

  // Generate individual post pages
  for (const post of posts) {
    // console.log(post); // Debugging: print the post object
    const postContent = generatePostContent(post);
    saveToFile(path.join(outputDirectory, `${post.slug}.html`), postContent);
  }

  console.log("Generated static Post Pages!");

}

// copyStylesheets function that is necessary when not using TaileindCSS

// const stylesheets = ['index.css', 'article.css'];

// async function copyStylesheets() {
// for (const stylesheet of stylesheets) {
// const srcPath = path.join(srcDirectory, './styles', stylesheet);
// const destPath = path.join(indexDirectory, stylesheet);

// fs.copyFile(srcPath, destPath, err => {
// if (err) {
// console.log(`Error copying ${stylesheet}:`, err);
// } else {
// console.log(`${stylesheet} copied successfully!`);
// }
// });
// }
// }

generateStaticSite();
// copyStylesheets();

Server Setup

The server itself is also kept very minimalistic to reduce any overhead. Essentially, it initializes the EJS templating engine and creates dynamic routes based on the files in the posts directory, created by the generateStaticSite function.

const express = require('express');
const path = require('path');
const fs = require('fs');
const app = express();
const port = 3000;
const expressLayouts = require('express-ejs-layouts');

app.use(express.json());

app.use(express.static('public'));

app.use('/scripts', express.static(__dirname + '/node_modules/flowbite/dist/'));

//Set Templating Engine
app.use(expressLayouts);
app.set("view engine", "ejs");

// Function to read the generated HTML files
function getHTMLContent(slug) {
  const filePath = path.join(__dirname, 'public', 'posts', `${slug}.html`);

  try {
    return fs.readFileSync(filePath, 'utf-8');
  } catch (error) {
    console.error(`Error reading HTML file for ${slug}:`, error);
    return null;
  }
}

// Dynamic route creation
function createDynamicRoutes() {
  const pagesDirectory = path.join(__dirname, 'public', 'posts');

  fs.readdirSync(pagesDirectory).forEach((file) => {
    if (file.endsWith('.html')) {
      const slug = file.replace('.html', '');
      app.get(`/posts/${slug}`, (req, res) => {
        const pageContent = getHTMLContent(slug);
        if (pageContent) {
          res.send(pageContent);
        } else {
          res.status(404).send(`${slug} page not found`);
        }
      });
    }
  });
}

createDynamicRoutes();

app.listen(port, () => {
  console.log(`Server is running at http://localhost:${port}`);
});

EJS templates

With that up and running, the only thing missing are the EJS templates.

EJS, or Embedded JavaScript, is a templating engine that allows us to generate HTML content by embedding JavaScript directly within the template.

The data that can be used in the templates is passed using the .render method, provided by the express-ejs-layouts npm package. In our case this is executed, when the static pages are generated using the generateStaticSite function.

This leaves us with the following templates:

indexTemplate.ejs

<!DOCTYPE html>
<html>
  <head>
    <title>Codesphere blog</title>
    <meta name="viewport" content="width=device-width, initial-scale=1.0" />
    <link href="./main.css" rel="stylesheet" />
    <link href="./index.css" rel="stylesheet" />
    <!-- Additional meta tags, styles, scripts... -->
    <script>
      // Your JavaScript filtering logic...
    </script>
  </head>
  <body>
    <%- include("/partials/header.ejs") %>
    <section class="w-full bg-dark-bg flex justify-center items-center pb-44 pt-60 -mt-20 h-fit relative">
        <% for(let i = 0; i < 1 && i < posts.length; i++) { %> 
            <% let post = posts[i]; %>
            <div class="container w-4/5 h-full flex flex-col align-middle items-center gap-12 mx-auto justify-between z-40 featured-post">
                <!-- Image Div -->
                <div class="h-auto object-cover flex flex-shrink-0 rounded-md overflow-hidden w-full lg:w-1/3 lg:items-start lg:justify-start">
                    <img class="h-auto object-cover m-0 w-full" src="<%= post.feature_image %>" alt="<%= post.title %>" />
                </div>
                <!-- Text Div -->
                <div class="flex-shrink flex gap-4 flex-col relative text-center    ">
                    <h2 class="text-white text-5xl font-semibold break-words"><%= post.title %></h2>
                    <p class="text-gray-500 text-sm"><%= new Date(post.published_at).toLocaleDateString('en-US') %></p>
                    <p class="text-gray-300 text-md font-300"><%= post.excerpt %></p>
                    <a class="text-white text-md font-semibold" href="/posts/<%= post.slug %>">Read the full article</a>

                </div>
            </div>
        <% } %>
    </section>

    <div class="container mx-auto px-8">
        <section class="w-full flex justify-start items-start pb-44 pt-60 -mt-20 h-fit relative flex-col">
            <h1 class="text-black text-4xl font-semibold break-words">All Articles</h1>
            <div class="posts-grid grid grid-cols-3 gap-8 pt-16">
              <% for(let i = 0; i < posts.length; i++) { %>
                  <div class="post-item flex w-full flex-col">
                      <!-- Image Div -->
                      <div class="post-image h-auto object-cover flex flex-shrink-0 rounded-md overflow-hidden w-full">
                          <img class="h-auto object-cover m-0 w-full" src="<%= posts[i].feature_image %>" alt="<%= posts[i].title %>" />
                      </div>
                      <!-- Text Div -->
                      <div class="post-text flex-shrink flex gap-4 flex-col relative text-left">
                          <h2 class="text-black text-2xl font-semibold break-words mt-4"><%= posts[i].title %></h2>
                          <p class="text-gray-500 text-sm"><%= new Date(posts[i].published_at).toLocaleDateString('en-US') %></p>
                          <p class="text-gray-500 text-md font-300"><%= posts[i].excerpt %></p>
                          <a class="text-black text-md font-semibold" href="/posts/<%= posts[i].slug %>">Read the full article</a>
                      </div>
                  </div>
              <% } %>
          </div>

        </section>
    </div>
  </body>
</html>

postTemplate.ejs

<!DOCTYPE html>

<html>
    <head>
        <title>Codesphere blog</title>
        <meta name="viewport" content="width=device-width, initial-scale=1.0" />
        <link href="../main.css" rel="stylesheet" />
        <link href="../article.css" rel="stylesheet" />
        <!-- Additional meta tags, styles, scripts... -->
      </head>
    <body>
        <!DOCTYPE html>
        <html lang="en">

        <head>
            <meta charset="UTF-8">
            <meta http-equiv="X-UA-Compatible" content="IE=edge">
            <meta name="viewport" content="width=device-width, initial-scale=1.0">
            <title><%= title %></title>
        </head>

        <body class="bg-gray-100">

            <%- include("/partials/header.ejs") %>
            <section class="text-gray-700 body-font">
                <div class="container mx-auto flex px-5 py-24 flex-col items-center">
                    <article class="mx-auto w-full max-w-2xl format format-sm sm:format-base lg:format-lg format-blue dark:format-invert">
                        <h1 class="title-font sm:text-4xl text-3xl mb-4 font-bold text-gray-900">
                            <%= title %>
                        </h1>
                        <p class="mb-8 leading-relaxed">
                            <%= excerpt %>
                        </p>
                        <div class="post-image h-auto object-cover flex flex-shrink-0 rounded-md overflow-hidden w-full">
                            <img class="h-auto object-cover m-0 w-full" src="<%= image %>" alt="<%= title %>" />
                        </div>
                        <div class="mt-16 w-full">
                            <%- content %>
                        </div>
                    </article>
                </div>
            </section>


        </body>
        </html>

    </body>
</html>

Starting the server

We created a few scripts that will suit your needs depending on if you want to actively work on your blog or run it on the server for production.

To start the server and have the templates and css files watched, run:

npm watch:generate

To start the server for production, run:

npm start

Or just use Codesphere to host your project in a matter of minutes!

Setting up in Codesphere

You can now deploy your static blog for free on Codesphere.

Just open up a new workspace using the GitHub Repo link, run the CI steps from the UI and you're done!

Wrapping up

In conclusion, building a custom static site generator might seem like a daunting task, but with the right approach, it's entirely feasible. By using a combination of Node.js, Express, EJS, and TailwindCSS, we were able to create a lean, efficient, and customizable generator tailored to our specific needs at Codesphere. Not only does this setup offer flexibility, but it also provides the speed and security benefits inherent to static sites.

Happy coding!

Building an Email Marketing Engine Part 5: Onboarding Automation Sequence with BullMQ

Alexander Voll — Tue, 08 Aug 2023 14:41:44 +0000

Introduction

Welcome back to our "Building an Email Marketing Engine" series where we use Node.js, Express.js, SQLite, SendGrid API and now BullMQ to create an email marketing engine to bypass pricey email marketing providers.

In our previous parts, we went about setting up the application, integrate it with SendGrid, creating a simple SQLite DB to store our contacts, create a login system using Auth0 and created Endpoints to add contacts to our db using external services.

In this part of our series, we will work on the trickiest part so far: automating email sending with custom delayed emails. We will do this by expanding the capabilities of our SQLite DB and implementing BullMQ for scheduling and handling emails.

As always, you can check the GitHub Repo for the latest status of our development.

Let's get to it!

Creating necessary tables

To accomodate automation, I firstly had to adjust our existing database structure. The goal was to create 3 new tables that would house automation tasks, sequences and templates for those sequences.

The logic here is as follows:

Automation Sequences

Automation Sequences are the highest level of our automation system. They store the name and the description of a sequence and are defined by the user.

Sequence Task Templates

Sequence Task Templates house the basic task information needed for a specific template. For an onboarding flow that might be 3 different emails being sent to the user for 3 days each after signing up to a service.

Automation Tasks

Automation Tasks are the actual tasks that are being scheduled. They extend the template with a few more columns, including the status, the due time or a task set id. The task set id is particularly important, since it makes tasks identifyable to a specific launched sequence. Staying in the example of an onbaording flow, each user would have 3 tasks (emails to be sent) and all of them would have the same set id.

Handling Storage of Tasks in SQLite

After those new tables were created, I started creating the logic for storing tasks in SQLite. Keep in mind that sequence templates and their according tasks are created by the user. We will implement this in a nice way when working on the frontend.

There are two main functions that handle the automation in the SQLite db are:

createTasksFromSeqId
sendTasks

Both of those make use of a number of helper functions I wrote beforehand. Most of them are very simple and do things like getting a specific task from the automation_tasks table or creating a unique id for task sets.

async function getSequenceTemplateBySeqId(sequenceId) {
  const db = await getDatabaseConnection();

  const sql = `SELECT * FROM sequence_tasks_template WHERE sequence_id = ?`;

  return new Promise((resolve, reject) => {
    db.all(sql, [sequenceId], (err, rows) => {
      if (err) {
        reject(err);
      } else {
        resolve(rows);
      }
    });
  });
}

To see all helper functions, take a look at the GitHub Repo, where you will find all of them in the utils.js file.

The `createTasksFromSeqId` function

This is an async function takes in two parameters: sequenceId and contactId.

The function generates a unique taskSetId , then tries to fetch a sequence template using the sequenceId. If successful, it sorts the sequence template based on the order_position property.

Next, it iterates through each task in the sequence template. For the first task, it sets the dueTime to the current date and the status to "pending". For subsequent tasks, it sets the dueTime to null and the status to "unscheduled".

Then, it calls a function called createAutomationTask passing in various parameters including the taskSetId, dueTime, and status for each task.

Finally, the function returns the taskSetId.

async function createTasksFromSeqId(sequenceId, contactId) {
    const taskSetId = generateTaskSetId();

    let sequenceTemplate;
    try {
        sequenceTemplate = await getSequenceTemplateBySeqId(sequenceId);
        sequenceTemplate.sort((a, b) => a.order_position - b.order_position);
    } catch (error) {
        console.error("Error fetching sequence template:", error);
    }

    for (let i = 0; i < sequenceTemplate.length; i++) {
        const task = sequenceTemplate[i];
        let dueTime;
        let status;

        if (i === 0) {
            // The first task
            dueTime = new Date(); // Set to "now"
            status = "pending";
        } else {
            dueTime = null; // Or whatever placeholder you want to use to indicate it's not scheduled yet
            status = "unscheduled";
        }

        await createAutomationTask(
            task.sequence_id,
            contactId,
            task.subgroup_id,
            task.template_id,
            task.order_position,
            task.delay,
            taskSetId,
            dueTime,
            status
        );
    }

    return taskSetId;
}

The `sendTasks` function

This is where the actual automation happens and tasks are being scheduled and sent to the BullMQ queue.

The sendTasks function is an async function too, that sends tasks to a specified task set. It is used to be called after the createTasksFromSeqId function, using the returned taskSetId.

It retrieves scheduled tasks for the given task set ID, sorts them based on their order position, and then iterates over each task.

For each task, it checks if the task's status is "pending" and if its due time has passed. If so, it retrieves the contact data for the task's associated contact ID and sends an email to the contact using a specified SendGrid template ID.

It then marks the task as completed in a SQL database. If there is a next task that is unscheduled, it schedules the next task based on the completed task's information and a delay. Finally, it breaks the loop after processing the first task that meets the conditions. If any errors occur during the process, they are logged to the console.

async function sendTasks(taskSetId) {
  try {
    const scheduledTasks = await getScheduledTasksBySetId(taskSetId);
    scheduledTasks.sort((a, b) => a.order_position - b.order_position);

    for (let i = 0; i < scheduledTasks.length; i++) {
      const currentTask = scheduledTasks[i];

      if (currentTask.status === "pending" && currentTask.due_time <= Date.now()) {
        const contactData = await getContactById(currentTask.contact_id);

        try {
          console.log("Sending email to:", contactData[0].email);
          await sendSingleEmail(contactData[0].email, currentTask.template_id);
          await completeTask(currentTask.id, Date.now());
          console.log("Task completed in SQL db");
        } catch (error) {
          console.error("Error sending email:", error);
          continue;
        }

        const nextTask = scheduledTasks[i + 1];
        if (nextTask && nextTask.status === "unscheduled") {
          const completedTask = await getTaskById(currentTask.id);
          const updatedTask = await scheduleNextTask(completedTask, nextTask);
          const scheduledTaskData = {
            sequence_id: updatedTask.sequence_id,
            contact_id: updatedTask.contact_id,
            subgroup_id: updatedTask.subgroup_id,
            template_id: updatedTask.template_id,
            order_position: updatedTask.order_position,
            delay: updatedTask.delay,
            task_set_id: updatedTask.task_set_id,
            dueTime: updatedTask.due_time,
            status: updatedTask.status
          };

          const delay = nextTask.delay * 1000;
          await scheduleBullTask(scheduledTaskData, delay);
        }

        break;
      }
    }
  } catch (error) {
    console.error("Error fetching scheduled tasks:", error);
  }
}

As you can see, if a nextTask that is unscheduled is found, its status is then set to "pending" by the scheduleNextTask helper function and its being scheduled and added to the bull queue using scheduleBullTask .

async function scheduleNextTask(lastCompletedTask, currentTask) {
  if (!lastCompletedTask || !lastCompletedTask.completed_at) {
    return { dueTime: new Date().toISOString(), status: "pending" };
  }

  const lastCompletedAtTimestamp = parseInt(lastCompletedTask.completed_at, 10);
  const lastCompletedAt = new Date(lastCompletedAtTimestamp);
  const delayDuration = currentTask.delay * 1000;
  const dueTime = new Date(lastCompletedAt.getTime() + delayDuration);
  const status = "pending";

  const db = await getDatabaseConnection();

  const sql = `UPDATE automation_tasks SET due_time = ?, status = ? WHERE id = ?`;

  return new Promise((resolve, reject) => {
    db.run(sql, [dueTime.getTime(), status, currentTask.id], (err) => {
      if (err) {
        reject(err);
      } else {
        // Fetch the updated row data
        const fetchSql = `SELECT * FROM automation_tasks WHERE id = ?`;
        db.get(fetchSql, [currentTask.id], (fetchErr, row) => {
          if (fetchErr) {
            reject(fetchErr);
          } else {
            resolve(row);
          }
        });
      }
    });
  });
}

Using BullMQ to automate sending of scheduled tasks

BullMQ is an advanced Node.js queue and job service solution built on top of the Redis database. With BullMQ, you can manage jobs in various states (waiting, active, delayed, etc.), set job priorities, schedule recurrent jobs, and handle delayed jobs.

This is incredibly useful for our case since it's meant for big application and very easy to handle.

A Bull queue essentially consists of 3 different parts:

Producers: They produce a specific job and add it to the queue
Job Queue: This is where the actual jobs are saved, it is housed in a Redis database
Consumers/Workers: They take care of handling the jobs in the job queue

Our setup looks like this:

Initialization and Producer

We initialize the Queue and give it a name. Then, we define our producer like a regular asynchronous function called scheduleBullTask that schedules a new task to be executed later. It adds a job to a task queue with some task data and a specified delay.

The delay is defined in the sendTasks when scheduleBullTask is called. It is retreived from the SQLite database. This is the reason why (for now) we use SQLite for storing the tasks while using BullMQs Redit database to store scheduled tasks. We want to make sure that tasks are only added t0 the job queue once they've been executed. This might change in the future though.

const { Queue } = require('bullmq');

//initialize new Queue
const taskQueue = new Queue("taskQueue", {
    connection: {
        host: "127.0.0.1",
        port: 6379
    }
});

//Producer: New Task is scheduled
async function scheduleBullTask(taskData, delay) {
    await taskQueue.add('myJobName', {
        sequence_id: taskData.sequence_id,
        contact_id: taskData.contact_id,
        subgroup_id: taskData.subgroup_id,
        template_id: taskData.template_id,
        order_position: taskData.order_position,
        delay: taskData.delay,
        task_set_id: taskData.task_set_id,
        dueTime: taskData.dueTime,
        status: taskData.status
    },
        {
            delay: delay
        });
}

module.exports = {
    scheduleBullTask
}

Worker

Our worker defines, that once a task is ready to be executed, it accesses the job data (which houses the task information from our SQLite DB) and executed the sendTasks function which then sends the task and schedules the next one (if it exists), in turn creating a new job queue entry. This happens until all tasks are set to "completed" in the SQLite database, which is when no more jobs are added to the queue.

const { Worker } = require('bullmq');

const { sendTasks } = require("../automationJobs")

//Initialize Worker for handling
const taskProcessorWorker = new Worker("taskQueue", async (job) => {
    const taskData = job.data;

    await sendTasks(taskData.task_set_id);

    console.log("sent email for task with task_set_id:", taskData.task_set_id);

    //mark the job as completed
    return {
        id: taskData.id,
        sequence_id: taskData.sequence_id,
        status: "completed"
    };
}, {
    connection: {
        host: "127.0.0.1",
        port: 6379
    }
})

taskProcessorWorker.on('completed', (job) => {
    console.log(`Job ${job.id} Completed`);
})

taskProcessorWorker.on('failed', (job, err) => {
    console.error(`Job ${job.id} failed with error ${err.message}`);
});

Creating an endpoint for automation

To make use of our newly created automation flow, I created a new endpoint which allows for external services to send a request, add a user to the database and start the automation flow.

For this, I made use of the addContact function we created in our last installment with some slight tweaks.

apiRouter.post("/addContactOnboarding", apiKeyMiddleware, async (req, res) => {
  try {
      const contactId = await addContact(req);
      console.log("contactId",contactId);
      const taskSetId = await createTasksFromSeqId(1, contactId);
      sendTasks(taskSetId);
      res.json({ message: `Onboarding flow started with taskSetId: ${taskSetId}` });
  } catch (error) {
      console.error(error);
      res.status(500).json({ error: error.message });
  }
});

This allows for us to send a POST request to the defined endpoint with body that includes some information about the new contact and then automatically start the onboarding flow.

{
  "contact": {
    "sendgrid_contact_id": "",
    "email": "max@mustermann.com",
    "first_name": "Max",
    "last_name": "Mustermann"
  },
  "subgroupIds": [1, 2]
}

Wrapping up

In this installment, we've successfully navigated the complexities of automating email sending using SQLite and BullMQ. By restructuring our database and leveraging BullMQ's capabilities, we've built a robust system that schedules and sends custom delayed emails efficiently. With the addition of a new endpoint, integrating external services and automating the onboarding flow is now seamless. As we continue to refine and enhance our email marketing engine, we will iterate on this, optimizing it even further.

Also, we're coming closer towards creating a frontend for our app!

As always, you can take a look at the latest status of the Email Marketing Engine in our GitHub Repo. To host the application yourself, just use our magic Codesphere link to get you started in only a few clicks!

Building an Email Marketing Engine Part 4: API Endpoints & Database Enhancements

Alexander Voll — Mon, 31 Jul 2023 15:29:23 +0000

Welcome back to our series on building an email marketing engine using Node.js, Express.js, SQLite, and SendGrid. In the previous parts, we set up our application, integrated it with SendGrid, and created a simple SQLite database to store our contacts. In Part 3, we introduced a custom login system with Auth0

In this article, we will be extending our database and creating API endpoints to upload and delete contacts. These enhancements will allow us to manage our contacts more effectively and provide a foundation for more complex operations in the future.

Extending our SQLite Database

SQLite is a great choice for our application due to its simplicity and ease of use. To start, we already created a Contacts table to store our contacts' information. Now, we're going to extend our database by introducing a subgroups table and a linking table called contact_subgroup.

Creating the `subgroups` Table

We want to be able to categorize our contacts into different subgroups for targeted marketing. To do this, we need a new table called subgroups. This table will have two columns:

id: A unique identifier for each subgroup.
name: The name of the subgroup.

Here is the SQL statement to create the subgroups table:

CREATE TABLE subgroups (
   id INTEGER PRIMARY KEY AUTOINCREMENT,
   name TEXT
);

Creating the `contact_subgroup` Linking Table

Since a contact can belong to multiple subgroups and a subgroup can have multiple contacts, we have a many-to-many relationship between contacts and subgroups. To represent this relationship in SQLite, we need a linking table.

The ContactsSubgroups table has two columns:

contact_id: The ID of a contact, which is a foreign key referencing contacts.id.
subgroup_id: The ID of a subgroup, which is a foreign key referencing subgroups.id.

Here's the SQL statement to create the contact_subgroup table:

CREATE TABLE contact_subgroup (
   contact_id INTEGER,
   subgroup_id INTEGER,
   PRIMARY KEY (contact_id, subgroup_id),
   FOREIGN KEY (contact_id) REFERENCES contacts (id),
   FOREIGN KEY (subgroup_id) REFERENCES subgroups (id)
);

Now, we have a much more flexible system for categorizing our contacts. We can assign a contact to multiple subgroups and select contacts based on their assigned subgroup.

Implementing API Endpoints

With our database now ready, we need a way to interact with it. That's where our API endpoints come in. We will create two endpoints: one for adding a contact and one for deleting a contact.

The `addContact` Endpoint

The addContact endpoint will receive a POST request with the contact's details in the request body. These details include the contact's email, first name, last name, and optionally, the ID of the subgroup that the contact belongs to.

For this, we create a new file called dataToDb.js where we define the addContact function:

function addContact(req, res) {
    // Check if the contact object is provided in the request body
    if (!req.body || !req.body.contact) {
        return res.status(400).json({ error: 'Contact data is missing in the request body.' });
    }

    // Deconstruct the contact and subgroupIds from the request body
    const { contact, subgroupIds } = req.body;

    // Validate the contact object
    const { sendgrid_contact_id, email, first_name, last_name } = contact;
    if (!sendgrid_contact_id || !email || !first_name || !last_name) {
        return res.status(400).json({ error: 'Some contact properties are missing.' });
    }

    // Open a connection to the database
    let db = new sqlite3.Database('./server/data/contacts.db', sqlite3.OPEN_READWRITE);

    // Prepare the SQL for inserting the contact
    let sql = 'INSERT INTO contacts (sendgrid_contact_id, email, first_name, last_name) VALUES (?, ?, ?, ?)';

    // Run the SQL query
    db.run(sql, [sendgrid_contact_id, email, first_name, last_name], function (err) {
        if (err) {
            return res.status(500).json({ error: err.message });
        }

        const contactId = this.lastID;

        // If subgroupIds were provided, associate the contact with each subgroup
        if (subgroupIds && Array.isArray(subgroupIds)) {
            const subgroupSql = 'INSERT INTO contact_subgroup (contact_id, subgroup_id) VALUES (?, ?)';

            // Iterate over each subgroupId
            subgroupIds.forEach(subgroupId => {
                db.run(subgroupSql, [contactId, subgroupId], function (err) {
                    if (err) {
                        return res.status(500).json({ error: err.message });
                    }
                });
            });

            // Respond with a success message
            res.json({ message: `Contact and subgroup relationships have been created with contactId ${contactId}` });
        } else {
            // Respond with a success message
            res.json({ message: `Contact has been inserted with rowid ${contactId}` });
        }
    });

    // Close the database connection
    db.close((err) => {
        if (err) {
            console.error(err.message);
        }
    });
}

Upon receiving a request, the addContact function validates the provided data, extracts essential information, and connects to the database. It then prepares an SQL query to insert the contact details into the contacts table. If successful, it obtains the contactId of the new contact and, if applicable, associates the contact with provided subgroups in the contact_subgroup table. The function responds with appropriate success messages, and it closes the database connection after the operations are completed.

The `deleteContact` Endpoint

The deleteContact endpoint receives a DELETE request with the ID of the contact to delete in the URL. It deletes the contact from the Contacts table and also removes any links between the contact and subgroups in the ContactsSubgroups table.

Here's how the deleteContact function looks in our dataToDb.js file:

function deleteContact(req, res) {
    const contactId = req.params.contactId;

    if (!contactId) {
        return res.status(400).json({ error: 'Contact id is required.' });
    }

    let db = new sqlite3.Database('./server/data/contacts.db', sqlite3.OPEN_READWRITE);

    // Prepare the SQL for deleting the contact
    let sql = 'DELETE FROM contacts WHERE id = ?';

    // Run the SQL query
    db.run(sql, [contactId], function (err) {
        if (err) {
            return res.status(500).json({ error: err.message });
        }

        // Respond with a success message
        res.json({ message: `Contact with id ${contactId} has been deleted` });
    });

    // Close the database connection
    db.close((err) => {
        if (err) {
            console.error(err.message);
        }
    });
}

The deleteContact expects the contactId as a parameter in the request. If the contactId is missing, it responds with an error. Otherwise, it connects to the contacts.db database, prepares an SQL query to delete the specified contact from the contacts table, and executes the query. If successful, it responds with a success message.

Protecting API Endpoints with an API Key

Security is a crucial aspect of any web application. For our API endpoints, we'll use a simple but effective method of protection: an API key. An API key is a secret token that a client must send in their request to access our endpoints.

We'll define a middleware function, apiKeyMiddleware, that checks if the incoming request includes a valid API key. This middleware will be used in our addContact and deleteContact routes.

Here's the middleware definition in our main server file:

const API_KEY = 'api-key';
const apiKeyMiddleware = (req, res, next) => {
  const reqKey = req.get('x-api-key');
  if (reqKey && reqKey === API_KEY) {
    next();
  } else {
    res.status(403).json({error: "Invalid or missing API key"});
  }
};

We then use this middleware in our routes as follows:

apiRouter.post('/addContact', apiKeyMiddleware, addContact);
apiRouter.delete('/deleteContact/:contactId', apiKeyMiddleware, deleteContact);

With this setup, only requests that include the correct API key in the x-api-key header will be able to access the addContact and deleteContact endpoints.

Adjusting the Server Setup

In our previous setup, we had all our routes handled directly by our main app object. We also didn't differentiate between different types of routes, like those for our API versus our application.

However, as we move forward, our application grows and we need to separate these concerns. We don't want to apply the same set of rules to all routes. For example, we want to secure our API with an API key and our application with Auth0. To achieve this, we introduce two new routers, apiRouter and appRouter, using express.Router().

const apiRouter = express.Router();
const appRouter = express.Router();

Routers are like mini Express applications that only have middleware and routing methods. We use the apiRouter to handle all our API routes and appRouter to handle application routes.

We then apply the API key middleware only to the apiRouter and Auth0 middleware to the appRouter:

apiRouter.post('/addContact', apiKeyMiddleware, addContact);
apiRouter.delete('/deleteContact/:contactId', apiKeyMiddleware, deleteContact);
...
appRouter.use(auth(auth0Config));
...
appRouter.post("/contacts", requiresAuth(), async (req, res) => {...});
appRouter.post('/upload', requiresAuth(), upload.single('contacts-upload'), (req, res) => {...});
appRouter.get('/profile', requiresAuth(), (req, res) => {...});

Now, the different routers are applied to different sets of routes. We use them in our main app like so:

app.use('/api', apiRouter);
app.use('/', appRouter);

This way, any request to the /api endpoint will go through the apiRouter and will be checked for the API key. Any other request will be handled by the appRouter, which is secured by Auth0.

With these changes, our application is not only more organized, but it also handles different types of requests with different security requirements. This allows us to better control access to our resources and provides a more secure and efficient system.

Wrapping Up

In this article, we extended our database and created API endpoints for adding and deleting contacts. We also went over how to handle many-to-many relationships in SQLite. With these enhancements, our email marketing engine is becoming more powerful and flexible.

In the next part of this series, we will be looking at how to send emails to specific subgroups and fixing up the frontend. Stay tuned!

Building an Email Marketing Engine Part 3: Securing Your Frontend with Auth0

Alexander Voll — Mon, 24 Jul 2023 13:19:56 +0000

In the previous articles of this series, we've covered the initial setup of our email marketing engine and how to send emails via the SendGrid API to contacts stored in a SQLite database. Now, let's dive into an essential part of any web application - security. In this article, we'll see how we can secure our frontend using Auth0.

Auth0 and Express-openid-connect

Auth0 is a flexible, drop-in solution to add authentication and authorization services to your applications. Your team and organization can avoid the cost, time, and risk that comes with building your own solution to authenticate and authorize users.

We used the express-openid-connect library, which is an Auth0 developed middleware that makes it super easy to set up authentication in your Express.js apps.

Let's delve into our code:

const { auth } = require('express-openid-connect');

require('dotenv').config();

const config = {
  authRequired: true,
  auth0Logout: true,
  secret: 'a long, randomly-generated string stored in env',
  baseURL: 'https://your_application_domain.com',
  clientID: 'your_client_id',
  issuerBaseURL: 'https://your_domain.region.auth0.com'
};

app.use(auth(config));

In this snippet, we require the auth function from express-openid-connect and dotenv to load environment variables. We then define a configuration object, config, that we pass to the auth function.

Here's what each configuration option does:

authRequired: This is a boolean that, when true, requires authentication for all routes.
auth0Logout: When true, this enables Auth0's logout feature.
secret: This is a long, randomly-generated string that is used to encrypt and sign the session cookie. It is important that this is kept secure and not checked into version control. It should be stored in an environment variable.
baseURL: This is the URL of your application. This is used by the middleware to determine the callback URL and the post-logout redirect URL.
clientID: This is your Auth0 application's Client ID.
issuerBaseURL: This is the URL of your Auth0 tenant.

Please note that these configuration values, especially secret, clientID, and issuerBaseURL, should not be hard-coded as they are sensitive pieces of information. They should be stored securely in environment variables and loaded into the application using dotenv or a similar package.

require('dotenv').config();

The dotenv package loads environment variables from a .env file into process.env. This means that when we run require('dotenv').config(), we can access the values in our .env file as if they were in the system's environment variables. This is a common practice to handle configuration in a secure, manageable way.

You can adjust the look and settings of your login page in the Auth0 dashboard. The default page will look like this:

Mounting and initializing auth()

This is a pretty basic setup. As soon as we mount and initialize auth(), all routes that are initialized afterwards will be protected by Auth0s login.

We insert this code snippet into our index.js which we set up in the previous parts of our series:

const express = require('express');
const app = express();
const port = 3000;

const multer = require('multer');
const upload = multer({ dest: 'uploads/' });

const { sendEmails } = require('./server/api');
const { processContacts } = require('./server/utils');
const { uploadToDb } = require('./server/uploadToDb');
const { auth } = require('express-openid-connect');

require('dotenv').config();

const config = {
  authRequired: true,
  auth0Logout: true,
  secret: process.env.SECRET,
  baseURL: process.env.BASE_URL,
  clientID: process.env.CLIENT_ID,
  issuerBaseURL: process.env.ISSUER_BASE_URL
};

app.use(auth(config));

app.use(express.json());
app.use(express.static('public'));

app.all("*", (req, res, next) => {
  console.log(`Received a ${req.method} request on ${req.originalUrl}`);
  next();
});

app.post("/contacts", async (req, res) => {
  console.log(req.body);

  const { templateId } = req.body;

  try {
    const emailGroups = await processContacts(99);

    // Send emails to each group
    for (let group of emailGroups) {
        await sendEmails(group, templateId);
    }

    res.json(emailGroups);
  } catch (error) {
    res.status(500).json({ error: "An error occurred while fetching contacts" });
  }
});

app.post('/upload', upload.single('contacts-upload'), (req, res) => {

  uploadToDb(req.file.path);

  res.send('File uploaded successfully');

});

app.listen(port, () => {
  console.log(`Server is running at http://localhost:${port}`);
});

Only allowing specific email domains

When building internal tools, you might want to restrict your login to company emails only.

Unfortunately, Auth0 doesn't support this for OAuth2 logins like Google or GitHub, however you can restrict the login to specific domains or emails when using regular Email/Password login. This is done by utilizing Auth0s Action Flows, which allow you to insert custom Node.js code in specific places of the login flow.

To restrict specific user Emails, you would use a "Pre User Registration" flow, which is called when a user tries to sign up before the signup in complete.

Auth0 Action Flow Menu

You can then use Auth0s neat drag and drop editor, to either insert predefined actions or create a custom one.

Auth0's drag and drop Action editor

To restrict email domains to a specific whitelist, you can create a small JS snippet which will then run on the Auth0 server.

In our case, we created the following custom snippet, which only allows users with a "codesphere.com" email to sign up:

exports.onExecutePreUserRegistration = async (event, api) => {
  console.log(event); // log the event object

  const whitelist = ['codesphere.com'];

  const emailSplit = event.user.email.split("@");
  const userDomain = emailSplit[emailSplit.length - 1].toLowerCase();

  // If userDomain is not part of the whitelist, deny access
  if (!whitelist.includes(userDomain)) {
    const LOG_MESSAGE = "User not part of Codesphere"
    const USER_MESSAGE = "Access to this tool is only available for employees of Codesphere"
    api.access.deny(LOG_MESSAGE, USER_MESSAGE);
  }
};

To make this more secure, it would be advisable to allow only specific emails and adjust the whitelist to accept new users if needed.

Conclusion

Securing your application is a critical task, and thankfully, with services like Auth0 and libraries like express-openid-connect, it becomes a lot simpler. By integrating Auth0 into our email marketing engine, we've added a robust layer of security with just a few lines of code and made sure that only specific domains can log in.

DEV Community: Alexander Voll

Building an AI powered search in Node.js using Qdrant Cloud

Introduction

What is Qdrant

Understanding Vector Databases

The Workflow for my Blog Search

Setting Up Qdrant in JavaScript

Qdrant Client Initialization

Installing Required Libraries

Text Encoding with Xenova

1. Creating a Collection with Qdrant

Function: recreateQdrantCollection

2. Checking Existing Articles in the Database

Function: scrollThroughPoints

3. Creating Embeddings for Missing Articles

Function: encodeText

4. Uploading to Qdrant

Function: uploadDocuments

Implementing AI-Powered Search

Function: searchArticles

Conclusion

OpenAIs GPTs with a third party API: Does it work?

Introduction

Setting up a new GPT

GPT Actions

OpenAPI Spec as the foundation

Instructions

API Calls

Limited Debugging Capabilities

Known bugs without known fixes

Conclusion

Migrating Node.js to Bun: Really a drop in replacement?

Intro

Buns Bundler

Watch mode

Entrypoints

Installing Dependencies

The Websocket issue

Conclusion

Improving Code Quality using Embold

Introduction

What is Embold?

Key Features of Embold

Getting Started with Embold

Conclusion

Building a Chrome Extension from Scratch using GitWit

Introduction

Starting out in GitWit

Writing a prompt

Our Workflow

Wrapping up

How to create a custom Static Site Generator for Blogs in 2023

Introduction

Project Setup

Static Site Generator

Server Setup

EJS templates

indexTemplate.ejs

postTemplate.ejs

Starting the server

Setting up in Codesphere

Wrapping up

Building an Email Marketing Engine Part 5: Onboarding Automation Sequence with BullMQ

Introduction

Creating necessary tables

Handling Storage of Tasks in SQLite

The createTasksFromSeqId function

The sendTasks function

Using BullMQ to automate sending of scheduled tasks

Initialization and Producer

Worker

Creating an endpoint for automation

Wrapping up

Building an Email Marketing Engine Part 4: API Endpoints & Database Enhancements

Extending our SQLite Database

Creating the subgroups Table

Creating the contact_subgroup Linking Table

Implementing API Endpoints

The addContact Endpoint

The deleteContact Endpoint

Function: `recreateQdrantCollection`

Function: `scrollThroughPoints`

Function: `encodeText`

Function: `uploadDocuments`

Function: `searchArticles`

The `createTasksFromSeqId` function

The `sendTasks` function

Creating the `subgroups` Table

Creating the `contact_subgroup` Linking Table

The `addContact` Endpoint

The `deleteContact` Endpoint