DEV Community: Chris Frewin

Two of My Favorite Custom MCP Tools I Use Every Day

Chris Frewin — Sun, 11 May 2025 10:50:33 +0000

My Two Favorite MCP Tools - True Game Changers

I've been building custom MCP tools for a month or so now, and just two of them have absolutely transformed how I work with Claude - they were the real 'aha!' moment for me and 'agentic' workflows. Now I get the MCP hype - at least partially :)

So, here they are: these two tools essentially give me Claude Code functionality without the extra cost:

list_repo_locations - A smart repository finder that uses fuzzy matching and helpful hints when matches aren't found
issue_terminal_command - A controlled terminal execution tool, with a 'yolo' option to go beyond read-only mode

Let me walk you through how to build these tools from scratch. By the end of this post, you'll have your own versions that you can customize to your heart's content!

The current MCP server I use locally on my own machine is published here: https://github.com/princefishthrower/mcp-server

I've been reading lots of complaints and confusion about MCP in general - while I agree some of the decisions and examples are flakey at best, the above link is a real MCP server that actually works and is useful.

Tool #1: `list_repo_locations`

I use this tool to scope what codebases I want Claude to find while just using natural language. Instead of trying to explain directory structures to Claude, I can just say something like "Take a look in the super_widgets repo and help me hunt down this bug..." and boom - it knows exactly where to look.

Create the Basic Structure

First, let's set up the basic TypeScript file for our list_repo_locations tool:

// src/tools/list_repo_locations.ts
import { levenshteinDistance } from "../utils/levenshteinDistance.js";

// We'll organize repos by environment for different projects
const workRepoMap: Array<{ keywords: string[], repoPath: string}> = [
    {
        keywords: ["main-project", "core", "backend"],
        repoPath: "/Users/yourname/work/main-project"
    },
    {
        keywords: ["documentation", "docs", "wiki"],
        repoPath: "/Users/yourname/work/documentation"
    },
    {
        keywords: ["frontend", "ui", "client"],
        repoPath: "/Users/yourname/work/frontend"
    }
];

const personalRepoMap: Array<{ keywords: string[], repoPath: string}> = [
    {
        keywords: ["blog", "website", "personal-site"],
        repoPath: "/Users/yourname/personal/blog"
    },
    {
        keywords: ["side-project", "experiment"],
        repoPath: "/Users/yourname/personal/side-project"
    }
];

Add Fuzzy Matching Magic

We'll use fuzzy matching using Levenshtein distance. When you (or Claude itself) types something close to what you mean, it still finds the right repo.

We're in a new era where it helps when your tools are built using natural language principles. Your AI assistant shouldn't require perfect spelling or exact keywords - it should understand instead the meaning of what you want, including both successes and mistakes, just like a human would.

export const listRepoLocations = async (environment: string, keyword: string) => {
    const keywordLower = keyword.toLowerCase();
    let keywordRepoMap: Array<{ keywords: string[], repoPath: string}> = [];

    // Select the right environment
    if (environment === "work") {
        keywordRepoMap = workRepoMap;
    } else if (environment === "personal") {
        keywordRepoMap = personalRepoMap;
    }

    // First, try exact or substring matches
    const exactMatches = keywordRepoMap.filter(repo => 
        repo.keywords.some(kw => kw.toLowerCase().includes(keywordLower))
    );

    if (exactMatches.length > 0) {
        return {
            content: [{
                type: "text",
                text: `Found matching repo(s): ${exactMatches.map(repo => repo.repoPath).join(", ")}`
            }],
            isError: false
        };
    }

    // If no exact matches, use fuzzy matching
    const DISTANCE_THRESHOLD = 3; // This is where the NLP magic happens!
    const fuzzyMatches: Array<{repo: typeof keywordRepoMap[0], keyword: string, distance: number}> = [];

    for (const repo of keywordRepoMap) {
        for (const kw of repo.keywords) {
            const distance = levenshteinDistance(keywordLower, kw.toLowerCase());
            if (distance <= DISTANCE_THRESHOLD) {
                fuzzyMatches.push({repo, keyword: kw, distance});
            }
        }
    }

    // Sort by distance (closest matches first)
    fuzzyMatches.sort((a, b) => a.distance - b.distance);

    if (fuzzyMatches.length > 0) {
        // This is where helpful error handling comes in - we're not just saying "not found"
        // We're providing context and suggestions, making the tool more conversational
        const suggestions = fuzzyMatches
            .slice(0, 5)
            .map(match => `${match.keyword} (${match.repo.repoPath})`)
            .join(", ");

        return {
            content: [{
                type: "text",
                text: `No exact matches found for '${keyword}'. Did you mean: ${suggestions}?`
            }],
            isError: true
        };
    }

    // Final fallback with full context - still being helpful!
    return {
        content: [{
            type: "text",
            text: `No matching repo found for '${keyword}'. Available keywords: ${keywordRepoMap.map(repo => repo.keywords.join(", ")).join("; ")}`
        }],
        isError: true
    };
};

Implement the Levenshtein Distance Util Function

This is where the NLP magic happens! Levenshtein distance calculates how many single-character edits (insertions, deletions, or substitutions) are needed to change one word into another. It's a classic algorithm that makes our tools much more human-friendly. (I'm glad I've got a bit of experience in these NLP things - you can see my old nearly 10+ year old experiments over at my archived site, NLP Champs!)

Here's the utility function you'll need:

// src/utils/levenshteinDistance.ts
export function levenshteinDistance(str1: string, str2: string): number {
    const m = str1.length;
    const n = str2.length;
    const matrix: number[][] = [];

    // Initialize matrix
    for (let i = 0; i <= m; i++) {
        matrix[i] = [i];
        for (let j = 1; j <= n; j++) {
            matrix[0][j] = j;
        }
    }

    // Calculate distance
    for (let i = 1; i <= m; i++) {
        for (let j = 1; j <= n; j++) {
            const cost = str1[i - 1] === str2[j - 1] ? 0 : 1;
            matrix[i][j] = Math.min(
                matrix[i - 1][j] + 1,     // deletion
                matrix[i][j - 1] + 1,     // insertion
                matrix[i - 1][j - 1] + cost // substitution
            );
        }
    }

    return matrix[m][n];
}

Tool #2: `issue_terminal_command`

This tool lets Claude execute terminal commands safely. Think of it as a "shell with guardrails" that prevents potentially destructive operations.

Set Up the Basic Command Executor

// src/tools/issue_terminal_command.ts
import { exec } from 'child_process';
import { promisify } from 'util';

// Safety first - toggle this to control read-only mode
const READ_ONLY_MODE = true;

// List of allowed commands (read-only operations)
// This list is not exhaustive - feel free to add more commands as needed!
const ALLOWED_COMMANDS = [
    'ls', 'dir', 'cat', 'head', 'tail', 'grep',
    'pwd', 'echo', 'date', 'whoami', 'df', 'du', 'ps',
    'wc', 'which', 'whereis', 'type', 'file', 'uname',
    'history', 'env', 'printenv', 'git', 'cd', 'find'
];

export const issueTerminalCommand = async (command: string) => {
    // Extract the base command (first word)
    const baseCommand = command.trim().split(/\s+/)[0];

    // Special handling for problematic commands
    if (baseCommand === 'find') {
        return {
            content: [{
                type: "text",
                text: `Warning: The 'find' command can be slow on large directories. Consider using 'ls' or 'tree' for specific directories instead.`
            }],
            isError: false
        };
    }

    // Security check with helpful error messages
    if (READ_ONLY_MODE && !ALLOWED_COMMANDS.includes(baseCommand)) {
        // Notice how we're not just saying "no" - we're providing context and alternatives
        // This natural language approach makes working with the tool much more intuitive
        return {
            content: [{
                type: "text",
                text: `Error: Command '${baseCommand}' is not allowed in read-only mode. Allowed commands: ${ALLOWED_COMMANDS.join(', ')}`
            }],
            isError: true
        };
    }

    // Execute the command
    try {
        const execPromise = promisify(exec);
        const { stdout, stderr } = await execPromise(command);

        if (stderr) {
            return {
                content: [{
                    type: "text",
                    text: `Error: ${stderr}`
                }],
                isError: true
            };
        }

        return {
            content: [{
                type: "text",
                text: stdout
            }],
            isError: false
        };
    } catch (error: any) {
        return {
            content: [{
                type: "text",
                text: `Error executing command: ${error.message}`
            }],
            isError: true
        };
    }
};

You could optionally extend this with more sophisticated security:

// Add these to your constants
const DANGEROUS_COMMANDS = ['rm', 'rmdir', 'mv', 'cp', 'chmod', 'chown', 'sudo'];

// Add this check before executing
if (DANGEROUS_COMMANDS.some(dangerous => command.includes(dangerous))) {
    return {
        content: [{
            type: "text",
            text: `Security Alert: This command appears to modify or delete files. Operation blocked.`
        }],
        isError: true
    };
}

// Add timeout protection
const COMMAND_TIMEOUT = 30000; // 30 seconds

try {
    const execPromise = promisify(exec);
    const { stdout, stderr } = await execPromise(command, {
        timeout: COMMAND_TIMEOUT
    });
    // ... rest of the code
} catch (error: any) {
    if (error.killed && error.signal === 'SIGTERM') {
        return {
            content: [{
                type: "text",
                text: `Command timed out after ${COMMAND_TIMEOUT/1000} seconds`
            }],
            isError: true
        };
    }
    // ... handle other errors
}

Wiring It All Together

Now let's integrate these tools into your MCP server:

Update Your Server Registration

// src/index.ts (partial)
server.setRequestHandler(ListToolsRequestSchema, async () => {
    return {
        tools: [
            {
                name: "list_repo_locations",
                description: "Given a keyword, returns the relevant repo path based on the keyword provided. If no matching keyword is found, it gives context of the nearest matches.",
                inputSchema: {
                    type: "object",
                    properties: {
                        keyword: { type: "string" },
                    },
                    required: ["keyword"]
                }
            },
            {
                name: "issue_terminal_command",
                description: "Executes a terminal command and returns the result.",
                inputSchema: {
                    type: "object",
                    properties: {
                        command: { type: "string" },
                    },
                    required: ["command"]
                }
            }
        ]
    };
});

Handle the Tool Calls

server.setRequestHandler(CallToolRequestSchema, async (request) => {
    if (request.params.name === "list_repo_locations") {
        const args = request.params.arguments as { keyword: string };
        const { keyword } = args;
        // Pass the environment from server startup
        return { toolResult: await listRepoLocations(environment, keyword) };
    }
    else if (request.params.name === "issue_terminal_command") {
        const args = request.params.arguments as { command: string };
        const { command } = args;
        return { toolResult: await issueTerminalCommand(command) };
    }

    throw new McpError(ErrorCode.MethodNotFound, "Tool not found");
});

Make It Your Own

The beauty of these tools is how customizable they are. And remember - we're not just building tools, we're building conversational interfaces! Every error message, every hint, every suggestion should feel natural and helpful. Here are some ideas:

Add more environments (home, work, client projects)
Include path shortcuts for common operations
Add command aliases for complex terminal operations
Create repo-specific tool mappings

Don't Forget: Safety First :)

Remember to:

Always use READ_ONLY_MODE unless you absolutely trust the environment
Test new commands in a safe environment first
Keep your allowed commands list minimal
Consider adding logging for audit trails

The Result? Claude Code, But Free!

With these two tools, in my opinion, I've basically recreated the core functionality of Claude Code without the subscription. Claude can now:

Intelligently navigate my desired repositories
Execute safe terminal commands
Understand project structure without constant hand-holding
Work across multiple environments seamlessly

Coming Soon: Deep Dive into MCP Development

If you found this helpful, I'm working on a comprehensive MCP Masterclass that will cover:

Advanced tool creation patterns
Natural language processing coding for tools
Building context-aware AI assistants
Performance optimization for MCP servers

Stay tuned for course details!

Thanks for Your Time!

These two tools have fundamentally changed how I work with Claude. The repository finder eliminates navigation friction, while the controlled terminal gives Claude real power to help with development tasks.

Give them a try, customize them to your workflow, and let me know what awesome variations you come up with!

-Chris

Building a Virtual and Declarative IDE: How Complex is it to Create a New Folder or File in an IDE, Really?

Chris Frewin — Mon, 14 Apr 2025 22:20:07 +0000

Just a heads up here: This post is highly technical and meta in nature. It quite literally describes how I re-create the file or folder creation process in an IDE in an event-sourced and declarative manner. However, if you enjoy posts about high-level software architecture and design decisions, this post is for you!

These new features are immediately available in our CodeVideo CLI tool, and will be rolled out to the CodeVideo Studio shortly.

In Short: It’s Complicated

Breaking it Down

In an event-sourced nature, when you, as a human editor, go to create a folder or file in an interactive GUI-based IDE like Visual Studio Code, it typically involves the following steps:

Go to the file explorer on the left of the IDE (or right — if you’re a barbarian) of your editor
Right-click in that area
Click either “Create New Folder…” or “Create New File…” that appears in the corresponding dropdown
Type into the resulting auto-focused input
Press enter to confirm the folder or file name
In the case of a file, the file is opened up blank, but in an initially clean and saved empty state, ready for you to edit

When I originally launched CodeVideo, all I had for all these discrete actions was file-explorer-create-file!

That meant I was missing at least 4–5 actions, none of which were even in the CodeVideo framework at all at that time. Big oof.

This is also where I realized I might as well implement a variety of context menus — one for right-clicking anywhere in the file explorer, one for right-clicking. There is still much more to do here, such as options when right-clicking in the editor, terminal, and so on. Even for the existing implementation, it’s the true “bare bones” strongly opinionated version of what I think any creator would need at minimum to make a coherent software course.

In full technical summary, creating a hello-world.txt file in a CodeVideo JSON used to look like this:

    [
        {
            "name": "file-explorer-create-file",
            "value": "hello-world.txt"
        }
    ]

It now looks like this:

    [
        {
            "name": "mouse-move-file-explorer",
            "value": "1",
        },
        {
            "name": "mouse-right-click",
            "value": "1",
        },
        {
            "name": "mouse-move-file-explorer-context-menu-new-file",
            "value": "1",
        },
        {
            "name": "mouse-left-click",
            "value": "1",
        },
        {
            "name": "file-explorer-type-new-file-input",
            "value": "hello-world.txt",
        },
        {
            "name": "file-explorer-enter-new-file-input",
            "value": "1",
        }
    ]

But we can go even further — above is what the standard “create a file with mouse” flow would look like. We also can make a file using the terminal, which should look like this:

    [
        {
            "name": "mouse-move-terminal",
            "value": "1",
        },
        {
            "name": "mouse-left-click",
            "value": "1",
        },
        {
            "name": "terminal-type",
            "value": "touch hello-world.txt",
        },
        {
            "name": "terminal-enter",
            "value": "1",
        }
    ]

Even then — it’s not the same behavior as creating through the file explorer — this flow will just create the file and you’ll see it in the file explorer. You’d have to mouse-move-file-explorer-file to hello-world.txt and mouse-left-click to actually open the file! Sheesh, all just to open a measly file… but hey, while I probably said it’s a lot of fun, I didn’t say any of this was very easy.

Some Good News

The good news is that this next level of the domain expansion helped me refine things heavily in the virtual-mouse and virtual-file-explorer domains. Until now, I didn’t even have a mouse-move action for any of the given IDE locations: mouse-move-file-explorer, mouse-move-file-explorer-file-name, mouse-move-file-explorer-folder-name, and so on! Heck, I didn’t even have the @fullstackcraft/codevideo-virtual-mouse implementation when starting this context menu insanity!

While this definitely complicates things for the user side, (but can be abstracted away with composite actions — which I’ll touch on in a bit), this heavily improves things on the UI side — as a developer working with these event names, I know exactly what is happening and where. When was previously just file-explorer-create-file - there were many logical, and thus visual jumps that would occur (mouse move to file name, file name appears, etc.), Now, we have very atomic actions that are easy to reproduce visually. In a way, while the user-facing side of things becomes more verbose, actually debugging any issues becomes much simpler - the actions are so atomic you can easily trace which one is the potential troublemaker.

Introducing Composite Actions!

It’s now time to share an idea I’ve finally solidified: composite actions. The general idea behind a composite action is that you, as a user in the CodeVideo Studio would select a “composite” action, which still has the standard name and value shape, like the simple ones above, i.e. file-explorer-create-file, however, you could have multiple options for this, such as composite-file-explorer-create-file-with-mouse, or composite-file-explorer-create-file-with-terminal. In the background, these composite actions would facilitate the actual atomic actions necessary for the proper state and UI changes that the CodeVideo framework guarantees.

To create composite actions, we can use generator functions to generate the steps needed. I’m still debating how to show these composite actions in the CodeVideo Studio; perhaps as a subset list of actions that are read-only, and only shown when you have a composite action chosen. The initial draft generator functions are currently a part of the @fullstackcraftllc/codevideo-types library, written initially there to ensure future type safety. These will likely become their own repository, as I’ve brainstormed a variety of other composite actions where we can use the virtual state layer itself to generate them (Want to close all tabs? easy. Collapse all open folders? also easy. Open multiple terminals? also easy). Anyway, this is another post for another time ;)

…Atomic Actions?

We could go even further as well, for example, going the opposite direction, to expand actions, for example, expanding typing actions to truly single keystrokes, i.e.:

    [
        {
            "name": "editor-type",
            "value": "console.log('hello, world!');"
        }
    ]

becomes in a truly atomic fashion:

    [
      { "name": "editor-type", "value": "c" },
      { "name": "editor-type", "value": "o" },
      { "name": "editor-type", "value": "n" },
      { "name": "editor-type", "value": "s" },
      { "name": "editor-type", "value": "o" },
      { "name": "editor-type", "value": "l" },
      { "name": "editor-type", "value": "e" },
      { "name": "editor-type", "value": "." },
      { "name": "editor-type", "value": "l" },
      { "name": "editor-type", "value": "o" },
      { "name": "editor-type", "value": "g" },
      { "name": "editor-type", "value": "(" },
      { "name": "editor-type", "value": "'" },
      { "name": "editor-type", "value": "h" },
      { "name": "editor-type", "value": "e" },
      { "name": "editor-type", "value": "l" },
      { "name": "editor-type", "value": "l" },
      { "name": "editor-type", "value": "o" },
      { "name": "editor-type", "value": "," },
      { "name": "editor-type", "value": " " },
      { "name": "editor-type", "value": "w" },
      { "name": "editor-type", "value": "o" },
      { "name": "editor-type", "value": "r" },
      { "name": "editor-type", "value": "l" },
      { "name": "editor-type", "value": "d" },
      { "name": "editor-type", "value": "!" },
      { "name": "editor-type", "value": "'" },
      { "name": "editor-type", "value": ")" },
      { "name": "editor-type", "value": ";" }
    ]

though sometimes that feels a bit too overboard. We’ll see — for a series of PNGs, or GIF export for example, these atomic actions are in fact the optimal choice of action discretization. Not so much for portability or readability, however!

So, I think I’ll keep this post at that for now. Let me know if you’d like more or less technicality in following posts!

Until the next one!

-Chris

CodeVideo: My 2+ Year Journey

Chris Frewin — Fri, 14 Mar 2025 11:35:00 +0000

It's Finally Here!

Today, after nearly two years in the making, I'm excited to be launching CodeVideo, an entire ecosystem for building high-quality software educational content.

Calling All Software and Coding Educators and Content Creators!

You know that dreaded feeling — you've got a great course idea, lessons laid out, but you're already discouraged at the massive effort you'll need to write the script, record the video, export the code per lesson, ensure the code is working and runnable at the end of every lesson, converting lessons to blog posts, pdfs, or presentations… the list goes on and on…

…what if — and this is a HUGE if (trust me, I didn't think it was possible either) — you could just write your course once, focusing on the best content possible, and do all of the other annoying administrative things instantly?

Export your course lessons to video? CodeVideo can do that.

Export to a markdown blog post? CodeVideo.

Export to a step-by-step PowerPoint presentation? CodeVideo.

Export PNGs, PDFs, Markdown, a website, or even an interactive React component?! CodeVideo can do all of this and more.

It gets even better: has something in your course changed or been deprecated? No problem. Load up your CodeVideo course, go to those series of steps, edit as needed, and republish everything instantly; no more pain of splicing video, searching to find at what timestamp the outdated info occurs.

CodeVideo In a Nutshell

CodeVideo is a framework for generating high-quality educational software content. The framework enables the creation of editable, replayable, or time-travelable IDE sessions, and can export those sessions to a wide variety of formats, including of course video, gif, PDF, markdown, .pptx (PowerPoint), and more — and many more to come.

Essentially, you create your course or lesson actions once and can export them anywhere. For example, here is the video export version (V3) of the framework in action, where I describe how I package my Go functions for a Netlify serverless environment:

https://www.youtube.com/embed/xZ6EMfPJUho?si=-GlFESxc0hzAdtMK

Or some PNG snapshots of the same course:

Or, maybe you'd like a zip of the final source code of all files from the same lesson:

https://coffee-app.sfo2.cdn.digitaloceanspaces.com/codevideo/zips/go-for-netlify-functions.zip

Each one of these different export types was created with just a single click! (and, in the case of every export type except video — generated almost instantly.)

Hopefully, you're starting to get a feel for how powerful the CodeVideo framework is!

However, it still sounds like a handful, right?! Funnily enough, building exporters for additional formats is often actually an “easier” part of writing software for CodeVideo and is often quite straightforward — provided you are creative with your configuration and options settings for the given output format.

Anyways, for those interested, we're going to take a deep dive into CodeVideo: how it first came to be, its “winters”, and what it is now!

The History of CodeVideo

Early Days

Initially, I started Code Video way, way back in December of 2022, after I was basically told that the startup I was working at would run out of money within 24 hours. Still stuck in startup mode, I reactionarily started looking at things to build myself. The original idea behind CodeVideo was just to take something like Carbon.io (which makes snapshots of an editor) and make a simple tool that generates videos of small code snippets. I actually managed to build that by the end of that December. In fact, that work from December 2022 is still on our homepage codevideo.io where we do some clever work with a ffmpeg ported to WASM. But for me, it was always just the proof of concept and nothing more than an MVP.

Ultimately, however, this solution were very fragile (for the frontend engine, you can still find the code in there of how the highlighter works — literally highlighting character by character and then recording a canvas — there is even still a TODO there!). In short, back in December 2022, I didn't really understand the domain and thought it was simply too time-consuming and challenging of a project as a solo dev. So for a long time — essentially all of 2023 — CodeVideo was looking like it was heading for the infamous Software Graveyard…

January / February 2024

In the quiet early months of 2024, my thoughts again returned to CodeVideo. With some discussions with a colleague, we discussed things like abstract syntax trees and making. By coincidence, I learned of Backdrop Build (I think it was V4) and for 5 weeks there was a bit of social pressure to keep shipping at least something every day. I started building out the classic “startup” essentials — a website, some draft ideas of repos, and much more. It was still clear however, I myself was still learning the domain and was rather unorganized here.

During this time, I managed to build a single Monaco editor driven by Puppeteer on Chromium that is still live today (and may forever stay live due to nostalgic reasons). Getting this API was a breakthrough moment, at least for a part of my vision: send and API endpoint a list of JSON actions, and receive a link to an mp4 of your single editor video, complete with life-like speech and realistic typing!

With this REST API, I built a neat demo video that showcased CodeVideo itself, which I'm pretty sure is what landed me a finalist position and got me in the special funding chat Telegram for Backdrop Build (though I haven't heard a peep since). It's a shame, because I thought the Backdrop Build program was a great idea really well organized!

Once again however, for most of 2024, CodeVideo experienced it's second winter…

Event Sourcing? What's That?

In October 2024, I did an event sourcing course with Ambar. The course was way above my non-existant event-sourcing skills, however. I didn't understand any of the terms at all. A few weeks later while working on CodeVideo, I remember literally asking ChatGPT what such a type of ‘auditable, and replayable framework' was official called, and if it was even possible. It of course responded “Are you thinking of event sourcing?” and I started cracking up. Perhaps what I subconsciously learned in the Ambar course was finally making its way through my head. However, I still wasn't overly sold on the traditional and what I thought was overly-complex event sourcing and decided to keep things as a sort of distilled and simpler event-sourcing for CodeVideo.

One thing I did realize, for sure, is that I would need to formalize the entire framework if I was ever going to have a solution that could be testable, debuggable, and sustainable into the future. I realized, for example, I would need an entire virtual layer before trying to render things in a UI. To do both at the same time (both the logic behind and IDE and UI itself) would certainly end in tears. It was shortly after that I began writing up CodeVideo as a formal framework in a white paper (again, I inspired by Ambar — their whitepaper is also linked on their own homepage).

The First Virtual Layer: Enter `virtual-code-block`

Returning to the concept of a “virtual layer”: virtual-code-block was my first attempt at a programmatic way to replay and audit actions in a single editor. (Even this name has changed; now it is codevideo-virtual-editor, you'll see why in about two paragraphs.)

Many times throughout the “virtual layer” phase in the journey, Donald Knuth's quote kept popping into my head:

If you find that you're spending almost all your time on theory, start turning some attention to practical things; it will improve your theories. If you find that you're spending almost all your time on practice, start turning some attention to theoretical things; it will improve your practice.

With this quote often bouncing around in my head, I would often hop between the virtual layer and the white paper itself. This is when the domain expansion really started to take off. I realized that if I were to do a good job with CodeVideo, a single virtual code block was not even close to enough to what I needed. I would also need virtual layers for the file explorer, the mouse, and even the terminal. Thus came the next series of the virtual layer: codevideo-virtual-file-system, codevideo-virtual-terminal, codevideo-virtual-author, and of course, codevideo-virtual-editor which I renamed from codevideo-virtual-code-block.

But wait, we're still not done! I then combined all four of those virtual layer components to a single entry point class and library: codevideo-virtual-ide!

This is the single class that drives the codevideo-ide-react component, currently what is used on the backend for creating videos.

I could go into more detail about how the V3 API uses the codevideo-ide-react component mounted within a static Gatsby page, how a go microservice is used to manage the puppeteer stream recorder, and how much of a pain in the *&&(@##$@ it was to get audio capture on a headless Chromium instance in Docker, but that's probably all better kept for later posts. For now, just enjoy the Studio!

November 2024: Applied to YCombinator!

I didn't get in. Probably I didn't use enough AI buzzwords :)

Major Theoretical Breakthrough #2: Parallelized Video Generation

It was in the past few months that I realized due to the event-sourced nature of CodeVideo, one could generate an animated video of a course of any level of complexity in mere seconds. How? Well, since we're able to reconstitute the IDE's state at any (and every) given action in a CodeVideo project, you can record short videos (essentially short clips) of just a single action in parallel and then stitch them all together at the end. Granted, this is given you have a damn good UI to render that state (more on this pain and suffering later), and a large amount of resources (direction of 64GB+ RAM). This means at maximum, no matter how many actions your course has, the time to generate a video for said course takes approximately as long as the longest single action plus a small amount of stitching / system overhead — typically anywhere from 10 to 20 seconds.

Final Release Push in Q1 2025

With all the groundwork I laid throughout the year in 2024, in 2025, I finally returned the client, essentially rewriting from the ground up both our homepage and studio. In the meantime, as a bit of a side experiment, I built an auditable / replayable mouse react component, and a variety of export functions that power CodeVideo's markdown, HTML, and PDF export formats. It was in January 2025 that I also achieved the first parallel recorded video on my pure HTML-based test bench, where each atomic step of a course was recorded as a separate video and then stitched together to form the full video — one part of the original vision of CodeVideo, formed more than two years prior, was in sight!

..and Now We're Here

We can't forget the launch itself, can we? Today, On Friday, March 14th, 2025 (shout out to Pi day!), I'm launching CodeVideo on all platforms that I know of: Indy Hackers, Producthunt, Hacker News, Substack, Medium, here on my blog, and many more.

Growth as a Software Engineer

In a way, (I hope), CodeVideo is a symbol of my maturity as a software engineer — if I had tried a few years back to tackle something like CodeVideo, I probably would have tried to do everything at once, got frustrated, and just rage quit entirely.

CodeVideo also illustrates a new step for me: it's the first product I've built fully open source and more importantly, across multiple repositories, where each repository typically at most represents a single class or at most a few classes. This separates concerns nicely and helps isolate any bugs down to their individual components. Of course, I'm biased in saying that I hope 99% of those interested in using CodeVideo would rather just pay for CodeVideo generation tokens than try and figure out how the 30+ repositories work together to help build CodeVideo projects and subsequently produce the desired export formats.

Getting Started

Head over to the CodeVideo Studio and sign up to get 50 free CodeVideo tokens — that's 5 video generations (10 tokens per video generation) of up to 250 steps (approximately up to a 5–10 minute video, depending on what your steps include). Our studio will always include Markdown and JSON exports for any sized course for free, forever.

Sign up today and join the revolution to supercharge your educational software content creation. With all the hype around LLMs and code generation, quality software courses and lessons built by real humans have never been more important.

I'm Super Happy You're Here!

In short — I'm glad you've taken the time to read this. CodeVideo has been an enormous effort to get to this state, so I appreciate you being here. There is, of course, plenty more to do. (I originally was going to write here something like “I think it will be quite some time before I can say CodeVideo is ‘finished'”. But with the complexity of a framework like this, I feel like this product will be more or less one of those lifelong projects :) we'll see.

On my side, this post was a nice walk down memory lane. If you made it this far, what do you think of CodeVideo? Is this product ultimately a crapshoot of extreme complexity that is doomed to fail? Do you even think that event-sourcing the entire IDE is even possible? Do you have some ideas for export formats, features, or anything I've overlooked? (Or just haven't had time to build yet!) Feel free to share in the comments!

Until next time… I have to get back to working on the avalanche of feature requests…

-Chris

Tales from the Software Graveyard: The Birth and Death of Squawk Market

Chris Frewin — Wed, 07 Jun 2023 13:16:49 +0000

The Squawk Market Service & Background

Since early March I've been honing my skills as a day trader, viewing it both as a compliment to my software engineering career and because I want to show my customers at The Wheel Screener that I've got ‘skin in the game' so to speak. While trading, I ran into many services that have audio based news, so you can keep them on in the background while staying focused on price action and the markets. As a shameless hop on the AI hype train as what I expected would bring more interest, I realized I could leverage both GPT4 and a variety of text-to-speech models to build a similar product but with lower latency and lower costs (and better reliability!) I posted about Squawk Market on the launch day - you can read that here.

If you're interested or just want to see what I mean by similar products, I recommended Financial Juice, which offers a 10 second delayed feed completely for free (no affiliation).

Ultimately however, even after advertising on Reddit and Twitter, I saw not a single trial or subscription, and even now writing this in mid June, I've had only two people approach me about the service.

However, not all is lost! In this post, I'm going to go into all the software aspects of Squawk Market, so as to turn this failed product into some software based positives going into the future.

Learnings and Takeaways

Go is Fast!

I already knew this but was reminded again — Go is very performant — scraping websites and converting them to speech as described below was done in sub second times.

Text to Speech Models — Largely Varied in Price and Quality

I ultimately ended up using three different text to speech APIs throughout the creation of squawk market:

I started with ElevenLabs, which in my opinion, has very impressive voice generation — sounding truly human with varied inflection and tones. Their free tier is for up to 10000 characters every month, which is only 6 pages (however, if you need only a fixed amount of voice snippets, this could be great for you) The only problem was even with the $5 per month subscription, I was hitting the character limit of 30000 too quickly.
I then moved on to Google Cloud's text to speech, but I realized Google did not have a free tier (if you start a new account with them you get up to $300 — my free credits are long gone!) and I quickly moved away from them after racking up $10–15 in charges.
I finally settled on Amazon's Polly, their free tier is super generous — a whopping 5 million characters, or 3000+ pages of text — that even with everything that Squawk Market was parsing, I never got outside of the free tier.

Ideally, I would have liked to stick to ElevenLabs, as but with no subscribers (i.e. no revenue), it just didn't make sense financially.

Here's the Go code for interfacing with each of those three services:

ElevenLabs:

func TextToSpeech(text string) []byte {
    log.Println("getting voice data...")

    // headers for elevenlabs
    headers := map[string]string{
    "Content-Type": "application/json",
    "xi-api-key":   os.Getenv("ELEVEN_LABS_API_KEY"),
    "Accept":       "audio/mpeg",
    }

    body := strings.NewReader(`{
    "text": "` + text + `", 
    "voice_settings": {
    "stability": 0.15, 
    "similarity_boost": 1.0
    }
    }`)

    // elli voice id MF3mGyEYCl7XYWbV9V6O
    data, err := http_helper.MakeHTTPRequest(
    "https://api.elevenlabs.io/v1/text-to-speech/MF3mGyEYCl7XYWbV9V6O",
    "POST",
    headers,
    nil,
    body,
    )
    if err != nil {
    log.Println(err)
    }

    // return the byte data generated by elevenlabs' boss bots
    return data
}

Amazon:

// TextToSpeech converts text to speech using Amazon Polly
func TextToSpeech(text string) ([]byte, error) {
    sess := session.Must(session.NewSession(&aws.Config{
    Region: aws.String("us-east-1"),
    }))
    svc := polly.New(sess)

    input := &polly.SynthesizeSpeechInput{
    OutputFormat: aws.String("mp3"),
    Text:         aws.String("<speak><prosody rate=\"fast\">" + utils.EscapeForSSML(text) + "</prosody></speak>"),
    VoiceId:      aws.String("Amy"),
    TextType:     aws.String("ssml"),
    // british language code
    LanguageCode: aws.String("en-GB"),
    }

    output, err := svc.SynthesizeSpeech(input)
    if err != nil {
    fmt.Println(err)
    return nil, err
    }

    mp3data, err := io.ReadAll(output.AudioStream)
    if err != nil {
    fmt.Println(err)
    return nil, err
    }

    return mp3data, nil
}

Google:

func TextToSpeech(text string) []byte {
    // Instantiates a client.
    ctx := context.Background()

    client, err := texttospeech.NewClient(ctx)
    if err != nil {
    log.Fatal(err)
    }
    defer client.Close()

    // Perform the text-to-speech request on the text input with the selected
    // voice parameters and audio file type.
    req := texttospeechpb.SynthesizeSpeechRequest{
    // Set the text input to be synthesized.
    Input: &texttospeechpb.SynthesisInput{
    InputSource: &texttospeechpb.SynthesisInput_Text{Text: text},
    },

    // Build the voice request, select the language code ("en-US") and the SSML
    // voice gender ("neutral").
    Voice: &texttospeechpb.VoiceSelectionParams{
    LanguageCode: "en-gb",
    SsmlGender:   texttospeechpb.SsmlVoiceGender_FEMALE,
    // Name:         "en-US-Studio-O", // nice one but expensive
    Name: "en-GB-Neural2-A",
    },

    // Select the type of audio file you want returned.
    // TODO: could eventually be client side configurable
    AudioConfig: &texttospeechpb.AudioConfig{
    AudioEncoding: texttospeechpb.AudioEncoding_MP3,
    SpeakingRate:  1.3,
    Pitch:         0,
    },
    }

    resp, err := client.SynthesizeSpeech(ctx, &req)
    if err != nil {
    log.Fatal(err)
    }

    // The resp's AudioContent is binary.
    return resp.AudioContent
}

We Can Also Parse Any Video with Speech to Text in Realtime!

While building Squawk Market, I found a way to convert video streams (m3u8) into text. You provide the URL to your YouTube video, and then text of that video comes out! How does it work under the hood?

First, we need to find an m3u8 file that we can actually parse! If you use a youtube video (preferrable a live video), there is a tool called yt-dlp that can do that for us. I wrote up a function called GetStreamUrlFromYoutubeVideoId , that calls yt-dlp to extract the m3u8 URL from the YouTube video ID:

func GetStreamUrlFromYoutubeVideoId(youtubeVideoId string) (string, error) {
    // use yt-dlp to get the stream url
    cmd := exec.Command("yt-dlp", "-f", "91", "-g", "https://www.youtube.com/watch?v="+youtubeVideoId)
    output, err := cmd.Output()
    if err != nil {
    return "", err
    }

    return string(output), nil
}

Then, I pass that on to a function M3U8VideoStreamToFile :

func M3U8VideoStreamToFile(m3u8VideoStreamURI string, fileName string, fileType string, secondsString string) error {
    // write to flac file
    cmd := exec.Command("ffmpeg", "-y", "-i", m3u8VideoStreamURI, "-t", secondsString, "-f", fileType, "-ar", fmt.Sprint(sampleRateHertz), fileName)
    err := cmd.Start()
    if err != nil {
    fmt.Println(err)
    return err
    }
    err = cmd.Wait()
    if err != nil {
    fmt.Println(err)
    return err
    }
    return nil
}

Finally, I use Open AI's audio to text to convert the audio in the file to text:

func SpeechToText(filePath string) (*string, error) {
    c := openai.NewClient(os.Getenv("OPEN_AI_SECRET_KEY"))
    ctx := context.Background()

    req := openai.AudioRequest{
    Model:    openai.Whisper1,
    FilePath: filePath,
    }
    resp, err := c.CreateTranscription(ctx, req)
    if err != nil {
    fmt.Printf("Transcription error: %v\n", err)
    return nil, err
    }
    return &resp.Text, nil
}

Altogether, we can define a function called YoutubeVideoIdToText that looks like this:

func YoutubeVideoIdToText(youtubeVideoId string) error {
    // get m3u8 video stream url
    videoStreamUrl, err := streamfinder.GetStreamUrlFromYoutubeVideoId(youtubeVideoId)
    if err != nil {
    return err
    }

    // filename for now is more or less an intermediate, may want to change later
    fileName := "audio.mp3"

    // convert video stream to audio file
    err = videostreaming.M3U8VideoStreamToFile(videoStreamUrl, fileName, "mp3", "10")
    if err != nil {
    return err
    }

    // convert audio file to text
    // open ai cost is $0.006 per minute of audio - i.e. $0.36 per hour of audio
    text, err := open_ai.SpeechToText(fileName)
    if err != nil {
    return err
    }

    // print text to log for now
    log.Println(*text)
    return nil
}

From there, it gets as interesting as you want — you could just take the text as is and display it somewhere, or continue on to use something like GPT4 to summarize the text, or provide sentiment analysis to boil down the transcript to a simple ‘BULLISH', ‘BEARISH', or ‘NEUTRAL' signal. Or chain it to a text to speech model like the ones mentioned above and just read it straight out.

Squawk Market is Now Open Source!

For those out there who would like to learn or see how I put a full stack app together, both the frontend and backend repositories of Squawk Market are open source! Check them out here:

Squawk Market Frontend

Squawk Market Backend

Please note that this post doesn't go into all the details of actually deploying the front and backend — you can checkout the READMEs of each on how to do that — if you are interested in running an instance yourself or having trouble reach out to me! I'm more than happy to consider a community led effort to get these services back up and running for everyone to share.

Thanks!

I hope you enjoyed this post and learned a thing or two about writing full stack applications with Go!

-Chris 🍻

I'm on a mission to teach 1,000,000 up-and-coming developers real-world software! Check out my blog for more:

https://chrisfrew.in/

and my Udemy profile & courses:

https://www.udemy.com/user/chris-frewin/

Full Stack Forays with Go and gRPC

Chris Frewin — Mon, 05 Jun 2023 08:55:05 +0000

What is gRPC?

If you’re like me, you may have never even heard of gRPC, let alone know what some of its advantages and disadvantages are. gRPC is a recursive acronym that stands for gRPC Remote Procedure Call. It is a combination of the HTTP 2.0 protocol and protocol buffers, also known as protobufs.

What is gRPC Good For?

A main advantage of gRPC is that it is a universal communication protocol, due to proto buffers being encoded down to binary (and you can’t get much simpler than that!). Thus it is perfect for distributed computing and exchanging messages between systems that are written in different languages or frameworks but need to nonetheless communicate. While something like an HTTP protocol could be used also for this crosss-ystem communication, this comes with the overhead that each system must also have a server exposing some sort of REST or REST-like service with valid endpoints and so on. Protobufs, while being at some levels even more strict than REST over HTTP, can be in code as easy as a one-liner.

gRPC is also much faster than traditional HTTP — again thanks to the binary format of the messages being transferred. This makes it excellent for low-latency communication between services.

Disadvantages of gRPC

Like everything in tech, nothing is a silver bullet, and gRPC is no exception, even with its impressive list of benefits. As of June 13th, 2022, when this post was published, there are key disadvantages with gRPC (all client-side issues, specifically web):

TypeScript support remains an experimental feature of gRPC.
A proxy is required to communicate from web clients to a server running gRPC, and there are only two choices for this proxy: a. The Improbable gRPC-Web client or b. The Google gRPC-Web client
Furthermore, currently, neither implementation conforms completely to the full gRPC specification.
WebSocket support as the transport layer is only available in the Improbable gRPC-Web client, and remains as experimental, not recommended for production use.

(By the way, if you’re wondering which to choose, only the Improbable gRPC-Web client supports bi-directional streaming at the moment, which is closer to the full spec of gRPC.)

Get Started

Backend Implementation

First, I started with gRPC’s recommended starter repository for learning gRPC, their **helloworld **example, which is a part of the official gRPC repository.

Using that code, I added a new utility method in Go called Reverse:

package utils

// Reverse returns its argument string reversed rune-wise left to right.
func Reverse(s string) string {
  r := []rune(s)
  for i, j := 0, len(r)-1; i < len(r)/2; i, j = i+1, j-1 {
  r[i], r[j] = r[j], r[i]
  }
  return string(r)
}

And modified the original implementation of the SayHello function to use this Reverse function:

func (s *server) SayHello(ctx context.Context, in *pb.HelloRequest) (*pb.HelloReply, error) {
  log.Printf("Received: %v", in.GetName())
  log.Printf("Returning: %v", utils.Reverse(in.GetName()))
  return &pb.HelloReply{Message: utils.Reverse(in.GetName())}, nil
}

Frontend Implementation

I scaffolded a React app with create-react-app, removed all extra fluff (styles, images, and such), and added some description text with a .

Now the cool part: I decided to use a custom React Hook to control the state of the message, the response, and interact with the gRPC call. The hook listens to the value of the message the user provides, and whenever it changes, the gRPC method reverser service is called. This comes out quite succinctly as a hook I called useGreeterService:

import { useEffect, useState } from "react";
import { sayHello } from "../services/greeterService";

export const useGreeterService = (): [
  string,
  React.Dispatch<React.SetStateAction<string>>,
  string
] => {
  const [message, setMessage] = useState("");
  const [responseMessage, setResponseMessage] = useState("");
  useEffect(() => {
    sayHello(message, setResponseMessage);
  }, [message]);

  return [message, setMessage, responseMessage];
};

Note the subtle power in that sayHello call - a one-liner! No fetch, no configuring headers, cookies, HTTP methods, or building up some custom API service - gRPC abstracts all of that away for us.

Usage of useGreeterService in App.tsx looks like this:

import { useState } from "react"
import { useGreeterService } from "./hooks/useGreeterService";

const App = () => {
  const [message, setMessage, responseMessage] = useGreeterService();

  return (
    <>
    <h1>Simple message reverser over gRPC</h1>
    <p>(Don't ask me why you would actually want to reverse a string server-side, this is just for fun!</p>
    <label title="Type a message in realtime over the gRPC wire">Type a message in realtime over the gRPC wire:</label>
    <br/>
    <textarea value={message} onChange={(event) => setMessage(event.target.value)}/>
    <p>And see the response message here: {responseMessage}</p>
    </>
  );
}

export default App;

gRPC Web Proxy

Once your backend and frontend are up and running, there is still one item that needs to be added: the gRPC Proxy. Luckily we have Improbable’s proxy ready to be used. First, install it with:

GOPATH=~/go ; export GOPATH
git clone https://github.com/improbable-eng/grpc-web.git $GOPATH/src/github.com/improbable-eng/grpc-web
cd $GOPATH/src/github.com/improbable-eng/grpc-web
dep ensure # after installing dep
go install ./go/grpcwebproxy # installs into $GOPATH/bin/grpcwebproxy

Then run it with:

GOPATH=~/go ; export GOPATH
$GOPATH/bin/grpcwebproxy \
    --backend_addr=localhost:50051 \
    --run_tls_server=false \
    --use_websockets \
    --allow_all_origins

Note: in my original experiment, I wanted to use WebSockets as the transport layer for optimum performance. However, upon further review into gRPC (and as mentioned above), using WebSockets as a transport layer is still technically an experimental feature, and so I would recommend removing the *--use_websockets** flag from the command above when running the web proxy in production.

(Though it’s unclear to me exactly what is experimental about using WebSockets as the transport layer. For this simple example I experienced no issues!)

Example code

The code from this little experiment can be found on GitHub. The README can give you even more detailed information on how to get it all up and running.

I have to say, from my initial fooling around, I was amazed by the performance I was seeing. From my perspective, I couldn’t have told you I saw any difference from just calling string.reverse() in the client on each keystroke - and yet those strings were produced by two gRPC calls making a round trip - first a call to the server with the string, and the server responding with the reversed string! (Granted, this was all running over localhost, so I'm not sure how this would look on a real server, but I can imagine it is nearly as fast). But like I said, from an initial feeling, it does feel fast!

Thanks!

If you were looking to dive into the world of gRPC, I hope this post convinced you to try and showed how easy it is to get up and running with a full stack gRPC call!

Cheers! 🍻

-Chris

I’m on a mission to teach 1,000,000 up-and-coming developers real-world software! Check out my blog for more:

https://chrisfrew.in/

and my Udemy profile & courses:

https://www.udemy.com/user/chris-frewin/

Create a Dockerized Go Application - With a Cron Job, Slack Messaging, and Test the Whole Thing

Chris Frewin — Thu, 25 May 2023 08:51:28 +0000

For the past few years, I've been using Go more and more to write backend software. I've been wanting to write a more in-depth article on Go for quite a while now and thought of a fun little application we could make together to illustrate some of the ways I write real-world production-ready Go apps.

This article corresponds with the e-book "Go for Real World Applications" on Gumroad

As well as the full course on Udemy

And the course preview on YouTube (first 9 of 19 lessons)

The example repository, one branch per lesson, is on GitHub

This article is only an overview of all the details that are in the full course and book, which does not include:

-Removing hardcoded variables and moving them to an environment file

-Adding more fancy formatting and style to the Slack messages

-Creating an automated CI/CD pipeline with Circle CI

Plugs are done, let's get into the full tutorial!

Why Go?

If you've read my blog or some of my other articles, you may have realized I am not any sort of language or framework evangelist — in fact, I try to stay language or framework agnostic: picking the right tool for the right job. However, there are many enticing reasons to consider adding Go to your stack:

1. Go is extremely performant

Reading files, mathematical operations, you name it- I'm often extremely impressed with the speeds at which Go operates and so far have never had to do deep dives on my code to improve performance — and even some of the stuff that I've built for The Wheel Screener and other finance apps, I was doing 10,000+ options contract calculations every second with nearly no slowdown or latency on a fairly average 4 core machine.

2. Go is extremely compact

Go projects compile down to a single binary. That means no monster external files, a no-headache package system all held within go.mod , and that's it. For Dockerizing these apps, as we'll see later in this article, it's the perfect scenario.

3. Go is extremely testable

Go has a testing framework built in. This alone should signal to you what kind of language Go is — one to get stuff done and get stuff done effectively and robustly.

4. Go is extremely API friendly

Go also has JSON support for types, serializing, and deserializing (known as marshaling and unmarshalling in the Go world) right out of the box. Go is the only language I know that can do this, and if you've done API stuff in something like JavaScript or C# when there is no client library to use, you know the pain it can be to properly serialize and deserialize JSON payloads. (Perhaps Rust or some other newer languages have built-in JSON support as well — leave a note in the comments if this is true!)

What We're Going to Build

So, I have hay fever. If you're one of the millions of people around the world with this annoying allergy, you may also know my pain :). In the northern hemisphere, it's a battle for me from about mid-May to mid-June every year. Luckily, where I live, in Austria, the Medizinische Universität Wien (Medical University of Vienna) has a neat website that predicts the "allergy risk" for the day, i.e. how much of the pollen is going to be in the air that day. (Things like rain and prevailing winds can actually reduce or increase the pollen in the air for a given day). I've sniffed around their website and found a few API calls that deliver this information. The first URL delivers an hourly risk level for the given day:

https://www.pollenwarndienst.at/index.php?eID=appinterface&action=getHourlyLoadData&type=zip&value=6800&country=AT&lang_id=0&pure_json=1&day=0

and the response looks like this:

{
    "success": 1,
    "result": {
        "total": 8,
        "dayrisk_personalized": false,
        "hourly": [
            5,
            4,
            4,
            3,
            5,
            4,
            3,
            2,
            3,
            6,
            8,
            8,
            8,
            8,
            8,
            8,
            8,
            8,
            8,
            8,
            8,
            8,
            8,
            8
        ]
    }
}

and the second gives the historical average and the actual pollen amounts that day:

https://www.pollenwarndienst.at/index.php?eID=appinterface&action=getCurrentChartData&poll_id=5&region_id=&zip=6800&season=2&lang_id=1&pure_json=1

this one's response shape looks like this:

{
    "success": 1,
    "results": [
    {
        "date": "2023-04-05",
        "current": 0.4,
        "average": 0.6,
        "season": "false",
        "datetime": 1680652800000
    },
    {
        "date": "2023-04-06",
        "current": 0.5,
        "average": 0.7,
        "season": "false",
        "datetime": 1680739200000
    },
    {
        "date": "2023-04-07",
        "current": 0.6,
        "average": 0.7,
        "season": "false",
        "datetime": 1680825600000
    },
    {
        "date": "2023-04-08",
        "current": 0.6,
        "average": 0.7,
        "season": "false",
        "datetime": 1680912000000
    },
    {
        "date": "2023-04-09",
        "current": 0.5,
        "average": 0.7,
        "season": "false",
        "datetime": 1680998400000
    },
    {
        "date": "2023-04-10",
        "current": 0.9,
        "average": 0.8,
        "season": "false",
        "datetime": 1681084800000
    }
    ]
}

However, it seems like the "current" value is typically delayed 4-5 days, so for real-time updates we'll have to rely on the hourly endpoint.

Requirements

We're going to build an app that does the following:

At a specified time, call the previously shown allergy API endpoints
Parse the response from each call and format it into a nice human-readable message
Send that message via Slack

All clear? Let's build this thing!

Writing the App

Install Go

First, you should ensure you have Go installed on your system. You may have it already, you can check by issuing:

go version

and you should get some sort of output similar to this:

go version go1.20.2 darwin/amd64

If you don't have Go installed yet, you can install it for Mac, Linux, or Windows by following the documenation on the official Go site.

Now let's start building our app!

Scaffold The App

I almost always forget how to do this in Go, and still think it is weird so I'll add it to this article as a reference. We'll create our Go app in a new folder called allergycron/. Then, we just need to initialize our module and we'll also create a new main.go file:

mkdir allergycron
cd allergycron
go mod init allergycron
touch main.go

I'd probably then open up the whole project in Visual Studio code:

open .

and open up main.go to get ready to write code.

Cron Job

Let's first add the cron job to our main.go file. For cron jobs in Go, I use the popular github.com/robfig/cron/v3 library. Adding a cron for a specific timezone, (I'd like for Vienna time, AKA Central European Time or CET, at 08:00:00 AM), can be done like so:

package main

import (
    "time"

    "github.com/robfig/cron"
)

func main() {
    // be sure to modify to run your desired timezone!
    loc, err := time.LoadLocation("Europe/Vienna")
    if err != nil {
    panic(err)
    }

    cronJob := cron.NewWithLocation(loc)

    cronJob.AddFunc("0 0 8 * * *", func() {

    })

    cronJob.Start()

    // run forever
    select {}
}

The time zones are looked up from the IANA Time Zone database. You can check out the names of all of them on Noda Time. With the cron job configured in this fashion, it will run in your desired timezone, regardless of where it is deployed in the world (like to the cloud as we'll see later). The cron string itself is comprised of six separate elements: seconds, minutes, hours, day of month, month, and day of the week, as specified in the cron godoc.

HTTP Utility

I'll be using my generic HTTP function to call the allergy API as well as sending the Slack message. This generic function is highly flexible, reusable, and is a staple in almost all of my Go codebases. We'll make a new folder utils/ and add a file make_http_request.go:

package utils

import (
    "encoding/json"
    "fmt"
    "io"
    "log"
    "net/http"
    "net/url"
    "strings"
)

// in the case of GET, the parameter queryParameters is transferred to the URL as query parameters
// in the case of POST, the parameter body, an io.Reader, is used
func MakeHTTPRequest[T any](fullUrl string, httpMethod string, headers map[string]string, queryParameters url.Values, body io.Reader, responseType T) (T, error) {
    client := http.Client{}
    u, err := url.Parse(fullUrl)
    if err != nil {
    return responseType, err
    }

    // if it's a GET, we need to append the query parameters.
    if httpMethod == "GET" {
    q := u.Query()

    for k, v := range queryParameters {
    // this depends on the type of api, you may need to do it for each of v
    q.Set(k, strings.Join(v, ","))
    }
    // set the query to the encoded parameters
    u.RawQuery = q.Encode()
    }

    // regardless of GET or POST, we can safely add the body
    req, err := http.NewRequest(httpMethod, u.String(), body)
    if err != nil {
    return responseType, err
    }

    // for each header passed, add the header value to the request
    for k, v := range headers {
    req.Header.Set(k, v)
    }

    // optional: log the request for easier stack tracing
    log.Printf("%s %s\n", httpMethod, req.URL.String())

    // finally, do the request
    res, err := client.Do(req)
    if err != nil {
    return responseType, err
    }

    if res == nil {
    return responseType, fmt.Errorf("error: calling %s returned empty response", u.String())
    }

    responseData, err := io.ReadAll(res.Body)
    if err != nil {
    return responseType, err
    }

    defer res.Body.Close()

    if res.StatusCode != http.StatusOK {
    return responseType, fmt.Errorf("error calling %s:\nstatus: %s\nresponseData: %s", u.String(), res.Status, responseData)
    }

    var responseObject T
    err = json.Unmarshal(responseData, &responseObject)

    if err != nil {
    log.Printf("error unmarshaling response: %+v", err)
    return responseType, err
    }

    return responseObject, nil
}

I won't go into the steps of this function here- if you are interested you can read about that in detail in my other article published under Better Programming

API Response Parsing

Parsing the response payload is also little trouble, as we mentioned above, we just need to utilize Go's built-in JSON features when defining our response type. We'll first make a new folder called allergy_api/ and create a new file called call_allergy_api.go. Let's first define the response types needed:

package allergy_api

type AllergyAPIResult struct {
    Total int `json:"total"`
    Hourly []int `json:"hourly"`
}

type AllergyAPIResponse struct {
    Success int `json:"success"`
    Result AllergyAPIResult `json:"result"`
}

Interestingly, on the University's website only shows the 'risk' level from 0 to 4, but the API numbers range anywhere from 0 to 8, so the UI on the website is doing some sort of normalization of the data. For now we'll assume that the 'normalization' is simply dividing by 2 and rounding to the nearest integer:

func GetHourlyLoadData() (*string, error) {
    queryParameters := url.Values{}
    queryParameters.Add("eID", "appinterface")
    queryParameters.Add("action", "getHourlyLoadData")
    queryParameters.Add("type", "zip")
    queryParameters.Add("zip", "6800")
    queryParameters.Add("country", "AT")
    queryParameters.Add("pure_json", "1")

    response, err := utils.MakeHTTPRequest("https://www.pollenwarndienst.at/index.php", "GET", nil, queryParameters, nil, HourlyLoadResponse{})
    if err != nil {
    return nil, err
    }

    // loop over hours and calculate the average
    averageLoad := 0
    for _, hour := range response.Result.Hourly {
    averageLoad += hour
    }
    averageLoad = averageLoad / len(response.Result.Hourly)

    scaledAverageLoad := averageLoad / 2

    formattedMessage := fmt.Sprintf("The average pollen load for today is %d", scaledAverageLoad)

    return &formattedMessage, nil

}

And for the current chart data endpoint:

func GetCurrentChartData() (*string, error) {
    queryParameters := url.Values{}
    queryParameters.Add("eID", "appinterface")
    queryParameters.Add("action", "getCurrentChartData")
    queryParameters.Add("poll_id", "5")
    queryParameters.Add("zip", "6800")
    queryParameters.Add("season", "2")
    queryParameters.Add("pure_json", "1")

    // use the utils.MakeHTTPRequest function to call the allergy api and return the result
    response, err := utils.MakeHTTPRequest("https://www.pollenwarndienst.at/index.php", "GET", nil, queryParameters, nil, CurrentChartDataResponse{})
    if err != nil {
    panic(err)
    }

    // get the average historical by matching the YYYY-MM-DD value
    currentYYYYMMDD := time.Now().Format("2006-01-02")
    averageHistorical := 0.0
    for _, result := range response.Results {
    if result.Date == currentYYYYMMDD {
    averageHistorical = result.Average
    }
    }

    scaledAverageHistorical := int(math.Round(averageHistorical / 2.0))

    formattedMessage := fmt.Sprintf("Historically, the average pollen load for today is %d", scaledAverageHistorical)

    return &formattedMessage, nil
}

Slack Messaging

Now let's get to the final step, messaging the predicted "allergy risk" for the day. We'll create a new file under the utils/ folder called send_slack_message.go:

cd utils
touch send_slack_message.go

Add the following to send_slack_message.go :

package utils

import (
    "bytes"
    "encoding/json"
    "os"
)

func SendSlackMessage(message string) error {
    body, err := json.Marshal(map[string]string{"text": message})
    if err != nil {
    return err
    }

    MakeHTTPRequest(os.Getenv("SLACK_WEBHOOK_URL"), "POST", nil, nil, bytes.NewBuffer(body), "")

    return nil
}

This code requires that we have SLACK_WEBHOOK_URL defined in our environment. You can create and set up your Slack app at api.slack.com

Hooking it All Together

Let's get back to main.go and connect all the functions we've written. Within the cron function, add the following:

dailyAverageMessage, err := allergy_api.GetHourlyLoadData()
if err != nil {
    panic(err)
}

historicalAverageMessage, err := allergy_api.GetCurrentChartData()
if err != nil {
    panic(err)
}

slackMessage := *dailyAverageMessage + "\n" + *historicalAverageMessage
err = utils.SendSlackMessage(slackMessage)
if err != nil {
    panic(err)
}
// log the message to the console as well
println("Successfully sent Slack message: " + slackMessage)

Note here we also log the constructed Slack message to the console for debugging.

Run It!

Let's try and run what we've built so far! To test 'immediately', you can modify the cron message to include the next minute and hour, i.e. for 10:35:00 AM:

cronJob.AddFunc("0 35 10 * * *", func() {
    // ...
})

and then run the main script:

go run main.go

Almost immediately after the clock hits 10:35:00 in the timezone we specified, we should see our Slack message in the console, and receive our Slack message on Slack!

2023/05/12 10:47:04 Successfully sent Slack message: The average pollen load for today is 3
Historically, the average pollen load for today is 4

Tests

Let's write a test for each of our functions. I like to put all my tests in a separate folder, let's call it — you guessed it — tests/ :

mkdir tests

Within this folder, we'll make a test for each of the functions we've made:

touch allergy_api_test.go
touch send_slack_message_test.go

Note that the file name must end with _test.go to be considered a valid test file by Go.

We won't make a test function explicitly for MakeHTTPRequest, because that function is called by the others that we are testing.

As mentioned, Go has testing built-in, and it's only a matter of importing the testing package and passing it to our test function:

package tests

import "testing"

func TestAllergyApi(t *testing.T) {

}

and:

package tests

import "testing"

func TestSendSlackMessage(t *testing.T) {

}

Note that the name of the function must start with the capitalized word 'Test' to be considered a valid test function by Go.

Now, within our test files, we call the function and see if there is an error or not:

func TestAllergyApi(t *testing.T) {
    message, err := allergy_api.GetHourlyLoadData()
    if err != nil {
    t.Errorf("Error getting hourly load data: %s", err)
    }
    if message == nil {
    t.Errorf("Error getting hourly load data: message is nil")
    }
    if *message == "" {
    t.Errorf("Error getting hourly load data: message is empty")
    }

    message, err = allergy_api.GetCurrentChartData()
    if err != nil {
    t.Errorf("Error getting current chart data: %s", err)
    }
    if message == nil {
    t.Errorf("Error getting current chart data: message is nil")
    }
    if *message == "" {
    t.Errorf("Error getting current chart data: message is empty")
    }
}

and for sending the Slack messages:

func TestSendSlackMessage(t *testing.T) {
    err := utils.SendSlackMessage("Test message!")
    if err != nil {
    t.Errorf("Error sending Slack message: %s", err)
    }
}

Our tests are done. To run our tests, we can use Go's built-in test command:

go test -p 1 -v ./tests

The flags here are as follows:

-p 1: Sets the number of parallel test workers to 1. This means that Go will run only one test at a time. If you have multiple tests, they will run one after the other, in serial.
-v: Enables verbose output. When you run your tests with this flag, Go will print detailed information about each test, including its name, its status (pass or fail), and any error messages.

Hopefully, if we're any good at our jobs, you should see output like the following:

=== RUN   TestAllergyApi
--- PASS: TestAllergyApi (1.17s)
=== RUN   TestSendSlackMessage
--- PASS: TestSendSlackMessage (0.00s)
PASS
ok      allergycron/tests        1.498s

Dockerize the Application

Putting our whole app in a Docker container is as easy as defining aDockerfile and docker-compose.yml :

FROM golang:1.20-alpine

WORKDIR /app
COPY . .

RUN go build -o /allergycron

CMD [ "/allergycron" ]

and:

    version: "3.9"
    services:
      allergycron:
        build: .
        restart: unless-stopped

With those files created, you can build and run the docker container with either:

docker compose build --no-cache && docker compose up -d

Or:

docker-compose build --no-cache && docker-compose up -d

Depending on if you have docker or docker-compose installed. After a bit of build output, you should finally see something like this:

[+] Running 2/2
    ⠿ Network "allergycron_default"  Created                                                                     4.8s
    ⠿ Container allergycron_allergycron_1    Started

Check that your application is running with:

docker ps -a

and you should see something like this:

CONTAINER ID   IMAGE                           COMMAND                  CREATED          STATUS                      PORTS                    NAMES
b35e7c28d564   alergycron_allergycron          "/allergycron"           21 seconds ago   Up 18 seconds                                        alergycron_allergycron_1

Done! Fin! Finito!

In just 20 minutes, you've got a Go application that is Dockerized, with a cron job sending Slack messages that calls an API, parses the response, and sends you a summarized Slack message! Oh, the wonders of Go! Hopefully, as I've pointed out throughout the article, this application would be easy to refactor to ping a different API URL and send a different type of Slack message.

Thanks!

I hope this article enticed you to try out the powers of Go for yourself — and showed you with what ease we can create Dockerized, tested, and robust backend applications!

-Chris 🍻

Hungry for more in-depth software tutorials, walk-throughs, courses, and books compiled from real-world experience from my time at large and small companies to startups? I'm on a mission to teach 1,000,000 up-and-coming developers real-world software! Check out my blog for more!

My Udemy profile & courses:

https://www.udemy.com/user/chris-frewin/

My Skillshare profile & courses:

https://www.skillshare.com/en/user/christopherfrewin

Craft a Stylish and Practical Developer Book with this LaTeX Template

Chris Frewin — Fri, 05 May 2023 06:37:27 +0000

The Code

The book template repository is on GitHub. There are some prerequisite steps to compile your first PDF book, so be sure you read the README in the repository or read on in this post!

Motivation

I've been drafting various software engineer and developer-targeted books and guides for the past few years, focusing on hands-on code snippets and applications I've used in real-world applications than theoretical-based books that so many books have. Often, developer or engineering-oriented books show you all the details of how to use a software language or library but do not actually show real-world use cases of the tools or libraries in action. My books focus on real-world recipes and examples that I've deployed to real apps — from startups to industry — in my 10+ year software career. I've returned to these books in the past few weeks and brushed up on a LaTeX template that I'll probably start with for all my books going forward. I'm also planning from here on out to create a PDF walkthrough or tutorial book for each video-based course I release. This way learners of different styles can purchase the book instead of the video if they so choose. Now that my template is fully functional, I figured I'd share it with everybody else!

Prerequisites

You'll need two libraries installed on your system before this book template will work:

-latex

-ghostscript

These tools are available for all operating systems — refer to their websites for installation information.

Template Layout

Title Page

The title page has its own folder and .tex file due to its special formatting. You'll find in the README how to set your own color and image. Then, simply. If you like the O'Reilly format, you can use dev.to's O'Reilly parody generator for image and color inspiration — they use the clip art images from the USF ClipArt ETC project.

Book

In another folder is the actual book content. Here's where you'll do your actual writing! In the repository, I've given examples of how to add titles, subtitles, bullet points, figures (i.e. images), and most importantly, code snippets. The code snippets can be written directly in the .tex file, or loaded from an external file. There are also some other goodies included in the .tex file, like how to include inline emojis, appendices, and an “about the author” section.

LaTeX 101

LaTeX itself deserves its own section. If you've never worked in higher areas of academia, you've probably never dealt with it, or maybe even heard of it before.

What Is LaTeX?

LaTeX defines itself as a “document preparation system”. Think of it like a very very fancy Microsoft Word. If you've ever seen an academic paper that had very complex glyphs of equations and matrices, that document was very likely prepared in LaTeX. Most academic journals share their preamble (you can think of a “preamble” as simply the header of the .tex file, with all its packages and rules defined) so authors can write their articles in a format that is consistent with the journal's style.

The LaTeX Language

LaTeX is... finicky to say the least. I've never really understood a lot of its rules from time to time, and it's rather easy to break your entire build. More often than not I find the rules and settings very brittle, and prone to breaking by mixing certain libraries with others. Think of CSS but worse. (Unfortunately yes, I think it's that bad, but this is just my personal opinion of LaTeX — you may love it! It could also be the fact that I haven't used LaTeX heavily for many years now.)

Here is a bit of an introduction:

\documentclass — defines what type of document you are creating, for example, an article, a book, a letter, and so on.

Creating custom commands can be done using the \newcommand directive, for example, if you forget that for LaTeX a hyperlink is always \href{text}{link}:

\newcommand{\link}[2]{\textbf{\textcolor{blue}{\href{#2}{#1}}}}

This command would create a blue text with a hyperlink. It can be used in the rest of the .tex file like so:

\link{The Google homepage.}{https://google.com}

Like any language or framework, there are a variety of packages you can use via the \usepackage command, some you'll see often are:

\usepackage{tikz}
\usepackage{color}

The tikz library is a drawing library, used to create lines, shapes, and so on on the document. The color library simply allows us to use colors (and define new ones via a hex or RGB code) around the document.

I'll leave it at that for now. Learning LaTeX is a whole endeavor in and of itself, and would need a whole series of other posts to get through.

Other Helpful (But Not Required) Tools

When I write in LaTex, I use VS Code with the LaTeX Workshop Extension. This extension creates a new pdf every time you save your .tex file. I then only have to run the custom bash script to combine the title page with the rest of the book to have a shippable book each time.

Start Writing!

Sounds good? I'm certain there are thousands of developers out there with a unique skill set and they can share their knowledge with the developer and engineering world! I hope my template repository can help jump-start your writing process.

The Template in Action

As I said, I've already gotten started on a book of my own, The Full Stack SaaS Product Cookbook — From Soup 🍜 to Nuts 🥜 — Create a Profitable SaaS Product as a Solo Developer in Days! Check it out here:
Full Stack SaaS Product Cookbook

This is actually my first true video-and-book course, and though it's still a work in progress, it's a monster — over 300 pages in the book and over 50 video lessons already. ☠️

This was probably too ambitious for my very first book, so I'm stepping back and will be publishing a much shorter and more focused book / course combo by this summer — stay tuned.

Thanks!

Enjoy the template, and enjoy writing!

-Chris

Kurynt: The last tool you will ever need to manage third-party code

Chris Frewin — Fri, 29 Jul 2022 16:49:18 +0000

Today's the day I release Kurynt!

I'm excited, but I'm also feeling sketched out; Kurynt is way more complex than my first successful SaaS product, The Wheel Screener. Additionally, I've built Kurynt only in my free time in between massive engineering work at InClub. I was able to build The Wheel Screener more or less full time for about 4 weeks when I was in between jobs in 2021. Kurynt has ultimate also been in about 4 weeks of total time, but in a much more on and off fashion, with the initial formulation in April 2022 (this year as of writing this post).

What is Kurynt?

Kurynt, pronounced "current", aims to fix a problem that software engineers know all too well: You've inherited a legacy codebase, and shrink at even thinking about trying to upgrade any packages. Or perhaps you've got that one package in an existing project you know has long needed to be upgraded, but you haven't even begun to pour into the docs and breaking changes of what an upgrade might entail. As a final example, there are the famous stories that in some projects there is always "that one" ancient package that "we don't touch because it immediately breaks everything for an unknown reason". Such behavior is just asking for security exploits and poor users experiences. We can do better.

I've long been unimpressed by similar tools like Dependabot and Snyk which only give, in my opinion, relatively 'dumb' upgrade suggestions. Those tools simply recommend you upgrade to the latest version, and even more heinously, simply spam your git repository with a pull request that includes nothing more than the package version updated. These tools don't analyze your codebase and warn you if parts of the package it has upgraded may break completely and essentially bring down your app. I know this all to well from development time at InClub, where a seemingly benign upgrade to a package in our React Native application causes the entire app to crash or even unable to boot up. Of course, we've caught these in our development and staging systems before any release, but this is essentially a trial-and-error pattern that is often frustrating. I wanted to build a tool that can manage everything about package upgrades, deprecations, and end-of-lfe entirely for you.

Kurynt gives you more intelligent and more detailed upgrade suggestions. In fact, you don't even need to upgrade to the latest version of any given package, you can go step by step, or even from any "source" version to a different "destination" version. Kurynt is also not very opinionated - you can think of it more as a collector and distiller of a variety of platforms. It's optimized to collect and crunch data to help you:

Upgrade your third-party software as quickly and safely as possible
Identify regressions or bugs in recently updated packages
Help you schedule and plan for future upgrades so your software is never using software that has been deprecated or worse - already reached end of life.

Lean Release

While reading the Lean Startup by Eric Ries (also partially as homework for InClub), I'm releasing Kurynt in a lean fashion (a very lean fashion). There are hints to many of the features I'd like to build, but hints only. Kurynt again differs significantly from The Wheel Screener, which was more or less a full SaaS product at the time I released it. Kurynt has very far to go until it's in a state that I think provides the most value, but I'd like to do this part in step with my early adopters.

Pushing Forward Despite Minor Bugs and Annoyances

Kurynt is also my first SaaS product which is directly built for developers. I'm excited to see how people react to it. Again, I'm quite nervous, as I expect engineers and developers to be the most critical - and Kurynt, as it is now, is riddled with minor bugs and annoyances. But it's usable.

Ultimately, despite these problems, I hope everyone who uses Kurynt can see the future vision and recognize its value and potential. At least on my side, it's something I use daily for InClub, and I've got a growing list of ideas I'd like to implement on the Kurynt platform. However (in the Lean Startup world), I'm going to wait to see what initial feedback I get from early adopters and the community at large. As far as I know, Kurynt is the first and only tool of its kind, combining analytics, some number crunching, and (eventually) natural language processing machine learning models to help you manage all the packages used in your codebases.

The Value-Add of Kurynt

My main assumptions about the market advantage of Kurynt are:

There is no service out there that combines analytics and NLP / machine learning to manage third-party packages.
There is no tool that combines upgrading, regression monitoring, and end of life guidance in a single spot for major frameworks, for example, Node.js, .NET, Go, and so on. Check the Framework Manager page on Kurynt to see what I'm getting at here).

For more information on the technical details of how Kurynt works, check out the about page on the Kurynt website.

Shortcuts and Missing Features

Just to get Kurynt to a state where I could actually release it to get feedback (I still consider it 'super-beta' in my opinion), I've had to take the following shortcuts:

Supporting only package.json parsing with NPM packages - (I started writing flows for requirements.txt (Python), .csproj files, and so on, but quickly realized the complexity would extend the beta release to months instead of weeks 😊)
Supporting only those NPM packages which have a corresponding GitHub repository
Not supporting any alerts or notifications yet
Not supporting any syncing of codebases from places like GitHub, GitLab, or Bitbucket, for example

So... Why are you Releasing Kurynt in Such a State?

Two reasons. The first is just a lame one, which is that next week I'm on vacation. I long said I would have this product 'ready' before this vacation, so the time has finally come. Second is that I've been reading more and more about shipping earlier rather than later, and while I could see hundreds of more hours of development time that could be put in to building new features for Kurynt, I want to see how people use it and what features are the most valuable. I've got the proof of concept up and working, and anything additional not informed by customers is just time which will likely take me away from my vision with Kurynt.

So, in conclusion, I'm at least mentally prepared for a bumpy release, but overall looking forward to it. It may seem to you that Kurynt does nothing right now! 😂 But I encourage you to check out Kurynt and give it a test drive. Any and all feedback is welcome - check out the contact page to do that.

With that, I'm going to enjoy my upcoming vacation. 😎

Cheers! 🍺

-Chris

P.S.: Fun fact: the final cracking dev week to push the release of Kurynt was accomplished only through playing Megadeth's Rust In Peace Album on near-repeat. (Heck, it's playing right now as I write this post!)

One Year as the Chief Technology Officer at InClub

Chris Frewin — Wed, 08 Jun 2022 16:40:50 +0000

This post is mirrored on my blog, chrisfrew.in

It's My 1 Year Anniversary at InClub!

So, today (June 8th, 2022) marks exactly my one year anniversary of my first commit to the InClub codebase:

What is InClub?

InClub is a platform that enables you to participate in a wide variety of experiences in your local area - or if you're traveling or otherwise abroad - experiences in the city or town you are residing. Our slogan is "Join & Host private experiences".

One reason of many that I joined InClub was because I began to notice the slowly decaying popularity of traditional "1.0 social" networks like Facebook, Instagram, and Snapchat. (If we had a secondary slogan, it might be "the anti-trend to Facebook"). InClub is an app that truly brings people back together again, in person, to participate in a variety of experiences, events, parties, and festivals - from experiences as local as going on a walk or drive with an InClub member, to exciting as sky diving.

I also had a great experience from a similar app Party with a Local which is mysteriously no longer available. It was great though - I still have a friend I met through this app and we've met a few times since. (If anyone has more information about what happened to this neat app I'd definitely be interested!)

What We've Accomplished at InClub in the Past Year

As the only contributing software engineer in the last year to the InClub software ecosystem, I'm proud to say that we've shipped:

A chat system that helps participants and organizers get to know each other before taking part in the experience together
A completely automated payment system using TWINT payments and IBAN payouts, whereby participants can request and pay for participation, and organizers can confirm or deny transactions, triggering the corresponding three-way settlement to the organizer, charity / NGO, and InClub.
A filtering / suggestion system based on nearby experiences, your preferences, or interests
A search system that highlights all matching tokens of your search terms, whether it be the experience name, category, or even name of the organizer
An experience template function, where you can take a huge variety of experience ideas and make it your very own as an organizer
A variety of emailing, cleanup, and update batch jobs that keep our infrastructure and ecosystem up-to-date to best serve experience participants and organizers alike

Main Takeaways and Tips for Tech Leads, Engineers, and Developers at Startups

Over last year's ups and downs, I'm happy to share the insights we've gained as a team when it comes to how to best run operations around our application and tech infrastructure. Here are the most important topics and points that I could think of:

1. Having Many Tests and Test-Driven Development (TDD) is Both Underrated and Truly Invaluable

There are many software engineers, developers, and coders out there who scoff at test-driven development. Indeed, TDD is a lot of additional work on top of the actual code behind business logic and features your company may need. But for small teams, especially expanding and growing small teams, the benefits of TDD are numerous:

You have clear, in-code documentation of how your system should work, on both the frontend and backend, as TDD should be applied to all parts of your software
With the help of tools like Gherkin and Cucumber, you have the descriptions of all your features written in steps, in plain English, and you can drive most tests directly from these features with the correct tools
Your tests inform and show any new onboarding employee, whether they be a developer or product manager, the exact current state of the system and product and how it works
The benefit of running all your tests before each release, so that you as a team are more sure and confident that your customers won't experience any bugs as they use your product
The requirement to write and think about tests while designing a new feature help improve the feature design itself, as well as cooperation between the product / management team and the software / engineering team - you will catch edge cases and challenges in the design more often when thinking about the feature in a TDD way
When or if a bug is found (let's be honest- it's always when!), writing a test against it prevents future regressions of said bug 🐛 - trust your tests, see and check why any of them are failing, and adapt and refine as necessary

2. Design, Build, and Run Many Tests... But Think About Your Tests!

What do I mean by this? For a B2C business, customers are king, and without fail or excuses, the product should always (ideally) ship with 0 known bugs for each release. The only way to do this realistically with a small startup team is to have a significant amount of automated tests, and I harped about the many advantages of many tests and TDD in point #1.

The challenge we, as engineers, have, especially in a startup environment, is to design our tests so that they don't take too much work to refactor or change as the business changes. In this way, we are nimble to modify the tests later as features grow, change, or are removed completely. I believe the typical industry rule of thumb is something like "80% unit tests, 15% integration tests, and 5% end-to-end tests". This is a good rule - unit tests are smaller and easier to refactor, change, and remove completely - just like your features. But likely your features are comprised of many unit tests - and you'll need to know what unit, integration, and end-to-end tests are touched by changing these features.

Furthermore, this 80-15-5 "rule" is just a rule of thumb. On our backend, we have about 90% unit tests and 10% integration tests, while on our React Native mobile app, we have some sort of gray area of almost 50 / 50 ratio of unit to integration tests.

3. UI/UX Testing is Hard... Very Hard

How do we test that timed animation?

What about that app switch to that payment platform which doesn't provide any sandbox or out-of-the-box mocks for us to use?

What about translated strings? Should we test those?

What about that WebView that interacts with 3rd party APIs? We don't need to test the third party stuff, right? Just how the WebView acts?

How much of our staging API should we mock, and for which test? All of them? Some of them? None of them?

These questions, as well as thousands of others, must at some point be answered as you build out tests on your UI. The challenge here actually isn't so much about the answers to each question, as they relate to tests in general which (should) have little risk of blowing up your customer's experience and or your production environment. (If you designed your systems and tests properly, your tests should never need to touch or run on a production instance anyway.) The challenge is more about documenting each choice and explaining why you made each choice. Often there are no "right" or "better" answers here... just... answers. Take a decision on what is mocked where and document why that decision was taken.

In addition to the complex animation, various libraries, and API calls from a client, another main challenge with UI testing is the impressive number of user journeys that are possible. Think about having just 3 or 4 yes/no decisions that dictate what screen your user sees next. If you want to run full integration tests on all of these paths, they quickly become numerous if you wish to test all paths:

2^4 = 16
2^5 = 32
2^6 = 64 (!!!)

You can see that the number of branches to test when it comes to user-based choices increases geometrically. Again, with a small team, this becomes a bit of a compromise. Do you try and keep the cool UI/UX of having multiple flexible flows, or simplify the choices themselves? Or, as an alternative (a strategy we've taken at InClub) tests each binary choice logic in isolation in a unit test, thereby testing "all" branches by proxy. We then pick 5-10 of the 'heaviest path' taken by the majority of users in our integration tests. Then even further, you would pick 1-3 of these branches to be run by your end-to-end tests (still getting there for InClub).

It also comes to how you implement your UI. Typically, the wisdom at large is to create UIs with components as reusable as possible, when using component-based frameworks like React, Angular, Vue, and the like. However, when it comes to a business heave component, like a payment button, or a share button, I encourage you to think a bit differently: leverage your UI as much as possible to control the UI flow of your application itself. Don't have a button that when pressed, calls a function, where a giant complex function determines what needs to happen with a specific behavior, action, or navigation side effect.

Rather, couple these business logic-heavy functions directly with the UI element. This way, even if you have many functionalities for a given button location on a screen where its functionality depends on the user or app state, then you are sure when that button is tapped, only that specific business logic associated with that button will be executed. While there will indeed be a bit more code for this type of component design pattern (and use this pattern sparingly!), you will benefit immensely overall from a component architecture that allows you to quickly rearrange UI screen flows and the like, while also being easier to test (and even rearrange your tests!)

4. Test Code Coverage... 🤷‍♂️

At InClub, we don't have any pipe dreams of 100% code coverage with our tests. Sure, if we had an army of engineers, this might be within the realm of possibility. However, we were (until about 2 weeks ago!) an Army of One™️ engineer, and we simply don't have the time as a team for me to spend hacking away to get the elusive 100% code coverage. Others argue that 100% code coverage through tests is always possible if you write your code in a very testable way. But I believe this to mostly be snake oil, or if it is the case, only for specific embedded or backend codebases. I would love to see the test coverage figures of extremely complex and dynamic social apps like Airbnb, Facebook, or Discord. I can almost guarantee they aren't hitting 100%, or even close to that.

After reading quite a lot on this topic, I've ultimately come to an engineer's favorite phrase, that for code coverage, "it depends". Unfortunately, I've lost the reference, but I recall reading somewhere that internally at Google, achieving even a 75% code coverage are considered outstanding. So, if you're on a small team that has even 40% or 50% code coverage in your tests, be proud. (As long as those tests correspond to the most important customer-facing features!)

5. Be a Perfectionist... a Perfectionist at Being Pragmatic

At a startup, you don't have time to build the "perfect, infinite scale, 0 bugs, 0 latency, 100% uptime" solution. (Ok, all technicalities aside, you could, but that would mean totally blocking all other business tasks for perhaps 2-3 months per feature... good luck!)

At a startup that is hopefully consistently growing, and consistently wanting to make its customers happy, and reap all the benefits that come with that (word of mouth / organic growth, etc.), you need to respond quickly to what the customers are asking for. To address these competing needs of infrastructure vs feature building, I've developed a sort of hierarchy of importance for each line of code that is written into the InClub ecosystem:

functional (it works)
commented (can other engineers in the organization immediately understand what the code does?)
documented (can other owners or product managers in the organization get exposure to or understand what this does?)
fully tested (is there an automated test that hits this specific line of code?)

I believe these four points are a sort of "pragmatic programmer's compass" for writing any line of code at a startup.

As a much more fun and realistic alternative to perfectionism, I recommend you to start thinking of your codebases as systems to manage, not inflexible perfect lines of text which never produce errors. Trust me, no matter how good you think you are or how many years of experience you have, you will write code either with bugs in it, unexpected behavior, memory issues, or reliability problems. If you haven't, please write to me, we would like to hire you! 😊

A company's tech infrastructure is just like a garden: you have to trim the hedges, do some weeding, and sometimes a plant or two will get sick, or brown spots on your grass will start cropping up. With time, you can fix and restore each of these problems, keeping your garden healthy overall. Neglecting any portion for too long, however, and you start to have larger problems. You also can't solve all problems simultaneously or all in one day, you have to plan strategically and move step by step to tackle each problem, keeping at it constantly to keep your garden looking sharp!

6. Address Tech Debt, and Address it Often

Following this metaphor of "garden maintenance" - you need to reserve days (or even a week sometimes) to address the technical debt at a growing startup. This of course blocks the time you have to build new features, and so it becomes a constant battle between the two. At InClub, we are still struggling to balance this perfectly, and based on what I've been reading, I'm starting to think it is impossible to balance perfectly because it is simply impossible to predict how your organization's tech debt will grow and change through time.

As your business changes along with your customers' needs, older features of the business may stay the same, change, or fall out of use entirely. The easy tech debt refactors are modules that you can just remove completely. This gives your engineers a sigh of relief as it's just one less thing to monitor and maintain. The features which will remain customer-facing but also need to be upgraded or cleaned up are more complex, however. You need to plan your changes and refactors in a way that they won't cause breaking changes for the customers, and these considerations make these sorts of tech debt challenges doubly long.

That's why it's so important to adhere to some of the modularization techniques and design patterns presented in literature like the gang of four. The cleaner and more isolated parts of your code are from one another, the easier it is to snip away old and unused parts, as well as improve and modify existing parts, or as all businesses may eventually need, breaking out modules into their own isolated microservices. Pair this clean design with all the tests you and your team should be writing (see points 1, 2, 3, and 4) and you should never be worried about how refactoring might break parts of your product or business logic!

What's Coming Up Next with InClub

This year will be one of the most challenging for InClub, at least from a technological perspective.

The challenge is that we need to successfully execute essentially two parallel tech paths: while we seek to expand and build on our customer base, behind the scenes we are also building out the next version of infrastructure, which we are simply calling internally "InClub Infrastructure 2.0". The nice advantage with the 2.0 infrastructure is that we can build the services we need from scratch, now that we better know and understand our customer's needs, as well as our own needs for exactly what each business logic needs to succeed. This code base is also going to be much more cloud-friendly and scalable than our current infrastructure - a system that will be tuned to handle hundreds of thousands to even (hopefully) millions of customers as we look to expand globally in 2023.

Thanks for Reading!

Whether you are an owner, product manager, or engineer at a startup, I hope these points and tips can help your teams build your product better.

Cheers 🍻

-Chris

Advanced Code Organization Patterns: The Case For One File Per Function

Chris Frewin — Wed, 16 Mar 2022 22:32:26 +0000

Disclaimer: this post was written with frontend projects in mind, like websites, React Native mobile apps, and Electron apps. Though there are some transferrable skills to backend programming, as I mention further down. Also, the title is a bit of a joke. 😄 This is not that advanced, but it IS often overlooked.

In this post, I'm going to talk about a pattern for organizing functional code in frontend projects that is very basic, but often totally overlooked... it's called... one function per file!

"You must be joking, right? File organization? Such a basic concept, can't believe this guy is posting about this..." Well, I'll push back and say its something that is considered so simple that it is often overlooked and forgotten about, and can lead to huge-sized 'god' files filled with dozens and dozens of functions over the long run. So consider reading this post! And respect the basics!

Two Code Organization Patterns

Consider two ways to organize your code's functions:

Grouping several related functions into one file. Example: A file called math.js, under the utils/ folder, filled with functions add, subtract, multiply, and divide:

utils
└── math.js
    ^ contains functions add, subtract, multiply, and divide

Placing each and every function in it's own file. Example: Files add.js, subject.js, multiply.js, divide.js, all under the utils/math/ folder:

utils
└── math
    ├── add.js
    ├── divide.js
    ├── multiply.js
    └── subtract.js

For quite some time, I used to think option 1. was better. Organizing related functions into the same file! Fancy! It seems to make sense, right? Fewer files, less complexity?

Over time, working with larger and larger projects, the more refreshing it is to work with small, manageable building blocks of code. We can anyway abstract the concept of 'similar' or 'related' functions by using a variety of folders to name the groupings.

Let's take a look at three reasons I've learned over my decade+ career as a software engineer that single function files are better.

Three Reasons Case for Single Function Files

1. Tests

When you want to unit test your functions, it's very clear when reading test code exactly what singular function you are importing - nothing more, nothing less. For integration tests, it's just as clear - we would expect to see an import for each function that is required for the integration test.

2. Code Merging

If you have a complex git tree with features and hotfix branches, putting many functions in a big 'utils' file or similar will only lead to headaches when you want to preserve the functionality (no pun intended) of each of the functions. When each of your functions is isolated to a single file, it is much easier to go through the git history per file and see what changes were made to the given function (and only that function).

3. Organization

It's easier to grok the organization of your code base by being immediately able to understand where the function is based on the import statement alone. With option 1., the add function is in its own self-named file, as is subtract, and so on. With option 2., There is an extra mental leap required to go from thinking "ok... utils/, math.js... aha! add is within math.js!"

If you're writing any modern app that has any sort of complexity at all, trust me: you're going to have a lot of utility functions, all collected around various actions, tasks, and activities - easily north of 50 or even 100 functions if you're compartmentalizing your code properly.

Drawbacks

There aren't many drawbacks to this pattern - there is only one I can think of, which is that your import statements (or require) will be more numerous if you compartmentalize each of your functions into a separate file. However, most linters and formatters are going to make an import multi-line anyway if you choose to take the path of option 1. above, i.e. formatting like:

import {
    add,
    subtract,
    multiple,
    divide
} from "./utils/math"

Instead of the organization pattern (option 2.) that I suggest:

import { add } from "./utils/math/add"
import { subtract } from "./utils/math/subtract"
import { multiply } from "./utils/math/multiply"
import { divide } from "./utils/math/divide"

And in fact, it's a line shorter for all you line hounds out there 😉

Other Considerations

Note that this pattern is for your own internal code. If you were shipping a library or package it would make sense to make all your exports from the same file, like and index.js or index.ts. Even then, this is only for the export interfacing and the source code of the library itself could (and in my opinion, should) use this pattern.

Again, the entirety of this post may seem very obvious. But I've seen a lack of code organization arise again and again in so many different projects that I thought I would reiterate the importance of this pattern.

This pattern is also transferrable to object-oriented programming. If you find you are writing too many private (or even public) methods in a class, it's a sign that you should break some of that logic into yet another class, ultimately leading to smaller, more readable, and easier-to-understand code.

Thanks!

How do you like to organize your functions in your client-facing apps? Would you choose option 1. or 2., and why?

Cheers,

-Chris 🍺

React with TypeScript: Optional Props with Default Values

Chris Frewin — Sun, 10 Oct 2021 15:40:09 +0000

Straight to the Code

There's an example repository scaffolded with create-react-app which I've created for this post. Probably of most interest is the component FancyTitle which uses the default props pattern discussed in this blog post.

Now let's get into the details of the pattern.

Optional Props Pattern

When using TypeScript with React, it's easy enough to define optional props. With your props interface, you simply apply the optional parameter symbol ?:

interface IMyComponentRequiredProps {
  someRequiredProp: string;
  someOptionalProp?: string;
}

Optional Props with Default Values Pattern

But what if we want default values for our optional props in the case when they are not specified?

A rather elegant way to do it is to define two separate interfaces, one for the required props, and one for the optional props. For example, if our component is called MyComponent:

// Required props
interface IMyComponentRequiredProps {

}

// Optional props
interface IMyComponentOptionalProps {

}

We can then use the TypeScript extends operator to combine these two interfaces to the final props interface we would use to actually type the props parameter:

interface IMyComponentProps
  extends IMyComponentRequiredProps,
    IMyComponentOptionalProps {}

We can then define our default props values by only taking the IMyComponentOptionalProps:

const defaultProps: IMyComponentOptionalProps = {
  color: "red",
  fontSize: 40,
};

and then being sure to set these defaultProps to the component's defaultProps:

MyComponent.defaultProps = defaultProps;

All Together Now...

Let's see this pattern in an actual React component. Here's the example component FancyTitle from the example repository which renders a customizable <h1> tag. The component has a required title string, and then a few optional style options. We can leverage the pattern discussed in the post to build the following:

import * as React from "react";

// Required props
interface IFancyTitleRequiredProps {
  title: string;
}

// Optional props
interface IFancyTitleOptionalProps {
  color: string;
  fontSize: number;
}

// Combine required and optional props to build the full prop interface
interface IFancyTitleProps
  extends IFancyTitleRequiredProps,
    IFancyTitleOptionalProps {}

// Use the optional prop interface to define the default props
const defaultProps: IFancyTitleOptionalProps = {
  color: "red",
  fontSize: 40,
};

// Use the full props within the actual component
const FancyTitle = (props: IFancyTitleProps) => {
  const { title, color, fontSize } = props;
  return <h1 style={{ color, fontSize }}>{title}</h1>;
};

// Be sure to set the default props
FancyTitle.defaultProps = defaultProps;

export default FancyTitle;

That's it! Required, optional, and default values for props in a React functional component with TypeScript!

Thanks!

I hope you enjoyed this post and you find this pattern useful.

Cheers 🍺

-Chris

🎃 Hacktoberfest 2021 Hype!

Chris Frewin — Thu, 30 Sep 2021 11:19:17 +0000

Well, Hacktoberfest 2021 starts tomorrow, I was looking around dev.to... and...

It's a graveyard out there! (Pun totally intended.) There's nearly no discussion or hype for Hacktoberfest, and it starts TOMORROW!

While I expect a lot more discussion and excitement tomorrow, it can't hurt to start the discussion off a bit early, right? So...

Who is looking for maintainers / pull requests / help on their project(s) this Hacktoberfest?!

I'll start:

react-use-please-stay

I have a fun and simple React hook called usePleaseStay that helps you animate your favicon and title like so:

The repository is react-use-please-stay on GitHub.

I've opened a few possible issues that I think could be further optimizations to this hook! I hope you'll consider taking a look at them or tackling one!

Hope to see you all out there this October on the front lines of open source software! Have fun!

DEV Community: Chris Frewin

Two of My Favorite Custom MCP Tools I Use Every Day

My Two Favorite MCP Tools - True Game Changers

Tool #1: list_repo_locations

Create the Basic Structure

Add Fuzzy Matching Magic

Implement the Levenshtein Distance Util Function

Tool #2: issue_terminal_command

Set Up the Basic Command Executor

Wiring It All Together

Update Your Server Registration

Handle the Tool Calls

Make It Your Own

Don't Forget: Safety First :)

The Result? Claude Code, But Free!

Coming Soon: Deep Dive into MCP Development

Thanks for Your Time!

Building a Virtual and Declarative IDE: How Complex is it to Create a New Folder or File in an IDE, Really?

In Short: It’s Complicated

Breaking it Down

Some Good News

Introducing Composite Actions!

…Atomic Actions?

CodeVideo: My 2+ Year Journey

It's Finally Here!

Calling All Software and Coding Educators and Content Creators!

CodeVideo In a Nutshell

The History of CodeVideo

Early Days

January / February 2024

Event Sourcing? What's That?

The First Virtual Layer: Enter virtual-code-block

November 2024: Applied to YCombinator!

Major Theoretical Breakthrough #2: Parallelized Video Generation

Final Release Push in Q1 2025

..and Now We're Here

Growth as a Software Engineer

Getting Started

I'm Super Happy You're Here!

Tales from the Software Graveyard: The Birth and Death of Squawk Market

The Squawk Market Service & Background

Learnings and Takeaways

Go is Fast!

Text to Speech Models — Largely Varied in Price and Quality

We Can Also Parse Any Video with Speech to Text in Realtime!

Squawk Market is Now Open Source!

Thanks!

Full Stack Forays with Go and gRPC

What is gRPC?

What is gRPC Good For?

Disadvantages of gRPC

Get Started

Backend Implementation

Frontend Implementation

gRPC Web Proxy

Example code

Thanks!

Create a Dockerized Go Application - With a Cron Job, Slack Messaging, and Test the Whole Thing

Why Go?

1. Go is extremely performant

2. Go is extremely compact

3. Go is extremely testable

4. Go is extremely API friendly

What We're Going to Build

Requirements

Writing the App

Install Go

Scaffold The App

Cron Job

HTTP Utility

API Response Parsing

Slack Messaging

Hooking it All Together

Run It!

Tests

Dockerize the Application

Done! Fin! Finito!

Thanks!

Craft a Stylish and Practical Developer Book with this LaTeX Template

The Code

Tool #1: `list_repo_locations`

Tool #2: `issue_terminal_command`

The First Virtual Layer: Enter `virtual-code-block`