DEV Community: Fabrikapp

TranscribeNow : Live Note-taking and Subtitles

Fabrikapp — Mon, 25 Nov 2024 03:33:29 +0000

This is a submission for the AssemblyAI Challenge : No More Monkey Business.

What I Built

For the AssemblyAI Challenge, I developed a real-time audio transcription and note-taking application. This project combines the power of AssemblyAI's Streaming API with a user-friendly interface to provide instant transcription, live note-taking, and AI-assisted content generation.

The application consists of three main components:

A Chrome extension for capturing tab audio, displaying subtitles, and fetching audio from any webpage or microphone
A server-side component for handling WebSocket connections and interacting with AssemblyAI's API
A frontend web application for displaying transcriptions and managing notes. The user will be able to rewrite and generate notes from a recorded session

Demo

Live demo : https://devto-assemblyai-hackaton-frontend-581527456637.us-central1.run.app/

Source Code

Source Code Repository

Screenshots

Review and edit transcribed sessions with additional context and notes

AI-assisted note generation based on the transcribed content

The dashboard interface showing live transcription and note-taking stats

Journey

Integrating AssemblyAI's Lemur API was an interesting part of this project:

API Integration: I added server-side actions to interact with Lemur for summarization, question answering, and action item generation. This involved learning the API endpoints and response structures.
User Interface: Incorporating Lemur's features into the frontend required some UI/UX considerations to make the AI-generated content accessible and useful to users.
Learning Curve: Getting familiar with Lemur's capabilities took some time. I experimented with different prompts and parameters to understand how to best utilize the API for our use case.
Added Value: Lemur allowed the application to go beyond simple transcription, offering users more insights from their audio content.

While integrating Lemur had its challenges, it ultimately enhanced the functionality of the application, providing users with AI-powered analysis of their transcribed content.

By integrating these additional tools, I was able to create a more comprehensive and powerful application that goes beyond simple transcription. The Chrome extension allows for seamless audio capture from any web content, while the AI-assisted content generation provides valuable insights and summaries to users, making their note-taking process more efficient and effective.

Throughout this project, I gained valuable experience in working with real-time audio processing, WebSocket communication, and integrating AI capabilities into a web application. The AssemblyAI Streaming API proved to be robust and reliable, enabling me to create a responsive and accurate transcription experience for users.

TranscribeNow : Live Note-taking and Subtitles

Fabrikapp — Mon, 25 Nov 2024 03:27:09 +0000

This is a submission for the AssemblyAI Challenge : Really Rad Real-Time.

What I Built

The application consists of three main components:

A Chrome extension for capturing tab audio, displaying subtitles, and fetching audio from any webpage or microphone
A server-side component for handling WebSocket connections and interacting with AssemblyAI's API
A frontend web application for displaying transcriptions and managing notes. The user will be able to rewrite and generate notes from a recorded session

Demo

Live demo : https://devto-assemblyai-hackaton-frontend-581527456637.us-central1.run.app/

Source Code

Source Code Repository

Screenshots

Review and edit transcribed sessions with additional context and notes

AI-assisted note generation based on the transcribed content

The dashboard interface showing live transcription and note-taking stats

Journey

Implementing AssemblyAI's Streaming API was an exciting process that involved several key steps:

Setting up the WebSocket connection: I used the Hono framework to create a WebSocket server that acts as a bridge between the client and AssemblyAI's API.
Audio capture and streaming: I developed a Chrome extension to capture tab audio and stream it to our server. The extension uses the chrome.tabCapture API to access the audio stream.
Real-time transcription display: On the frontend, I created a React component to display the live transcription as it comes in from the WebSocket.
AI-assisted note generation: I implemented a feature that allows users to generate notes based on the transcribed content using AI-powered instructions.

One of the most challenging aspects was ensuring a smooth, real-time experience while handling the continuous stream of audio data and transcription results. I had to carefully manage state updates and optimize rendering to prevent lag or jitter in the user interface.

I am really happy to be able to connect AssemblyAI To any webpage for instant note taking and more !

I created a Realtime Voice Assistant for my ESP-32, here is my journey - Part 2 : Node, OpenAI, Langchain

Fabrikapp — Tue, 19 Nov 2024 23:08:11 +0000

From Hardware to Smartware: Building the Brain of Our ESP32 Voice Assistant

Welcome back to part two of our ESP32 voice assistant series! In part one, we laid the hardware foundation by configuring the ESP32 to capture voice input and play back responses. Now, we're diving into the exciting realm of giving our assistant intelligence using a Node.js server powered by **LangChain* and OpenAI.*

The Journey So Far

In the first installment, we got our hands dirty with the hardware side of things—connecting microphones, setting up speakers, and ensuring our ESP32 could handle audio input and output. We managed to make the ESP32 record audio and play it back—an essential first step.

But let's be real, an assistant that only echoes what you say isn't particularly smart. So, the next logical step was to infuse it with some intelligence. That's where LangChain and OpenAI come into play.

If you missed the part one, here are the relevant medias.

The Ugly Beast

Video

Source Code

The complete source code for this project is available on GitHub:
ESP32 AI Assistant Repository

The Server Challenge

Initially, I thought, "I've got this—Node.js is my jam!" However, as I delved deeper, I realized that integrating real-time audio streaming, managing WebSocket connections, and handling AI processing was...quite a ride !

After numerous caffeine-fueled debugging sessions, frantic whiteboard scribbles, and eureka moments in the shower, I developed a robust server architecture that I'm excited to share with you.

Behind the Scenes: How It All Works

Imagine this workflow:

Voice Capture: Your voice is captured by the ESP32's microphone.
Data Transmission: The audio data is sent via WebSocket streams to a Node.js server.
AI Processing: The server processes the audio in real-time, uses LangChain to orchestrate the agent and maange communication with OpenAI.
Response Playback: The server sends the response back to your ESP32 for playback—all in a matter of seconds.

Here are the key components that make this system work:

WebSocket Server: The custom WebSocket server handles audio streams smoothly while maintaining stable connections with the ESP32.
Audio Processing: Raw audio gets transformed into OpenAI-compatible formats, with careful buffer management to keep the stream flowing.
Playback: The system ensures audio responses play back smoothly on the ESP32, making conversations feel natural.
LangChain Agent: This is where the intelligence comes in - LangChain helps you manage your IA flow "easily".

Let's look at how these pieces work together, starting with the core intelligence.

The Brain of the Operation

The heart of the server is the Agent Management system (in lib/agent.ts). It acts as the command center, processing incoming speech and determining appropriate responses.

Voice Assistant Server Implementation (Node.js/TypeScript)

Core Components

1. Agent Management (`lib/agent.ts`)

The OpenAIVoiceReactAgent class encapsulates the core functionality of the voice assistant, including buffer handling from start to finish.

export class OpenAIVoiceReactAgent {
    // Protected properties
    protected connection: OpenAIWebSocketConnection;
    protected instructions?: string;
    protected tools: Tool[];
    protected BUFFER_SIZE = 4800;

    // Public properties
    public buffer = new Uint8Array();
    public audioBuffer: Buffer | undefined;

    // Private properties
    private audioManager: AudioManager;
    private recording: boolean = false;

    constructor(params: OpenAIVoiceReactAgentOptions) {
        this.audioManager = new AudioManager();
        this.connection = new OpenAIWebSocketConnection({
            url: params.url,
            apiKey: params.apiKey,
            model: params.model,
            audioConfig: params.audioConfig,
            audioManager: this.audioManager
        });
        this.instructions = params.instructions;
        this.tools = params.tools ?? [];
    }

    // Starts a new recording session
    public startRecordingSession(): void {
        if (this.recording) {
            this.audioManager.resetRecording();
        }
        this.recording = true;
        this.audioManager.startRecording();
        console.log('Started new recording session');
    }

    // Stops recording and processes the audio
    public async stopRecordingAndProcessAudio(): Promise<void> {
        if (!this.recording) {
            console.log('No active recording to stop');
            return;
        }

        this.recording = false;

        try {
            this.audioManager.closeFile(); // Finalize the audio file
            const buffer = this.audioManager.getCurrentBuffer();
            if (buffer.length === 0) {
                console.log('No audio data captured');
                return;
            }

            const base64 = convertAudioToPCM16(buffer);
            await this.sendAudioEvent(base64);

        } catch (error) {
            console.error('Error processing audio:', error);
        } finally {
            this.audioManager.resetRecording();
            console.log('Recording session ended and cleaned up');
        }
    }

    // Sends the audio event to OpenAI
    private async sendAudioEvent(base64Audio: string): Promise<void> {
        if (!base64Audio) {
            console.log('No audio data to send');
            return;
        }

        const eventAudio = {
            type: 'conversation.item.create',
            item: {
                type: 'message',
                role: 'user',
                content: [{
                    type: 'input_audio',
                    audio: base64Audio
                }]
            }
        };

        try {
            this.connection.sendEvent(eventAudio);
            // Wait for the audio event to be processed
            await new Promise(resolve => setTimeout(resolve, 2000));
            this.connection.sendEvent({
                type: 'response.create'
            });
        } catch (error) {
            console.error('Error sending audio event:', error);
            throw error;
        }
    }

    // WebSocket Connection Methods
    async connect(
        websocketOrStream: AsyncGenerator<string> | WebSocket,
        sendOutputChunk: (chunk: string) => void | Promise<void>
    ): Promise<void> {
        // ...connection setup and event handling...
    }

    // ...additional methods...
}

Key Features:

Buffer Handling from Start to Finish:
- Manages the initiation, recording, processing, and resetting of audio buffers.
- Ensures that audio data is handled efficiently and accurately throughout the session.
Manages WebSocket connections.
Controls audio recording sessions.
Handles tool execution.
Processes event streams.

2. Audio Management (`lib/audio.ts`)

The AudioManager class is responsible for audio processing and buffering, a critical component in handling the audio data from the ESP32.

export class AudioManager {
    private configHighDef: AudioConfig = {
        sampleRate: 44100,
        channels: 1,
        bitDepth: 16
    };

    private fileWriter: wav.FileWriter | undefined;
    private writeTimeout: NodeJS.Timeout | null = null;
    private isProcessing: boolean = false;
    private config = this.configHighDef;
    private audioBuffer: Buffer = Buffer.alloc(0);

    constructor() {
        // Initialization code
    }

    // Resets the recording session
    public resetRecording(): void {
        if (this.writeTimeout) {
            clearTimeout(this.writeTimeout);
            this.writeTimeout = null;
        }

        if (this.fileWriter) {
            this.fileWriter.end();
            this.fileWriter = undefined;
        }

        this.audioBuffer = Buffer.alloc(0);
        this.isProcessing = false;
    }

    // Starts a new recording session
    public startRecording() {
        this.resetRecording();
        const randomId = Math.random().toString(36).substring(2, 15);
        const filename = path.join(__dirname, '../../tmp', `recording-${randomId}.wav`);
        this.initializeFileWriter(filename);
        console.log('Started new recording:', filename);
    }

    // Handles incoming audio buffers
    public handleAudioBuffer(buffer: Buffer): void {
        // Buffer management logic
    }

    // Processes and writes buffer to file
    private processAndWriteBuffer(): void {
        // Buffer processing logic
    }

    // Retrieves the current buffer
    public getCurrentBuffer(): Buffer {
        if (!this.fileWriter) {
            console.error('No file writer available');
            return Buffer.alloc(0);
        }

        try {
            const audioFilePath = this.fileWriter.path;
            const fileBuffer = fs.readFileSync(audioFilePath);
            console.log(`Successfully read audio file: ${audioFilePath}`);
            return fileBuffer;
        } catch (error) {
            console.error('Error reading current audio buffer:', error);
            return Buffer.alloc(0);
        }
    }

    // Closes the file writer
    public closeFile(): void {
        console.log('Closing WAV file writer');
        if (this.writeTimeout) {
            clearTimeout(this.writeTimeout);
        }
        if (this.audioBuffer.length > 0) {
            this.processAndWriteBuffer();
        }
    }

    // ...additional methods...
}

Features:

Buffer Management:
- Efficient handling of audio buffers to prevent memory issues.
- Timely processing and writing of buffers to maintain real-time performance.
Audio Processing Pipeline:
- Converts raw audio data into formats compatible with OpenAI's APIs.
- Includes methods to start, stop, and reset recordings.
WAV File Operations:
- Handles writing audio data to WAV files for processing.
- Manages file writers and ensures proper closure and cleanup.

3. WebSocket Connection and Buffer Handling (`src/index.ts`)

The index.ts file manages the WebSocket connections and orchestrates the flow between the client and the server.

import "dotenv/config";
import { WebSocket } from "ws";
// ...additional imports...

const app = new Hono();
const WS_PORT = 8888;
const connectedClients = new Set<WebSocket>();

// ...app setup...

app.get(
  "/device",
  upgradeWebSocket((c) => ({
    onOpen: async (c, ws) => {
      // ...connection setup...

      // Wait 1000ms before connecting to allow WebSocket setup
      await new Promise(resolve => setTimeout(resolve, 1000));
      await agent.connect(rawWs, broadcastToClients);
    },
    onClose: (c, ws) => {
      // ...cleanup...
    },
  }))
);

// ...server setup...

Key Features:

WebSocket Management:
- Handles bi-directional communication between the ESP32 and the server.
- Manages binary data handling for audio streams.
Broadcasting to Clients:
- Sends AI-generated audio responses back to connected ESP32 devices.
- Manages chunking of audio data to accommodate ESP32 buffer limitations.

4. Utility Functions (`lib/utils.ts`)

Various utility functions assist in handling buffers and working with audio data.

export function convertAudioToPCM16(
    audioFile: Buffer, 
    sourceRate: number = 44100, 
    targetRate: number = 24000
): string {
    // Convert audio file to PCM16 24kHz mono format from 44.1kHz
    // ...conversion logic...
}

Key Utilities:

Buffer Conversion:
- Converts audio files to the appropriate format and sample rate.
- Handles resampling and encoding to base64 for transmission.

Buffer Handling: From Start to Finish

One of the most critical aspects of this project is efficient buffer handling. Let's take a journey through how audio data is captured, processed, and transmitted.

1. Capturing Audio on the ESP32

The ESP32 captures audio data from the microphone and sends it over a WebSocket connection to our Node.js server. Due to the ESP32's limited resources, we need to ensure that the data is sent in manageable chunks.

2. Receiving Data on the Server

On the server side, the AudioManager class in lib/audio.ts is responsible for handling incoming audio buffers.

public handleAudioBuffer(buffer: Buffer): void {
    try {
        if (this.audioBuffer.length + buffer.length > this.MAX_BUFFER_SIZE) {
            this.processAndWriteBuffer();
        }

        this.audioBuffer = Buffer.concat([this.audioBuffer, buffer]);

        if (this.writeTimeout) {
            clearTimeout(this.writeTimeout);
        }

        this.writeTimeout = setTimeout(() => {
            if (this.audioBuffer.length > 0) {
                this.processAndWriteBuffer();
            }
        }, this.WRITE_DELAY);

        if (this.audioBuffer.length >= this.MIN_BUFFER_SIZE && !this.isProcessing) {
            this.processAndWriteBuffer();
        }

    } catch (error) {
        console.error('Error handling audio buffer:', error);
        this.audioBuffer = Buffer.alloc(0);
        this.isProcessing = false;
    }
}

What's Happening Here?

Buffer Accumulation: As data arrives, we accumulate it into this.audioBuffer.
Memory Management: We check if adding the new buffer would exceed our MAX_BUFFER_SIZE (to prevent memory overflows). If so, we process and write the buffer to disk.
Write Delay: We use a writeTimeout to batch the writing process, allowing for efficient file I/O operations.
Processing Triggers: If the buffer reaches a minimum size before the timeout, we process it immediately to maintain responsiveness.

3. Processing and Writing the Buffer

The processAndWriteBuffer method handles the actual writing of the audio data to a WAV file.

private processAndWriteBuffer(): void {
    if (!this.fileWriter || this.audioBuffer.length === 0 || this.isProcessing) {
        return;
    }

    try {
        this.isProcessing = true;
        const bufferToWrite = this.audioBuffer;
        this.audioBuffer = Buffer.alloc(0);
        this.fileWriter.write(bufferToWrite);
        console.log(`Successfully wrote ${bufferToWrite.length} bytes of audio data`);
    } catch (error) {
        console.error('Error writing audio buffer:', error);
    } finally {
        this.isProcessing = false;
    }
}

Key Points:

Atomic Operations: We ensure that writing to the file and resetting the buffer are atomic to prevent data loss.
Concurrency Control: The isProcessing flag prevents concurrent writes, which could corrupt the audio file.
Error Handling: Robust error handling ensures that any issues do not crash the application and are properly logged.

4. Finalizing the Audio Data

When the recording session is over, we finalize the audio data:

public closeFile(): void {
    console.log('Closing WAV file writer');
    if (this.writeTimeout) {
        clearTimeout(this.writeTimeout);
    }
    if (this.audioBuffer.length > 0) {
        this.processAndWriteBuffer();
    }
}

Why This Matters:

Ensuring Data Integrity: We process any remaining data before closing the file to ensure we don't lose the tail end of the audio.
Cleanup: Clears any timeouts and resets the state to prepare for the next recording session.

5. Sending the Audio to OpenAI

Once we have the processed audio buffer, we need to send it to the OpenAI API for transcription and understanding.

private async sendAudioEvent(base64Audio: string): Promise<void> {
    if (!base64Audio) {
        console.log('No audio data to send');
        return;
    }

    const eventAudio = {
        type: 'conversation.item.create',
        item: {
            type: 'message',
            role: 'user',
            content: [{
                type: 'input_audio',
                audio: base64Audio
            }]
        }
    };

    try {
        this.connection.sendEvent(eventAudio);
        await new Promise(resolve => setTimeout(resolve, 2000));
        this.connection.sendEvent({
            type: 'response.create'
        });
    } catch (error) {
        console.error('Error sending audio event:', error);
        throw error;
    }
}

Base64 Encoding: The audio buffer is converted to a base64 string for transmission.
Event Dispatching: We send a conversation.item.create event to the OpenAI service, which triggers the AI processing.

Handling the Speaker Output

An essential part of a voice assistant is being able to speak back to the user. This involves receiving audio data from OpenAI and sending it back to the ESP32 for playback.

1. Receiving Audio Responses

The server listens for audio data in the event stream from OpenAI.

private async handleSpeakerOutput(
    data: any,
    toolExecutor: VoiceToolExecutor,
    sendOutputChunk: (chunk: string) => void | Promise<void>
): Promise<void> {
    const { type } = data;

    if (type === "response.audio.delta" || type === "response.audio_buffer.speech_started") {
        await sendOutputChunk(JSON.stringify(data));
    } else if (type === "error") {
        console.error("error:", data);
    } else if (!EVENTS_TO_IGNORE.includes(type)) {
        console.log(type);
    }
}

Streaming Audio Data: The response.audio.delta event contains chunks of audio data that form the AI's spoken response.
Sending Chunks to Clients: We use the sendOutputChunk function to send these audio chunks back to the connected clients (our ESP32 devices).

2. Broadcasting to ESP32 Clients

In src/index.ts, we broadcast the audio data to all connected ESP32 clients, making sure to handle their buffer limitations.

const broadcastToClients = (data: string) => {
    connectedClients.forEach(client => {
        if (client.readyState === WebSocket.OPEN) {
            try {
                const parsed = JSON.parse(data);
                if (parsed.type === "response.audio.delta" && parsed.delta) {
                    const audioBuffer = Buffer.from(parsed.delta, 'base64');
                    // Send audio data in chunks that ESP32 can handle
                    const CHUNK_SIZE = 1024; // ESP32 friendly chunk size
                    for (let i = 0; i < audioBuffer.length; i += CHUNK_SIZE) {
                        const chunk = audioBuffer.slice(i, i + CHUNK_SIZE);
                        client.send(chunk);
                    }
                    return;
                }
            } catch (e) {
                // If parsing fails, send original data
            }
        }
    });
};

Key Considerations:

Chunking the Data: ESP32 devices have limited buffer sizes, so we split the audio data into chunks that they can handle (CHUNK_SIZE of 1024 bytes).
Binary Data Transmission: The audio data is sent in binary format. We ensure the WebSocket connection is set to handle binary data appropriately.

3. Playback on the ESP32

On the ESP32 side, the device receives the audio chunks and plays them through its connected speaker. The efficient handling and timely delivery of these chunks ensure that the playback is smooth and without noticeable latency.

Setting Up the Project

Clone the Repository

   git clone https://github.com/FabrikappAgency/esp32-realtime-voice-assistant.git
   cd esp32-ai-assistant

Install Server Dependencies

   cd server_langchain
   npm install

Configure Environment Variables

Copy .env.example to .env
Add your OpenAI API key

Start the Server

   npm run dev

Usage

Power Up Your ESP32

Ensure it's connected to the same network as your server.

Interact with the Assistant

Press the designated button to start recording.
Speak your query or command.
Release the button to stop recording.

Receive Responses

The AI processes your input, and the ESP32 plays back the response.

Troubleshooting Buffer Issues

Audio Quality Problems
- Verify microphone wiring and grounding.
- Ensure sample rates between ESP32 and server match.
- Check and adjust audio buffer sizes.
WebSocket Connection Errors
- Confirm network connectivity.
- Validate the WebSocket server address in the ESP32 code.
- Use logs to monitor connection status.
Playback Stutters
- Increase the CHUNK_SIZE if the ESP32 can handle larger buffers.
- Optimize the network to reduce latency.

Future Improvements

Here are some planned improvements and future directions for the project:

Enhanced Tool Integration

Open Interpreter Integration
- Add support for Open Interpreter to enable code execution capabilities
- Implement sandboxed environment for safe code execution
- Enable real-time code generation and execution feedback
Extended Tool Ecosystem
- Implement RAG (Retrieval Augmented Generation) for local knowledge base
- Add web scraping capabilities for real-time information
- Create custom function registry for easy tool addition
- Enable tool chaining for complex multi-step operations

Hardware Enhancements

Camera Integration
- Add ESP32-CAM module support for visual input
- Implement computer vision capabilities using TensorFlow Lite
- Enable QR code/barcode scanning functionality
- Add gesture recognition for hands-free control
Smart Voice Activation
- Implement wake word detection (e.g., "Hey Assistant")
- Add voice activity detection (VAD) for automatic recording
- Create noise cancellation algorithms
- Enable continuous conversation mode

Performance Optimizations

Buffer Management
- Implement adaptive buffer sizing based on network conditions
- Add circular buffer implementation for smoother audio handling
- Create buffer monitoring and optimization tools
- Enable compression for reduced bandwidth usage
Code Quality
- Implement comprehensive error handling
- Add unit tests and integration tests
- Create documentation using standardized formats
- Optimize memory usage and reduce heap fragmentation

User Experience

Feedback Systems
- Add LED indicators for system status
- Implement haptic feedback for button-free operation
- Create audio cues for different states
- Display connection quality metrics
Configuration Interface
- Build web-based configuration portal
- Enable OTA (Over-The-Air) updates
- Add customizable voice and personality settings
- Create user profiles for personalized responses

Conclusion

This project showcases the potential of integrating embedded systems with advanced AI capabilities. By combining the ESP32's hardware with a sophisticated Node.js server and leveraging LangChain and OpenAI, we've created a responsive and intelligent voice assistant. This platform opens up numerous possibilities for home automation, personal assistance, and educational applications.

Now you can add any tool you need easily, including RAG, scraping, custom functions etc.

Feel free to contribute to the project or adapt it to suit your specific needs. Happy coding!

Now you can add any tool you need easily, including RAG, scraping, custom functions, etc.

Feel free to contribute to the project or adapt it to suit your specific needs. The possibilities are endless, and this is just the beginning.

Happy coding!

If you enjoyed this article or have questions, let's continue the conversation in the comments below.

I created a Realtime Voice Assistant for my ESP-32, here is my journey - Part 1 : Hardware, PlatformIO & C++

Fabrikapp — Tue, 19 Nov 2024 22:46:05 +0000

Building a Real-time Voice Assistant with ESP32: Hardware and C++ Implementation

In this two-part series, we're exploring how to build a real-time voice assistant using an ESP32 microcontroller and the power of AI. In this first installment, we'll focus on setting up the hardware and configuring the ESP32 to handle audio input and output. Then, in part two, we'll dive deep into the C++ implementation, exploring buffer handling, speaker output, and integrating with PlatformIO to bring our voice assistant to life.

The Journey Starts

What was supposed to be a cool 2-day project turned into a 2-week struggle, filled with transistors and WebSocket nightmare. But the result is pretty exciting! Now I can talk with my AI Agent anytime, just by pressing a button.

Disclaimers and Considerations

Before we embark on this project, let's address a few important points:

Beginner's Perspective: As a newcomer to the world of C++, I welcome any suggestions and feedback to enhance the code and improve the overall implementation.
System Architecture: In this setup, the ESP32 serves as the hub for audio input and output, while a Node.js server handles the WebSocket communication. Although running the server directly on the ESP32 is an intriguing possibility, leveraging the familiarity of JavaScript/TypeScript for the server-side logic proved to be a more practical choice.
Audio Challenges: Brace yourself for a few audio-related hurdles along the way. Diagnosing whether issues stem from hardware mishaps (like faulty soldering) or software misconfigurations can be the trickiest part of the process.

Project Overview

This voice assistant seamlessly merges the realms of embedded systems and modern AI, creating a responsive and intuitive user experience. The key components that power this project include:

ESP32 Microcontroller: The brain of the operation, handling audio input/output and Wi-Fi connectivity.
WebSocket Communication: Enabling real-time data transfer between the ESP32 and the server.
I²S Protocol: Ensuring high-quality, synchronized audio data transfer between devices.
C++ Implementation: Optimizing performance and resource utilization on the ESP32.

Video

Source Code

The complete source code for this project is available on GitHub:
ESP32 AI Assistant Repository

Repository Structure

The project is organized into three main components:

1. ESP32 Firmware (`/esp32`)

Core firmware for audio capture and playback
WebSocket client implementation
Hardware configuration and I2S setup
Button handling and WiFi management

2. Audio Processing Server (`/server_audio`)

Node.js server handling WebSocket connections
Real-time audio streaming and buffering
OpenAI Whisper integration for speech-to-text
Text-to-speech conversion using OpenAI APIs

3. LangChain Server (`/server_langchain`)

TypeScript/Node.js server for AI processing
LangChain integration for natural language understanding
Custom tool system for extensible commands
OpenAI GPT integration for response generation

Each component plays a crucial role in the system:

The ESP32 handles all hardware interactions and audio I/O
The audio server manages speech processing and synthesis
The LangChain server provides the AI "brain" for understanding and responding

Required Materials

To bring this project to life, you'll need the following hardware components:

ESP32-S3 Development Board
I²S Digital Microphone (e.g., INMP441)
I²S Amplifier (e.g., MAX98357A)
Small Speaker (3W, 4Ω)
Push Button
Resistors (10kΩ for button pull-up)
Jumper Wires
Breadboard
Soldering Equipment

With these materials in hand, you're ready to embark on the hardware setup and dive into the ESP32 implementation. This should cost you less than $40.

Setting Up the Development Environment

Before we delve into the code, let's set up our development environment using PlatformIO, a powerful open-source ecosystem for IoT development.

Why PlatformIO?

Integrated IDE: Works seamlessly with VSCode, offering code completion, debugging, and more.
Library Management: Easily manage libraries and dependencies.
Board Support: Extensive support for various microcontrollers, including ESP32.

Installing PlatformIO

Install VSCode: If you haven't already, download and install Visual Studio Code.
Install PlatformIO Extension:
- Open VSCode.
- Go to Extensions (Ctrl+Shift+X).
- Search for "PlatformIO IDE" and install it.

Setting Up the ESP32 Project

Create a New Project:
- Open PlatformIO Home (Ctrl+Shift+P -> "PlatformIO: Home").
- Click on "New Project".
- Name: esp32-ai-assistant.
- Board: Select your ESP32 board model (e.g., ESP32 Dev Module).
- Framework: Arduino.
- Click "Finish".
Organize the Project Structure:
- In the src folder, add your .cpp files (main.cpp, mic.cpp, etc.).
- In the include folder, add your header files (config.h, mic.h, etc.).
Configure platformio.ini File:

   [env:esp32dev]
   platform = espressif32
   board = esp32dev
   framework = arduino
   monitor_speed = 115200
   build_flags = 
       -DCORE_DEBUG_LEVEL=5
       -DBOARD_HAS_PSRAM

Configure ESP32-WROOM Model:

You may get in some issues while running your code. For ESP32-WROOM boards. If you are using a similar S3 Wroom Board, you may want to use the config file in the bottom of the article. You can also try specific configuration to your platformio.ini:

   [env:esp32dev]
   platform = espressif32
   board = esp32dev
   framework = arduino
   monitor_speed = 115200
   board_build.partitions = huge_app.csv
   board_build.flash_mode = qio
   board_build.f_cpu = 240000000L
   board_build.f_flash = 80000000L
   build_flags = 
       -DCORE_DEBUG_LEVEL=5
       -DBOARD_HAS_PSRAM
       -mfix-esp32-psram-cache-issue

This configuration is optimized for ESP32-WROOM modules with:

Partition scheme for larger applications
QIO flash mode for better performance
CPU frequency set to 240MHz
Flash frequency set to 80MHz
PSRAM cache fix enabled

Download the board configuration file, and add it to your platform.io boards folder.

Understanding the C++ Implementation

Our ESP32 code handles multiple tasks:

Capturing audio from a microphone.
Sending audio data to a server via WebSockets.
Receiving processed audio data and playing it back through a speaker.
Managing buffer handling to ensure smooth data flow.
Handling user inputs via a button.

Let's break down each of these components.

Buffer Handling: From Start to Finish

Efficient buffer management is critical for real-time audio processing. We need to ensure that audio data is captured, processed, and transmitted without delays or overflows.

Microphone Handler (`mic.cpp`)

The micTask function handles audio data from the microphone.

void micTask(void *parameter) {
    while (true) {
        if (isRecording) {
            // Read audio data from the I2S microphone
            size_t bytesRead = 0;
            esp_err_t result = i2s_read(I2S_PORT_MIC, &soundBuffer, bufferLen, &bytesRead, portMAX_DELAY);

            if (result == ESP_OK && bytesRead > 0) {
                // Process the audio data
                detectSound(soundBuffer, bytesRead / sizeof(int16_t));

                // Transmit audio data over WebSocket
                if (isWebSocketConnected) {
                    sendBinaryData(soundBuffer, bytesRead);
                }
            }
        }
        vTaskDelay(pdMS_TO_TICKS(1)); // Yield to other tasks
    }
}

Key Points:

Non-Blocking Read: i2s_read with portMAX_DELAY ensures it waits for data without blocking other tasks.
Buffer Management: Uses soundBuffer to store audio samples.
Condition Checks: Ensures data is only processed and sent when recording is active and WebSocket is connected.

Buffer Allocation (`utils.cpp`)

Efficient memory allocation is essential for handling audio data.

void *audio_malloc(size_t size) {
    // Allocate memory in PSRAM if available, else use regular heap
    void *ptr = psramFound() ? heap_caps_malloc(size, MALLOC_CAP_SPIRAM) : malloc(size);
    if (!ptr) {
        Serial.println("Failed to allocate memory");
    }
    return ptr;
}

Highlights:

PSRAM Usage: ESP32 has limited RAM. Using external PSRAM helps manage larger buffers.
Error Handling: Checks if allocation is successful.

Audio Buffer Class (`audioMemoryBuffer.cpp`)

Manages the circular buffer for audio data.

class AudioMemoryBuffer {
public:
    AudioMemoryBuffer() {
        buffer = (int16_t*)malloc(BUFFER_SIZE * sizeof(int16_t));
        if (!buffer) {
            Serial.println("Failed to allocate audio buffer memory");
        }
        clear();
    }

    bool write(const int16_t* data, int length) {
        if (samplesAvailable + length > BUFFER_SIZE) {
            return false; // Buffer overflow
        }
        // Write data to buffer
        // ...
        samplesAvailable += length;
        return true;
    }

    bool read(int16_t* data, int length) {
        if (samplesAvailable < length) {
            return false; // Not enough data
        }
        // Read data from buffer
        // ...
        samplesAvailable -= length;
        return true;
    }

    void clear() {
        samplesAvailable = 0;
        memset(buffer, 0, BUFFER_SIZE * sizeof(int16_t));
    }

private:
    int16_t* buffer;
    int writeIndex = 0;
    int readIndex = 0;
    int samplesAvailable = 0;
};

Usage:

Writing Data: Called when new audio data is captured.
Reading Data: Used when sending data over WebSocket or playing back.
Circular Buffer: Efficiently manages continuous audio streams.

Speaker Output Handling (`lib_speaker.cpp`)

Manages playback of received audio data.

void speaker_play(uint8_t *payload, uint32_t len) {
    const float volume = 0.2f;
    size_t bytes_written;

    // Convert payload to samples
    int16_t *samples = (int16_t *)payload;
    size_t num_samples = len / sizeof(int16_t);

    // Adjust volume and play
    for (size_t i = 0; i < num_samples; i++) {
        samples[i] = (int16_t)(samples[i] * volume);
    }

    i2s_write(I2S_PORT_SPEAKER, samples, len, &bytes_written, portMAX_DELAY);
}

Key Elements:

Volume Control: Adjusts playback volume.
Playback: Uses i2s_write to send audio data to the speaker.
Buffer Management: Ensures data sent aligns with the speaker's capabilities.

Configuring I2S for Microphone and Speaker

We need to switch between microphone input and speaker output modes.

void InitI2SSpeakerOrMic(AudioMode mode) {
    // Uninstall existing driver
    i2s_driver_uninstall(mode == MODE_MIC ? I2S_PORT_MIC : I2S_PORT_SPEAKER);

    // Configure I2S based on mode
    i2s_config_t i2s_config = {/*...*/};

    // Install I2S driver
    esp_err_t err = i2s_driver_install(port, &i2s_config, 0, NULL);
    if (err != ESP_OK) {
        Serial.printf("Failed to install I2S driver: %s\n", esp_err_to_name(err));
        return;
    }

    // Set I2S pins
    i2s_pin_config_t pin_config = {/*...*/};
    err = i2s_set_pin(port, &pin_config);
    if (err != ESP_OK) {
        // Handle error
    }

    // Start I2S
    err = i2s_start(port);
    if (err != ESP_OK) {
        // Handle error
    }

    // Update state flags
    if (mode == MODE_MIC) {
        is_mic_installed = true;
        digitalWrite(LED_MIC, HIGH);
    } else {
        is_speaker_installed = true;
        digitalWrite(LED_SPKR, HIGH);
    }
}

Considerations:

Driver Uninstallation: Clean up before switching modes.
Pin Configuration: Ensures correct pins are used for each mode.
State Management: Keeps track of the current mode.

Main Program Flow (`main.cpp`)

The main.cpp orchestrates the initialization and main loop.

Initialization

void setup() {
    Serial.begin(115200);
    setupWiFi();
    connectToWebSocket();
    InitI2SSpeakerOrMic(MODE_MIC);
    xTaskCreatePinnedToCore(micTask, "Microphone Task", 4096, NULL, 1, NULL, 0);
    pinMode(BUTTON_PIN, INPUT_PULLUP);
}

Components:

Serial Communication: For debugging.
Wi-Fi Setup: Connects to network.
WebSocket Connection: Establishes communication with the server.
Microphone Task: Runs on Core 0 to handle audio input.
Button Setup: Configured for user input.

Main Loop

void loop() {
    button.loop();

    if (button.justPressed()) {
        sendButtonState(1);
        setRecording(true);
        i2s_start(I2S_PORT_MIC);
    }

    if (button.justReleased()) {
        sendButtonState(0);
        setRecording(false);
        i2s_stop(I2S_PORT_MIC);
        delay(100);
        i2s_start(I2S_PORT_SPEAKER);
    }

    loopWebsocket();
}

Key Actions:

Button Monitoring: Starts or stops recording based on button presses.
Audio State Management: Switches between microphone and speaker modes.
WebSocket Loop: Keeps the connection alive and handles incoming messages.
Important: I need to refactor this part as I'm not using the switch function, it caused me some I2S initialization issues.

Wi-Fi Management (`lib_wifi.cpp`)

Handles Wi-Fi connectivity.

void setupWiFi() {
    WiFi.disconnect();
    WiFi.mode(WIFI_STA);
    WiFi.begin(WIFI_SSID, WIFI_PASSWORD);

    while (WiFi.status() != WL_CONNECTED) {
        delay(1500);
    }

    Serial.println("WiFi connected");
}

Notes:

Connection Handling: Waits until a connection is established.
Status Updates: Provides feedback via serial output.

Important: Create a config.cpp file with the following :


#include "config.h"

const char* WIFI_SSID = "YOURWIFISSID";
const char* WIFI_PASSWORD = "YOURPASS";
const char* WEBSOCKET_HOST = "YOURWEBSOCKETIP";

Button Handling (`lib_button.cpp`)

Monitors button state changes.

class ButtonChecker {
public:
    void loop() {
        lastTickState = thisTickState;
        thisTickState = !digitalRead(BUTTON_PIN); // Active Low
    }

    bool justPressed() {
        return thisTickState && !lastTickState;
    }

    bool justReleased() {
        return !thisTickState && lastTickState;
    }

private:
    bool lastTickState = false;
    bool thisTickState = false;
};

Usage:

Debouncing: Manages state transitions to avoid false triggers.
Integration: Used in the main loop to respond to user inputs.

WebSocket Communication (`lib_websocket.cpp`)

Facilitates data exchange with the server.

void onMessageCallback(WebsocketsMessage message) {
    if (!message.isBinary()) {
        Serial.println("Received non-binary message: " + message.data());
        return;
    }

    uint8_t *payload = (uint8_t *)message.c_str();
    size_t length = message.length();

    if (length == 0) {
        Serial.println("Received empty audio data");
        return;
    }

    speaker_play(payload, length);
}

void connectToWebSocket() {
    client.onMessage(onMessageCallback);
    client.connect(WEBSOCKET_HOST);
}

Features:

Message Handling: Processes incoming messages and plays audio data.
Connection Management: Establishes and maintains WebSocket connections.

Building and Uploading the Project

With PlatformIO, building and flashing the code to the ESP32 is straightforward.

Step-by-Step Guide

Connect ESP32 to Your Computer:
- Use a USB cable to connect your ESP32.
Select the Correct Serial Port:
- In VSCode, click on the PlatformIO icon.
- Go to "Devices" and note the port (e.g., COM3, /dev/ttyUSB0).
Build the Project:
- Click on the checkmark (✓) in the status bar or run PlatformIO: Build from the command palette.
- Resolve any compilation errors.
Upload the Firmware:
- Click on the right arrow (→) in the status bar or run PlatformIO: Upload.
- Monitor the output for successful upload messages. You can also run Build & Monitor.
Monitor Serial Output:
- Use PlatformIO: Monitor to view serial logs.
- Adjust the baud rate if necessary (configured in platformio.ini).

Common Issues and Resolutions

Compilation Errors:
- Ensure all dependencies and libraries are included.
- Check for typos and syntax errors.
Upload Failures:
- Verify the correct port is selected.
- Press the boot button on the ESP32 during upload if necessary.
Runtime Errors:
- Use serial logs to debug.
- Ensure Wi-Fi credentials are correct in config.cpp.

Testing the Voice Assistant

Once the ESP32 is running, you can start interacting with your voice assistant.

Steps

Press the Button:
- This initiates recording.
Speak Clearly:
- The microphone captures your voice.
Release the Button:
- Recording stops, and the audio is sent to the server.
Listen to the Response:
- The processed response is played through the speaker.

Troubleshooting Tips

No Audio Playback:
- Check speaker connections.
- Ensure the ESP32 is receiving audio data.
Poor Audio Quality:
- Verify sample rates and buffering.
- Check for hardware noise or interference.
Connection Issues:
- Ensure the server is running and accessible.
- Verify network configurations.

Conclusion and Next Steps

By integrating the ESP32 with a robust C++ implementation and managing buffer handling efficiently, we've created a capable real-time voice assistant. This project not only demonstrates the potential of embedded systems but also provides a foundation for further enhancements.

Potential Improvements

Optimize Buffer Sizes: Adjust based on performance and memory usage.
Enhance Error Handling: Make the system more robust against failures.
Add Features: Integrate LEDs for status indication, add more interactive inputs.

Bridging to the AI Backend

With the hardware and firmware in place, the next phase is to connect our device to an AI backend. In the upcoming section, we'll explore how to integrate a Node.js server powered by LangChain and OpenAI to interpret voice commands and generate intelligent responses.

Stay tuned for the final part, where we bring true intelligence to our voice assistant!

Ready for Part 2?

Continue to Part 2: From Hardware to Smartware : Node, OpenAI, Langchain, where we'll explore how to:

Build our Websocket server using TypeScript and connect our ESP32
Integrate LangChain ReAct agent
Connect OpenAI's Realtime APIs
Handle real-time audio streaming with WebSockets
Give to your assistant the total control of your computer (in our 3rd Episode)

If you found this article helpful or have questions, feel free to leave a comment below.

Additional Resources

PlatformIO Documentation: https://docs.platformio.org/
ESP32 Wroom Board Config: https://github.com/sivar2311/freenove-esp32-s3-platformio/blob/main/freenove_esp32_s3_wroom.json
ThatProject: A huge repo of ESP32 project https://github.com/0015/ThatProject

Community Projects

For those interested in similar projects, check out these related implementations:

Open Interpreter ESP32 Client

The Open Interpreter ESP32 Client provides another approach to building voice assistants with ESP32. It's a much more complete solution, and I'm actually on my way to see how I can integrate it.

Exploring alternative implementations like this can provide valuable insights for improving your own projects.

How to Make a VS Code Extension Using TypeScript: A Step-by-Step Guide

Fabrikapp — Sat, 27 Apr 2024 15:52:42 +0000

Visual Studio Code (VS Code) has become one of the most popular code editors among developers, thanks to its powerful features, customizability, and extensive extension ecosystem. Creating your own VS Code extension allows you to enhance the coding experience by adding new functionalities, integrating with other tools, or tailoring the editor to your specific needs. In this step-by-step guide, we'll walk you through the process of building a VS Code extension using TypeScript.

Why TypeScript for VS Code Extensions?

TypeScript, a typed superset of JavaScript, is an excellent choice for developing VS Code extensions. It offers several advantages:

Static typing: TypeScript's static typing helps catch errors early in the development process, improving code quality and maintainability.
Enhanced tooling: VS Code provides excellent support for TypeScript, with features like IntelliSense, code navigation, and refactoring.
Improved readability: TypeScript's type annotations make the code more self-documenting and easier to understand.
Compatibility: TypeScript compiles down to JavaScript, ensuring compatibility with the VS Code extension API.

Prerequisites

Before you start building your extension, make sure you have the following installed:

Node.js (version 10 or higher)
Git
TypeScript compiler: Install globally using npm install -g typescript
Yeoman and VS Code Extension Generator: Install globally using npm install -g yo generator-code

Step 1: Scaffold Your Extension Project

To quickly set up the structure for your extension, use the Yeoman VS Code Extension Generator:

Open a terminal and run yo code.
Select "New Extension (TypeScript)" from the list of options.
Follow the prompts to provide a name, identifier, and description for your extension.
Choose "TypeScript" as the language.

The generator will create a new directory with the necessary files and folders for your extension.

Step 2: Understand the Extension Structure

Navigate to the generated extension directory and familiarize yourself with its structure:

package.json: The manifest file containing metadata about your extension, such as its name, version, and contribution points (commands, keybindings, etc.).
src/extension.ts: The main TypeScript file where you'll write the logic for your extension.
tsconfig.json: The TypeScript compiler configuration file specifying how the TypeScript code should be compiled to JavaScript.

Step 3: Write Your Extension Code

Open the src/extension.ts file and locate the activate function. This function is called when your extension is activated in VS Code. Here, you can register commands, subscribe to events, and implement the desired functionality.

Let's create a simple "Hello World" command as an example:

import * as vscode from 'vscode';

export function activate(context: vscode.ExtensionContext) {
  let disposable = vscode.commands.registerCommand('extension.helloWorld', () => {
    vscode.window.showInformationMessage('Hello World from my VS Code extension!');
  });

  context.subscriptions.push(disposable);
}

In this code snippet, we register a command with the identifier extension.helloWorld. When the command is triggered, it displays an information message with the text "Hello World from my VS Code extension!".

Step 4: Debug and Test Your Extension

VS Code provides built-in debugging support for extensions. To test your extension:

Open the Run view in VS Code by clicking on the Run icon in the Activity Bar or pressing Ctrl+Shift+D (Windows/Linux) or Cmd+Shift+D (macOS).
Select "Run Extension" from the dropdown menu.
VS Code will compile your TypeScript code and launch a new Extension Development Host window.
In the Extension Development Host window, open the Command Palette (Ctrl+Shift+P or Cmd+Shift+P) and type the name of your command (e.g., "Hello World").
Select the command to execute it and see the information message displayed.

You can set breakpoints in your TypeScript code to debug and inspect variables using VS Code's debugging tools.

Step 5: Package and Publish Your Extension

Once you're satisfied with your extension, it's time to package and publish it to the Visual Studio Marketplace:

Install the vsce command-line tool globally: npm install -g vsce.
Open a terminal in your extension directory and run vsce package. This command will generate a .vsix file.
Create a publisher account on the Visual Studio Marketplace.
Upload the .vsix file to the Marketplace and provide the necessary information about your extension.

Congratulations! Your extension is now available for others to install and use.

Best Practices

To ensure a high-quality extension, consider the following best practices:

Use a linter: Employ a linter like ESLint with TypeScript support to maintain consistent code style and catch potential issues.
Modularize your code: Organize your extension code into separate modules or files for better maintainability and readability.
Leverage asynchronous APIs: Use asynchronous APIs provided by VS Code to avoid blocking the editor's UI thread.
Test thoroughly: Write unit tests for your extension's functionality using testing frameworks like Mocha or Jest.
Provide clear documentation: Include a README.md file with installation instructions, usage examples, and a changelog.

Conclusion

Building a VS Code extension using TypeScript is a fantastic way to customize and extend the capabilities of your favorite code editor. By following this step-by-step guide, you've learned how to set up your development environment, create an extension project, implement functionality, debug and test your extension, and publish it to the Visual Studio Marketplace.

Remember to keep your extension focused, maintainable, and well-documented. Engage with the VS Code community, gather feedback, and continuously improve your extension based on user needs.

Happy coding, and may your extension empower developers to be more productive and efficient!

How to Create Powerful Customer Personas in 2024

Fabrikapp — Tue, 23 Apr 2024 23:06:36 +0000

If you are an Indie Maker, or a Saas Builder, you know that you can't skip the marketing stuff. But you're skiping it aren't you ?... (Well, then at least read this article, and reach the end of this article for a surprise !)

And those 34 visitors last week, and no paying users, didn't bring you the cash flow for the nomad lifestyle you bought.

But fear not ! Sales is a game of volume. But of targeted volume. Find your niche, find your clients, and you'll make money.

The only reason why you can fail, is if you go red-headed into building a solution for months before caring about your users needs.

But yes, we are developers and we love to code.

So how to make your marketing time and spending just as efficient as possible, so that you can spend more time building ?

You will ruin your efforts and bankrupt if you are spending money on the wrong horse ! And you will also go bankrupt if you don't invest in marketing.

So, let's jump in !

Understanding your potential customers is the key to driving business growth. One of the most effective ways to truly know your audience is by developing detailed customer personas.

A customer persona is a semi-fictional representation of your ideal customer based on real data and research. Personas help you understand your customers better - their behaviors, motivations, goals, and challenges. With this valuable insight, you can tailor your products, messaging, and overall experience to meet their specific needs.

Creating personas is an essential exercise for businesses of all sizes, products, and industries. Whether you're a small startup or a global enterprise, having well-crafted personas will fuel more resonant marketing, smoother user experiences, and ultimately, better sales.

In this guide, we'll walk through the full process for developing rich, insightful customer personas that will transform how you approach your business in 2024 and beyond.

Why Are Customer Personas So Valuable?

Before we dive into the steps of persona creation, let's discuss why this exercise is so worthwhile:

Personas humanize your customers. Rather than looking at people as numbers and data points, personas help you truly empathize with real people's wants, needs, and pain points. This human-centric view shapes better experiences.
They provide focus and alignment. Personas give your whole team a clear, unified vision of who you're designing for. This focus prevents going too broad and trying to be everything to everyone.
Personas fuel more relevant marketing. With detailed personas, you can craft messaging that speaks directly to your audience's interests, challenges, and motivations. No more one-size-fits-all content.
They improve user experiences. Understanding how different user personas behave and what they value allows you to design smoother product experiences tailored to their needs.
Personas minimize assumptions. Rather than relying on hunches, personas are grounded in real research about your customers' lives. This insight replaces guesswork.

Spending 3 hours detailing your Persona is just what you need !
(Source : me)

The Key Steps of Creating Customer Personas

Now that we understand the "why" behind personas, let's dig into the actual process of developing them. Here are the essential steps:

Step 1: Conduct audience research

The first step is gathering as much data as possible about your target customers through quantitative and qualitative research methods like:

User surveys and interviews
Website analytics and heatmaps
Customer service logs and inquiries
Social media listening
Reviewing reviews and testimonials
Studying industry reports and market research

During this phase, you'll want to ask exploratory questions to uncover key persona details like:

Demographics

Age, gender, location, income level, job, education
Family status, living situation

Psychographics

Goals, motivations, and aspirations
Values, interests, and personality traits
Attitudes, lifestyles, and hobbies

Behaviors

Typical day and responsibilities
Media and information sources used
Buying process and objections
Challenges, pain points, and needs

Product Experience

How they discover and use your product/service
Desired features and use cases
Reasons for purchasing or churning

The more comprehensive your research, the richer and more insightful your eventual personas will be. Don't make assumptions - let the data tell their real stories.

Step 2: Identify your major persona groups

Once you've gathered a wealth of audience data, look for patterns and commonalities that allow you to segment your customers into distinct persona groups.

Some common ways to categorize personas include:

Demographics (age, location, income level, etc.)
Roles and responsibilities (job titles, industries, etc.)
Goals and motivations (what they want to achieve)
Skill levels and experience (beginner, advanced user, etc.)
Attitudes and behaviors (pragmatic, spontaneous, etc.)

Most businesses will identify between 3-7 key customer personas that represent their major audience segments. Any more than that and your personas become too fragmented to be truly useful.

Step 3: Flesh out your persona details

With your main persona groups defined, it's now time to build out comprehensive profiles for each one. Treat these personas as real people by giving them names, faces, identities, and backstories.

Here are some of the key details to capture for each persona:

Persona Name and Photo: Choose a representative name and stock photo to make the persona feel realistic.
Demographics: Age, gender, location, job, income level, education, family details.
Identifiers: A few quotes or descriptive phrases that capture their voice and personality.
Goals and Motivations: Their main personal and professional goals, as well as core motivations.
Values and Interests: Their beliefs, attitudes, hobbies, media habits, and general lifestyle.
Challenges and Pain Points: The problems, frustrations, and obstacles they're facing.
Buying Process: How they discover, evaluate, and make purchasing decisions for your product/service.
Product Needs: The key features and use cases they require to achieve their goals.
Sources of Information: Where they go for recommendations, news, and industry information.
Objections and Concerns: Potential roadblocks that could prevent them from buying.

The more specific and narrative you can make each persona, the easier it will be to truly understand them as real people. Don't just list out dry facts - craft a story around who they are.

To illustrate what a comprehensive persona looks like, let's walk through a fictional example:

Persona Name: Health-Conscious Hannah

*Photo: *
A smiling woman in her late 20s/early 30s doing yoga
Identifiers: "I try to live as naturally as possible." "My wellness routine is really important to me."

Background:
Hannah is a 29-year-old wellness blogger and social media influencer living in Los Angeles. She's passionate about holistic health, fitness, and sustainable living.

After struggling with an autoimmune condition in her early 20s, Hannah adopted a clean, plant-based diet and committed to an active lifestyle. She now feels healthier than ever and wants to inspire others to live their best lives too.

Hannah has a bachelor's degree in communications and previously worked in marketing before leaving to build her personal brand full-time. She lives with her husband and rescue dog in a trendy LA neighborhood.

Demographics:

Age: 29
Location: Los Angeles, CA
Job: Wellness Blogger / Influencer
Income: $85,000 / year
Education: Bachelor's Degree
Relationship: Married

Goals & Motivations:

Grow her blog and social following to become an influential voice in the wellness space
Maintain a healthy, balanced lifestyle aligned with her natural living values
Inspire and empower others to take control of their wellbeing
Earn enough income from her blog and partnerships to work for herself

Challenges & Pain Points:

Difficulty managing her time and staying organized with content creation
Feeling overwhelmed by her many projects, commitments, and responsibilities
Skepticism from some audiences about her lack of professional health certifications
Concerns about burnout and maintaining a sustainable work/life balance

Values & Interests:

Passionate about holistic health, fitness, nutrition, and sustainability
Loves being outdoors, doing activities like yoga, hiking, and gardening
Interested in personal growth, mindfulness, and spiritual practices
Enjoys cooking, traveling, and spending time with loved ones

Buying Process:
Hannah is heavily influenced by recommendations from other influencers and trusted sources she follows. She's willing to invest in premium products and services aligned with her wellness values.

Before making purchases, Hannah extensively researches online by reading reviews, watching videos, and joining discussions in communities she's a part of. She's wary of"greenwashing" marketing and values authenticity and transparency.

Product Needs:

Healthy food and beverage products that are organic, plant-based, and sustainably sourced
Fitness, yoga, and meditation apps with high-quality content and instructors she resonates with
Productivity and content creation tools to streamline her blogging workflow
Natural beauty, personal care, and home products in line with her natural living ethos

Information Sources:

Follows other wellness influencers, bloggers, and thought leaders on Instagram and YouTube
Subscribes to health and wellness podcasts and newsletters to stay informed about the latest trends and research.
Participates in online forums and Facebook groups dedicated to wellness and natural living.
Attends wellness events, workshops, and retreats to connect with like-minded individuals and brands.

Objections & Concerns:

Wary of products that make unsubstantiated health claims or use misleading marketing tactics.
Concerned about the environmental impact of products and seeks out brands with strong sustainability practices.
Prefers to support companies that are transparent about their ingredients and sourcing.

With this detailed persona example, you can see how every aspect of Hannah's life influences her buying decisions and interactions with brands. By understanding her deeply, a company can tailor its marketing efforts, product development, and customer service to meet her specific needs and preferences.

Step 4: Validate and refine your personas

After creating your initial persona drafts, it's crucial to validate them with real users. Share your personas with members of your target audience and gather feedback. Are the personas accurate and relatable? Do they miss any critical details? Use this feedback to refine and adjust your personas until they truly resonate with your audience.

Step 5: Share and apply your personas across your organization

Once your personas are finalized, it's important to share them with your entire team. Ensure that everyone from marketing to product development to customer service understands who they are serving. Incorporate persona insights into every aspect of your business strategy, from content creation to UX design to sales approaches.

Best Practices for Effective Persona Creation

To ensure your personas are as effective as possible, follow these best practices:

Base your personas on real data, not assumptions. The more research you do, the more accurate your personas will be.
Keep your personas focused and avoid unnecessary details that don't impact decision-making.
Use a consistent template for all your personas to make them easy to understand and compare.
Avoid stereotypes and make your personas as realistic as possible.
Regularly review and update your personas as your audience and market evolve.

Recommended Tools and Templates for Persona Creation

Several tools and templates can help streamline the persona creation process:

PersonaBuilder.co : The most complete Persona Builder tool. You just need a short description, or a link to your website to generate your persona.
Hubspot's Make My Persona tool: An interactive online tool that guides you through the persona creation process.
Xtensio user persona templates: A platform offering customizable persona templates that you can adapt to your needs.
Hotjar's persona building guide: A comprehensive guide with tips and best practices for creating data-driven personas.

So, ready to stop coding and start marketing ?

Creating detailed customer personas is a powerful way to deepen your understanding of your audience and tailor your business strategies to meet their needs. By following the steps outlined in this guide and using the recommended tools and templates, you can develop personas that will drive more effective marketing, product development, and customer experiences.

Remember, personas are not static; they should evolve as your business and market change. Keep researching, refining, and applying your personas, and you'll be well on your way to building stronger connections with your customers in 2024 and beyond.

Update April 2024 :

We created a Tool and get a complete persona profile adapted to your product in minutes !

Check our the Persona Creation tool : PersonaBuilder.co

Prisma vs. Drizzle: A Comprehensive Guide for Your NextJS Project

Fabrikapp — Tue, 23 Apr 2024 10:31:08 +0000

When embarking on a new NextJS project, one of the critical decisions you'll face is choosing the right Object-Relational Mapping (ORM) tool. Two popular choices in the TypeScript ecosystem are Prisma and Drizzle. Both are designed to facilitate database operations, but they differ significantly in their approach, philosophy, and features. In this article, we'll explore the differences between Prisma and Drizzle, and provide guidance on which one might be the best fit for your project. Additionally, we'll invent custom examples to illustrate how each ORM could be used in a practical scenario.

Understanding Prisma and Drizzle

Prisma is an ORM that aims to solve many of the problems associated with traditional ORMs, such as bloated model instances, mixing business logic with storage logic, and unpredictable queries due to lazy loading. It uses a Prisma schema to define application models declaratively and generates SQL migrations from this schema. Prisma Client, a type-safe database client for Node.js and TypeScript, provides CRUD operations.

Drizzle, on the other hand, is a traditional SQL query builder that allows you to compose SQL queries using JavaScript/TypeScript functions. It offers a Queries API for a higher level of abstraction from SQL, enabling you to read nested relations. Drizzle's schema is defined in TypeScript files, which are then used to generate and execute SQL migrations against a database.

API Design and Level of Abstraction

Drizzle's philosophy is "If you know SQL, you know Drizzle ORM." It mirrors SQL in its API, providing a familiar environment for those well-versed in SQL. Prisma Client offers a higher-level abstraction designed with application developers in mind, making common tasks more accessible without requiring deep SQL knowledge.

Data Modeling

Prisma models are defined in the Prisma schema, while Drizzle uses TypeScript functions for table definitions. Prisma's data modeling DSL is lean and intuitive, with a powerful VS Code extension that enhances developer productivity. Drizzle's use of TypeScript offers additional flexibility and the power of code reuse.

Migrations

Both Drizzle and Prisma ORM handle migrations similarly, generating SQL files based on model definitions and providing a CLI to execute them. The SQL files can be modified before execution, allowing custom database operations with either system.

Querying

Query construction is natural in both Drizzle and Prisma ORM. Drizzle's Queries API and Prisma's querying methods are similar, but Drizzle requires the use of its SQL-like APIs for mutations, while Prisma provides a more fluent API for creating, updating, or deleting records.

Relations

Prisma ORM simplifies working with foreign key-connected records through virtual relation fields, offering a fluent API, nested writes, filters on related records, and type-safe querying of nested data. Drizzle requires more complex SQL construction to achieve similar results.

Filtering

Drizzle exposes SQL dialect-specific filter and conditional operators, while Prisma ORM provides a more generic and intuitive set of operators, such as contains, startsWith, and endsWith.

Pagination

Drizzle offers limit-offset pagination, whereas Prisma ORM provides APIs for both limit-offset and cursor-based pagination, making it more versatile.

Observability

Both ORMs allow query logging and SQL generation logging. Prisma ORM has additional features like metrics and tracing, which can be integrated with external tools for performance tracking.

Additional Products and Ecosystem

Prisma and Drizzle offer additional products alongside their ORMs. Prisma Studio and Drizzle Studio allow users to interact with their databases via a GUI. Prisma also offers commercial products like Prisma Accelerate and Prisma Pulse, which integrate with Prisma ORM for comprehensive data tooling.

Prisma ORM is well-established and included as the data layer tool of choice in many meta-frameworks and development platforms. Its community has developed a plethora of useful tools to assist with various workflows.

Database Support

Both ORMs support multiple databases, with Drizzle achieving this through driver implementations and Prisma primarily using built-in drivers. Prisma also supports databases like CockroachDB, Microsoft SQL Server, and MongoDB, which Drizzle does not currently support.

Custom Example: Blogging Platform

Let's consider a blogging platform where users can create, update, and delete blog posts. We'll see how Prisma and Drizzle can be used to handle this scenario.

Prisma Example

// Define models in Prisma schema
model User {
  id    Int     @id @default(autoincrement())
  name  String?
  email String  @unique
  posts Post[]
}

model Post {
  id        Int     @id @default(autoincrement())
  title     String
  content   String?
  published Boolean @default(false)
  authorId  Int?
  author    User?   @relation(fields: [authorId], references: [id])
}

// Create a new post
const newPost = await prisma.post.create({
  data: {
    title: 'My First Post',
    content: 'This is the content of my first post.',
    published: true,
    authorId: 1,
  },
});

// Update a post
const updatedPost = await prisma.post.update({
  where: { id: 1 },
  data: { title: 'Updated Title' },
});

// Delete a post
const deletedPost = await prisma.post.delete({
  where: { id: 1 },
});

Drizzle Example

import { pgTable, text, boolean, integer, serial } from 'drizzle-orm/pg-core';

// Define tables using TypeScript functions
export const users = pgTable('users', {
  id: serial('id').primaryKey(),
  name: text('name'),
  email: text('email').unique(),
});

export const posts = pgTable('posts', {
  id: serial('id').primaryKey(),
  title: text('title').notNull(),
  content: text('content'),
  published: boolean('published'),
  authorId: integer('author_id').references(() => users.id),
});

// Create a new post
const newPost = await db.insert(posts).values({
  title: 'My First Post',
  content: 'This is the content of my first post.',
  published: true,
  authorId: 1,
});

// Update a post
const updatedPost = await db
  .update(posts)
  .set({ title: 'Updated Title' })
  .where(eq(posts.id, 1))
  .returning();

// Delete a post
const deletedPost = await db
  .delete(posts)
  .where(eq(posts.id, 1))
  .returning();

Conclusion

Choosing between Prisma and Drizzle for your NextJS project depends on your team's familiarity with SQL, the level of abstraction you prefer, and the specific features you need. Prisma offers a more abstracted, developer-friendly approach, while Drizzle provides a closer-to-SQL experience with type safety. Both have their strengths and can be effectively used to build robust applications.

For teams that use SQL daily, Drizzle's familiar SQL-like syntax may be more appealing. For teams with varying levels of database experience, Prisma's comprehensive and easy-to-learn approach may be more suitable. Ultimately, the decision should be based on your project requirements, team expertise, and the trade-offs you are willing to make.

So, what's your ORM of choice ?

How to Use Typed Routes in Next.js

Fabrikapp — Mon, 22 Apr 2024 10:06:37 +0000

Next.js is a powerful React framework that simplifies the creation of web applications by providing a set of conventions and built-in functionalities. One of the key features of Next.js is its routing system, which allows developers to create pages and navigate between them with ease. With the introduction of typed routes, Next.js enhances the developer experience by providing type safety for routing, especially when using TypeScript.

Understanding Typed Routes in Next.js

Typed routes in Next.js are an experimental feature that provides statically typed links within your application. This feature is part of the Next.js App Router and requires TypeScript to be used in your project. The main advantage of typed routes is that they help ensure that the links within your application are correct at compile time, reducing the likelihood of runtime errors due to incorrect URLs or missing query parameters.

To enable typed routes in your Next.js project, you need to set the typedRoutes option to true in your next.config.js file. Here's an example configuration:

// next.config.js
const nextConfig = {
  experimental: {
    typedRoutes: true,
  },
};

module.exports = nextConfig;

This feature is still in beta and currently does not support rewrites and redirects. However, it is a significant step towards improving type safety in Next.js applications.

Using `nextjs-routes` for Typed Routing

While the built-in typed routes feature in Next.js is still experimental, there is a third-party library called nextjs-routes that provides type-safe routing for both the app directory and pages. This library scans your pages directory and generates route types based on your application's routes, ensuring that your links are type-safe with zero runtime overhead.

Installation and Setup

To use nextjs-routes, you need to install the package and update your next.config.js file as follows:

npm install nextjs-routes
# or
yarn add nextjs-routes
# or
pnpm add nextjs-routes

Then, update your next.config.js:

// next.config.js
const nextRoutes = require("nextjs-routes/config");
const withRoutes = nextRoutes();

const nextConfig = {
  reactStrictMode: true,
};

module.exports = withRoutes(nextConfig);

After running npx next dev or npx next build, a @types/nextjs-routes.d.ts file will be generated, which augments the type definitions for next/link and next/router.

Examples of Typed Routing with `nextjs-routes`

Here are some examples of how you can use typed routing with nextjs-routes:

Link Component

import Link from "next/link";

// Typed href based on your application routes
<Link
  href={{
    pathname: "/posts/[postId]",
    query: { postId: "123" },
  }}
>
  Read Post
</Link>;

// For routes without parameters, you can use a string
<Link href="/about">About Us</Link>;

useRouter Hook

import { useRouter } from "next/router";

const router = useRouter();

// Typed push method
router.push({
  pathname: "/posts/[postId]",
  query: { postId: "456" },
});

// Typed replace method
router.replace("/");

// Typed query object
const { query } = useRouter();
// query is typed as a union of all query parameters defined by your application's routes

getServerSideProps

import type { GetServerSidePropsContext } from "nextjs-routes";

export function getServerSideProps(
  context: GetServerSidePropsContext<"/posts/[postId]">
) {
  // context.params will include `postId` as a string
  const { postId } = context.params;
}

Custom Example: Typed Blog Post Routes

Let's create a custom example to illustrate how typed routes can be used in a practical scenario. Imagine we have a blog application with dynamic routes for individual blog posts. We want to ensure that our links to these posts are type-safe.

First, we define our blog post page under the pages/posts/[postId].tsx file. Then, we use nextjs-routes to generate the typed routes.

// pages/posts/[postId].tsx
import { useRouter } from "next/router";
import Link from "next/link";

const PostPage = () => {
  const router = useRouter();
  const { postId } = router.query as { postId: string };

  return (
    <div>
      <h1>Post {postId}</h1>
      <Link href="/posts">
        <a>Back to posts</a>
      </Link>
    </div>
  );
};

export default PostPage;

Now, when we link to a post, we can use the typed href to ensure that we pass the correct postId:

// pages/posts/index.tsx
import Link from "next/link";

const PostsList = ({ posts }) => {
  return (
    <ul>
      {posts.map((post) => (
        <li key={post.id}>
          <Link
            href={{
              pathname: "/posts/[postId]",
              query: { postId: post.id },
            }}
          >
            {post.title}
          </Link>
        </li>
      ))}
    </ul>
  );
};

export default PostsList;

With nextjs-routes, if we accidentally mistype the route or forget to pass the required postId, TypeScript will alert us at compile time, preventing potential runtime errors.

Conclusion

Typed routes in Next.js provide a robust way to ensure that your application's routing is type-safe, reducing the risk of errors and improving the developer experience. While the built-in typed routes feature in Next.js is still experimental, the nextjs-routes library offers a stable solution that works with both the app directory and pages. By following the examples provided, you can implement typed routes in your Next.js projects and enjoy the benefits of type safety in your routing logic.

Remember to keep an eye on the Next.js documentation and release notes for updates on the typed routes feature as it moves towards stability. Happy coding!

The Evolution of Next.js: A Deep Dive into the Latest Updates Since Version 14.2

Fabrikapp — Mon, 22 Apr 2024 09:12:37 +0000

Next.js, the popular React framework, has been on a continuous journey of improvement and innovation. With the release of version 14.2, the Next.js team has introduced a slew of enhancements that cater to both developers and end-users. In this article, we will explore the latest updates since version 14.2, providing insights into the new features, performance improvements, and the overall impact on the development experience.

Turbopack for Development (Release Candidate)

One of the most significant updates in Next.js 14.2 is the introduction of Turbopack for local development. As a release candidate, Turbopack has shown promising results, with 99.8% of integration tests passing and compatibility with the top 300 npm packages used in Next.js applications. The integration of Lightning CSS, a fast CSS bundler and minifier written in Rust, has also contributed to the performance boost.

Developers can now experience:

Up to 76.7% faster local server startup.
Up to 96.3% faster code updates with Fast Refresh.
Up to 45.8% faster initial route compile without caching.

To try out Turbopack, developers can use the command next dev --turbo. The focus now shifts to improving memory usage, implementing persistent caching, and optimizing the next build --turbo command.

Build and Production Improvements

Next.js 14.2 has brought forward optimizations that benefit all Next.js applications, regardless of whether they use Pages or App Router. These improvements include:

Tree-shaking

An optimization has been identified that allows for tree-shaking unused exports, significantly reducing production JavaScript bundle sizes. For instance, importing a single Icon component from a file with "use client" no longer includes all other icons from that package. This optimization led to a 51.3% reduction in the final bundle size for a popular library like react-aria-components.

Build Memory Usage

For large-scale Next.js applications, the team addressed out-of-memory crashes during production builds by refactoring the bundling logic and optimizing the compiler. Early tests show a decrease in memory usage and cache file size from 2.2GB to under 190MB on average.

CSS Optimization

The production build process for CSS has been updated to chunk CSS and avoid conflicting styles when navigating between pages. The order and merging of CSS chunks are now defined by the import order, ensuring the correct CSS order is maintained.

Caching Improvements

Caching is crucial for fast and reliable web applications. Next.js 14.2 introduces staleTimes, an experimental option that allows developers to configure the client-side router cache invalidation period. This feature aims to give developers more control over caching heuristics.

Error DX Improvements

The readability of error messages and stack traces has been improved in development mode. The error overlay design has been updated to support light and dark modes, and dev and build logs have been made more visually appealing.

React 19 Preparation

In anticipation of React 19, the Next.js team is working on integrating the latest features and improvements. New features like Actions and related hooks will be available for all React applications.

Other Notable Improvements

New documentation on Video Optimization and instrumentation.ts.
New overrideSrc prop for next/image.
New revalidateReason argument to getStaticProps.
Refactored streaming logic for faster page streaming in production.
Support for nested Server Actions and localization in generated Sitemaps.
Visual improvements to dev and build logs.
Skew protection is now stable on Vercel.

Community and Contributions

Next.js has surpassed 1 million monthly active developers, thanks to the support and contributions from the community. The framework is a result of the combined efforts of individual developers, industry partners, and the core team at Vercel.

Custom Example: Implementing a Dynamic Icon Component with Tree-shaking

To illustrate the power of the new tree-shaking optimization, let's create a dynamic Icon component that imports only the necessary icons from a library.

// icons.tsx
export const HomeIcon = () => <svg>...</svg>;
export const SearchIcon = () => <svg>...</svg>;
export const ProfileIcon = () => <svg>...</svg>;
// Use "use client" to enable tree-shaking
export const icons = { HomeIcon, SearchIcon, ProfileIcon };

// IconComponent.tsx
import { icons } from './icons';

type IconComponentProps = {
  name: keyof typeof icons;
};

export const IconComponent: React.FC<IconComponentProps> = ({ name }) => {
  const Icon = icons[name];
  return <Icon />;
};

// Usage in a page
import { IconComponent } from './IconComponent';

const Page = () => (
  <div>
    <IconComponent name="HomeIcon" />
    {/* Only the HomeIcon will be included in the bundle */}
  </div>
);

By using the IconComponent, we ensure that only the HomeIcon is included in the bundle, thanks to the tree-shaking optimization introduced in Next.js 14.2.

CSS Handling and Optimization

Updates to CSS optimization during production builds ensure smoother transitions between pages:

Enhanced Chunking: CSS chunking now prevents conflicting styles when navigating between pages.
Streamlined Order: Import hierarchy now determines the order and merging of CSS chunks, maintaining consistency in styling.

Custom Example: CSS Chunking and Order

To maintain the correct CSS order, you should use CSS Modules and import them in a single JS/TS file. For example:

/* base-button.module.css */
.primary {
  background-color: blue;
}

/* page.module.css */
.primary {
  background-color: red;
}

// base-button.js
import styles from './base-button.module.css';

export function BaseButton() {
  return <button className={styles.primary}>Base Button</button>;
}

// page.js
import { BaseButton } from './base-button';
import pageStyles from './page.module.css';

export function Page() {
  return <div className={pageStyles.primary}><BaseButton /></div>;
}

In this setup, the BaseButton component's styles will take precedence over the Page component's styles due to the import order.

Enhanced Caching Mechanisms

The caching strategies have been substantially enhanced with the addition of the staleTimes configuration for the Client-side Router Cache. This new feature empowers developers to define the duration for which a cached page remains valid before being marked as stale.

Custom Example: staleTimes Configuration

You can configure the cache invalidation periods in next.config.js:

// next.config.js
module.exports = {
  experimental: {
    staleTimes: {
      dynamic: 30, // seconds
      static: 180, // seconds
    },
  },
};

With this configuration, dynamic routes will be cached for 30 seconds, and static routes for 180 seconds before revalidation.

Developer Experience Improvements

In Next.js 14.2, there's a strong emphasis on improving the developer experience through enhancements to error messages and debugging tools. The introduction of a new error overlay that accommodates both light and dark themes simplifies issue troubleshooting.

Custom Example: Error DX Improvements

When encountering a React Hydration error, the new error overlay will provide clearer information, including the file name where the error occurred, making it easier to pinpoint and resolve the issue.

Conclusion

Next.js 14.2 has set a new standard for the framework, with a focus on performance, developer experience, and future-proofing for upcoming React releases. The updates discussed in this article are just the beginning, as the Next.js team continues to innovate and push the boundaries of what's possible in web development.

Stay tuned for more updates and improvements as Next.js evolves to meet the needs of developers and users alike. Whether you're upgrading your existing Next.js application or starting a new project, these enhancements are sure to make your development journey smoother and more efficient.

How to Implement a LangGraph (LangChainJS) in TypeScript (in 5 minutes)

Fabrikapp — Mon, 22 Apr 2024 00:41:21 +0000

The ability to create stateful, multi-actor applications is crucial for developing complex systems that can handle a variety of tasks. LANGCHAIN LangGraph is a library designed for this purpose, allowing developers to build applications with Large Language Models (LLMs) by modeling steps as edges and nodes in a graph. In this article, we will explore how to implement a LANGCHAIN LangGraph in TypeScript, complete with code examples and a custom tool that we will invent.

Understanding LangGraph

LangGraph is a part of the LangChain framework, which is a set of tools for building applications powered by LLMs. LangGraph extends the LangChain Expression Language with the ability to coordinate multiple chains or actors across multiple steps of computation in a cyclic manner. It is inspired by Pregel and Apache Beam and is not optimized for only Directed Acyclic Graph (DAG) workflows. Instead, it is designed for adding cycles to your LLM application, which is essential for agent-like behaviors.

Installation

To get started with LangGraph in TypeScript, you need to install the LangGraph package using npm:

npm install @langchain/langgraph

Additionally, you should install the LangChain OpenAI integration package:

npm i @langchain/openai

Make sure to export your OpenAI API key as an environment variable:

export OPENAI_API_KEY=sk-...

Creating a Simple LangGraph

Let's create a simple LangGraph that contains a single node called "oracle" that executes a chat model and returns the result. Here's how you can do it:

import { ChatOpenAI } from "@langchain/openai";
import { HumanMessage, BaseMessage } from "@langchain/core/messages";
import { END, MessageGraph } from "@langchain/langgraph";

const model = new ChatOpenAI({ temperature: 0 });

const graph = new MessageGraph();

graph.addNode("oracle", async (state: BaseMessage[]) => {
  return model.invoke(state);
});

graph.addEdge("oracle", END);

graph.setEntryPoint("oracle");

const runnable = graph.compile();

// For Message graph, input should always be a message or list of messages.
const res = await runnable.invoke(new HumanMessage("What is 1 + 1?"));

The code above initializes a model and a MessageGraph, adds a single node that calls the model with the given input, and compiles the graph. When executed, the graph adds the input message to the internal state, passes the state to the "oracle" node, and outputs the final state after execution.

Adding Conditional Edges

LangGraph also allows for conditional edges, which can route execution to a node based on the current state using a function. Here's an example of adding a "calculator" node that conditionally executes based on the model's output:

import { ToolMessage } from "@langchain/core/messages";
import { Calculator } from "langchain/tools/calculator";
import { convertToOpenAITool } from "@langchain/core/utils/function_calling";

const model = new ChatOpenAI({
  temperature: 0,
}).bind({
  tools: [convertToOpenAITool(new Calculator())],
  tool_choice: "auto",
});

const graph = new MessageGraph();

graph.addNode("oracle", async (state: BaseMessage[]) => {
  return model.invoke(state);
});

graph.addNode("calculator", async (state: BaseMessage[]) => {
  const tool = new Calculator();
  const toolCalls = state[state.length - 1].additional_kwargs.tool_calls ?? [];
  const calculatorCall = toolCalls.find(
    (toolCall) => toolCall.function.name === "calculator"
  );
  if (calculatorCall === undefined) {
    throw new Error("No calculator input found.");
  }
  const result = await tool.invoke(
    JSON.parse(calculatorCall.function.arguments)
  );
  return new ToolMessage({
    tool_call_id: calculatorCall.id,
    content: result,
  });
});

graph.addEdge("calculator", END);

const router = (state: BaseMessage[]) => {
  const toolCalls = state[state.length - 1].additional_kwargs.tool_calls ?? [];
  if (toolCalls.length) {
    return "calculator";
  } else {
    return "end";
  }
};

graph.addConditionalEdges("oracle", router, {
  calculator: "calculator",
  end: END,
});

const runnable = graph.compile();

In the example above, if the "oracle" node returns a message expecting a tool call, the "calculator" node is executed. Otherwise, execution ends.

Creating a Custom Tool

Now, let's invent a custom tool that we can integrate into our LangGraph. We'll create a simple "SentimentAnalyzer" tool that takes a text input and returns a sentiment score.

import { BaseTool, ToolMessage } from "@langchain/core/messages";

class SentimentAnalyzer extends BaseTool {
  async invoke(input: string): Promise<ToolMessage> {
    // For simplicity, we'll return a fixed sentiment score.
    // In a real-world scenario, you would integrate with a sentiment analysis API.
    const sentimentScore = 0.8; // Let's assume a positive sentiment
    return new ToolMessage({
      content: `Sentiment score: ${sentimentScore}`,
    });
  }
}

// Now, let's bind the SentimentAnalyzer to our model as a tool.
const sentimentAnalyzer = new SentimentAnalyzer();
const modelWithSentiment = model.bind({
  tools: [convertToOpenAITool(sentimentAnalyzer)],
  tool_choice: "auto",
});

// You can now add the SentimentAnalyzer node to your graph and use it as needed.

Conclusion

LangGraph is a powerful library for building stateful, multi-actor applications with LLMs. By following the steps outlined in this article, you can implement a LangGraph in TypeScript, add conditional edges, and even create custom tools to extend the functionality of your application. With LangGraph, you can create complex AI systems that can handle a wide range of tasks with agent-like behaviors.

Remember to explore the LangChain documentation and GitHub repository for more examples and detailed information on how to use LangGraph and other components of the LangChain framework. Happy coding!

References

LangGraph Documentation: LangGraph | 🦜️ LangChain
LangChain GitHub Repository: langchain-ai/langchain

(Note: The code examples provided in this article are based on the documentation and may require adjustments based on the actual LangChain and LangGraph library versions and API changes.)

Building a Production-Ready Web App with T3 Stack

Fabrikapp — Fri, 19 Apr 2024 22:32:22 +0000

Building a Production-Ready Web App with React: A Comprehensive Tutorial

In this in-depth tutorial, we will walk through the process of building a fully-featured, production-ready web application using modern React development practices and libraries. By the end of this course, you will have a strong understanding of how to architect and implement a scalable, performant, and maintainable React application.

We will cover a wide range of topics, including:

Setting up a development environment with TypeScript, Tailwind CSS, and the T3 stack
Implementing user authentication and authorization using Clerk
Managing application state with libraries like Zustand
Handling data fetching and mutations with React Query and Prisma
Optimizing performance with Next.js features like server-side rendering and incremental static regeneration
Deploying the application to Vercel
Monitoring errors with Sentry
Collecting analytics with PostHog

Throughout the tutorial, we will work on building a real-world application - an image gallery that allows users to upload, view, and manage their photos. We'll implement features incrementally, focusing on best practices and common pitfalls to avoid.

Prerequisites

This is an intermediate to advanced level tutorial. To get the most out of it, you should have a solid foundation in HTML, CSS, JavaScript, and React fundamentals. Familiarity with TypeScript and Next.js is helpful but not required.

Project Setup

Let's get started by scaffolding a new Next.js project using the create-t3-app tool. This will set us up with a bunch of useful libraries and conventions out of the box.

Open your terminal and run:

pnpm create t3-app@latest

You'll be prompted to choose a set of options for your project:

? What will your project be called? -> t3-gallery
? Will you be using TypeScript or JavaScript? -> TypeScript
? Which packages would you like to enable? -> Tailwind CSS
? Initialize a new git repository? -> Yes
? Would you like us to run 'npm install'? -> Yes, use pnpm

After the installation finishes, change into the new project directory:

cd t3-gallery

To make sure everything is working, start the development server:

pnpm dev

Then open http://localhost:3000 in your browser. You should see the default Next.js starter page.

Cleaning Up the Starter Code

The create-t3-app template comes with some placeholder content that we won't need for our image gallery app. Let's clean that up.

Replace the contents of src/pages/index.tsx with:

const Home = () => {
  return (
    <main className="flex min-h-screen flex-col items-center justify-center">
      <h1 className="text-4xl font-bold">Welcome to the T3 Gallery!</h1>
    </main>
  );
};

export default Home;

This gives us a simple homepage with a title. The className attributes are using Tailwind utility classes to style the elements.

Delete the src/pages/api directory since we won't be using API routes in this project.

Also delete the src/styles/globals.css file and remove its import from src/pages/_app.tsx, since we'll be styling everything with Tailwind.

Deploying to Vercel

Before we start building out the features of our app, let's deploy it to Vercel. This will let us easily share our progress and monitor the application in production.

First, push your code to a new GitHub repository (you can skip this if you initialized the repo through GitHub when creating the project):

git add .
git commit -m "Initial commit"
git branch -M main
git remote add origin https://github.com/yourusername/t3-gallery.git
git push -u origin main

Now go to https://vercel.com, sign up for an account, and click "New Project". Connect your GitHub account and give Vercel permission to access your repositories.

Select your t3-gallery repo and click "Import". On the next screen, leave all the default settings and click "Deploy". In a minute or two, your app will be live at a URL like https://t3-gallery.vercel.app.

Whenever you push changes to the main branch of your GitHub repo, Vercel will automatically redeploy your application. You can also preview changes from pull requests before merging.

Let's get coding !

In this section, we set up the initial codebase for our image gallery application and deployed it to production on Vercel.

We used the create-t3-app template to quickly bootstrap a Next.js project with TypeScript and Tailwind CSS preconfigured. Then we cleaned up some of the starter code to prepare for implementing our own features.

By deploying to Vercel from the beginning, we can easily share our progress, get feedback, and monitor the app's real-world performance as we continue to build it out.

In the next sections, we'll start adding core functionality like user authentication, image uploads, and data persistence. You'll see how to integrate various libraries and services to efficiently develop production-ready features.

The tutorial continues on from here to cover the core features outlined in the introduction. I can continue generating the content if you'd like. Let me know if you have any other specific instructions or topics you want me to focus on in the subsequent sections. I'm aiming to provide comprehensive, well-structured, and example-driven explanations suitable for an online programming course.

User Authentication with Clerk

Let's implement user registration and login using Clerk. Clerk provides a complete authentication solution with a prebuilt, customizable UI and secure backend logic. It supports various authentication methods like email/password, OAuth, and magic links.

First, sign up for a free account at https://clerk.com. Create a new application and make note of your "Frontend API" key.

Install the Clerk React SDK:

pnpm add @clerk/nextjs

Wrap your src/pages/_app.tsx component with the ClerkProvider:

import { ClerkProvider } from '@clerk/nextjs';

function MyApp({ Component, pageProps }) {
  return (
    <ClerkProvider frontendApi={process.env.NEXT_PUBLIC_CLERK_FRONTEND_API}>
      <Component {...pageProps} />
    </ClerkProvider>
  );
}

export default MyApp;

Add your Clerk frontend API key to .env.local:

NEXT_PUBLIC_CLERK_FRONTEND_API=your_api_key_here

Now add the Clerk <SignIn /> and <SignUp /> components to your homepage:

import { SignIn, SignUp } from "@clerk/nextjs";

const Home = () => {
  return (
    <main className="flex min-h-screen flex-col items-center justify-center">
      <h1 className="text-4xl font-bold">Welcome to the T3 Gallery!</h1>
      <div className="mt-8">
        <SignIn path="/sign-in" routing="path" signUpUrl="/sign-up" />
      </div>
      <div className="mt-8">
        <SignUp path="/sign-up" routing="path" signInUrl="/sign-in" />
      </div>
    </main>
  );
};

This displays the Clerk sign in and sign up forms on our homepage. The path and routing props specify that we want to use path-based routing for these pages. The signUpUrl and signInUrl props create links between the forms.

Start your development server and test out the login and registration flow. You should be able to create an account, log in, and log out.

To show different content based on the user's authentication status, use the useUser hook from @clerk/nextjs:

import { useUser } from "@clerk/nextjs";

const Home = () => {
  const { isLoaded, isSignedIn, user } = useUser();

  if (!isLoaded) {
    return null;
  }

  return (
    <main className="flex min-h-screen flex-col items-center justify-center">
      {isSignedIn ? (
        <>
          <h1 className="text-4xl font-bold">Welcome {user.firstName}!</h1>
          <button onClick={() => signOut()} className="mt-8 rounded-md bg-blue-500 py-2 px-4 font-bold text-white hover:bg-blue-600">
            Sign Out
          </button>
        </>
      ) : (
        <>
          <h1 className="text-4xl font-bold">Welcome to the T3 Gallery!</h1>
          <div className="mt-8">
            <SignIn path="/sign-in" routing="path" signUpUrl="/sign-up" />
          </div>
          <div className="mt-8">  
            <SignUp path="/sign-up" routing="path" signInUrl="/sign-in" />
          </div>
        </>
      )}
    </main>
  );
};

Here, we conditionally render a welcome message and sign out button if the user is signed in, or the sign in and sign up forms if they're signed out.

Clerk also provides a <UserButton /> component that displays the user's profile image and a dropdown menu with account management options:

import { UserButton } from "@clerk/nextjs";

const Home = () => {
  const { isLoaded, isSignedIn } = useUser();

  if (!isLoaded) {
    return null;
  }

  return (
    <main className="flex min-h-screen flex-col items-center justify-center">
      {isSignedIn ? (
        <>
          <UserButton />
          {/* ... */}
        </>
      ) : (
        {/* ... */}
      )}
    </main>  
  );
};

With just a few lines of code, we've added complete user authentication to our app! In the next section, we'll look at how to protect certain routes and use the user's information to personalize their experience.

Securing Pages and API Routes

By default, all pages in a Next.js app are publicly accessible. To limit access to authenticated users, we can use Clerk's withServerSideAuth and withAuthMiddleware helpers.

Create a new file src/pages/gallery/index.tsx:

import { withServerSideAuth } from "@clerk/nextjs/ssr";

export const getServerSideProps = withServerSideAuth(async ({ req }) => {
  const { userId } = req.auth;

  // Load the user's images based on their userId
  const images = await db.image.findMany({
    where: { userId },
  });

  return { props: { images } };
});

const GalleryPage = ({ images }) => {
  return (
    <div>
      <h1>My Gallery</h1>
      {/* Display the user's images */}
    </div>
  );
};

export default GalleryPage;

The withServerSideAuth helper verifies the user's authentication status and attaches their userId to the request object. If the user is not signed in, they will automatically be redirected to the sign in page.

We can access the userId in getServerSideProps to fetch data specific to that user, like their uploaded images in this example.

For API routes, use the withMiddlewareAuthRequired helper:

import { withMiddlewareAuthRequired } from "@clerk/nextjs/server";

const handler = async (req, res) => {
  const { userId } = req.auth;

  switch (req.method) {
    case "POST":
      // Create a new image for this user
      break;
    case "PUT":
      // Update one of the user's images
      break;  
    case "DELETE":
      // Delete one of the user's images
      break;
    default:
      res.status(405).json({ message: "Method not allowed" });
  }
};

export default withMiddlewareAuthRequired(handler);

The withMiddlewareAuthRequired helper attaches the userId to the req.auth object, allowing you to perform user-specific actions in your API route handlers. If the request is not authenticated, it will return a 401 Unauthorized response.

By using these helpers throughout our app, we can easily implement granular, user-based access control to pages, API routes, and data.

Managing Application State

As our app grows in complexity, we'll need a way to manage state that's shared across multiple components. We could use React's built-in state and prop drilling, but that quickly becomes cumbersome and hard to maintain.

Instead, we'll use Zustand, a lightweight state management library. It allows us to create a centralized store and update it from anywhere in our component tree.

Install Zustand:

pnpm add zustand

Create a new file src/stores/useGalleryStore.ts:

import { create } from "zustand";

type Image = {
  id: string;
  url: string;
  // ... other fields
};

type GalleryState = {
  images: Image[];
  selectedImageId: string | null;
  setImages: (images: Image[]) => void;
  setSelectedImageId: (id: string | null) => void;
};

export const useGalleryStore = create<GalleryState>((set) => ({
  images: [],
  selectedImageId: null,
  setImages: (images) => set({ images }),
  setSelectedImageId: (id) => set({ selectedImageId: id }),
}));

Here we define the shape of our gallery's state with the GalleryState type. It includes an array of images, the selectedImageId for the currently focused image, and setter functions to update those values.

The useGalleryStore hook is created by passing an initial state object to Zustand's create function.

Now we can import and use this hook in any component:

import { useGalleryStore } from "../stores/useGalleryStore";

const GalleryPage = ({ initialImages }) => {
  const images = useGalleryStore((state) => state.images);
  const setImages = useGalleryStore((state) => state.setImages);

  useEffect(() => {
    setImages(initialImages);
  }, [initialImages, setImages]);

  return (
    <div>
      {images.map((image) => (
        <img key={image.id} src={image.url} alt={image.name} />
      ))}
    </div>  
  );
};

We use the useGalleryStore hook to access the images array and setImages function from our global state. The component receives initialImages from getServerSideProps, which we use to populate the store on mount via useEffect.

Zustand automatically re-renders components that use the hook whenever the state values they depend on are updated. This lets us efficiently sync the UI with our global state.

Some other cool features of Zustand:

Computed state values with memoization
Transient updates (state changes that don't trigger re-renders)
Async actions with Thunk-like syntax
Persist state to local/session storage
TypeScript support
Tiny bundle size (< 1kb gzipped)

Data Fetching and Mutations with React Query

For data that's loaded from or saved to an external API, we'll use React Query. It provides a powerful set of hooks for fetching, caching, synchronizing, and updating server state in our React app.

Install React Query and the Prisma Client:

pnpm add @tanstack/react-query @prisma/client

Initialize Prisma with a PostgreSQL database:

pnpm prisma init

This command creates a new Prisma schema file and .env for your database connection URL.

Update the Prisma schema in prisma/schema.prisma to define your app's data models. For our image gallery app, it might look something like:

datasource db {
  provider = "postgresql"
  url      = env("DATABASE_URL")
}

generator client {
  provider        = "prisma-client-js" 
  previewFeatures = ["referentialIntegrity"]
}

model User {
  id       String    @id @default(cuid())
  email    String    @unique
  name     String?
  images   Image[]
}

model Image {
  id       String  @id @default(cuid())
  url      String
  creator  User    @relation(fields: [userId], references: [id])
  userId   String
}

Here we define two models: User and Image. A user can have multiple images, and each image is associated with a user via the userId foreign key and @relation attribute.

After defining your schema, create the database tables with:

pnpm prisma migrate dev --name init

Next, instantiate the Prisma Client in src/server/db/client.ts:

import { PrismaClient } from "@prisma/client";

export const prisma = new PrismaClient();

Now we can define our React Query hooks for loading and mutating data. Create a new file src/utils/useImages.ts:

import { useQuery, useMutation, useQueryClient } from "@tanstack/react-query";
import { prisma } from "../server/db/client";

const IMAGES_QUERY_KEY = "images";

export const useImages = (userId: string) => {
  return useQuery([IMAGES_QUERY_KEY, userId], async () => {
    const images = await prisma.image.findMany({
      where: { userId },
    });
    return images;
  });
};

export const useAddImage = () => {
  const queryClient = useQueryClient();

  return useMutation(
    async (newImage: { url: string; userId: string }) => {
      const addedImage = await prisma.image.create({
        data: newImage,
      });
      return addedImage;
    },
    {
      onSuccess: () => {
        queryClient.invalidateQueries([IMAGES_QUERY_KEY]);
      },
    }
  );
};

export const useDeleteImage = () => {
  const queryClient = useQueryClient();

  return useMutation(
    async (imageId: string) => {
      await prisma.image.delete({ where: { id: imageId } }); 
    },
    {
      onSuccess: () => {
        queryClient.invalidateQueries([IMAGES_QUERY_KEY]);
      },  
    }
  );
};

Here we define three hooks:

useImages - A query hook that loads a user's images from the database
useAddImage - A mutation hook for adding a new image
useDeleteImage - A mutation hook for deleting an image

The query hook specifies a unique key that identifies the query (IMAGES_QUERY_KEY + userId). It uses Prisma Client to load the images from the database.

The mutation hooks define two async functions: one to make the actual mutation (create/delete an image), and an onSuccess callback that invalidates the IMAGES_QUERY_KEY. This tells React Query to refetch the images on the next render, keeping the cache in sync with our database.

To use these hooks in a component:

const GalleryPage = () => {
  const { userId } = useUser();
  const { data: images, isLoading } = useImages(userId);
  const addImage = useAddImage();
  const deleteImage = useDeleteImage();

  if (isLoading) {
    return <div>Loading...</div>;
  }

  return (
    <div>
      {images.map((image) => (
        <div key={image.id}>
          <img src={image.url} alt={image.id} />
          <button onClick={() => deleteImage.mutate(image.id)}>Delete</button>
        </div>
      ))}
      <button onClick={() => addImage.mutate({ url: "https://example.com/new-image.jpg", userId })}>
        Add Image
      </button>
    </div>
  );
};

Here we use the useUser hook from Clerk to get the current user's ID. We pass that into the useImages hook to load the user's images. The isLoading flag lets us show a loading state while the query is fetching.

We destructure the addImage and deleteImage mutation functions from their respective hooks. These are used in the button onClick handlers to add and delete images.

React Query automatically handles loading/error states, caching, refetching on window focus, and more. It abstracts away all the boilerplate of CRUD-ing server state, letting you focus on your app's business logic.

Optimizing Performance with Next.js

Next.js offers several features to improve the loading speed and responsiveness of your app out of the box. Let's look at how to leverage them effectively.

Incremental Static Regeneration (ISR)

ISR allows you to update existing pages by re-rendering them in the background as traffic comes in. Inspired by stale-while-revalidate, this ensures traffic is served static pages quickly, while new pages are being rendered.

To enable ISR, use the revalidate option in your getStaticProps function:

export const getStaticProps: GetStaticProps = async (context) => {
  const images = await prisma.image.findMany();

  return {
    props: {
      images,
    },
    revalidate: 60, // Regenerate the page every 60 seconds
  };
};

Now this page will be statically generated at build time, but also regenerated in the background every 60 seconds as traffic comes in. This keeps your page speed fast while ensuring content is never stale for too long.

Dynamic Imports and Lazy Loading

To reduce your JavaScript bundle size and speed up loading, you can dynamically import components that aren't needed immediately on page load. This is especially useful for large or complex components like modals, charts, or rich texteditors.

Use Next's dynamic function to lazily load a component:

import dynamic from "next/dynamic";

const ImageModal = dynamic(() => import("../components/ImageModal"));

const GalleryPage = () => {
  const [isModalOpen, setIsModalOpen] = useState(false);

  return (
    <div>
      {/* ... */}
      {isModalOpen && <ImageModal onClose={() => setIsModalOpen(false)} />}
    </div>
  );
};

The ImageModal component will only be loaded when the modal is opened, reducing the initial bundle size.

You can also use next/image to automatically optimize images and lazily load them as they enter the viewport:

import Image from "next/image";

const GalleryImage = ({ image }) => {
  return (
    <Image
      src={image.url}
      alt={image.id}
      width={500}
      height={500}
      loading="lazy"
    />
  );
};

The loading="lazy" prop defers loading the image until it's scrolled into view. Next will also automatically resize, optimize, and serve the image in modern formats like WebP.

Measuring Performance

To identify performance issues and opportunities for optimization, use the React Profiler and Chrome DevTools Performance tab to record and analyze rendering.

Wrap your component tree in a Profiler component to measure rendering performance:

import { Profiler } from "react";

function onRenderCallback(
  id,
  phase,
  actualDuration,
  baseDuration,
  startTime,
  commitTime
) {
  console.log(`${id} render took ${actualDuration}ms`);
}

const App = () => {
  return (
    <Profiler id="App" onRender={onRenderCallback}>
      {/* ... */}
    </Profiler>
  );
};

The onRender callback logs render durations to the console, helping you spot slow components. You can also use the React DevTools Profiler to visually explore this data.

To go deeper, use the Chrome DevTools Performance tab to record and analyze runtime performance. This lets you see CPU usage, network requests, and more over time.

By leveraging ISR, lazy loading, and performance profiling, you can ensure your Next.js app stays fast and responsive as it grows in complexity.

Monitoring Errors with Sentry

To track and debug errors in production, we'll integrate Sentry into our app. Sentry is an error tracking platform that captures exceptions, logs, and performance data to help you identify and fix issues.

First, sign up for a free account at https://sentry.io. Create a new project and make note of your DSN (Data Source Name).

Install the Sentry SDK:

pnpm add @sentry/nextjs

Configure Sentry in your next.config.js file:

const { withSentryConfig } = require("@sentry/nextjs");

const moduleExports = {
  // Your existing Next.js config
};

const sentryWebpackPluginOptions = {
  silent: true,
};

module.exports = withSentryConfig(moduleExports, sentryWebpackPluginOptions);

Initialize Sentry in src/pages/_app.tsx:

import { init } from "@sentry/nextjs";

init({
  dsn: process.env.SENTRY_DSN,
});

function MyApp({ Component, pageProps }) {
  return (
    <ClerkProvider frontendApi={process.env.NEXT_PUBLIC_CLERK_FRONTEND_API}>
      <Component {...pageProps} />
    </ClerkProvider>
  );
} 

export default MyApp;

Add your Sentry DSN to .env.local:

SENTRY_DSN=your_dsn_here

With this setup, Sentry will automatically capture unhandled exceptions in your app. You can also manually capture errors and add context:

import * as Sentry from "@sentry/nextjs";

const MyComponent = () => {
  const { user } = useUser();

  function handleClick() {
    try {
      // Do something that might throw an error
    } catch (error) {
      Sentry.captureException(error, {
        extra: {
          userId: user.id,
        },
      });
    }
  }

  return (
    <button onClick={handleClick}>
      Click me
    </button>
  );
};

The captureException function sends the error to Sentry along with additional context like the current user ID. This makes it easier to reproduce and debug the issue.

Sentry also provides performance monitoring, release tracking, and issue management features to give you better visibility into your app's health and user experience.

By proactively identifying and fixing bugs with Sentry, you can provide a more stable and reliable application to your users.

Collecting Analytics with PostHog

To understand how users interact with your app, it's crucial to collect analytics events. This lets you track key metrics, identify bottlenecks in your conversion funnels, and make data-driven decisions about what features to build next.

We'll use PostHog, an open-source product analytics platform, to capture and analyze events in our app.

Sign up for a free PostHog account at https://app.posthog.com. Create a new project and make note of your Project API Key.

Install the PostHog client library:

pnpm add posthog-js

Initialize PostHog in src/pages/_app.tsx:

import posthog from "posthog-js";

posthog.init(process.env.NEXT_PUBLIC_POSTHOG_KEY, {
  api_host: "https://app.posthog.com",
});

function MyApp({ Component, pageProps }) {
  return (
    <ClerkProvider frontendApi={process.env.NEXT_PUBLIC_CLERK_FRONTEND_API}>
      <Component {...pageProps} />
    </ClerkProvider>
  );
}

export default MyApp;

Add your PostHog project API key to .env.local:

NEXT_PUBLIC_POSTHOG_KEY=your_api_key_here

Now you can capture events anywhere in your app:

const ImageUpload = () => {
  const [isUploading, setIsUploading] = useState(false);
  const addImage = useAddImage();

  async function handleSubmit(event) {
    event.preventDefault();
    setIsUploading(true);

    posthog.capture("Image Uploaded", {
      userId: user.id,
    });

    await addImage.mutate(event.target.image.value);
    setIsUploading(false);
  }

  return (
    <form onSubmit={handleSubmit}>
      <input type="file" name="image" />
      <button type="submit" disabled={isUploading}>
        {isUploading ? "Uploading..." : "Upload"}
      </button>
    </form>
  );
};

Here we capture an "Image Uploaded" event with the posthog.capture function. We include the user ID as an event property to let us analyze events per user in PostHog.

Some other common events you might want to track:

Page views
Sign up / sign in
Search queries
Add to cart / checkout
Feature usage

You can also use the posthog.identify function to attach user properties like email or name to the user ID. This lets you segment your analytics based on user attributes.

PostHog provides an insights dashboard where you can explore your event data, create funnels, and set up retention charts. By understanding how users flow through your app, you can optimize your UX and maximize conversion and retention.

PostHog also supports session recording, feature flagging, A/B testing, and self-hosting, making it a powerful and flexible alternative to SaaS analytics platforms.

Conclusion

In this comprehensive tutorial, we walked through the process of building a production-ready image gallery application with React, TypeScript, Next.js, and the T3 stack.

We covered a wide range of topics, including:

User authentication with Clerk
Global state management with Zustand
Data fetching and mutations with React Query
Optimizing performance with ISR, lazy loading, and profiling
Tracking errors with Sentry
Collecting analytics with PostHog

By leveraging these technologies and following best practices, you can build modern, full-stack React apps efficiently and focus on delivering value to your users.

Some key takeaways:

Use a pre-built authentication solution like Clerk to save time and ensure security
Manage global UI state with a lightweight library like Zustand
Fetch and cache server state with React Query for optimal performance and developer experience
Lazy load non-critical components and use ISR to speed up pages
Track errors in production with Sentry to proactively fix bugs
Collect analytics with PostHog to understand user behavior and make data-driven decisions

I hope this tutorial has given you a solid foundation for building your own production React apps. Remember to keep learning, experiment with new libraries and techniques, and always prioritize user experience and developer productivity.

Happy coding!

Mastering Next.js 13/14 - Advanced Techniques

Fabrikapp — Fri, 19 Apr 2024 00:09:24 +0000

Deployment and optimization are essential aspects of any production-ready application. We'll explore different deployment options for Next.js applications, such as Vercel, AWS, and Google Cloud. You'll learn how to optimize your application's performance using techniques like static site generation, incremental static regeneration, and other performance optimization strategies.

Throughout the course, we'll also touch upon advanced topics like custom server configuration, authentication strategies, internationalization (i18n), and testing Next.js applications. These topics will help you extend your application's functionality and ensure its reliability and maintainability.

7. Deployment and Optimization

Next.js provides built-in support for deployment and optimization, making it easy to deploy your application to production and ensure optimal performance.

Deploying Next.js Applications

Next.js applications can be deployed to various platforms, including Vercel, AWS, Google Cloud, and more. The deployment process typically involves building your application and starting the Next.js server.

Here's an example of deploying to Vercel:

Sign up for a Vercel account and install the Vercel CLI.
Navigate to your Next.js project directory.
Run the following command to deploy your application:

vercel

Vercel will automatically detect your Next.js application, build it, and deploy it to a unique URL.

Static Site Generation (SSG)

Next.js supports Static Site Generation (SSG), which allows you to generate static HTML files at build time. This improves performance by serving pre-rendered pages to users.

To enable SSG for a page, you need to export a getStaticProps function that fetches the necessary data and returns it as props.

Here's an example:

// pages/blog/[slug].js
export async function getStaticProps({ params }) {
  const post = await getPostBySlug(params.slug);
  return {
    props: {
      post,
    },
  };
}

export async function getStaticPaths() {
  const posts = await getAllPosts();
  const paths = posts.map((post) => ({
    params: { slug: post.slug },
  }));

  return {
    paths,
    fallback: false,
  };
}

export default function BlogPostPage({ post }) {
  return (
    <div>
      <h1>{post.title}</h1>
      <p>{post.content}</p>
    </div>
  );
}

In this example, getStaticProps fetches a blog post based on the slug parameter, and getStaticPaths generates the list of paths for all blog posts. The generated static files are served to users, providing fast loading times.

Incremental Static Regeneration (ISR)

Incremental Static Regeneration (ISR) is a feature in Next.js that allows you to update static pages without rebuilding the entire application. It enables you to regenerate specific pages on-demand or at a specified interval.

To enable ISR for a page, you need to add the revalidate property to the getStaticProps return object.

Here's an example:

// pages/blog/[slug].js
export async function getStaticProps({ params }) {
  const post = await getPostBySlug(params.slug);
  return {
    props: {
      post,
    },
    revalidate: 60, // Regenerate the page every 60 seconds
  };
}

In this example, the revalidate property is set to 60, indicating that the page should be regenerated every 60 seconds. If a request comes in after the specified interval, Next.js will regenerate the page in the background and serve the updated content.

Performance Optimization Techniques

Next.js provides several performance optimization techniques out of the box, such as automatic code splitting, server-side rendering, and asset optimization. However, you can further optimize your application's performance using the following techniques:

Lazy loading: Use dynamic imports to lazy load components or modules that are not immediately required, reducing the initial bundle size.
Image optimization: Optimize images by using the next/image component, which automatically resizes and optimizes images based on the device and viewport size.
Caching: Implement caching strategies for API routes and static assets to reduce the load on the server and improve response times.
Compression: Enable compression for server responses to reduce the size of the transferred data.
CDN: Use a Content Delivery Network (CDN) to serve static assets from geographically distributed servers, reducing latency and improving load times.

By applying these optimization techniques, you can ensure that your Next.js application performs well and provides a smooth user experience.

8. Advanced Topics

Next.js offers several advanced features and concepts that allow you to customize and extend your application's functionality. Let's explore a few of these topics.

Custom Server Configuration

By default, Next.js uses its built-in server to handle requests and serve pages. However, you can customize the server configuration to add additional functionality or integrate with other server frameworks.

To customize the server, you need to create a server.js file in the root of your project and export a custom server instance.

Here's an example using Express.js:

// server.js
const express = require('express');
const next = require('next');

const dev = process.env.NODE_ENV !== 'production';
const app = next({ dev });
const handle = app.getRequestHandler();

app.prepare().then(() => {
  const server = express();

  // Custom server logic goes here

  server.all('*', (req, res) => {
    return handle(req, res);
  });

  server.listen(3000, (err) => {
    if (err) throw err;
    console.log('> Ready on http://localhost:3000');
  });
});

In this example, we create an Express.js server and integrate it with Next.js. You can add custom server logic, such as handling API routes or implementing middleware, before passing the request to Next.js for rendering.

Authentication Strategies

Next.js provides flexibility in implementing authentication strategies. You can choose from various authentication providers like Auth0, Firebase Authentication, or build your own authentication system.

Here's an example using Next.js with Firebase Authentication:

// pages/api/login.js
import { getAuth, signInWithEmailAndPassword } from 'firebase/auth';
import { initializeApp } from 'firebase/app';

const firebaseConfig = {
  // Your Firebase configuration
};

const app = initializeApp(firebaseConfig);
const auth = getAuth(app);

export default async function handler(req, res) {
  if (req.method === 'POST') {
    const { email, password } = req.body;

    try {
      const userCredential = await signInWithEmailAndPassword(auth, email, password);
      const user = userCredential.user;
      // Generate and send a token or session data
      res.status(200).json({ user });
    } catch (error) {
      res.status(401).json({ message: 'Invalid credentials' });
    }
  } else {
    res.status(405).json({ message: 'Method not allowed' });
  }
}

In this example, we use the Firebase Authentication SDK to handle user login. When a POST request is made to the /api/login route with the user's email and password, we attempt to sign in the user using Firebase Authentication. If successful, we can generate and send a token or session data to the client for subsequent authenticated requests.

Internationalization (i18n)

Next.js provides built-in support for internationalization (i18n) to create multilingual applications. You can define translations for different languages and switch between them based on the user's locale.

Here's an example of setting up i18n in Next.js:

Create a next.config.js file in the root of your project and add the following configuration:

// next.config.js
module.exports = {
  i18n: {
    locales: ['en', 'fr', 'es'],
    defaultLocale: 'en',
  },
};

In this example, we define the supported locales (en, fr, es) and set the default locale to en.

Create translation files for each locale in the public/locales directory:

public/
  locales/
    en.json
    fr.json
    es.json

Each translation file contains key-value pairs representing the translations for that locale.

Use the useRouter hook to access the current locale and the t function from the next-i18next library to translate texts:

// pages/index.js
import { useRouter } from 'next/router';
import { useTranslation } from 'next-i18next';

export default function Home() {
  const router = useRouter();
  const { t } = useTranslation();

  return (
    <div>
      <h1>{t('title')}</h1>
      <p>{t('description')}</p>
      <button onClick={() => router.push('/', '/', { locale: 'fr' })}>
        {t('switch_to_french')}
      </button>
    </div>
  );
}

In this example, we use the useRouter hook to access the current locale and the t function to translate texts based on the keys defined in the translation files. We also provide a button to switch the locale to French.

Next.js will automatically detect the user's locale based on the URL or the browser's language settings and load the corresponding translation file.

Testing Next.js Applications

Testing is an essential part of building robust and maintainable applications. Next.js integrates well with popular testing frameworks like Jest and React Testing Library.

Here's an example of writing a test for a Next.js component:

// components/Button.js
export default function Button({ label, onClick }) {
  return <button onClick={onClick}>{label}</button>;
}

// tests/Button.test.js
import { render, screen, fireEvent } from '@testing-library/react';
import Button from '../components/Button';

describe('Button', () => {
  it('renders the button with the correct label', () => {
    render(<Button label="Click me" />);
    const buttonElement = screen.getByText('Click me');
    expect(buttonElement).toBeInTheDocument();
  });

  it('calls the onClick handler when clicked', () => {
    const handleClick = jest.fn();
    render(<Button label="Click me" onClick={handleClick} />);
    const buttonElement = screen.getByText('Click me');
    fireEvent(buttonElement);
    expect(handleClick).toHaveBeenCalledTimes(1);
  });
});

In this example, we use Jest and React Testing Library to test the Button component. We write two test cases:

It renders the button with the correct label.
It calls the onClick handler when clicked.

We use the render function from React Testing Library to render the component, and the screen object to query the rendered elements. We then make assertions using Jest's expect statements.

By writing tests for your Next.js components and pages, you can ensure that they behave as expected and catch any regressions during development.

9. Real-World Projects

To reinforce your understanding of Next.js and apply the concepts you've learned, let's explore a few real-world project examples.

Building a Blog Application

A blog is a common use case for Next.js. You can create a blog application with the following features:

Display a list of blog posts on the homepage.
Allow users to view individual blog posts.
Implement a search functionality to find specific posts.
Add pagination or infinite scrolling for better user experience.
Integrate a content management system (CMS) like Sanity or Contentful to manage blog content.

Here's a basic example of a blog homepage component:

// pages/index.js
import Link from 'next/link';

export async function getStaticProps() {
  const posts = await fetchPosts();
  return {
    props: {
      posts,
    },
  };
}

export default function Home({ posts }) {
  return (
    <div>
      <h1>My Blog</h1>
      <ul>
        {posts.map((post) => (
          <li key={post.id}>
            <Link href={`/posts/${post.slug}`}>
              <a>{post.title}</a>
            </Link>
          </li>
        ))}
      </ul>
    </div>
  );
}

In this example, we use getStaticProps to fetch the list of blog posts at build time. We then render the list of posts on the homepage, with each post title linking to its individual page.

Creating an E-commerce Store

Next.js is well-suited for building e-commerce applications. You can create an online store with features like:

Display products with images, descriptions, and prices.
Implement a shopping cart functionality.
Allow users to add and remove products from the cart.
Integrate a payment gateway for secure transactions.
Implement user authentication and order tracking.

Here's a basic example of a product detail page component:

// pages/products/[slug].js
import { useRouter } from 'next/router';
import Image from 'next/image';

export async function getStaticProps({ params }) {
  const product = await fetchProductBySlug(params.slug);
  return {
    props: {
      product,
    },
  };
}

export async function getStaticPaths() {
  const products = await fetchProducts();
  const paths = products.map((product) => ({
    params: { slug: product.slug },
  }));

  return {
    paths,
    fallback: false,
  };
}

export default function ProductDetailPage({ product }) {
  const router = useRouter();

  const handleAddToCart = () => {
    // Logic to add the product to the cart
    router.push('/cart');
  };

  return (
    <div>
      <h1>{product.name}</h1>
      <Image src={product.image} alt={product.name} width={500} height={500} />
      <p>{product.description}</p>
      <p>Price: ${product.price}</p>
      <button onClick={handleAddToCart}>Add to Cart</button>
    </div>
  );
}

In this example, we use getStaticProps and getStaticPaths to pre-render the product detail pages at build time. We display the product information, including the image, description, and price. We also provide an "Add to Cart" button that adds the product to the shopping cart and redirects the user to the cart page.

Developing a Social Media Dashboard

Next.js can be used to build interactive dashboards for social media platforms. You can create a dashboard with features like:

Display user profile information and statistics.
Show real-time data updates for posts, likes, comments, and followers.
Implement data visualization using charts and graphs.
Allow users to create and schedule new posts.
Integrate with social media APIs to fetch and update data.

Here's a basic example of a social media dashboard component:

// pages/dashboard.js
import { useEffect, useState } from 'react';
import Chart from 'chart.js';

export default function Dashboard() {
  const [userData, setUserData] = useState(null);

  useEffect(() => {
    const fetchUserData = async () => {
      const response = await fetch('/api/user-data');
      const data = await response.json();
      setUserData(data);
    };

    fetchUserData();
  }, []);

  useEffect(() => {
    if (userData) {
      const ctx = document.getElementById('followerChart').getContext('2d');
      new Chart(ctx, {
        type: 'line',
        data: {
          labels: userData.followerHistory.map((entry) => entry.date),
          datasets: [
            {
              label: 'Followers',
              data: userData.followerHistory.map((entry) => entry.count),
              backgroundColor: 'rgba(75, 192, 192, 0.2)',
              borderColor: 'rgba(75, 192, 192, 1)',
              borderWidth: 1,
            },
          ],
        },
        options: {
          scales: {
            y: {
              beginAtZero: true,
            },
          },
        },
      });
    }
  }, [userData]);

  if (!userData) {
    return <div>Loading...</div>;
  }

  return (
    <div>
      <h1>Dashboard</h1>
      <p>Welcome, {userData.name}!</p>
      <p>Total Posts: {userData.totalPosts}</p>
      <p>Total Likes: {userData.totalLikes}</p>
      <p>Total Comments: {userData.totalComments}</p>
      <canvas id="followerChart" width="400" height="200"></canvas>
    </div>
  );
}

In this example, we fetch user data from an API route (/api/user-data) and store it in the component's state using the useState hook. We use the useEffect hook to initialize a chart using the Chart.js library to visualize the user's follower history. We display the user's name, total posts, likes, and comments, along with the follower chart.

These real-world project examples demonstrate how Next.js can be used to build various types of applications, from blogs and e-commerce stores to social media dashboards. By applying the concepts and techniques covered in this course, you can create dynamic and interactive web applications with Next.js.

10. Conclusion

Congratulations on completing the "Mastering Next.js 13/14: Building Dynamic Web Applications" course! You've gained a comprehensive understanding of Next.js and its powerful features for building modern web applications.

Throughout this course, we covered a wide range of topics, including:

Setting up a Next.js development environment
Understanding the fundamentals of Next.js, such as pages, components, and routing
Exploring different styling options in Next.js
Learning various data fetching methods and their use cases
Diving into the App Router introduced in Next.js 13/14 and its new features
Creating API routes and handling server-side logic
Deploying and optimizing Next.js applications
Exploring advanced topics like custom server configuration, authentication, and internationalization
Applying Next.js concepts to real-world projects

With the knowledge and skills you've acquired, you're now well-equipped to build dynamic, high-performance web applications using Next.js.

As you continue your journey with Next.js, remember to stay updated with the latest releases and features. Next.js has a vibrant community and active development, so make sure to follow the official documentation, join community forums, and explore open-source projects to keep learning and growing.

Here are some additional resources to help you further expand your Next.js knowledge:

Official Next.js Documentation: https://nextjs.org/docs
Next.js GitHub Repository: https://github.com/vercel/next.js
Next.js Blog: https://nextjs.org/blog
Next.js Community: https://github.com/vercel/next.js/discussions
Next.js Examples: https://github.com/vercel/next.js/tree/canary/examples

Remember, practice is key to mastering any skill. Continue building projects, experimenting with new features, and challenging yourself to create innovative web applications using Next.js.

I hope this course has been informative and enjoyable, and I wish you all the best in your future endeavors with Next.js. Happy coding!

Ps, this article is part of a group of 3 Articles, continue your reading :

Fundamentals : https://dev.to/fabrikapp/mastering-nextjs-1314-building-dynamic-web-applications-part-1-fundamentals-gh1
NextJS App Router and API Routes : https://dev.to/fabrikapp/mastering-nextjs-1314-app-router-and-api-routes-fbn
NextJS Advanced Techniques : https://dev.to/fabrikapp/mastering-nextjs-1314-advanced-techniques-5ba

DEV Community: Fabrikapp

TranscribeNow : Live Note-taking and Subtitles

What I Built

Demo

Source Code

Screenshots

Journey

TranscribeNow : Live Note-taking and Subtitles

What I Built

Demo

Source Code

Screenshots

Journey

I created a Realtime Voice Assistant for my ESP-32, here is my journey - Part 2 : Node, OpenAI, Langchain

From Hardware to Smartware: Building the Brain of Our ESP32 Voice Assistant

The Journey So Far

The Ugly Beast

Video

Source Code

The Server Challenge

Behind the Scenes: How It All Works

The Brain of the Operation

Voice Assistant Server Implementation (Node.js/TypeScript)

Core Components

1. Agent Management (lib/agent.ts)

2. Audio Management (lib/audio.ts)

3. WebSocket Connection and Buffer Handling (src/index.ts)

4. Utility Functions (lib/utils.ts)

Buffer Handling: From Start to Finish

1. Capturing Audio on the ESP32

2. Receiving Data on the Server

3. Processing and Writing the Buffer

4. Finalizing the Audio Data

5. Sending the Audio to OpenAI

Handling the Speaker Output

1. Receiving Audio Responses

2. Broadcasting to ESP32 Clients

3. Playback on the ESP32

Setting Up the Project

Usage

Troubleshooting Buffer Issues

Future Improvements

Enhanced Tool Integration

Hardware Enhancements

Performance Optimizations

User Experience

Conclusion

I created a Realtime Voice Assistant for my ESP-32, here is my journey - Part 1 : Hardware, PlatformIO & C++

Building a Real-time Voice Assistant with ESP32: Hardware and C++ Implementation

The Journey Starts

Disclaimers and Considerations

Project Overview

Video

Source Code

Repository Structure

1. ESP32 Firmware (/esp32)

2. Audio Processing Server (/server_audio)

3. LangChain Server (/server_langchain)

Required Materials

Setting Up the Development Environment

Why PlatformIO?

Installing PlatformIO

Setting Up the ESP32 Project

Understanding the C++ Implementation

Buffer Handling: From Start to Finish

Microphone Handler (mic.cpp)

Buffer Allocation (utils.cpp)

Audio Buffer Class (audioMemoryBuffer.cpp)

Speaker Output Handling (lib_speaker.cpp)

Configuring I2S for Microphone and Speaker

Main Program Flow (main.cpp)

Initialization

Main Loop

Wi-Fi Management (lib_wifi.cpp)

Button Handling (lib_button.cpp)

WebSocket Communication (lib_websocket.cpp)

Building and Uploading the Project

Step-by-Step Guide

Common Issues and Resolutions

Testing the Voice Assistant

1. Agent Management (`lib/agent.ts`)

2. Audio Management (`lib/audio.ts`)

3. WebSocket Connection and Buffer Handling (`src/index.ts`)

4. Utility Functions (`lib/utils.ts`)

1. ESP32 Firmware (`/esp32`)

2. Audio Processing Server (`/server_audio`)

3. LangChain Server (`/server_langchain`)

Microphone Handler (`mic.cpp`)

Buffer Allocation (`utils.cpp`)

Audio Buffer Class (`audioMemoryBuffer.cpp`)

Speaker Output Handling (`lib_speaker.cpp`)

Main Program Flow (`main.cpp`)

Wi-Fi Management (`lib_wifi.cpp`)

Button Handling (`lib_button.cpp`)

WebSocket Communication (`lib_websocket.cpp`)

Using `nextjs-routes` for Typed Routing

Examples of Typed Routing with `nextjs-routes`