DEV Community: Abhishek Mishra

Build Voice AI in Python: Complete Speech-to-Text Developer Guide (2026)

Abhishek Mishra — Tue, 10 Feb 2026 15:04:38 +0000

TL;DR – Quick Integration Overview

API Platform: Pulse STT by Smallest AI – a state-of-the-art speech-to-text API supporting real-time streaming and batch audio transcription.

Key Features:

Transcribes in 32+ languages with automatic language detection
Ultra-low latency: ~64ms time-to-first-transcript for streaming
Rich metadata: word timestamps, speaker diarization, emotion detection, age/gender estimation, PII redaction

Integration Methods:

Pre-Recorded Audio: POST https://waves-api.smallest.ai/api/v1/pulse/get_text – upload files for batch processing
Real-Time Streaming: wss://waves-api.smallest.ai/api/v1/pulse/get_text – WebSocket for live transcription

Developer Experience: Use any HTTP/WebSocket client or official SDKs (Python, Node.js). Authentication via a single API key.

Why Pulse STT? Compared to other providers, Pulse offers faster response (64ms vs 200-500ms for typical cloud STT) and all-in-one features (no need for separate services for speaker ID, sentiment, or PII masking).

Quick Links:

API Console – Get your API key
Documentation – Full API reference
Python SDK – Official client

Introduction: Why Voice Integration Matters

Voice is becoming the next frontier for user interaction. From virtual assistants and voice bots to real-time transcription in meetings, speech interfaces are making software more accessible and user-friendly. Developers today have access to Automatic Speech Recognition (ASR) APIs that convert voice to text, opening up possibilities for hands-free control, live captions, voice search, and more.

However, integrating voice AI is more than just getting raw text from audio. Modern use cases demand speed and accuracy – a voice assistant needs to transcribe commands almost instantly, and a call center analytics tool might need not just the transcript but also who spoke when and how they said it.

Latency is critical. A delay of even a second feels laggy in conversation. Traditional cloud speech APIs often have 500–1200ms latency for live transcription, with better ones hovering around 200–250ms. This has pushed the industry toward ultra-low latency – under 300ms – to enable seamless real-time interactions.

In this guide, we'll walk through how to integrate an AI voice & speech API that meets these modern demands using Smallest AI's Pulse STT. By the end, you'll know how to:

Transcribe audio files (WAV/MP3) to text using a simple HTTP API
Stream live audio for instantaneous transcripts via WebSockets
Leverage advanced features like timestamps, speaker diarization, and emotion detection
Use both Python and Node.js to integrate voice capabilities

Understanding Pulse STT

Pulse is the speech-to-text/ ASR(automatic speech recognition) model from Smallest AI's "Waves" platform. It's designed for fast, accurate, and rich transcription with industry-leading latency – around 64 milliseconds to first transcribed word TTFT for streaming audio. This is an order of magnitude faster than many alternatives.

Highlight Features

Feature	Description
Real-Time & Batch Modes	Stream live audio via WebSocket or upload files via HTTP POST
32+ Languages	English, Spanish, Hindi, French, German, Arabic, Japanese, and more with auto-detection
Word/Sentence Timestamps	Know exactly when each word was spoken (great for subtitles)
Speaker Diarization	Differentiate speakers: "Speaker A said X, Speaker B said Y"
Emotion Detection	Tag segments with emotions: happy, angry, neutral, etc.
Age/Gender Estimation	Infer speaker demographics for analytics
PII/PCI Redaction	Automatically mask credit cards, SSNs, and personal info
64ms Latency	Time-to-first-transcript in streaming mode

Getting Started: Authentication

Step 1: Get Your API Key

Step 2: Test Your Key

curl -H "Authorization: Bearer $SMALLEST_API_KEY" \
  https://waves-api.smallest.ai/api/v1/lightning-v3.1/get_voices

Authentication Header

All requests require this header:

Authorization: Bearer <YOUR_API_KEY>

Part 1: Transcribing Audio Files (REST API)

The Pre-Recorded API is perfect for batch processing voicemails, podcasts, meeting recordings, or any existing audio files.

Endpoint

POST https://waves-api.smallest.ai/api/v1/pulse/get_text

Query Parameters

Parameter	Type	Description
`model`	string	Model identifier: `pulse` (required)
`language`	string	ISO code (`en`, `es`, `hi`) or `multi` for auto-detect
`word_timestamps`	boolean	Include word-level timing data
`diarize`	boolean	Enable speaker diarization
`emotion_detection`	boolean	Detect speaker emotions
`age_detection`	boolean	Estimate speaker age group
`gender_detection`	boolean	Estimate speaker gender

Supported Languages (32+)

Italian, Spanish, English, Portuguese, Hindi, German, French, Ukrainian, Russian, Kannada, Malayalam, Polish, Marathi, Gujarati, Czech, Slovak, Telugu, Odia, Dutch, Bengali, Latvian, Estonian, Romanian, Punjabi, Finnish, Swedish, Bulgarian, Tamil, Hungarian, Danish, Lithuanian, Maltese, and auto-detection (multi).

cURL Example

curl --request POST \
  --url "https://waves-api.smallest.ai/api/v1/pulse/get_text?model=pulse&language=en&diarize=true&word_timestamps=true&emotion_detection=true" \
  --header "Authorization: Bearer $SMALLEST_API_KEY" \
  --header "Content-Type: audio/wav" \
  --data-binary "@/path/to/audio.wav"

Python Example

import os
import requests

API_KEY = os.getenv("SMALLEST_API_KEY")
audio_file = "meeting_recording.wav"

url = "https://waves-api.smallest.ai/api/v1/pulse/get_text"
params = {
    "model": "pulse",
    "language": "en",
    "word_timestamps": "true",
    "diarize": "true",
    "emotion_detection": "true"
}
headers = {
    "Authorization": f"Bearer {API_KEY}",
    "Content-Type": "audio/wav"
}

with open(audio_file, "rb") as f:
    audio_data = f.read()

response = requests.post(url, params=params, headers=headers, data=audio_data)
result = response.json()

# Print transcription
print("Transcription:", result.get("transcription"))

# Print word-level details with speaker info
for word in result.get("words", []):
    speaker = word.get("speaker", "N/A")
    print(f"  [Speaker {speaker}] [{word['start']:.2f}s - {word['end']:.2f}s] {word['word']}")

# Check emotions
if "emotions" in result:
    print("\nEmotions detected:")
    for emotion, score in result["emotions"].items():
        if score > 0.1:
            print(f"  {emotion}: {score:.1%}")

Node.js Example

const fs = require('fs');
const axios = require('axios');

const API_KEY = process.env.SMALLEST_API_KEY;
const audioFile = 'meeting_recording.wav';

const url = 'https://waves-api.smallest.ai/api/v1/pulse/get_text';
const params = new URLSearchParams({
  model: 'pulse',
  language: 'en',
  word_timestamps: 'true',
  diarize: 'true',
  emotion_detection: 'true'
});

const audioData = fs.readFileSync(audioFile);

axios.post(`${url}?${params}`, audioData, {
  headers: {
    'Authorization': `Bearer ${API_KEY}`,
    'Content-Type': 'audio/wav'
  }
})
.then(res => {
  console.log('Transcription:', res.data.transcription);

  // Print words with speaker info
  res.data.words?.forEach(word => {
    console.log(`  [Speaker ${word.speaker}] [${word.start}s - ${word.end}s] ${word.word}`);
  });
})
.catch(err => {
  console.error('Error:', err.response?.data || err.message);
});

Example Response

{
  "status": "success",
  "transcription": "Hello, this is a test transcription.",
  "words": [
    {"start": 0.0, "end": 0.88, "word": "Hello,", "confidence": 0.82, "speaker": 0, "speaker_confidence": 0.61},
    {"start": 0.88, "end": 1.04, "word": "this", "confidence": 1.0, "speaker": 0, "speaker_confidence": 0.76},
    {"start": 1.04, "end": 1.20, "word": "is", "confidence": 1.0, "speaker": 0, "speaker_confidence": 0.99},
    {"start": 1.20, "end": 1.36, "word": "a", "confidence": 1.0, "speaker": 0, "speaker_confidence": 0.99},
    {"start": 1.36, "end": 1.68, "word": "test", "confidence": 0.99, "speaker": 0, "speaker_confidence": 0.99},
    {"start": 1.68, "end": 2.16, "word": "transcription.", "confidence": 0.99, "speaker": 0, "speaker_confidence": 0.99}
  ],
  "utterances": [
    {"start": 0.0, "end": 2.16, "text": "Hello, this is a test transcription.", "speaker": 0}
  ],
  "age": "adult",
  "gender": "female",
  "emotions": {
    "happiness": 0.28,
    "sadness": 0.0,
    "anger": 0.0,
    "fear": 0.0,
    "disgust": 0.0
  },
  "metadata": {
    "duration": 1.97,
    "fileSize": 63236
  }
}

Part 2: Real-Time Streaming (WebSocket API)

For live audio – voice assistants, live captioning, call center analytics – use the WebSocket API for sub-second latency with partial results as audio streams in.

WebSocket Endpoint

wss://waves-api.smallest.ai/api/v1/pulse/get_text

Query Parameters

Parameter	Type	Default	Description
`language`	string	`en`	Language code or `multi` for auto-detect
`encoding`	string	`linear16`	Audio format: `linear16`, `linear32`, `alaw`, `mulaw`, `opus`
`sample_rate`	string	`16000`	Sample rate: `8000`, `16000`, `22050`, `24000`, `44100`, `48000`
`word_timestamps`	string	`true`	Include word-level timestamps
`full_transcript`	string	`false`	Include cumulative transcript
`sentence_timestamps`	string	`false`	Include sentence-level timestamps
`redact_pii`	string	`false`	Redact personal information
`redact_pci`	string	`false`	Redact payment card information
`diarize`	string	`false`	Enable speaker diarization

Python Streaming Example

From the official cookbook:

import asyncio
import json
import os
import numpy as np
import websockets
import librosa
from urllib.parse import urlencode

WS_URL = "wss://waves-api.smallest.ai/api/v1/pulse/get_text"

# Configurable features
LANGUAGE = "en"
ENCODING = "linear16"
SAMPLE_RATE = 16000
WORD_TIMESTAMPS = False
FULL_TRANSCRIPT = True
SENTENCE_TIMESTAMPS = False
DIARIZE = False
REDACT_PII = False
REDACT_PCI = False

async def transcribe(audio_file: str, api_key: str):
    params = {
        "language": LANGUAGE,
        "encoding": ENCODING,
        "sample_rate": SAMPLE_RATE,
        "word_timestamps": str(WORD_TIMESTAMPS).lower(),
        "full_transcript": str(FULL_TRANSCRIPT).lower(),
        "sentence_timestamps": str(SENTENCE_TIMESTAMPS).lower(),
        "diarize": str(DIARIZE).lower(),
        "redact_pii": str(REDACT_PII).lower(),
        "redact_pci": str(REDACT_PCI).lower(),
    }

    url = f"{WS_URL}?{urlencode(params)}"
    headers = {"Authorization": f"Bearer {api_key}"}

    # Load audio with librosa (handles any format)
    audio, _ = librosa.load(audio_file, sr=SAMPLE_RATE, mono=True)
    chunk_duration = 0.1  # 100ms chunks
    chunk_size = int(chunk_duration * SAMPLE_RATE)

    async with websockets.connect(url, additional_headers=headers) as ws:
        print("✅ Connected to Pulse STT WebSocket")

        async def send_audio():
            for i in range(0, len(audio), chunk_size):
                chunk = audio[i:i + chunk_size]
                pcm16 = (chunk * 32768.0).astype(np.int16).tobytes()
                await ws.send(pcm16)
                await asyncio.sleep(chunk_duration)
            await ws.send(json.dumps({"type": "end"}))
            print("📤 Sent end signal")

        async def receive_responses():
            async for message in ws:
                result = json.loads(message)

                if result.get("is_final"):
                    print(f"✓ {result.get('transcript')}")

                    if result.get("is_last"):
                        if result.get("full_transcript"):
                            print(f"\n{'='*60}")
                            print("FULL TRANSCRIPT")
                            print(f"{'='*60}")
                            print(result.get("full_transcript"))
                        break

        await asyncio.gather(send_audio(), receive_responses())

# Usage
if __name__ == "__main__":
    api_key = os.environ.get("SMALLEST_API_KEY")
    asyncio.run(transcribe("recording.wav", api_key))

Install dependencies:

pip install websockets librosa numpy

Run:

export SMALLEST_API_KEY="your-api-key"
python transcribe.py recording.wav

Node.js Streaming Example

From the official cookbook:

const fs = require("fs");
const WebSocket = require("ws");
const wav = require("wav");

const WS_URL = "wss://waves-api.smallest.ai/api/v1/pulse/get_text";

// Configurable features
const LANGUAGE = "en";
const ENCODING = "linear16";
const SAMPLE_RATE = 16000;
const WORD_TIMESTAMPS = false;
const FULL_TRANSCRIPT = true;
const DIARIZE = false;
const REDACT_PII = false;
const REDACT_PCI = false;

async function loadAudio(audioFile) {
  return new Promise((resolve, reject) => {
    const reader = new wav.Reader();
    const chunks = [];

    reader.on("format", (format) => {
      reader.on("data", (chunk) => chunks.push(chunk));
      reader.on("end", () => {
        const buffer = Buffer.concat(chunks);
        const samples = new Int16Array(buffer.buffer, buffer.byteOffset, buffer.length / 2);
        resolve(samples);
      });
    });

    reader.on("error", reject);
    fs.createReadStream(audioFile).pipe(reader);
  });
}

async function transcribe(audioFile, apiKey) {
  const params = new URLSearchParams({
    language: LANGUAGE,
    encoding: ENCODING,
    sample_rate: SAMPLE_RATE,
    word_timestamps: WORD_TIMESTAMPS,
    full_transcript: FULL_TRANSCRIPT,
    diarize: DIARIZE,
    redact_pii: REDACT_PII,
    redact_pci: REDACT_PCI,
  });

  const url = `${WS_URL}?${params}`;
  const audio = await loadAudio(audioFile);
  const chunkDuration = 0.1; // 100ms
  const chunkSize = Math.floor(chunkDuration * SAMPLE_RATE);

  return new Promise((resolve, reject) => {
    const ws = new WebSocket(url, {
      headers: { Authorization: `Bearer ${apiKey}` },
    });

    ws.on("open", async () => {
      console.log("✅ Connected to Pulse STT WebSocket");

      for (let i = 0; i < audio.length; i += chunkSize) {
        const chunk = audio.slice(i, i + chunkSize);
        ws.send(Buffer.from(chunk.buffer, chunk.byteOffset, chunk.byteLength));
        await new Promise((r) => setTimeout(r, chunkDuration * 1000));
      }
      ws.send(JSON.stringify({ type: "end" }));
      console.log("📤 Sent end signal");
    });

    ws.on("message", (data) => {
      const result = JSON.parse(data.toString());

      if (result.is_final) {
        console.log(`✓ ${result.transcript}`);

        if (result.is_last) {
          if (result.full_transcript) {
            console.log("\n" + "=".repeat(60));
            console.log("FULL TRANSCRIPT");
            console.log("=".repeat(60));
            console.log(result.full_transcript);
          }
          ws.close();
        }
      }
    });

    ws.on("close", resolve);
    ws.on("error", reject);
  });
}

// Usage
const apiKey = process.env.SMALLEST_API_KEY;
transcribe("recording.wav", apiKey).then(() => console.log("Done!"));

Install dependencies:

npm install ws wav

Run:

export SMALLEST_API_KEY="your-api-key"
node transcribe.js recording.wav

WebSocket Response Format

{
  "session_id": "sess_12345abcde",
  "transcript": "Hello, how are you?",
  "full_transcript": "Hello, how are you?",
  "is_final": true,
  "is_last": false,
  "language": "en",
  "words": [
    {"word": "Hello,", "start": 0.0, "end": 0.5, "confidence": 0.98, "speaker": 0},
    {"word": "how", "start": 0.5, "end": 0.7, "confidence": 0.99, "speaker": 0},
    {"word": "are", "start": 0.7, "end": 0.9, "confidence": 0.97, "speaker": 0},
    {"word": "you?", "start": 0.9, "end": 1.2, "confidence": 0.99, "speaker": 0}
  ]
}

Key Response Fields

Field	Description
`is_final`	`false` = partial/interim transcript; `true` = finalized segment
`is_last`	`true` when the entire session is complete
`transcript`	Current segment text
`full_transcript`	Accumulated text from entire session (if enabled)
`words`	Word-level timestamps (if enabled)

Part 3: Advanced Features

Speaker Diarization

Enable diarize=true to identify different speakers:

params = {"model": "pulse", "language": "en", "diarize": "true"}

Response includes speaker labels:

{
  "words": [
    {"word": "Hello", "speaker": 0, "speaker_confidence": 0.95},
    {"word": "Hi", "speaker": 1, "speaker_confidence": 0.92}
  ],
  "utterances": [
    {"text": "Hello, how can I help?", "speaker": 0},
    {"text": "I have a question.", "speaker": 1}
  ]
}

Emotion Detection

Enable emotion_detection=true to analyze speaker sentiment:

{
  "emotions": {
    "happiness": 0.28,
    "sadness": 0.0,
    "anger": 0.0,
    "fear": 0.0,
    "disgust": 0.0
  }
}

PII/PCI Redaction

For compliance (HIPAA, PCI-DSS), enable redact_pii=true or redact_pci=true:

{
  "transcript": "My credit card is [CREDITCARD_1] and SSN is [SSN_1]",
  "redacted_entities": ["[CREDITCARD_1]", "[SSN_1]"]
}

Age and Gender Detection

Enable age_detection=true and gender_detection=true:

{
  "age": "adult",
  "gender": "female"
}

Comparing STT Providers

Provider	Latency	Languages	Diarization	Emotion	PII Redaction	Price (per 1000 min)
Pulse STT	~64ms	32+	✅	✅	✅	Competitive
Google Cloud STT	200-300ms	125+	✅	❌	❌	~$16
Deepgram	100-200ms	36+	✅	❌	✅	~$4-5
AssemblyAI	200-400ms	30+	✅	✅	✅	~$3.50
OpenAI Whisper	Batch only	99+	❌	❌	❌	~$6

Why Pulse STT stands out:

Fastest time-to-first-transcript (64ms)
All-in-one features (no separate services needed)
Competitive accuracy across diverse accents
Built for real-time voice AI applications

Best Practices

Audio Quality

Use 16kHz, mono, 16-bit PCM for best results
WAV or FLAC formats are ideal
Minimize background noise when possible

Error Handling

try:
    response = requests.post(url, params=params, headers=headers, data=audio_data, timeout=120)
    response.raise_for_status()
except requests.exceptions.HTTPError as e:
    if e.response.status_code == 429:
        # Rate limited - implement exponential backoff
        time.sleep(2 ** retry_count)
    elif e.response.status_code == 401:
        # Invalid API key
        raise ValueError("Invalid API key")

Rate Limiting

Add 500ms+ delay between batch requests
Use webhooks for long audio files
Implement exponential backoff for 429 errors

Bonus: Full Demo Application

Want to see everything working together? Check out the demo app in the code samples repository — a complete Next.js web application featuring:

File upload transcription with word-level timestamps (hover to see timing)
Real-time microphone streaming with live transcript display
Secure WebSocket proxy that keeps your API key server-side
Modern UI with Smallest AI brand colors
Language selection (English, Hindi, Spanish, French, German, Portuguese, Auto-detect)
Emotion detection and speaker diarization display

Quick Start

cd demo-app
npm install

Create a .env.local file with your API key:

echo 'SMALLEST_API_KEY=your-api-key' > .env.local

Start both servers (Next.js + WebSocket proxy):

npm run dev:all

Then open http://localhost:3000 in Chrome or Safari (for microphone access).

How It Works

The demo runs two servers:

Next.js (port 3000) — Serves the React UI and handles file upload via /api/transcribe
WebSocket Proxy (port 3001) — Securely proxies audio from browser to Pulse STT WebSocket API

Browser → WebSocket Proxy (3001) → Pulse STT (wss://waves-api.smallest.ai)
Browser → Next.js API (3000) → Pulse STT (REST API)

This architecture keeps your API key secure on the server while enabling real-time streaming.

Project Structure

demo-app/
├── src/
│   └── app/
│       ├── api/
│       │   └── transcribe/
│       │       └── route.ts    # REST API for file upload
│       ├── page.tsx            # Main UI
│       └── layout.tsx
├── ws-server.js                # WebSocket proxy server
├── .env.local                  # Your API key (create this)
└── package.json

Scripts

Command	Description
`npm run dev`	Start Next.js only
`npm run dev:ws`	Start WebSocket proxy only
`npm run dev:all`	Start both (recommended)

This architecture pattern is recommended for production apps — API keys stay server-side while the React frontend provides a smooth user experience with both file upload and real-time microphone transcription.

Conclusion

Integrating voice and speech capabilities into your workflow and apps can greatly enhance user experience. With Pulse STT, developers can achieve high-accuracy, low-latency transcription with just a few API calls.

When to use REST API:

Podcast transcription
Meeting recordings
Voicemail processing
Batch analytics

When to use WebSocket API:

Live captioning
Voice assistants
Call center real-time analytics
Interactive voice applications

The code patterns in this guide translate directly to production. Start with the REST API for prototyping, then add WebSocket streaming when real-time interaction becomes a requirement.

Resources

Smallest AI Console — API key management
Waves Documentation — Full API reference
Discord Community — Developer support

How to do efficient fine-Tuning for LLMs using SLoRA

Abhishek Mishra — Wed, 07 Aug 2024 08:23:17 +0000

You're a passionate AI developer, eager to try to utilize the power of large language models (LLMs) for your latest project. You've got brilliant ideas, but there's a catch – fine-tuning these massive models feels like trying to parallel park a cruise ship in a crowded marina. It's resource-intensive, time-consuming, and frankly, a bit overwhelming.
Sound familiar? You're not alone in this boat.

Before we dive deeper, let's clarify what we mean by fine-tuning:

Fine-tuning is the process of further training a pre-trained model on a specific task or dataset to adapt its knowledge for a particular application.

The world of AI has been grappling with a significant challenge: how to efficiently fine-tune LLMs without breaking the bank or melting your hardware – we all are GPU poor except NVIDIA.
Traditional fine-tuning methods are like using a sledgehammer to crack a nut, – they get the job done, but at what cost?
Enter SLoRA – Sparse Low-Rank Adaptation, – the unsung hero for efficient model fine-tuning.

Fine-tuning large language models (LLMs) has become a crucial step in achieving state-of-the-art results in various natural language processing (NLP) tasks. However, this process often comes with significant computational costs, memory requirements, and time constraints. In this article, we'll explore SLoRA, a novel approach to efficient model fine-tuning that promises to revolutionize the way we work with LLMs. By leveraging sparse low-rank adaptation, SLoRA offers a faster, more cost-effective, and more sustainable way to fine-tune LLMs without sacrificing performance.

What is SLoRA?

To understand SLoRA, it's essential to first grasp the concept of Low-Rank Adaptation (LoRA). LoRA is a parameter-efficient fine-tuning method that updates only a small subset of model parameters while keeping the rest frozen. This approach has shown remarkable success in reducing computational costs and memory requirements for model fine-tuning.

However, LoRA still updates a relatively large number of parameters, which can be computationally expensive and memory-intensive. This is where SLoRA comes in – by introducing sparsity into the LoRA framework, SLoRA further reduces the number of updated parameters to approximately 1% of the original model's parameters. This drastic reduction in updated parameters leads to significant computational savings and faster convergence rates
SLoRA stands for Sparse Low-Rank Adaptation, a method designed to enhance the efficiency of fine-tuning LLMs. It builds on the concept of LoRA, which constrains the update of pre-trained weights using low-rank decomposition. SLoRA introduces sparsity into this approach, focusing only on a subset of parameters that significantly impact the model's performance.

Here's a simple analogy -- Imagine you have a giant jigsaw puzzle. LoRA would focus on updating a specific section of the puzzle, while SLoRA would pinpoint only the most important pieces within that section.

How SLoRA Works?

SLoRA employs a sparse matrix approach where the weight updates are constrained to a low-rank format and further sparsified. This involves decomposing the weight matrix into a product of two smaller matrices and applying updates only to a sparse subset of the original parameters. This method reduces the density of updates to about 1%, significantly cutting down on the resources needed for training.

Think of a weight matrix in an LLM as a giant grid of numbers. These numbers represent the connections between different parts of the model. SLoRA uses a technique called sparse matrix decomposition to break down this grid into smaller, more manageable pieces.

Imagine slicing a pizza into smaller triangles. SLoRA only updates the toppings on a select few of those triangles, leaving the rest untouched. This dramatically reduces the amount of data we need to process and store.

Comparison with Traditional Fine-Tuning Methods

Traditional fine-tuning involves updating all model parameters, which is resource-intensive and time-consuming. In contrast, SLoRA updates only a small, significant subset of parameters, achieving similar performance with much lower computational overhead.

SLoRA in the nutshell:

SLoRA builds on the foundation of LoRA (Low-Rank Adaptation), which already constrains updates to a small subset of parameters.
It takes this a step further by introducing sparsity – focusing on an even smaller, more crucial set of parameters.
The result? You're updating only about 1% of the model's parameters, dramatically reducing computational load and memory requirements.

You can read more here

Benefits:

Reduced Computational Requirements: By updating only a sparse subset of parameters, SLoRA dramatically lowers the computational load.
Faster Fine-Tuning Process: The sparse updates enable quicker convergence, speeding up the fine-tuning process.
Lower Memory Usage: The reduced number of updates translates to lower memory requirements, making it feasible to deploy models on devices with limited memory.
Potential for Improved Model Performance: Efficient parameter updates can lead to models that are not only faster but also potentially more robust and adaptable to specific tasks.

How to use SLoRA in Your Projects?

Initialize the Sparse Matrix: Begin by setting up the sparse matrix with low-rank decomposition.
Apply Sparse Updates: Update only the critical parameters as identified by the sparsity constraints.
Train the Model: Proceed with the fine-tuning process, leveraging the computational efficiency of SLoRA.

For more detailed guide checkout this paper: https://arxiv.org/pdf/2308.06522

SLoRA's Secret Sauce:

Here's where SLoRA really flexes its muscles. By dramatically reducing the number of parameters updated during fine-tuning, SLoRA processes fewer tokens. It's like having a car that can drive the same distance using only a fraction of the fuel.

Fewer tokens processed = Lower costs
Efficient updates = More bang for your buck
Optimized token usage = Stretch your budget further

Let's say you're fine-tuning a model on a specific task. Traditional methods might process millions of tokens, updating every parameter. With SLoRA, you're looking at a fraction of that - potentially cutting your token usage (and thus, your costs)!

Conclusion

SLoRA represents a significant leap forward in efficient model fine-tuning. It empowers developers to unlock the full potential of LLMs while being mindful of resources and costs.

Ready to give SLoRA a try? At Tune AI, we implemented this approach to, make models mode accessible and provide an efficient way to fine-tune and serve LLMs.
Head over to Tune Studio and start exploring! Share your experiences and let's build a more accessible and sustainable future for AI together.

Don't Be That Dev: Avoiding the XY Problem Trap

Abhishek Mishra — Tue, 20 Feb 2024 19:55:52 +0000

Have you ever found yourself in a situation where you're trying to solve a problem, but you're so focused on your attempted solution that you're not even sure what the actual problem is? This is a common pitfall known as the XY problem.

What is the XY Problem?

The XY problem is a communication issue that often arises in software development. It occurs when someone seeking help focuses on their attempted solution (X) rather than the actual problem they're trying to solve (Y). This can lead to misunderstandings, wasted time, and ineffective solutions.

Here's a classic example:

Imagine a developer named Abhishek is working on a web application that displays product information. He encounters an issue where product images take a long time to load, impacting the user experience.

The XY Problem Approach:

Instead of clearly stating the problem of slow image loading, Abhishek asks a question focusing on his chosen solution:

"How can I implement caching for product images in my application?"

Why it's a problem:

While caching might seem like a logical solution, focusing on it without explaining the context can lead to issues:

Misdiagnosis: The person providing help might suggest specific caching techniques without understanding the root cause of slow loading. This could be inefficient if the issue lies elsewhere, like slow image optimization or network problems.
Limited solutions: Exploring only caching might overlook alternative solutions like image compression or a Content Delivery Network (CDN) that could be more effective.
Wasted time: Both Abhishek and the other dev spend time discussing caching, which might not be the real solution.

Better approach:

Abhishek should rephrase his question to focus on the actual problem:

"I'm experiencing slow loading times for product images in my web application. This is affecting the user experience. Can you suggest ways to improve the loading speed?"

Benefits:

Clearer communication: The helper understands the desired outcome (faster image loading) and can suggest various solutions, including caching, optimization, or CDN, depending on the root cause.
Effective solutions: Exploring a wider range of options increases the chances of finding the most efficient and suitable solution.
Saved time: Focusing on the actual problem avoids unnecessary discussions about potentially irrelevant solutions.

Avoiding the XY Problem

Here are some tips to avoid falling into the XY problem trap:

Clearly define the problem you're facing: What are you trying to achieve? What specific issue are you encountering?
Provide context and relevant details: What have you tried so far? What is the expected outcome?
Be open to exploring different solutions: Don't get fixated on your initial idea.

By following these tips, you can ensure effective communication and collaboration when seeking help with software development problems.

Remember: It's always better to focus on the Y (the actual problem) before jumping to the X (solutions).

Reference:

For more information on the XY problem, you can refer to the Wikipedia page: https://en.wikipedia.org/wiki/XY_problem

Understanding Strings in Rust: String vs str

Abhishek Mishra — Mon, 10 Jul 2023 18:01:05 +0000

If you're a Python developer exploring the Rust language, you'll find that working with strings in Rust is quite different from Python. Rust has two main types to represent strings: String and str.

Let's dive into what they are and how they differ.

What are `String` and `str`?

String is a growable, heap-allocated data structure that allows you to store a sequence of UTF-8 characters. This is similar to Python's str type, but with one key difference - String in Rust is mutable and can be modified (Rust Doc: String).

On the other hand, str (pronounced 'string slice') is an immutable reference or 'view' into a string. It can point to the entire string or a subsection of it (Rust Doc: str).
You will most commonly encounter str in its borrowed form, represented as &str.
Also, &mut str is a mutable reference to a str. You can use it to perform in-place modifications to the string, as long as those modifications maintain valid UTF-8.

In Rust, str itself is a type that represents a sequence of valid UTF-8 characters, while &str and &mut str are references to such a sequence. The difference is that &str is an immutable reference (meaning you can't use it to change the underlying string), while &mut str is a mutable reference (so you can use it to change the underlying string, as long as you maintain valid UTF-8).

Why does Rust have two string types?

The two string types in Rust cater to different use cases.

String is used when you need to own a string and change its contents, like appending characters or changing a character at a certain position.

str is used for more 'read-only' operations. Since it's an immutable reference, it does not own the underlying memory and cannot alter it.

&mut str allows you to perform some in-place modifications to a string, as long as you maintain valid UTF-8. However, its use is less common and usually requires more care.

However, the mutable operations that you can do on &mut str are limited. For example, as the user mentioned, you can use the make_ascii_lowercase method to change all ASCII alphabetic characters to lowercase in place, but you can't use it to append to the string or change its length, because that could invalidate the UTF-8.

This is a different from Python, where you're used to working with just one main string type (str), which is always immutable.

Example

Let's look at an example:

fn main() {
    let mut s = String::from("hello");
    s.push_str(", world");  // This works fine
    println!("{}", s);
}

In this Rust program, we create a String and then append to it using the push_str method, something Python developers might be surprised to see, given that strings in Python are immutable.

Now, let's see what happens when we try to modify (change a specific character) a str:

fn main() {
    let mut s = "hello";
    s[0] = 'H';  // This will cause an error
    println!("{}", s);
}

In this Rust program, we're trying to replace the first character of the str s with 'H'. If s were a mutable String, we could do this with the replace_range method. But because s is a str, we cannot change its contents, and this program will not compile because str is immutable.

If we want to append a character to the string, we first have to convert it to a String with s.to_string(). Then, we can use the push method to append a character to the String.

For example:

fn main() {
    let s = "hello";
    let mut t = s.to_string();  // Convert to String to allow changes
    t.push('!');  // Append a character to the string
    println!("{}", t);
}

This program will compile and print "hello!".

Example of using &mut str

fn main() {
    let mut s = String::from("HELLO");
    let s_slice = &mut s[..];  // Obtain a &mut str from the String
    s_slice.make_ascii_lowercase();  // This works fine
    println!("{}", s);
}

In this program, we create a String and then obtain a &mut str slice from it using &mut s[..]. We then call the make_ascii_lowercase method on the &mut str to change all ASCII alphabetic characters to lowercase. The program will print "hello".

Conclusion

String and str serve different purposes. Use String when you need to own and modify string data, and use str when you need a temporary, read-only view into a string.

Rust string types give you more control over memory allocation and can help prevent bugs related to memory access. This design choice aligns with Rust's core goals of performance and safety. It's one of the ways Rust differs significantly from Python and many other languages, and it's something you'll get used to as you write more Rust code.

For more detailed information, refer to the official Rust documentation for String and str.

Creating a Low-Latency Cache with Dozer and PostgreSQL

Abhishek Mishra — Tue, 30 May 2023 05:56:29 +0000

In the realm of software development and data management, the efficiency of data retrieval plays a crucial role in the overall performance of an application. One effective way to enhance this efficiency is by implementing a cache. A cache is a high-speed data storage layer that stores a subset of data, typically transient in nature so that future requests for that data are served up faster than is possible by accessing the data’s primary storage location.

This blog post explores how to enhance these aspects by implementing a low-latency cache using Dozer and PostgreSQL. We delve into the importance of caching in data management, the limitations of not having a cache, and the unique advantages of using Dozer as a cache layer. We also provide a guide on how to use Dozer for real-time data ingestion from PostgreSQL. This post is a must-read for developers seeking to improve their application's performance and responsiveness, and for those looking to simplify their data infrastructure.

Why Do We Need a Cache on Top of PostgreSQL?

PostgreSQL is a powerful, open-source object-relational database system. It is highly extensible and enables high performance on complex queries and heavy loads. However, like any other database, it has its limitations.

When a database grows in size and complexity, the time taken to retrieve data can increase significantly. This is especially true for complex queries that involve multiple joins and aggregations. In such scenarios, every millisecond counts, and the latency can add up quickly, leading to a slow and unresponsive application.

This is where a cache comes into play. A cache is a high-speed data storage layer that stores a subset of data so that future requests for that data are served up faster than is possible by accessing the data’s primary storage location. By storing frequently accessed data in a cache, we can significantly reduce the time taken to access this data.

What's the Problem with Not Having a Cache?

Without a cache, every time a client requests data, the application has to query the database. This can be a time-consuming operation, especially if the database is large or the query is complex. This can lead to increased latency and a poor user experience.

Moreover, without a cache, the database can become a bottleneck, especially under heavy load. Every read and write operation involves a significant amount of work, and as the number of operations increases, the time taken to complete these operations also increases. This can lead to increased CPU and memory usage, and in extreme cases, it can even lead to database crashes.

Why Not PostgreSQL Read Replica?

While a PostgreSQL read replica can certainly help offload some of the read operations from your primary database, it's not without its limitations. The process of querying a read replica still involves a network round trip and the overhead of a database query. These factors can contribute to latency, especially when dealing with large volumes of data or complex queries. A cache like the one provided by Dozer, on the other hand, can serve data much faster than a read replica.

In terms of using a read replica instead of a cache layer like Dozer, it depends on your specific use case and requirements. A read replica can help improve read performance by distributing read traffic across multiple instances, but it may not provide the same level of performance improvement as a cache layer like Dozer.

Moreover, Dozer doesn't make a distinction between types of data sources. Developers can get a seamless experience building products with application databases such as Postgres and MySQL, data warehouses such as SnowFlake and cloud storage such as S3 and Deltalake. Dozer can also consume real-time events and Ethereum data.

Why Dozer?

In the context of database management, most OLTP workloads involve random disk I/O usage. Given that disks, including SSDs, are slower in performance than RAM, database systems use caching to increase performance. Caching is all about storing data in memory (RAM) for faster access at a later point in time.

One approach to database caching involves replicating data using Change Data Capture (CDC) to an alternative database optimized for your queries. This method is increasingly being adopted for operations that require extensive data reading. However, it comes with its own set of challenges such as ensuring real-time data, managing indexing, guaranteeing availability, handling schema changes, and considering costs. Dozer addresses these pain points by providing an end-to-end system that takes care of the complexity of caching logic, allowing developers to focus on building their applications.

Dozer, an open-source data API backend, provides a solution to this. It connects to any of your data sources, transforms and stores the data in an embedded cache powered by LMDB, automatically creates secondary indexes, and instantly generates low-latency REST and gRPC APIs. Dozer is implemented fully in Rust for fast processing of data and is less resource intensive.

Without a cache layer, data retrieval would rely solely on the performance of the PostgreSQL database and its internal caching mechanisms. This can result in slower response times and increased load on the database server. By using a cache layer like Dozer, you can improve the performance of data retrieval and reduce the load on the database server.

How to Use Dozer as a Cache

Architecture Diagram

Here's a simple architecture diagram showing how Dozer can be used as a cache layer for a PostgreSQL database:

In a nutshell:

Dozer begins by ingesting data in real-time from the PostgreSQL database using the PostgreSQL connector and stores this data in its cache.
When a client sends a request for data, Dozer processes the request using the low-latency APIs it has generated, and retrieves the required data from its cache to serve the client.
For write requests, these are facilitated through the backend API, which directly communicates with the PostgreSQL database. Once the write operation is confirmed, the PostgreSQL database sends a write confirmation back to the backend API.

The key point here is that data ingestion into Dozer's cache and the serving of data to the client through Dozer's APIs occur in parallel. This setup ensures quick and efficient access to data, enhancing the performance of your application."

Dozer provides a PostgreSQL connector that allows it to ingest data in real-time from your PostgreSQL database. The configuration can be as simple as the following:

app_name: movie-booking
connections:
  - config: !Postgres
      user: postgres
      password: postgres
      host: 0.0.0.0
      port: 5438
      database: database_name
    name: connection_name

sql: |
  -- transformation and
  -- analytical queries here.

sources:
  - name: source_name_1
    table_name: table_name
    columns:
    connection: !Ref connection_name

  ...

endpoints:
  - name: endpoint_name
    path: /endpoint_path
    table_name: table_name
    index:
      primary_key:
        - primary_key_column_name

This configuration allows Dozer to ingest data in real-time from your PostgreSQL database and store it in a high-performance cache. This cache can then be used to serve low-latency data APIs to your application.

Conclusion

Implementing Dozer with its ability to ingest data in real-time from PostgreSQL can significantly improve the performance of your application by reducing the time taken to access frequently used data. This approach can significantly enhance your application's performance, making it more efficient and responsive in today's demanding tech scenario. By leveraging Dozer's real-time SQL engine, automatic secondary indexing, and instant API generation, developers can drastically lower the cost, complexity, and effort involved in putting together the data infrastructure necessary to build data applications.

Happy coding, Happy Data APIng! 🚀👩‍💻👨‍💻

Secure Your Data APIs with Dozer: Adding Authorization to APIs

Abhishek Mishra — Thu, 25 May 2023 14:35:50 +0000

In today's world of API-driven applications, securing access to data is very important. As the amount of data generated and consumed continues to grow, the need for reliable, and efficient data access management becomes increasingly critical. One essential aspect of data security is ensuring proper authorization for API calls, allowing only the right people, applications, and services to access the required data. Managing access control for your APIs is crucial to guarantee that only authorized users can access and perform specific actions. With the increasing complexity of API ecosystems, secure and efficient authentication and authorization mechanisms are necessary.

Dozer addresses this challenge by adding authorization layer to your data APIs in an efficient way. One of the ways Dozer provides authorization is through the use of JSON Web Tokens (JWT). JWT is a widely used industry-standard method for representing claims securely between two parties. You can learn more about JWT here. Using JWT tokens simplifies the process of adding authorization to your data APIs.

To demonstrate this, we'll build a simple movie ticket booking app that has two roles - user (public) and admin. The user can only book tickets and view their bookings, whereas the admin has full access to all the APIs. We will use @dozerjs/dozer and dozer-react to enhance the functionality of our app. @dozerjs/dozer is a TypeScript wrapper over gRPC APIs generated by Dozer, facilitating seamless communication with Dozer, while dozer-react provides a set of React helpers for using Dozer as a data provider. The sample application's repository can be found here, you can aslo find the instructions to run the application directly.

By following the step-by-step guide we've provided, you can safeguard your application from unauthorized access, ensuring data integrity and user trust.

TL;DR

Clone the repo and change to the project directory:

git clone git@github.com:getdozer/dozer-samples.git && cd dozer-samples/usecases/api-auth/

Start the PostgreSQL database:

cd database && docker-compose up -d && cd ..

Generate and export a master token for auth:

dozer api generate-token
export MASTER_TOKEN=your_token_here

Start the Dozer app:
```
dozer
```
In a new terminal, install server dependencies and run the server app:
```
cd server && yarn install && node index.js
```
In a new terminal, navigate to the client directory, install client dependencies and start the app:

    cd ../client && yarn install && yarn start

Visit localhost:3000 to use the app. For the admin dashboard, go to localhost:3000/admin/dashboard with "admin" as username and password.

User Page

Admin dashboard

Introduction

In this blog, we'll guide you through the process of constructing a movie ticket booking application using Dozer, React, and PostgreSQL. The PostgreSQL database is utilized directly in our server-side application to manage user creation and ticket bookings.

To elaborate, the server app is specifically designed to handle user login, create new user entries, and register ticket bookings. It interacts directly with the PostgreSQL database to handle these tasks. The information required for booking tickets, such as booking details, is sent from the React frontend application to the Node.js server app, which then inserts these details into the PostgreSQL database.

However, when it comes to consuming real-time data like the movie list, dashboard updates, or ongoing bookings, we use Dozer, which fetches the data from PostgreSQL using the PostgreSQL connector. This strategy allows us to get the most recent booking updates and display them on the dashboard in real-time.

The PostgreSQL connector in Dozer bridges our database with the Dozer server and enables the seamless transfer of data in real-time. You can learn more about this connector at Dozer's documentation page here: PostgreSQL Connector.

On the client side, we're using a React app to display the movie data and booking interface, pulling data from Dozer. We employ the @dozerjs/dozer and dozer-react libraries to simplify interactions with Dozer and access real-time data without having to hardcode API endpoints in the code.

The @dozerjs/dozer library manages gRPC requests between the frontend and the Dozer server, allowing effective communication between the two. In contrast, the dozer-react library provides a set of React helpers, making it simpler to use Dozer as a data provider in our React app. Find more about these libraries at their respective GitHub repositories: @dozerjs/dozer and dozer-react.

Let's dive into the process of building this application, demonstrating how you can leverage Dozer to amplify your app development experience.

Server-Side Setup (Express.js)

First, let's set up the server-side code using Express.js. The code is available in server/index.js:

const express = require('express')
const app = express()
const {ApiClient, AuthClient} = require("@dozerjs/dozer");
const axios = require('axios');
const cors = require('cors');
const { faker } = require('@faker-js/faker');

//... Rest of the code

We import the required libraries, including express, @dozerjs/dozer, axios, cors, and @faker-js/faker.

Database Connection for Server-Side

Setting up a connection with a PostgreSQL database is crucial to perform operations like user creation and booking entries. We accomplish this using the pg client.

const { Client } = require('pg')

const client = new Client({
    host: '127.0.0.1',
    port: 5432,
    database: 'omdb',
    user: 'postgres',
    password: 'postgres',
});

client.connect();

In the configuration object, we specify the parameters like host, port, database name, user, and password. With the successful connection established via client.connect(), we can execute queries to our database in our Express.js routes.

This setup enables two primary functionalities in our application:

User creation: With the database connection ready, we can insert new user details into the users table. This is used in the /public/new_user endpoint, where a new user is created with a unique UUID and a randomly generated name.
Booking entries: The connection also allows us to create new entries in the bookings table. This functionality is used in the /public/book_movie endpoint, where a user can book a movie, resulting in a new booking record in the database with the user's id and the provided movie id.

Admin Login Endpoint

This endpoint validates the provided username and password. If valid, a post request is made to the authentication server to get a token. The token is then sent back to the client. Error handling is also set up to send a 500 status code in case of a server error and a 400 status code for incorrect credentials.

app.post('/admin/login', function(req, res){
    // Check if the provided credentials are valid
    // Generate JWT token with full access to all API endpoints
    // Send the JWT token to the client
})

Public New User Endpoint

The new user endpoint generates a random username and UUID using the faker library. It then inserts this new user into the database. After successful insertion, a JWT token is generated for the new user with limited access. Error handling is set up in case the server encounters an error while processing the request.

app.get('/public/new_user', function(req, response){
    // Generate random username and UUID
    // Insert new user into the database
    // Generate JWT token with limited access for the new user
    // Handle server errors
})

Public Book Movie Endpoint

The book movie endpoint allows users to book a movie by sending their JWT token in the headers. The server first validates the user's token. If valid, it inserts a new record in the bookings table in the database with the user's id and the provided movie id. The response includes error handling as well.

app.post('/public/book_movie', function(req, response){
    // Validate the user's token
    // Insert a new record in the bookings table
    // Handle server errors
})

Error Handling Middleware

Lastly, we also add a middleware function to handle any errors that may slip through our route handlers. This middleware logs the error and sends a generic error message to the client.

app.use((err, request, response, next) => {
    // log the error, for now just console.log
    console.log(err)
    response.status(500).send('Something broke!')
})

After setting up these routes, the Express.js application is ready to listen for requests on port 4000.

app.listen(4000)

Our error handling middleware ensures that any errors that occur during these database operations are effectively captured and appropriately communicated to the user.
In this way, our server is set up to handle the creation of new users, allow users to book movies, and allow an admin to log in, all while ensuring errors are handled appropriately. .

Database Setup For Movie data (PostgreSQL)

We use PostgreSQL as our database, and the configuration is available in the docker-compose.yml file. The file defines a postgres service, which includes the container name, image, volumes, and environment variables.

version: '3.8'
services:
  postgres:
    container_name: omdb-postgres
    image: debezium/postgres:13
    volumes:
      - ./pg_hba.conf:/var/lib/foo/pg_hba.conf
      - ./data/init.sql:/docker-entrypoint-initdb.d/init.sql
      - ./data:/data
    command: postgres -c hba_file=/var/lib/foo/pg_hba.conf
    environment:
      POSTGRES_DB: omdb
      POSTGRES_USER: postgres
      POSTGRES_PASSWORD: postgres
      ALLOW_IP_RANGE: 0.0.0.0/0
    ports:
      - '5438:5432'
    healthcheck:
      test: [ "CMD-SHELL", "pg_isready -U postgres -d omdb" ]
      interval: 5s
      timeout: 5s
      retries: 5

The database is populated with the Open Movie Database (OMDB) dataset, which provides a comprehensive list of movie-related data. For further details about this dataset, you can visit the official OMDB repository here: OMDB on GitHub

Front-End Setup (React)

The frontend of our application is designed with React, a popular JavaScript library used for building interactive UIs. The frontend setup for our movie booking application involves a series of React components that interact with our backend server using HTTP requests. These components handle user authentication, data fetching, and real-time updates to ensure users have an interactive and up-to-date interface.

Let's take a look at the structure of our project:

├── README.md
├── package.json
├── src
│   ├── App.css
│   ├── App.js
│   ├── App.test.js
│   ├── components
│   │   ├── admin
│   │   │   └── dashboard.js
│   │   ├── login.js
│   │   └── public
│   │       ├── movies.js
│   │       └── moviesList.js
│   ├── index.css
│   ├── index.js
│   ...
└── yarn.lock

Our React components are placed in the src/components directory, separated by user type (public users and admins).

Login Component

The login.js component facilitates the login functionality for the admin users. It includes text fields for the admin to enter their username and password, and a login button to submit the login details.

On submission, a POST request is made to the http://localhost:4000/admin/login endpoint with the username and password as the body of the request. If the credentials are valid, the response will contain a token, which is stored in the browser's local storage and used to authenticate future requests.

Admin Dashboard Component

The dashboard.js component acts as the main interface for admin users. On load, it checks if a token exists in local storage. If not, it redirects the user to the login page.

Using the movies_with_bookings query, it fetches the list of movies with their booking counts and displays them in a table. This component uses real-time updates to ensure the admin always has the most current booking data.

Movie List Component

The moviesList.js component displays the list of movies and their current booking counts to public users. It allows users to "buy" a ticket for a movie, which is simulated by making a POST request to the http://localhost:4000/public/book_movie endpoint with the selected movie's id. The state of this component is updated in real-time as bookings are made.

Movies Component

The movies.js component acts as a wrapper for the moviesList.js component. On load, it checks for a token in local storage. If a token doesn't exist, it makes a GET request to the http://localhost:4000/public/new_user endpoint to generate a new token. Once a token is obtained, it passes the token as a prop to the MoviesList component.

Integrating Dozer JWT in the Application

Having established the foundation of our application in the previous sections, it's time to delve into the secure integration of JWT authentication and Role-Based Access Control (RBAC) using Dozer. Our focus will be on illustrating how this can be applied in our movie ticket booking application.

Outlined below are the key steps to enable secure API authorization in your application:

Generate a master token using Dozer's API server.
Using the master token, generate custom access tokens for each set of APIs to establish granular-level access.
Incorporate the generated custom access tokens in subsequent client-side API calls.

1. Master JWT Token Generation

Kick off the process by running the following command to generate a master JWT token:

dozer api generate-token

This command triggers the generation of the master token, which is outputted to stdout. The master token has the privilege of accessing all API endpoints specified in the config file. To access the APIs, include the token in your request header as demonstrated below:

MASTER_TOKEN = 'your_master_token_here'

2. Custom Access Token Generation

For a more granular access control, Dozer allows for the creation of custom access tokens for specific API calls. These tokens can be generated through a POST request to the /auth/token endpoint, providing an access filter and the previously generated MASTER_TOKEN in the request header.

This affords you the opportunity to designate specific access controls for different users or roles within your application. For a more detailed guide on this, you may refer to the Dozer documentation page on Authentication & Authorization.

In the context of our movie ticket booking app, we generate a JWT token comprising the necessary access controls whenever an admin logs in or a new user is registered. This token is then dispatched to the client and utilized in future API calls.

Here is an illustration of the token generation process in the index.js file located in the server directory:

//...other code
const MASTER_TOKEN = process.env.MASTER_TOKEN;
app.post('/admin/login', function(req, res){
    //...other code
    const data = JSON.stringify("All");
    const config = {
        method: 'post',
        url: 'http://127.0.0.1:8080/auth/token',
        headers: {
            'Authorization': 'Bearer ' + MASTER_TOKEN,
            'Content-Type': 'application/json'
        },
        data : data
    };
//...other code

Furthermore, we can customize the access token to limit access based on the user's role:

//...other code
// Generate token with limited access based on user role
const data = JSON.stringify({
   //...other code
});
//...other code

In the code snippet above, the custom access token restricts access to only the need_to_Add_endpoint_name API endpoint. The access filter can be tweaked to include other endpoints and set specific permissions anchored on user roles.

3. Incorporating Tokens in Client-Side API Calls

Once the JWT tokens are generated, they can be utilized client-side for secure API access. This can be seen in the movies.js file and other components housed in the public directory.

//...other code
function Movies() {
    let [token, setToken] = useState(null);

    useEffect(() => {
        if (localStorage.getItem('public_jwt')) {
            setToken(localStorage.getItem('public_jwt'));
        } else {
            //...other code
        }
    }, []);
//...other code

Whenever the client makes an API call, the server validates the JWT token and checks if the user has the necessary permissions to access the requested endpoint. If the token is valid and the user will get the valid response.

Wrapping Up:

Using Dozer's powerful security capabilities, you can easily add authorization to your data APIs, ensuring that your application remains secure and accessible only to authorized users. Whether you're building a movie ticket app or any other application, Dozer provides a flexible and straightforward way to manage access control for your APIs.

The beauty of Dozer lies in its flexibility—it moulds itself according to your needs, allowing you to craft custom access controls for different user roles, thereby ensuring the right balance between functionality and security.

Through our movie ticket booking application, we've demonstrated just a fragment of the potential that Dozer holds. Regardless of your application's domain—be it e-commerce, social media, healthcare, or any other field—you can leverage the power of Dozer to secure your data APIs.

In conclusion, Dozer presents a robust, scalable, and straightforward solution to facilitate API security and data management. It's a testament to the flexibility of modern software tools and their ability to adapt to a wide range of application needs. We hope this guide has clarified the process of secure API management and inspired you to integrate Dozer into your next project.

Happy coding, Happy Data APIng! 🚀👩‍💻👨‍💻

ES Modules or CommonJS: What's the Deal and Why Should I Care ?

Abhishek Mishra — Wed, 10 May 2023 10:53:12 +0000

Today, I'm on a quest to untangle a little confusion that's been bothering me (as a JS noob 👶)- and maybe you, too.

It's all about JavaScript module systems: ES Modules and CommonJS. Funny enough, I stumbled upon this while trying out a library, @dozerjs/dozer, a JavaScript library to interact with Dozer, a data API backend.

Alright, here's the deal. Back in the day, JavaScript didn't have a built-in module system. So, dev's couldn't easily separate their code into different files or manage dependencies. It was kind of a mess. But then, Node.js stepped up with its own module system, CommonJS. And later, ES6 (ES2015) introduced ES Modules.

Now, you might be thinking, "Why should I, as a web dev, even care about this? 🤔" Well, it's simple. The module system you choose can affect your code's structure, how fast it runs, and even its compatibility. Trust me, I learned this the hard way when I faced some import issues while running a node app with @dozerjs/dozer. Let's dive in!

CommonJS: This is like the godfather of JavaScript module systems. It uses the require() function to import modules and module.exports to export them.

The cool part? You can use require pretty much anywhere - inside functions, loops, or based on some conditions.
The not-so-cool part? Imports are synchronous, meaning they could slow things down a bit.

Here's a little example:

// Importing a module
const express = require('express');

// Exporting a module
module.exports = someFunction;

ES Modules: These are the new kids on the block, coming in with ES6. Instead of require() and module.exports, they use import and export. These modules are statically analyzed, which means the imports and exports are determined before the code runs. This can make your code run faster and gives you some extra tooling perks.

Here's how it looks:

// Importing a module
import express from 'express';

// Exporting a module
export default someFunction;

So, which one do you choose? 🤹‍♀️**

If you're writing code for the browser, ES Modules are a great choice because modern browsers support them natively. For Node.js, it does support ES Modules, but you might still find yourself using CommonJS for compatibility reasons.

Got some older browsers or Node.js versions to support? They might not be fans of ES Modules. In that case, you could stick with CommonJS or use a tool like Babel to compile your ES Modules to CommonJS.

The trend is definitely leaning towards ES Modules, but there's still a ton of CommonJS code out there. So, it's a good idea to know your way around both.

Remember, the choice is yours. Pick the module system that fits your project and team the best.

A Quick Example: Switching to CommonJS in Node.js

I thought it might be handy to wrap this up with a little example. So, let's say you're working on a Node.js app and you've been using ES Modules. And if want to switch back to CommonJS. How do you do it?

Well, first, you'll need to change your import and export statements. Here's an example:

// ES Module import
import express from 'express';

// Change to CommonJS require
const express = require('express')

But wait, there's another step! You also need to update the type in your package.json file. If you have "type": "module", which allows you to use ES Modules, you'll want to remove it to switch back to CommonJS.

Here is the package.json example:

{
  "name": "dozer_lib_js_test",
  "version": "1.0.0",
  "main": "index.js",
  "license": "MIT",
  "type": "commonjs",
  "dependencies": {
    "@dozerjs/dozer": "^0.0.6"
  }
}

I hope this helps!

Announcing the Dozer Appathon!

Abhishek Mishra — Wed, 03 May 2023 05:59:55 +0000

👋🏼 Are you looking for a platform that simplifies the process of building and deploying data-driven applications? Look no further than Dozer ! 💻 is a powerful Data API backend that simplifies the process of building and deploying data-driven applications & APIs for real-time data processing and analysis, predictive analytics, data caching, and much more.

And now, you have the chance to showcase your coding skills and creativity by participating in the Dozer Appathon! 🚀 This exciting event is open to everyone who is interested in building applications with Dozer. The submission period runs from May 3 to May 9, 2023; so you still have time to get your app ready! ⏰

Dozer Appathon Hacker Guide 👩‍💻👨‍💻

Introduction 🚀

Welcome to the Dozer Appathon! This is your chance to showcase your coding skills and creativity by building amazing applications using our platform. Dozer is a powerful Data API backend that simplifies the process of building and deploying data-driven applications & APIs.

Sample use cases📝

To help you get started, we have prepared a list of sample use cases and ideas for your app. You can use these as inspiration for your project, or come up with your own unique ideas.

Some sample use cases are

- Real-time data processing and analysis
- Data aggregation and consolidation from multiple sources
- Machine learning and predictive analytics
- Data caching and optimization for faster performance
- IoT data processing and analysis

Some wonderful project ideas💡

Real-time Eth layer 2 crypto dashboard📈
NodeJS-based Twilio notifications with real-time alerts using Dozer📲
Real-time ML predictions using ONNX model🧠
React Native push notifications📱
Klarna clone with user profiles and transactions in React Native💳
Real-time vehicle tracking on Google Maps with traffic density alerts (Web or React Native)🚗
Real-time inventory dashboard for a retail chain with multiple warehouses🛍️
Fraud detection for credit card transactions using ML models on Python Lambda💳🕵️‍♂️
Model training and scoring with data injection through Python Lambda 🧑‍💻📈

Prizes 🎁

We have some exciting prizes up for grabs!

The top 5 submissions will win an amazing Mechanical Keyboard and a Dozer branded Swagpack 🏆
The next 10 successful submissions win Dozer Branded swagpack which includes an awesome tee and stickers. 🎉
Additionally, all participants will receive a certificate of participation.📜

Resources 📚

To help you build your app, we have some resources for you:

Website: https://getdozer.io/ 🌐
Docs: https://getdozer.io/docs/dozer 📖
Blogs: https://getdozer.io/blog 📰
Blog Tutorial: https://getdozer.io/blog/building-real-time-data-app-using-dozer-react-postgresql/ 📚
Video Tutorial: https://youtu.be/pVfyMfNrlD4 📹
Discord Community support: If you have any questions or need help with your project, you can reach out to #dozer-appathon channel of the Dozer Discord for support. 💬

Judging Criteria 🧐

We will rate as per optimal utilisation of Dozer in your app on a scale of 0-10, based on factors like real-time data, multiple data sources, and caching requirements. We'll also consider the problem you're solving and the code quality of your app.

Submission Method 📩

Submit a PR following the Contribution Guidelines with your project detail.
Fill out this form post PR submission.
Winners will be announced in the Discord Server.💪🏆

Stargazers’ Raffle✨

Star our repository and get a chance to win an amazing Swagpack!!🎁🎉
We will announce the winners in our Discord server. So, don’t forget to join.🤗

We're excited to see what you can create with Dozer!
Good luck and Happy Hacking!, Happy Data APIng 🚀 👩‍💻👨‍💻

For more details, checkout: https://getdozer.io/blog/dozer-appathon

Unleashing the Power of Dozer Lambda Runtime for Real-time and Event-driven Data Apps

Abhishek Mishra — Tue, 02 May 2023 14:21:50 +0000

Real-time data processing is essential for modern applications, as it ensures that insights are timely and accurate. The open-source project Dozer now provides a runtime environment that allows developers to execute lambda functions or custom user-defined functions based on specific data conditions.
This new functionality allows for more dynamic and event-driven applications, as Lambda functions can be triggered based on specific data change conditions. Just imagine triggering a Twilio notification based on a SQL statement (data change condition)!

In this blog post, we will discuss how you can use Dozer to respond to real-time data changes by triggering functions based on data conditions.
In this example, we will create a React app that generates a real-time graph based on room temperature data (room temperature monitoring). We will use the pydozer_log module to react to data change conditions and trigger a Twilio message if the room temperature goes above 22.9°C.

Dozer Lambda Runtime: A Glimpse

The Dozer Lambda runtime is a major enhancement to the Dozer project, enabling developers to write Lambda functions in JavaScript and Python that react to Dozer events.

To help you better understand the Dozer Lambda runtime, we will examine the following state diagram:

DCC: Data Change Condition

Data Source: Your application's data source, such as a database, sends data to the Dozer pipeline.
Dozer Pipeline: The data is processed through the Dozer pipeline, which can apply various transformations and aggregations based on the configuration you provide.
Lambda Runtime: The Dozer Lambda runtime monitors the pipeline for specific data change conditions that you define. If a data change condition is met, the corresponding Lambda function is triggered.
Action/Trigger: The triggered Lambda function performs the desired action, such as sending a Twilio notification or updating a database record.

The reponse of trigger can be used to write back top dozer, for example using gRPC service.

JavaScript and Python Bindings for Reading Dozer Logs

Dozer now provides bindings for both JavaScript and Python to read logs, making it easier for developers to work with the languages they are most comfortable with. These bindings are essential components of the Dozer Lambda runtime, as they enable developers to create Lambda functions that respond to
events.

React App Example: Room Temperature Monitoring

In this example, we will create a React app that generates a real-time graph based on room temperature data. We will use the pydozer_log module to react to data change conditions and trigger a Twilio message if the room temperature goes above 22.9°C.

Here are the steps to run the app:

Clone the Dozer Samples repository and install dependencies:

git clone https://github.com/getdozer/dozer-samples.git
cd dozer-samples/usecases/lambda/
pip install -r requirements.txt

Remove any existing Dozer configuration and start the Dozer:

rm -rf ./.dozer && dozer -c config/dozer-config.yaml

Run the Python scripts to read data from the pipeline:

Note: make sure to replace the Twilio configuration in reader.py, with your account id and auth token. To learn more about Twilio SMS setup checkout this getting started guide.

python reader.py
python main.py

Navigate to the React app folder, install the necessary packages, and start the development server:

cd react
yarn install && yarn start

Once the app is running, you should see a real-time graph of the room temperature data. If the temperature exceeds 22.9°C, a Twilio message will be triggered, notifying the relevant parties of the situation.

Conclusion 🎯

By using the Dozer Lambda runtime, you can create event-driven applications that automatically respond to changes in your data, making your applications more efficient, scalable, and responsive to user needs. By leveraging the Python and JavaScript libraries provided by Dozer, developers can easily build applications that respond to real-time data changes, making it an invaluable resource for modern applications.

Happy coding, Happy Data APIng! 🚀👩‍💻👨‍💻

🚀 Getting Started with Dozer with Docker: A Step-by-Step Tutorial

Abhishek Mishra — Mon, 24 Apr 2023 14:00:00 +0000

In the world of data-driven applications, accessing and manipulating data in real-time is crucial. Dozer is a powerful tool that helps you create low-latency data APIs (gRPC and REST) from any data source, enabling seamless integration with frontend applications. In this tutorial, we'll walk you through setting up Dozer with Docker and connecting it to PostgreSQL using Docker.

🌟 Why Dozer?

Dozer addresses the challenges of accessing and manipulating data in real-time by providing an easy-to-use solution to create efficient data APIs. With Dozer, you can quickly set up APIs that can be easily integrated with your frontend applications. Furthermore, using Docker allows you to containerize your application, making it portable and easy to manage.

In this step-by-step tutorial, we'll guide you through connecting PostgreSQL with Dozer using Docker. Let's get started!

🔧 Prerequisites:

Docker/Docker desktop installed on your machine
Basic knowledge of Docker and PostgreSQL

📝 Overall steps involved:

Create a docker-compose file for PostgreSQL
Create a Dozer config with PostgreSQL connector and API configuration
Run PostgreSQL (via Docker)
Run Dozer (via Docker)
Query your API

🔹 Step 1: Create a `docker-compose.yml` file

First, create a docker-compose.yml file and copy the following configuration to it or download the file here:

version: '3.9'
services:
  postgres:
    platform: linux/amd64
    container_name: quick-start-postgres
    image: public.ecr.aws/getdozer/dozer-samples-pg-stocks:latest
    command: postgres -c hba_file=/var/lib/stock-sample/pg_hba.conf
    environment:
      POSTGRES_DB: stocks
      POSTGRES_USER: postgres
      POSTGRES_PASSWORD: postgres
      ALLOW_IP_RANGE: 0.0.0.0/0
    ports:
      - '5434:5432'
    healthcheck:
      test: [ "CMD-SHELL", "pg_isready -U postgres  -h 0.0.0.0 -d stocks" ]
      interval: 5s
      timeout: 5s
      retries: 5

volumes:
  sample-data:

🔹 Step 2: Create a Dozer config file

Next, create a dozer-config.yaml file with the following content:

app_name: 1-hypercharge-postgres-sample
connections:
  - config: !Postgres
      user: postgres
      password: postgres
      host: host.docker.internal
      port: 5434
      database: stocks
    name: stocks
sources:
  - name: stocks
    table_name: stocks
    columns:
      - id
      - ticker
      - date
      - open
      - high
      - low
      - close
      - adj_close
      - volume
    connection: !Ref stocks
  - name: stocks_meta
    table_name: stocks_meta
    columns:
      - symbol
      - security_name
    connection: !Ref stocks
endpoints:
  - name: stocks
    path: /stocks
    table_name: stocks
    index:
      primary_key:
        - id
  - id: null
    name: stocks_meta
    path: /stocks-meta
    table_name: stocks_meta
    index:
      primary_key:
        - symbol

🔹 Step 3: Run PostgreSQL with sample stock data

Open a terminal or command prompt and navigate to the directory containing the docker-compose.yml file. Run the following command to start PostgreSQL with sample stock data:

docker-compose up

🔹 Step 4: Run Dozer with PostgreSQL connector (via Docker)

Open another terminal or command prompt and navigate to the directory containing the dozer-config.yaml file. To run Dozer as a Docker container, execute the following command:

docker run -it \
       -v "$PWD":/usr/dozer \
       -p 8080:8080 \
       -p 50051:50051 \
       --platform linux/amd64 \
       public.ecr.aws/k7k6x1d4/dozer \
       dozer

Congratulations 🥳 , you should see Dozer up and running!

🎉 Query your API

Dozer automatically generates APIs in both REST and gRPC formats, along with documentation, OpenAPI, and proto files. In the following sections, we'll showcase some possible queries. Refer to the APIs section for more details.

🔸 Query Using gRPC:

You can use gRPCurl or Postman to interact with gRPC APIs.

gRPC query example:

grpcurl -plaintext localhost:50051 dozer.generated.stocks_meta.StocksMetas/query

🔸 Query Using REST:

You can use curl or Postman to interact with REST APIs.

REST query example:

curl --location --request GET 'localhost:8080/stocks-meta'

That's it! You've successfully set up Dozer with Docker and connected it to PostgreSQL. Now you can easily create APIs for your data and integrate them into your frontend applications.

If you're interested in learning more about how to use Dozer with applications, we recommend reading our previous blog post: "Building a Real-Time Data App with Dozer, React, and PostgreSQL". In this blog, you'll find a detailed tutorial that demonstrates how to integrate Dozer with a React frontend application and PostgreSQL as a data source. This comprehensive guide will give you a better understanding of how to build powerful, real-time data-driven applications using Dozer.

Happy coding, Happy Data APIng! 🚀👩‍💻👨‍💻

Dozer Goes Open Source: Empowering the Community to Build Real-time Data Apps

Abhishek Mishra — Fri, 21 Apr 2023 15:56:07 +0000

We are excited to announce that Dozer, a powerful platform for building low-latency data APIs (gRPC and REST) from any data source, is now open source! The Apache 2.0 license 🎉. With this move, we aim to empower the community to build and scale real-time data applications more effectively.

Dozer simplifies the process of connecting applications to various data sources, such as PostgreSQL, Kafka, or other databases & sources, enabling developers to easily create and manage real-time data APIs. By making Dozer open source, we are inviting developers to contribute to its growth and help shape the future of real-time data applications.

What is Dozer?

Dozer is a platform that enables developers to build low-latency data APIs (gRPC and REST) from any data source. It simplifies the process of connecting applications to data sources like databases, APIs, or other services, and automatically manages data streaming and synchronization. This allows developers to focus on building powerful applications that utilize real-time data without worrying about the complexities of managing data connections and APIs.
With a few lines of SQL and a simple YAML configuration, you can build, deploy and maintain full data backends. It also has support for client libraries in JS and React to easily integrate with front-end applications.

Why Open Source?

There are several reasons why we decided to make Dozer open source:

Community-driven innovation: We want developers from all over the world to contribute their ideas,and provide valuable insights, improvements, and fixes, leading to a more innovative and robust platform, and faster development cycles.
Transparency and trust: We want to enable users to view and understand the underlying code, fostering trust in the platform and ensuring that it meets their needs and expectations. Especially how data is being stored and processed.
Collaboration and learning: We encourage developers to collaborate, share ideas, and learn from one another. We want to foster a strong community that helps developers grow their skills and expertise in the real-time data space.

Getting Started with Dozer

To help you get started with Dozer, we've created documentation and tutorial blogs that guide you through the process of setting up, configuring, and using Dozer for your data applications. You can find these resources on our official documentation site and can also follow Dozer on dev.to or additional resources on getting to know & use Dozer better!

How to Contribute?

We welcome contributions from the community! If you're interested in contributing to Dozer, please check out our GitHub repository for the guidelines. Whether you want to submit a bug report, suggest a new feature, or contribute code, we appreciate your help in making Dozer even better.

We also encourage you to check out the following resources to know other ways of contributions:

Show and tell GitHub discussion forum: We have setup a GitHub discussion forum, where you can share your Dozer projects, and experience, ask questions, do a feature request, and connect with fellow Dozer community members. Feel free to join the conversation and share your own projects & ideas! Additionally, we have also created a discord channel to give developers an opportunity to showcase and talk about their projects. Join our discord channel, If you’ve built something using Dozer, we’d love to see it!
Dozer Samples Repository: We have created a Dozer Samples repository on GitHub, which contains sample applications built using Dozer. This repository can be a good resource for learning how to use Dozer and getting inspiration for your projects. You can also contribute to the dozer-sample repository by submitting a pull request with your projects. This not only helps showcase your work but also provides good examples for other developers to learn from and take inspiration to build their own projects.

You can also join our community discord to see what we are cooking at Dozer :female-cook: :male-cook:

Conclusion

We believe that by making Dozer open source, we are empowering the developer community to build amazing real-time data applications. We look forward to seeing the project and ideas that arise from this collaborative effort, and we're excited to work together to shape the future of real-time data!
Together, we can shape the future of real-time data and contribute to the ongoing success of open source software. 🚀

Happy coding, Happy Data APIng! 🚀👩‍💻👨‍💻

Building a Real-time Data App with Dozer, React, and PostgreSQL

Abhishek Mishra — Wed, 19 Apr 2023 08:13:29 +0000

In today's data-driven world, having access to real-time data is crucial and gaining a value from the data is more cumbersome. Another challenge developers face when working with real-time data is efficiently and quickly fetching, processing, and displaying the insights from the data.

Enter Dozer, it simplifies the process of creating low-latency data APIs (gRPC and REST) from any data source easily with OpenAPI specification. Dozer is a real-time data engineering tool built with Rust, which enables seamless integration of data streams(sources), efficient data processing, and lightning-fast response times. This allows developers to focus on building applications without worrying about the complexity of managing real-time data.

To know more about it's capabilities and features you can checkout Dozer repository.

In this blog, we'll walk you through the process of creating a real-time flight data application using Dozer, React.
For this particular blog, we are using PostgreSQL as our data source, but it's important to note that there can be multiple data sources that can be connected to Dozer and subsequently used by your React application. By using Dozer, you can quickly create efficient data APIs that enable seamless integration with your frontend applications.
PostgreSQL is connected to Dozer via PostgreSQL connector and fetches data from multiple Postgres tables that has a flight data and combines them in real time based on the queries and produces fast READ APIs.

On the client side, React app is being used to display the flight data on map coming from Dozer. The dozerjs and dozer-react libraries are used to make it easy to interact with Dozer and access real-time data without having to hardcode API endpoints in the code.
The dozerjs library manages gRPC requests between the frontend and the Dozer app, while the dozer-react library provides React hooks for easy integration with React components.

So let's dive in and learn how to harness the power of Dozer to create amazing real-time data-driven applications! 💪✨

🐘 Setting Up PostgreSQL:

Clone the sample repository and navigate to the appropriate directory:

git clone https://github.com/getdozer/dozer-samples.git
cd dozer-samples/usecases/pg-flights

Checkout the react branch:

git checkout react

In the usecases/pg-flights/ directory, run the following command to start PostgreSQL using Docker Compose:

docker-compose up

This command will create a PostgreSQL container with the necessary configuration and sample flight data as shown below:

To test the connection to the PostgreSQL container, you can use psql or any PostgreSQL client. Here's an example using psql:

psql -h localhost -p 5437 -U postgres -d flights

This command connects to the flights database using the postgres user on the default port 5437. Once connected, you can run SQL queries to explore the sample flight data.

With PostgreSQL set up and running, we can now proceed to build the React application and configure Dozer.

⚛️ Building & Running the React App 🚀:

Open a new termincal and navigate to the /usecases/react directory in the cloned dozer-samples repository:

cd dozer-samples/usecases/react

This directory contains a sample React app that displays flight data on a map component.

Install the required packages for the app using yarn install:

yarn install

In the project directory, run the following command to start the development server:

yarn start

This command runs the app in development mode.

Open http://localhost:3000 to view it in your browser. The page will automatically reload when you make changes, and you may also see any lint errors in the console. On this page, you will the Dozer logo, now navigate to http://localhost:3000/airports to see the map UI for respective ariport and it's flight booking data as follow:

This React app uses the @dozerjs/dozer and @dozerjs/dozer-react libraries to interact with Dozer, enabling seamless integration and real-time data updates.

Since, we have not configured the Dozer with PostgreSQL yet, there is not data on the map.
Let' quickly setup the Dozer and see the magic happening!

🔌 Configuring Dozer and PostgreSQL Connector:

Start by installing Dozer. In for this blog, we'll use Homebrew for the installation. To learn about other installation methods, visit the official Dozer installation documentation.

open a new terminal, ensuring you're in the same directory as your React project, and run:

brew tap getdozer/dozer
brew install dozer

Once the installation is complete, create a dozer-config.yaml file in your project directory and paste the following configuration into it:

# dozer-config.yaml content

app_name: flight-microservices
connections:
  - name: flights_conn
    config: !Postgres
      user: postgres
      password: postgres
      host: 0.0.0.0
      port: 5437
      database: flights

sql: |
  select f.arrival_airport as airport, a.coordinates as coordinates, COUNT(t.ticket_no) as tickets
  INTO airports_count
  from tickets t
  join ticket_flights tf on t.ticket_no = tf.ticket_no
  join flights f on tf.flight_id = f.flight_id
  join airports a on f.arrival_airport = a.airport_code
  group by f.arrival_airport, a.coordinates;

  select extract(HOUR FROM f.window_start) as start, count(f.window_start) as dep_count
  INTO departures_count
  from TUMBLE(flights, scheduled_departure, '4 HOURS') f
  group by extract(HOUR FROM f.window_start)

sources:
  - name: tickets
    table_name: tickets
    columns:
    connection: !Ref flights_conn

  - name: flights
    table_name: flights
    columns:
    connection: !Ref flights_conn

  - name: ticket_flights
    table_name: ticket_flights
    columns:
    connection: !Ref flights_conn

  - name: airports
    table_name: airports
    columns:
    connection: !Ref flights_conn

  - name: airports_flights_schema
    table_name: airports
    columns:
    schema: flights_schema
    connection: !Ref flights_conn

endpoints:

  - name: tickets
    path: /bookings/tickets
    table_name: tickets
    index:
      primary_key:
        - ticket_no

  - name: flights
    path: /bookings/flights
    table_name: flights
    index:
      primary_key:
        - flight_id

  - name: airports
    path: /bookings/airports
    table_name: airports
    index:
      primary_key:
        - airport_code


  - name: ticket_flights
    path: /bookings/ticket_flights
    table_name: ticket_flights
    index:
      primary_key:
        - ticket_no
        - flight_id

  - name: airports_count
    path: /airports_count
    table_name: airports_count
    index:
      primary_key:
        - airport
        - coordinates

  - name: departures_count
    path: /departures_count
    table_name: departures_count
    index:
      primary_key:
        - start

This configuration defines the PostgreSQL connector and sets up the API endpoints that our React application will use to fetch data.
For more information about the PostgreSQL connector, refer to documentation.
Dozer supports multiple connectors, allowing you to integrate various data sources. Check out the list of available connectors for more options.

With the configuration in place, run Dozer in action by executing the following command:

dozer -c dozer-config.yaml

Now, you should see Dozer logs indicating that the endpoints is ready to be queried by the React app.

You can now access your application at http://localhost:3000/airports to see the real-time flight data populating on the map component as shown below:

Additionally, You can also query the API using gRPC and REST endpoints to test the data. In this case, you can use the REST endpoint using curl or Postman http://localhost:8080/airports/

Additionally, you can check the OpenAPI specification documentation that Dozer generates for endpoints by requesting http://localhost:<port>/<endpoint_path>/oapi as shown below.

🌟 Conclusion:

Congratulations! You've successfully built a real-time flight data app using Dozer, React, and PostgreSQL. This powerful combination enables you to create interactive and data-driven applications with ease.
Feel free to expand on this example and explore other data sources and connectors to enhance your app's capabilities. Enjoy exploring the application and customizing it further to suit your needs!

Happy coding, Happy Data APIng! 🚀👩‍💻👨‍💻

DEV Community: Abhishek Mishra

Build Voice AI in Python: Complete Speech-to-Text Developer Guide (2026)

TL;DR – Quick Integration Overview

Introduction: Why Voice Integration Matters

Understanding Pulse STT

Highlight Features

Getting Started: Authentication

Step 1: Get Your API Key

Step 2: Test Your Key

Authentication Header

Part 1: Transcribing Audio Files (REST API)

Endpoint

Query Parameters

Supported Languages (32+)

cURL Example

Python Example

Node.js Example

Example Response

Part 2: Real-Time Streaming (WebSocket API)

WebSocket Endpoint

Query Parameters

Python Streaming Example

Node.js Streaming Example

WebSocket Response Format

Key Response Fields

Part 3: Advanced Features

Speaker Diarization

Emotion Detection

PII/PCI Redaction

Age and Gender Detection

Comparing STT Providers

Best Practices

Audio Quality

Error Handling

Rate Limiting

Bonus: Full Demo Application

Quick Start

How It Works

Project Structure

Scripts

Conclusion

Resources

How to do efficient fine-Tuning for LLMs using SLoRA

What is SLoRA?

How SLoRA Works?

Comparison with Traditional Fine-Tuning Methods

Benefits:

How to use SLoRA in Your Projects?

SLoRA's Secret Sauce:

Conclusion

Don't Be That Dev: Avoiding the XY Problem Trap

Understanding Strings in Rust: String vs str

What are String and str?

Why does Rust have two string types?

Example

Conclusion

Creating a Low-Latency Cache with Dozer and PostgreSQL

Why Do We Need a Cache on Top of PostgreSQL?

What's the Problem with Not Having a Cache?

Why Not PostgreSQL Read Replica?

Why Dozer?

How to Use Dozer as a Cache

Architecture Diagram

Conclusion

Secure Your Data APIs with Dozer: Adding Authorization to APIs

TL;DR

Introduction

Server-Side Setup (Express.js)

Database Connection for Server-Side

Admin Login Endpoint

Public New User Endpoint

Public Book Movie Endpoint

Error Handling Middleware

Database Setup For Movie data (PostgreSQL)

Front-End Setup (React)

Login Component

Admin Dashboard Component

Movie List Component

Movies Component

Integrating Dozer JWT in the Application

What are `String` and `str`?

🔹 Step 1: Create a `docker-compose.yml` file