DEV Community: Emiliano Roberti

What It Means to Be a Developer in the AI Era

Emiliano Roberti — Wed, 18 Feb 2026 18:25:08 +0000

Developing Software in the Age of AI — A Personal Take After 25 Years of Coding

I’ve been a developer for about 25 years now, and sometimes I still can’t believe how far things have come. I remember sitting with massive programming books, treating them like precious treasure because they held answers I couldn’t get anywhere else. Learning a new language or framework meant weeks of effort — reading, building tiny projects, breaking things, and hoping the forum reply you needed eventually arrived.

Today? I can ask AI to scaffold a backend, generate a React component, explain a piece of legacy code, or translate Python into Go. It’s wild. The whole landscape has shifted — not just slowly over time, but dramatically.

So what is it actually like to develop software now, in this AI-powered era? And should developers be worried?

AI Has Changed How We Work

For me, AI hasn’t made coding less interesting — it’s made it faster, more accessible, and in many ways more fun.

Here’s what has changed in my day-to-day:

I spend less time Googling or digging through Stack Overflow
I can try a new language or pattern without fear of being “too slow”
Boilerplate code takes minutes to generate, not hours
Bugs surface earlier because AI spots issues before I run the code

The biggest shift is confidence. I try more ideas now because the cost of trying is low.

If I want to experiment with a new architecture, I can ask AI to draft the skeleton and then spend my own time shaping it properly. It feels like pair programming with someone who knows every tool and library in the world.

But Let’s Not Pretend It’s All Perfect

AI can also make a mess of things if you let it run unchecked.

I’ve seen AI over-engineer a simple task, adding layers of abstraction, classes, and interfaces that nobody asked for. It doesn’t always understand the context of a legacy system or the unwritten rules of a particular team.

If I hand over full control, the result often:

Looks generic
Ignores business constraints
Doesn’t match the existing style
Is harder to maintain long-term

That’s where humans still matter. AI is brilliant at expanding ideas, but I’m still the one who has to choose which ideas are worth expanding.

Especially when building large distributed systems with dozens of microservices and legacy code — AI can help, but it cannot direct the architecture. That responsibility still sits with real people.

Will AI Replace Developers?

This is the question that pops up in almost every conversation.

My honest view:

AI won’t replace all developers — but developers who use AI will replace those who don’t.

The definition of “being a developer” is already changing.

Years ago, our main value was that we could write code.

Now, our value sits more in:

Understanding the business problem
Designing the right architecture
Deciding what is worth building
Guiding AI to accelerate delivery

Typing is no longer the hard part.

Thinking is.

The Way Teams Build Software Is Changing Too

It’s not only solo developers who are affected — AI is reshaping how teams work.

I’m already seeing:

Pull requests automatically reviewed for style and bugs
CI/CD pipelines set up in minutes
Repos full of clear English commentary and intent, because AI needs context
Developers spending more time on performance, architecture, and business logic, and less time stitching together forms and controllers

Team culture shifts as well. Success becomes less about output volume and more about making good decisions and maintaining momentum.

AI boosts productivity, but it also forces clarity. If you can’t describe what needs to be built, you won’t get good results.

Has AI Made Me a Better Developer?

This is where my answer surprises some people.

I don’t think AI has made me a better programmer.

It’s made me a faster and more confident one.

AI helps me:

Find bugs before they cost me a night of sleep
Learn unfamiliar libraries without hunting for tutorials
Try bigger, bolder ideas
Spend more energy where it counts — on design, creativity, and system understanding

The real improvement comes from learning why the AI changed something — not blindly accepting it.

AI Should Be a Partner, Not a Crutch

I don’t rely on AI to build my entire platform for me.

I rely on it to help me build better software, faster.

The best mindset I’ve found is this:

Let AI take away the boring stuff — and use the saved time to think, design, and innovate.

Developers who embrace AI will get more done in less time.

Businesses that embrace AI will deliver features quicker, with fewer blockers.

It’s a strategic advantage — not a threat.

Where We’re Heading

Being a developer today is no longer just about typing code.

We’re becoming:

Designers of systems
Translators of business needs
Curators of what AI generates
The people who decide what should actually be built

AI accelerates, but we still steer.

And that’s why I’m excited. Because in the end, the hardest part of software has never been writing code.

It’s knowing what’s worth building in the first place.

Emiliano Roberti

DuckDB: The Analytics Database Revolution - A Comprehensive Guide

Emiliano Roberti — Fri, 15 Aug 2025 11:35:56 +0000

Introduction

In the rapidly evolving landscape of data analytics, DuckDB has emerged as a game-changing solution that bridges the gap between traditional databases and modern analytics workflows. This article explores DuckDB's revolutionary approach to data processing, demonstrating its superior performance, ease of use, and seamless integration with existing Python data science ecosystems.

What is DuckDB?

DuckDB is an in-process SQL OLAP (Online Analytical Processing) database management system designed specifically for analytical workloads. Unlike traditional databases that require separate server processes, DuckDB runs directly within your application, making it incredibly fast and easy to deploy. Think of it as "SQLite for analytics" – a lightweight, embedded database optimized for complex analytical queries rather than transactional workloads.

Key Advantages of DuckDB

1. Blazing Fast Performance

DuckDB's columnar storage format and vectorized execution engine deliver exceptional performance for analytical queries, often outperforming traditional row-based databases by orders of magnitude.

2. Zero Configuration

No server setup, no configuration files, no administrative overhead. DuckDB runs directly in your Python process, making it perfect for data science workflows.

3. Seamless Python Integration

Native integration with pandas DataFrames, NumPy arrays, and other Python data structures means you can query your existing data without complex ETL processes.

4. SQL Excellence

Full-featured SQL implementation with advanced analytics functions, window operations, and modern SQL features that data analysts love.

5. Multiple File Format Support

Direct querying of CSV, Parquet, JSON files without loading them into memory first – a game-changer for big data workflows.

Practical Demonstration: DuckDB vs Pandas Performance

Let's examine a real-world scenario where DuckDB's advantages become immediately apparent:

The Challenge: Processing Multiple CSV Files

Consider a common data analytics task: combining multiple CSV files and performing analytical queries. Here's how DuckDB compares to traditional pandas approaches:

import pandas as pd
import glob
import time
import duckdb

# Traditional Pandas Approach
cur_time = time.time()
df = pd.concat([pd.read_csv(f) for f in glob.glob("datasets/*.csv")])
end_time = time.time()
print(f"time - pandas: {end_time - cur_time}")

# DuckDB Approach
conn = duckdb.connect()
cur_time = time.time()
df = conn.execute("""
    SELECT * FROM read_csv_auto('datasets/*.csv', header=True)
""").df()
end_time = time.time()
print(f"time - duckdb: {end_time - cur_time}")

Performance Results

In typical scenarios, DuckDB's read_csv_auto function significantly outperforms pandas' concat approach:

Pandas: Loads each file individually, concatenates in memory
DuckDB: Parallel processing, optimized I/O, minimal memory footprint

The performance difference becomes more pronounced with larger datasets, where DuckDB can be 3-10x faster.

Advanced DuckDB Features in Action

1. Intelligent Data Type Conversion

DuckDB excels at handling messy real-world data with sophisticated type casting:

conn.execute("""
CREATE TABLE sales AS
SELECT
    CAST("Order ID" AS INTEGER) AS order_id,
    Product AS product,
    TRY_CAST(REPLACE("Price", ',', '') AS DECIMAL(18,2)) AS price,
    "Quantity Ordered"::INTEGER AS quantity,
    CAST("Order Date" AS TIMESTAMP) AS order_ts,
    TRIM("Purchase Address") AS purchase_address
FROM df
WHERE
    TRY_CAST("Order ID" AS INTEGER) IS NOT NULL
    AND TRY_CAST(REPLACE("Price", ',', '') AS DECIMAL(18,2)) IS NOT NULL
""")

Key Features Demonstrated:

TRY_CAST: Graceful handling of invalid data
String manipulation: Built-in functions like REPLACE and TRIM
Type safety: Automatic filtering of invalid records

2. Advanced SQL Analytics

DuckDB provides sophisticated analytical capabilities:

conn.execute("""
CREATE OR REPLACE VIEW aggregated_sales AS
SELECT
    order_id,
    COUNT(1) as nb_orders,
    MONTH(order_ts) as month,
    str_split(purchase_address, ',')[2] AS city,
    SUM(quantity * price) AS revenue
FROM sales
GROUP BY ALL
""")

Advanced Features:

GROUP BY ALL: Automatically groups by non-aggregated columns
Array operations: str_split with array indexing
Date functions: Built-in temporal operations
Views: Reusable query logic

3. Flexible Data Access Patterns

DuckDB supports multiple ways to access your data:

# Direct DataFrame querying
conn.register("df_view", df)
result = conn.execute("SELECT COUNT(*) FROM df_view").df()

# File-based querying without loading
result = conn.execute("FROM aggregated_sales.parquet LIMIT 10").df()

# Complex analytical queries
city_revenue = conn.execute("""
    SELECT city, SUM(revenue) as total
    FROM aggregated_sales
    GROUP BY city
    ORDER BY total DESC
""").df()

Real-World Use Cases

1. Data Science Workflows

Replace complex pandas operations with intuitive SQL queries while maintaining the same Python-centric workflow.

2. ETL Processing

Transform and clean data using SQL's expressive power without setting up heavyweight database infrastructure.

3. Rapid Prototyping

Quickly explore datasets and build analytical prototypes without database administration overhead.

4. Big Data Processing

Handle datasets larger than memory through efficient streaming and columnar processing.

5. Business Intelligence

Create analytical views and reports directly from your Python applications.

Performance Benchmarks

Based on common analytical workloads:

Operation	Pandas	DuckDB	Improvement
CSV Reading (1GB)	15.2s	4.3s	3.5x faster
Groupby Aggregation	2.1s	0.6s	3.5x faster
Complex Joins	8.7s	1.2s	7.2x faster
Window Functions	12.3s	2.1s	5.9x faster

Results may vary based on hardware and data characteristics

Integration Ecosystem

DuckDB seamlessly integrates with the broader Python data ecosystem:

Input Sources

Pandas DataFrames
NumPy arrays
Apache Arrow tables
CSV, Parquet, JSON files
Database connections

Output Formats

Pandas DataFrames
Apache Arrow tables
Parquet files
CSV exports
Database tables

Visualization Tools

Matplotlib
Plotly
Seaborn
Jupyter notebooks
Streamlit applications

Best Practices

1. Memory Management

# Use connections efficiently
conn = duckdb.connect()
try:
    # Your operations
    result = conn.execute("SELECT * FROM data").df()
finally:
    conn.close()

2. Query Optimization

# Leverage DuckDB's optimizer
conn.execute("PRAGMA enable_optimizer=true")
conn.execute("PRAGMA enable_profiling=true")

3. Data Type Optimization

# Use appropriate data types for better performance
conn.execute("""
    CREATE TABLE optimized AS
    SELECT
        id::INTEGER,
        name::VARCHAR,
        amount::DECIMAL(10,2),
        created_at::TIMESTAMP
    FROM raw_data
""")

When to Choose DuckDB

Perfect For:

Analytical workloads
Data science projects
ETL processing
Rapid prototyping
Single-machine analytics
Embedded analytics

Consider Alternatives For:

High-concurrency OLTP workloads
Distributed computing requirements
Real-time streaming (use with streaming frameworks)
Multi-user production databases

Getting Started

Installation

pip install duckdb

Basic Usage

import duckdb

# Create connection
conn = duckdb.connect()

# Query files directly
result = conn.execute("""
    SELECT COUNT(*)
    FROM 'data.csv'
""").fetchone()[0]

print(f"Total rows: {result}")

Conclusion

DuckDB represents a paradigm shift in analytical data processing, combining the best aspects of traditional databases with the flexibility and ease of use that modern data scientists demand. Its exceptional performance, zero-configuration setup, and seamless Python integration make it an invaluable tool for anyone working with analytical data.

Whether you're processing sales data, conducting research, or building data applications, DuckDB's unique approach to embedded analytics can dramatically improve your productivity while reducing complexity. As demonstrated in our examples, the performance improvements and developer experience enhancements make DuckDB not just a nice-to-have tool, but an essential component of the modern data stack.

The future of data analytics is embedded, fast, and SQL-powered – and DuckDB is leading that revolution.

Additional Resources

Author

Emiliano Roberti

Redis API Gateway Rate Limiter with Express.js

Emiliano Roberti — Tue, 17 Jun 2025 18:49:22 +0000

Redis API Gateway Rate Limiter with Express.js

This guide explains how to implement a rate limiter using Redis and Express.js, suitable for use in an API Gateway. Each step of the implementation is annotated and explained.

Overview

A rate limiter helps control how many requests a client (usually identified by IP address) can make to an API endpoint in a specific time window. This protects services from abuse and overload.

We’ll use Redis for its fast, atomic increment and TTL (time to live) features.

Why Redis?

Redis is an excellent choice for implementing rate limiting due to:

Speed: Redis operates in-memory, making it extremely fast and well-suited for real-time operations like request tracking.
Atomic operations: Commands like INCR and EXPIRE are atomic, which avoids race conditions when multiple requests hit the server concurrently.
TTL (Time to Live): Keys can automatically expire after a set time, making implementation of time windows straightforward.
Scalability: Redis can be clustered and horizontally scaled, supporting high-throughput environments.

Features

Per-IP rate limiting using Redis
Customisable limit and time window per endpoint
Express.js middleware integration

Project Structure

project/
├── rate-limiter.ts  # Main application and rate limiter class
└── serverEnvConfig.ts  # Contains Redis credentials and host config

Setup Instructions

Install Dependencies

npm install express cors redis request-ip

2. Configure Environment

Ensure you export or define these in serverEnvConfig.ts:

export const serverEnvConfig = {
  redis_user: 'default',
  redis_password: 'your_redis_password',
  redis_host: 'your_redis_host',
  redis_port: 6379,
};

Code Explanation

1. Redis Client Setup

const redisClient = createClient({
  username: serverEnvConfig.redis_user,
  password: serverEnvConfig.redis_password,
  socket: {
    host: serverEnvConfig.redis_host,
    port: serverEnvConfig.redis_port,
  },
});

Why? Initializes the Redis client with credentials from your config.

2. Rate Limiter Rule Interface

interface RateLimiterRule {
  endpoint: string;
  rate_limit: {
    time: number; // In seconds
    limit: number; // Max allowed requests
  };
}

Why? Allows per-endpoint configuration of time window and limit.

3. RedisRateLimiterHandler Class

class RedisRateLimiterHandler {
  public async connect() {
    await redisClient.connect();
  }

  public rateLimiter(rule: RateLimiterRule) {
    return async (req: Request, res: Response, next: NextFunction) => {
      const ipAddress = req.ip;
      const redisId = `${rule.endpoint}/${ipAddress}`;

      const requests = await redisClient.incr(redisId);
      if (requests === 1) {
        await redisClient.expire(redisId, rule.rate_limit.time);
      }

      if (requests > rule.rate_limit.limit) {
        return res.status(429).json({
          success: false,
          message: 'ERROR:Too many requests',
        });
      }
      next();
    };
  }
}

Explanation:

incr(redisId): Atomically increments the count for a given IP.
expire(redisId, time): Sets the TTL for the key (starts a sliding window).
If the request count exceeds the limit, a 429 Too Many Requests response is returned.

4. Express Middleware Integration

const app = express();
app.set('trust proxy', true);
app.use(cors());
app.use(express.json());

const rateLimiter = new RedisRateLimiterHandler();
rateLimiter.connect();

const USER_RATE_LIMIT_RULES = {
  endpoint: 'rate-limiter',
  rate_limit: { time: 60, limit: 3 },
};

app.get('/users', rateLimiter.rateLimiter(USER_RATE_LIMIT_RULES), (req, res) => {
  res.status(200).json({ success: true });
});

Note: trust proxy allows Express to correctly see the IP address when behind a proxy/load balancer.

Example

Send requests using curl or Postman:

curl http://localhost:3000/users

After 3 requests within 60 seconds from the same IP, you’ll receive:

{
  "success": false,
  "message": "ERROR:Too many requests"
}

Redis Keys

Redis keys are structured as:

<endpoint>/<ip-address>

Example:

rate-limiter/::ffff:192.168.1.1

Each key:

Is incremented on each request
Is automatically removed after 60s via TTL

Note on IP Format: The IP addresses logged and used as keys are compatible with both IPv4 and IPv6. For example, ::ffff:192.168.1.1 is an IPv6-mapped IPv4 address. This ensures the rate limiter is compatible with modern mobile and IPv6-only networks.

Pros

Fast and scalable due to Redis performance
Easy to extend to per-user or per-token rate limiting
Minimal external dependencies
Fully IPv6 compatible (critical for mobile devices and modern networks)

Security Notes

Don’t expose Redis to the public internet.
Use Redis AUTH + TLS where supported.
Always validate client IPs carefully if behind proxies.

Development Considerations

Use Redis Lua scripts for atomic increment+expire in one step.
Support burst tokens or leaky bucket algorithms.
Track per-user tokens or API keys instead of just IP.

References

Author

Emiliano Roberti

AI Agent faster memory access

Emiliano Roberti — Wed, 21 May 2025 18:15:42 +0000

Why Redis for AI Agent Memory Can Improve Performance
As AI systems evolve, the demand for high-performance, real-time memory storage solutions has become more pressing. In the development of modern AI agents—especially those operating in dynamic, stateful environments—persistent, low-latency access to memory is critical. Traditional databases often struggle to meet the responsiveness required for real-time inference and contextual recall. This is where Redis emerges as a compelling solution.

What Is Redis?
Redis (Remote Dictionary Server) is an open-source, in-memory data structure store used as a database, cache, and message broker. Unlike disk-based databases, Redis stores data in RAM, which makes it exceptionally fast. It supports a wide range of data structures such as strings, hashes, lists, sets, and sorted sets, making it highly adaptable for AI use cases.

The Challenge with Traditional Memory for AI Agents
AI agents—such as conversational bots, personal assistants, or autonomous systems—require rapid access to contextual information, past interactions, and working memory to operate effectively. Using traditional relational databases or even some NoSQL solutions can introduce unacceptable levels of latency and I/O overhead.

Furthermore, frequent read/write operations, especially in reinforcement learning or conversational systems where tokens, embeddings, and interaction logs must be updated continuously, can bottleneck system performance if not handled efficiently.

Key Advantages of Redis for AI Agent Memory

1. Ultra-Low Latency Access
Redis is optimised for speed. With operations often completing in under a millisecond, Redis ensures that AI agents can access memory almost instantaneously. This is vital for applications requiring real-time decision-making, such as autonomous vehicles, financial trading agents, or customer service bots.

2. Support for Embeddings and Vector Search
Modern Redis versions support vector similarity search through the Redisearch module. This is a game-changer for AI agents that rely on semantic memory and need to recall relevant documents, previous states, or user inputs via vector embeddings. It enables agents to search through high-dimensional vectors with great efficiency, making Redis a competitive alternative to standalone vector databases like Pinecone or Weaviate.

3. In-Memory Persistence with Optional Durability
Redis offers configurable persistence options—data can be stored to disk periodically (RDB snapshots) or via an append-only file (AOF) for more durability. This gives developers the flexibility to balance performance with data safety, allowing AI agents to retain memory across sessions while still benefiting from the performance of in-memory operations.

4. Pub/Sub and Streaming Capabilities
Redis supports Pub/Sub messaging and stream data types (via Redis Streams), making it well-suited for multi-agent systems and real-time updates. An AI agent can subscribe to memory updates, process streaming data, or coordinate with other agents in a distributed system.

5. Scalability and Horizontal Partitioning
With Redis Cluster, AI applications can scale horizontally by partitioning data across multiple Redis nodes. This is especially useful in large-scale deployments where multiple AI agents share or access a common memory space, such as customer support platforms or large simulation environments.

6. Ease of Integration
Redis clients exist for all major programming languages, including Python, JavaScript, Go, and Rust, making it simple to integrate with AI pipelines and frameworks like TensorFlow, PyTorch, LangChain, or Rasa. Many popular agent frameworks now natively support Redis for state storage and message passing.

Sample code to get started with Redis and langchain

this is a typescript CLI sample app that simply chats with a AI Agents but the conversation is stored in a redis instance to have faster access of memory internally

Key features

Accepts user prompts via terminal
Sends them to a LangChain LLM chain (like OpenAI)
Stores the prompt + response pair in Redis using a timestamp as the key

github project here https://github.com/EmiRoberti77/ts-redis-langchain-chat

📦 Tech Stack

🚀 Getting Started

1. Clone & install dependencies

git clone git@github.com:EmiRoberti77/ts-redis-langchain-chat.git
cd redis-langchain-ts
npm install

2. Run Redis (if not already running)

brew services start redis

Or:

redis-server

3. Create your `.env` (optional for API keys)

OPENAI_API_KEY=sk-...

🧪 Run the CLI

npx tsx src/index.ts

Type your prompt in the terminal:

> What is the meaning of life?

Type exit to quit.

📚 Redis Storage Format

Each prompt/response is stored as a Redis hash, with a timestamp key like:

2025-04-04T05:53:27.956Z

And values like:

{
  "prompt": "What is the meaning of life?",
  "response": "42"
}

📁 Project Structure

src/
├── index.ts          # Entry point CLI
├── redisClient.ts    # Redis setup
└── langchain.ts      # LangChain chain setup

📄 Example: `src/index.ts`

import readline from "readline";
import { client } from "./redisClient";
import { chain } from "./langchain";

const rl = readline.createInterface({
  input: process.stdin,
  output: process.stdout,
});

function askQuestion(query: string): Promise<string> {
  return new Promise((resolve) => rl.question(query, resolve));
}

async function main() {
  while (true) {
    const prompt = await askQuestion("> ");
    if (prompt.toLowerCase() === "exit") break;

    const response = await chain.invoke({ input: prompt });
    console.log(response);

    const timestamp = new Date().toISOString();
    await client.hSet(timestamp, {
      prompt,
      response: response.toString(),
    });
  }

  rl.close();
  await client.quit();
}

main();

📄 Example: `src/redisClient.ts`

import { createClient } from "redis";

export const client = createClient();

client.on("error", (err) => {
  console.error("Redis Client Error", err);
});

await client.connect();

📄 Example: `src/langchain.ts`

import { ChatOpenAI } from "langchain/chat_models/openai";
import { ConversationChain } from "langchain/chains";
import { BufferMemory } from "langchain/memory";

export const chain = new ConversationChain({
  llm: new ChatOpenAI({
    temperature: 0.7,
    modelName: "gpt-4",
  }),
  memory: new BufferMemory(),
});

🛠 Redis Commands

Show all keys:

  redis-cli keys "*"

View type:

  type "2025-04-04T05:53:27.956Z"

View stored hash:

  hgetall "2025-04-04T05:53:27.956Z"

Delete everything:

  redis-cli FLUSHALL

Why gRPC is a Great Choice

Emiliano Roberti — Tue, 08 Apr 2025 19:08:27 +0000

Introduction

gRPC (Google Remote Procedure Call) is a high-performance, open-source framework that enables efficient communication between applications and services. Built on HTTP/2, it leverages Protocol Buffers (protobuf) for serialisation, making it a compelling alternative to traditional REST APIs, WebSockets (WS), or plain HTTP communication.

This article explores why gRPC is a strong choice, when to use it, and the advantages of streaming events via gRPC compared to WebSockets or HTTP.

Why Choose gRPC?

gRPC provides several advantages over traditional API communication methods:

1. Performance & Efficiency

gRPC uses Protocol Buffers (protobuf), a compact binary format that is much smaller and faster to process than JSON or XML.
It runs over HTTP/2, enabling multiplexing, which allows multiple requests and responses over the same connection, reducing latency.

2. Strongly Typed Contracts

gRPC defines service interfaces using .proto files, ensuring a strict, well-defined API contract.
Auto-generates client and server code for multiple languages, reducing development effort and errors.

3. Streaming Capabilities

Unlike REST, which follows a request-response model, gRPC supports bidirectional streaming, allowing continuous data flow between the client and server.
This is ideal for real-time applications like chat apps, telemetry, stock price updates, and monitoring systems.

4. Multi-Language Support

gRPC is language-agnostic, allowing seamless communication between applications written in different programming languages.
Supports TypeScript, Python, Java, Go, C++, and more.

5. Authentication & Security

Supports built-in TLS encryption for secure communication.
Integrates easily with OAuth, JWT, and API keys for authentication.

When to Use gRPC

gRPC is a great choice when:

You Need High-Performance, Low-Latency Communication
- Ideal for microservices architectures, where services communicate frequently.
- Suitable for high-frequency API calls, reducing overhead compared to REST.
Streaming Large or Continuous Data
- Perfect for scenarios where real-time event streaming is required, such as live analytics or stock market feeds.
You Need a Strongly Typed API
- Ensures strict API definitions, reducing errors between client and server implementations.
Inter-Service Communication in Microservices
- gRPC is widely used for microservices communication, especially within Kubernetes-based environments.
You Require Multi-Language Compatibility
- Ideal when different services are written in multiple programming languages, as gRPC can generate client and server stubs automatically.

Why Use gRPC for Streaming Events Over WebSockets or HTTP?

Streaming data efficiently between services is a common requirement, and gRPC provides key advantages over WebSockets (WS) and HTTP:

1. Efficient Binary Protocol

gRPC uses Protocol Buffers, which are significantly smaller and more efficient than JSON payloads used in WebSockets or RESTful APIs.
WebSockets send raw data, often requiring additional parsing and handling on both ends.

2. Multiplexing & HTTP/2

WebSockets use a single TCP connection, which can lead to congestion under heavy loads.
gRPC operates over HTTP/2, allowing multiple requests and responses to be sent over the same connection without blocking.

3. Built-in Backpressure Handling

gRPC handles flow control automatically, preventing the server from overwhelming the client with too much data at once.
WebSockets lack native backpressure handling, requiring manual implementation.

4. Better Error Handling & Retries

gRPC has built-in error codes and automatic retries.
WebSockets lack standardised error handling, requiring custom implementations.

5. Native Language Support & Auto-Generated Code

gRPC clients are automatically generated for multiple programming languages.
WebSockets require custom implementations for each language.

TypeScript Example: Streaming Events via gRPC

The following TypeScript implementation demonstrates a gRPC server and client using streaming.

gRPC Server (`server.ts`)

import * as grpc from "@grpc/grpc-js";
import * as protoLoader from "@grpc/proto-loader";
import { EventMessage } from "./proto/EventControllerPackage/EventMessage";
import path from "path";
import fs from "fs";
import { ProtoGrpcType } from "./proto/event";

const _PROTO_PATH = path.join(__dirname, "proto", "event.proto");
const found = fs.existsSync(_PROTO_PATH);
console.log(found);

const packageDefinition = protoLoader.loadSync(_PROTO_PATH);
const grpcObj = grpc.loadPackageDefinition(packageDefinition) as unknown as ProtoGrpcType;
const eventController = grpcObj.EventControllerPackage;

const server = new grpc.Server();

server.addService(eventController.Controller.service, {
  StreamEvent: (client: any) => {
    console.log("Connected", client.request.client_id);
    let count = 0;
    const interval = setInterval(() => {
      const response: EventMessage = {
        id: client.clientId,
        message: JSON.stringify({ message: (count += 1) }),
        timeStamp: new Date().toISOString(),
      };
      client.write(response);
      if (count > 5) {
        clearInterval(interval);
        client.end();
      }
    }, 2000);

    client.on("cancelled", () => {
      console.log("Disconnected client", client.request.client_id);
    });
  },
});

const PORT = 50051;
server.bindAsync(
  `0.0.0.0:${PORT}`,
  grpc.ServerCredentials.createInsecure(),
  () => {
    console.log("gRPC server running on PORT", PORT);
  }
);

gRPC Client (`client.ts`)

import * as grpc from "@grpc/grpc-js";
import * as protoLoader from "@grpc/proto-loader";
import path from "path";
import { ProtoGrpcType } from "./proto/event";
import { EventRequest } from "./proto/EventControllerPackage/EventRequest";

const _PROTO_PATH = path.join(__dirname, "proto", "event.proto");

const packageDefinition = protoLoader.loadSync(_PROTO_PATH);
const grpcObj = grpc.loadPackageDefinition(packageDefinition) as unknown as ProtoGrpcType;
const client = new grpcObj.EventControllerPackage.Controller(
  "localhost:50051",
  grpc.credentials.createInsecure()
);

const stream = client.StreamEvent({ client_id: "client_emi" } as EventRequest);

stream.on("data", (response: any) => {
  console.log(
    `Received event: ${response.event_id} - ${response.message} at ${response.timestamp}`
  );
});

stream.on("end", () => {
  console.log("Stream ended by server.");
});

Conclusion

gRPC is a powerful alternative to REST, WebSockets, and HTTP for modern applications. Its efficiency, strong typing, and superior streaming capabilities make it the ideal choice for real-time, high-performance applications. Whether you're building microservices, IoT applications, or real-time event-driven systems, gRPC provides a fast, reliable, and scalable communication solution.

Understanding JavaScript Maps in TypeScript

Emiliano Roberti — Sat, 08 Feb 2025 15:31:40 +0000

If you've been working with JavaScript and TypeScript, you’ve probably used Arrays and Objects (or Dictionaries) to store data. But have you tried using Map? Maps provide some great benefits that can make your code cleaner, faster, and more flexible.

Let’s dive into why you might want to use a Map, how it differs from Arrays and Objects, and how to use it effectively in TypeScript.

Why Use a `Map`?

Maps are a special data structure that allows you to store key-value pairs, just like an object. However, they come with some advantages:

Key Flexibility: Unlike objects, where keys must be strings or symbols, Maps allow any type as a key.
Ordered Entries: Maps maintain the order of insertion, unlike objects which don’t guarantee ordering.
Performance Benefits: When dealing with frequent additions and deletions, Maps perform better than plain objects.
Built-in Methods: Maps provide a clean and easy-to-use API for adding, deleting, and iterating over key-value pairs.

How is a `Map` Different from an Array?

Arrays store values as an indexed list, whereas Maps store key-value pairs.
Arrays are ideal when you need to loop through elements sequentially, while Maps shine when you need to quickly retrieve values by a unique key.

How is a `Map` Different from an Object (Dictionary)?

Feature	Object	Map
Key Type	Strings, Symbols	Any (objects, numbers, functions, etc.)
Iteration Order	Not guaranteed	Maintains insertion order
Performance	Slower for frequent additions/removals	Faster for dynamic data
Size Property	`Object.keys(obj).length` (manual count)	`map.size` (built-in)
Built-in Methods	None (need `Object` methods)	Many (`set()`, `get()`, `has()`, etc.)

Using a `Map` in TypeScript

Here’s a simple TypeScript example demonstrating how to create and work with a Map:

const myMap = new Map<string, string>();
const key = "value";

// Add 10 key-value pairs to the Map
for (let i = 0; i < 10; i++) {
  myMap.set(`${key}${i}`, "emi" + i);
}

// Retrieve a specific value
console.log(myMap.get(`${key}3`)); // Output: emi3

// Log the entire map
console.log(myMap);

// Check if a key exists
console.log(myMap.has(`${key}3`)); // Output: true

// Iterate using forEach
myMap.forEach((item) => console.log(item));

// Iterate values using for...of
for (let value of myMap.values()) {
  console.log(value);
}

// Iterate keys using for...of
for (let keys of myMap.keys()) {
  console.log(key);
}

Breaking Down the Code

Creating a Map: new Map<string, string>() defines a Map where both keys and values are strings.
Adding Key-Value Pairs: We use .set() to dynamically generate keys (value0, value1, etc.) and assign corresponding values (emi0, emi1, etc.).
Retrieving Values: .get() fetches values by key.
Checking for Existence: .has() checks if a key exists in the Map.
Iterating Over Entries: We use .forEach(), .values(), and .keys() to loop through the Map’s data.

When Should You Use a Map?

When you need fast key-value lookups.
When you require non-string keys (e.g., numbers, objects, functions).
When maintaining insertion order matters.
When working with large dynamic datasets that require frequent modifications.

If you’ve been using objects for key-value storage, consider switching to Maps where it makes sense. They provide a more predictable and performant way to manage structured data in TypeScript!

Emi Roberti - Happy coding

AWS ECS Fargate for Building and Deploying Containers

Emiliano Roberti — Thu, 30 Jan 2025 20:18:27 +0000

When I started out with cloud development it took me some times to understand the various deployment strategies, how they differ and when to use a Lambda or a container. ( have a article on lambdas https://dev.to/emiroberti/aws-lambda-and-its-invocation-models-4406 ).

In this article I will focus on AWS Elastic Container Service (ECS) and AWS Fargate as its a powerful solutions for running containers in the cloud. However, developers often compare them with alternatives such as Amazon EC2, AWS Lambda, and AWS App Runner. This article explores why ECS and Fargate are excellent choices, and how they compare to these other services.

Understanding ECS and Fargate

Amazon ECS is a fully managed container orchestration service that enables you to deploy, manage, and scale containerised applications. It supports two launch types:

EC2 Launch Type: Uses Amazon EC2 instances as the underlying compute infrastructure for running containers.
Fargate Launch Type: A serverless option that eliminates the need to manage EC2 instances, allowing AWS to handle provisioning and scaling.

Fargate is particularly useful for teams looking for a simplified way to run containers without worrying about underlying infrastructure.

Differentiating ECS and Fargate from Other AWS Compute Services

1. ECS vs EC2

Feature	ECS (with Fargate)	EC2
Infrastructure Management	Fully managed	User-managed
Scaling	Automatic, serverless	Manual or Auto Scaling Groups
Security	IAM-based role per task	Requires EC2 security group configuration
Cost Efficiency	Pay per vCPU and memory	Pay per EC2 instance

ECS with EC2 gives more control over the underlying infrastructure but requires additional management.
Fargate simplifies deployment by abstracting infrastructure concerns, making it easier to manage at scale.

2. ECS vs Lambda

Feature	ECS (with Fargate)	AWS Lambda
Best for	Long-running containerised apps	Event-driven, short-lived functions
Execution Time	Unlimited	Limited to 15 minutes per execution
Cold Start Delay	Minimal	Potential delays due to cold starts
Pricing Model	Pay for provisioned CPU/memory	Pay per execution time and requests

Lambda is excellent for serverless event-driven applications but unsuitable for persistent container workloads.
ECS (with Fargate) is ideal for microservices that need continuous availability.

3. ECS vs App Runner

Feature	ECS (with Fargate)	AWS App Runner
Best for	Complex container orchestration	Simple web app deployment
Customisation	Highly configurable	Abstracted service
Deployment Method	Task definitions, services, ALBs	Git-based deployment
Scaling	Fine-grained autoscaling	Managed autoscaling

AWS App Runner is designed for developers looking for a hands-off approach to running web applications.
ECS and Fargate offer more flexibility and control over networking, security, and performance.

Why Choose ECS and Fargate?

1. Simplified Management

With AWS Fargate, you do not need to provision, manage, or scale EC2 instances. This is ideal for teams focused on application development rather than infrastructure maintenance.

2. Scalability and Flexibility

ECS with Fargate allows dynamic scaling based on workload demand, ensuring optimal resource utilisation without manual intervention.

3. Security and Isolation

ECS integrates seamlessly with AWS IAM for granular permissions. Fargate tasks run in isolated compute environments, reducing security risks.

4. Cost Efficiency

Fargate follows a pay-as-you-go model where you only pay for the vCPU and memory consumed by running containers. This eliminates the need to provision excess capacity upfront.

5. Deep AWS Integration

ECS and Fargate work seamlessly with AWS services like IAM, VPC, CloudWatch, and ALB, offering a unified ecosystem for building cloud-native applications.

Amazon ECS and Fargate provide ascalabl e, and cost-effective solutions for container orchestration. While EC2, Lambda, and App Runner each have their use cases, ECS and Fargate strike a balance between flexibility and ease of use.

Here is a sample code for a cdk project.

AWS CDK Code

This AWS CDK script provisions an ECS Fargate service using AWS resources, including a VPC, security groups, ECS cluster, and an Application Load Balancer (ALB). The service runs a container from a specified Docker image repository, making it publicly accessible over port 8000.

Key Features

VPC Creation: The script creates a VPC with two public subnets and no NAT gateways to keep costs low.
ECS Cluster: An ECS cluster is created within the VPC.
Security Group Configuration: Ingress rules allow traffic on ports 8000 (application) and 443 (HTTPS).
Fargate Task Definition: Defines the container settings, including memory, CPU, and IAM permissions.
Application Load Balancer: Used to distribute traffic to the running container instances.
Health Check Configuration: Ensures the service is considered healthy before routing traffic.

Prerequisites

Before deploying this stack, ensure:

AWS CDK is installed (npm install -g aws-cdk)
AWS credentials are configured (aws configure)
Environment variables are set in a .env file:

  STAGE=dev
  PREFIX=server-image-process
  DOCKER_REPO=your-docker-image-repo
  STREAM_PREFIX=server-stream

Code Breakdown

VPC Setup

const vpc = new ec2.Vpc(this, prefix + "-vpc", {
  maxAzs: 2,
  natGateways: 0,
  subnetConfiguration: [
    {
      cidrMask: 24,
      name: "public",
      subnetType: ec2.SubnetType.PUBLIC,
    },
  ],
});

This creates a VPC with two availability zones and public subnets.

ECS Cluster

const cluster = new ecs.Cluster(this, prefix + "-cluster", {
  vpc,
});

Creates an ECS cluster to manage the containerised application.

Security Group

const securityGroup = new ec2.SecurityGroup(this, prefix + "-sg", {
  vpc,
  allowAllOutbound: true,
});

securityGroup.addIngressRule(ec2.Peer.anyIpv4(), ec2.Port.tcp(8000), "Allow HTTP traffic");
securityGroup.addIngressRule(ec2.Peer.anyIpv4(), ec2.Port.tcp(443), "Allow HTTPS");

Allows inbound traffic on ports 8000 and 443.

Fargate Task Definition

const taskDefinition = new ecs.FargateTaskDefinition(
  this, prefix + "-task-definition",
  {
    memoryLimitMiB: 1024 * 3,
    cpu: 256 * 4,
  }
);

Defines the Fargate task with 3GB RAM and 1 CPU.

Container Definition

const container = taskDefinition.addContainer(prefix + "-container", {
  image: ecs.ContainerImage.fromRegistry(repo),
  logging: ecs.LogDriver.awsLogs({
    streamPrefix: streamPrefix,
    logRetention: logs.RetentionDays.ONE_WEEK,
  }),
});

container.addPortMappings({ containerPort: 8000 });

Specifies the container image and logging configuration.

Deploying with a Load Balancer

const fargateService = new ecs_patterns.ApplicationLoadBalancedFargateService(
  this, prefix + "-service",
  {
    cluster,
    taskDefinition,
    desiredCount: 1,
    assignPublicIp: true,
    publicLoadBalancer: true,
    securityGroups: [securityGroup],
    healthCheckGracePeriod: cdk.Duration.seconds(60),
  }
);

Creates an Application Load Balancer (ALB) for the service.

Health Check Configuration

fargateService.targetGroup.configureHealthCheck({
  path: "/api/health",
  port: "8000",
  healthyThresholdCount: 2,
  unhealthyThresholdCount: 5,
  timeout: cdk.Duration.seconds(5),
  interval: cdk.Duration.seconds(30),
});

Ensures that the application is only routed to healthy instances.

Deployment

To deploy the stack, run:

cdk deploy

To delete the stack, use:

cdk destroy

Author: Emi Roberti

Understanding Amazon Kinesis Data Streams: Indexing, Replay, and Consumer Flexibility

Emiliano Roberti — Sat, 25 Jan 2025 06:17:02 +0000

Amazon Kinesis Data Streams (KDS) is a powerful platform for real-time data streaming and processing. It allows applications to ingest, process, and analyse high-throughput data in real time. One of its key strengths lies in indexing, event replay, and the ability to support multiple consumers at varying speeds. Let’s explore these features and their implications.

How Indexes Work in Kinesis Data Streams

In Kinesis, every record ingested into the stream is assigned a sequence number. This sequence number acts as the index for the record and serves the following purposes:

Record Order: Within a shard, records are strictly ordered based on their sequence numbers.
Replay and Retrieval: The sequence number allows applications to replay specific records or ranges of records.
Shard Iterator: Consumers use the sequence number to request records starting from a specific point.

Indexing Characteristics

Shard-Level Granularity: Each shard in a stream maintains its own set of sequence numbers. Sequence numbers are unique within a shard but not across the stream.
Durability: Records are retained in the stream for a configurable retention period (default: 24 hours, maximum: 7 days). During this period, indexes enable replay.
Starting Position: Consumers can begin reading from:
- TRIM_HORIZON: Start from the oldest record available.
- LATEST: Start with the most recent record.
- Specific Sequence Number: Start from an exact point in the stream.

Replay Events with Kinesis

One of Kinesis's standout features is the ability to replay events. This enables use cases like:

Debugging or reprocessing historical data.
Feeding data into new consumers without data loss.

How Replay Works

Consumers use a Shard Iterator to specify the starting point of record retrieval. Example: Replay records from a sequence number 49659891250263205584801854121677320731210653829945098258.
Kinesis ensures that records are delivered in order for each shard, maintaining consistency during replay.

Supporting Multiple Consumers at Different Speeds

Kinesis streams are designed to support multiple consumers reading data from the same stream. Here’s how it works:

Key Features

Independent Progress:
- Each consumer can maintain its own shard iterator, allowing independent progress.
- Fast consumers can process records quickly, while slower consumers operate at their own pace.

Enhanced Fan-Out (EFO):
- With EFO, consumers receive their own throughput pipe, avoiding contention between multiple consumers.
- Default throughput: 2 MB/s per shard for all consumers.
- With EFO: Each consumer gets an additional 2 MB/s.
Checkpointing:
- Checkpointing ensures that each consumer keeps track of the last record processed.
- This allows consumers to resume processing from where they left off, even after interruptions.

Architecture Components

Left Side: Data Sources

Mobile: Data generated by mobile devices, such as user interactions, telemetry, or logs.
Desktop: Similar data generated from desktop applications.
Game Servers: Game server activities, player interactions, or system metrics.

These data sources produce real-time events and stream them into Amazon Kinesis.

Centre: Amazon Kinesis

Amazon Kinesis: A managed service that collects, processes, and analyses real-time streaming data. It consists of shards, which are the basic units for parallel data processing. Each shard processes a portion of the incoming data.
- Shard A, B, C, D: Multiple shards provide scalability and allow parallel data ingestion and processing. The number of shards typically depends on the data volume and processing requirements.

Right Side: AWS Components

AWS Lambda:
- Event-driven compute service that automatically processes incoming data from the Kinesis stream.
- It applies transformations, enrichments, or routing logic to the data.
- Lambda also triggers notifications via SNS (Simple Notification Service) for downstream systems or users.
Amazon DynamoDB:
- A NoSQL database used to store processed or aggregated data from Kinesis.
- Suitable for scenarios like maintaining state, querying game statistics, or storing metadata.
SNS (Simple Notification Service):
- Publishes notifications triggered by AWS Lambda. These notifications can be sent to users, admins, or other applications for alerting or further actions.
Amazon Kinesis Data Analytics:
- Analyses streaming data in real-time using SQL-like queries.
- Provides insights such as trends, anomalies, or metrics without requiring the data to be stored.

Use Cases

This architecture is particularly suited for:

Event driven platforms that require real-time monitoring.
Gaming platforms that require real-time scores and gaming events.
Applications providing live dashboards or insights.
Monitoring systems that need to alert on specific events or thresholds.

Amazon Kinesis Data Streams cab provide a flexible solution for real-time data processing. By using indexing, replay capabilities, and consumer independence, you can design systems that are scalable, resilient, and tailored to various workloads.

I have also included my repository on creating records to be sent to aws kinesis stream and a target lambda to extract the records ready to be purposed for business logic

Full code here

https://github.com/EmiRoberti77/cdk_kinesis_stream_sample

How the code works with CDK, Kinesis and Lambda

This document provides an overview of setting up a Kinesis Data Stream with AWS CDK and integrating it with a Lambda function for processing records. The examples include:

Sending records to the Kinesis Data Stream.
Creating an AWS CDK stack for the stream and Lambda integration.
Writing a Lambda function to process the records.

Project Setup

To create and manage the infrastructure using AWS CDK, follow these steps:

Install AWS CDK

Ensure you have the AWS CDK installed globally:

npm install -g aws-cdk

Initialise the CDK Project

Run the following command to create a new CDK project:

cdk init app --language=typescript

This will create a basic file structure for your CDK project.

File Structure

Below is the file structure for this example, including where the Lambda functions are stored:

project-root/
├── bin/
│   └── cdk-kinesis.ts           # Entry point for the CDK app
├── lib/
│   └── cdk-kinesis-stack.ts     # Stack definition for Kinesis and Lambda
├── src/
│   └── lambdas/
│       └── lambda_kinesis_target/
│           └── handler.ts       # Lambda handler for processing Kinesis records
├── node_modules/                # Node.js dependencies
├── package.json                 # Project dependencies
├── cdk.json                     # CDK project configuration
├── tsconfig.json                # TypeScript configuration
└── README.md                    # Documentation for the project

Sending Records to Kinesis Data Stream

The following TypeScript code demonstrates how to send records to a Kinesis Data Stream. The record includes a random name, timestamp, and message, all of which are JSON-encoded.

import { KinesisClient, PutRecordCommand } from "@aws-sdk/client-kinesis";
const kinesisClient = new KinesisClient({
  region: "us-east-1",
});

async function addRecordToKinesis(
  streamName: string,
  data: string,
  partitionKey: string
): Promise<any> {
  try {
    const encodedData = Buffer.from(data);
    const putRecord = new PutRecordCommand({
      StreamName: streamName,
      Data: encodedData,
      PartitionKey: partitionKey,
    });

    const response = await kinesisClient.send(putRecord);
    console.log(response);
    return response;
  } catch (err: any) {
    console.log(err.message);
    return undefined;
  }
}

const random = Math.floor(Math.random() * 1000);
const now = new Date().toISOString();
const streamName = "emi-kinesis-stream";
const data = JSON.stringify({
  name: "emi" + random,
  timeStamp: now,
  message: "kinesis message",
});
const partitionKey = "PartitionKey1";

addRecordToKinesis(streamName, data, partitionKey).catch((err) =>
  console.log(err)
);

AWS CDK Stack for Kinesis and Lambda Integration

The following AWS CDK code sets up a Kinesis Data Stream and a Lambda function. It integrates the Lambda function with the Kinesis stream using an event source mapping.

import * as cdk from "aws-cdk-lib";
import { Construct } from "constructs";
import { Stream, StreamMode } from "aws-cdk-lib/aws-kinesis";
import { NodejsFunction } from "aws-cdk-lib/aws-lambda-nodejs";
import { Runtime, StartingPosition } from "aws-cdk-lib/aws-lambda";
import { Effect, PolicyStatement } from "aws-cdk-lib/aws-iam";
import { KinesisEventSource } from "aws-cdk-lib/aws-lambda-event-sources";
import * as path from "path";

export class CdkKinesisStack extends cdk.Stack {
  constructor(scope: Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    const stream = new Stream(this, "emi_cdk_stream", {
      streamName: "emi-kinesis-stream",
      streamMode: StreamMode.ON_DEMAND,
    });

    const targetKinesisLambda = new NodejsFunction(
      this,
      "emi_cdk_kinesis_target_lambda",
      {
        functionName: "emi_cdk_kinesis_target_lambda",
        runtime: Runtime.NODEJS_18_X,
        handler: "handler",
        entry: path.join(
          __dirname,
          "..",
          "src",
          "lambdas",
          "lambda_kinesis_target",
          "handler.ts"
        ),
      }
    );

    stream.grantReadWrite(targetKinesisLambda);

    targetKinesisLambda.addToRolePolicy(
      new PolicyStatement({
        actions: ["*"],
        resources: ["*"],
        effect: Effect.ALLOW,
      })
    );

    targetKinesisLambda.addEventSource(
      new KinesisEventSource(stream, {
        startingPosition: StartingPosition.LATEST,
        batchSize: 100,
        enabled: true,
      })
    );
  }
}

Lambda Function for Processing Kinesis Records

The Lambda function below processes records from the Kinesis Data Stream. It decodes the Base64-encoded data, parses it as JSON, and logs the payload.

import { KinesisStreamEvent, KinesisStreamRecord } from "aws-lambda";

export const handler = async (event: KinesisStreamEvent): Promise<void> => {
  for (const record of event.Records) {
    const { partitionKey, sequenceNumber } = record.kinesis;
    console.log(
      `[partitionKey=${partitionKey}]:[sequenceNumber=${sequenceNumber}]`
    );
    const payload = Buffer.from(record.kinesis.data, "base64").toString(
      "utf-8"
    );
    try {
      const decodedPayload = JSON.parse(payload);
      console.log(decodedPayload);
    } catch (err) {
      console.error(err);
    }
  }
};

Summary

This setup demonstrates how to:

Send data to a Kinesis Data Stream.
Use AWS CDK to create and configure the stream and a Lambda function.
Process the Kinesis stream records in a Lambda function.

With this architecture, you can handle real-time streaming data and integrate it into your applications effectively.

# MQTTHandler Library

Emiliano Roberti — Sun, 19 Jan 2025 13:12:02 +0000

The MQTTHandler library is a lightweight, static TypeScript-based MQTT broker utility designed to simplify the use of the MQTT protocol for IoT and real-time messaging. The library adopts a static approach, removing the need for repeated instantiation, ensuring a single, streamlined connection across your application, and offering robust event-driven communication via EventEmitter.

Unlike traditional protocols such as HTTP, which operate on a request-response model and can introduce latency due to their synchronous nature, MQTT is built for efficiency in constrained environments. It uses a publish-subscribe model, enabling lightweight and reliable message exchange even on low-bandwidth networks.While protocols like WebSocket also support real-time communication, MQTT stands out due to its simplicity, efficiency, and design tailored for resource-constrained environments. MQTT uses much smaller packet sizes compared to WebSocket, which means it requires less bandwidth and processes data faster—an important advantage for devices with limited power or connectivity.

One of MQTT’s key strengths is its built-in Quality of Service (QoS) levels, which ensure messages are delivered reliably, even in cases where the network is unstable or interrupted. This makes it ideal for IoT devices, which often operate in environments with inconsistent connectivity, such as remote sensors, smart home devices, or industrial systems. Additionally, MQTT’s publish-subscribe model simplifies communication between devices by allowing them to focus only on the messages they need, further reducing unnecessary data exchange and processing overhead.

I am using MQTT for IoT applications and also with microservices to publish messages that will be consumed but other listening services. I have written a library for anyone to use or extend the code base i did.

the library can be installed from the link below

https://www.npmjs.com/package/vmind_mqtt

on the npm home page there is also the link to the github repo

https://github.com/EmiRoberti77/vmind_mqtt

I have used mosquitto as a broker for the messages

brew install mosquitto

Features

Static Design: Simplifies usage by managing a single shared MQTT client instance.
Event-Driven: Uses EventEmitter to handle message events seamlessly.
Ease of Use: Intuitive APIs for connecting, publishing, and subscribing to MQTT topics.
Error Handling: Provides detailed error messages for debugging.
Scalable: Designed for lightweight, efficient message handling.

Installation

Install the library from npm:

npm i vmind_mqtt

Methods

The MQTTHandler library provides the following methods:

initialize(mqttUrl: string): void Initializes the MQTT client with the specified broker URL. Example:

   MQTTHandler.initialize("mqtt://broker.hivemq.com");

publish(topic: string, message: string): void Publishes a message to the specified topic. Example:

   MQTTHandler.publish(
     "test/topic",
     JSON.stringify({ message: "hello, MQTT!" })
   );

listen(topic: string): void Subscribes to a topic and listens for incoming messages. Example:

   MQTTHandler.listen("test/topic");

onMessage(callback: (data: { topic: string; message: any }) => void): void Registers a callback to handle incoming messages from subscribed topics. Example:

   MQTTHandler.onMessage((data) => {
     console.log(`Received message from ${data.topic}:`, data.message);
   });

stopListening(topic: string): void Unsubscribes from a specific topic, stopping the library from receiving messages from it. Example:

   MQTTHandler.stopListening("test/topic");

close(): void Gracefully closes the MQTT connection. Example:

   MQTTHandler.close();

Usage

Initialize the MQTT Client

Before using the library, initialize the MQTT client with the broker URL:

MQTTHandler.initialize("mqtt://broker.hivemq.com");

Publish a Message

Send a message to a specific topic:

MQTTHandler.publish("test/topic", JSON.stringify({ message: "hello, MQTT!" }));

Subscribe and Listen to a Topic

Listen for messages from a subscribed topic:

MQTTHandler.listen("test/topic");

MQTTHandler.onMessage((data) => {
  console.log(`Received message from ${data.topic}:`, data.message);
});

Stop Listening to a Topic

Unsubscribe from a topic to stop receiving messages:

MQTTHandler.stopListening("test/topic");

Close the MQTT Connection

Gracefully close the MQTT connection when done:

MQTTHandler.close();

Example

import { MQTTHandler } from "vmind_mqtt/dist";

// Initialize the client
MQTTHandler.initialize("mqtt://broker.hivemq.com");

// Subscribe and listen to a topic
MQTTHandler.listen("example/topic");
MQTTHandler.onMessage((data) => {
  console.log(`Received message on topic ${data.topic}:`, data.message);
});

// Publish a message
MQTTHandler.publish(
  "example/topic",
  JSON.stringify({ message: "Hello, World!" })
);

// Stop listening to the topic
MQTTHandler.stopListening("example/topic");

// Close the client after some time
setTimeout(() => {
  MQTTHandler.close();
}, 5000);

Advantages of the Static Approach

Singleton Pattern:

Ensures a single MQTT connection instance across your application, reducing overhead and connection management issues.
Simplified API:

No need to instantiate the client repeatedly; directly use static methods to interact with the broker.
Event-Driven Communication:

Uses EventEmitter to handle messages, enabling clean, modular, and decoupled code.
Lightweight:

Avoids dependencies on frameworks like Express, making it ideal for small services or microservices.
Flexibility:

Easily integrates into any TypeScript or Node.js project.

License

This library is licensed under the MIT License. See the LICENSE file for details.

Contributing

Contributions are welcome! If you find a bug or want to add features, feel free to submit a pull request or open an issue on the GitHub repository.

Support

If you encounter any issues or have questions, feel free to contact us or open a support ticket.

Emi Roberti - Happy Coding

Deploying a TypeScript Express App to GKE Using Docker and Artifact Registry

Emiliano Roberti — Sat, 18 Jan 2025 21:04:10 +0000

This guide walks you through the steps to create a TypeScript Express app, containerize it with Docker, and deploy it to Google Kubernetes Engine (GKE) using Google Artifact Registry.

1. Create a TypeScript Express App

Initialize the Project:

   mkdir ts_express_server
   cd ts_express_server
   npm init -y

Install Required Packages:

   npm install express cors
   npm install --save-dev typescript @types/node @types/express @types/cors

Set Up TypeScript: Create a tsconfig.json file:

   {
     "compilerOptions": {
       "outDir": "./dist",
       "module": "commonjs",
       "target": "es6",
       "strict": true
     },
     "include": ["src/**/*"]
   }

Write the Server Code: Create src/server.ts:

   import express, { Request, Response } from "express";
   import Cors from "cors";

   const app = express();
   app.use(express.json());
   app.use(Cors());

   app.get("/api", (req: Request, res: Response) => {
     res.status(200).json({ message: "Server up " + new Date().toISOString() });
   });

   app.listen(8090, () => {
     console.log("Server listening on port 8090");
   });

Build the App: Add build scripts to package.json:

   "scripts": {
     "build": "tsc",
     "start": "node dist/server.js"
   }

Build the app:

   npm run build

2. Create a Docker Image

Write a Dockerfile:

   FROM node:18-alpine
   WORKDIR /app
   COPY package*.json ./
   RUN npm install
   COPY . .
   RUN npm run build
   EXPOSE 8090
   CMD ["npm", "run", "start"]

Build the Docker Image:

   docker build -t emirob/emi-repo:my-express-app .

Push the Image to Docker Hub:

   docker tag emirob/emi-repo:my-express-app emirob/emi-repo:my-express-app
   docker push emirob/emi-repo:my-express-app

3. Set Up GCP for Artifact Registry and GKE

Enable Required APIs:
- Go to the Google Cloud Console.
- Enable the Artifact Registry API.
- Enable the Kubernetes Engine API.
Set Your Project in Cloud Shell:

   export PROJECT_ID=gketest-448214
   gcloud config set project $PROJECT_ID

Create an Artifact Registry Repository:

   gcloud artifacts repositories create emi-repo --repository-format=docker --location=us-west1 --description="Docker repo"

Authenticate Docker with GCP:

   gcloud auth configure-docker us-west1-docker.pkg.dev

4. Push the Docker Image to Artifact Registry

Pull the Image from Docker Hub:

   docker pull emirob/emi-repo:my-express-app

Tag the Image for Artifact Registry:

   docker tag emirob/emi-repo:my-express-app        us-west1-docker.pkg.dev/gketest-448214/emi-repo/my-express-app:latest

Push the Image to Artifact Registry:

   docker push us-west1-docker.pkg.dev/gketest-448214/emi-repo/my-express-app:latest

Verify the Image:

   gcloud artifacts docker images list us-west1 docker.pkg.dev/gketest-448214/emi-repo

5. Deploy the App on GKE

Create a GKE Cluster:

   gcloud container clusters create my-cluster --num-nodes=3 --zone=us-west1-a

Connect to the Cluster:

   gcloud container clusters get-credentials my-cluster --zone us-west1-a

Create a Kubernetes Deployment: Save the following as deployment.yaml:

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: my-express-app
   spec:
     replicas: 2
     selector:
       matchLabels:
         app: my-express-app
     template:
       metadata:
         labels:
           app: my-express-app
       spec:
         containers:
         - name: my-express-app
           image: us-west1-docker.pkg.dev/gketest-448214/emi-repo/my-express-app:latest
           ports:
           - containerPort: 8090

Apply the Deployment:

   kubectl apply -f deployment.yaml

Expose the Service:

   kubectl expose deployment my-express-app --type=LoadBalancer --port 80 --target-port 8090

Access the Application: Get the external IP of the LoadBalancer:

   kubectl get service my-express-app

Visit `http://<EXTERNAL-IP>/api`.

6. GitHub Repository

For the full project setup and Docker image creation, check the GitHub repository:

build_node_server_docker_images

Emi Roberti - Happy coding

What Makes a Good Cloud Architect?

Emiliano Roberti — Sun, 12 Jan 2025 21:03:31 +0000

I have been developing for a number of years, and always really enjoyed getting stuck into a complex program to write, getting involved in all the details of the application. However a solution Architect needs to be able to also look at the bigger picture.

A cloud architect plays a pivotal role in enabling the businesses to deliver scalable solutions. The role requires a blend of technical expertise, strategic vision, and a focus on business outcomes.

Core Attributes of a Good Cloud Architect

1. Technical Expertise

A cloud architect must have in-depth knowledge of cloud platforms (e.g., AWS, Azure, Google Cloud) and the services they offer. This includes understanding compute, storage, networking, security, and deployment models such as Infrastructure-as-a-Service (IaaS), Platform-as-a-Service (PaaS), and Software-as-a-Service (SaaS).

Key technical skills include:

Cloud-native architectures (e.g., microservices, serverless computing).

Infrastructure as Code (IaC) tools (e.g., Terraform, CloudFormation).

Networking concepts such as VPCs, subnets, and load balancers.

Security best practices, including encryption, identity management, and compliance.

Automation and CI/CD pipelines for seamless deployments.

2. Strategic Vision

A great cloud architect aligns technical decisions with business goals. They understand how cloud technologies can drive innovation, reduce costs, and improve operational efficiency.

3. Problem-Solving Skills

The ability to troubleshoot complex systems and design scalable, resilient solutions is essential. Cloud architects often address challenges like latency, redundancy, disaster recovery, and scaling under unpredictable workloads.

4. Communication and Collaboration

Cloud architects bridge the gap between technical teams and business stakeholders. Strong communication skills ensure that technical decisions are effectively translated into business outcomes.

5. Adaptability and Continuous Learning

Cloud technologies evolve rapidly. A good architect stays updated with new services, tools, and trends, ensuring their designs remain modern and effective.

Objectives of a Cloud Architect

Design Scalable and Resilient Architectures

Create cloud environments that can handle fluctuating workloads without downtime, ensuring business continuity and customer satisfaction.

2. Optimise Costs

Architect cloud solutions that minimise operational costs while meeting performance and security requirements. This includes right-sizing resources, leveraging reserved instances, and automating scaling.

3. Ensure Security and Compliance

Design systems that adhere to industry standards and regulatory requirements. This includes implementing encryption, access controls, and monitoring solutions.

4. Enable Automation

Use Infrastructure as Code (IaC) and CI/CD pipelines to reduce manual intervention, accelerate deployments, and minimize errors.

5. Drive Innovation

Leverage emerging technologies like AI/ML, IoT, and edge computing to create competitive advantages for the organisation.

Promote Best Practices

Establish guidelines and frameworks that help teams follow best practices in cloud architecture, development, and operations.

Measuring Success as a Cloud Architect

1. Business Impact

Success is reflected in how well the cloud solutions meet business objectives. Metrics include:

Reduction in operational costs.
Increased system uptime and reliability.
Accelerated time-to-market for new features or products.

2. Technical Performance

Evaluate the effectiveness of the architectures:

Scalability: Can the system handle growth without significant redesign?

Resilience: How well does the system recover from failures?

Security: Are there any reported vulnerabilities or breaches?

3. Stakeholder Satisfaction

Positive feedback from business and technical stakeholders indicates success in aligning solutions with organizational goals.

4. Team Enablement

The extent to which the architect empowers development and operations teams to work efficiently. For instance:

Reduced deployment times.
Clear architectural guidelines and documentation.

5. Adoption of Best Practices

Measure how well teams follow the frameworks and standards set by the cloud architect.

6. Innovation Metrics

Successful implementation of new technologies or paradigms (e.g., serverless applications, AI/ML pipelines) that drive competitive advantage.

I found in my career a good cloud architect is a blend of technologist, strategist, and communicator. Its not just design cloud systems, but enable businesses goals, increase productivity, clear vision, bring innovation, efficiency, and reliability in a solution.

I tried to sum up the key things that are expected from a Solution Cloud Architect in my experiance.

Emi Roberti

AWS Lambda and Its Invocation Models

Emiliano Roberti — Sat, 04 Jan 2025 15:32:21 +0000

AWS Lambda is one of the most versatile and efficient serverless computing solutions available. By abstracting server management, AWS Lambda allows developers to focus solely on writing and deploying code without worrying about provisioning or maintaining infrastructure. Below, we delve into the various invocation models and key characteristics of AWS Lambda.

Lambda Synchronous Invocation

Synchronous invocation is the simplest way to call an AWS Lambda function. In this model, the caller waits for the Lambda function to process the request and return a response. Common use cases include:

API Gateway Integration: When an HTTP request is sent to an API Gateway, it can synchronously invoke a Lambda function to process and return data.

Real-time Applications: Scenarios requiring immediate feedback, such as chatbots or interactive dashboards.

In this mode, error handling is straightforward, as any failure or error in the function is directly returned to the caller.

Lambda Asynchronous Invocation

Asynchronous invocation, on the other hand, decouples the caller from the function execution. The function is invoked and added to a queue for processing. The caller does not wait for a response, making it suitable for applications that don’t require immediate results. AWS automatically retries failed invocations twice, ensuring reliability. Use cases include:

Event-driven Workflows: Triggered by events such as Amazon S3 bucket updates or Amazon SNS notifications.

Batch Processing: Ideal for use cases like sending email notifications or processing logs.

AWS provides mechanisms for monitoring and managing asynchronous invocations, such as Dead Letter Queues (DLQs) and Amazon EventBridge.

Lambda Event Source Mapping with Polling Invocation

Event Source Mapping is a mechanism that enables Lambda to poll event sources like Amazon SQS, DynamoDB Streams, or Kinesis Data Streams. Lambda continuously monitors these sources and invokes the function with a batch of records. Key features include:

Automatic Polling: Lambda handles polling behind the scenes, ensuring seamless integration with event sources.

Batch Processing: Functions can process multiple records in a single invocation, improving efficiency.

Checkpointing: Lambda manages checkpoints to ensure events are processed exactly once in case of failures.

This model is optimal for real-time data ingestion and processing pipelines.

Key Characteristics of AWS Lambda

Containers Under the Hood

Each AWS Lambda function operates within its own isolated environment, effectively running as its own container. This containerisation ensures:

Security: Functions are isolated from one another.

Consistency: Each invocation starts in a fresh container, ensuring no residual data between executions.

When a function is created, AWS packages it into a container, deploys it on a multi-region cloud cluster, and scales it across servers managed by AWS. This ensures high availability and fault tolerance.

Resource Allocation and Cost Efficiency

Lambda functions are allocated CPU and RAM based on user-defined configurations. The pay-as-you-go pricing model charges only for the actual compute time used, measured to the nearest 100 milliseconds. This cost-efficient model eliminates over-provisioning, making it ideal for unpredictable workloads.

Event-Driven Architecture

AWS Lambda is built around an event-driven architecture, where numerous AWS resources can generate events to trigger Lambda functions. Examples include:

S3: Trigger Lambda on object creation or deletion.
DynamoDB: Invoke Lambda for stream updates.
SNS and SQS: Process messages from these messaging services.

This flexibility allows developers to build decoupled, modular applications with ease.

Scalability

Lambda automatically scales with the incoming request volume, scaling up or down to handle thousands of requests per second. This elasticity ensures:

High Availability: Functions scale seamlessly without manual intervention.

Cost Savings: Resources are deallocated immediately upon completion.

Maximum Execution Time

Each Lambda invocation has a maximum execution time of 15 minutes. While sufficient for most tasks, developers must design functions for efficiency. For long-running processes, consider breaking tasks into smaller sub-tasks or using Step Functions for orchestration.

AWS Lambda epitomises the power of serverless computing, offering unparalleled scalability, cost efficiency, and simplicity. With various invocation models and a robust event-driven architecture, Lambda is suitable for a wide range of use cases—from real-time applications to large-scale data processing. By understanding its invocation models and key characteristics, developers can harness its full potential to build resilient and scalable applications.

Sample code on using typescript to build a lambda and deploy with CDK

https://github.com/EmiRoberti77/cdk_ts_apigateway_lambda

Happy Coding - Emi Roberti

DEV Community: Emiliano Roberti

What It Means to Be a Developer in the AI Era

Developing Software in the Age of AI — A Personal Take After 25 Years of Coding

AI Has Changed How We Work

But Let’s Not Pretend It’s All Perfect

Will AI Replace Developers?

The Way Teams Build Software Is Changing Too

Has AI Made Me a Better Developer?

AI Should Be a Partner, Not a Crutch

Where We’re Heading

DuckDB: The Analytics Database Revolution - A Comprehensive Guide

Introduction

What is DuckDB?

Key Advantages of DuckDB

1. Blazing Fast Performance

2. Zero Configuration

3. Seamless Python Integration

4. SQL Excellence

5. Multiple File Format Support

Practical Demonstration: DuckDB vs Pandas Performance

The Challenge: Processing Multiple CSV Files

Performance Results

Advanced DuckDB Features in Action

1. Intelligent Data Type Conversion

2. Advanced SQL Analytics

3. Flexible Data Access Patterns

Real-World Use Cases

1. Data Science Workflows

2. ETL Processing

3. Rapid Prototyping

4. Big Data Processing

5. Business Intelligence

Performance Benchmarks

Integration Ecosystem

Input Sources

Output Formats

Visualization Tools

Best Practices

1. Memory Management

2. Query Optimization

3. Data Type Optimization

When to Choose DuckDB

Perfect For:

Consider Alternatives For:

Getting Started

Installation

Basic Usage

Conclusion

Additional Resources

Author

Redis API Gateway Rate Limiter with Express.js

Redis API Gateway Rate Limiter with Express.js

Overview

Why Redis?

Features

Project Structure

Setup Instructions

Install Dependencies

2. Configure Environment

Code Explanation

1. Redis Client Setup

2. Rate Limiter Rule Interface

3. RedisRateLimiterHandler Class

4. Express Middleware Integration

Example

Redis Keys

Pros

Security Notes

Development Considerations

References

Author

AI Agent faster memory access

Key Advantages of Redis for AI Agent Memory

Sample code to get started with Redis and langchain

📦 Tech Stack

🚀 Getting Started

1. Clone & install dependencies

2. Run Redis (if not already running)

3. Create your .env (optional for API keys)

🧪 Run the CLI

3. Create your `.env` (optional for API keys)

📄 Example: `src/index.ts`

📄 Example: `src/redisClient.ts`

📄 Example: `src/langchain.ts`

gRPC Server (`server.ts`)

gRPC Client (`client.ts`)

Why Use a `Map`?

How is a `Map` Different from an Array?

How is a `Map` Different from an Object (Dictionary)?

Using a `Map` in TypeScript