DEV Community: Oussama Belhadi

Kafka | everything you need to get started

Oussama Belhadi — Mon, 13 Oct 2025 13:48:06 +0000

Decoding Kafka: The Architectural Shift to Scalable, Decoupled Systems

Apache Kafka is one of the most transformative technologies in modern system design, moving applications from sluggish, tightly coupled messes to real-time, scalable data streams. But what is it, and why is everyone so excited about it?

Let's dive into the problem Kafka solves and its fundamental concepts, using the real-life example of an e-commerce platform.

1. The Tightly Coupled Trap

When a startup builds an e-commerce application, they often start with the simplest microservices architecture where services call each other directly.

Imagine an Order Service that needs to tell the Payment, Inventory, Analytics, and Notification services about a new order. The communication is synchronous:

"Hey Inventory, update your stock. Wait for the confirmation. Hey Payment, process this. Wait again..."

It looks like this in Programming :

While straightforward at first, this approach leads to a nightmare when traffic spikes, such as during a busy holiday season:

Tight Coupling: If the Payment Service or the Inventory Service goes down, the entire order process freezes, blocking the Order Service from completing its task.
Synchronous Communication: Each order becomes a game of dominoes. One slow service delays the entire chain reaction, causing customers to stare at loading screens.
Single Points of Failure: An outage for one essential service can mean hours of order backlogs and lost sales.

The solution is to decouple these services by introducing a highly reliable middleman.

2. Introducing Kafka: The Post Office Analogy

Think of Kafka as a central Post Office or mail delivery service.

When you ship a package, you don't personally drive to every person who needs to know about that shipment. You hand it over to the post office and go home. You trust the post office (the middleman) to handle the delivery.

Kafka is that middleman. Instead of services calling each other directly, they hand their information to Kafka and immediately get back to work.

This simple change moves the architecture from a synchronous "hot potato" game to an asynchronous, scalable conveyor belt of items.

3. Kafka's Core Concepts

To understand how Kafka works, we need to know its main components:

Concept	E-commerce Example	Post Office Analogy
Event	An order placed, a payment failed, stock updated.	The letter or package being sent.
Producer	The Order Service writes a new "Order Placed" event to Kafka.	The sender dropping off a package at the counter.
Broker	A Kafka server that stores and manages the events.	A post office branch or the entire mail delivery infrastructure.
Topic	A category or feed for a stream of related events, e.g., `orders`, `payments`, `inventory`.	A dedicated section for a type of mail (letters, large packages, international mail).
Consumer	Services subscribed to a Topic, e.g., Notification Service, Inventory Service.	The recipient who is notified and picks up their mail.

The Flow of an Event

The Order Service (Producer) creates an Event (an order payload with all details) and writes it to the orders Topic.
The Order Service immediately moves on to the next task—it doesn't wait.
The Notification Service (Consumer), Inventory Service (Consumer), and Analytics Dashboard (Consumer) are all subscribed to the orders Topic.
Kafka notifies all subscribed consumers about the new event.
Each consumer performs its own action independently and in parallel:
- Notification: Sends a confirmation email to the customer.
- Inventory: Updates the stock level in the database.
- Payment: Generates the invoice.

Because they are decoupled, the failure of the Inventory Service will not stop the Notification Service from sending the email or the Order Service from accepting new orders.

4. Achieving Scalability: Partitions and Consumer Groups

Kafka's true power lies in its ability to handle millions of events per second. It achieves this primarily through two concepts: Partitions and Consumer Groups.

Partitions: Scaling Writes and Reads

A Topic is divided into ordered, immutable sequences called Partitions.

Analogy: If the "Letters" section of the Post Office gets overloaded, you add more workers. But instead of random assignments, you distribute work based on a criteria: "Ann processes letters for Europe, Steve handles the US."
In Kafka: The orders Topic might be partitioned into EU_orders, US_orders, and Asia_orders. This allows Producers to write data to different partitions in parallel, significantly increasing throughput. Partitions also allow the storage load to be distributed across multiple Brokers (Kafka servers).

Consumer Groups: Scaling Processing

When orders come in too fast, a single consumer service (like Inventory) can get overwhelmed.

Analogy: The letters are arriving, but the single recipient is getting buried under the pile. You need helpers to sort through the mail.
In Kafka: You can start additional instances (replicas) of the same service (e.g., three instances of the Inventory Service). These replicas form a Consumer Group.
Kafka automatically distributes the partitions among the consumers in the group. If the orders Topic has three partitions, each of the three Inventory Service instances will be assigned one partition, allowing them to process the data in parallel.

5. Beyond Messaging: The Power of Streams

Unlike traditional message queues, Kafka persists every event/message as long as you need (based on a configurable retention policy). This enables powerful use cases like Stream Processing.

Traditional Message Queue: Once a message is read, it's deleted (like watching live TV—if you miss it, it's gone).
Kafka (Stream Platform): Events are saved long-term, and Consumers can read them anytime, even multiple times (like watching a streaming service—you can pause, replay, or start from the beginning).

This capability is essential for real-time analytics and complex event processing, like this Driver Live location example:

Real-time Analytics: The Analytics Service can continuously stream all Order and Payment events to update sales dashboards and revenue numbers in real time.
Stateful Processing: An application can constantly process incoming Inventory events, check if the stock dropped below a certain threshold, and immediately trigger a low-inventory alert, initiating an automatic restock order.

By decoupling services, enabling massive scale through partitioning, and providing stream-processing capabilities, Apache Kafka becomes the circulatory system for a modern, data-driven application.

I have found this video very helpful, most screenshots and points discussed are extracted from it.

Software Engineering/Architecture in a Nutshell: A Four-Year Journey

Oussama Belhadi — Tue, 02 Sep 2025 13:10:51 +0000

Whether you're a software engineer, system architect, or a student, system design is a crucial skill for building robust, scalable, and reliable applications. It's the art of creating a blueprint for a software system that meets specific requirements. This guide will walk you through the fundamental principles, common architectures, and essential concepts you'll need to master to design and build systems that can stand the test of time.

Here are the concepts I will pass by in this blog :

Diagram Types in System Design
Production App Architecture
The Pillars of System Design
The Heart of System Design
CAP Theorem (Brewer’s Theorem)
Building Resilience into a System
Measuring Speed
Network Basics
API Design & Best Practices
Scaling & Performance Strategies

This document is your one-stop resource for understanding everything from the core pillars of a well-designed system to practical strategies like caching, load balancing, and network protocols. Let's dive in.

I will start with the most interesting part for me which is Diagrams.
Diagrams help you see a zoomed out picture of a system and visualize the foggy parts when things are so interconnected.

Diagram Types in System Design

When building any system, from a simple app to a large-scale enterprise platform, it's essential to have a clear blueprint. This blueprint is created using various types of diagrams that help visualize the architecture at different levels of detail. Diagrams range from a broad overview to a granular, component-specific view.

High-Level Diagrams

These diagrams provide a zoomed-out view of the system, focusing on its main components and how they interact with each other and the outside world.

Context Diagram: This is a very high-level diagram that shows the system as a single black box. Its purpose is to show how the system interacts with external entities (users, other systems, etc.). It helps define the scope of the system.
Container Diagram: A step down in detail from a context diagram, this view shows the major containers or applications within the system (e.g., a web application, a database, a mobile app). It focuses on how these applications work together to deliver the system's functionality.

Low-Level Diagrams

These diagrams provide a detailed, zoomed-in view of the system's internal workings, showing how specific parts are structured and how they behave.

Component Diagram: This diagram shows the smaller, modular parts within a container and their dependencies.
Class Diagram: This type of diagram illustrates the static structure of a system by showing the different classes, their attributes, methods, and relationships. It's used to define the system's blueprint.

Sequence Diagram: A sequence diagram shows the dynamic behavior of a system by illustrating the order of events and the interactions between objects or components over time. It's great for visualizing a specific use case or flow.

ERD (Entity-Relationship Diagram): This diagram is a blueprint of the database. It shows the data entities (tables), their attributes (columns), and the relationships between them. It is crucial for understanding how data is stored and connected.

The C4 Model: A Framework for Diagrams

The C4 model is a popular framework for visualizing software architecture that organizes these different diagrams into a logical hierarchy. It stands for Context, Containers, Components, and Code. Each "C" represents a specific level of abstraction, or "zoom level," designed for a different audience.

Context (Level 1): The highest level, matching the Context Diagram.
Containers (Level 2): Zooms in to show the major applications and services, matching the Container Diagram.
Components (Level 3): Breaks down a single container into its internal parts, similar to your Component Diagram.
Code (Level 4): The lowest level, showing the actual code structure (e.g., classes and their relationships), which aligns with your Class and Sequence Diagrams.

Production App Architecture example

I will pass the parts that are known for most people which are CI/CD with Github Actions or Jenkins,

and move to implementing a load balancer like Ngnix to handle user requests, ensuring a reliable storage, using external logging and monitoring services like Sentry or PM2, that send alerts to an alert service that shows alerts to the connected users and sends immediate alert messages to the developers using a communication channel like Slack so action can be taken immediately to fix the issue.
and the first thing developers are gonna look at is the logs, so devs go searching for the root of the problem :

The golden rule is to never debug in a production environment but create a staging/testing area to ensure that the users wont get infected by the debugging process

The Pillars of system design

Scalability : system growth.
Maintainability : ensuring future developers can understand and build on top of the system.
Efficiency : best use of existing resources.
Reliability : planning for failure, and ensuring that its handled smoothly and not only working on best case scenarios, (maintaining composure when things go wrong)

The Heart of System Design

Moving Data : ensuring that data can flow seamlessly from one part of the system to another, user requests seeding the servers or transfer between databases.
Storing Data : Choosing between Relational and No-Relational based databases and having a deep understanding of the following concepts :
- access patterns
- indexing strategies
- backup solutions

Ensuring that the data is not just stored securely but readably available when needed.

Transforming Data : taking raw data and turning it into meaningful information, for example aggregating log files for analysis or converting user input to understandable format like json.

CAP Theorem or Brewer’s Theorem

That’s a set of principles that guide us in making informed trade-offs between 3 key components of a distributed system :

Consistency : ensures that all nodes of the same system have the same data at the same time.

Availability : the system should be operational and available for requests 24/7, regardless of what happens behind the scenes

the SLO is the objective of the service, what’s expected and SLA is the agreement with the users about the expected uptime, what we are committing to provide, for example if we commit to 99.9% availability and drop below that, we might have to provide refunds or other compensations

Partition Tolerance : the system ability to continue functioning even when a network partition occur, meaning if there’s a disruption in communication between nodes, the system still works.

Following to CAP’s Theorem, a system can only achieve 2 of these principles/properties at the same time. if we appreciate Consistency and Partition Tolerance we might have to compromise on Availability and vise versa.

Building Resilience into a system

Building resilience into a system means expecting the unexpected, that could mean implementing a resilient system, ensuring that there’s always a backup ready to take over in case of failure

Measuring Speed: Throughput & Latency

we have two unity measures apparently Throughput and Latency

Throughput : Its mainly how much data a system can handle in a given time frame

Server throughout - measured in RPS (Request per second), how many requests our system can take per second, the higher the better.
DB throughput - measured in QPS (Query Per second), how many queries per second can the system take.
Data throughput - measured in Bits/s, that reflects the amount of data transferred over the network or processed by the system in a given period of time.

Latency : Its simply the amount of time taken to process a single request

Optimizing for one can often lead to sacrifices in the other.

Designing a system poorly can lead to several issues from performance bottlenecks to security vulnerabilities, if you think refactoring code is hard then redesigning a system is x10 times harder (monumental task), so designing the right system could lay the right foundation to support futuristic features and user growth.

Network Basics

We are basically discussing how computers communicate with each other.

IP Layer :

Each device in the internet is represented by an IP Address, these days its IPv4 which is based on 32bits and can offer 4Billion unique IP addresses but with the growth of number of devices in the world, a migration to an IPv6 will be necessary, which uses 128-bit significantly increasing the number of unique addresses to 340 Tera addresses.

when devices communicate between each other they send packets of data, each packet contains an IP header which contains some meta data, like the sender and the receiver IP addresses to ensure that the data reaches the right destination, this process is governed by the Internet Protocol (IP) which is a set of rules that defines how data is sent and received.

Application Layer :

here where we store the data related to the application protocol, the data in this package is formatted according to specific application protocol data like HTTP for web browsing, so the data is interpreted correctly by the receiving device

Transport Layer :

where TCP (Transmission Control Protocol) and UDP (User DataGram Protocol) come to play

TCP ensures reliable communication like a delivery guy that makes sure that the package to the right destination AND that nothing is missing, each data packet also includes a TCP Header which includes important information like the ports and control flags necessary for managing the connection and data flow;

TCP is known for reliability and its famous 3 way hand-shake which establishes connection between 3 devices.

UDP is faster but not as reliable as TCP, it doesnt establish connection before sending data nor ensures the delivery or order of the package, makes it preferrable for time sensitive communication like video calls or live streaming where speed is crucial and some data loss is acceptable

To tie all these concepts together we have DNS.

DNS (Domain Name Server) is basically like the internet phone book translating human friendly domain names into IP-Adrresses, when u enter a domain in the browser, the browser sends a DNS query that searches for the corresponding IP Address allowing it to establish the connection and retrieve the web page,

the functioning of DNS is overseen by

ICANN(Internet Corporation For Assigned Names And Numbers), which coordinates the global IP address space and domain name system, and Domain Name Registererers like Namecheap or GoDaddy are accredited by ICANN to sell domain names to the public.

DNS, uses different types of records like
A Records : which map the domain to it’s corresponding IPv4, ensuring that the requests reach the correct destination

or

AAAA Records: which basically do the same thing but map the domains to an IPv6 Address.

Networking Basic Infrastructure :

Public IP Addresses : Unique across the internet

Private IP Addresses : Unique only within the network

and IP Address can be Static, permanently assigned to a device or Dynamic, changing over time (commonly used in residential areas, Cafes and public spaces)

Devices within a Local Area Network can communicate with each other directly, and usually a firewall is used to monitor and filter the ingoing and outgoing packets from that local network.

Device Services are Identified by ports which when combined with an IP Address create a Unique Identifier for a network service.

some ports are reserved for specific services like :

HTTP - 80
HTTPS - 443
MySQL - 3306
SSH - 22

HTTP; Request - Response Protocol with no memory, it doesnt have to store any request context
the POST/GET requests contain all the necessary information like the RequestURL, The Request Method, Status, Policy …etc.

each status code represents a specific state of the request

each request has a method GET, POST, PUT, DELETE (CRUD)

Web Protocols

HTTP is a one way connection, if we want real time two way connection (Real time updates) we use websockets.
WebSockets; provides a 2 way communication channel over a single long-lived connection, allowing servers to push real time updates to clients, like Chat Applications, without the overhead of HTTP Request-Response Cycles

Email Related Protocols :

SMTP; an ideal protocol for sending emails
IMAP; used to retrieve emails from a server (internet Message Access Protocol)
POP3; Used for downloading emails from a server to a local client.

File Transfer Protocols :

FTP : transferring files over the internet
SSH: For Command-line login and file; used for operating network services securely on an unsecured network; exp: logging into a remote machine and executing commands or transferring files

Real-time Communication Protocols :

WebRTC : enables browser to browser applications for voice calling, video chat and file sharing without internal or external plugins, essential for live streaming or video conferencing.

MQTT : Lightweight messaging protocol (message queuing Telemetry Transport); ideal for devices with limited processing power (IoT devices)

RPC (Remote Procedure Call); a protocol that allows a program on one computer to execute code on a server or another computer, its a method used to invoke a function as if it was a local call while in reality the code is executed in a remote machine.

in Web services, HTTP requests can make RPC calls to execute code on behalf of a client to execute a specific functionality.

API Design

In API design we are concerned with defining :

the inputs; (what the users enters), like products details for a new product which is provided by a seller.
the ouputs the information returned when someone queries a product of an API.

The focus is mainly on how the CRUD operations are exposed to the user interface

There different types of APIs like GraphQL, REST and gRPC, each with its own set of principles, protocols and standards, the most used one is REST which stands for REpresational State Transfer,

gRPC stands for google Remote Procedure Call, in the image the positive and negative parts of each type could be seen.

the way an API is structured can largely vary from use case to use case but that’s the ideal format to follow, here’s an example of fetching the orders for a specific user in a E-Commerce application :

common queries using best practices can also include limits and pagination like so :

a Get Request must be Idempotent, the result never changes upon querying the get request repeatedly and a Get request must never affect changes, we have POST request for that.

When modifying endpoints it’s important to maintain Backward Compatibility and Versioning, that means that we need to ensure that changes don’t break existing clients, a common practice is to introduce new versions, like version to product; version 1 API can still serve the old client and the version 2 API should serve the new client

Another best practice is to integrate a Rate Limiter to prevent DDOS attacks, it basically prevents a user from sending multiple requests a user can make in a certain time frame , a common practice is to also set CORS settings (cross origin resource sharing) to only accept requests from specific domains to prevent unwanted cross site interactions.

Now image a company is hosting a website in a server on google cloud data centers in Finland, the response time for users across Europe would be 100ms but for users from US or AFRICA or Mexico the respond time might get longer up to 4000ms or more.

but fortunately, there are strategies to minimize this request latency for people who are far away

these strategies are :

Caching and CDNs

caching is a technique used to improve the performance and efficiency of a system, it involves storing a copy of data in temporary storage so that future request for that data can be stored faster, there are 4 common places where cache can be stored :

Browser Caching;

using websites resources on the user’s local computer (usually bundled files html,css and js) typically in a dedicated cache directory managed by the browser so when the user revisits the site the browser can load the cache rather than fetching everything from the server again, there’s a cache control header to tell the browser how long this content should be cached, we can check that on the developer tools.

   cache hit means cache was searched for and found.

cache miss means cache was searched for and not found.

the higher the cache ratio the better (more reliable cache)

Server Caching;

involves storing frequently used data on the server side, reducing the use of expensive operations like querying the database, they are usually stored on a server or a separate cache server, either on memory like Redis or on disk.

There are several ways to deal with the caching on the server side :

Write around cache; The server checks if the data exists on the cache first, if it wasn’t found then it queries the database and stores it in the cache.
Write-Through Cache; data simultaneously written to cache and the database to ensure asynchronization but it might be slower than write-around cache.

Write-Back cache; where data is first returned to the cache and then to permanent storage at a later time, but the storage can get quickly full like that, that why we have

Eviction Policies :

Database Caching;

that’s another crucial concept, it includes storing query results to improve the performance of database driven applications, It’s often done within the database system itself or via an external caching layer like Redis or Memcache, when the query is made we check the cache first to see if the result of that query has been already stored, if it is we return the cached data, if not we query the database and the result is stored on the cache for future use, that’s super beneficial for Read Focused applications, also the same eviction policies used on the server side caching, apply here too

CDNs;

They are basically a network of servers distributed graphically, they are usually used to serve static content like html, css and javascript, images and other assets.

they cache the content from the original server and deliver it to the end user from the nearest CDN server. There are 2 types of CDNs: Pull-based and Push-Based.

in Pull-based, it first checks if the assets are existing in the nearest server, if not it pulls them from the main server.

In the Push-based we usually put the assets in the main server and it automatically pushed them to the CDN servers, requires more management.

Proxy Servers

they act as an intermediate between the Client asking for resources and the server providing that resources, the proxy servers can serve many other purposes.

there are different types of Proxies :

The most commonly used proxies are Forward Proxy and Reverse Proxy :

A forward proxy acts as a middleman for a client to access the internet. It protects the client's identity by making requests on their behalf, hiding their IP address and providing a layer of security and privacy. Forward proxies are often used in corporate or school networks for content filtering and enforcing usage policies.

A reverse proxy acts as a middleman for a server (or group of servers) to accept requests from the internet. It protects the servers by receiving all incoming traffic and then forwarding it to the appropriate internal server, which hides the server's identity. Reverse proxies are used for things like load balancing, SSL encryption, and protecting against DDoS attacks.

Load Balancers

A load balancer acts as a traffic controller, distributing incoming network traffic across multiple servers to prevent any single server from becoming overwhelmed. It ensures high availability, improves application performance, and allows for seamless scaling.

Types of Load Balancers

There are three main types of load balancers:

Hardware Load Balancers: These are physical appliances with dedicated hardware and software, offering high performance and security for large-scale, on-premise deployments.
Software Load Balancers: These are applications that run on standard servers, offering more flexibility and lower cost. They can be installed on-premise or in the cloud.
Cloud Load Balancers: These are fully managed services provided by cloud providers like AWS or Google Cloud. They are scalable, easy to configure, and integrate seamlessly with other cloud services.

Used Algorithms

Load balancing algorithms determine how traffic is distributed. They can be broadly categorized into two types:

Static Algorithms

These algorithms don't consider the current state of the servers (e.g., their load or health).

Round Robin: Distributes requests sequentially to each server in a rotating fashion. Simple and effective for servers with equal capacity.
Weighted Round Robin: Assigns a "weight" to each server based on its capacity, sending more requests to more powerful servers.
IP Hash: Uses a hash of the client's IP address to ensure requests from the same client are always sent to the same server, which is useful for maintaining session persistence.

Dynamic Algorithms

These algorithms actively monitor server health and load to make more intelligent routing decisions.

Least Connections: Directs new requests to the server with the fewest active connections. This is good for environments where connection times vary.
Least Response Time: Sends requests to the server with the lowest response time, ensuring the fastest service.
Resource-Based: Uses an agent on each server to report real-time metrics (like CPU or memory usage) and directs traffic to the server with the most available resources.

Databases (Sharding, Replication, ACID, Vertical & Horizontal Scaling)

Databases

Databases are organized collections of data, typically stored electronically. To handle increasing amounts of data and traffic, different strategies are used to manage and scale them.

Sharding

Sharding is a technique that breaks up a large database into smaller, more manageable parts called shards. Each shard is a separate database that contains a subset of the total data. The main goal of sharding is to improve performance by distributing the load across multiple servers. Instead of one server handling all the queries, multiple servers handle a portion of the workload, reducing the strain on any single machine. This is a form of horizontal scaling.

Replication

Replication is the process of creating and maintaining multiple copies of a database. It's used for two primary reasons:

High Availability: If one server fails, another copy (or replica) can take over, ensuring the database remains accessible.
Load Balancing: Read traffic can be distributed across multiple replicas, reducing the load on the primary server.

There are different types of replication, but the most common model is Primary-Replica (also known as Master-Slave), where one database is the primary (writes are allowed), and others are replicas (reads are allowed).

ACID Properties

Relational Databases are ACID compliant is a set of properties that guarantee a database transaction is processed reliably. They are fundamental to ensuring data integrity, especially in relational databases.

Atomicity: A transaction is treated as a single, indivisible unit. It either completes entirely or doesn't happen at all. There are no partial transactions.
Consistency: A transaction must bring the database from one valid state to another, ensuring all data integrity rules are maintained.
Isolation: Concurrent transactions are isolated from each other. The result of a transaction is the same as if it were the only transaction running.
Durability: Once a transaction is committed, it will remain so even in the event of a system failure (like a power outage).

Scaling

Scaling is the ability to handle increased demand. There are two main ways to scale a database:

Vertical Scaling (Scaling Up): This involves increasing the resources of a single server, such as adding more CPU, RAM, or storage. It's simpler to implement but has a physical limit and can be more expensive. Think of it as upgrading a single computer with better parts.
Horizontal Scaling (Scaling Out): This involves adding more servers to the system. It's more complex to manage but provides greater flexibility and is often more cost-effective for large-scale systems. Sharding and replication are examples of horizontal scaling. Think of it as adding more computers to the network.

For the record, the images and diagrams used in this guide are from FreeCodeCamp.

n8n self-hosting with persistent workflows

Oussama Belhadi — Mon, 25 Aug 2025 08:10:39 +0000

🚀 How I Self-Host n8n with Docker + Git for Persistent Workflows

If you’re like me and love automating things with n8n, self-hosting is the best way to have full control over your workflows.

But there’s a catch:

👉 By default, workflows are stored inside the container’s DB. If something breaks, you risk losing them.

👉 And moving workflows between devices isn’t exactly plug & play.

So, I built a simple setup that makes self-hosting persistent, Git-friendly, and portable, using Docker + Git hooks.

You can find the repo here

📦 Project Structure

Here’s the layout of my n8n-docker repo:

n8n-docker/
├── docker-compose.yml      # Docker Compose setup for n8n
├── export-workflows.sh     # Script to export all workflows as JSON
├── workflows/              # Folder where exported workflows are stored (Git-tracked)
├── n8n\_data/               # Docker volume for SQLite DB, credentials, and user account
└── .git/hooks/
└── pre-commit          # Git hook to auto-export workflows before commit

⚙️ How it Works

Docker Compose Setup
- docker-compose.yml runs n8n in a container.
- Data is persisted in n8n_data/ so you don’t lose your account or workflows after restart.
Automatic Workflow Export
- A helper script export-workflows.sh uses docker exec to dump all workflows as JSON into the workflows/ folder.
- This makes it easy to track workflows in Git.
Git Pre-Commit Hook
- Before every commit, .git/hooks/pre-commit automatically runs the export script and stages updated JSONs.
- Result: your Git history always has the latest workflows.

No more “oh crap I forgot to export that workflow.”

🚀 Usage

Start n8n with Docker

docker compose up -d

Commit changes to Git

Create or edit workflows in n8n.
Run a commit:

git commit -m "Update workflows"

The hook auto-exports workflows → stages them → includes them in your commit.

Move to another device

Clone the repo.
Run docker compose up -d.
Import workflows from the workflows/ folder.
(Note: credentials are not included — you’ll need to recreate those manually).

⚠️ Notes

The container must be running for workflow export to work.
Credentials are stored encrypted in n8n_data/, not exported to Git.
Add a .gitignore to keep secrets and .env files safe.

🔗 References

This setup has been a game-changer for me — it feels like using n8n with built-in Git support.
Now, even if I nuke my server, I can recover my workflows in minutes.

Predicting Customer Churn with TensorFlow – A Beginner-Friendly Guide

Oussama Belhadi — Mon, 18 Aug 2025 10:21:12 +0000

Introduction

Customer churn is when customers leave a company. Predicting churn helps businesses retain valuable customers and increase revenue.

In this tutorial, I’ll show you how to use TensorFlow, pandas, and scikit-learn to build a neural network that predicts churn based on a real dataset.

You can find a working ready to test/use example in my Github

No heavy theory — just step-by-step coding, explanations, and visuals.

We have a .csv file that holds the customers data and we will use it as a dataset to train our model today, It's also available in the Github Repo

Step 1: Setting Up the Environment

We need these libraries:

pip install pandas numpy scikit-learn tensorflow matplotlib

pandas → for data manipulation
numpy → for numeric computations
scikit-learn → preprocessing, scaling, train/test splitting
tensorflow → building neural networks
matplotlib → plotting results

Step 2: Load and Inspect the Dataset

Load the dataset with pandas:

import pandas as pd

df = pd.read_csv("customer_churn.csv")
df.head()

Tip: ⚠️ Always check your column names. Spaces or extra characters can break code later:

df.columns = df.columns.str.strip().str.replace(" ", "_")

Step 3: Clean the Data

Convert numeric columns with potential issues:

df['Total_Charges'] = pd.to_numeric(df['Total_Charges'], errors='coerce')

Drop missing rows and irrelevant columns:

df = df.dropna()
df.drop('Customer_ID', axis=1, inplace=True)

Step 4: Encode Categorical Variables

Neural networks cannot process text. Convert categories to numbers:

from sklearn.preprocessing import LabelEncoder

df['Churn'] = df['Churn'].map({'Yes': 1, 'No': 0})

cat_cols = df.select_dtypes(include='object').columns
le = LabelEncoder()
for col in cat_cols:
    df[col] = le.fit_transform(df[col])

Example: Male → 1, Female → 0. Similarly for other categories.

Step 5: Split Features and Target

Separate input features (X) and output (y);
Before scaling, each feature (column) has its own mean and standard deviation. Neural networks learn better when features are roughly in the same range.

Mean: average value of the feature
Standard Deviation: measures how spread out the values are

*The formula for the mean (μ) of a dataset with N values is:
*

Standard Scaler subtracts the mean and divides by the standard deviation,
The formula for the standard deviation is:

After scaling, each feature has mean ~0 and std ~1.

X = df.drop('Churn', axis=1)
y = df['Churn']

Scale features (important for neural networks):

from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)

Split into train/test sets:

from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X_scaled, y, test_size=0.2, random_state=42)

Step 6: Build and Train the Neural Network

from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense

model = Sequential([
    Dense(32, activation='relu', input_shape=(X_train.shape[1],)),
    Dense(16, activation='relu'),
    Dense(1, activation='sigmoid')
])

Why these layers and activations?

Dense(32) and Dense(16) → number of neurons in each hidden layer. Experiment to see what works best.
ReLU activation → introduces non-linearity, helps the network learn complex patterns.
Sigmoid in output → outputs a probability between 0 and 1, perfect for binary classification.

Optimizer: Adam
model.compile(
    optimizer='adam',
    loss='binary_crossentropy',
    metrics=['accuracy']
)

Why Adam?

Adaptive optimizer: adjusts learning rate automatically
Combines advantages of Momentum and RMSProp
Works well out-of-the-box for most problems
Loss function: binary_crossentropy → suitable for predicting 0/1 outcomes.
Metric: accuracy → how often the model predicts correctly.
Training

history = model.fit(
    X_train, y_train,
    validation_data=(X_test, y_test),
    epochs=20,
    batch_size=32
)

Epochs = 20 → model sees the dataset 20 times.

Batch size = 32 → updates weights every 32 samples.

Step 7: Evaluate and Visualize

import matplotlib.pyplot as plt

plt.plot(history.history['accuracy'], label='Train')
plt.plot(history.history['val_accuracy'], label='Validation')
plt.title('Accuracy over Epochs')
plt.legend()
plt.show()

Ouput Example :

Train vs Validation curves → check for overfitting/underfitting.

Step 8: Predict Churn for a New Customer

import numpy as np
import pandas as pd

new_customer = pd.DataFrame([{
    'Gender': 0, 'Senior_Citizen': 0, 'Partner': 1, 'Dependents': 0,
    'tenure': 12, 'Phone_Service': 1, 'Multiple_Lines': 0, 'Internet_Service': 0,
    'Online_Security': 2, 'Online_Backup': 0, 'Device_Protection': 1,
    'Tech_Support': 0, 'Streaming_TV': 0, 'Streaming_Movies': 1, 'Contract': 0,
    'Paperless_Billing': 1, 'Payment_Method': 2, 'Monthly_Charges': 50.0, 'Total_Charges': 500.0
}])

new_customer_scaled = scaler.transform(new_customer)
churn_prob = model.predict(new_customer_scaled)[0][0]
churn_label = int(churn_prob > 0.5)

print(f"Churn Probability: {churn_prob:.2f}")
print(f"Churn Prediction: {churn_label} ({'Yes' if churn_label==1 else 'No'})")

Conclusion

You now have a complete pipeline to:

Clean and preprocess data
Train a neural network in TensorFlow
Evaluate model performance
Predict churn for new customers

This workflow is reusable for other tabular datasets and binary classification problems.

Automating Script Execution and Building a Production-Ready Data Pipeline with GitHub Actions

Oussama Belhadi — Mon, 21 Jul 2025 12:24:37 +0000

Learn how to set up a fully automated workflow to fetch external data, update your web app’s content, and trigger redeployment using GitHub Actions. In this guide, I’ll use news fetching as a practical example, but the approach applies to any data pipeline. You’ll see real-world CI/CD, automation tips, and how to keep your site’s data up-to-date—no manual intervention required!

🚀 The Goal

I wanted my Next.js app to always display the latest news from a third-party site, without me having to manually update a JSON file or trigger a redeploy. The solution? Automate everything: scraping, data update, and deployment.

🛠️ The Stack

Next.js (App Router)
Tailwind CSS + DaisyUI
Puppeteer for scraping
GitHub Actions for CI/CD
Vercel for hosting

🧩 The Workflow

1. Scraping News with Puppeteer
I wrote a Node.js script using Puppeteer to scrape news headlines and details, then save them to src/data/news.json in my repo.
Key code snippet:

const puppeteer = require('puppeteer');
const fs = require('fs');

async function scrapeNews() {
  const browser = await puppeteer.launch({
    headless: true,
    args: ['--no-sandbox', '--disable-setuid-sandbox'] // Required for CI!
  });
  const page = await browser.newPage();
  await page.goto('https://www.news-website.com/', { waitUntil: 'networkidle2' });
  // ...scraping logic...
  fs.writeFileSync('src/data/news.json', JSON.stringify(news, null, 2));
  await browser.close();
}
scrapeNews();

Pro tip:
If you run Puppeteer in CI (like GitHub Actions), you must use the --no-sandbox and --disable-setuid-sandbox flags, or Chromium will fail to launch.

2. Automating with GitHub Actions

I created a workflow (.github/workflows/scrape.yml) to:
Run on a schedule (every 12 hours) or manually
Install dependencies
Run the scraper
Commit and push the updated news.json back to GitHub
Key workflow steps:

permissions:
  contents: write

jobs:
  scrape:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - uses: actions/setup-node@v4
        with:
          node-version: '18'
      - run: npm ci
      - run: node utils/News/scrape-news.js
      - run: |
          git config user.name "github-actions[bot]"
          git config user.email "github-actions[bot]@users.noreply.github.com"
      - run: |
          git add src/data/news.json
          git diff --cached --quiet || git commit -m "chore: update news.json [auto]"
          git push

Don’t forget:
Add permissions: contents: write at the top level of your workflow, or you’ll get a 403 error when trying to push.

3. Automatic Vercel Redeploys
Vercel is connected to my GitHub repo. When the workflow pushes a new commit (with updated news), Vercel automatically rebuilds and redeploys my app. The site always shows the latest scraped news—no manual intervention!

🐛 Common Pitfalls & Fixes

Puppeteer fails to launch in CI:
Add args: ['--no-sandbox', '--disable-setuid-sandbox'] to your launch options.

Workflow can’t push to repo:
Add permissions: contents: write to your workflow YAML.
Case-sensitive paths:
Linux (CI) is case-sensitive! Double-check your file and folder names.

The Result
News is scraped and updated automatically, every 12 hours.
My Next.js app always displays fresh content.
No manual data updates or redeploys needed.
The workflow is robust, production-ready, and easy to maintain.

Learning COBOL as a Beginner — A Simple Step-by-Step Guide

Oussama Belhadi — Wed, 16 Jul 2025 12:21:36 +0000

Hey there! 👋
I'm currently learning COBOL — yes, that old programming language from the 1960s! I know it looks outdated, but it’s still used in critical systems like banks, airlines, and government software.

I decided to document my learning journey here, in the simplest and shortest way possible, in case it helps someone else out there who's just getting started like me.

📚 1. What is COBOL?

COBOL stands for Common Business-Oriented Language

It was created in 1959 to process business data (e.g., invoices, reports)

Still used today in huge legacy systems

Super readable (like plain English), but very strict in structure

🧱 2. Structure of a COBOL Program

COBOL programs are divided into four main divisions, always in this order:

IDENTIFICATION : Metadata like name, author, etc.
ENVIRONMENT : System configuration (often skipped by beginners)
DATA : Variable declarations
PROCEDURE: The actual logic — your code goes here!

🧾 Example Program

      IDENTIFICATION DIVISION.
       PROGRAM-ID. HelloWorld.

       ENVIRONMENT DIVISION.

       DATA DIVISION.

       PROCEDURE DIVISION.
           DISPLAY "Hello, World!".
           STOP RUN.

📏 3. COBOL Columns (Important!)

COBOL was made for punch cards, so it has strict rules about where you write your code:

1 – 6 : Sequence numbers (optional, can ignore)
7 : * for comments, - for line continuation
8–72 : Actual code goes here (indentation often starts at column 8)
73–80 : Ignored by compiler

📝 4. Declaring Variables (DATA DIVISION)

       DATA DIVISION.
       WORKING-STORAGE SECTION.
       01 Name     PIC A(20).
       01 Age      PIC 99.

Explanation:

PIC = Picture Clause, defines the variable type

A(20) = 20 alphabetic characters

99 = 2-digit number

🧾 5. Input and Output

       DISPLAY "Enter your name: ".
       ACCEPT Name.
       DISPLAY "Hello, " Name.
DISPLAY shows a message

ACCEPT takes input from the user

🔁 6. IF Conditions

  IF Age > 18
           DISPLAY "You are an adult."
       ELSE
           DISPLAY "You are a minor."

Pretty readable, right?

🔄 7. Loops with PERFORM

  WORKING-STORAGE SECTION.
       01 I      PIC 9 VALUE 1.

       PROCEDURE DIVISION.
           PERFORM VARYING I FROM 1 BY 1 UNTIL I > 5
               DISPLAY "Count: " I
           END-PERFORM.

Prints:

Count: 1
Count: 2
Count: 3
Count: 4
Count: 5

That’s what I’ve learned so far!
This post is meant to be a beginner-to-beginner guide, I’ll keep sharing more as I go deeper into COBOL.
If you’re also learning COBOL or just curious about it, feel free to follow along! Let’s bring a bit of life into this powerful (but old-school) language.

How to Work With Different People on Your Team

Oussama Belhadi — Tue, 10 Jun 2025 10:16:54 +0000

Teamwork: Mastering the Art of Working with Anyone

Learning to navigate differences and finding ways to work effectively with different kinds of people isn’t just smart, it’s a practical strategy to get things done without losing your mind.

If you take the time to notice people around you, you'll find that they all have unique styles in the way they talk, how they work, and what motivates or drives them. Each person has a distinct personality that may appeal to you because it’s relatable, or bother you because you can’t comprehend the way they think, feel, and act.

But we hardly pay attention to others. We hardly take the time to understand what makes them act a certain way. Not understanding others not only leads to misunderstandings, frustration, and conflict, it makes us miserable because we simply can’t stand people we need to work with.

For example:

If you’re a fast-paced go-getter, you may feel slowed down by someone who’s more detail-oriented, while they may feel rushed and pushed into a decision.

You prefer collaborating closely, while someone else likes to work independently with minimal interference. You may find them as cold and unresponsive, while they may think of you as annoying.

Unlocking the "Idiots" Code

I recently read a book that shed a lot of light on these dynamics: "Surrounded by Idiots" by Thomas Erikson. The title is quite catchy, but the insights are profound. Erikson introduces a simple yet powerful framework, categorizing different behavioral types into four colors: Red, Yellow, Green, and Blue. It’s like having a guide to truly understanding why people do what they do.

This book really resonated with me because it offers a practical lens for viewing team interactions. Instead of getting frustrated by what seems like inexplicable behavior, you can start to identify different communication styles and, more importantly, adapt your own approach to connect better with colleagues.

So, how can we leverage this understanding to work more effectively with everyone on the team?

Start with You: Know Your Own Style

Before you try to understand others, take a moment to understand yourself. What’s your dominant communication style? Are you direct and results-focused, or do you prefer to brainstorm and socialize? Do you prioritize harmony and stability, or are you driven by facts and logic? Recognizing your own tendencies is the first step to adapting how you interact with others. Knowing your style helps you predict how you might come across to someone with a different approach.

Embrace the Blend
Imagine a team where everyone thought and acted exactly the same way. It would be pretty uninspired, wouldn't it? Diversity in personalities is a strength, not a hindrance. Different perspectives lead to more robust solutions and a richer work environment. Embrace the unique mix of people around you.

Decode Their Communication
While you don't need to formally "color-code" your colleagues, noticing how people prefer to communicate is incredibly helpful.

The "Reds" (Direct & Driven): They want action and results. Give them the bottom line.
The "Yellows" (Social & Enthusiastic): They love to talk and brainstorm. Engage them with ideas and collaboration.
The "Greens" (Stable & Harmonious): They value peace and teamwork. Be patient and emphasize shared support.
The "Blues" (Analytical & Precise): They need facts and details. Present information logically and with data.

Adjust Your Approach, Don't Force Theirs
This is perhaps the most crucial takeaway. It’s not about trying to force your colleagues to adopt your communication style. Instead, the responsibility is on you to adapt.

For the direct types (Red): Be concise, focus on outcomes.
For the social types (Yellow): Be engaging, open to new ideas.
For the harmonious types (Green): Be patient, empathetic, and emphasize team well-being.
For the analytical types (Blue): Be factual, provide details, and present information logically.

This isn't about compromising your authenticity, but rather being flexible in your delivery to ensure your message lands effectively.

Observe and Listen Deeply
Beyond general communication styles, pay close attention to individual cues. What truly motivates your colleagues? What are their preferred ways of working? Do they respond better to email, instant messages, or in-person discussions for certain topics? Observe their reactions, listen to their concerns, and ask open-ended questions to gain a deeper understanding.

Build a Safe Space
Regardless of personality type, everyone thrives in an environment where they feel safe to express ideas, ask questions, and even make mistakes without fear of judgment. Encourage open dialogue, be approachable, and create a culture of respect where differences are celebrated.

Focus on the Finish Line
Ultimately, despite our individual differences, we are united by common objectives. Regularly remind your team of the overarching goals and how each person's unique contribution is vital to achieving them. This shared purpose can help bridge communication gaps and foster a strong sense of collective responsibility.

Working effectively with different people on your team doesn't have to be a constant uphill battle. By understanding that seemingly "difficult" behaviors are often just different ways of navigating the world, and by actively adapting your communication and approach, you can transform potential friction into powerful synergy. So, go forth, embrace the incredible diversity of your team, and watch your collective success soar!

EU Tech Law Update: What Devs Need to Know

Oussama Belhadi — Tue, 10 Jun 2025 08:21:49 +0000

The EU has been busy making some changes to its tech laws, and as developers and startup founders, it's super important to know how these updates impact our work.
Good news: some areas are getting a bit easier!
Bad news: new tech means new responsibilities.

1. GDPR (General Data Protection Regulation) for Small Businesses is Getting Easier (Finally!)

Less Paperwork: If you're a small or mid-sized startup (especially under 750 employees), you might see a break on some of the heavy paperwork that GDPR usually demands. The EU wants to cut down on "red tape."
Focus on High-Risk Data: This doesn't mean you can ignore GDPR! You'll still need to be super careful with "high-risk" personal data (like health info or financial data). But for simpler data processing, the burden might be less.
What to do: Keep making privacy a core part of your design. But know that some of the stricter documentation rules might not apply to you if your data handling is low-risk.

2. AI Liability: Software is a "Product" Now!

No Dedicated AI Liability Law (for now): The EU scrapped its specific AI liability proposal. This means there isn't a new, separate law just for AI accidents.
Existing Product Rules Apply: Instead, our existing Product Liability Directive has been updated. The BIG change? Software, including AI systems, is now officially considered a "product."
What this means: If your AI or software causes harm because it's "defective," you could be held responsible – even if it wasn't your fault directly. This is called "strict liability." So, make sure your AI is robust, well-tested, and secure. Document everything!
AI Act Still Important: Don't forget the separate EU AI Act! If your AI is "high-risk" (e.g., in healthcare, hiring), you still have to follow strict rules for safety, transparency, and human oversight.

3. ePrivacy (The "Cookie Law"): Still Around!

No New Cookie Regulation: After years of talk, the EU has actually withdrawn its proposal for a new ePrivacy Regulation.
Old Rules Still Apply: This means the original ePrivacy Directive (the "Cookie Law") is still in force.
What this means: You still need proper cookie consent on your websites and apps. That means:
Clear banners: Tell users what cookies you use.
Real choice: Let them accept, reject, or customize. No pre-ticked boxes!
Easy opt-out: Make it simple to change their mind later.
Direct Marketing: If you send marketing emails or messages, you generally need consent for those too.

Bottom Line for Devs:

The message is clear: build safely and with privacy in mind from the start. While some of the administrative burden for smaller businesses is easing, your core responsibility for user data and product safety remains. If you're building with AI, understand its risks and document your development process thoroughly. And yes, those cookie banners aren't going away anytime soon!

Stay sharp, keep coding!

Is PostgreSQL the Swiss Army Knife of Web Development? Rethinking Your Stack

Oussama Belhadi — Thu, 06 Mar 2025 09:45:40 +0000

We all know and (maybe sometimes) love PostgreSQL as that reliable relational database workhorse. But what if I told you it could be so much more? Recently, I've been exploring the incredible potential of Postgres and it has completely shifted my perspective on what it's capable of.

It turns out that with its amazing flexibility and a vibrant ecosystem of extensions, PostgreSQL can actually replace a whole bunch of specialized tools in your web development stack. Intrigued? I was! Let's dive into some of the key areas where Postgres can shine, potentially simplifying your architecture and reducing tool sprawl.

Beyond Relational: Postgres as a Multi-Tool Powerhouse

Postgres, armed with its built-in features and extensions, can step in for tools you might be currently relying on:

NoSQL Database? JSONB to the Rescue! Need to handle schemaless data or flexible documents? Postgres' JSONB data type lets you store and efficiently query JSON data right within your relational database. Forget spinning up a separate NoSQL database for certain use cases – Postgres has you covered.
Cron Jobs? Say Hello to pg_cron! Scheduling tasks directly from your database? The pg_cron extension allows you to schedule cron-like jobs within Postgres itself, simplifying your infrastructure and keeping scheduled tasks close to your data.
In-Memory Cache? Unlogged Tables for Speed! Need a fast, temporary cache? Unlogged tables in Postgres provide in-memory-like performance without the complexity of setting up and managing a dedicated cache like Redis for certain scenarios.
Vector Database? pgvector and pgai for AI Magic! Jumping into the world of vector embeddings and AI? Extensions like pgvector and pgai turn Postgres into a viable vector database, enabling similarity searches and vector-based operations directly within your database.
Full-Text Search? Built-in Power! Need robust search capabilities? Postgres has powerful built-in full-text search functionality. You might not need to reach for dedicated search engines for many applications.
GraphQL API? pg_graphql Makes it Easy! Want to expose your data via GraphQL? The pg_graphql extension lets you build GraphQL APIs directly on top of your Postgres database, streamlining API development.
Real-time Data Sync? ElectricSQL for the Win! Need real-time data synchronization? ElectricSQL can bring real-time capabilities to Postgres, opening up possibilities for collaborative applications and live updates.
Authentication? pgcrypto and pgjwt for Security! Handling authentication? Extensions like pgcrypto and pgjwt provide cryptographic functions and JWT support directly within Postgres, enhancing your database's security capabilities.
Time-Series Analytics? pg_mooncake for Insights! Working with time-series data? pg_mooncake extension adds time-series analytics capabilities to Postgres, making it suitable for monitoring and trend analysis.
REST APIs? PostgREST for Instant Endpoints! Need to quickly expose your data via REST APIs? PostgREST can automatically generate RESTful APIs from your Postgres database, significantly speeding up backend development.
Frontend Hosting? Serve Static Files Directly! Believe it or not, you can even serve frontend code directly from Postgres in certain situations! While maybe not for high-traffic production, it's a fascinating capability for simpler setups or internal tools.

Neon: Serverless Postgres in the Spotlight

If you're looking to explore Postgres in a serverless environment, Neon is definitely worth checking out.

The "But..." - Considerations for Real-World Use

Now, before you go ripping out half your tech stack and replacing it with Postgres, it's crucial to remember: just because you can do it, doesn't mean you always should.

It's essential to consider:

Performance: While Postgres is powerful, specialized tools are often optimized for their specific tasks. Evaluate if Postgres can handle the performance demands of replacing a dedicated tool in your specific use case.
Maintainability: Complexity can sometimes shift. While you might reduce the number of tools, ensure that managing a more multifaceted Postgres setup doesn't become more complex in the long run.
The Right Tool for the Job: Sometimes, a dedicated tool is simply the best choice. Don't force Postgres into a role where another tool might be significantly more efficient or better suited.

Final Thoughts: Embrace Postgres' Versatility, But Choose Wisely

This exploration has really opened my eyes to the sheer versatility of PostgreSQL. It's not just a relational database anymore; it's a platform capable of handling a surprising range of web development needs.

While it's tempting to jump on the "one database to rule them all" bandwagon, the key takeaway is thoughtful evaluation. Postgres offers incredible potential to simplify your stack, but always consider the specific requirements of your project and choose the right tool for each job.

Azure Pipelines

Oussama Belhadi — Tue, 04 Mar 2025 08:32:45 +0000

1. Introduction to Azure Pipelines

Azure Pipelines is a cloud service that automates the building and testing of your code and deploys it to any target. It works with any language, platform, and cloud.

Key Concepts:

Triggers: Events that initiate a pipeline run (e.g., code commits, scheduled times).
Pools: Groups of agents (virtual machines or containers) that execute pipeline jobs.
Tasks: Pre-built or custom actions performed in a pipeline (e.g., compiling code, running tests).
Steps: Ordered sequences of tasks within a job.
Variables: Values that can be used throughout the pipeline to customize behavior.
Artifacts: Files or packages produced by a pipeline run.
YAML Syntax: Pipelines are defined using YAML (YAML Ain't Markup Language) files.

YAML Syntax Basics:

Indentation is crucial for defining the structure.
Key-value pairs define settings.
Lists are defined with hyphens (-).

2. Java (Maven) Pipeline Explained

Purpose:

This pipeline automates the build, test, and deployment of a Java project using Apache Maven.

YAML File Breakdown:

# Maven
# Build your Java project and run tests with Apache Maven.
# Add steps that analyze code, save build artifacts, deploy, and more:
# [https://docs.microsoft.com/azure/devops/pipelines/languages/java](https://docs.microsoft.com/azure/devops/pipelines/languages/java)
trigger:
- Test

pool: Server Name 

steps:
- task: Maven@3
  inputs:
    mavenPomFile: 'JobAPI/pom.xml'
    goals: 'clean compile process-resources package'
    publishJUnitResults: true
    testResultsFiles: '**/surefire-reports/TEST-*.xml'
    javaHomeOption: 'JDKVersion'
    jdkVersionOption: '1.8'
    mavenVersionOption: 'Default'
    mavenOptions: '-Xmx3072m'
    mavenAuthenticateFeed: false
    effectivePomSkip: false
    sonarQubeRunAnalysis: false

- task: CopyFiles@2
  inputs:
    SourceFolder: '$(Build.SourcesDirectory)/JobAPI'
    Contents: |
      pom.xml
      target/*.jar
    TargetFolder: '$(Build.ArtifactStagingDirectory)'
  condition: and(succeeded(), ne(variables['Build.Reason'], 'PullRequest'))

- task: PublishBuildArtifacts@1
  inputs:
    PathtoPublish: '$(Build.ArtifactStagingDirectory)'
    ArtifactName: 'drop'
    publishLocation: 'Container'
  condition: and(succeeded(), ne(variables['Build.Reason'], 'PullRequest'))

- task: Maven@3
  inputs:
    mavenPomFile: 'JobAPI/pom.xml'
    goals: 'deploy'
    publishJUnitResults: true
    testResultsFiles: '**/surefire-reports/TEST-*.xml'
    javaHomeOption: 'JDKVersion'
    jdkVersionOption: '1.8'
    mavenVersionOption: 'Default'
    mavenOptions: '-Xmx3072m'
    mavenAuthenticateFeed: false
    effectivePomSkip: false
    sonarQubeRunAnalysis: false
  condition: and(succeeded(), ne(variables['Build.Reason'], 'PullRequest'))

trigger: - Test: Runs the pipeline when code is pushed to the Test branch.
pool: ServerName: Uses a self-hosted agent pool named "Server."
steps:: Defines the sequence of tasks.
- Maven@3 (Clean, Compile, Package):
  - mavenPomFile: Specifies the path to the pom.xml file.
  - goals: Defines the Maven goals to execute (clean, compile, process-resources, package).
  - publishJUnitResults: Publishes JUnit test results.
  - testResultsFiles: Specifies the location of test result files.
  - javaHomeOption, jdkVersionOption, mavenVersionOption, mavenOptions: Configures the Java and Maven environment.
- CopyFiles@2 (POM and JAR):
  - Copies the pom.xml and the generated JAR file to the artifact staging directory ($(Build.ArtifactStagingDirectory)).
  - condition: Prevents these tasks from running during pull requests.
- PublishBuildArtifacts@1: Publishes the staged artifacts as "drop."
- Maven@3 (Deploy):
  - Runs the maven deploy goal, to push the artifact to a maven repository.
condition: and(succeeded(), ne(variables['Build.Reason'], 'PullRequest')): Ensures specific steps only run on successful builds and not on pull requests.

Explanation of Maven Goals and Options:

clean: Deletes the target directory.
compile: Compiles the Java source code.
process-resources: Copies resources to the target directory.
package: Packages the compiled code into a JAR file.
deploy: pushes the artifact to a repository.
Xmx3072m: Allocates 3GB of memory to Maven.

Artifact Handling:

The pipeline copies the pom.xml and JAR file to the artifact staging directory and then publishes them as an artifact named "drop."

Deployment Explanation:

The second maven task with the deploy goal, will push the resulting jar file to a maven repository, that is configured inside of the pom.xml file.

3. .NET Core (.NET Framework) Pipeline Explained

Purpose:

This pipeline builds and tests an ASP.NET Core project targeting the full .NET Framework.

YAML File Breakdown:

# ASP.NET Core (.NET Framework)
# Build and test ASP.NET Core projects targeting the full .NET Framework.
# Add steps that publish symbols, save build artifacts, and more:
# [https://docs.microsoft.com/azure/devops/pipelines/languages/dotnet-core](https://docs.microsoft.com/azure/devops/pipelines/languages/dotnet-core)

trigger:
- Test

pool:
  vmImage: 'windows-latest'

variables:
  solution: '**/*.sln'
  buildPlatform: 'Any CPU'
  buildConfiguration: 'Release'

steps:
- task: NuGetToolInstaller@1
- task: NuGetCommand@2
  displayName: 'NuGet restore'
  inputs:
    command: 'restore'
    restoreSolution: '$(solution)'
    feedsToUse: config
    nugetConfigPath: 'TTLineIMSBackoffice/NuGet.Config'
    externalFeedCredentials: 'Telerik Nuget'

- task: VSBuild@1
  inputs:
    solution: '$(solution)'
    msbuildArgs: '/p:DeployOnBuild=true /p:WebPublishMethod=Package /p:PackageAsSingleFile=true /p:SkipInvalidConfigurations=true /p:DesktopBuildPackageLocation="$(build.artifactStagingDirectory)/WebApp.zip" /p:DeployIisAppPath="Default Web Site"'
    platform: '$(buildPlatform)'
    configuration: '$(buildConfiguration)'

- task: VSTest@2
  inputs:
    platform: '$(buildPlatform)'
    configuration: '$(buildConfiguration)'

- task: PublishBuildArtifacts@1
  inputs:
    PathtoPublish: '$(Build.ArtifactStagingDirectory)'
    ArtifactName: 'drop'
    publishLocation: 'Container'

trigger: - Test: Triggers the pipeline on commits to the Test branch.
pool: vmImage: 'windows-latest': Uses a Microsoft-hosted Windows agent.
variables:: Defines pipeline variables.
- solution: Path to the solution file.
- buildPlatform: Build platform (Any CPU).
- buildConfiguration: Build configuration (Release).
steps::
- NuGetToolInstaller@1: Installs the NuGet tool.
- NuGetCommand@2 (Restore):
  - Restores NuGet packages using the specified NuGet.Config file.
  - externalFeedCredentials: provides credentials to a telerik nuget feed.
- VSBuild@1: Builds the solution using MSBuild.
  - msbuildArgs: Configures the build process, including creating a web package (WebApp.zip).
- VSTest@2: Runs unit tests.
- PublishBuildArtifacts@1: Publishes the build artifacts.

Understanding the AS400 and its Connection to DB2

Oussama Belhadi — Mon, 24 Feb 2025 15:12:17 +0000

The AS400, now known as the IBM i, is a unique and powerful system that has played a significant role in the world of business computing. Often misunderstood, it's more than just a legacy system; it's a robust, integrated platform with a rich history and a surprisingly modern architecture. A key component of its strength is its tight integration with the DB2 database. This article will delve into the AS400, its evolution, and its inseparable link to DB2.

What is the AS400 / IBM i?

Originally introduced in 1988 as the Application System/400, the AS400 was designed as a mid-range computer system aimed at businesses of all sizes.

It quickly gained popularity for its reliability, ease of use, and integrated architecture. Over the years, it has evolved significantly, undergoing name changes (first to iSeries, then to System i, and now to IBM i) and numerous technological advancements. However, its core principles of integrated hardware and software, object-based architecture, and strong focus on business applications remain.

Key Characteristics of the AS400 / IBM i:

Integrated Hardware and Software: The AS400 was designed as a complete, integrated system. The operating system (initially OS/400, now IBM i) was tightly coupled with the hardware, providing a highly optimized environment. This integration simplified system management and contributed to its renowned reliability.
Object-Based Architecture: The AS400's operating system is object-based, meaning that everything within the system – programs, data files, even hardware resources – is treated as an object. This approach allows for a high degree of encapsulation and modularity, making development and maintenance easier.
Robust and Reliable: The AS400 has a well-deserved reputation for its reliability. Its architecture, combined with features like journaling and commitment control, helps ensure data integrity and system availability.
Business-Oriented: From its inception, the AS400 was designed with business applications in mind. It provides a robust platform for running ERP systems, CRM applications, and other critical business software.
Modern Capabilities: While often perceived as "legacy," the IBM i is a modern platform that supports current technologies like web services, open-source languages (Python, Node.js), and cloud integration.
DB2 and the AS400 / IBM i: A Symbiotic Relationship

DB2 is IBM's family of relational database management systems (RDBMS). On the AS400/IBM i, DB2 is not just a database; it's the database. It's deeply integrated into the operating system and is the primary means of storing and managing data. This tight integration provides several advantages:

Performance: The close relationship between DB2 and the operating system allows for significant performance optimizations. Data access is highly efficient, and the system is designed to minimize overhead.
Simplified Management: Because DB2 is integrated, database administration is often simpler than on systems where the database is a separate component. Many database management tasks can be performed using the same tools and interfaces used for other system functions.
Data Integrity: The AS400's architecture, combined with DB2's features like journaling and commitment control, helps ensure data integrity. This is crucial for business applications that require accurate and reliable data.
Seamless Application Development: The tight integration of DB2 makes it easy for developers to work with data within their applications. The system provides tools and APIs that streamline database access.
The Evolution of DB2 on IBM i:

Over the years, DB2 on IBM i has evolved alongside the operating system. It has kept pace with industry standards and introduced new features to meet the changing needs of businesses. This includes support for SQL, advanced query processing, and integration with other database technologies.

Why is this Integration Important?

The tight integration of DB2 with the AS400/IBM i is a key factor in the system's success. It provides a stable, reliable, and high-performance platform for running business applications. This integration simplifies system management, improves data integrity, and makes application development easier.

The AS400 / IBM i Today:

While often overlooked, the IBM i remains a relevant and powerful platform for businesses. Its focus on reliability, security, and integration, combined with its modern capabilities, makes it a strong choice for organizations that need a robust and dependable system. The tight integration with DB2 continues to be a crucial advantage, providing a highly optimized environment for data-intensive applications.

In Conclusion:

The AS400/IBM i and DB2 are inextricably linked. Their close relationship is a defining characteristic of the platform and a major contributor to its long-standing success. Understanding this connection is essential for anyone working with or considering this powerful system. While the name has changed and the technology has evolved, the core principles of integration, reliability, and business focus remain, making the IBM i a compelling option for many organizations.

FAQ

Is AS400 an Operating System?

The AS400 is not just an operating system, but it's more accurate to describe it as a complete, integrated system. This system includes both hardware and software components that are designed to work together seamlessly.

Here's a breakdown to clarify:

AS400 as a Hardware Platform: Initially, AS400 referred to a specific line of mid-range computer hardware introduced by IBM in 1988. This hardware was designed with a focus on reliability and business applications.
AS400 as a Software Platform: Alongside the hardware, the AS400 also included its own operating system, initially called OS/400. This operating system was tightly integrated with the hardware and provided a unique object-based architecture.
Evolution and Renaming: Over the years, both the hardware and the operating system evolved significantly. The hardware line went through several name changes (iSeries, System i, and now IBM Power Systems). The operating system also underwent name changes (i5/OS, IBM i) to reflect its modernization.

Key takeaway: While the term "AS400" is often used to refer to the system as a whole, it's important to remember that it encompasses both the hardware and the operating system working together.

Think of it like a car:

The car itself (hardware) is like the AS400 system.
The engine (operating system) is a crucial part of the car, but it doesn't define the entire car.

In the same way, the operating system (IBM i) is a vital component of the AS400 system, but it's not the whole story. The AS400 is a combination of hardware and software designed to provide a robust and integrated platform for business computing.

Monorepos

Oussama Belhadi — Thu, 20 Feb 2025 13:07:21 +0000

Mastering Monorepos: How to Set Up Next.js & NestJS with Turborepo

Introduction

Monorepos are becoming a popular approach for managing multiple related projects in a single repository. If you're building a full-stack application with Next.js for the frontend and NestJS for the backend, using a monorepo setup can significantly improve code sharing, consistency, and development speed.

In this article, we'll walk through setting up a monorepo using Turborepo, which optimizes builds and runs tasks efficiently.

Why Use a Monorepo?

A monorepo helps manage multiple applications or services in a single repository, making it ideal for projects with shared code and multiple services.

✅ Benefits of a Monorepo

Code Sharing: Reuse components, utility functions, and TypeScript types across frontend and backend.
Single Dependency Management: All dependencies are managed in one place, reducing version conflicts.
Faster Development: Turborepo caches previous builds, making incremental changes faster.
Simplified CI/CD: One pipeline for the entire project, making deployment easier.
Better Collaboration: Developers can work across services without managing multiple repositories.

Setting Up a Monorepo with Next.js & NestJS

Let's create a monorepo structure that includes:

Frontend: Next.js
Backend: NestJS
Shared Code: Common UI components, utilities, and TypeScript types

1️⃣ Initialize the Monorepo

Turborepo helps manage monorepos efficiently by caching and running only necessary tasks.

Run the following command to create a new monorepo:

npx create-turbo@latest personal-finance-app
cd personal-finance-app

This sets up a Turborepo workspace with apps and packages folders.

2️⃣ Create Folder Structure

Organize your project like this:

/personal-finance-app
 ├── /apps
 │   ├── /frontend  (Next.js)
 │   ├── /backend   (NestJS)
 ├── /packages
 │   ├── /ui        (Shared UI components)
 │   ├── /utils     (Shared utility functions)
 │   ├── /types     (Shared TypeScript types)
 ├── package.json
 ├── turbo.json
 ├── tsconfig.json
 ├── .gitignore

3️⃣ Install Next.js for the Frontend

Navigate to the apps/frontend folder and create a Next.js app:

cd apps/frontend
npx create-next-app .

Then, add workspace support to package.json:

"workspaces": ["../../packages/*"]

4️⃣ Install NestJS for the Backend

Navigate to the apps/backend folder and set up NestJS:

cd ../backend
npx nest new .

Similarly, add workspace support to package.json:

"workspaces": ["../../packages/*"]

5️⃣ Configure the Root `package.json`

Modify the root package.json to enable workspaces and Turborepo commands:

{
  "name": "personal-finance-app",
  "private": true,
  "workspaces": ["apps/*", "packages/*"],
  "scripts": {
    "dev": "turbo run dev",
    "build": "turbo run build"
  },
  "devDependencies": {
    "turbo": "latest"
  }
}

6️⃣ Run Everything!

Install dependencies for all apps and packages:

yarn install

Then, start both frontend and backend:

yarn dev

Now, Turborepo runs Next.js and NestJS in parallel! 🎉

How Monorepos Work Under the Hood

Workspaces for Shared Code

You can create reusable packages under packages/, such as:

packages/ui → Shared React components for frontend.
packages/utils → Utility functions used in both frontend & backend.
packages/types → Shared TypeScript interfaces.

To use these packages, simply install them in apps/frontend and apps/backend:

yarn workspace frontend add ../../packages/types

Turborepo’s Smart Caching

Turborepo caches previous builds, so when you change the frontend, it won’t unnecessarily rebuild the backend.

🚀 Deploying a Monorepo

Frontend (Next.js) Deployment

Deploy easily to Vercel:

yarn build
vercel deploy

Backend (NestJS) Deployment

For a backend, Railway or Render works well:

yarn build
railway up

Conclusion

A monorepo with Next.js and NestJS using Turborepo makes full-stack development faster, more efficient, and easier to manage. 🚀

Key Takeaways:

✅ Shared code reduces duplication.

✅ Faster builds with caching.

✅ One command to run both frontend & backend.

✅ Simplified deployments.