DEV Community: Mohamed Sambo

When Kafka Goes Kaiju: Building Monster-Scale Streaming Architectures

Mohamed Sambo — Sat, 29 Nov 2025 19:37:43 +0000

🔥 Kafka: The Event Backbone Behind Real-Time Systems

In every modern distributed system, one subtle truth emerges:

Your system is only as real-time, resilient, and trustworthy as the event pipeline behind it.

From fintech fraud scoring to telecom usage tracking to global e-commerce personalization, Apache Kafka has become the backbone of that pipeline.

There’s a quote often attributed to Franz Kafka:

“Start with what is right rather than what is acceptable.” Franz Kafka
And ironically, that’s exactly how Apache Kafka should be architected.

Many engineers think Kafka is “just a message queue,” but the moment you try to build a real-time, end-to-end, fault-tolerant event platform, you quickly realize:

Kafka is a distributed log, a real-time streaming engine, a replication protocol, a storage layer, and an ecosystem — all in one.

This blog will walk through Kafka not as a theoretical concept but through a story-driven, practical, senior-level exploration of how real production systems use it.

🧭 What You’ll Learn (Explained for Senior Engineers)

This post covers the full lifecycle of events in Kafka:

How brokers, partitions, and replicas actually work
Why leader partitions exist and how failover affects data safety
How producers choose partitions and how Kafka guarantees ordering
How TLS vs mTLS protects event traffic
How certificates, keystores, and truststores fit together
How Kafka behaves on Kubernetes (and why it’s tricky)
When to use a Kafka operator like Strimzi
Where Kafka Streams fits into modern architectures

Everything is explained through a real e-commerce event pipeline, which mirrors how today’s largest digital platforms operate.

🏬 A Real Scenario: The E-Commerce Event Firehose

Imagine you're leading platform engineering for a major e-commerce business.
Every second, thousands of events flow through your systems:

Product views
Search queries
Add-to-cart actions
Purchases
Fraud anomalies
Inventory updates
Delivery status changes

Each downstream team wants a specific live stream of this data:

Data Engineering → analytics
ML/Recommendations → personalization
Fraud & Security → anomaly detection
Finance → auditing + reconciliation
Mobile/Web Teams → real-time experiences

And every team expects:

No downtime
No data loss
Guaranteed ordering (per user/session)
Low latency
End-to-end encryption
Horizontal scalability

A traditional message queue can’t do this.
A database absolutely can’t do this in real time.

So you choose Apache Kafka and its real value becomes visible only when you understand how it works internally.

🎬 Turning Abstractions Into Reality: The User-Interactions Stream

People often talk about Kafka in abstract terms:
“Kafka for real-time analytics.”

Let’s make it real.

Here’s the event the frontend emits on every user action:

{
  "user_id": "213884",
  "event_type": "product_view",
  "product_id": "P-4440",
  "timestamp": 1732707200,
  "device": "iOS",
  "session_id": "S-9912"
}

Examples include:

page views
product views
add-to-cart
checkout started
purchase completed
login attempts
search queries
ad clicks

All of these events flow into a single Kafka topic:

user-interactions

But raw events alone don’t create value — processing them does.

And that’s where Kafka Streams comes in.

🧠 Kafka Streams: The Real-Time Brain of the Event Pipeline

Kafka Streams is not a separate cluster or a heavy engine.
It’s a lightweight library embedded inside your microservices.

Think of it as:

“A distributed execution engine inside your services that reads from Kafka, transforms data, maintains state, and writes enriched results back to Kafka — with exactly-once semantics.”

Your e-commerce architecture typically has multiple Kafka Streams applications working together.

A. Recommendation Engine Stream Processor

##Input:

user-interactions topic

##Processing:

sessionization
affinity scoring
behavior aggregation
similar-item calculations

##Output:

processed-events

Your recommendation service subscribes to this topic to update UI suggestions in under 200 ms.

B. Real-Time Analytics / Clickstream Pipeline

##This Streams app:

counts events
aggregates metrics per minute/hour
computes funnel drop-offs
pushes results into Druid / ClickHouse / BigQuery

##Output:

aggregated-metrics

C. Fraud Detection Stream

##This service monitors patterns like:

rapid login attempts
too many purchases in a short window
mismatched session IDs
abnormal add-to-cart actions

##Output:

fraud-events

Downstream systems react instantly.

⚙️ How Kafka Distributes Partitions & Replicas Internally

A Deep Dive Into Leadership Balancing, Replica Placement, and Cluster Stability

One of Kafka’s most critical architectural responsibilities is determining where partitions live and which broker becomes their leader.
This decision directly affects latency, fault-tolerance, throughput, and cluster stability.

To understand how Kafka optimizes these properties, we analyze a realistic, production-grade scenario.

🎯 Scenario Setup

You operate a Kafka cluster with:

3 Brokers

Broker-01
Broker-02
Broker-03

2 Topics, Each topic has 3 partitions

topicA → p0, p1, p2
topicB → p0, p1, p2

Replication factor = 3

This means:
Each partition is replicated across all brokers (1 leader + 2 followers).

But the important question is:
How does Kafka decide which broker is the leader for each partition?

This is where Kafka’s internal placement strategy becomes crucial.

🎛️ Why Kafka Must Distribute Leaders Evenly

If Kafka randomly or naïvely assigned all leaders to one broker, that broker would immediately become a bottleneck.

Every leader handles:

All incoming writes from producers
All read requests from consumers (unless follower-fetching is enabled)
All coordination with followers (replication, ISR management, log divergence detection)

If one broker hosted all leaders:

Write load → concentrated on a single machine
Consumer fetch load → same single machine
Network traffic → same machine
Risk of failure → extremely high

Kafka prevents this collapse by intentionally distributing leaders evenly across brokers.
This design principle is known as:

Partition Leadership Balancing

📦 Example: How Kafka Balances topicA

Kafka tries to assign leaders in a round-robin fashion across brokers.

Topic A (3 partitions, replication factor = 3):
Partition   Leader          Followers
p0          Broker-01   Broker-02, Broker-03
p1          Broker-02   Broker-01, Broker-03
p2          Broker-03   Broker-01, Broker-02

Result:
Each broker leads exactly one partition.

This ensures leadership load is evenly distributed:

Broker-01 leads p0
Broker-02 leads p1
Broker-03 leads p2

📦 Example: How Kafka Balances topicB

Kafka repeats the same balancing logic for each topic independently.

Topic B:
Partition   Leader          Followers
p0          Broker-02   Broker-01, Broker-03
p1          Broker-01   Broker-02, Broker-03
p2          Broker-03   Broker-01, Broker-02

Again, every broker receives one leader:

Broker-01 → p1
Broker-02 → p0
Broker-03 → p2

This avoids hotspots and keeps the cluster balanced for both topics.

🔍 Why This Architecture Matters
✅ 1. High Throughput

Producers send data only to the leader of a partition.

By distributing leaders evenly:

Write traffic is spread out.
No single broker becomes saturated.
Replication remains efficient because followers are also spread across nodes.

✅ 2. High Availability

Because each partition is replicated to all brokers:

Any broker can fail.
A different replica can immediately take over as leader.
Cluster continues operating smoothly.

Kafka ensures high availability by maintaining the ISR (In-Sync Replica) list, which tracks replicas fully caught up with the leader.

✅ 3. Predictable Latency

Consumers also read from the leader (unless using follower-fetching).

Balanced leadership:
prevents consumer load concentration,
and produces predictable read latency across the system.

✅ 4. Clean, Safe Failover

When a broker fails, Kafka automatically triggers leader election.

Example:

Broker-01 fails → leaders of topicA/p0 and topicB/p1 must move.

Kafka chooses the next leader from the ISR list:

p0 → Broker-02 or Broker-03
p1 → Broker-02 or Broker-03

No data loss occurs if:

unclean.leader.election = false
acks=all
min.insync.replicas=2

This ensures Kafka promotes only replicas that are fully caught up, preventing log truncation or loss of committed messages.

🧠 Summary: How Kafka Achieves Balanced, Fault-tolerant Partition Placement

Kafka’s internal placement logic ensures:

Leaders are evenly distributed across all brokers.
Replicas are also evenly spread.
High availability is automatically maintained.
Producers and consumers experience consistent performance.
Failures trigger safe ISR-driven elections (when configured properly).

This intelligent distribution architecture is what makes Kafka capable of:

linear scalability,
fault isolation,
stable throughput, and
predictable performance in large production deployments.

How Events Reach Kafka (Producers):

Modern distributed systems rely heavily on event-driven architectures, and Apache Kafka sits at the center as the backbone of high-throughput, low-latency ingestion.

How Events Are Produced: The Producer Application

A Kafka producer is any application, service, or component that sends data into Kafka.

Producers can be built in many forms:
🔹 1.1 Native Kafka Clients (Official Libraries)

Kafka provides native client libraries in different languages; these libraries directly manage:
Connections, Batching, Retries, Compression, Partition selection

How Events Are Serialized Before Sending

Kafka accepts bytes, not objects, so the producer must serialize the event. Common choices:

🔹 2.1 JSON Serialization
Simple, human-readable, slower, larger size.

🔹 2.2 Avro Serialization
Schema-based
Requires a Schema Registry
Backward/forward compatibility

🔹 2.3 Protobuf / gRPC
Strong typed
Efficient
Great for evolving microservices

🔹 2.4 Thrift / FlatBuffers
Ultra-low latency use cases.

🔹 2.5 Custom binary formats
For extreme performance.

How Producers Connect to Kafka

A producer connects to bootstrap brokers: bootstrap.servers = broker1:9092, broker2:9092

The bootstrap server is NOT responsible for all messages — it only provides:
Cluster metadata, List of brokers, Topic→partition→leader mapping

After the initial metadata fetch, the producer does direct leader communication.

How Kafka Decides Which Partition Gets the Message

Kafka MUST assign every message to exactly one partition.

✅ 4.1 Key-based Partitioning (Most Common)
producer.send("customer_events", key="customer_123", value="event...")
Kafka hashes the key:
partition = hash(key) % number_of_partitions
➡️ Ensures ordering per key

✅ 4.2 Round-Robin Partitioning (No Key)
If key = null:
Producer sends one batch → partition 0
Next batch → partition 1 And so on
➡️ Ordering NOT guaranteed

✅ 4.3 Custom Partitioners
Used for special routing logic.

✅ 4.4 Sticky Partitioning (newer behavior)
Kafka 2.4+ introduced a "sticky" policy:
If no key is provided, producer sticks to a single partition for batching efficiency, then switches after a batch is sent.

Producer Batching & Compression

Before sending, producers batch records:
Batching improves throughput
More events per network call → less overhead

Compression types:
gzip (high CPU)
snappy (balanced)
lz4 (high-performance)
zstd (best modern choice)

Producers compress batches before sending.

TLS / mTLS Authentication (Optional but Common in Production)

For production clusters with security:
Producer must provide:
Truststore → to validate broker cert
Contains:
cluster CA certificate
intermediate CAs

Keystore → to validate producer identity (mTLS)
Contains:
client certificate signed by CA
client private key

Broker verifies:
client → broker: trusted CA
broker → client:
client certificate signed by cluster CA

Producer Sends the Message Over TCP

When the batch is ready:
Producer opens a TCP connection to the partition leader broker
Uses binary Kafka protocol
Sends a ProduceRequest
Broker responds with ProduceResponse

Broker Handles the Message

The leader:
Writes event to local log segment on disk
Appends metadata and offsets
Replicates the event to ISR (In-Sync Replicas) followers
Depending on producer durability settings…

Producer Durability Settings (acks)

acks=0 → fire and forget (might lose data)
acks=1 → leader writes only (faster, risk of leader crash)
acks=all → safest, waits for ISR replicas

Most financial systems use:
acks=all
min.insync.replicas=2

What Happens If Leader Broker Fails?

If leader down → Kafka elects new leader:
If unclean.leader.election=false → only ISR replicas can become leader (no data loss)
If true → an out-of-sync replica may become leader (possible data loss)

Producers retry automatically using:
retries
retry.backoff.ms
max.in.flight.requests

RDP to your EC2-Ubuntu

Mohamed Sambo — Wed, 26 Mar 2025 20:09:32 +0000

1- Create security group with outpund to everywhere and inbound on port 3350 and 3389 to your ip only

2- Create role of policy SsmManagedInstanceCore

3- Install ubuntu machine with public ip enabled in public subnet, without keypair
attach previous security group created at step 1 to the machine and role created in step 2 to the instance profile

4-Connect to machine using ssm session

then run the following commands

sudo su - 
apt-get update
apt install xrdp 
systemctl enable xrdp

###add-apt-repository tool that adds new software repositories to your system's APT (Advanced Package Tool) sources.
###ppa:gnome3-team/gnome3 a Personal Package Archive (unofficial Ubuntu repository). maintained by the GNOME 3 development team.

add-apt-repository ppa:gnome3-team/gnome3
apt-get install gnome-shell ubuntu-gnome-desktop
passwd ubuntu

5-then connect using remote desktop application from your local machine to public ip of the ec2 machine

RDP to your EC2-Ubuntu

Mohamed Sambo — Wed, 26 Mar 2025 20:09:32 +0000

1- Create security group with outpund to everywhere and inbound on port 3350 to your ip only

2- Create role of policy SsmManagedInstanceCore

3- Install ubuntu machine with public ip enabled in public subnet, without keypair

4-Connect to machine using ssm session

then run the following commands

sudo su -
apt-get update
apt install xrdp
systemctl enable xrdp
add-apt-repository ppa:gnome3-team/gnome3
apt-get install gnome-shell ubuntu-gnome-desktop
passwd ubuntu

5-then connect using remote desktop application from your local machine to public ip of the ec2 machine

Server Certificate Chain

Mohamed Sambo — Wed, 31 Jul 2024 22:37:00 +0000

What happens while curl https domains?

-The simplest command to use for example is curl "https://www.vodafone.com/"
-curl is making a GET request and returns the page source without any error because the server uses Trusted CA Signed SSL Certificates.
This means that the server is using a certificate that was signed by a trusted authority.

What is certificate chain?

-when you curl a domain name there are sequences happen in order to
establish a secure connection between the remote server and client,
beyond DNS resolution, TCP handshake and SSL/TLS handshake, Server sends
its SSL/TLS certificate chain to the client [including all intermediate
certificates up to (but not including) the root certificate].
-certificate chain is an ordered list of certificates and each certificate
in the chain is signed by the entity identified by the next certificate
in the chain, that enables the receiver to verify that the sender and all
CAs are trustworthy.

Certificate chain typically includes:
1-Leaf Certificate:
Issued to the domain (e.g., www.vodafone.com). This certificate is
used to encrypt the communication between the client and the server.
And known as SSL/TLS Certificate
2-Intermediate Certificate(s):
-Any certificate that sits between the SSL/TLS Certificate and the
Root Certificate is called a chain or Intermediate Certificate.
-The Intermediate Certificate is the signer/issuer of the SSL/TLS
Certificate which is the leaf one or previous intermediate one.
-The Root CA Certificate is the signer/issuer of the Intermediate
or last intermediate Certificate.
-If the Intermediate Certificate is not installed on the server (where
the SSL/TLS certificate is installed) it may prevent some browsers,
mobile devices, applications, etc. from trusting the SSL/TLS
certificate.
-In order to make the SSL/TLS certificate compatible with all clients,
it is necessary that the Intermediate Certificate be installed.
3-Root CA Certificate: Trusted Root Certificate Authority List
-The root CA (Certificate Authority) certificate is the top-most
certificate in the certificate chain. It is self-signed and must be
verified up to The Root CA Certificate, meaning the issuer and
subjects are the same.
-It serves as the ultimate trust anchor for all certificates issued
under it.
-Root CA certificates are typically distributed with operating systems
and browsers so that they are trusted by default.

Use Case: checking "https://www.vodafone.com" https://www.ssllabs.com/ssltest/analyze.html?d=vodafone.com
and you can find the leaf certificate www.vodafone.com is issued by the chain one DigiCert SHA2 Secure Server CA, and the chain certificate is issued by trusted CA DigiCert Global Root CA.

You can connect to the server and retrieve the certificate chain using openssl s_client. This command will show you the certificates sent by the server
-showcerts: Displays all certificates sent by the server

[sh@ip-10-160-78-8 ~]$ openssl s_client -connect www.vodafone.com:443 -showcerts

What may cause a problem?

-If the server has 2 intermediate certificates, and sends only one intermediate certificate (the one directly issuing the leaf certificate) but omits the second intermediate certificate (the one issuing the first intermediate certificate), the client might not be able to validate the certificate chain properly
for more issues and troubleshooting

For more info: https://www.digicert.com/blog/what-is-a-certificate-authority

Nginx Ingress Controller-Part01

Mohamed Sambo — Sat, 25 May 2024 18:37:46 +0000

one of opensource projects for Kubernetes ingress controllers, for ex: nginx-ingress-controller

What you are truly deploying for your services is ingress resources, but ingress Controller is required so that ingress resources come to life.
So please keep in mind that ingress-resource is different from ingress-controller

What is Ingress?
Ingress is an API object for routing and load balancing requests to a kubernetes service. Ingress can run on HTTP or HTTPS protocols and performs redirection by applying the rules we define as developers.

What is Ingress Controller?
Ingress Controller is a backend service developed with the Ingress API. It reads Ingress objects and takes actions to properly route incoming requests. Ingress Controllers can perform load balancing as well as forwarding operations. There are many Ingress Controllers in use
ingress-controllers

Install Ingress Controller firstly, you need to deploy nginx-controller by helm chart

helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm upgrade --install ingress-nginx ingress-nginx \ 
             --repo https://kubernetes.github.io/ingress-nginx \ 
             --namespace ingress-nginx \
             --create-namespace
             --set-string controller.service.annotations."service\.beta\.kubernetes\.io/aws-load-balancer-type"="nlb"

-- the default type of chart service type is LoadBalancer: https://github.com/kubernetes/ingress-nginx/blob/3b1908e20693c57a97b55d8a563da284a5dbf36c/charts/ingress-nginx/values.yaml#L482

-- and to define that created LoadBalancer should be NLB, it is defined as annotation at nginx-ingress-controller service:

controller:
  service:
    annotations:
      service.beta.kubernetes.io/aws-load-balancer-type: "nlb"

For example, if we have public domain "tools.com"
-- to Set SSL/TLS termination on AWS load balancer:
simply this is a way of abstracting TLS handling is to terminate on load balancer and have HTTP inside the cluster by default.
by requesting a public certificate for your custom domain "api.tools.com" and wildcard custom domain "*.api.tools.com", and don't forget to record the certificate CNAME under your public hosted zone to validate it.
then use the ACM certificate arn in controller service annotation and define the ssl port as "https"
and of course, don't forget to record all needed sub domains "api.tools.com" and "*.api.tools.com" to route to your NLP as record type A but the certificate is CNAME
Now you can set not only one ingress controller, but multiples and each ingress-controller has its own NLP and hostname for example
hostnameA: ui.api.tools.com -> to route to all ui websites
hostnameB: services.api.tools.com -> to route to restful apis services
and how the ingress resource knows which ingress-nginx-controller ?
--- each one will have unique nginx-controller-class

Another important note that ingress-controller should have the minimum permissions to allow it to create loadbalancer on aws, and this should use an IRSA role and passed as annotation to serviceAccount inside the helm chart

controller:
  service:
    annotations:
      service.beta.kubernetes.io/aws-load-balancer-ssl-cert: arn:aws:acm:eu-central-1:55xxxxxxx:certificate/5k0c5513-a947-6cc5-a506-b3yxxx
      service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "https"

-- Choosing Publicly Accessible
This will configure the AWS load balancer for public access

controller:
  service:
    annotations:
      service.beta.kubernetes.io/aws-load-balancer-scheme: "internet-facing"

-- NLB with NGINX Ingress Controller maybe overwrite client IP, how to retain actual client IP:
you need to have proxy protocol enabled on your NLB and have the appropriate configuration in ingress-nginx.

controller:
  config:
    use-proxy-protocol: "true"
    real-ip-header: "proxy_protocol"
    use-forwarded-headers: "true"

--So Finally maybe all you needs

controller:
  config:
    use-proxy-protocol: "true"
    real-ip-header: "proxy_protocol"
    use-forwarded-headers: "true"
  service:
    annotations:
      service.beta.kubernetes.io/aws-load-balancer-type: nlb
      service.beta.kubernetes.io/aws-load-balancer-ssl-cert: arn:aws:acm:eu-central-1:55xxxxxxx:certificate/5k0c5513-a947-6cc5-a506-b3yxxx
      service.beta.kubernetes.io/aws-load-balancer-ssl-ports: https
      service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing
      service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: "*"

-- what the chart deploys:
-ingress-nginx namespace
-ingress-nginx-controller-7ed7998c-j2er5 pod
-ingress-nginx-controller service of type LoadBalancer
-ingress-nginx-controller-admission service of type ClusterIP (Validating admission controller which helps in preventing outages due to wrong ingress configuration)
-EXTERNAL-IP -> in turn points to AWS Load Balancer DNS Name which gets created when the Ingress Controller is installed cause of service of type LoadBalancer created by the chart

-- In details:
The controller deploys, configures, and manages Pods that contain instances of nginx, which is a popular open-source HTTP and reverse proxy server. These Pods are exposed via the controller’s Service resource, which receives all the traffic intended for the relevant applications represented by the Ingress and backend Services resources. The controller translates Ingress and Services’ configurations, in combination with additional parameters provided to it statically, into a standard nginx configuration. It then injects the configuration into the nginx Pods, which route the traffic to the application’s Pods.
The Ingress-Nginx Controller Service is exposed for external traffic via a load balancer. That same Service can be consumed internally via the usual ingress-nginx-controller.ingress-nginx.svc.cluster.local cluster DNS name.

Create Deployment and Expose it as a service

# create deployment
kubectl create deployment demo --image=nginx --port=80 
# expose deployment as a service
kubectl expose deployment demo
# Create Ingress resource to route request to demo service
kubectl create ingress demo --class=nginx \
  --rule your-public-domain/=demo:80

Refereces:

Your First FastApi+JWT token

Mohamed Sambo — Mon, 01 Jan 2024 23:05:09 +0000

Objectives the blog will discuss

Hashing
OAuth2 with Password
SQLMODEL
Bearer with JWT tokens

GITHUB REPO: https://github.com/sambo2021/python-dev/tree/master/fastapi-auth-project

We'll proceed to installing the necessary dependencies needed in this guide. Copy the below content to requirements.txt.
pip install -r requirements.txt

Hashing:

Our database will handle users signing in for now, but you do not want to store the password as it is plaintext, converting the original plain-text passwords
adrian123

into irreversible, fixed-length hash codes
$2b$12$fNiX.PSSs4XQg0YYC5PEF.t5.aDjEvhIVYHIN5UxLXO2.9LIRHnO6

This way, even if the hashed data is somehow obtained by an attacker, they cannot reverse-engineer it to reveal the original passwords.

We implement it by installing the necessary dependencies.
pip install "passlib[bcrypt]"

Python package to handle password hashes with the recommended algorithm is “Bcrypt”.

from passlib.context import CryptContext
pwd_context = CryptContext(schemes=["bcrypt"], deprecated="auto")

def hash_password(password:str):
    return pwd_context.hash(password)
if __name__ == "__main__":
    print(hash_password("adrian123"))

output:

python3 main.py 
python3 main.py 
(trapped) error reading bcrypt version
Traceback (most recent call last):
  File "/home/sambo/.local/lib/python3.10/site-packages/passlib/handlers/bcrypt.py", line 620, in _load_backend_mixin
    version = _bcrypt.__about__.__version__
AttributeError: module 'bcrypt' has no attribute '__about__'
$2b$12$fNiX.PSSs4XQg0YYC5PEF.t5.aDjEvhIVYHIN5UxLXO2.9LIRHnO6

note:
pwd_context = CryptContext(schemes=["bcrypt"], deprecated="auto"): This creates an instance of the CryptContext class, specifying that the bcrypt hashing algorithm should be used.
The deprecated="auto" parameter means that if bcrypt becomes deprecated in the future, passlib will automatically choose a more secure scheme.

OAuth2 with Password and Bearer:

as mentioned in the docs-> https://fastapi.tiangolo.com/tutorial/security/simple-oauth2/

We are going to use FastAPI security utilities to get the username and password.
OAuth2 specifies that when using the "password flow" (that we are using) the client/user must send a username and password fields as form data.
And the spec says that the fields have to be named like that. So user-name or email wouldn't work.
But don't worry, you can show it as you wish to your final users in the frontend.
And your database models can use any other names you want.
But for the login path operation, we need to use these names to be compatible with the spec (and be able to, for example, use the integrated API documentation system).
The spec also states that the username and password must be sent as form data (so, no JSON here).

but before going into that part lets dive into

Create Database and Tables on startup:

I am using sqlmodel to connent to database, create table and do the sql operations to store users into the table and use them into other endpoints especially /token for generating jwt token

database.py

from sqlmodel import Field, SQLModel, create_engine

class User(SQLModel, table=True):
    __table_args__ = (UniqueConstraint("email"),)
    username: str = Field( primary_key=True)
    fullname: str
    email: str
    hashed_password: str
    join: datetime = Field(default=datetime.utcnow())
    disabled: bool = Field(default=False)

sqlite_file_name = "database.db"
sqlite_url = f"sqlite:///{sqlite_file_name}"
connect_args = {"check_same_thread": False}
engine = create_engine(sqlite_url, echo=True, connect_args=connect_args)
def create_db_and_tables():
    SQLModel.metadata.create_all(engine)

adrianholland = User(
        username = "adrianholland", 
        fullname = "Adrian Holland",
        email = "Adrian.Holland@gmail.com",
        hashed_password = hash_password("a1dri2an5@6holl7and"))

def initiate_admin():
    admin = get_user("adrianholland")
    if not admin:
        add_user(adrianholland)

For that, we will import SQLModel (plus other things we will also use) and create a class User that inherits from SQLModel and represents the table model for our users, and on start up of the main app we will use:

main.py

@app.on_event("startup")
def on_startup():
    create_db_and_tables()
    initiate_admin()

but of course we can not do it before adding main database functions we need, so:

database.py

from sqlmodel import Field, SQLModel, Session, create_engine,select
from passlib.context import CryptContext
from datetime import datetime
from sqlalchemy import UniqueConstraint
from fastapi import HTTPException
from email_validator import EmailNotValidError, validate_email
from disposable_email_domains import blocklist

sqlite_file_name = "database.db"
sqlite_url = f"sqlite:///{sqlite_file_name}"
connect_args = {"check_same_thread": False}
engine = create_engine(sqlite_url, echo=True, connect_args=connect_args)


def create_db_and_tables():
    SQLModel.metadata.create_all(engine)

class User(SQLModel, table=True):
    __table_args__ = (UniqueConstraint("email"),)
    username: str = Field( primary_key=True)
    fullname: str
    email: str
    hashed_password: str
    join: datetime = Field(default=datetime.utcnow())
    disabled: bool = Field(default=False)

# to hash the passowrd as we did before    
pwd_context = CryptContext(schemes=["bcrypt"], deprecated="auto")

def hash_password(password: str):
    return pwd_context.hash(password)

##https://github.com/s-azizkhan/fastapi-email-validation-server/blob/main/main.py
def validate_email_data(email: str):
    try:
        # Validate against general email rules
        v = validate_email(email, check_deliverability=True)

        # Check if the domain is in the disposable email blocklist
        domain = email.split("@")[1]
        if domain in blocklist:
            raise HTTPException(
                status_code=400,
                detail=f"Disposable email addresses are not allowed: {email}",
            )

        return True
    except EmailNotValidError as e:
        return False
    except Exception as e:
        return False

def validate_data(user: User):
    return validate_email_data(user.email) and type(user.username) == str

def get_user(username: str):
    with Session(engine) as session:
        user = session.get(User, username)
        return user

def add_user(user: User):
    exist_user = get_user(user.username)
    if not exist_user:
        with Session(engine) as session:
            session.add(user,_warn=True)
            session.commit()
    else:
        raise HTTPException(status_code=409, detail=f"user {exist_user.fullname} exists and username is {exist_user.username}")


def get_all_users():
    with Session(engine) as session:
        statement = select(User)
        users = session.exec(statement).fetchall()
        return users

adrianholland = User(
        username = "adrianholland", 
        fullname = "Adrian Holland",
        email = "Adrian.Holland@gmail.com",
        hashed_password = hash_password("a1dri2an5@6holl7and"))

def initiate_admin():
    admin = get_user("adrianholland")
    if not admin:
        add_user(adrianholland)

so now we have our main skeleton to build our app, get_user, add_user, and get_users, and of course, helper functions like hash_passowrd to store the password you entered as hashed one, and the validator function to validate mainly the email

now we get back to the main.py to add all endpoint we need

main.py

from datetime import timedelta
from fastapi import Depends, FastAPI, HTTPException, Request, status
from fastapi.security import OAuth2PasswordRequestForm
from models import Token, User, authenticate_user, create_access_token, get_current_active_user
from database import create_db_and_tables, get_user, add_user, get_all_users, hash_password, initiate_admin, validate_data
import json


app = FastAPI()

@app.on_event("startup")
def on_startup():
    create_db_and_tables()
    initiate_admin()

@app.get("/",tags=["root"])
async def read_root(current_user: User = Depends(get_current_active_user)):
    return {"message":"Welcome inside first FastApi api",
            "owner": current_user}


# when u login u redirected to /token to generate token by username and password
# u enter the username
@app.post("/token", response_model=Token)
async def login_for_access_token(form_data: OAuth2PasswordRequestForm = Depends()):
    user = authenticate_user(form_data.username, form_data.password)
    if not user:
        raise HTTPException(status_code=status.HTTP_401_UNAUTHORIZED,
                            detail="Incorrect username or password", headers={"WWW-Authenticate": "Bearer"})
    access_token_expires = timedelta(minutes=30)
    access_token = create_access_token(
        data={"username" : user.username, "email": user.email, "fullname": user.fullname }, expires_delta=access_token_expires)
    return {"access_token": access_token, "token_type": "bearer"}


@app.get("/users/me", tags=["my-user-data"])
async def read_users_me(current_user: User = Depends(get_current_active_user)):
    return {
            "owner": current_user
            }


@app.get("/users/{user_name}",tags=["get-one-user"])
async def read_item(user_name: str , request: Request , current_user: User = Depends(get_current_active_user)):
    user = get_user(user_name)
    try:
        return {"user_name": user.username,
                "user_fullname": user.fullname,
                "user_email": user.email,
                "query_params": request.query_params,
                "request_headers": request.headers,
                "owner": current_user
                }
    except:
        raise HTTPException(status_code=status.HTTP_404_NOT_FOUND,
                            detail=f"user {user_name} doesnot exist", headers={"WWW-Authenticate": "Bearer"})




@app.get("/users",tags=["get-all-users"])
async def read_item(request: Request , current_user: User = Depends(get_current_active_user)):
    return {
            "users": get_all_users(),
            "request_headers": request.headers,
            "owner": current_user
            }

@app.post("/users",tags=["add-new-user"])
async def read_item(request: Request , current_user: User = Depends(get_current_active_user)):
    request_body  = await request.body()
    json_str = request_body.decode('utf-8')
    json_data = json.loads(json_str)
    new_user=User(
        username = json_data["username"], 
        fullname = json_data["fullname"],
        email = json_data["email"],
        hashed_password = hash_password(json_data["password"])
    )
    if validate_data(new_user):
        add_user(new_user)
        return {
                "request_headers": request.headers,
                "owner": current_user
                }
    else:
        raise HTTPException(status_code=status.HTTP_404_NOT_FOUND,
                            detail=f"{new_user.username} or {new_user.email} may be not valid", headers={"WWW-Authenticate": "Bearer"})

Bearer with JWT tokens

but you still need some functionality to generate the jwt token and verify it against any login so have a look on :

models.py
and you will find everything described, but for SECRET_KEY used in encoding the token you need to generate your own local and use it
$ openssl rand -hex 32

from fastapi import Depends,HTTPException, status
from fastapi.security import OAuth2PasswordBearer
from pydantic import BaseModel
from datetime import datetime, timedelta
from jose import JWTError, jwt
from passlib.context import CryptContext # we use it for password hasher
from database import get_user,add_user,User


#openssl rand -hex 32
#we gonna use it to encode and decode the token

SECRET_KEY = "2cfea755f57d42cc34ce427475f5891aa92361bff9af79a360d1dafd296853d9"

# lets consider this as out data base already which have users and thier hashed password
# what is the actual password -> adrian123
# how u hashed it -> print(get_password_hash("adrian123"))



class Token(BaseModel):
    access_token: str
    token_type: str

class TokenData(BaseModel):
    username: str or None = None


#This parameter specifies how to handle deprecated hashing algorithms. "auto" means 
#that FastAPI will automatically handle deprecated hashing schemes and update them as needed.
#"bcrypt" is chosen. Bcrypt is a popular and secure password hashing algorithm
#This parameter specifies how to handle deprecated hashing algorithms. "auto" means that 
#FastAPI will automatically handle deprecated hashing schemes and update them as needed.
pwd_context = CryptContext(schemes=["bcrypt"], deprecated="auto")


#OAuth2PasswordBearer: This is a class provided by the fastapi.security module for handling OAuth2 password bearer authentication. 
#OAuth2 is a protocol that allows secure authorization in a standardized way.
#tokenUrl="token": This parameter specifies the URL where clients can request a token. 
#In this case, it's set to "token," meaning that when clients want to authenticate using OAuth2 password flow, 
#they should send their credentials to the "/token" endpoint.
#With this code, you've created an instance of the OAuth2PasswordBearer class named oauth2_scheme, and you can use it as a dependency in 
#your FastAPI routes. When a route depends on oauth2_scheme, FastAPI will expect clients to include an OAuth2 token in the "Authorization" 
#header of their requests. The token will be validated and can be used to identify and authenticate the user.
oauth2_scheme = OAuth2PasswordBearer(tokenUrl="token")


def authenticate_user(username: str, password: str):
    user = get_user(username)
    # if the user u entered is not exist 
    if not user:
        return False
    # if the user u entered exists but lets compare the password text u entered with the hashed one
    if not pwd_context.verify(password, user.hashed_password):
        return False

    return user


def create_access_token(data: dict, expires_delta: timedelta or None = None):
    to_encode = data.copy()
    print(f"data: {to_encode}")
    if expires_delta:
        expire = datetime.utcnow() + expires_delta
    else:
        expire = datetime.utcnow() + timedelta(minutes=15)

    to_encode.update({"exp": expire})
    print(f"data: {to_encode}")

    encoded_jwt = jwt.encode(to_encode, SECRET_KEY, algorithm="HS256")
    print(f"jwt: {encoded_jwt}")
    return encoded_jwt


async def get_current_user(token: str = Depends(oauth2_scheme)):
    credential_exception = HTTPException(status_code=status.HTTP_401_UNAUTHORIZED,
                                         detail="Could not validate credentials", headers={"WWW-Authenticate": "Bearer"})

    try:
        payload = jwt.decode(token, SECRET_KEY, algorithms=["HS256"])
        print(f"payload: {payload}")
        username = payload.get("username")
        print(f"username: {username}")
        if username is None:
            raise credential_exception

        token_data = TokenData(username=username)
        print(f"token_data: {token_data}")
    except JWTError:
        raise credential_exception

    user = get_user(username=token_data.username)
    if user is None:
        raise credential_exception

    return user


async def get_current_active_user(current_user: User = Depends(get_current_user)):
    if current_user.disabled:
        raise HTTPException(status_code=400, detail="Inactive user")

    return current_user

finally your app structure shall be something like

app
|__ database.py
|__ models.py
|__ main.py

once u run your app uvicorn main:app --port 9095 --reload
you can go through your brower directly http://localhost:9095/docs
you should have endpoints for user login and remaining endpoints

and once you go into /token endpoint and enter the username and password

you will have a valid token

you can see a lock on each endpoint because we already used
current_user: User = Depends(get_current_active_user)

so you can not use any of them if you dont have a valid token at first point

next enhancement:
1- add more endpoints delete-user, update-user, block-user
2- Using Postgres sql database
3- containerize our app and database, and make them microservices

Your first ARGO-CD

Mohamed Sambo — Mon, 25 Dec 2023 18:32:25 +0000

What are we going to do in the next steps?

We are going to set up Argo CD on a Kubernetes cluster that we initiated in the last blog 1- Your First K8S+Istio.
Also, we will make argo-cd behind a reverse proxy, so we gonna use what we installed through Istio to reach the argo-cd ui through the browser

How will we install the argo-cd at first?

We'll install it with Helm, create an application to use the app-of-apps pattern, and set Argo CD up so that it can update itself.

What is Argo CD?

Argo CD is a GitOps tool to automatically synchronizes the cluster to the desired state defined in a Git repository. Each workload is defined declaratively through a resource manifest in a YAML file. Argo CD checks if the state defined in the Git repository matches what is running on the cluster, and synchronizes it if changes were detected.

Step 01: Initialize our argo-cd Helm chart

We will use Helm to install Argo CD with the community-maintained chart from argoproj/argo-helm because The Argo project doesn't provide an official Helm chart.
We will render thier helm chart for argocd locally on our side, manipulate it and overrides its default values, and also we can helm lint the chart and templating to see if there is some errors or not, We gonna use the chart version 5.50.0 which matches appVersion: v2.8.6 you can find all details for the chart
and also we gonna override some values @ default-values.yaml

configs:
  params:
    server.insecure: true
    server.basehref: '/argocd'
    server.rootpath: '/argocd'
dex:
  enabled: false
notifications:
  enabled: false
applicationSet:
  enabled: false

We start the server with the --insecure flag to serve the Web UI over HTTP.
For this tutorial, we're using a local k8s server without a TLS setup.

also, we should override the basehref and rootpath to the subpath we gonna use to access the argo-cd UI -> http://localhost:9080/argocd/

Disable the dex component (integration with external auth providers).

Disable the notifications controller (notify users about changes to the application state).

Disable the ApplicationSet controller (automated generation of Argo CD Applications).

and BTW in the render-helm script, I deleted the part of highly available argocd deployment, so we can deploy non-HA version of Argo CD by default. If you want to run Argo CD in HA mode please have a look on README.md

just go inside helm_render.sh
and run the script, it will generate for you argo-cd

sure every time you want a higher version just look at their GitHub-repo and use the chart version you need and don't forget the appVersion also -> you can find the chart version at tags for ex: argo-cd-5.50.0, add values to Chart.yaml and helm_render.sh run the script helm_render.sh again.

to check whether the manifests in templates are good or corrupted:

~/helm-charts/charts/argocd-test/ $ helm lint ./argo-cd/ --debug

==> Linting ./argocd-test/argo-cd/
[INFO] Chart.yaml: icon is recommended

1 chart(s) linted, 0 chart(s) failed

Step 02: Installing our argo-cd Helm chart

We have to do the initial installation manually from our local machine
Later we set up Argo CD to manage itself (meaning that Argo CD will automatically detect any changes to the helm chart and synchronize it):

~/helm-charts/charts/argocd-test/ $ helm install argo-cd argo-cd/

After a minute all resources should have been deployed:

Accessing the Web UI
all you need now to add the path of argo-cd under the virtual service we did at the previous blog

the service

the virtual service

then you can go directly to UI http://localhost:9080/argocd
username-> The default username is admin
passowrd-> is auto-generated, we can get it with:

$ kubectl get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d

if something happened to istio deployment or you deployed argocd before istio then To access the Web UI we have to port-forward to the argocd-server service on port 443:

$ kubectl port-forward svc/argo-cd-argocd-server 9081:80

then you can go directly to UI http://localhost:9081/argocd
After logging in, we'll see the empty Web UI:
At this point, Argo CD applications could be added through the Web UI or CLI, but we want to manage everything in a declarative way (Infrastructure as code). This means need to write Application manifests in YAML, and commit them to our Git repo.

Step 03: manage root-app

In general, when we want to add an application to Argo CD, we need to add an Application resource in our Kubernetes cluster. The resource needs to specify where to find manifests for our application.

The root-app is a Helm chart that renders Application manifests. Initially, it has to be added manually, and after, we will commit Application manifests with Git, and it will be deployed automatically to argo-cd apps

Creating the root-app Helm chart
***note: we will add at first step the templates/root-app.yml application so don't add the templates/argo-cd.yml now-> only the templates/root-app.yml

https://github.com/sambo2021/helm-charts/blob/master/charts/root-app/Chart.yaml

apiVersion: v2
name: root-app
version: 1.0.0

and empty values.yaml -> https://github.com/sambo2021/helm-charts/blob/master/charts/root-app/values.yaml

then the root-app -> https://github.com/sambo2021/helm-charts/blob/master/charts/root-app/templates/root-app.yml

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: root-app
  finalizers:
  - resources-finalizer.argocd.argoproj.io
spec:
  project: default
  source:
    repoURL: https://github.com/sambo2021/helm-charts.git
    path: charts/root-app/
    targetRevision: master
  destination:
    server: https://kubernetes.default.svc
    namespace: default
  syncPolicy:
    automated:
      selfHeal: true

The above Application watches our root-app Helm chart (under https://github.com/sambo2021/helm-charts/blob/master/charts/root-app/templates/), and if changes are detected, synchronizes (meaning that it will render the Helm chart and apply the resulting manifests on the cluster) it.

How does Argo CD know our application is a Helm chart? It looks for a Chart.yaml file under path in the Git repository.

Argo CD will not use helm install to install charts. It will render the chart with helm template and then apply the output with kubectl.
This means we can't run helm list on a local machine to get all installed releases.

after pushing your charts to the remote repo

Now let's apply the manifest in our Kubernetes cluster. The first time we have to do it manually

~/helm-charts/charts/ $ helm template root-app/ | kubectl apply -f -

***note: api-server will understand the kind of that manifest because you already provided it by necessary crds when you deployed arg-cd

Now Argo CD manage the root-app and synchronize it automatically:

Step 04: let argo-cd manage itself

finally it is the moment of adding the argo-cd app that referring to our helm chart that we applied before, at the same level of root-app.yaml
https://github.com/sambo2021/helm-charts/blob/master/charts/root-app/templates/argo-cd.yml

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: argo-cd
  finalizers:
  - resources-finalizer.argocd.argoproj.io
spec:
  project: default
  source:
    repoURL: https://github.com/sambo2021/helm-charts.git
    path: charts/argocd-test/argo-cd/
    targetRevision: master
  destination:
    server: https://kubernetes.default.svc
    namespace: default
  syncPolicy:
    automated:
      selfHeal: true

and push it to the remote repo and let argo-cd to do the magic
We let the Argo CD controller watch for changes to the argo-cd helm chart in our repo (under https://github.com/sambo2021/helm-charts/tree/master/charts/argocd-test/argo-cd), render the Helm chart, and apply the resulting manifests. It's done using kubectl and asynchronous.

note: sometimes some apps get stuck and hanging while being deleted or resync, a small tip, is to remove the finalizer of one/multiple argo-cd applications
because if an Application or an ApplicationSet is stuck while deleting. It means it needs to wait for a response from "finalizers". So, the solution is to remove the "finalizers" from JSON

kubectl get applications  -o=jsonpath='{range .items[?(@.status.health.status=="Unknown")]}{.metadata.name}{"\n"}' | xargs -I {} kubectl patch application {}  --type=json -p='[{"op": "remove", "path": "/metadata/finalizers"}]'

Step 05: manage istio charts by argocd

we deployed istio-base, istiod and istio-ingress before by helm install, now this is the step to migrate them to our rago-cd
the same we did for argo-cd, in every component we pull the chart local by the helm_render script and using the argocd app

1- istio-base
the chart -> https://github.com/sambo2021/helm-charts/tree/master/charts/istio-base-test/
the argocd-app-> https://github.com/sambo2021/helm-charts/blob/master/charts/root-app/templates/istio-base.yml

2- istiod
the chart -> https://github.com/sambo2021/helm-charts/tree/master/charts/istio-istiod-test
the argocd-app-> https://github.com/sambo2021/helm-charts/blob/master/charts/root-app/templates/istio-istiod.yml

3- istio-ingress
the chart -> https://github.com/sambo2021/helm-charts/tree/master/charts/istio-ingress-test
the argocd-app-> https://github.com/sambo2021/helm-charts/blob/master/charts/root-app/templates/istio-ingress.yml

finnaly :

an issue appeared to me for istiod-v1.20.1, specially istiod-default-validator
but a quick fix to add ignore diff parameter to istio-base argoapp as the third link mentioned :
1-https://github.com/istio/istio/issues/46727
2-https://github.com/istio/istio/issues/45738
3-https://github.com/argoproj/argo-cd/issues/9323

Your First K8S+Istio

Mohamed Sambo — Sun, 24 Dec 2023 13:25:01 +0000

What do u need to build up k8s cluster

I am using a Linux subsystem on my Windows 10 machine, so I was searching for the best way to install a quick Kubernetes cluster for dev/test purposes let's dive in quickly
Using the k3d tool which lightweight wrapper to run k3s (Rancher Lab’s minimal Kubernetes distribution) in docker.
Requirements:
1- Install Ubuntu on WSL2 on Windows 10
what if i have virtual machine linux on my windows ?
then you need to configure the network -> https://serverfault.com/questions/225155/virtualbox-how-to-set-up-networking-so-both-host-and-guest-can-access-internet
2- docker to be able to use k3d at all
Note: k3d v5.x.x requires at least Docker v20.10.5 (runc >= v1.0.0-rc93) to work properly
3- kubectl: to interact with the Kubernetes cluster
4- helm: to use it later to install istio helm charts
5- k9s: terminal-based UI to interact with your Kubernetes clusters

wget https://github.com/derailed/k9s/releases/download/v0.32.7/k9s_linux_amd64.deb && apt install ./k9s_linux_amd64.deb && rm k9s_linux_amd64.deb

step 1: install k3d tool v5.6.0 which comes with default k8s v1.27

k3d official page

$ curl -s https://raw.githubusercontent.com/rancher/k3d/main/install.sh | bash
$ k3d --version
  k3d version v5.6.0                                                                                                                                           
  k3s version v1.27.4-k3s1 (default)

step 2: build up your cluster

$ k3d cluster create DevOps --agents 2 --api-port 0.0.0.0:6443 -p '9080:80@loadbalancer' --k3s-arg "--disable=traefik@server:*"

cluster name: DevOps
cluster master nodes: 1
cluster worker nodes: 2
api-server works on port: 6443
- note: if you are using a firewall you can allow this port through

$ sudo ufw allow 6443

disable=traefik@server:* :k3d will not deploy the Traefik ingress controller because we will use istio in a modern fashion way
9080:80@loadbalancer: means that the load balancer(in docker, which is exposed), will forward requests from port 9080 on your machine browser to 80 in the k8 cluster, you can check this out after creation by running docker ps

$ docker ps

k3d uses NGINX (stream mode) as the load balancer NGINX divides traffic using round-robin: k3d-DevOps-agent-0:80 k3d-DevOps-agent-1:80 k3d-DevOps-server-0:80

These names are Docker DNS hostnames inside the same network.

sambo@DESKTOP-FQ599KT:~$ docker exec -it abbf49 sh
/ # cat /etc/nginx/nginx.conf
###################################
# Generated by confd 2025-11-21 19:43:00.997738828 +0000 UTC m=+0.010213309 #
#             #######             #
#             # k3d #             #
#             #######             #
###################################

error_log stderr notice;

worker_processes auto;
events {
  multi_accept on;
  use epoll;
  worker_connections 1024;
}

stream {

  upstream 6443_tcp {
    server k3d-DevOps-server-0:6443 max_fails=1 fail_timeout=10s;
  }

  server {
    listen        6443;
    proxy_pass    6443_tcp;
    proxy_timeout 600;
    proxy_connect_timeout 2s;
  }

  upstream 80_tcp {
    server k3d-DevOps-agent-0:80 max_fails=1 fail_timeout=10s;
    server k3d-DevOps-agent-1:80 max_fails=1 fail_timeout=10s;
    server k3d-DevOps-server-0:80 max_fails=1 fail_timeout=10s;
  }

  server {
    listen        80;
    proxy_pass    80_tcp;
    proxy_timeout 600;
    proxy_connect_timeout 2s;
  }

}

now you can check ur cluster
$ kubectl cluster-info

By using k9s

but you need to taint your master node to prevent any pod to be scheduled on, This is the default behavior of k3s because tainting the control plane node is not a Kubernetes requirement especially for the dev/test environments.

$ kubectl taint nodes k3d-devops-server-0 node-role.kubernetes.io/master:NoSchedule

step 3: installing istio using helm

note: I insttalled almost the latest vrsion of istio Istio repository contains the necessary configurations and Istio charts for installing Istio. The first step is to add it to Helm by running the command below.

$ helm repo add istio https://istio-release.storage.googleapis.com/charts

Now, update Helm repository to get the latest charts:

$ helm repo update

Install Istio base chart, Enter the following command to install the Istio base chart, which contains cluster-wide Custom Resource Definitions (CRDs). (Note that this is a requirement for installing the Istio control plane.)

$ helm install istio-base istio/base -n istio-system --create-namespace --set defaultRevision=default
$ helm install istiod istio/istiod -n istio-system --wait
$ helm install istio-ingress istio/gateway -n istio-ingress --
create-namespace --wait

to list all helm releases deployed in namespace istio-system
$ helm ls -n istio-system

to get status of istio-system

$ helm status istio-base -n istio-system
$ helm status istiod -n istio-system
$ helm status istio-ingress -n istio-ingress

to get all deployed for relaese of istio-system

$ helm get all istio-base -n istio-system
$ helm get all istiod -n istio-system
$ helm get all istio-ingress -n istio-ingress

Label namespace to onboard Istio
For Istio to inject sidecars, we need to label a particular namespace. Once a namespace is onboarded (or labeled) for Istio to manage, Istio will automatically inject the sidecar proxy (Envoy) to any application pods deployed into that namespace.
Use the below command to label the default namespace with the Istio injection-enabled tag:
$ kubectl label namespace default istio-injection=enabled

if u gonna use argocd to deploy those charts, u need to getch those repos local on ur side for more visibility

$ helm fetch istio/base --untar
$ helm fetch istio/istiod --untar
$ helm fetch istio/gateway --untar

step 4: enable istio envoy access logs

Istio offers a few ways to enable access logs. Use of the Telemetry API is recommended
The Telemetry API can be used to enable or disable access logs:
istio envoy access log

apiVersion: telemetry.istio.io/v1alpha1
kind: Telemetry
metadata:
  name: mesh-default
  namespace: istio-system
spec:
  accessLogging:
    - providers:
      - name: envoy

The above example uses the default envoy access log provider, and we do not configure anything other than default settings.
Similar configuration can also be applied on an individual namespace, or to an individual workload, to control logging at a fine grained level.

step 5: try your first apps using istio gateway

we have in that example 2 simple apps [pod+service] deploy them in default namespace, which you enabled istio injection by default

echo app

apiVersion: v1
kind: Pod
metadata:
  name: echo-server
  labels:
    app: echo-server
spec:
  containers:
  - name: echoserver
    image: gcr.io/google_containers/echoserver:1.0
    imagePullPolicy: IfNotPresent
    ports:
    - containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
  name: echo-service
  labels:
    app: echo-server
spec:
  selector:
    app: echo-server
  ports:
  - port: 8080
    targetPort: 8080
    name: http

hello app


apiVersion: v1
kind: Service
metadata:
  name: hello-service
spec:
  ports:
  - port: 8080
    name: http
  selector:
    app: hello-app
---
apiVersion: v1
kind: Pod
metadata:
  labels:
    app: hello-app
  name: hello-app
spec:
  containers:
  - command:
    - /agnhost
    - netexec
    - --http-port=8080
    image: registry.k8s.io/e2e-test-images/agnhost:2.39
    name: agnhost
    ports:
    - containerPort: 8080

Now that the 2 services are up and running, you need to make the 2 applications accessible from outside of your Kubernetes cluster, e.g., from a browser. A gateway is used for this purpose and virtual service to match the URI requested, we gonna link our gateway to istio-ingress to see traffic going into the cluster

gateway

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: ingress-gateway
  namespace: istio-ingress
spec:
  selector:
    app: istio-ingress
    istio: ingress
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
    - "*"

virtualservice

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: virtualservice
  namespace: istio-ingress
spec:
  hosts:
  - "*"
  gateways:
  - ingress-gateway
  http:
  - match:
    - uri:
        prefix: /echo
    route:
    - destination:
        host: echo-service.default.svc.cluster.local
        port:
          number: 8080
  - match:
    - uri:
        prefix: /hello
    route:
    - destination:
        host: hello-service.default.svc.cluster.local
        port:
          number: 8080

hitting the browser
http://localhost:9080/echo

http://localhost:9080/hello

look at the taffic going into cluster through istio-ingress

and the istio side car logs on each app

Finally, this is how your cluster looks like almost not exact :D

Extra-Tip: Furthermore I need to configure specific domain and use TLS connection so We gonna need one more port mapped through the nginx-load balancer so we can make it by editing the cluster

k3d cluster edit DevOps --port-add '9443:443@loadbalancer'

Your First K8S+Istio

Mohamed Sambo — Sun, 24 Dec 2023 13:25:01 +0000

What do u need to build up k8s cluster

I am using a Linux subsystem on my Windows 10 machine, so I was searching for the best way to install a quick Kubernetes cluster for dev/test purposes let's dive in quickly
Using the k3d tool which lightweight wrapper to run k3s (Rancher Lab’s minimal Kubernetes distribution) in docker.
Requirements:
1- docker to be able to use k3d at all
Note: k3d v5.x.x requires at least Docker v20.10.5 (runc >= v1.0.0-rc93) to work properly (see #807)
2- kubectl: to interact with the Kubernetes cluster
3- helm: to use it later to install istio helm charts
4- k9s: terminal-based UI to interact with your Kubernetes clusters

Note:

step 1: install k3d tool v5.6.0 which comes with default k8s v1.27

k3d official page

$ curl -s https://raw.githubusercontent.com/rancher/k3d/main/install.sh | bash
$ k3d --version
  k3d version v5.6.0                                                                                                                                           
  k3s version v1.27.4-k3s1 (default)

step 2: build up your cluster

$ k3d cluster create DevOps --agents 2 --api-port 0.0.0.0:6443 -p '9080:80@loadbalancer --k3s-arg "--disable=traefik@server:*"

cluster name: DevOps
cluster master nodes: 1
cluster worker nodes: 2
api-server works on port: 6443
- note: if you are using a firewall you can allow this port through $ sudo ufw allow 6443
disable=traefik@server:* :k3d will not deploy the Traefik ingress controller because we will use istio in a modern fashion way
9080:80@loadbalancer: means that the load balancer(in docker, which is exposed), will forward requests from port 9080 on your machine browser to 80 in the k8 cluster, you can check this out after creation by running docker ps $ docker ps
now you can check ur cluster $ kubectl cluster-info By using k9s

step 3: installing istio using helm

note: I insttalled almost the latest vrsion of istio Istio repository contains the necessary configurations and Istio charts for installing Istio. The first step is to add it to Helm by running the command below. $ helm repo add istio https://istio-release.storage.googleapis.com/charts Now, update Helm repository to get the latest charts: $ helm repo update

$ helm install istio-base istio/base -n istio-system --create-namespace --set defaultRevision=default
$ helm install istiod istio/istiod -n istio-system --wait
$ helm install istio-ingress istio/gateway -n istio-ingress --
create-namespace --wait

to list all helm releases deployed in namespace istio-system
$ helm ls -n istio-system

to get status of istio-system

$ helm status istio-base -n istio-system
$ helm status istiod -n istio-system
$ helm status istio-ingress -n istio-ingress

to get all deployed for relaese of istio-system

$ helm get all istio-base -n istio-system
$ helm get all istiod -n istio-system
$ helm get all istio-ingress -n istio-ingress

if u gonna use argocd to deploy those charts, u need to getch those repos local on ur side for more visibility

$ helm fetch istio/base --untar
$ helm fetch istio/istiod --untar
$ helm fetch istio/gateway --untar

step 4: enable istio envoy access logs

Istio offers a few ways to enable access logs. Use of the Telemetry API is recommended
The Telemetry API can be used to enable or disable access logs:
istio envoy access log

apiVersion: telemetry.istio.io/v1alpha1
kind: Telemetry
metadata:
  name: mesh-default
  namespace: istio-system
spec:
  accessLogging:
    - providers:
      - name: envoy

step 5: try your first apps using istio gateway

we have in that example 2 simple apps [pod+service] deploy them in default namespace, which you enabled istio injection by default

echo app

apiVersion: v1
kind: Pod
metadata:
  name: echo-server
  labels:
    app: echo-server
spec:
  containers:
  - name: echoserver
    image: gcr.io/google_containers/echoserver:1.0
    imagePullPolicy: IfNotPresent
    ports:
    - containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
  name: echo-service
  labels:
    app: echo-server
spec:
  selector:
    app: echo-server
  ports:
  - port: 8080
    targetPort: 8080
    name: http

hello app


apiVersion: v1
kind: Service
metadata:
  name: hello-service
spec:
  ports:
  - port: 8080
    name: http
  selector:
    app: hello-app
---
apiVersion: v1
kind: Pod
metadata:
  labels:
    app: hello-app
  name: hello-app
spec:
  containers:
  - command:
    - /agnhost
    - netexec
    - --http-port=8080
    image: registry.k8s.io/e2e-test-images/agnhost:2.39
    name: agnhost
    ports:
    - containerPort: 8080

gateway

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: ingress-gateway
  namespace: istio-ingress
spec:
  selector:
    app: istio-ingress
    istio: ingress
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
    - "*"

virtualservice

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: virtualservice
  namespace: istio-ingress
spec:
  hosts:
  - "*"
  gateways:
  - ingress-gateway
  http:
  - match:
    - uri:
        prefix: /echo
    route:
    - destination:
        host: echo-service.default.svc.cluster.local
        port:
          number: 8080
  - match:
    - uri:
        prefix: /hello
    route:
    - destination:
        host: hello-service.default.svc.cluster.local
        port:
          number: 8080

hitting the browser
http://localhost:9080/echo

http://localhost:9080/hello

look at the taffic going into cluster through istio-ingress

and the istio side car logs on each app