DEV Community: Mohamed Radwan

Stop Paying for Availability You Don’t Actually Have

Mohamed Radwan — Fri, 11 Sep 2026 09:34:42 +0000

Cloud platforms distribute workloads across Availability Zones (AZs) by default, for resilience. But cross-AZ traffic, replicated storage, and unnecessary spread of stateless and stateful workloads generate ongoing charges.
For platforms that don’t need a strict multi-AZ SLA, intentionally co-locating an application’s compute and its stateful components
not just the database, but any StatefulSet: message queues, search indices, caches, custom stateful services — inside one AZ can reduce infrastructure costs by more than 10%, depending on internal traffic volume and architecture.

Where the cost comes from

Cross-AZ charges typically originate from:

Application pods talking to a StatefulSet (database, queue, cache, search index) in a different AZ
EBS/PD-backed volumes located in a different AZ from the pods that use them
Chatty internal APIs crossing AZ boundaries at high volume
Monitoring/logging traffic unnecessarily spread across zones
Replication traffic crossing AZs without a real availability requirement behind it

This traffic stays inside the region, but most providers still bill it as regional data transfer. On database- or queue-heavy platforms, this becomes a real monthly cost.

The basic optimization

Any workload where the application tier repeatedly exchanges large volumes of data with a stateful backend — SQL/NoSQL databases, Kafka/RabbitMQ, Elasticsearch/OpenSearch, Redis, custom stateful services is a candidate for this optimization.

The storage reality: your StatefulSet is already single-AZ

This is the part most cost discussions skip, and it’s the strongest technical argument for this pattern.
Block storage such as AWS EBS is zonal.
A StatefulSet pod backed by such a volume can only ever run in the AZ where its volume lives.
Kubernetes cannot move it, and a scheduling rule cannot migrate it. It can only place other pods relative to it.

That means: if you have not built real cross-AZ replication for that stateful workload (e.g., Postgres streaming replica, Kafka multi-AZ, Elasticsearch multi-zone cluster), then that workload is already exposed to a single-AZ failure domain today regardless of how many AZs your stateless application pods are spread across.

So if AZ-2 (where your only Postgres/queue/cache instance lives) goes down:

An application tier spread across AZ-1/2/3 still fails for all users, because every app pod depends on the same single-AZ backend.
Spreading the app tier bought you nothing for this workload's availability; it only paid cross-AZ transfer fees while the backend was healthy.

Co-locating the app pods with that StatefulSet in its AZ does not meaningfully change your real availability in this common case. It just removes the wasted cost of the app pods talking to it cross-AZ during normal operation. You are not trading availability for cost here; you are removing a cost that was never buying you the availability it appeared to.
If you do have genuine multi-AZ stateful replication, this argument doesn’t apply.

Going from single-AZ to multi-AZ typically moves you from ~99.5% to ~99.99%, roughly the difference between ~1.8 days/year and ~1 hour/year of potential downtime, for the redundant components.
That gap is the real price of true HA. It only applies to components that are actually replicated across AZs, not to a stateless app tier sitting in front of a single-AZ backend.

Why dynamic policy (Kyverno) beats manual nodeAffinity

nodeAffinity/nodeSelector are the underlying Kubernetes scheduling primitives, but they’re static and per-workload. Applied manually across many tenants/workloads, you’d need to:

Identify the current AZ of each StatefulSet anchor per tenant
Hardcode that AZ into every related Deployment
Keep it updated whenever the StatefulSet pod is recreated in another AZ
Avoid ever applying it to PVC-backed pods or DaemonSets That’s manual, error-prone, and drifts over time.

Kyverno centralizes it: at pod admission time it looks up the anchor’s live AZ and injects the constraint automatically, only for eligible stateless pods.

Example — Generic anchor via label

Recommended approach: label whichever StatefulSet pod should act as the AZ anchor (database, queue, cache, search node — anything), instead of hardcoding a name pattern.

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: pin-stateless-pods-to-anchor-az
spec:
  background: false          # only mutate newly-created pods; never retroactive
  rules:
    - name: pin-to-anchor-zone
      match:
        any:
          - resources:
              kinds: [Pod]
      context:
        # 1. Namespace must opt in via a generic "tenant" label (any value)
        - name: tenant
          apiCall:
            urlPath: "/api/v1/namespaces/{{request.namespace}}"
            jmesPath: 'metadata.labels."example.com/tenant-name" || ''none'''
        # 1b. Is the namespace deactivated?
        - name: deactivated
          apiCall:
            urlPath: "/api/v1/namespaces/{{request.namespace}}"
            jmesPath: 'metadata.labels."deactivated" || ''false'''
        # 2. Anchor = any running pod explicitly labelled as the AZ anchor
        #    (could be a database, queue, cache, search node, or any StatefulSet)
        - name: anchorNode
          apiCall:
            urlPath: "/api/v1/namespaces/{{request.namespace}}/pods"
            jmesPath: 'items[?metadata.labels."example.com/az-anchor"==''true''] | [?status.phase==''Running''] | [0].spec.nodeName || ''none'''
        # 3. Read that node's AZ
        - name: zone
          apiCall:
            urlPath: "/api/v1/nodes"
            jmesPath: 'items[?metadata.name==''{{anchorNode}}''] | [0].metadata.labels."topology.kubernetes.io/zone" || ''none'''
      preconditions:
        all:
          - key: "{{ tenant }}"
            operator: NotEquals
            value: "none"
          - key: "{{ deactivated }}"
            operator: NotEquals
            value: "true"
          - key: "{{ anchorNode }}"
            operator: NotEquals
            value: "none"
          - key: "{{ zone }}"
            operator: NotEquals
            value: "none"
          # skip DaemonSets (must run on every node)
          - key: "{{ request.object.metadata.ownerReferences[?kind=='DaemonSet'] | length(@) }}"
            operator: Equals
            value: 0
          # skip ANY pod that mounts a PVC — protects the anchor itself and every other StatefulSet
          - key: "{{ request.object.spec.volumes[?persistentVolumeClaim] | length(@) }}"
            operator: Equals
            value: 0
      mutate:
        patchStrategicMerge:
          spec:
            nodeSelector:
              topology.kubernetes.io/zone: "{{ zone }}"

Apply the anchor label once, on whichever StatefulSet should define the AZ for that namespace:

kubectl -n <namespace> label pod <anchor-statefulset>-0 example.com/az-anchor=true

RBAC required

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: kyverno:pin-stateless-pods-to-anchor-az
  labels:
    app.kubernetes.io/component: admission-controller
    app.kubernetes.io/instance: kyverno
    app.kubernetes.io/part-of: kyverno
rules:
  - apiGroups: [""]
    resources: ["namespaces", "pods", "nodes"]
    verbs: ["get", "list"]

Rollout strategy

Measure — cross-AZ transfer cost, which StatefulSets are heavy talkers, current pod/volume AZs.
Classify — genuinely multi-AZ-replicated stateful workloads (keep spread) vs. single-AZ-bound ones (safe to co-locate around).
Label anchors — tag the StatefulSet(s) that should define each namespace’s AZ.
Pilot — apply the policy in TEST/DEV, restart a few Deployments, confirm placement and zero Pending pods.
Validate — compare cross-AZ bytes and cost before/after; confirm latency/error rates unchanged.
Expand gradually — roll out per tenant group; explicitly exclude any workload with a real multi-AZ HA requirement.

Recommendation

Keep genuinely multi-AZ-replicated stateful workloads spread — that's a deliberate, paid-for availability trade-off (roughly 99.5% → 99.99%, i.e., ~1.8 days/year → ~1 hour/year of potential downtime).
For everything else, recognize that an unreplicated StatefulSet is already a single-AZ failure domain — co-locating its stateless clients with it doesn't reduce real-world availability, it just stops paying cross-AZ transfer fees for no benefit.
Use Kyverno, not manual nodeAffinity, to apply and maintain this at scale.
Test in DEV/TEST first, then measure the actual savings and availability impact — don't assume a fixed percentage; calculate it from your own traffic and billing data.

How "Vibe Coding" Accidentally Turned My EC2 Instance Into a Cryptominer

Mohamed Radwan — Wed, 01 Jul 2026 11:37:43 +0000

A few days ago I got an email I did not expect: an abuse report from AWS Trust & Safety, saying my EC2 instance in my personal AWS account had been caught scanning other hosts on the internet.
My first thought was the obvious one someone hacked my box.
What I actually found was more interesting (and honestly, kind of embarrassing): a single line in a package.json file was all it took.

Here's the full story of how it happened, how I tracked it down, and what actually fixed it.

The abuse report
The email included a forwarded report from a third party network. They'd logged multiple completed TCP handshakes from my instance's public IP, hitting a handful of destination IPs on ports like 9200 (commonly Elasticsearch), 443, and 80.
The "completed handshake" detail matters — it rules out IP spoofing. Both sides have to respond for a handshake to finish, so this wasn't some rando forging my IP. My instance really was reaching out and probing other hosts.
Time to figure out why.

First theory: someone broke in
Outbound scanning traffic screams "compromised box," so I started where anyone would: how did they get in?

SSH brute force?
Nope. SSH was key only, and every entry in auth.log was Accepted publickey from IPs I recognized. No password attempts, no unfamiliar sources.

Accepted publickey for ubuntu from <trusted-ip> port 57322 ssh2: RSA SHA256:...

An exposed service?

Everything else was completely unreachable from the internet.

The security group sat in front of all of it. Dead end.
So: no brute force, no exposed service. Whatever this was, it didn't come in through the front door.

Finding the malware
A quick find / -mtime -2 (files modified in the last 2 days) turned up something ugly, sitting right in my frontend app's directory:

scanner_linux
xmrig.tar.gz
scanner_deployed.log
exploited.log
failed.log
monitor.log
data.log

xmrig is a legitimate (but heavily abused) Monero mining tool. scanner_linux was a binary I'd never seen before — and ps aux confirmed it was actively running, as root, chewing through CPU:

root  285679  27.7% CPU  ./scanner_linux -t 1000

There it was. That process was almost certainly what generated the traffic AWS flagged.

Both files were owned by root, written within about a minute of each other, at a time that didn't line up with any of my own SSH sessions. Whatever dropped them had root level filesystem access but never touched SSH to get it.

Following the trail outward
If nothing got in from the outside, something on the inside must have reached out. I checked the security group's outbound rule:
Outbound: All traffic → 0.0.0.0/0

The actual root cause
Digging through the frontend app's package.json, one line jumped out immediately:

"dependencies": {
    "zod": "latest",
    "child_process": "latest",
    "aws-amplify": "latest",
    "axios": "latest"
}

child_process is not a real npm package. It's a built-in Node.js core module, no install required, ever.
There is zero legitimate reason for it to show up as an external dependency.
But somebody had published a package under that exact name on the public npm registry. This is a known attack pattern: squat on a name that looks like a trusted standard-library module, and wait for someone (a developer copy-pasting a snippet, or these days, an AI coding assistant hallucinating a dependency) to install it by mistake.

These packages almost always ship a postinstall script, which npm runs automatically the second npm install finishes — silently, with whatever permissions the install process has. In my case, that was root, inside a container with a bind-mounted host directory. Every rebuild of that container had been quietly re-running it.

The fix: remove the child_process line from package.json, wipe node_modules and the lockfile, and rebuild from scratch with docker compose up -d --build — the --build flag matters, since a plain restart just resumes the already-infected image.

What I'm taking away from this

A locked-down security group only stops external attackers. It does nothing once malicious code is already executing locally with your own privileges.
Wide-open outbound rules are underrated as a risk. 0.0.0.0/0 outbound is a common default, but it's exactly what let a tiny postinstall script fetch a cryptominer. Restricting egress would've stopped this cold even after the bad package installed.
Any dependency with the same name as a Node built-in is an instant red flag. child_process, fs, http, etc. should never appear in your dependencies.
Stop pinning things to "latest". Half my dependency list was unpinned, which makes it way harder to spot when something new and unwanted sneaks in.
npm audit and a manual skim of package.json should be routine, not just a post incident activity.

[Boost]

Mohamed Radwan — Sat, 19 Jul 2025 20:39:04 +0000

Mohamed Radwan for AWS Community Builders

Jun 21 '25

Enriching Keycloak with LinkedIn VanityName, Headline & Profile Picture via Custom SPI

#keycloak #opensource

3 min read

[Boost]

Mohamed Radwan — Sun, 13 Jul 2025 12:39:14 +0000

Mohamed Radwan for AWS Community Builders

Jul 13 '25

Extract Invoice Data Automatically Using LangChain

#ai #langchain #aws #cognito

1 min read

Extract Invoice Data Automatically Using LangChain

Mohamed Radwan — Sun, 13 Jul 2025 12:01:54 +0000

In this article, I’m sharing an app I built to automate invoice processing using image recognition and language models. The goal is simple: take scanned or photographed invoices (in JPG, PNG, or PDF format) and extract structured data in JSON.

Under the hood, the system uses OpenAI’s GPT-4o (GPT-4 Vision) model via LangChain, and it’s wrapped in a lightweight FastAPI backend built with Python. The app can batch process files, run locally or in a container, and outputs clean JSON ready for downstream systems like accounting tools or CRMs.

Tech Stack & Architecture

The app is structured with a modern, API-first architecture:

Backend: Python with FastAPI, using LangChain + GPT-4o for invoice processing

Authentication: AWS Cognito for secure, scalable user auth

Database: MongoDB for storing processed invoice data and metadata

Frontend: A Next.js app handles the UI and connects to the backend via API

Backend Code

The authentication layer with AWS Cognito makes it easy to manage user sign-ups, login access. Invoice data and any synced product info are stored in MongoDB, which works well with JSON-like structures.
On the frontend, Next.js provides a fast and reactive UI that lets users upload invoice images, view extracted data, and manage syncing.

Sync Invoice Items with Product Barcodes

Once the app extracts line items from the invoice such as product names, you can take it a step further by syncing these items with your internal product database. This allows you to:

Match items by name
Automatically assign or update barcodes
Link products to existing inventory systems
Detect mismatches or missing items

Enriching Keycloak with LinkedIn VanityName, Headline & Profile Picture via Custom SPI

Mohamed Radwan — Sat, 21 Jun 2025 10:57:36 +0000

When integrating LinkedIn as an Identity Provider in Keycloak, you may quickly discover a limitation: by default, LinkedIn only provides basic user information such as email, first name, and last name.

However, in many real-world applications, you may also want to retrieve and store additional profile details, including:

Vanity name (public LinkedIn profile ID)

High-resolution profile picture

Professional headline (user's job title or tagline)

To achieve this, you'll need to create a custom Identity Provider Mapper SPI (Service Provider Interface) in Keycloak. This extension allows you to fetch additional data from LinkedIn's API and map it into the Keycloak user model during the authentication process.

Step 1: Set Up Your LinkedIn App Permissions

Before you begin implementing the custom SPI, make sure your LinkedIn developer app has the correct permissions. You'll need to request access to the following

These scopes allow your Keycloak instance to retrieve detailed user information such as their profile picture and professional headline.
Configuring LinkedIn as an Identity Provider in Keycloak (Tested on Keycloak 26.2.5)

After you've created a LinkedIn App and obtained the required credentials, follow these steps to configure LinkedIn in your example Keycloak realm:

Navigate to your Keycloak admin console.
Go to:
Identity Providers → Select LinkedIn.
Fill in the following fields:

Client ID: (from your LinkedIn app)
Client Secret: (from your LinkedIn app)

In the Scopes field, add the following permissions:

openid profile email r_basicprofile

These scopes are necessary to:

Authenticate the user (openid)
Retrieve profile information (profile, r_basicprofile)
Fetch the user’s verified email address (email)

Enable the following options:
Store Tokens (This is required for the SPI to extract the LinkedIn access token and make additional API calls.)
Trust Email (Assumes the email provided by LinkedIn is verified.)

Click Save.

To use this LinkedIn SPI extension in your Keycloak setup:

Download the JAR file
Either build it yourself using Maven, or download a prebuilt version from the GitHub Releases page.
Copy the JAR to Keycloak's provider directory

cp linkedin-profile-mapper-1.0.0.jar /opt/keycloak/providers/

Rebuild Keycloak to recognize the new provider

/opt/keycloak/bin/kc.sh build

Restart Keycloak

    /opt/keycloak/bin/kc.sh start

Source Code & Releases
You can find the full source code and latest releases at:

https://github.com/maradwan/keycloak-linkedin-profile-mapper

Then you need to add Identity Provider Mapper

Then you need to add attributes for vanityName, profilePicture, and headline in the User profile that is in the Realm settings of your example realm

Create an example-client in Keycloak

In your Keycloak Admin Console:

Go to your realm (e.g., example), then navigate to Clients → click Create client.
Enter example-client as the Client ID, choose OpenID Connect, and click Next.
Set Root URL to http://localhost:3000, and Valid Redirect URI to http://localhost:3000/callback, then click Save.
Go to the Credentials tab → copy the Client Secret.
Under Settings, enable Standard Flow, optionally enable Direct Access Grants.
Use these details in your app:

Test the Integration (Optional FastAPI Example)

You can test the LinkedIn login flow and see the enriched tokens by using a simple FastAPI application:

Install the required packages:

pip install fastapi uvicorn requests

# app.py
from fastapi import FastAPI, Request
from fastapi.responses import RedirectResponse
import requests

app = FastAPI()

# === CONFIG ===
KEYCLOAK_BASE_URL = "https://idp.example.com"
REALM = "example"
CLIENT_ID = "example-client"
CLIENT_SECRET = "XXXXXXXXX"  # Only needed for confidential clients
REDIRECT_URI = "http://localhost:3000/callback"  # Your app's redirect URI

# === ROUTES ===

@app.get("/login")
def login():
    return RedirectResponse(
        f"{KEYCLOAK_BASE_URL}/realms/{REALM}/protocol/openid-connect/auth"
        f"?client_id={CLIENT_ID}"
        f"&redirect_uri={REDIRECT_URI}"
        f"&response_type=code"
        f"&scope=openid"
    )

@app.get("/callback")
def callback(request: Request):
    code = request.query_params.get("code")
    if not code:
        return {"error": "No authorization code provided"}

    # Exchange code with Keycloak (not LinkedIn)
    token_response = requests.post(
        f"{KEYCLOAK_BASE_URL}/realms/{REALM}/protocol/openid-connect/token",
        data={
            "grant_type": "authorization_code",
            "code": code,
            "redirect_uri": REDIRECT_URI,
            "client_id": CLIENT_ID,
            "client_secret": CLIENT_SECRET,
        },
        headers={"Content-Type": "application/x-www-form-urlencoded"},
    )

    if token_response.status_code != 200:
        return {"error": "Token exchange failed", "details": token_response.json()}

    token_data = token_response.json()

    # Decode the token to inspect (optional)
    id_token = token_data.get("id_token")
    access_token = token_data.get("access_token")
    return {
        "access_token": access_token,
        "id_token": id_token,
        "message": "Use this token to call secured APIs or decode for LinkedIn data",
    }

Run the App

uvicorn app:app --reload --port 3000

Sources:
https://learn.microsoft.com/en-us/linkedin/marketing/quick-start?view=li-lms-2025-03#step-1-apply-for-api-access

[Boost]

Mohamed Radwan — Mon, 13 Jan 2025 09:23:36 +0000

Tetiana Mostova for AWS Community Builders

Jan 12 '25

How I automated Certificate expiration alerts with AWS

#aws #security #automation #devops

5 min read

[Boost]

Mohamed Radwan — Wed, 25 Dec 2024 12:34:15 +0000

Ravindra Singh for AWS Community Builders

Sep 27 '24

Installing ArgoCD and Securing Access Using Amazon Cognito

#kubernetes #eks #devops #cognito

6 min read

Optimize AWS Cloud Costs

Mohamed Radwan — Mon, 11 Nov 2024 18:01:21 +0000

Optimize AWS cloud costs by implementing the following strategies:

VPC Optimization:

Manage Data Transfer Costs: NAT gateways incur costs; in most cases, two subnets with 99.9% availability are sufficient. For 99.99% availability, use three subnets with three NAT gateways. Two subnets are generally sufficient for most scenarios.
Remove IP v4 addresses that are not needed or associated with Network Interface Card (NIC).
Use VPC endpoints to privately connect to supported AWS services and VPC endpoint services, rather than relying on NAT gateways, to reduce costs. Additionally, AWS PrivateLink is often required for other purposes, such as compliance, irrespective of cost considerations.

EKS/EC2/Auto Scaling Group (ASG) Optimization:

Implement Instance Auto scaling: Configure autoscaling with Spot instances for worker nodes by using Karpenter to adjust resources based on demand dynamically.
Use Reserved Instances and Savings Plans: Optimize for a single node type, such as m5a.large or m5a.xlarge , for most applications to gain cost efficiencies through reserved capacity.
Rightsize Resources: Regularly reassess applications and nodes to evaluate utilization and rightsize resources for optimized performance and cost.
Use scheduled actions in the Auto Scaling Group ASG for test and development environments to automatically stop instances at specified times when compute nodes are not in use and start them when needed.
CPU Type: ARM-based CPUs (such as AWS Graviton) are more cost-effective but may have compatibility issues with certain applications. Developers should ensure application compatibility to support multiple CPU types.
Upgrades: Cloud providers may impose fees for clusters that are not upgraded to a supported version. For EKS, using outdated Kubernetes Cluster requires Extended Support, resulting in additional expenses.

EBS Optimization:

Ensure that EBS volumes are configured to use the GP3 storage type, which is 20% more cost-effective than GP2.
Delete unused EBS volumes and snapshots that are no longer in use.

S3 Optimization:

Optimize Storage Costs for Object Storage: Configure lifecycle policies, especially when using versioning, and choose the appropriate storage class (e.g., Standard or Infrequent Access). Consider S3 Intelligent Tiering for automated cost-efficient storage management.

Database Optimization:

For RDS, consider rightsizing, using Graviton-based instances, and performing mass engine version upgrades if applications support the new version.
Switching to Amazon Aurora I/O-Optimized increases instance costs by approximately 25% to include I/O operations, making it an ideal choice for high I/O workloads. This option offers improved cost predictability and can potentially reduce total costs when I/O demand is significant.
Upgrades: Cloud providers may impose fees for clusters that are not upgraded to a supported version. For RDS, using outdated database engines may require Extended Support, resulting in additional expenses.

Take IT Shipping with Serverless Technology

Mohamed Radwan — Thu, 21 Mar 2024 12:35:59 +0000

In this article, I'll take you through the journey of creating a mobile application utilizing serverless architecture.

The app idea is a platform that facilitates and supports peer-to-peer shipping services, connecting requesters and travelers.

FrontEnd Flutter Code
Backend Lambda Code

The App Architecture:

Frontend Framework (Flutter):

Flutter serves as the frontend framework for developing the mobile application.
Flutter allows for cross-platform development, enabling the app to run on both iOS and Android devices from a single codebase.

Backend Services (Lambda):

The backend service is built using Lambda, a serverless computing platform.
Lambda functions are deployed in response to events triggered by user actions, ensuring scalability and cost-efficiency.
"The Take IT" app utilizes Python Flask within the Lambda functions

Authentication (Cognito):

Cognito, a serverless authentication service, manages the sign-up/sign-in process in the Take IT app. Additionally, it seamlessly integrates with identity providers such as Google or Facebook. Authentication is typically managed using JWT (JSON Web Tokens) for secure and efficient user authentication.

API Gateway:

Integrating AWS API Gateway with Cognito offers enhanced security and scalability for the Take IT app.
API Gateway ensures only authenticated users can access backend services.

Database (DynamoDB):

DynamoDB serves as the serverless database utilized to store all data in the Take IT app. It is leveraged to manage user preferences, trip details, contacts, as well as the status of sent and received requests (including accepted, pending, and declined requests).

Adding a Trip

The function looks like this:

def add_trip(item):
    return query_table.put_item(Item=item)

Below is an example of how a trip is saved into DynamoDB in JSON format:

{
 "username": "4fb1dce7-6a50-41a8-8d7d",
 "created": "2024-10-04-19-34-18-736061",
 "acceptfrom": "2024-10-05",
 "acceptto": "2024-10-30",
 "allowed": {
  "Clothes": {
   "cost": "3.0",
   "kg": "10.0"
  },
  "Electronics": {
   "cost": "20.0",
   "kg": "5.0"
  }
 },
 "currency": "EUR",
 "fromcity": "Berlin-Germany",
 "fromto": "Berlin-Germany_Cairo-Egypt",
 "tocity": "Cairo-Egypt",
 "trdate": "2024-10-31",
 "tstamp": 1732961762
}

By utilizing the TTL (Time to Live) feature in DynamoDB, you can automatically delete records after a specified time period. For instance, in the trip record, there is an attribute called "tstamp" that determines the deletion time of the record.

Find available trips

The attributes in the trip record, such as "fromCity," "toCity," or "fromTo," are utilized for search functionality when users seek trips. I employ a global secondary index in DynamoDB to retrieve trips based on the originating city, destination city, or the combination of both.

The function looks like this:

Index: This refers to the name of the global secondary index.
Key: This represents the DynamoDB attribute used for indexing, typically referring to "fromCity" or "toCity."
City: Denotes the name of the city being referenced.
Limit: Specifies the maximum number of records to retrieve.
Today_date: This indicates the current date, used to filter and display only trips available from today onwards.
LastKey: Utilized for pagination purposes, facilitating the retrieval of subsequent sets of records beyond the initial limit.

def get_global_index(index,key,city,limit,today_date,lastkey=None):

    if lastkey:
        return query_table.query(
        IndexName=index,KeyConditionExpression=Key(
            key).eq(city),Limit=int(limit),FilterExpression=Attr('acceptto').gte(today_date),ExclusiveStartKey=json.loads(lastkey))

    return query_table.query(
        IndexName=index,KeyConditionExpression=Key(
            key).eq(city),Limit=int(limit),FilterExpression=Attr('acceptto').gte(today_date))

Users Sent/Received Requests
DynamoDB supports querying data with a key that begins with a specific value. When users send or receive requests in the Take IT app, these requests are stored in DynamoDB with statuses such as "pending," "accepted," "request," or "declined." This allows for efficient querying and retrieval of requests based on their status, enabling seamless management and tracking of request statuses within the application.

The function looks like this:

item: This refers to "pending," "accepted," "request," or "declined."

query_table.query(KeyConditionExpression=Key("username").eq(username) & Key("created").begins_with(item))

Here's an example of how a user request is saved into DynamoDB in JSON format:

{
 "username": "user2-466b-a4b9-94f90",
 "created":  "request_2022-10-15-22-35-18-213147_user1-4fb1dce7-6a50",
 "dtime": "2022-10-15-22-42-41-599518",
 "tripid": "2022-10-15-22-35-18-213147",
 "tstamp": 1669766400
},
{
 "username": "user1-4fb1dce7-6a50",
 "created": "pending_2022-10-15-22-35-18-213147_user2-466b-a4b9-94f90",
 "dtime": "2022-10-15-22-42-41-599518",
 "tripid": "2022-10-15-22-35-18-213147",
 "tstamp": 1669766400
}

Remove Account
When a user decides to delete their account in the Take IT app, several steps are initiated:

Querying User Data: All data associated with the user is queried from DynamoDB, including trip history, pending requests, and any other relevant information.

Deleting User Data: Each record associated with the user's account is deleted from DynamoDB.
This includes trip records, request records (both sent and received), and any other user-specific data stored in the database.

Removing User from Cognito: The user is removed from the Cognito user pool, deleting their account and associated authentication tokens.

The Cognito delete function looks like this:

cognito = boto3.client('cognito-idp',region_name = region_name, verify=True)
cognito.admin_delete_user(UserPoolId= userpoolid, Username= username)

The delete function looks like this:

def delete_records(username):
   items= query_table.query(
       KeyConditionExpression=Key("username").eq(username)
   )
   for i in range(len(items['Items'])):
       query_table.delete_item(
       Key={
           'username': username,
           'created' : items['Items'][i]['created']
           }
           )
   return True

Configure a privately hosted Git repository for EMR Studio

Mohamed Radwan — Thu, 12 Oct 2023 09:13:02 +0000

By default, you can access GitHub and GitLab from the studio workspace. If you are using a private Git repository, follow these steps:

Create a Studio: You should use a VPC with private subnets that have a NAT gateway to enable communication with the internet.
Ensure that the security group for the default workspace has an outbound rule allowing HTTPS traffic on port 443 to the destination 0.0.0.0/0
Create VPC Endpoint:

Make sure the endpoint uses private subnets.
Confirm that the security group attached to the endpoint allows inbound rules for HTTPS traffic on port 443 with the source set to either "Your VPC Address" or "0.0.0.0/0".
Upload the following configuration file into your Amazon S3 storage location that is used for your Studio in a folder called life-cycle-configuration:
s3://BUCKET-NAME/life-cycle-configuration/configuration.json

GitServerDnsName - The DNS name of your Git server. For example "git.example.com".

GitServerIpV4List - A list of IPv4 addresses that belong to your Git servers, the example VPC CIDR is 10.0.0.0/16, DNS is 10.0.0.2

[
    {
        "Type": "PrivatelyHostedGitConfig",
        "Value": [
            {
                "DnsServerIpV4": "10.0.0.2",
                "GitServerDnsName": "git.example.com",
                "GitServerIpV4List": [
                    "1.2.3.4"
                ]
            }
        ]
    }
]

"If you are facing issues, you may need to stop the workspace and start it again."

Fix Cert-Manager Conflict with EKS

Mohamed Radwan — Wed, 29 Mar 2023 21:45:03 +0000

I was facing issue with multiple managed worker nodes running on EKS clusters.

The issue was appearing randomly in different nodes, I cannot access the pods or get the logs by kubectl.

x509: cannot validate certificate for 10.0.83.153 because it doesn’t contain any IP SANs

Kube API in the CloudWatch showing the following errors:

E0327 08:54:17.406029 11 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"error dialing backend: x509: cannot validate certificate for 10.0.83.153 because it doesn't contain any IP SANs"}: error dialing backend: x509: cannot validate certificate for 10.0.83.153 because it doesn't contain any IP SANs

After investigating the issue with the AWS EKS support team, we found that cert-manager-webhook is causing the issue.
Kubelet certificate chain is being used from cert-manager-webhook-ca.

Run the following command on the non-working node:

openssl s_client -connect localhost:10250

CONNECTED(00000003)
---
Certificate chain
 0 s:
   i:/CN=cert-manager-webhook-ca
---
Server certificate
-----BEGIN CERTIFICATE-----
-----END CERTIFICATE-----
subject=
issuer=/CN=cert-manager-webhook-ca
---

Run the following command on the working healthy node:

openssl s_client -connect localhost:10250

CONNECTED(00000003)
---
Certificate chain
 0 s:/O=system:nodes/CN=system:node:ip-10-0-31-151.eu-west-1.compute.internal
   i:/CN=kubernetes
---
Server certificate
-----BEGIN CERTIFICATE-----
-----END CERTIFICATE-----
subject=/O=system:nodes/CN=system:node:ip-10-0-31-151.eu-west-1.compute.internal
issuer=/CN=kubernetes
---

The cert-manager-webhook deployment uses port 10250 which is also used for kubelet.

The solution is change the port of cert-manager-webhook to 10260.

By setting webhook.securePort to 10260

helm install \
  cert-manager jetstack/cert-manager \
  --namespace cert-manager \
  --create-namespace \
  --version v1.10.0 \
  --set webhook.securePort=10260

Sources:

https://cert-manager.io/docs/concepts/webhook/
https://cert-manager.io/docs/installation/compatibility/#aws-eks