DEV Community: Rakan

Python package conflicts

Rakan — Thu, 05 Sep 2024 09:15:04 +0000

When developing a Python package, users might encounter dependency conflicts if different versions of the same dependency are required. For example, if your package requires requests==2.26.0, but the user's system needs requests==2.25.1, both cannot coexist since Python doesn’t allow multiple versions of the same package to be installed simultaneously.

Approaches to Avoid Dependency Conflicts:

A. Vendor Approach:

Vendoring Dependencies: This involves including necessary dependencies directly in your package. It's useful for controlling versions but may increase package size.
Pure-Python Packages: Vendoring works well for pure Python packages without their own dependencies.
Packages with Dependencies: Vendoring becomes problematic if the vendored package has its own dependencies, leading to potential conflicts.

Issues:

Dependency Clashes: Vendoring a package with dependencies may lead to conflicts in the user's environment.
Version Control: Keeping vendored dependencies updated is crucial for security.
Size: Vendoring can increase package size.

Example:

Scenario 1: If requests had no dependencies, bundling it with your package ensures the correct version is used.
Scenario 2: Since requests relies on libraries like urllib3, including it may cause conflicts if other packages require different versions of urllib3.

Note: If you do vendoring, you need to comply with the vendoring policy. Check it here.

B. Virtual Environment Approach:

Dependency conflicts are often out of your control, especially in third-party apps, even if virtual environments are used.

Issues:

Out of Our Control: How users set up virtual environments is beyond our influence.
Third-Party Apps: They might still face conflict issues, even in virtual environments.

C. Fork Approach:

You can fork the conflicting package, rename it (e.g., mypackage-requests==2.26.0), and use the forked version in your package.

Issues:

Maintenance: Forking requires keeping the fork updated with the original package.
Child Dependencies: If the forked package has dependencies, you may need to fork and manage those as well.

Conclusion:

Each approach has its benefits and challenges, and the choice depends on your specific use case and how much control you want over the dependencies. As a rule of thumb, it’s better to resolve conflicts by maintaining the package properly, ensuring compatibility with the broader Python ecosystem.

Resources:

JWT can fit as an authentication system with a blacklist technique

Rakan — Sun, 04 Feb 2024 19:34:32 +0000

As mentioned before, JWT has some drawbacks that make it unsuitable as an authentication system. However, there are ways to work around these drawbacks and make JWT more secure.

One way to protect our system is to blacklist JWT tokens (although JWT is stateless and was not designed to be blacklisted). But as they say, tools can be used in ways they were not designed for.

Here's how it works:

First, make sure to add the issued_at (iat) claim to the payload of the JWT token when the server creates it. It's a timestamp that represents the time when the token was issued/created. For example:

{
    "name": "John Doe",
    "user_id": "123",
    "iat": 1643961600
}

Then, this is how the process will work when a user logs out or you want to block someone from accessing the system through an admin dashboard or something:

When a user logs out, we will add their user_id and minimum_issued_at to a blacklist table in the database.

INSERT INTO blacklist (user_id, minimum_issued_at) VALUES (1, NOW());

When a user tries to access the system via a JWT token, it will go through this process:
- The server will check if the JWT token is valid (not expired, not tampered with, etc.).
- If the JWT token is valid, the server will check if the user_id that is in the JWT payload exists in the blacklist table.
- If it exists, the server will check if the iat of the token is greater than minimum_issued_at in the blacklist table for that user_id.
  - NOTE: If the user has multiple records in the blacklist table, the server will check for the record with the latest minimum_issued_at value since it's the most recent one.
- If the iat of the token is greater than minimum_issued_at, the server will allow the user to access the system.
- If it's not greater, the server will deny the user access to the system (goodbye, you are blocked 😊 here is a 403 status code).

Now, the blacklist table will have a lot of records over time (which is not good for performance/efficiency). So, we need to clean it up from time to time. We can do this by creating a cron job that runs every day and deletes all records that have minimum_issued_at less than the current time minus JWT token TTL (Time To Live). This way, we can keep the blacklist table clean and small.

DELETE FROM blacklist WHERE minimum_issued_at < NOW() - INTERVAL 15 MINUTE;

Additionally, you can also do global token invalidation by changing the secret key (not recommended, but it's an option). Or in the case of the Blacklist table, you can insert user_id with the value "all" and minimum_issued_at with the current time. And let the server check for this record first before checking for the user's record. If it exists, deny access to the system.

Moreover, you can add more columns to the blacklist table to make it more flexible. For example, you can add a platform column to store the platform that the token was issued for (web, mobile, etc.). You can add a device column to store the device that the token was issued for (laptop, phone, etc.). This way, you can invalidate tokens based on the platform or device too.

Drawbacks of Blacklisting JWT:

You need to query the database for each request to check if the token is blacklisted or not (since the table will be cleaned up from time to time, it will be small and fast to query + you can add an index on user_id and minimum_issued_at columns to make it faster to query + you can use a cache like Redis to store the blacklist table in memory to make it even faster).

Conclusion:

Blacklisting JWT tokens is a good way to make JWT more secure and flexible even though it's not designed to be blacklisted. It's a good way to work around the drawbacks of JWT as an authentication system.

Resources:

How JWTs Could Be Dangerous and Its Alternatives

Is JWT really a good fit for authentication?

Rakan — Sat, 03 Feb 2024 12:30:50 +0000

UPDATE: JWT can fit as an authentication system with a blacklist technique

JWT (JSON Web Token) is a very popular way to authenticate users. It's a way to securely exchange data between client and server through a token. Here is how it works:

User sends their credentials (i.e. username and password) to the server.
The server checks if the credentials are correct.
If valid, server creates a JWT token as follows:
- To make a JWT token, server will use 3 things: Header, Payload, and Secret.
  - Header: It contains type of token and hashing algorithm used to sign a signature. json { "alg": "HS256", "typ": "JWT" }
  - Payload: It contains data that we want to send. For example, user's ID, username, email, etc. json { "id": 1, "name": "John Doe", "admin": true }
  - Secret: It's a secret key that is used to sign a signature. It's only known by server. "MySecretKeyThatNoBodyKnowsButTheServer"
  - The server will use header (base64UrlEncode), payload (base64UrlEncode), and secret in order to create a signature. signature = HMACSHA256( base64UrlEncode(header) + "." + base64UrlEncode(payload), secret )
  - The output structure of JWT token will be like this: base64UrlEncode(header) + "." + base64UrlEncode(payload) + "." + signature
The server sends JWT token to client.

And that's it. Client will send JWT token in header of each request to server.

Very important notes:

JWT token is not encrypted, it's just base64UrlEncoded. So, don't put any sensitive information in payload. Meaning, if for some reason an access token is stolen, an attacker will be able to decode it and see information in payload.Check it here.
Keep access token expiry short (e.g., 15 minutes) to limit attacker misuse if stolen.
Because access token is kept on client side, it's vulnerable to XSS attacks. Server can't set HttpOnly flag on it since client needs access to include it in requests.
Dont use an access token to generate a new access token when old one expires. If access token is stolen, an attacker will be able to generate new access tokens forever. Instead, use a refresh token to generate new access tokens.
Refresh tokens should have a long expiration time, and they are stored in HttpOnly cookies to prevent XSS attacks (only server can access them).
Protect a refresh token from CSRF attacks by using SameSite attribute in cookie. Check it here so a browser will only send a cookie to server if a request originated from same site that set cookie in first place (server).
As a security measure for refresh tokens. Create a table for refresh tokens in database and store refresh token's related information (user ID, expiration time, last used time, device, user agent, etc) to be able to revoke refresh token if it's stolen or check if refresh token was used before or not.

Big Drawbacks of JWT:

Logout doesn't really log you out:
- Once a JWT token is created, it's valid until it expires. There is no way to invalidate it. So, if a user logs out, you can delete the access token from client side, but it's still valid until it expires.
Blocking a user is not possible:
- If a user is blocked, there is no way to block them from accessing the system until their access token expires. Imagine you are a bank and one of the users's access token is stolen, you can't block it or do anything about it until it expires which is a big security risk for a bank that deals with money.
Changes by system are not reflected in real time:
- If a user's role is changed from admin to regular user, there is no way to reflect this change in real time. The user will still have access to admin features until their access token expires.
You can't know how many current users are logged-in:
- Since JWT tokens are stateless, there is no way to know how many users are logged in at any given time in the system.

My conclusion:

Use traditional session-based authentication. It's more secure and flexible than JWT.
JWT is a good fit for cases/situations where you want to issue a one-time token to be used for a specific purpose. what is JWT good for, then?

Resources:

Type Declaration Files

Rakan — Sat, 20 Jan 2024 15:30:24 +0000

Type Declaration Files are files that describe the whole API of a library that was written in Javascript. These files are the magic behind TypeScript.

Let's break it down with a simple example:
Suppose there is a JavaScript library called mathLibrary.js with the following code:

const add = (a, b) => {
    return a + b;
}

If we try to use this library in a TypeScript file, we will get an error saying that add function doesn't exist. This is because TypeScript doesn't know anything about this library (because it's written in JavaScript without any type annotations). So, we need to create a type declaration file for this library. Let's call it mathLibrary.d.ts. It will look like this:

declare function add(a: number, b: number): number;

Now, when we try to use this library in a TypeScript file, we will not get any errors. This is because TypeScript now knows about this library and its functions. It knows that add function takes 2 numbers and returns a number. So, if we try to pass a string, it will show an error.

Typescript depends on type declaration files heavily. It's what makes TypeScript so powerful. If you do npm install typescript, you will see that it comes with a lot of type declaration files ".d.ts" files in the node_modules.

Note: There is a tool called DefinitelyTyped which has a lot of type declaration files for popular JavaScript libraries. You can install them using npm install @types/<library-name>. For example, if you want to install type declaration files for Googlemaps JS library, you can do npm install @types/googlemaps after that TypeScript will automatically find the type declaration files in the node_modules/@types folder and use them to check your code for errors.

Resources:

Type Declarations

TypeScript: Type Annotations and Type Inference

Rakan — Fri, 19 Jan 2024 20:07:43 +0000

As I go through the typescript course, here are some things I have learned. In TypeScript, there are two important concepts: Type annotations and Type inference.

Type Annotations: We, as developers, use type annotations to tell TypeScript the type of value a variable will have. For example:

let apples: number = 5;

Here, we're saying that apples will always be a number. If we try to assign a string, TypeScript will show an error.

Type Inference: TypeScript tries to figure out the type of a variable by itself. For example:

let apples = 5;

When we declare and initialize a variable in one line, TypeScript guesses its type. In this case, it's a number. If we try to assign a string, TypeScript will show an error.

When to use each one?

Type Inference: Use it whenever possible. Let TypeScript do the work of figuring out types for you.

Type Annotations:

When declaring a variable and initializing it in separate lines:

let apples; // should be let apples: number;

When a function needs to clarify the type it returns:

const numberToString = (num: number): string => {
    return num.toString();
}

When a variable's type can't be inferred(guessed by TypeScript):

When Typescript can't figure out the intended type (a scenario where a variable could potentially have multiple types), we need to use type annotations. For example:

let words = ['red', 'green', 'blue'];
let foundWord = false;

for (let i = 0; i < words.length; i++) {
    if (words[i] === 'green') {
        foundWord = words[i];
    }
}

to fix this, we need to add type annotations to foundWord variable:

let words = ['red', 'green', 'blue'];
let foundWord: boolean | string = false;

Resources:

NestJS: The Complete Developer's Guide

Learning TypeScript and NestJS

Rakan — Fri, 19 Jan 2024 20:06:07 +0000

In upcoming days I will shift my focus to learning NestJS and TypeScript for various reasons.

Javascript is a very popular language used by many small, big and huge companies (everywhere).
Has a huge community and a lot of resources.
Can be used for many things (Frontend, Backend, Mobile, Desktop, etc).
In performance wise, its very fast and efficient language (very good for high traffic applications, example: Twitter/X).
Its a must have skill/tool for any developer that wants to find a work in the tech industry.

So, I have enrolled in a course on Udemy to learn TypeScript and NestJS. I am on the first section of the course and here what I have learned so far:

Why does TypeScript exist?

Typescript is basically a superset of Javascript. It adds types to Javascript. It's like a layer on top of Javsacript. Typescript = Javascript + Type System.
Helps you catch errors before you run your code.
Helps you write better/Maintainable code.
Doesnt run in the browser or nodejs. It needs to be compiled to Javascript first.
It doesnt provide any performance benefits.
Its like a friend sitting next to you and telling you "Hey, you made a mistake here, you should fix it before you run your code".

What is a type?

A type is something that describes the different properties and functions that a value has. Lets take exmple of a string:

let color = 'red';

The type of color is string. It has properties like length and functions like toUpperCase().

Another example:

let count = 5;

The type of count is number. It has properties like toFixed() and functions like toString().

And there are 2 categories of types:

Primitive Types: (number, string, boolean, symbol, void, undefined, null)
They directly store/hold the value.
If you have variable a storing number 5, its like saying you have a number 5 in your hand.
Object Types: (functions, arrays, classes, objects).
They store a reference to the value.
They don't hold the actual data, but point to where the data is stored.
If you have a variable a storing an an object called Person, its like saying you have a piece of paper in your hand that has the address where the details about that Person are stored, not the Person itself.

Resources:

NestJS: The Complete Developer's Guide

Redis Scaling with RAM

Rakan — Sat, 13 Jan 2024 21:59:33 +0000

Ok, So I have been using Redis for a long time in many applications, for storing cache, sessions, and queues. And a question came up that I need to an answer for: Can we scale Redis? and if yes, how?

So, I did some research and here is what I found:

Redis is in-memory data structure store, which means it stores data in RAM.
The amount of data that can be stored in Redis is limited by the amount of RAM available. The more RAM you have, the more data you can store.
When Redis uses all the available RAM, it supports swapping to disk. It can help to store more data and avoid crashing, but it will cause a huge performance drop.
Based on a real-world test done by Redis team. Redis can handle up to 2^32 keys (4294967296 keys). And Each key can hold a value with its own limits based on the chosen data structure. For example, a string can hold up to 512MB, a list can hold up to 2^32 elements. So, if you store 1 string in each key, you can store up to 4294967296 * 512MB = 2048000000000 MB which is roughly 2 TB of data. That's a lot of data (if you have that much RAM that is 😄).
To estimate the storage capacity of Redis, you can use this formula: Redis Storage Capacity = Total Available RAM − (Redis Metadata Overhead + OS RAM Usage) where Total Available RAM is the total RAM available in the server, Redis Metadata Overhead Redis uses some amount of RAM for its internal metadata to manage keys, values, and other data structures efficiently. OS RAM Usage is the amount of RAM used by the operating system and other processes running on the server.

So, how can we scale Redis?

Add more RAM to the server. 😄
Its all about the RAM. The more RAM you have, the more data you can store. Its the bottleneck of Redis.
When your redis server reach 2^32 keys, you can use Redis Cluster to scale it horizontally by adding more nodes to the cluster. And the load balancer will distribute the requests between the nodes.
If your app is using Redis for sessions, you can use sticky sessions(in the load balancer) to make sure that all the subsequent requests from the same user will be routed to the same Redis node where the user's session is stored, otherwise the user will be logged out because the session is not found when the request is routed to a different Redis node by the load balancer.

Resources:

Notes to Self

Rakan — Thu, 11 Jan 2024 19:08:38 +0000

These are some thoughts I have learned/read somewhere that I would like to always make them stick in my mind and I want to write them down here so I can always come back to them and remind myself of them.

ITS NOT YOU!. No matter your job or how high up you are, it's not because you're smarter, better, stronger, or hard worker than others. It's not about your connections. It's because God gave you this, and you should be always grateful for it. Don't let it get to your head, and be always humble.
YOU ONLY HAVE CONTROL OVER YOUR ACTIONS!. You can't control your life, your thoughts, your feelings, other people, or anything else. You have 0 control over everything except your actions. Focus on what you can control.
EVERY ONE IS REPLACEABLE!. No matter how good you are at what you do. Here is the bitter truth: You are replaceable and everyone is.

And always remmber, things can be seen from different perspectives/angles. 😄

Data Lake Vs Data Warehouse Vs Data Mart

Rakan — Tue, 09 Jan 2024 21:15:10 +0000

Well, data is everywhere every second of every day.As a backend developer, I used to dodge terms related to data engineering. However, due to a recent project, I've started learning more about it.

So, I came across these terms: Data Lake, Data Warehouse, and Data Mart. I will break them down into simple terms that I can understand.

The format will be as follows:
- Definition: (Definition)
- Characteristics: (Characteristics)
- Why it exists: (Why it exists)
- Tools: (Tools that can be used to implement it)

Data Lake:
- Definition: A huge storage space for all raw data (For example: JSON, Videos, Database dumps, etc) where everything is dumped without organization.
- Characteristics:
  - Stores raw data without modification.
  - Store structured, semi-structured, and unstructured data.
  - Can be Used for the entire data lifecycle.
- Why it exists: Data is valuable nowdays and it can be used for many things. So, store it and you can use it later when you need it.
- Tools:
  - Free: Hadoop Distributed File System (HDFS)
  - Paid: Amazon S3, Azure Data Lake Storage, Google Cloud Storage
Data Warehouse:
- Definition: An organized storage place where data is structured and cleaned.
- Characteristics:
  - Stores data in a structured way.
  - Requires transformed and cleaned data.
  - Time-variant data, meaning any existing data will be archived after perid of time (Example: 1 year) and stored in the Data Lake.
- Why it exists: Since data is stored in a structured way, it can be used for reporting and analysis.
- Tools:
  - Free: PostgreSQL, MySQL, MariaDB (limitions: not scalable for HUGE data and not optimized for analytics purposes)
  - Paid: Amazon Redshift, Google BigQuery
Data Mart:
- Definition: A subset of a Data Warehouse, with a focus on specific topics.
- Characteristics:
  - Users don't need advanced technical knowledge.
  - Subset of a Data Warehouse, smaller and topic-focused.
  - Users have read-only access to specific information.
- Why it exists: Provides users a quick and easy access to data for specific topics.
- Tools:
  - Free: Microsoft Power BI (limited features)
  - Paid: Microsoft Power BI, Tableau, QlikView, Looker

Resources:

Data Lake vs Data Warehouse vs Data Mart

How Docker containers works?

Rakan — Tue, 09 Jan 2024 16:42:28 +0000

Have you ever wondered how Docker containers work?

First, lets make these things clear:

VMs have their own kernel, making them heavier and slower compared to Docker containers.
Docker containers share the host machine's kernel, making them lightweight and faster than VMs.
There are different ways to create containers, but Docker is the most popular one (example: Podman, LXC, LXD, etc).
A container must follow the OCI (Open Container Initiative) standards to be considered a container.

Now, lets see how (Docker Linux containers) works.

They are normal Linux processes that run in an isolated environment. And they rely on the following Linux features:

Namespaces: This keeps each container separate from the others. It isolates things like processes, network, user IDs, and file systems, but they still share the same basic system (kernel).
Control groups (cgroups): These control and limit how much of the computer's resources (like CPU, memory, disk, and network) each container can use. It helps in managing and isolating resource usage for a group of processes.

So, now what happens when you execute a command like docker run -it ubuntu bash?

Docker will create a new container process.
a new namespace will be created for the container.
a new cgroup will be created for the container.
a new root filesystem will be created for the container.
a new network interface will be created for the container.

Resources:

What Is a Standard Container

What is a hypervisor?

Rakan — Mon, 08 Jan 2024 18:24:42 +0000

What is a hypervisor? A hypervisor is a software that creates and runs virtual machines (VMs). It allows you to run multiple operating systems on a single physical machine and share the underlying hardware resources. It will allocate the resources (CPU, RAM, Storage, etc) to each VM.

There are two types:

Type 1: Bare Metal Hypervisor (Native Hypervisor)
- What? It directly communicates with hardware, making it efficient and secure without a middleman OS.It can allocate more resources to virtual machines than your server actually has. For instance, with 128GB of RAM and eight VMs, you can assign 24GB to each, totaling 192GB. However, VMs use only the necessary RAM for their tasks, not the full allocated amount.
- Who? Used by enterprises and data centers (e.g., Amazon, Google) for running multiple OS on one machine.
- When to Use: Large data centers or server farms for maximum performance and direct hardware access.
- Example: VMware ESXi, Microsoft Hyper-V.
Type 2: Hosted Hypervisor
- What? Operates within the regular OS, less powerful but user-friendly. Type 1 hypervisors allocate resources dynamically based on VM needs, making it easier. Type 2 hypervisors use fixed allocations, so if a user assigns 8GB to a VM, it takes that amount even if not fully used. This can lead to resource issues on the host machine.
- Who? Used by developers and individuals for running multiple OS on one machine.
- When to Use: Simple setups or software development when top performance is not critical.
- Example: VMware Workstation, Oracle VM VirtualBox.

Here is a diagram that explains it better than words. 😄

Resources:

What does Kubernetes mean?

Rakan — Sat, 06 Jan 2024 18:31:43 +0000

I recently found myself lacking knowledge about Kubernetes. I have read some posts about it, and I know it's like a management tool. It's somewhat similar to Composer and NPM for managing packages, but Kubernetes is for managing Docker containers. I didn't dive into it in the past because it was not relevant to me, and to be completely honest, I was running away from it (maybe scared). Now it's time to face it. 😬

As they say, knowing the name of the enemy is the first step towards winning! Today, I want to shed light on its name. Kubernetes!? For some reason, I didn't like it. It's a long word, and I always find myself struggling to pronounce it. 🤔

After researching, I found that Craig McLuckie, one of the founders of the Google-based Kubernetes project, said:

We had 13 other names we couldn’t get past Google’s legal department. It was the last day, and I had to pick something. I was driving into work, and I thought, ‘Well, [the technology] is like driving a container ship. What would the helmsman be called?’ So I tried to find something exotic. I had no idea what the Greek for that was. I had to look it up.

Basically, the word "κυβερνήτης" (Check How to pronounce κυβερνήτης) is a Greek term that translates to "helmsman" or "steersman" in English.

To make sure, I opened Google Translator and typed "helmsman" then translated it into Greek, which gave me "πηδαλιούχος" pronounced as "pidalioúchos." That's incorrect; it's not even close to Kubernetes's word sound. I dug deeper and found the Liddell & Scott Greek-to-English lexicon, published in 1909, which had these meanings: "steersman, helmsman, guide, governor" for "κυβερνήτης." on page 397, which I was looking for.

Now, what does "helmsman" mean?

A helmsman or helm (sometimes driver or steersman) is a person who steers a ship, sailboat, submarine, or any other type of maritime vessel or spacecraft.

Another question: Why do people mention it sometimes as K8s?

"K8s" is shorthand for Kubernetes because there are 8 letters between "K" and "s" in the word "Kubernetes" making it shorter and easier to type without spelling mistakes.

Now that I know the origin of "Kubernetes" I can sleep in peace. 😴

Resources: