DEV Community: jairajsahgal

Understanding Eventual Consistency

jairajsahgal — Mon, 01 Jan 2024 19:42:31 +0000

Introduction

In the world of distributed systems, data management and synchronization are critical aspects that ensure the smooth operation of the system. One of the models that has gained popularity in this domain is Eventual Consistency.

What is Eventual Consistency?

Eventual consistency is a model that ensures that data will eventually become consistent across all nodes or replicas in the system. It allows for temporary states of inconsistency during the propagation of updates.

When a change is made to a piece of data, that change is first applied locally. The new version is then propagated to other nodes or replicas in the system over time. However, there could be a period where different nodes or replicas may have different versions of the same data due to network latency or concurrent updates.

The key concept here is that, as time progresses and all updates are eventually propagated, the system will become consistent across all nodes. This model is also known as “Brewer’s Conjecture” or the CAP theorem. It’s a trade-off between consistency (always reflecting the latest data) and availability (ensuring that data is always accessible), making it suitable for applications where eventual consistency works well.

Is Eventual Consistency limited to NoSQL databases?

Eventual consistency is not limited to NoSQL databases. The concept can be applied in various types of distributed systems, including both SQL and NoSQL databases, as well as other distributed applications.

The choice between eventual consistency and strong consistency depends on the specific requirements of an application. Eventual consistency is often chosen when performance is a priority, the system is designed to tolerate temporary inconsistencies, or the application allows eventual consistency because it has built-in mechanisms to handle data conflicts.

On the other hand, strong consistency is preferred when data accuracy and real-time consistency are essential, the application doesn’t tolerate temporary inconsistencies or data conflicts, or the system is designed to prioritize data integrity over other considerations.

Example

Let's consider a simple example of an eventual consistent system using a NoSQL database, like MongoDB. We have a collection of documents representing user profiles. Each document stores the user's
name, email address, and profile picture:

{
    "_id": "user1",
    "name": "John Doe",
    "email": "john@example.com",
    "profilePicture": "https://example.com/john.jpg"
}
{
    "_id": "user2",
    "name": "Jane Smith",
    "email": "jane@example.com",
    "profilePicture": "https://example.com/jane.jpg"
}

Suppose that we have two nodes (let's call them Node A and Node B) in our MongoDB cluster, each storing a replica of this collection. When an update is made on one node, it might not be immediately propagated
to the other node due to network latency or other factors.

Let's say that we receive an update to change Jane Smith's email address from "jane @ example.com" to "new_email @ example.com". This update is first applied on Node A, but not yet propagated to Node B. If we
were to query the data from both nodes at this moment, we would see different versions of Jane Smith's profile:

Node A:

{
    "_id": "user2",
    "name": "Jane Smith",
    "email": "new_email@example.com",
    "profilePicture": "https://example.com/jane.jpg"
}

Node B:

{
    "_id": "user2",
    "name": "Jane Smith",
    "email": "jane@example.com",
    "profilePicture": "https://example.com/jane.jpg"
}

In this situation, the system is in a state of temporary inconsistency. If we perform the same query on both Node A and Node B after some time has passed, the data will eventually be consistent:

Node A (eventually):

{
    "_id": "user2",
    "name": "Jane Smith",
    "email": "new_email@example.com",
    "profilePicture": "https://example.com/jane.jpg"
}

Node B (eventually):

{
    "_id": "user2",
    "name": "Jane Smith",
    "email": "new_email@example.com",
    "profilePicture": "https://example.com/jane.jpg"
}

Conclusion

Eventual consistency is a powerful model for managing data in distributed systems. It provides a balance between data consistency and system availability, making it a popular choice for many large-scale applications. However, it’s important to understand the trade-offs involved and choose the right consistency model based on the specific requirements of your application.

Wikipedia Article

𝗗𝗶𝗿𝘁𝘆 𝗿𝗲𝗮𝗱, 𝗡𝗼𝗻-𝗿𝗲𝗽𝗲𝗮𝘁𝗮𝗯𝗹𝗲 𝗿𝗲𝗮𝗱, 𝗮𝗻𝗱 𝗣𝗵𝗮𝗻𝘁𝗼𝗺 𝗿𝗲𝗮𝗱 𝘄𝗶𝘁𝗵 Real 𝗹𝗶𝗳𝗲 𝗲𝘅𝗮𝗺𝗽𝗹𝗲𝘀

jairajsahgal — Sat, 30 Dec 2023 20:10:44 +0000

In the field of database systems, concurrency control is a crucial aspect to ensure that multiple transactions can execute simultaneously without causing inconsistencies or data corruption. To manage this, databases use various isolation levels and techniques like locking mechanisms to protect data integrity. However, these mechanisms may introduce certain issues known as Dirty Read, Non-repeatable Read, and Phantom Read.

𝟭. 𝗗𝗶𝗿𝘁𝘆 𝗥𝗲𝗮𝗱:

A dirty read occurs when a transaction reads uncommitted data from the database. In other words, it reads data that has been modified by another
transaction but not yet committed. This can lead to incorrect results being displayed to the user. To prevent this issue, databases generally use the Two-Phase Locking (2PL) protocol, which ensures that a transaction holds locks only for the duration of the transaction.

# Example using SQL and Python's psycopg2 library:
import psycopg2
conn = psycopg2.connect(database="your_db", user="your_user", password="your_password")
cursor = conn.cursor()

def transaction1():
    cursor.execute("UPDATE table SET column=column+1 WHERE condition")
    conn.commit()  # commits the changes

def transaction2():
    cursor.execute("SELECT * FROM table")  # reads uncommitted data
    print(cursor.fetchall())

𝟮. 𝗡𝗼𝗻-𝗿𝗲𝗽𝗲𝗮𝘁𝗮𝗯𝗹𝗲 𝗥𝗲𝗮𝗱:

A non-repeatable read occurs when a transaction re-reads data that has been modified by another concurrent transaction. This issue arises because databases do not guarantee that the data will remain unchanged between two reads of the same transaction, even if no other transactions are modifying it. To address this, 2PL is also used to prevent non-repeatable reads.

# Example using SQL and Python's psycopg2 library:
import psycopg2
conn = psycopg2.connect(database="your_db", user="your_user", password="your_password")
cursor = conn.cursor()

def transaction1():
    cursor.execute("SELECT * FROM table WHERE condition")  # reads data
    rows = cursor.fetchall()
    cursor.execute("UPDATE table SET column=column+1 WHERE condition")  # modifies data
    conn.commit()  # commits the changes
    cursor.execute(rows)  # reads data again, but results can be different due to non-repeatable read

𝟯. 𝗣𝗵𝗮𝗻𝘁𝗼𝗺 𝗥𝗲𝗮𝗱:

A phantom read occurs when a transaction retrieves multiple rows from a query based on certain conditions, but later, when it re-executes the
same query with the same conditions, additional rows appear that were inserted by other concurrent transactions in the meantime. To address this
issue, databases use multiversion concurrency control (MVCC) and version store, which allow multiple versions of data to coexist until a commit
or rollback is performed.

# Example using SQL and Python's psycopg2 library:
import psycopg2
conn = psycopg2.connect(database="your_db", user="your_user", password="your_password")
cursor = conn.cursor()

def transaction1():
    cursor.execute("SELECT * FROM table WHERE condition")  # reads data
    rows = cursor.fetchall()
    cursor.execute("INSERT INTO table (column) VALUES ('some value')")  # inserts new row
    conn.commit()  # commits the changes
    cursor.execute("SELECT * FROM table WHERE condition")  # re-reads data, but results can be different due to phantom read

To avoid these issues, databases use various techniques like 2PL, MVCC, and version store, depending on the isolation level chosen for the transaction. By ensuring data integrity, these mechanisms prevent dirty reads, non-repeatable reads, and phantom reads, allowing transactions to run concurrently without causing inconsistencies or data corruption.

𝗥𝗲𝗮𝗹 𝗟𝗶𝗳𝗲 𝗘𝘅𝗮𝗺𝗽𝗹𝗲

Imagine you are withdrawing money from your bank account. You have two options for checking the balance: Option A and Option B.

Option A: Dirty Read, Non-repeatable Read

You check the balance of your account, showing $10,000.
Another transaction transfers $5,000 from your account to another account without committing yet (dirty read).
You again check the balance of your account and see $5,000.
The transfer transaction gets rolled back due to an error.
If you try to re-check the balance using Option A, it will show $10,000, but it should ideally be $5,000 (non-repeatable read).

Option B: Phantom Read

You check all transactions in your account from last month, showing a total of $10,000.
Another transaction makes a deposit of $2,000 into your account without committing yet.
You try to re-check the transactions using Option B, but it will now show an additional $2,000 deposit (phantom read).
If you try to re-execute the query, it will show both the original transactions and the new $2,000 deposit (phantom read persists).

In this example, Option A demonstrates dirty reads and non-repeatable reads, while Option B illustrates a phantom read. Databases employ
isolation levels, locking mechanisms, or multiversion concurrency control to prevent these issues and maintain data integrity during concurrent
transactions.

Chapter on Concurrency Controls : https://www.cs.uct.ac.za/mit_notes/database/pdfs/chp13.pdf

How to Increase File Upload Size in Django Backend

jairajsahgal — Wed, 27 Dec 2023 22:32:14 +0000

Handling file uploads in Django is usually a breeze with the convenient request.data or request.FILES. However, there are situations where you might find yourself needing to up the ante on the maximum allowed file upload size for your Django backend server. Fear not! This guide is here to walk you through the process by tweaking the right knobs in Django's configuration.

Understanding Memory Size Settings

In the Django universe, two unsung heroes govern the realm of file uploads: DATA_UPLOAD_MAX_MEMORY_SIZE and FILE_UPLOAD_MAX_MEMORY_SIZE. These settings are like the unsung heroes that ensure optimal memory usage during the file upload dance. Getting to know them is key to preventing hiccups and ensuring silky-smooth file uploads on your Django backend server.

DATA_UPLOAD_MAX_MEMORY_SIZE

This setting in Django dictates the maximum size in bytes that a request body can be before sounding the alarms with a SuspiciousOperation. Defaulting to 2.5 MB (2621440 bytes), this setting keeps a watchful eye on incoming data.

FILE_UPLOAD_MAX_MEMORY_SIZE

This setting is the maestro that controls the maximum size (in bytes) a file can be before taking it off the memory stage and onto the file system. Imagine a user uploading a file through a Django form—the file starts off in memory. But, if it decides to be a bit too big for its digital britches and crosses the set limit, off it goes to temporary storage (a.k.a. the disk) during the upload performance.

Fine-Tuning the Symphony: DATA_UPLOAD_MAX_MEMORY_SIZE and FILE_UPLOAD_MAX_MEMORY_SIZE

Let's paint a picture to illustrate this harmony. Suppose DATA_UPLOAD_MAX_MEMORY_SIZE is set to 20M, and FILE_UPLOAD_MAX_MEMORY_SIZE is strutting its stuff at 15M. If someone tries to upload a file larger than 25 MB, brace yourself—error incoming. To dodge this digital bullet, simply crank up DATA_UPLOAD_MAX_MEMORY_SIZE to a safer 30M. Voila! No more errors.

# settings.py

DATA_UPLOAD_MAX_MEMORY_SIZE = 30 * 1024 * 1024  # 30 MB
FILE_UPLOAD_MAX_MEMORY_SIZE = 15 * 1024 * 1024  # 15 MB

Now, when users upload files less than 15 MB, the data gracefully pirouettes in memory. However, if it decides to be a heavyweight contender and crosses the 15 MB threshold, it elegantly glides to the disk, ensuring a perfect balance between memory usage and efficient file handling.

With these settings in your Django symphony, you can customize the upload experience, making it error-free and seamless for users interacting with your Django backend server. Happy uploading!

Core Arguments of Django Rest Framework Serializer

jairajsahgal — Sun, 17 Dec 2023 19:00:45 +0000

𝗖𝗼𝗿𝗲 𝗔𝗿𝗴𝘂𝗺𝗲𝗻𝘁𝘀 𝗼𝗳 𝗦𝗲𝗿𝗶𝗮𝗹𝗶𝘇𝗲𝗿𝘀 𝗶𝗻 𝗗𝗷𝗮𝗻𝗴𝗼 𝗥𝗲𝘀𝘁 𝗙𝗿𝗮𝗺𝗲𝘄𝗼𝗿𝗸

In Django Rest Framework (DRF), serializers are responsible for converting complex data types such as querysets and model instances to Python data types that can then be easily rendered into JSON, XML, or other content types. They also provide deserialization, allowing parsed data to be converted back into complex types after validating incoming data. To achieve this, serializers use various field classes, each with a set of core arguments that control their behavior.

Below are core arguments and their purpose:

𝗿𝗲𝗮𝗱_𝗼𝗻𝗹𝘆
This argument determines whether a field is used only for serialization and not for deserialization. When set to True, the field will be included in the output representation but ignored for creating or updating an instance. By default, it is set to False.

𝘄𝗿𝗶𝘁𝗲_𝗼𝗻𝗹𝘆
If set to True, the field can be used for creating or updating an instance but won't be included in the serialized output. This is useful for fields that should not be exposed, like passwords. Its default value is False.

𝗿𝗲𝗾𝘂𝗶𝗿𝗲𝗱
This argument specifies whether a field must be present in the input data during deserialization. If set to False, the field is not mandatory. Additionally, the attribute or key for this field may be omitted in the serialized output if absent. By default, it is True, but if using ModelSerializer, it can default to False based on the model's field configuration (e.g., blank=True).

𝗱𝗲𝗳𝗮𝘂𝗹𝘁
This provides a default value for the field if no input is given. It can be a static value, a callable, or a function that is evaluated each time the default is accessed. If a default is provided, the field automatically becomes non-required. Note that for partial updates, the default is not used, and when serializing, it's used only if the object attribute or dictionary key is missing.

𝗮𝗹𝗹𝗼𝘄_𝗻𝘂𝗹𝗹
By setting this argument to True, you allow the field to accept None as a valid value. This doesn't imply a default value for deserialization. The default for allow_null is False.

𝘀𝗼𝘂𝗿𝗰𝗲
Specifies the attribute used to populate the field. You can use a method (like source='get_absolute_url') or a dotted path to navigate through attributes (source='user . email'). If using dotted notation, it is often prudent to specify a default value to avoid attribute errors during serialization. Special value source='*' means that the entire object should be passed to the field.

𝘃𝗮𝗹𝗶𝗱𝗮𝘁𝗼𝗿𝘀
A list of validator functions that are called on the field's input. They should raise serializers.ValidationError (or Django's ValidationError) if validation fails.

𝗲𝗿𝗿𝗼𝗿_𝗺𝗲𝘀𝘀𝗮𝗴𝗲𝘀
A dictionary mapping error codes to their respective error messages, allowing for custom error messages.

𝗹𝗮𝗯𝗲𝗹
A string that serves as a human-readable name for the field, which can be displayed in forms or other interfaces.

𝗵𝗲𝗹𝗽_𝘁𝗲𝘅𝘁
A description of the field, typically displayed alongside it in forms or other interfaces.

𝗶𝗻𝗶𝘁𝗶𝗮𝗹
A value used for pre-populating the field in forms. It can also be a callable, just like Django's field initial.

𝘀𝘁𝘆𝗹𝗲
A dictionary containing rendering options for the field. This can affect how the field is displayed in the browser. Examples include setting input_type for password fields or using a different base template for choice fields.

These core arguments provide the flexibility needed to define the behavior of serializer fields in DRF, ensuring that API representations are precise, secure, and beneficial for both API providers and consumers.

Docs: https://www.django-rest-framework.org/api-guide/fields/#core-arguments

technology #careers

softwareengineering

programing

cloudcomputing

engineering

coding

django

backend

Core Arguments of Django Rest Framework Serializer

jairajsahgal — Sun, 17 Dec 2023 19:00:45 +0000

𝗖𝗼𝗿𝗲 𝗔𝗿𝗴𝘂𝗺𝗲𝗻𝘁𝘀 𝗼𝗳 𝗦𝗲𝗿𝗶𝗮𝗹𝗶𝘇𝗲𝗿𝘀 𝗶𝗻 𝗗𝗷𝗮𝗻𝗴𝗼 𝗥𝗲𝘀𝘁 𝗙𝗿𝗮𝗺𝗲𝘄𝗼𝗿𝗸

Below are core arguments and their purpose:

𝗲𝗿𝗿𝗼𝗿_𝗺𝗲𝘀𝘀𝗮𝗴𝗲𝘀
A dictionary mapping error codes to their respective error messages, allowing for custom error messages.

𝗹𝗮𝗯𝗲𝗹
A string that serves as a human-readable name for the field, which can be displayed in forms or other interfaces.

𝗵𝗲𝗹𝗽_𝘁𝗲𝘅𝘁
A description of the field, typically displayed alongside it in forms or other interfaces.

𝗶𝗻𝗶𝘁𝗶𝗮𝗹
A value used for pre-populating the field in forms. It can also be a callable, just like Django's field initial.

Docs: https://www.django-rest-framework.org/api-guide/fields/#core-arguments

technology #careers

softwareengineering

programing

cloudcomputing

engineering

coding

django

backend

Creating a file uploader in python

jairajsahgal — Sat, 03 Apr 2021 10:21:33 +0000

About

In this blog you will learn how to create a file uploader that runs from a terminal and asks the user to locate the file using a Graphical User Interface and uploads the file using an API from the File.io, the uploaded link is then displayed into the terminal and copied into clipboard, it also creates a QR code for the link.

Imports

Requests (Posting file in API)
Tkinter (For GUI)
Json (To convert result in a dictionary)
Pandas ( To copy the link into a clipboard)
qrcode (To create a QR Code of the link with filename)

import requests
from tkinter import Tk
from tkinter.filedialog import askopenfilename
import json
import pandas as pd
import qrcode
import time

Program

Structure

The program is created in Python and frequently uses methods from different libraries. Also the proram only works in Unix only because of the / and \ difference and I do not want to make distinction of operating system in this simple program.

We hide the main window in tkinter using Tk().withdraw() so that user can only use tkinter to select the file and get its path.

python

print("Please select your file")
time.sleep(1)
filelocation = askopenfilename()
file_name=filelocation.split("/")[-1]
print(file_name)

Asks the user to select the file and waits for 1 second, then a dialog box opens and the user selects a file, user cannot select a directory. The path is stored in a file location and then the file name is extracted using string and array manipulation which is explained below.

file_name=filelocation.split("/")[-1]
Remember filelocation is a variable of class type string, hence using split("/") will split it into an array with elements distributed where / is encountered.
Example of filelocation string

/home/username/Downloads/Documents/file.html

This is what a file location in unix based operating system will show when asked for a file location. Then [-1] ensures that I get the file name.

python

files = {
        'file': (file_name, open(filelocation, 'rb')),
    }

response = requests.post('https://file.io/', files=files)

Then we create a dictionary that will have a key of 'files' and name of the file we are uploading and an object of open() with file location with parameter of reading it as a binary file.
Using requests library function post we post the file in file.io with the parameter as required by their API, we fill the parameter files with our created dictionary i.e. files in our syntax above.

python

data=json.loads(response.text)
if data["success"]==True:
    print("One use hyperlink with your file -> ",data["link"]," copied to your clipboard")
    img=qrcode.make(data["link"])
    img.save(file_name.split(".")[0]+".png")
    df=pd.DataFrame([data["link"]])
    df.to_clipboard(index=False,header=False)
    print("Created QR code for link with name: "+file_name.split(".")[0]+".png")

else:
    print("Error: \n Please check your network \n Possible uploading same file too many times\n Else:\n Server problem")

We convert the text from response from the API which is in a json format into a python dictionary.
We check if the file was uploaded successfully with data["success"]==True
If the file is uploaded successfully, we then print the link into the terminal
python

print("One use hyperlink with your file -> ",data["link"]," copied to your clipboard")

Then we make the QR code, using the following code
python

img=qrcode.make(data["link"])
img.save(file_name.split(".")[0]+".png")

qrcode.make(data["link"]) is passing the link string into make method of qrcode which returns an object of image which we stored in img variable and then using save() we saved the image into program directory.

We then copy the link into our clipboard using below code
python

df=pd.DataFrame([data["link"]])
df.to_clipboard(index=False,header=False)

We create a series in pandas of our link string and then convert the series into a dataframe, and then used to_clipboard() to copy the link into clipboard.
I used pandas library because it is maintained and frequently used by people who handle data.
We tell the user that the QR code of the link has been created and that the link has been copied into computer's clipboard.

Link to the github repository :
Project_Link

Thank you for visiting my article.

DEV Community: jairajsahgal

Understanding Eventual Consistency

Introduction

What is Eventual Consistency?

Is Eventual Consistency limited to NoSQL databases?

Example

Conclusion

𝗗𝗶𝗿𝘁𝘆 𝗿𝗲𝗮𝗱, 𝗡𝗼𝗻-𝗿𝗲𝗽𝗲𝗮𝘁𝗮𝗯𝗹𝗲 𝗿𝗲𝗮𝗱, 𝗮𝗻𝗱 𝗣𝗵𝗮𝗻𝘁𝗼𝗺 𝗿𝗲𝗮𝗱 𝘄𝗶𝘁𝗵 Real 𝗹𝗶𝗳𝗲 𝗲𝘅𝗮𝗺𝗽𝗹𝗲𝘀

How to Increase File Upload Size in Django Backend

Understanding Memory Size Settings

DATA_UPLOAD_MAX_MEMORY_SIZE

FILE_UPLOAD_MAX_MEMORY_SIZE

Fine-Tuning the Symphony: DATA_UPLOAD_MAX_MEMORY_SIZE and FILE_UPLOAD_MAX_MEMORY_SIZE

Core Arguments of Django Rest Framework Serializer

technology #careers

softwareengineering

programing

cloudcomputing

engineering

coding

django

backend

Core Arguments of Django Rest Framework Serializer

technology #careers

softwareengineering

programing

cloudcomputing

engineering

coding

django

backend

Creating a file uploader in python

About

Imports

Program

Structure

technology #careers

technology #careers