DEV Community: Hamzza K

$0 Architecture: Full-Stack application on Serverless Cloud from scratch

Hamzza K — Sun, 24 Sep 2023 20:20:35 +0000

Welcome to the ultimate tutorial that will guide you through the exhilarating journey of building a complete website from scratch. With a pinch of coding wizardry and a dash of creativity, you'll learn the art of crafting a captivating frontend using Next.js. But we won't stop there! We'll delve into the realm of backend development, where you'll master the server-side sorcery with Python and a powerful framework like FastAPI. From integrating databases to creating dynamic content, every aspect of website development will be unveiled.

What you'll learn

Consider this tutorial as your all-access pass to the world of cutting-edge technologies mentioned in job descriptions. And even if they're not explicitly listed, you'll be equipped with the skills to tackle any challenge that shares a similar architecture. You'll learn how to

create an API using FastAPI
scrape data using scrapy
insert data into a cloud database
create lambda functions to trigger the scraper
create a cloud run service to deploy your API

Pre-requisites

Before you begin, make sure you have the following tools installed in your working environment

Docker
Python (≥ 3.8) and pip
Npm & Npx (≥ 9.x.x)

Acquiring Data with Scrapy

Our first step on this exciting journey will be to acquire data, and we'll do so by leveraging the dynamic capabilities of Scrapy, a reliable companion to Python. While static websites may lull you into a slumber (unless, of course, you're creating a portfolio for sheep enthusiasts), we're aiming for a more engaging experience. Scrapy's refined design architecture, akin to a debonair secret agent's tuxedo, exudes both elegance and efficiency. Of course, alternative methods too are available for the adventurous souls among us.

Without further ado, fire up your shell and type the following to install scrapy.
pip install scrapy
Check if the installation was a success by typing:
scrapy --version
You should see the following output:

Before you start scraping, you will have to set up a new Scrapy project. Enter a directory where you'd like to store your code and run: (taken this from https://docs.scrapy.org/en/latest/intro/tutorial.html)
scrapy startproject tutorial
This will create a tutorial directory with the following contents:

tutorial/
    scrapy.cfg            # deploy configuration file
    tutorial/             # project's Python module, you'll import your code from here
        __init__.py
        items.py          # project items definition file
        middlewares.py    # project middlewares file
        pipelines.py      # project pipelines file
        settings.py       # project settings file
        spiders/          # a directory where you'll later put your spiders
            __init__.py

This is the code for our first Spider. Save it in a file named quotes_spider.py under the tutorial/spiders directory in your project:

import scrapy

class QuotesSpider(scrapy.Spider):
    name = "quotes"
    start_urls = [
        "<https://quotes.toscrape.com/page/1/>",
    ]
    def parse(self, response):
        for quote in response.css("div.quote"):
            yield {
                "text": quote.css("span.text::text").get(),
                "author": quote.css("small.author::text").get(),
                "tags": quote.css("div.tags a.tag::text").getall(),
            }
        next_page = response.css("li.next a::attr(href)").get()
        if next_page is not None:
            next_page = response.urljoin(next_page)
            yield scrapy.Request(next_page, callback=self.parse)

Now, from your tutorial/spiders directory, run the following command to check if your spider is working well:
scrapy crawl quotes
You should see a couple of debug logs in your terminal along with the scraped data and a summary at the end with the stats:

See the output as above? Good, now let's set up our database for storing all the scraped data. I'm choosing the serverless XataDB for this. There are couple of other serverless DBs that are as good as not more than this. However, for our $0 architecture and ease of use, I'll be using XataDB.
Open up https://xata.io/ and create a free account. Set up your database name, location and click next.

create a new table and click next, next..

You'll see the following dashboard. This is your table where the data will reside.

We'll modify the schema to add a few features to our table. Go to Schema tab and create three columns, author (string) - quote (text) - date (Datetime)

Click on your database name and go to it's settings.

Note the endpoint of your db, you'll need this for the connection string.

go back and click on your profile. Open Account settings.

In the Personal API keys section, create a new API. You'll be using this to access your table to store the data. Do not share your API. Click save and note the key.

We'll need the Python Xata SDK to make it work in our scrapy project. Open up your shell and type
pip install xata
This will install the 1.xx version in your environment. If, for some reason, the version installed is 0.xx, there are not that major changes in the functionality and you can still use that to perform transactions all the same. Refer to (https://xata.io/docs/sdk/python/examples) for more information.

Before we connect our database we have to modify our spider a bit. First, create a new Item in tutorial/items.py file.

import scrapy
from itemloaders.processors import TakeFirst

class QuotesItem(scrapy.Item):
    text   = scrapy.Field(output_processor=TakeFirst())
    date   = scrapy.Field(output_processor=TakeFirst())
    author = scrapy.Field(output_processor=TakeFirst())

Now, open up your quotes_spider.py file and update your spider.

import scrapy
from datetime import datetime
from zero.items import QuotesItem
from scrapy.loader import ItemLoader

class QuotesSpider(scrapy.Spider):
    name = "quotes"
    start_urls = [
        "<https://quotes.toscrape.com/page/1/>",
    ]
    def parse(self, response):
        date = datetime.now().strftime("%Y-%m-%dT%H:%M:%SZ")

        for quote in response.css("div.quote"):
            loader = ItemLoader(item=QuotesItem())
            loader.add_value("text", quote.css("span.text::text").get())
            loader.add_value("author", quote.css("small.author::text").get())
            loader.add_value("date", date)
            yield loader.load_item()
        next_page = response.css("li.next a::attr(href)").get()
        if next_page is not None:
            next_page = response.urljoin(next_page)
            yield scrapy.Request(next_page, callback=self.parse)

Create a new class or modify the existing one in your tutorial/pipelines.py file.

from xata.client import XataClient
from itemadapter import ItemAdapter

API_KEY = "<YOUR API>"
DB_URL  = "<YOUR ENDPOINT>"

class ZeroPipeline:
    def __init__(self):
        self.client = XataClient(api_key=API_KEY, db_url=DB_URL)
    def process_item(self, item, spider):
        text   = item['text']
        date   = item['date']
        author = item['author']
        record = {
            "author": author,
            "quote": text,
            "date": date
        }

        resp = self.client.records().insert("zero", record)
        spider.logger.info("Record id is: %s" %resp.json()["id"])
        return item

💡I don't recommend using your secret keys in the same file in a plain string text as above but for this tutorial, I'll skip.

Finally, in your tutorial/settings.py file, enable the pipeline. You'lll see the following setting commented out. Un-comment it. Or, you're using a custom classname then change the "ZeroPipeline" with the name you specified.

ITEM_PIPELINES = {
   "zero.pipelines.ZeroPipeline": 300,
}

In your terminal, run the following command to start your spider:
scrapy crawl quotes
Check the debug logs and make sure there aren't any 400 errors. If successful, you'll see the data populated in your table.

Congratulations, you're halfway through.
Now it's time to make an API to fetch that data.

Creating an API using FastAPI

Being the buzz-word in the backend word, I'm using FastAPI to leverage it's fast capabilities. One thing about FastAPI I find impressive is the use of strong typing model (pydantic) to validate the models you build. You can also use Flask for this task but considering how everyone is now demanding 10 years of experience in FastAPI, it should be lucrative to get to know it.
Launch your shell and type the following command
pip install fastapi
You will also need an ASGI server, for production such as Uvicorn or Hypercorn.
pip install "uvicorn[standard]"
Create a new directory called api, next to your web-scraper. Create a new file api/main.py and enter the following content:

from typing import Any, Union
from datetime import datetime
from pydantic import BaseModel
from xata.client import XataClient
from fastapi import FastAPI, HTTPException, Query, Body

app = FastAPI()

API_KEY = "<YOUR API>"
DB_URL = "<YOUR ENDPOINT>"

xata = XataClient(api_key=API_KEY, db_url=DB_URL)

class FilterParams(BaseModel):
    """Base Model class to pass into the FastAPI routes for xata client."""
    table_name: str
    columns: list
    filter: Union[dict, None] = None
    sort: Union[dict, None] = None

@app.get("/")
def read_root():
    return {"message": "All OK"}

@app.post("/read-records/")
def read_records(filter_params: FilterParams):
    """Reads the records from XataDB"""
    try:
        resp = xata.data().query(filter_params.table_name, {
            "columns": filter_params.columns,
            "filter": filter_params.filter,
            "sort": filter_params.sort
        })
        if resp.is_success():
            records = resp.json()["records"]
            for record in records:
                date_str = record["date"]
                date_obj = datetime.strptime(date_str, "%Y-%m-%dT%H:%M:%SZ")
                formatted_date = date_obj.strftime("%A, %B %d, %Y")
                record["formatted_date"] = formatted_date
            return records
        else:
            raise HTTPExecption(status_code=resp.status_code, detail="Error reading records")
    except Exception as error:
        raise HTTPException(status_code=500, detail=str(error)) from error

Save the file and write the following in your shell
uvicorn main:app --reload
This will start your server on port 8000 on your localhost.

Click on the link or go to localhost:8000 and you'll be greeted with the message:

Let's go the the docs and interact with our API. append /docs at the end of your link e.g localhost:8000/docs and FastAPI will generate the UI using swagger.

Click on the POST method /read-records/ to test whether our API fetches the required data. Modify the body to reflect the changes below. Change the date accordingly.

{
  "table_name": "zero",
  "columns": [
    "date", "author", "quote"
  ],
  "filter": {"date": "2023-09-18T11:34:27Z"},
  "sort": {"date": "asc"}
}

Click on execute and check the response

You should see the following output.

If you've made it this far, good job.

Dockerizing your API

It's time to flex your Docker skills. Actually, not much is needed here except a few lines of code.
First, create api/requirements.txt file and enter the following requirements.

annotated-types==0.5.0
click==8.1.7
deprecation==2.1.0
fastapi==0.101.1
h11==0.14.0
httptools==0.6.0
orjson==3.9.5
pydantic==2.2.1
pydantic_core==2.6.1
starlette==0.27.0
uvicorn==0.23.2
uvloop==0.17.0
watchfiles==0.19.0
websockets==11.0.3
xata==1.0.1

Create a Dockerfile and copy the following content into it.

FROM python:3.9-slim@sha256:980b778550c0d938574f1b556362b27601ea5c620130a572feb63ac1df03eda5

ENV PYTHONUNBUFFERED True
ENV APP_HOME /app
WORKDIR $APP_HOME
COPY . ./
ENV PORT 1234
RUN pip install --no-cache-dir -r requirements.txt
CMD exec uvicorn main:app --host 0.0.0.0 --port ${PORT} --workers 1

Build your image by running the following command:
docker build -t api .
Check if the container is working correctly
docker run -dp 8000:1234 -e PORT=1234 api
Go to localhost:8000 and you should see a response of {"message": "All OK"} You can skip the next part if you intend to only work locally. However, I would suggest not to.

Deploying your API on Google Cloud Run

Register for the google cloud account if you haven't already. Once you're in, create a new project.

Enter the project name and click Create.

Select the project to start working on it.

Make sure your billing account is linked to this project, if not, link it. Note the project ID to be used. In this case, it is zerodollar-399414

I'll be using Gcloud CLI to manage and deploy my service. Refer → https://cloud.google.com/sdk/docs/install to install it on your system.

Link your project id with the gcloud CLI
gcloud config set project <YOUR PROJECT ID>

Authenticate your account by running the command. This will prompt you for your google credentials by opening up a modal.
gcloud auth login
After log-in, check whether these services are enabled. If not, you have to enable them by using the following command: gcloud services enable <servicename>.googleapis.com

Create a new repository in Artifact registry. Google provides 0.5gb of free storage, make sure that you do not exceed this to remain inside the $0 budget.
gcloud artifacts repositories create zero --repository-format=docker --location=us-east1 --description="Zero Dollar API"
Before pushing the images to your repository, you first have to configure docker authentication. Type: (change the region accordingly)
gcloud auth configure-docker us-east1-docker.pkg.dev

Tag your image by entering the following command:

docker tag api:latest us-east1-docker.pkg.dev/<YOUR PROJECT ID>/zero/api

Finally, push your image to the Artifact registry:
docker push us-east1-docker.pkg.dev/<YOUR PROJECT ID>/zero/api

Deploying your Image to Cloud Run

It's time to deploy your image as a service using cloud run. Run the following command:

gcloud run deploy zero-dollar-api --image us-east1-docker.pkg.dev/zerodollar-399414/zero/api
 --allow-unauthenticated

After a while, you will see the following message on success.

Go to your google cloud console and click on the hamburger icon on the left of the screen and find Cloud Run. Click on it.

You'll see your service running. Click on it.

Go to the URL specified in the above service. Hopefully, you'll see the OK message indicating the success of your deployed API.
Wonderful! now your API is publically available. How does it feel? amazing, right?

💡 Given the number of words exceeded for a typical tutorial article here, I'll be skipping the scheduling process using Lambda function using EventBridge. And also, deploying the scrapy spiders.

Front-end using Next.js

Next.js is chosen as the frontend for our website because it has many advantages. It helps the website load faster and perform better. It's easy to scale your website as it grows. Next.js also makes it simple to develop and deploy your website. Create a new project by typing:
npx create-next-app@latest
select the default values and press enter

cd into your project and create new folders: pages/api/ Your folder structure now should look like this.

Inside your /api folder, create a new file called fetch-quotes.ts and enter the following code.

import type { NextApiRequest, NextApiResponse } from "next";

type ResponseData = {
    date: string,
    author: string,
    quote: string,
}
export default async function handler(req: NextApiRequest, res: NextApiResponse) {
    try {
        const URL: string = "http://localhost:8080/read-records/"
        const response = await fetch(URL, {
            method: "POST",
            headers: {
                "Content-Type": "application/json",
            },
            body: JSON.stringify({
                "table_name": "zero",
                "columns": ["date", "author", "quote"],
                "filter": {"date": "2023-09-18T11:34:27Z"},
                "sort": {"date": "asc"}
                })
        });
        const data: Promise<ResponseData> = await response.json();
        res.status(200).json(data);
    } catch (error) {
        console.error("ERROR: fetching data", error);
        res.status(500).json({error: "Error fetching data"});
    }
}

💡 The reason we're making Next api is to mask our google cloud API.
We'll first be using our locally run docker API to fetch the data. Make sure that your container is running on port 8080.
Create a new file Table.tsx in your components folder and enter the following content

'use client'
import { useEffect, useState } from 'react';

interface QuoteData {
  date: string;
  author: string;
  quote: string;
  formatted_date: string;
}
const Table = () => {
  const [quoteData, setQuoteData] = useState<QuoteData[]>([]);
  useEffect(() => {
    const fetchData = async () => {
      try {
        const response = await fetch('/api/fetch-quotes');
        if (response.ok) {
          const data: QuoteData[] = await response.json();
          setQuoteData(data);
        } else {
          console.error('Error fetching data:', response.statusText);
        }
      } catch (error) {
        console.error('Error fetching data:', error);
      }
    };
    fetchData();
  }, []);
  return (
    <table className="w-full border-collapse">
      <thead>
        <tr>
          <th className="border p-2">Date</th>
          <th className="border p-2">Author</th>
          <th className="border p-2">Quote</th>
        </tr>
      </thead>
      <tbody>
        {quoteData.map((quote, index) => (
          <tr key={quote.date}>
            <td className={`border p-2 ${index !== quoteData.length - 1 ? 'border-b' : ''}`}>{quote.formatted_date}</td>
            <td className={`border p-2 ${index !== quoteData.length - 1 ? 'border-b' : ''}`}>{quote.author}</td>
            <td className={`border p-2 ${index !== quoteData.length - 1 ? 'border-b' : ''}`}>{quote.quote}</td>
          </tr>
        ))}
      </tbody>
    </table>
  );
};
export default Table;

Finally, use your component in your page.tsx

import Table from "@/components/Table"

export default function Home() {
  return (
    <div>
      <Table />
    </div>
  )
}

Run: npm run dev to start your next app. Hopefully, you'll see the following table.
Now edit your Table.tsx and replace the URL with your Cloud Run API url

import type { NextApiRequest, NextApiResponse } from "next";

type ResponseData = {
    date: string,
    author: string,
    quote: string,
}
export default async function handler(req: NextApiRequest, res: NextApiResponse) {
    try {
        const URL: string = "<Your API>/read-records/"
        const response = await fetch(URL, {
            method: "POST",
            headers: {
                "Content-Type": "application/json",
            },
            body: JSON.stringify({
                "table_name": "zero",
                "columns": ["date", "author", "quote"],
                "filter": {"date": "2023-09-18T11:34:27Z"},
                "sort": {"date": "asc"}
                })
        });
        const data: Promise<ResponseData> = await response.json();
        res.status(200).json(data);
    } catch (error) {
        console.error("ERROR: fetching data", error);
        res.status(500).json({error: "Error fetching data"});
    }
}

The page will refresh and populate the data from the API. You can look at the metrics for verification.

Congratulations! you can now build an end-to-end applications incorporating serverless cloud services.

For the complete tutorial including Lambda Functions and Spiders deployment go here

You can find all the source code here

What are these method in 🐍 you sssssay? Look no further!

Hamzza K — Wed, 20 Sep 2023 12:28:39 +0000

Have you ever used or seen the __init__() or __str__() method in Python classes? These are called dunder methods! Dunder, short for "double underscore", is a term used in Python to refer to special methods that allow programmers to modify the behavior of objects in a logical and intuitive way. In this article, we will examine Python's dunder methods and show how to use them to create more Pythonic code. We'll cover the basics of dunder methods, their common use cases, and best practices for using them effectively.

How do Dunder Methods work?

Dunder (double underscore) methods, are unique in the Python programming language that have double underscores defined on either side of their names, such as __init__(), __str__(), or __eq__(). These methods allow you to modify the behaviour of objects and give a more user-friendly interface for dealing with them. You can specify how instances of the class should act when they are generated, printed, compared, or utilised in arithmetic operations by specifying dunder methods in a class declaration.

Common Use Cases for Dunder Methods

Dunder methods are extensively used in Python libraries and frameworks, and have become a standard convention in the Python community. Here are some common use cases for dunder methods:

Initialization

The __init__() method is used to initialize a new instance of a class. It is called when a new object is created, and is used to set the initial state of the object. This method takes a self parameter, which refers to the instance of the class being initialized, and any additional parameters that are needed to set the initial state of the object. The __init__ method is most commonly used to initialize the attributes of an object when it is created.

class Person:
    def __init__(self, name, age):
        self.name = name
        self.age = age

person = Person("John", 25)
print(person.name) # Output: Alice
print(person.age) # Output: 25

String Representation

The __str__() and __repr__() methods serve the purpose of defining a string representation of an object.

While __str__() is used to define a human-readable string representation, __repr__() is used to define a machine-readable format for the object.

class Person:
    def __init__(self, name, age):
        self.name = name
        self.age = age

    def __str__(self):
        return f"{self.name} ({self.age} years old)"

person = Person("John", 25)
print(person) # Output: John (25 years old)

class Person:
    def __init__(self, name, age):
        self.name = name
        self.age = age

    def __repr__(self):
        return f"Person(name='{self.name}', age={self.age})"

person = Person("John", 25)
print(person) # Output: Person(name='John', age=25)

The __init__ method can be used in scenarios involving multiple inheritance to initialize attributes of all parent classes.

class Animal:
    def __init__(self, species):
        self.species = species

    def speak(self):
        pass

class Mammal:
    def __init__(self, mammal_type):
        self.mammal_type = mammal_type

    def feed_young(self):
        pass

class Dog(Animal, Mammal):
    def __init__(self, name, age):
        Animal.__init__(self, "Dog")
        Mammal.__init__(self, "Canine")
        self.name = name
        self.age = age

    def speak(self):
        return "Woof"

dog = Dog("Fido", 3)
print(dog.species) # Output: Dog
print(dog.mammal_type) # Output: Canine
print(dog.name) # Output: Fido
print(dog.age) # Output: 3
print(dog.speak()) # Output: Woof

Container Operations

Dunder methods can be used to define the behavior of container operations such as len(), indexing ([]), assignment to an indexed value (obj[index] = value), deletion of an indexed value (del obj[index]), and membership testing (in)

__len__()

The __len__() method is used to define the behavior of the len() function for objects of a class. It should return the number of items in the container.

class MyList:
    def __init__(self, items):
        self.items = items

    def __len__(self):
        return len(self.items)

my_list = MyList([1, 2, 3])
print(len(my_list)) # Output: 3

__getitem__()

The __getitem__()
method is used to define the behavior of indexing ([]) for objects of a class. It should return the item at the given index.

class MyList:
    def __init__(self, items):
        self.items = items

    def __getitem__(self, index):
        return self.items[index]

my_list = MyList([1, 2, 3])
print(my_list[2]) # Output: 3

__setitem__()

The __setitem__() method is used to define the behavior of assignment to an indexed value (obj[index] = value) for objects of a class. It should set the item at the given index to the given value.

class MyList:
    def __init__(self, items):
        self.items = items

    def __setitem__(self, index, value):
        self.items[index] = value

my_list = MyList([1, 2, 3, 4, 5])
my_list[2] = 66
print(my_list.items) # Output: [1, 2, 66, 4, 5]

__delitem__()

The __delitem__() method is used to define the behavior of deletion of an indexed value (del obj[index]) for objects of a class. It should delete the item at the given index.

class MyList:
    def __init__(self, items):
        self.items = items

    def __delitem__(self, index):
        del self.items[index]

my_list = MyList([1, 2, 3, 4, 5])
del my_list[2]
print(my_list.items) # Output: [1, 2, 4, 5]

__contains__()
The __contains__() method is used to define the behavior of membership testing (in) for objects of a class. It should return True if the given value is present in the container, and False otherwise.

class MyList:
    def __init__(self, items):
        self.items = items

    def __contains__(self, value):
        return value in self.items

my_list = MyList([1, 2, 3, 4, 5])
print(2 in my_list) # Output: True
print(10 in my_list) # Output: False

These examples demonstrate how container operations can customize the behavior of Python objects. By defining these methods in a class definition, you can provide a more intuitive interface for interacting with objects and make your code more flexible and customizable.

Comparison

Dunder methods can be used to define the behavior of comparison operators such as ==, <, >, <=, and >= for objects of a class. These methods should return True or False depending on whether the comparison is true or false.

__eq__()

The __eq__() method is used to compare two objects for equality.

class Person:
    def __init__(self, name, age):
        self.name = name
        self.age = age

    def __eq__(self, other):
        return self.name == other.name and self.age == other.age

person1 = Person("Alice", 25)
person2 = Person("Bob", 30)
person3 = Person("Alice", 25)

print(person1 == person2) # Output: False
print(person1 == person3) # Output: True

__lt__()

The __lt__() method is used to compare two objects to check if one is less than the other.

class Person:
    def __init__(self, name, age):
        self.name = name
        self.age = age

    def __lt__(self, other):
        return self.age < other.age

person1 = Person("Alice", 25)
person2 = Person("Bob", 30)

print(person1 < person2) # Output: True
print(person2 < person1) # Output: False

__gt__()

The __gt__() method is used to compare two objects to check if one is greater than the other.

class Person:
    def __init__(self, name, age):
        self.name = name
        self.age = age

    def __gt__(self, other):
        return self.age > other.age

person1 = Person("Alice", 25)
person2 = Person("Bob", 30)

print(person1 > person2) # Output: False
print(person2 > person1) # Output: True

__le__()

The __le__() method is used to compare two objects to check if one is less than or equal to the other. Here's an example:

class Person:
    def __init__(self, name, age):
        self.name = name
        self.age = age

    def __le__(self, other):
        return self.age <= other.age

person1 = Person("Alice", 25)
person2 = Person("Bob", 30)
person3 = Person("Charlie", 25)

print(person1 <= person2) # Output: True
print(person2 <= person1) # Output: False
print(person1 <= person3) # Output: True

__ge__()

The __ge__() method is used to compare two objects to check if one is greater than or equal to the other. Here's an example:

class Person:
    def __init__(self, name, age):
        self.name = name
        self.age = age

    def __ge__(self, other):
        return self.age >= other.age

person1 = Person("Alice", 25)
person2 = Person("Bob", 30)
person3 = Person("Charlie", 25)

print(person1 >= person2) # Output: False
print(person2 >= person1) # Output: True
print(person1 >= person3) # Output: True

Arithmetic Operations

Dunder methods can be used to define the behavior of arithmetic operators such as +, -, *, and / for objects of a class. These methods should return a new object that represents the result of the arithmetic operation.

__add__()

The __add__() method is used to define the behavior of the + operator when applied to two objects. Here's an example:

    class Vector:
        def __init__(self, x, y):
            self.x = x
            self.y = y

        def __add__(self, other):
            return Vector(self.x + other.x, self.y + other.y)

    v1 = Vector(1, 2)
    v2 = Vector(3, 4)
    v3 = v1 + v2

    print(v3.x) # Output: 4
    print(v3.y) # Output: 6

__sub__()

The __sub__() method is used to define the behavior of the - operator when applied to two objects. Here's an example:

class Vector:
    def __init__(self, x, y):
        self.x = x
        self.y = y

    def __sub__(self, other):
        return Vector(self.x - other.x, self.y - other.y)

v1 = Vector(1, 2)
v2 = Vector(3, 4)
v3 = v2 - v1

print(v3.x) # Output: 2
print(v3.y) # Output: 2

__mul__()

The __mul__() method is used to define the behavior of the * operator when applied to two objects. Here's an example:

class Vector:
    def __init__(self, x, y):
        self.x = x
        self.y = y

    def __mul__(self, other):
        return Vector(self.x * other, self.y * other)

v1 = Vector(1, 2)
v2 = v1 * 3

print(v2.x) # Output: 3
print(v2.y) # Output: 6

__truediv__()

The __truediv__() method is used to define the behavior of the / operator when applied to two objects. Here's an example:

class Vector:
    def __init__(self, x, y):
        self.x = x
        self.y = y

    def __truediv__(self, other):
        return Vector(self.x / other, self.y / other)

v1 = Vector(3, 6)
v2 = v1 / 3

print(v2.x) # Output: 1.0
print(v2.y) # Output: 2.0

Best Practices for Using Dunder Methods

While dunder methods can be a powerful tool for customizing the behavior of objects in Python, it is important to use them thoughtfully and appropriately. Here are some best practices for using dunder methods effectively:

Use dunder methods to define object behavior in a way that is consistent with Python conventions and libraries.
Use descriptive names for dunder methods that accurately reflect their purpose and behavior.
Be aware of the specific names and behaviors of dunder methods used by Python itself, and use them appropriately when defining your own dunder methods.
Use dunder methods judiciously. Too many can make your code more complex and harder to understand.
Thoroughly test your dunder methods to ensure that they behave as expected and do not introduce unexpected side effects.

Conclusion

Dunder methods are a powerful feature of Python that enables developers to customize the behavior of objects in a consistent and intuitive way. By using dunder methods effectively, you can make your code more readable, maintainable, and flexible. You can also ensure that it is consistent with the expectations of other Python developers. Understanding the usage of dunder methods is an important part of writing Pythonic code, and can help you become a more effective Python developer.

Automate your Hyper-V Windows through Docker Ansible

Hamzza K — Sat, 02 Sep 2023 16:20:47 +0000

But why, you may wonder?
Let me explain the problem I'm currently facing and the rationale behind my chosen approach. One of my job responsibilities involves deploying software that exclusively operates on Windows.
Performing this task manually is exceptionally tedious, to say the least.
So, why not automate it, you might ask?
The challenge lies in the fact that my work laptop runs on Windows, and I'm unable to create Linux-based VMs due to company policies. Unfortunately, Ansible doesn't function within Windows when running inside Hyper-V.
Although I've contemplated using WSL, I wasn't in the mood for it, plus I have a preference for Docker.
Consequently, I arrived at this solution after spending several days troubleshooting these issues. I hope this explanation proves beneficial to anyone encountering a similar predicament.

Pre-requisites

before beginning, make sure you configure the following:

Hyper-V Virtual Switch
WinRM on your Target
Firewall Ports on your Target

Hyper-V Virtual Switch

Since we are working with Windows Hyper-V we need to set up the connection type for our VM to use.

Go to Actions → Virtual Switch Manager → New virtual network switch
to create a new connection. Make sure to connect you LAN and select External Network also allow your OS to share the network adapter otherwise you won’t be able use internet on your host machine.
💡

Sharing your adapter whilst on WiFi might behave unexpectedly.

💡

Internal Network isolates your VM from the network. You won’t be able to communicate with it using your host machine. However, it is possible to ping/communicate through another VM running in the same hyper-V manager. Private Network is almost the same except it restricts communication from other devices that are not on the same network.

apply and save the changes. Select your target VM, make sure it is turned-off, right click and go to → settings, assign the virtual switch you just created.

log into your VM and open up powershell/cmd.
type ipconfig and note the IPv4 address.

Now open up the shell on your host machine (if you are on the same network) and ping the ip-address. if you get the response, you’re good to go.

WinRM

WinRM stands for Windows Remote Management, which is a Microsoft technology that allows for remote management of Windows machines. It uses the open standard WS-Management protocol for secure and reliable communication between the machines.
fire up the powershell and type:

if the service is not running the type: Start-Service -Name WinRM
Now enable the remote connection by typing Enable-PSRemoting -Force

Chocolatey

Install the chocolatey package which is to be used by ansible to install the softwares.
Set-ExecutionPolicy Bypass -Scope Process -Force; iex ((New-Object System.Net.WebClient).DownloadString('https://chocolatey.org/install.ps1'))

Ports

check the following ports (5985, 5986) in your Windows defender firewall. These ports are required by ansible_winrm to communicate.

if you don’t see these ports then simply create a new Inbound rule for the given ports and enable it for all Profiles.

Host Setup

Now it’s time to set up the host node which will control your hyper-V VM. Before diving, make sure you have Docker installed. Create a new directory and cd into it. Create the following three files and leave them empty for now.

Dockerfile
inventory.ini
playbook.yaml (ignore .gitignore, LICENSE and README for now)

Playbook

Ansible playbooks are files containing a series of tasks that can be executed in order to automate IT infrastructure configurations and deployment processes. Playbooks are written in YAML format and can be used to execute tasks on a single or multiple hosts, depending on the requirements. They are a powerful tool for automating complex tasks and can greatly simplify IT operations.
Enter the following contents into the playbook and save it.

---
- name: Install Software on Windows VM
  hosts: windows_vm
  gather_facts: false
  vars:
    ansible_winrm_transport: ntlm
  tasks:
    - name: Install NeoVim
      win_command: choco install neovim -y

    - name: Install Git
      win_command: choco install git -y

    - name: Install Google Chrome
      win_command: choco install googlechrome -y

Inventory
In Ansible, an inventory is a file or collection of files that defines the hosts and groups of hosts on which Ansible commands and playbooks will run. It serves as the source of truth for Ansible about the infrastructure it manages. The inventory can include details about hostnames, IP addresses, remote connection information, host grouping, and variables that provide essential context for executing automation tasks across a network.
Enter the following contents and save the file.
[windows_vm] <ip-address> ansible_user=<username> ansible_password=<password> ansible_connection=winrm ansible_winrm_server_cert_validation=ignore

In the context of Ansible and WinRM, ansible_winrm_transport is an Ansible variable that specifies the transport method used for communication between the Ansible control node and the Windows VM. The two common options for this variable are ntlm and kerberos.

NTLM (NT LAN Manager): NTLM is a Microsoft authentication protocol that uses a challenge-response mechanism. It is an older authentication method and may be suitable for simpler setups or legacy systems.
Kerberos: Kerberos is a network authentication protocol that provides strong security and is the recommended authentication method for WinRM communication. It relies on a trusted third-party authentication server and provides secure mutual authentication between the Ansible control node and the Windows VM. To determine which transport method to use, you should consider the configuration of your Windows VM and the authentication mechanisms supported by your environment. If your Windows VM is part of an Active Directory domain and Kerberos is configured, it is generally recommended to use Kerberos for more secure authentication. However, if you are working in a non-domain environment or have specific requirements, NTLM can be used as an alternative.

Dockerfile

open up your editor and enter the following contents into the file and save it. (You can also use the official ansible image)

Build your Dockerfile

Now its time to build and run your image. Enter the following command:

docker build -t <tag-name> .

You should see the following output:

Now run your image:
docker run <tag-name>

if successful, you should see the following output:

Go to your target VM and check the tools installed by running the following command:

choco list

Troubleshooting Errors

If you get the following error: “auth method ntlm requires a username”

then check if the

service is running
ports are opened NTLM uses port 5986, check whether the WinRM Listener is listening on the required port. open up the powershell and type Get-WSManListener Make sure the port is correct. If the port is correct then the issue might be the certificates NTLM requires. Run the following command:

New-Item -Path WSMan:\localhost\Listener -Transport HTTPS -Address * -CertificateThumbPrint <thumbprint> -Force

Replace with the actual thumbprint from your certificate:
Get-ChildItem -Path Cert:\LocalMachine\My

If you only have the self-signed certificate (localhost) then just omit the -CertificateThumbPrint parameter otherwise it will result in an error.
New-Item -Path WSMan:\localhost\Listener -Transport HTTPS -Address * -Force

if Windows cannot find the certificate due to any issue then create a new one
$cert = New-SelfSignedCertificate -DnsName "localhost" -CertStoreLocation "cert:\LocalMachine\My"

Make a note of the thumbprint value displayed after this command:
$cert.Thumbprint

Finally, run this command to include the local certificate:
New-Item -Path WSMan:\localhost\Listener -Transport HTTPS -Address * -CertificateThumbPrint $cert.Thumbprint -Force

Restart the WinRM service to apply the changes:
Restart-Service -Name WinRM

Hopefully ansible works after this.

You can find the Dockerfile here:
https://github.com/hamzza-K/docker-ansible-hyperV.git

Mastering Monad Design Patterns: Simplify Your Python Code and Boost Efficiency

Hamzza K — Sun, 26 Feb 2023 16:17:58 +0000

Monad Design Pattern

Monad is a functional programming design pattern that enables you to combine several calculations or functions into a single expression while also managing error circumstances and side effects. Every function in the chain should, in theory, return a new monad that may be used as input by the function after it.

Types of Monad

In functional programming, there are several types of Monads that are commonly used to represent different kinds of computations. Here are a few examples:
Maybe Monad: Represents a computation that may or may not return a value. This is useful for handling error conditions or optional values.
State Monad: Represents a computation that maintains an internal state that is passed from one function to the next. This can be useful for modeling simulations or other computations that require tracking changes over time.
Reader Monad: Represents a computation that has access to a shared environment or configuration data. This can be useful for parameterizing computations and making them more reusable.
Writer Monad: Represents a computation that generates output or side effects. This can be useful for logging, debugging, or other kinds of diagnostics.
IO Monad: Represents a computation that performs input/output operations or other kinds of side effects. This can be useful for interacting with external systems, such as databases or web services.

Each Monad has its own set of operations that define how computations can be chained together and how values can be transformed or combined. However, all Monads share the property of being composable and modular, which makes them a powerful tool for building complex computations in a functional style.

Maybe Monad

In Python, the Monad design pattern can be implemented using classes and operator overloading. Here's an example implementation of the Maybe Monad, which represents a computation that may or may not return a value:

class Maybe:
    def __init__(self, value):
        self._value = value

    def bind(self, func):
        if self._value is None:
            return Maybe(None)
        else:
            return Maybe(func(self._value))

    def orElse(self, default):
        if self._value is None:
            return Maybe(default)
        else:
            return self

    def unwrap(self):
        return self._value

    def __or__(self, other):
        return Maybe(self._value or other._value)

    def __str__(self):
        if self._value is None:
            return 'Nothing'
        else:
            return 'Just {}'.format(self._value)

    def __repr__(self):
        return str(self)

    def __eq__(self, other):
        if isinstance(other, Maybe):
            return self._value == other._value
        else:
            return False

    def __ne__(self, other):
        return not (self == other)

    def __bool__(self):
        return self._value is not None

def add_one(x):
    return x + 1

def double(x):
    return x * 2

result = Maybe(3).bind(add_one).bind(double)
print(result)  # Just 8

result = Maybe(None).bind(add_one).bind(double)
print(result)  # Nothing

result = Maybe(None).bind(add_one).bind(double).orElse(10)
print(result)  # Just 10

result = Maybe(None) | Maybe(1)
print(result) # Just 1

In this example, the Maybe class represents a computation that may or may not return a value. The bind method takes a function as input and returns a new Maybe instance that represents the result of applying the function to the original value, if it exists. The | operator can be used to combine two Maybe instances, returning the first one that contains a value.
The add_one and double functions represent computations. These functions can be chained together using the bind method to create more complex computations that can handle error conditions and side effects.
Note that the Monad design pattern is not a commonly used pattern in Python, as it is more commonly associated with functional programming languages like Haskell. However, the pattern can still be useful in certain situations where you need to chain computations together in a more modular and reusable way.

State Monad

The state monad allows you to encapsulate a stateful computation as a pure function that takes an initial state and returns a new state and a result. The state is typically represented as a data structure, and the function performs computations that update the state as needed. The state monad is often used in functional languages like Haskell and Scala, but it can also be implemented in Python.
In Python, you can implement the state monad using classes and closures. The basic idea is to define a class that represents a stateful computation, and use closures to create new stateful computations that depend on the current state. The call method of the class is used to define the actual computation, and it returns a new instance of the class with an updated state and a result.
Here's a simple example of how to implement the state monad in Python to perform a stateful computation that counts the number of times a function is called:

class State:
    def __init__(self, state):
        self.state = state

    def __call__(self, value):
        return (self.state[1], State((self.state[0] + 1, value)))

# create a stateful computation that counts the number of times it is called
counter = State((0, 0))

# call the computation multiple times and print the current count
for i in range(5):
    result, counter = counter(i)
    print(f"Computation result: {result}, count: {counter.state[0]}") 

#Computation result: 0, count: 1
#Computation result: 0, count: 2
#Computation result: 1, count: 3
#Computation result: 2, count: 4
#Computation result: 3, count: 5

In this example, we define a State class that encapsulates a stateful computation. The init method initializes the state, which is represented as a tuple with two values: the count and the result. The call method is the actual computation, which returns a tuple containing the result and a new instance of the State class with an updated state.
We then create an instance of the State class called counter that represents the stateful computation that counts the number of times it is called. We call the computation multiple times using a loop, and print the current count and result after each call.
The benefits of using the state monad in Python include the ability to write pure functions that encapsulate stateful computations, which can improve code clarity and maintainability. By separating the stateful computation from the rest of the code, you can write more modular and testable code that is easier to reason about. Additionally, the use of closures can make it easier to write stateful computations that depend on the current state, and can simplify code that would otherwise be more complex to write and maintain.

Reader Monad

The Reader monad is a functional programming concept that allows you to pass around an immutable environment to a function, so that the function can access values from the environment without having to explicitly pass them as arguments.
In the Reader monad, the environment is modeled as a function that takes a single argument and returns a value. The function that uses the environment is then wrapped in a monadic context, so that it can be composed with other monadic functions.
Here's an example that demonstrates the basic usage of the Reader monad in Python:

from typing import Any, Callable, TypeVar

T = TypeVar('T')
def reader(f: Callable[[Any], T]) -> Callable[[Any], T]:
    def wrapped(*args):
        return f(*args)
    return wrapped

def greet(name: str) -> str:
    return f"Hello, {name}!"

greet_reader = reader(greet)

# call greet_reader with the name argument
result = greet_reader("Alpha")

print(result) # output: "Hello, Alpha!"

In this example, the reader function is a helper function that returns a wrapped function that takes a single argument. The wrapped function calls the original function with the argument.
Here, greet function takes a single argument, name, and returns a string. The greet_reader function is created by calling the reader function with the greet function as an argument. The greet_reader function takes a single argument, name, and returns the result of calling greet with the name argument.
Using Reader monad for Configuration:

from typing import Dict, Callable, TypeVar

T = TypeVar('T')
def reader(f: Callable[..., T]) -> Callable[..., T]:
    def wrapped(*args, **kwargs):
        config = kwargs.get('config')
        return f(config, *args)
    return wrapped

@reader
def greet(config: Dict[str, str]) -> str:
    return f"Hi, {config['name']}"

result = greet(config={'name':'Beta'})
print(result)

In this example, the reader function takes a function as an argument and returns a wrapped function. The wrapped function takes an additional keyword argument, config, which is used to pass a configuration dictionary to the function.
By decorating a function with the reader decorator, you are creating a new function that expects a config keyword argument and passes it to the decorated function. This allows you to separate the configuration data from the rest of your function's logic.
In the example code, the greet function is decorated with the reader decorator. This means that when you call the greet function using greet(config={"name": "Beta"}), the config dictionary is passed to the decorated function, and the resulting string is returned.
The greet function itself takes a config dictionary as its argument and returns a string greeting the person whose name is specified in the config dictionary. The config argument is passed to the greet function via the wrapped function created by the reader decorator.
These are just some simple examples of using the Reader monad in Python. The concept can be applied to a wide range of scenarios where there are dependencies between functions.
Overall, the Reader monad can be a powerful tool for building functional programs in Python, especially when working with complex and nested data structures.

Writer Monad

The Writer Monad allows us to perform computations while accumulating a log or other auxiliary information. It is similar to the Reader monad in that it separates some aspect of your program's behavior (in this case, logging or accumulation) from the rest of your application logic.
In Python, you can implement the Writer monad using a combination of a tuple and a function that takes a value and a log, and returns a new value and log. This function is usually called the "writer function".

from typing import Tuple

def writer(value, log):
    return (value, log)

def add(x, y):
    result = x + y
    log = f"Adding {x} and {y} to get {result}.\n"
    return writer(result, log)

def multiply(x, y):
    result = x * y
    log = f"Multiplying {x} and {y} to get {result}.\n"
    return writer(result, log)

# Chain together add and multiply using the Writer monad
add_result, add_log = add(2, 3)
mul_result, mul_log = multiply(add_result, 4)
result = mul_result
log = add_log + mul_log
print(f"Result: {result}")
print(f"Log: {log}")

#result: 20
#log: Adding 2 and 3 to get 5.
#Multiplying 5 and 4 to get 20.

In this example, the writer function takes a value and a log as its arguments and returns a tuple containing the value and log. The add and multiply functions perform addition and multiplication, respectively, and also generate log messages using formatted strings.
This demonstrates how the Writer monad can be used to accumulate log messages as your program runs, making it easier to debug and understand the behavior of your code.
Another example of the Writer monad might involve accumulating a list of values as your program runs, or maintaining a running total of some quantity. The basic idea is the same: use a tuple and a writer function to accumulate values or logs, and chain together functions using the partial function to combine them into a larger computation.

IO Monad

The IO monad is a way of dealing with input and output in a purely functional way. In Python, the IO monad can be implemented using a class with a single method call that takes no arguments and returns the result of the IO operation.
Here's an example of how you might use the IO monad in Python to read a file and print its contents:

class IO:
    def __init__(self, effect):
        self.effect = effect

    def __call__(self):
        return self.effect()

def read_file(filename):
    def read_file_effect():
        with open(filename, 'r') as f:
            return f.read()

    return IO(read_file_effect)

def print_contents(contents):
    def print_effect():
        print(contents)

    return IO(print_effect)

# chain the IO operations manually
contents = read_file('example.txt')()
print_contents(contents)()

In this example, we call read_file() to create an IO object that reads the contents of a file. We then call the call() method of this object to execute the IO operation and retrieve the contents of the file. We store the contents in the contents variable and pass it as an argument to print_contents() which creates another IO object that prints the contents to the console. Finally, we call the call() method of the print_contents() object to execute the IO operation and print the contents of the file to the console.

Conclusion

In conclusion, Monad is a design pattern that is used to structure functional programs. It is a powerful abstraction that helps developers to deal with side effects and provides a way to compose complex operations in a declarative manner.
In Python, monads can be used to write clean and expressive code that is easy to understand and reason about. By using monads, we can write code that is more modular, composable, and easier to test. Monads provide a way to handle side effects without sacrificing the purity of our functions. While monads can be challenging to learn at first, once you understand the concepts behind them, they can be a powerful tool in your programming arsenal.

DEV Community: Hamzza K

$0 Architecture: Full-Stack application on Serverless Cloud from scratch

What you'll learn

Pre-requisites

Acquiring Data with Scrapy

Creating an API using FastAPI

Dockerizing your API

Deploying your API on Google Cloud Run

Deploying your Image to Cloud Run

Front-end using Next.js

What are these __method__ in 🐍 you sssssay? Look no further!

How do Dunder Methods work?

Common Use Cases for Dunder Methods

Initialization

String Representation

Container Operations

Comparison

Arithmetic Operations

Best Practices for Using Dunder Methods

Conclusion

Automate your Hyper-V Windows through Docker Ansible

Pre-requisites

Hyper-V Virtual Switch

WinRM

Chocolatey

Ports

Host Setup

Playbook

Dockerfile

Build your Dockerfile

Troubleshooting Errors

Mastering Monad Design Patterns: Simplify Your Python Code and Boost Efficiency

Monad Design Pattern

Types of Monad

Maybe Monad

State Monad

Reader Monad

Writer Monad

IO Monad

Conclusion

What are these method in 🐍 you sssssay? Look no further!