DEV Community: sangarshanan

Curb your data with Huffman coding

sangarshanan — Sun, 27 Dec 2020 20:44:56 +0000

Transmitting messages meant converting the message into a sequence of symbols, The symbol or the sequence of symbols associated with a message is called the message code while the entire number of messages being transmitted might be called a message ensemble. The mutual agreement between the transmitter and the receiver about the meaning of the code for each message in the ensemble is called the ensemble code.

The concept of minimum redundancy code will be defined as the code used by a message ensemble consisting of a finite number of members N and for the given number of coding digits D yields the lowest possible average message length

This "Minimum Redundancy" or rather "Optimal" coding method had two rules

No two messages will consist of an identical arrangement of coding digits
No additional information would be needed to specify where the message code begins and ends once the starting point of a sequence of messages is known, this representation is called a prefix code.

The second rule or the prefix code is important because we are dealing with a variable-length encoding so there is a chance for ambiguity, i.e we might not know where the encoding of one symbol ends and the other begins, but designing out encoding as a prefix code means that the bit string representing some particular symbol is never a prefix of the bit string representing any other symbol which means no ambiguity whatsoever

01, 102, 111, 202 is a valid prefix message code for an ensemble of four members, which means that
1111022020101111102 can be broken up without any ambiguity into
111-102-202-01-01-111-102

However, if the message code includes 11, 111, 102, and 02 then a 11 would result in ambiguity cause we will not know whether it corresponds to 11 or to 111

This rule forms the basis of Huffman Code an optimal algorithm for lossless data compression.

He proved his encoding as The most optimal way of assigning ones/zeros to a single character in his paper title A Method for the Construction of Minimum-Redundancy Codes which is the shortest paper I have ever read ! so quick shoutout to my man David for that.

Huffman coding turned the existing Shannon coding upside down and used a Bottoms up Approach i.e Merging from the leaves to the root to generate the tree.

Let's suppose we need to encode A B C D and all of them equal probabilities of occurrence 1/4

We generate a tree out of the probabilities such that the second level nodes sum up to 1/2 and the root node sums up to 1. We then use this tree to paint all the left nodes 1 and right nodes 0. Then the encodings are generate

A - 11
B - 10
C - 01 
D - 00

Length = 2,2,2,2
Since the probabilities are the same the length of the code is always 2

Well now if we chose to point the left nodes as 0 and right nodes as 1
the coding we would be the reverse but will still end up being a prefix code and optimal.

Entropy of the code H = Σ Pi log(1/Pi) = 0.5+0.5+0.5+0.5 = 2

Expected code length = L = Σ pi Li = 4(1/4*2) = 2

                      Entropy (H)
Efficiency (E) =  ━━━━━━━━━━━━━━━━━━━━
                Expected Word length (L)

E = 2/2 = 1 ~ 100 percent :)

But this was easy cause the probabilities were all powers of 2 which might not always be the case

A - 0
B - 10
C - 110 
D - 1110 
E - 1111

Length = 1, 2, 3, 4, 4

In this above diagram, the probabilities are not powers of 2

Entropy of the code H = Σ Pi log(1/Pi) 
= 2(1/3*log_2 (3)) + 3(1/9 * log_2 (9))
= 2.113

Expected code length = L = Σ pi Li
= (1/3 * 1) + (1/3 * 2) +(1/9 * 3) +(1/9 * 4) +(1/9 * 4) 
= 2.222

Efficiency (E) = 2.113/ 2.222 = 0.95 ~ 95 Percent Efficiency

Here is the algorithm, For a given string of characters

Calculate the frequency of each character in the string.
Sort the characters in increasing order of the frequency
Make each unique character as a leaf node.
Create an empty node. Assign the minimum frequency as the left child and the second minimum frequency as the right child of the node.
Assign the empty node a value that is the sum of the above two minimum frequencies.
Repeat till we reach the maximum frequency character that would have a node one step away from the root node, which means the least number of bits to represent it.

The basic idea is to create a binary tree and operate on it in a bottoms-up manner so that the least two frequent characters are as far as possible from the root. In this way, the most frequent character gets the smallest code and the least frequent character gets the largest code.

Implementation

Huffman Coding can be implemented with a priority queue, Here is a rough pseudoscope

Count the occurrences of each character in the string
Place characters and counts into a priority queue
Use priority queue to create Huffman tree
Traverse the tree and assign right nodes 1 and left nodes 0
Encode the string with the binary mapping

The code is in the repo and has a Notebook that visualizes the generated huffman tree

https://github.com/sangarshanan/huffman-coding/blob/master/notebook.ipynb

Reference

Morse Code & The Beginning of an era

sangarshanan — Sun, 27 Dec 2020 20:38:34 +0000

Original post: https://sangarshanan.github.io/2020/12/20/morse-codes/

Humans are social animals, we rely on friends and family maybe now it's just for social juice but when we were still living in jungles staying together meant better higher chances of survival. We have evolved by cooperating as a civilization and we were able to continue that cooperation for generations with effective communication.

Humans are also expansionists we like to explore into the unknown whether it is across towns, cities, countries, planets, solar systems, galaxies, or even superclusters and as we humans explored places on earth and set up colonies it got a bit harder to communicate across colonies that were a bit far apart, we started by using natural resources for long term communication, smoke signals and drumbeats were common means of communications in almost all ancient civilizations. But this was limited by the weather and line of sight. We mitigated the line of sight issue by setting up communication channels on hilltops by this too was limited by the weather.

We then started developing infrastructure for long term communication, animals like pigeons, and even human messengers were trained to deliver messages across long distances. This was obviously was slow but kinda worked out for a while

Enter Electricity

Around 600 BC the Greek mathematician Thales of Miletus observed something interesting, certain stones like amber when rubbed against fur would showcase a strange force that could attract small fibers. Not only this it was also observed that when you try to touch this invisible force it would give you a shock and then immediately disappear. Also, not every object would allow the flow of this force, today we know these objects as conductors.

In 1752 Benjamin Franklin would discover that the tiny shocks we get from the force of friction between certain objects as discussed above were also the same force observed in lightning as it gave him the same shock, Not only this he was also able to store this so-called energy in a glass jar filled with water a protruding wire and discharge it all will by connecting wires or touching the jar. This was the very first Capacitor.

Once we discovered about free-flowing of electrons across opposing charges, its resulting magnetic field, and an effective way to store and transmit this electrical power over a conductor there was no turning back. Electricity kinda like fire was a quicksave in the species game.

The Electric Telegraph was a result of two developments in the field of electricity. First, in 1800, the Italian physicist Alessandro Volta invented the battery, which reliably stored electric current and allowed it to be used in a controlled environment. Second, in 1820, the Danish physicist Hans Christian Oersted demonstrated the connection between electricity and magnetism by deflecting a magnetic needle with an electric current.

In the 1830s, the British team of Cooke and Wheatstone developed a telegraph system, It was suddenly possible to communicate with electricity over long distances by just laying some wires.

But there was an issue they could only communicate through pulses of electricity. This where Samuel Morse came in he developed a sorta code, more particularly morse code to translate these pulses to numbers. He later sought help from Alfred Vail and Leonard Gale to expand its scope to letters and other characters.

The code assigned a sequence and short and long pulses to numbers and letters. These pulses can also be described as dots and dashes, For example, the code for A is 🔴 🚥 a dot followed by a dash. Each dot represents one time unit and a dash represents three time units, there is also a silence of 1 time unit in between every pulse.

Here is the interesting problem statement

Which specific sequence to dots and dashes do you assign to a character ?

To do this they actually studied the frequency of usage of every character in the English language and assigned easier sequences to the more commonly used characters. So E the most commonly used letter is represented with a single 🔴 dot.

Morse codes were a gamechanger and was a huge step towards enabling humans to communicate over really long distances like never before. Even now in maritime and a bunch of other places we use SOS as a signal for distress calls and the combination of these letters were chosen purely for their simplicity in morse code.

On an entirely different note, Samuel Morse was a painter by profession. The city of New York promised him a $1,000 commission to paint Marquis de Lafayette and while working on the painting he got a letter from a man on horseback and the letter said that his wife Lucretia had passed away and by the time he rushed back to see her one last time she was already buried, perhaps it was this heartbeat and pain caused by slow communication that gave us morse code.

Despite being a little hard and not intuitive to learn once mastered morse code is actually one of the easiest means of communication, It has been used by spies to deliver secret messages which to an unsuspecting onlooker might seem like weird blinking or tapping. There have also been multiple references to it in pop culture, more recently I saw one in the movie parasite where (spoiler alert) the man who lives inside the mansion communicates with its owner by banging his dead and switching the lights on and off in morse code.

Now, this seems all too familiar to like minds, morse code using two types of pulses short and long to encode information which is kinda what every computer ever does right ? so is this all just an elaborate ploy by the omnipresent 1 and 0s

So its all binary then ?

Ehhhhh not exactly, so the think is with morse code we have a gap of one time unit between dots and dashes, to separate letters we use the three time units and to separate words we use seven time units

So morse code is no exactly a binary system but a ternary system cause we also need to account for silence or gaps i.e time units when there are no pulses.

That's is not stopping us from representing this in binary

A dot is denoted by a 10 1 representing the time unit and 0 representing the end of the pulse
A dash is denoted by 110 where 111 represents 3 time units and it too ends with a 0 to denote the one time unit gap or the end of a pulse
A gap is denoted by 00 which is two time units of nothingness so when combined with either a dot or a dash mean a letter separator and 3 continuos gaps represent the end of the word

And there you have it !

|--------+------+--------+-----------|
| Letter | Code | Length | Frequency |
|--------+------+--------+-----------|
| E      | 🔴    |      1 |    12.49% |
| T      | 🚥    |      3 |     9.28% |
| A      | 🔴🚥   |      4 |     8.04% |
| O      | 🚥🚥🚥  |      9 |     7.64% |
| I      | 🔴   |      2 |     7.57% |
| N      | 🚥🔴   |      4 |     7.23% |
| S      | 🔴  |      3 |     6.51% |
| R      | 🔴🚥🔴  |      5 |     6.28% |
| H      | 🔴 |      4 |     5.05% |
| L      | 🔴🚥🔴 |      6 |     4.7% |
| D      | 🚥🔴  |      5 |     3.82% |
| C      | 🚥🔴🚥🔴 |      8 |     3.34% |
| U      | 🔴🚥  |      5 |     2.73% |
| M      | 🚥🚥   |      6 |     2.51% |
| F      | 🔴🚥🔴 |      6 |     2.40% |
| P      | 🔴🚥🚥🔴 |      8 |     2.14% |
| G      | 🚥🚥🔴  |      7 |     1.87% |
| W      | 🔴🚥🚥  |      7 |     1.68% |
| Y      | 🚥🔴🚥🚥 |     10 |     1.66% |
| B      | 🚥🔴 |      6 |     1.48% |
| V      | 🔴🚥 |      6 |     1.05% |
| K      | 🚥🔴🚥  |      7 |     0.54% |
| X      | 🚥🔴🚥 |      8 |     0.23% |
| J      | 🔴🚥🚥🚥 |     10 |     0.16% |
| Q      | 🚥🚥🔴🚥 |     10 |     0.12% |
| Z      | 🚥🚥🔴 |      8 |     0.09% |
|--------+------+--------+-----------|

The representations in morse code was designed so that the most frequently used letters have the shortest codes. In general, code length increases as frequency decreases.

This is actually a form of data compression and perhaps one of the earliest if not the first one.

More work on compression was brought by pioneers of information theory, In 1949 Claude Shannon and Robert Fano devised a systematic way to assign codewords based on probabilities of blocks

Even the morse code can be represented in graphical form, We have dots, dashes, word space and letter spaces and There are two states depending on whether or not a space was the last symbol transmitted. If so, then only a dot or a dash can be sent next and the state always changes. If not, any symbol can be transmitted and the state changes if a space is sent or remains the same.

This concept of Modelling Sequences of random events using states and transitions between them is called a Markov Chain

This markov chain in the image above was presented in Claude Shannon's paper titled The Mathematical Theory of Communication, the paper which laid the foundations of modern information theory.

It was also in this paper that Shannon introduced the Shannon-Fano coding, a technique for lossless data compression, this was preceded by a more optimal algorithm proposed by David Huffman in 1951, it was called Huffman encoding.

Ah yes back when naming things after yourself was cOoL

Reference material

https://www.history.com/topics/inventions/telegraph
https://www.youtube.com/watch?v=HY_OIwideLg of course it's a Vsauce video
https://www.khanacademy.org/computing/computer-science/informationtheory/

'Tis Build-a-Loadbalancer Day

sangarshanan — Sat, 17 Oct 2020 09:29:24 +0000

Introduction

Loadbalancing a way of effectively distributing incoming network traffic across a group of backend servers, so normally in services you have clients and you have server that process the request of your client and when we wanna scale we just add more servers for the clients to get served from, load balancer sits in the middle of this process by accepting requests from clients and routing it to the server, it also does this in a way that optimizes for speed, effective utilization and makes sure that things work even when one of the servers go down.

A very cool analogy I read for this is that of a support center, we are clients whereas the support staff are servers and when we call them the phone line is essentially a loadbalancer cause it has to route our call to one of the available support staff.

So now that we know the loadbalancer distributes the load, lets talk about how it does it

Round Robin

Here, we distribute the load sequentially among servers.
In this strategy we can either assume that all backends have the same processing power or use weights set relative to the processing power

Least connections

We prefer the server with the fewest connections to the client. The relative computing capacity of each server is also factored into determining which one has the least connections

Least time

Sends requests to the server selected by a formula that combines the
fastest response time and fewest active connections.

We can also define custom strategies, it all depends on what we wanna do which for me is mostly falling back to defaults.

Execution

So our end result should look something like this

https://hub.docker.com/r/strm/helloworld-http/

This is a docker-compose file with hello world servers and an nginx loadbalancer to distribute the requests, so we just need to replace that with the one we a new one.

Lets define the steps before actually building it

Our service is running Huzzah !!
Hoomans start calling our service, sometimes many and sometimes simultaneously
For every human that request to use our service we first receive the request start to determine the server that needs to serve the request, we do some magic() to get the server that needs to serve the traffic (We discussed some of those magics in the introduction)
We then serve the traffic through that server, Hooman is happy now !!

Lets actually name our loadbalancer, I hereby declare thee anubis, cause the god nicely balances souls and their worth

Now that the naming ceremony is over, To be able to do this "magic" we discussed earlier we need metadata on all the servers that are actually serving requests like if its alive, the computing capacity, number of connections to the client and maybe even more.

# Servers have the same computing capacity
class Server:
    url: str # localhost:8080
    alive: bool # Dead or Alive

In the end we will have a list of servers that can serve the client.

[Server1, Server2, Server3] <========>✨ Magic✨ <========> Route to Server1

Let's write this magic, well gonna call it Strategy for sanity

class Strategy:
    servers = [Server1, Server2, Server3]
    alive_servers = get_alive_servers(servers)
    def round_robin(self):
        robin_goes_round(self.alive_servers)

So the last step is to route use the client to the server returned by the output, to do this we are gonna use a simple reverse proxy, its kinda similar to a loadbalancer in the sense that it basically accepts a request from a client, forwards it to a server that can fulfill it, and returns the server’s response to the client. it often makes sense to deploy a reverse proxy when you have just one server cause then you have don't have the need for distributing requests.

https://www.nginx.com/resources/glossary/reverse-proxy-vs-load-balancer/

We can write a simple reverse proxy with requests

"""Reverse Proxy."""
import http.server  
import socketserver
from requests import Session
class Handler(http.server.BaseHTTPRequestHandler):
  session = Session()
  def do_GET(self):
    resp = session.get("www.google.com", allow_redirects=True)
    self.send_response(resp.status_code)
    self.send_header('Content-Length', len(resp.content))
    self.end_headers()
    self.wfile.write(resp.content)
# Fetches content from localhost:8000
httpd = socketserver.TCPServer(('', 9000), Handler)
httpd.serve_forever()

Run this and when you goto localhost:8080 you get redirected to google, Finally !! google running on my localhost. Screw the Internet. But this is a very stupid reverse proxy that only works on GET request and in pages with no dynamic elements

But then I had a question, why don't we just do a redirect how is it different from a reverse proxy. I looked online and found an explanation on stackoverflow (obviously). So with a redirect the server tells the client to look elsewhere for the resource. The client will be aware of this new location. The new location must be reachable from the client. A reverse proxy instead forwards the request of the client to some other location itself and sends the response from this location back to the client. This means that the client is not aware of the new location and that the new location does not need to be directly reachable by the client

Code is up on https://github.com/Sangarshanan/anubis

What I noticed immediately was that Python's http.serve is hella slow and when I tried opening multiple tabs it straight up died on me, which is kinda bad this is when I found out about ThreadedHTTPServer and ForkingMixIn. So we can either spawn multiple threads or processes to handle requests https://pymotw.com/2/BaseHTTPServer/index.html#module-BaseHTTPServer

Well its kinda better now but not awesome, we need something like https://github.com/http-party/http-server to take full advantage of asynchronous IO for concurrent handling of requests. We can only scale so much serialising requests and with python we will always be blocked by GIL so not the perfect candidate to do this but a fun exercise nonetheless.

Swiper

Realtime channels with FastAPI + Broadcaster

sangarshanan — Tue, 30 Jun 2020 19:06:13 +0000

Websockets are awesome, just learnt that before websockets people were polling (eww). Well actually polling helps when you know the exact time interval of your refresh But we are gonna be the cool realtime streamers.

Websockets allow full-duplex, bidirectional connections between a client and a server over the web with a single TCP connection (A protocol for sending and receiving packets of data across IPs in a reliable way, simply by acknowledgement)

We are gonna be using Fastapi and Starlette to define Websocket endpoint and Broadcaster to publish messages to this websocket. Others can just subscribe to the websocket endpoint to receive the published messages in real time

So this is what my API looks like, there is just one endpoint to publish messages to a channel (lebowski). Now the goal is to have a websocket that hoomans/robots can subscribe to follow the updates to lebowski in real time.



import json
from fastapi import FastAPI
from pydantic import BaseModel

class Publish(BaseModel):
    channel: str = "lebowski"
    message: str

app = FastAPI()

@app.post("/push")
async def push_message(publish: Publish):
    return Publish(channel =publish.channel, 
    message =json.dumps(publish.message))

We are gonna use broadcaster and Starlette to define a websocket endpoint

With starlette we can use WebSocketEndpoint and use it to create a WebsocketRoute, WebSocketEndpoint has three overridable methods for handling specific ASGI websocket message types: on_connect, on_receive, on_disconnect.



class Echo(WebSocketEndpoint):
    encoding = "text"
    async def on_connect(self, websocket):
        await websocket.accept()
    async def on_receive(self, websocket, data):
        await websocket.send_text(f"Message text was: {data}")
    async def on_disconnect(self, websocket, close_code):
        pass

routes = [WebSocketRoute("/ws", Echo)]

This is a simple echo websocket that send back what it receives. But I am not worrying about all this I am using broadcaster. With broadcaster we can define a simple broadcasting API onto a number of different backend services like Redis PUB/SUB, Kafka, Postgres LISTEN/NOTIFY or an in-memory one.



from starlette.concurrency import run_until_first_complete
from starlette.routing import WebSocketRoute

from broadcaster import Broadcast

broadcast = Broadcast("postgresql://postgres@localhost/test")

async def events_ws(websocket):
    await websocket.accept()
    await run_until_first_complete(
        (events_ws_receiver, {"websocket": websocket}),
        (events_ws_sender, {"websocket": websocket}),
    )


async def events_ws_receiver(websocket):
    async for message in websocket.iter_text():
        await broadcast.publish(channel="events", message=message)


async def events_ws_sender(websocket):
    async with broadcast.subscribe(channel="events") as subscriber:
        async for event in subscriber:
            await websocket.send_text(event.message)
routes = [
    WebSocketRoute("/events", events_ws, name="events_ws"),
]

We have defined two async functions to receive and publish messages and passed it to a starlette WebSocketRoute. Used Postgres as a backend for the broadcaster.

Now that we have defined a websocket route with broadcaster, lets just add it FastAPI and seal the deal



from fastapi import FastAPI

app = FastAPI(
    routes=routes, 
    on_startup=[broadcast.connect],
    on_shutdown=[broadcast.disconnect],
)

@app.post("/push")
async def push_message(publish: Publish):
    await broadcast.publish(publish.channel, 
    json.dumps(publish.message))
    return Publish(channel =publish.channel, 
    message =json.dumps(publish.message))

I have added the websocket route to the FastAPI app and am publishing to the channel on every call to the API.

Now to test things I am writing a dummy subscriber to listen to the broadcast (With the API is running on port 1234)



import json
import asyncio
import websockets
from websockets.exceptions import ConnectionClosed

async def connect(uri):
    async with websockets.connect(uri) as websocket:
        print("Connected..")
        while True:
            message = await websocket.recv()
            action = json.loads(message)
            print(action)

async def hello():
    uri = "ws://localhost:1234/events"
    try:
        await connect(uri)
    except ConnectionClosed:
        await asyncio.sleep(3)
        print("Not able to connect.. Retying in 3 seconds")
        await connect(uri)

asyncio.get_event_loop().run_until_complete(hello())

That's it :) In action below 👇

Ok lastly, FastAPI is awesome and I have been using a lot lately. It's powerful, easy to learn and the async community powering the whole ecosystem makes me wanna cry happy tears 😭

:wq

Behind your Web Application (WSGI + Web server)

sangarshanan — Sat, 16 Nov 2019 09:41:39 +0000

When I started writing simple flask web applications back in college I did not actually undertand what was happening in the back when I run the flask application

So when I run python app.py , this pops up

 * Serving Flask app "somename"
 * Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)

Now I am able to interact with the applications by sending requests and getting a responses but how am I able to that... I did not do anything to make that happen.. I just wrote the logic and Flask did it for me and I was able to get stuff done really fast

It's almost similar even when you look at other frameworks like Django

Django provides tons of stuff out of the box for you like routing, views, templates, authentication, middleware and a whole lot more with a very rich documentation so that you are always in your comfort zone. And with a huge community any issue I faced with Django was one stackoverflow answer away

These frameworks wrap everthing up so you just write in your logic and you are able to send requests and handle responses

By running your application and sending HTTP requests, you realise 3 things happening in the HTTP flow

You open a TCP connection: (In my case it's the localhost) The TCP connection is used to send a request, or several, and receive an answer. The client may open a new connection, reuse an existing connection, or open several TCP connections to the servers
Send an HTTP request message with your application

GET / HTTP/1.1
Host: developer.mozilla.org
Accept-Language: fr

You get a response from the server which you process with your application

HTTP/1.1 200 OK
Date: Sat, 09 Oct 2010 14:28:02 GMT
Server: Apache
Last-Modified: Tue, 01 Dec 2009 20:18:22 GMT
ETag: "51142bc1-7449-479b075b2891b"
Accept-Ranges: bytes
Content-Length: 29769
Content-Type: text/html

<!DOCTYPE html... (here comes the 29769 bytes of the requested web page)

Now you close or reuse the connection

But then how the hell do these python framework handle HTTP requests ? Cause it seems like magic... and I would love to learn a trick or two so let's break it down :D

Well maybe your site is static and you have all your files saved

Then for every request you already have corresponding static file as a reponse... awesome this is really fast, you could do tons of caching techniques and it's all really easy and straightforward but I mean it's static which means that you have to manually go and edit the html files every time

To solve this issue arose the Common gateway interface which invokes a script that dynamically generates the web page

Common Gateway Interface (CGI) offers a standard protocol for web servers to execute programs that execute like console applications (also called command-line interface programs) running on a server that generates web pages dynamically

The HTTP request that comes in breaks down into ENV variables which act as input and invokes a script which acts as a function and you get your output as an STDOUT... So if you just print hello world, it would be yout http response

As cool and simple as all this sounds, you can't afford to run the script every time you get a request

So we need a seperate out the web server and the python application

Enter WSGI

Or as the cool people call it Web server gateway interface

WSGI can be implemented with a simple function and every time there is a request we just call the function instaead of running an entire script

It is a specification that describes how a web server communicates with web applications, and how web applications can be chained together to process one request

WSGI is not a server, a python module, a framework, an API or any kind of software. It is just an interface specification by which server and application communicate

Well this means freedom cause we don't really need to worry about web servers like nginx or apache given that they understood this common interface and we could easily switch anytime we wanted to

Also there is a really cool article comparing these two web servers https://serverguy.com/comparison/apache-vs-nginx/

And if you wanted to build your own framework then you don't have to know a whole lot of HTTP and instead build something that implements this simple function (The one below is Gunicorn)

def app(environ, start_response):
    data = b"Hello, World!\n"
    start_response("200 OK", [
        ("Content-Type", "text/plain"),
        ("Content-Length", str(len(data)))
    ])
    return iter([data])

This function take two arguments

start_response: This is invoked with a status code, headers and an iterable data that you wanna send back (here it's the generic hello world)
environ: Well this is dictionary with some information like REQUEST_METHOD, PATH_INFO, SERVER_PROTOCOL etc

WSGI servers like Gunicorn are designed to handle many requests concurrently. Frameworks are not made to process thousands of requests and determine how to best route them from the server. They can also communicate with multiple web servers and keep multiple processes of the web application running. They can also async (Bjoern which is actually faster but not compatible with HTTP/1.1)

Gunicorn describes itself as a Pre-fork web server which means that a master creates forks which handle each request. A fork is a completely separate process and the "pre" part actually means that worker processes are created in advance, so that time is not wasted forking only when a worker is needed

A very commonly used combo includes Nginx + Gunicorn + Django where Gunicorn acts as a middleware between nginx and the Django

But now with Gunicorn why the hell do we need Nginx... seems unnecessary to me

Well with Gunicorn Ngnix acts as a reverse proxy server which can be used to provide load balancing, provide web acceleration through caching or compressing inbound and outbound data, and provide an extra layer of security by intercepting requests headed for back-end servers and also gunicorn is designed to be an application server that sits behind a reverse proxy server that handles load balancing, caching, and preventing direct access to internal resources.

Actually python comes with it's own built-in web server that provides standard GET and HEAD request handlers. You can use this to turn any directory in your system into your web server directory. I have actually used this to share code and stuff with my peers and it's pretty handy <3

Now to reaffirm the fact that i have learnt somethings let's write sum fresh code with Sockets... ughh...Now I'm glad I don't have to do this everytime I write a webapp

Now let's run this code to start a simple web server on port 8000

import socket
HOST = '' ## Symbolic name meaning all available interfaces
PORT = 8000 ## Port 8000

'''
AF_INET is an address family that is used to designate the 
type of addresses that your socket can communicate with (in this case, Internet Protocol v4 addresses). 

SOCK_STREAM is a constant indicating the type of socket (TCP),
'''
listen_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)

'''
Set the value of the given socket option
When retrieving a socket option, or setting it, 
you specify the option name as well as the level. When level = SOL_SOCKET, 
the item will be searched for in the socket itself.

For example, suppose we want to set the socket option 
to reuse the address to 1 (on/true),
we pass in the "level" SOL_SOCKET and the value we want it set to.

This will set the SO_REUSEADDR in my socket to 1.
'''

listen_socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)

## Bind to the host and port
listen_socket.bind((HOST, PORT))

## Listen to the host/port
listen_socket.listen(1)
print(f'Serving your very own HTTP on port {PORT} ...')

while True:
    client_connection, client_address = listen_socket.accept()
    """
    The recv() function receives data on a socket 
    with descriptor socket and stores it in a buffer. 
    """
    request_data = client_connection.recv(1024)
    print(request_data.decode('utf-8')) ## Decode the data assuming UTF=8 Endoding

    http_response = b"""\
HTTP/1.1 200 OK

Sample Response to be parsed!
"""
    ## Send a response and close the connection
    client_connection.sendall(http_response)
    client_connection.close()

Run the file.. I've saved it as webserver.py

$ python webserver.py                                                                                                                 
Serving your very own HTTP on port 8888 ...

Now let's

curl -v http://localhost:8000/


*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 8000 (#0)
> GET / HTTP/1.1
> Host: localhost:8000
> User-Agent: curl/7.58.0
> Accept: */*
> 
< HTTP/1.1 200 OK
* no chunk, no close, no size. Assume close to signal end
< 
Sample Response to be parsed!
* Closing connection 0

Maybe also try

telnet localhost 8000

Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.

HTTP/1.1 200 OK

Sample Response to be parsed!
Connection closed by foreign host.

There you go a simple web server to send requests and get responses

Now you can write your own web framework to interact with this web server and you would have essentially written your own application from scratch

Essentially you get parsed responses for processing / storing or whatever it is you are into

import requests
r = requests.get('http://127.0.0.1:8000/') 
print(r.content)

Different web frameworks communicate with web servers in different ways

In pyramid you have an application object that comes back from make

from pyramid.config import Configurator
from pyramid.response import Response


def hello_world(request):
    return Response(
        'Hello world from Pyramid!\n',
        content_type='text/plain',
    )

config = Configurator()
config.add_route('hello', '/hello')
config.add_view(hello_world, route_name='hello')
app = config.make_wsgi_app()

Flask is actually build around Werkzeug which is a WSGI web application library and Flask wraps Werkzeug, using it to handle the details of WSGI while providing more structure and patterns for defining powerful applications. In flask we actually define this as the app that we use as a decorator for our views and it's got routing and other functionalities built into it

from flask import Flask
from flask import Response
flask_app = Flask('flaskapp')


@flask_app.route('/hello')
def hello_world():
    return Response(
        'Hello world from Flask!\n',
        mimetype='text/plain'
    )

app = flask_app.wsgi_app

Contains stolen content from

https://github.com/rspivak/lsbaws
https://www.youtube.com/watch?v=WqrCnVAkLIo

You scrolled to the end :)