DEV Community: salem ododa

Writing Custom Log Handlers In Python

salem ododa — Wed, 02 Nov 2022 13:50:26 +0000

Have you ever wondered how logging services like logz .io or loggly .com ship your logs, onto their platform? For example logz .io has an opensource logging handler which they use to transport logs from your application to their platform where further analysis are carried out on these logs to enable you derive substantial insights on your application's performance. In this short article we are briefly going to cover the basics of writting user-defined log handlers using the python standard logging package.

You might be wondering why i chose to write about this rather "odd" topic, but the importance of a good logging system cannot be overstated, whether your application is a web application, mobile or even an operating system, logging enables you to derive useful informations on how your application is performing. A good logging system helps in catching errors as they happen, instead of relying on users to report them, the majority of whom simply wouldn’t or wouldn’t know how to do it.

The logging.StreamHandler

The python standard logging package provides a Handler class which basically defines how a particular log message is handled, we can configure a simple handler that prints log messages to the terminal like this:



LOGGING = {
    "version": 1, 
    "disable_existing_loggers": False,
    "formatters": {
        "stdformatter": {"format": "DateTime=%(asctime)s loglevel=%(levelname)-6s  %(funcName)s() L%(lineno)-4d %(message)s call_trace=%(pathname)s L%(lineno)-4d"},
    },
    "handlers": {
        "stdhandler": {
            "class": "logging.StreamHandler",
            "formatter": "stdformatter",
            'stream': 'ext://sys.stdout'
        },
    },
    "loggers" : {
        "root": {
            "handlers": ["stdhandler"], 
            "level": "DEBUG",
            "propagate": True
            }
        }
}

We can then initialize this configuration, in a log.py assuming the log.py file is the application we want to log



import logging
from logging.config import dictConfig


dictConfig(LOGGING)
logger = logging.getLogger()
logger.debug("logger started")

def logToConsole():
    logger.info("Application started, succesfully")
    logger.info("Logging to the console")
    logger.info("Finished logging to console!")

# call the function
logToConsole()

When we run the log.py file python3 log.py; we can see a similar output to this shown on the terminal

As you can see we've been able to configure a handler that logs directly to the console(stdout), keep in mind you don't need to create any congifuration to log to the terminal, since the StreamHandler is the default handler for the logging package. The logging package also comes with an array of already made Handlers you can use asides the StreamHandler, some of which are listed below:

FileHandler: which sends logging output to a disk file
NullHandler: It is essentially a ‘no-op’ handler for use by library developers. Which i've never had any reason to use :)
BaseRotatingHandler: which is basically another type of FileHandler for special use cases
SocketHandler: which sends logging output to a network socket.
SysLogHandler: which sends logging messages to a remote or local Unix syslog.
SMTPHandler: which sends log messages to an email address, using SMTP.
HTTPHandler: which supports sending logging messages to a web server, using either GET or POST semantics.

There are several handlers provided by the logging package that can be found here, some of which you might never have a use case for since they are built for very specific use cases.

Custom Handlers

Supposing we are building a cloud based logging as a service software like logz.io, where we'd like to store, retrieve and analyze our logs for better insights; python provides a low-level "api" to enable us transports logs from python applications easily, using the logging.Handler class. In the following example we are going to create a simple Custom Handler that writes to a ".txt" file. Obviously log messages are not written to txt files but this is to illustrate how to write Custom Log Handlers in a simple way.



class SpecialHandler(logging.Handler):

    def __init__(self )-> None:
        self.sender = LogSender()
        logging.Handler.__init__(self=self)

    def emit(self, record) -> None:
        self.sender.writeLog(record)

In the code above we declare a class named SpecialHandler, which inherits from the logging.Handler base class, this basically allows us to extend the base Handler so as to make it fit our use. We then declare the __init__ method and initiates the class which handles each log record as it is retrieved i.e LogSender, the emit(self, record) method is quite special as it responsible for sending each log record to the specified reciever.



 class LogSender:
    def __init__(self) -> None:
        pass

    def writeLog(self, msg: logging.LogRecord) -> None:
        with open("application.txt",'a',encoding = 'utf-8') as f:
            f.write(f"{msg} \n")

The LogSender class acts as the retriever of log records as they are sent from the emit method of the Handler's class; the LogSender class declares a writeLog method, which accepts any argument of type msg: logging.LogRecord. On reception of a record it opens a txt file application.txt in append mode and writes the log message to the file, adding a new line at the end of it.
This is basically all it takes to create a customHandler, we can configure our LOGGING Dictionary to recognize this new handler with a few lines of code, like so:



LOGGING = {
    "version": 1, 
    "disable_existing_loggers": False,
    "formatters": {
        "stdformatter": {"format": "DateTime=%(asctime)s loglevel=%(levelname)-6s  %(funcName)s() L%(lineno)-4d %(message)s call_trace=%(pathname)s L%(lineno)-4d"},
    },
    "handlers": {
        "stdhandler": {
            "class": "logging.StreamHandler",
            "formatter": "stdformatter",
            'stream': 'ext://sys.stdout'
        },
        "specialhandler":{
            "class" : "writer.SpecialHandler",
            "formatter": "stdformatter",
        }
    },
    "loggers" : {
        "root": {
            "handlers": ["stdhandler", "specialhandler"], 
            "level": "DEBUG",
            "propagate": True
            }
        }
}

Keep in mind, the writer.SpecialHandler is our custom handler class written in a writer.py file. To initialize this changes we can import the writer.py file into our log.py file and load the logging configuration into dictConfig again, but this time with a custom handler that writes to our application.txt file.



import logging
from logging.config import dictConfig
import writer #writer.py contains our SpecialHandler and LogSender class respectively

LOGGING = {
    "version": 1, 
    "disable_existing_loggers": False,
    "formatters": {
        "stdformatter": {"format": "DateTime=%(asctime)s loglevel=%(levelname)-6s  %(funcName)s() L%(lineno)-4d %(message)s call_trace=%(pathname)s L%(lineno)-4d"},
    },
    "handlers": {
        "stdhandler": {
            "class": "logging.StreamHandler",
            "formatter": "stdformatter",
            'stream': 'ext://sys.stdout'
        },
        "specialhandler":{
            "class" : "writer.SpecialHandler",
            "formatter": "stdformatter",
        }
    },
    "loggers" : {
        "root": {
            "handlers": ["stdhandler", "specialhandler"], 
            "level": "DEBUG",
            "propagate": True
            }
        }
}

dictConfig(LOGGING)
logger = logging.getLogger()
logger.debug("logger started")

def logToConsole():
    logger.info("Application started, succesfully")
    logger.info("Logging to the console")
    logger.info("Finished logging to console!")

# call the function
logToConsole()

When we run python3 log.py we can see an application.txt file is created within the current directory, containing all the log records.

Conclusion

Logging applications can save us hours in debugging hidden bugs within our application, as well as enabling to derive useful insights about the performance, operatability and needs of applications without any hassles. In this short article, we've seen how to write basic custom log handlers for our applications, extending to fit our specific needs.

Let me know what you think about this article and possibly any amendments that can be made to improve it's user experience, thanks for reading and have a great time!

message brokers, a brief walk-through

salem ododa — Mon, 12 Sep 2022 17:30:23 +0000

In an ever changing world of software development,utilizing microservices and distributed systems is the de facto standard for architecting software solutions, there is always a need to employ efficient means of message transfer between these various microservices.
One of the fundamental aspect of developing these services infact, is to incorporate an Inter-service communication between them. And although there are numerous methods of approaching this; Message Brokers provide an efficient, scalable and cheaper means of data sharing betwixt these services.
In this article we are briefly going to uncover the basics of message brokers, and build a distributed alert system that publishes alert messages to listed email addresses and a twitter account; using fastapi golang/gin and rabbitmQ.

Message Brokers

A broker in layman's term is someone who negotiates/arranges a transaction between two entities (be it people, businesses e.t.c) similarly; Message brokers are an inter-service communication technology, that utilizes standard messaging protocols to promote rapid development, and data-sharing between distributed applications. Message brokers provide data-sharing, data-marshaling, and data persistences features to traditional middleware applications out of the box, and they as well guarantee the transfer of messages with low latency. They ensure delivery of messages even when a consumer(client) is inactive, this enhances reliability/uptime of application's flow since each service is independent. A typical case for example is a service responsible for sending emails to recently signed up users, now assuming the service went down at the time a user is being registered, the downtime doesn't interupt the user's experience since the native backend only has to publish to the sign-up queue, which the consuming application can process from whenever it is restored.

Publishers, Consumers and Queues

Message brokers primarily rely on three fundamental concepts for message transfer, they are Publishers, Consumers and Queues and although they might differ in name and behaviour pending on the message brokers you use, they can still be used interchangably(or not) with Producers, Subscribers and Channels; So lets have a brief walkthrough of each of these concepts.

Publishers: In messaging, publishers (or producers) represents an application that sends/publishes messages to a message broker instance. Depending on the protocol, the publisher sends the message to a broker which further routes the message to the approriate queue/channel.

Consumers: consumers (or subscribers) represents application that readily listens on a queue or channel for new messages from the publisher. In order to consume messages, a consumer has to be registered to a queue.

Queues: queues as we know them are data structures that provides two kind of operations; enqueue whereby data is enterred into the queue through the rear or tail, and dequeue whereby data is removed from the queue through the front or head. In message brokers, queues are referenced via their names.

Data Sharing

Since the essence of this article revolves around data tranfer, i'd quickly like to deviate your attention to commonly employed design practices when defining how applications should communicate with each other.
In the process of building a distributed system where message sharing is priority, you should be able to answer the following questions:-

What data does your services need to share?
What service(s) are the primary providers of the data, can they be identified?
Does the nature of the data imply the introduction of a third-party software to share it?

Answering these questions are quintessential to the development of solid distributed ecosystems. When we start to write the codes for this article, we'd incoporate this services and see if our solution answer these questions.

When and Why Should We Use Message Brokers?

Unlike the typical client-server paradigm, where a client (eg. web browser) sends a requests to a server, and blocks till it gets a response back from the server, message brokers allows a client to continue with other operations, whilst it processes the former requests. This is incredible because it saves time, and enhances the user's experience; Imagine a scenerio where you have to successfully send a user their registration email, before redirecting them to their profile or the login page, the time spent waiting on the email service would imply a very poor user experience.
Suffice to say it's best to use message brokers whenever you're building something whose average processing time spans beyond the traditional http requests/responses duration. Keep in mind time isn't the only yardstick to consider when trying out message brokers, but it is one to always have in mind.

Building an alert dispatch service

Up until this point we've seen ideal situations where we would want to build our applications using message brokers. This section aims at furthering your practical knowledge on the use of message brokers, and we are going to be building an Alert Dispatch Service, which is essentially an API that recieves a json post request:

the API then serializes this json and publishes it to a message broker (RabbitMq) which in turn broadcasts it to any consumer listening on the queue, the consuming service serializes the recieved data and publishes the message to a twitter account and a list of email addresses.
The Api is written in python fastapi framework and the consumer is written in golang, this example elaborates the use cases of message brokers as we can see how we've enabled communication between two independent microservices, written in entirely different languages; whether this service has a use case in real world scenerios isn't considered as i only thought of a perfect example for using message brokers, which would be straight to beginners.

Fastapi Service (Publisher)

As i mentioned earlier, the Api and publisher service is written with python's modern web api framework fastapi, it is quite easy to set up. First we need to set up our virtual environment, do this by running python -m venv <name_of_your_virtual_env> you can activate your virtual environment on windows by running

<name_of_venv>/bin/activate.bat or source <name_of_venv>/bin/activate on linux/macOs. Next we need to install fastapi by running:
pip install fastapi
this installs fastapi, alongside other dependencies it needs to function properly. Then we need to install uvicorn which will act as the ASGI (Asynchronous server gateway interface) server for our application; to do this run pip install uvicorn. Once you've completed this setups, we can move on to writing our codes;

From the code above,on line one and two we imported the FastAPI class from fastapi and also the BaseModel class from pydantic
we then instantiate the fastapi class with a variable called app on line 4, we will see the use of this app variable shortly.On line 8 we create an Alert class which is the schema of the json data our api expects, and within the class we declare two fields, topic which is the topic of the alert and message which is the alert's content. Notice how our alert class inherits the pydantic.BaseModel class.

Next we write our api function on line 14, the app variable defines the http method required by the function and also declares it's route, then the function basically checks if an http request body contain's an alert type, notice how all the Serialization is handled neatly by Fastapi behind the scenes; if the alert is valid we calls our rabbitmQ publisher function else, we throw an error at the client. The Publisher function:

For the publisher, we import two packages Pika which is a python library for communicating with rabbitmQ and json to enable us serialize the alert instance. Then we create a connection to our rabbitmQ instance, and declare a Queue named alert_queue, when you declare a queue, rabbitmQ checks to see if the queue already exists, if it doesn't it creates the queue else it just goes ahead to publish the message. Next we declare a dictionary, to map our Alert's value, this is to enable us send messages in json format to our message broker, which can be delivered and serialized by our consumer service. Finally we publish the message to the alert_queue and return from the function body.

Golang Service (Consumer)

The consumer service is written in go, to elaborate the essence of this article i.e using message brokers to transfer messages between different, independent microservices.
Unfortunately i won't be able to detail the code bit by bit since this article is becoming too lengthy, but it's quite straight-forward if you write Go regularly;

Basically this code, connects to our rabbitmQ using the streadway/amqp package which is the golang standard library for communicating with rabbitmQ, next we declare a unique channel to enable us pass across message to a specific queue. Then we call the Consume function which takes in a parameter of *amqp.Channel which is the channel we created previously.

So the Consume function basically reads all the messages currently available in the queue and serializes them to an alert variable, then it sends a tweet to a specified twitter account with the alert's message as the tweet's content, it also sends an email with the alert's topic as subject and the alert's message as the email's content to my email address. I'm obviously not gonna go into the details of how to programmatically create a tweet or send an email. But if you're interested in learning how to send Emails with golang, you should definitely take look at my article Sending E-mails with Go. All the sample codes written here are publicly available on this github repository with instructions to run them.

Conclusion

Being able to effectively exchange messages across multiple decoupled software components is vital for every distributed ecosystem; message brokers provides an easy, scalable and reliable means of achieving this.
In this article we've briefly covered the basics of publishing and consuming messages with rabbitmq, we went further to build an alert dispatch service that distributes alerts to listed email addresses and a twitter account, using python and golang.

Let me know what you think about this article and possibly any amendments that can be made to improve it's user experience, thanks for reading and have a great time!

Sending E-mails with Go.

salem ododa — Sun, 21 Aug 2022 22:05:57 +0000

Sending email notifications are essential for effective communication between users and service providers. Beneath this notable piece of technology lies several layers of abstractions which might require more than a blog post to have a clear or complete idea of; which in retrospect isn't the aim of this post.
In this short article i will quickly try to uncover some issues i faced whilst trying to implement the email notifications for a service i was building at work.
By default using the standard SMTP package provided by golang i.e net/smtp suffices for most use cases, but depending on the Email service provider you're using, you might experience some bottle necks using the smtp package.
We use Webmail where i work and the first time i tried using net/smtp with it, i experienced some irregularities, for instance my client was successfully able to authenticate my credentials, and send the mail message using the smtp.SendMail method, with the sample code below:

package mails
import (
         "log"
         "net/smtp"
)

var (
    From_mail     = os.Getenv("FROM_MAIL")
    Mail_password = os.Getenv("MAIL_PASSWORD")
    SMTP_Host     = os.Getenv("HOST")
)
func SendMail(msg string, recipient []string) {
    // Message.
    message := []byte("This is a test email message.")

    // Authentication.
    auth := smtp.PlainAuth("", From_mail, Mail_password, SMTP_Host)

    fmt.Println(auth)
    // Sending email.
    if err := smtp.SendMail(fmt.Sprintf("%s:%d", SMTP_Host, 587), auth, From_mail, recipient, message); err != nil {
        log.Printf("Error sending mail %v", err)
        return
    }

    log.Println("Email Sent Successfully!")
}

but on the recieving end no mails were delivered.

After series of relentless searching for a reasonable explanations to why i was experiencing such, i got to learn some email service providers like the one i used preferred sending mails over port 465 requiring an ssl connection from the very beginning (without starttls), as compared to the standard 587 which uses plain TCP to send the mail traffic to the server with subsequent calls using Starttls.
There's been argument over these ports for a long time, but generally sending mails over port 587 is more recommended as it is provides a much secured transmission layer compared to port 465. How these protocols are implemented depends mainly on the service providers you're using and the network protocol they choose for each port, for example Gmail uses SSL for the SMTP server on port 465 and TLS for port 587, how SSL and TLS are implemented and used for secure data tranmission on the internet is beyond the scope of this article, but you can read more about these protocols here.

Luckily i found this github "gist", which solved the problem we've been trying to figure out. I refactored the code to fit my use case, but you can get the actual solution from the "gist" link above, now unto the actual code:

package main

import (
    "crypto/tls"
    "fmt"
    "log"
    "net/mail"
    "net/smtp"
    "os"
    "sync"
)

var (
    From_mail     = os.Getenv("FROM_MAIL")
    Mail_password = os.Getenv("MAIL_PASSWORD")
    SMTP_Host     = os.Getenv("HOST")
    Mail_subject  string
    Mail_body     string

    from      *mail.Address
    auth      smtp.Auth
    tlsconfig *tls.Config
    mailwg    sync.WaitGroup
)

type Container struct {
    m       sync.Mutex
    Headers map[string]string
}

func NewContainer() *Container {
    return &Container{
        Headers: make(map[string]string),
    }
}

func main() {

    SendMails("subject", "article message", []string{"testuser1@gmail.com", "testuser2@gmail.com"})
}

func init() {
    from = &mail.Address{Name: "Test-mail", Address: From_mail}
    auth = smtp.PlainAuth("", From_mail, Mail_password, SMTP_Host)
    tlsconfig = &tls.Config{
        InsecureSkipVerify: true,
        ServerName:         SMTP_Host,
    }
}

func SendSSLMail(subject, msg string, recipient string) {
    to := mail.Address{Name: "", Address: recipient}

    Mail_subject = subject
    Mail_body = msg

    // initialize new container object
    container := NewContainer()
    // call mutex.lock to avoid multiple writes to
    // one header instance from running goroutines
    container.m.Lock()
    container.Headers["From"] = from.String()
    container.Headers["To"] = to.String()
    container.Headers["Subject"] = Mail_subject
    // unlock mutex after function returns
    defer container.m.Unlock()

    // Setup message
    message := ""
    for k, v := range container.Headers {
        message += fmt.Sprintf("%s: %s\r\n", k, v)
    }
    message += "\r\n" + Mail_body

    conn, err := tls.Dial("tcp", fmt.Sprintf("%s:%d", SMTP_Host, 465), tlsconfig)
    if err != nil {
        log.Printf("Error sending mail %v", err)
        return
    }

    c, err := smtp.NewClient(conn, SMTP_Host)
    if err != nil {
        log.Printf("Error sending mail %v", err)
        return
    }

    // Auth
    if err = c.Auth(auth); err != nil {
        log.Printf("Error sending mail %v", err)
        return
    }

    // To && From
    if err = c.Mail(from.Address); err != nil {
        log.Printf("Error sending mail %v", err)
        return
    }

    if err = c.Rcpt(to.Address); err != nil {
        log.Printf("Error sending mail %v", err)
        return
    }

    // Data
    w, err := c.Data()
    if err != nil {
        log.Printf("Error sending mail %v", err)
        return
    }

    _, err = w.Write([]byte(message))
    if err != nil {
        log.Printf("Error sending mail %v", err)
        return
    }

    err = w.Close()
    if err != nil {
        log.Printf("Error sending mail %v", err)
        return
    }

    if err = c.Quit(); err != nil {
        return
    }
}

// Concurrently sending mails to multiple recipients
func SendMails(subject, msg string, recipients []string) {
    mailwg.Add(len(recipients))
    for _, v := range recipients {
        go func(recipient string) {
            defer mailwg.Done()
            SendSSLMail(subject, msg, recipient)
        }(v)
    }
    mailwg.Wait()
}

In the code above we declared variables to hold our smtp credentials, then we declared a struct named Container which basically contains a mutex m field, that essentially allows us to avoid Race Conditions i.e when different threads/goroutines try to access and mutate the state of a resource at the same time, this is to lock the Headers field which is the actual email header containing meta-data of each email to be sent.
The init() function allows us to initialize some of the resource to be used before the function is called, you can see we initialize the auth variable with our mail credentials, which returns an smtp.Auth type; we then move further to set up tlsconfig which basically determines how the client and server should handle data transfer.
The SendMails function lets us send mails concurrently to avoid calling the SendSSLMail function for each number of recipients we want to distribute the mail to, I believe the SendSSLMail function body is quite straight-forward and doesn't require much explanation, as at the time of publishing this article, this service is being used so i'm pretty sure it is stable enough to be used by anyone.

Conclusion

Sending mails are very essential to the growth of any business, as they allow direct communication with the customers. Knowing how to effectively build stable email services is quitessential to every software developer out there. I hope this article helps solve any challenge you might encounter while writing mail services with Go.

Let me know what you think about this article and possibly any amendments that could be made to improve it's user experience, thanks for reading and have a great time!

Working with Shell Alias and Functions in linux

salem ododa — Sat, 26 Mar 2022 10:41:00 +0000

Alias helps to pass commands to the linux terminal to run a specific instruction.
The linux terminal is very resourceful, and as such utilizing Alias could be a major boost to your productivity as a developer.
As a "lazy" dev like myself :)) i use aliases for any command that spans beyond 3 words, don't blame me, cos why should i bother typing a long array of words when i could pass in two letters that represents the same command set.
Now you might not be a fan of shortening commands when writing codes, but i find alias quite helpful when you're trying to remember long commands, as they allow you map short, self-made commands rather than the default that comes with the compiler/interpreter or whatever interface you're working on.
For example this is an alias i use whenever i'm trying to run a unittest in my python projects:
alias test="python -m unittest"
you can see it is much more shorter and saves me the stress of typing everything out repeatedly.

Creating Alias

Creating an alias is pretty straight-forward, normally user-defined alias can be created in either the .bashrc or .bash_aliases files, i'd personally recommend using the .bashrc file for easy organization and referencing.
The syntax for adding an alias is:
sudo nano ~/.bashrc
alias name=command
to create an alias you have to indicate with the alias command.

Flags when working with alias

Flags are optional arguments passed along side a command to perform specific purposes, they're not unique to only alias as they are used with almost every cli tool.
For instance,you can print out a list of file within a directory using the ls command, additionally you can use

ls -a
ls -la
to get similar but specific outputs.
Flags are usually accompanied with a - or -- symbol, to indicate they are optional.
There are two common flags when working with alias in linux, one is the:

-p flag: that prints out all user-defined alias in a reuseable format. A common use case is running alias -p which yields

alias pip='pip3'
alias pip2='pip'
alias push='git push -u origin master'
alias py='python3'
alias python2='python'
alias run='go run'
alias runapp='python manage.py runserver'
alias test='python -m unittest'
alias weather='curl wttr.in'

on my pc obviously :)
Another common flag is the;

--help flag: that returns a help page, instructing how to navigate using alias. Running alias --help yields

$ alias --help
alias: alias [-p] [name[=value] ... ]
    Define or display aliases.

    Without arguments, `alias' prints the list of aliases in the reusable
    form `alias NAME=VALUE' on standard output.

    Otherwise, an alias is defined for each NAME whose VALUE is given.
    A trailing space in VALUE causes the next word to be checked for
    alias substitution when the alias is expanded.

    Options:
      -p    print all defined aliases in a reusable format

    Exit Status:
    alias returns true unless a NAME is supplied for which no alias has been defined.

Unaliasing

Unaliasing is basically removing a defined alias, so that a command can't be reached via it's alias.
You can unalias a command by running:
unalias commandName
for instance i can unalias push in the above alias list by running:
unalias push
when i run push on the terminal again, i get:

$ push
bash: push: command not found

obviously this isn't permanent as once the *.bashrc *_file is reloaded into the terminal memory again, all previously removed commands via **_unalias** are available again.
To permanently delete an alias, you would have to edit the .bashrc file and erase that specific command.

Adding Parameters to Alias with Shell Functions

I know! that dosen't sound right, alias do not accept parameters aside their regular pre-defined flags.
So how do we go about a case where we'd like to pass arguments to our alias?
For instance i usually prefer using an alias
build NameOfExedir/NameOfExe to build golang executables rather than typing go build -o NameOfExedir/NameOfExe -v .
so i'd rather do
build bin/myproject instead of go build -o bin/myproject -v .
In situations like this we apply the use of functions, although functions might be out of context for our current topic of discussion, i find them very useful when working with aliases.

Functions are commands in linux which are used to create functions or methods. Just like in many programming languages, functions are re-usable chunks of codes that carry out an instruction set.
There are two ways we can declare functions in .bashrc files,

Using a Function keyword: functions can be defined using the function keyword before specifying the function's name.
Syntax
function build { }
Using parenthesis: functions can also be defined using the function's name with parenthesis next to it.
Syntax
build() { }

Run help function for more information on working with shell functions.

Back to the above gobuild question;
we could create an alias that runs the go build functionality and then uses a function to parse the additional parameter, i.e the location the executable would reside. So:

alias gobuild="go build -o "
and then:

build(){
        gobuild "$1" -v .
        echo "Finished building project executable"
}

so when i run, build bin/myproject
i get:

$ build bin/myproject
Finished building project executable

and then when i run ls | grep 'bin'

$ ls | grep 'bin'
bin

you can see there's a binary folder containing my project's executable, all done using a shell function that calls an alias to run a go build command within a directory of my choosing. This is resourceful as i can now use this build function in whichever project i'd like to workon.

Conclusion

In this short article we've seen how to optimize our development process by using aliases and functions to shorten long terminal commands.
I hope you enjoyed the article and in the meantime, let me know what you think about the article and possibly any amendments that could be made to improve it's user experience, thanks for reading and have a great time!

Replacing django's default Statreloader with Facebook's Watchman

salem ododa — Fri, 25 Mar 2022 14:23:19 +0000

Ever thought what went on behind the scene when you run
python manage.py runserver
in your project's base directory during development?
Normally you get a familiar output:

$ python manage.py runserver
Watching for file changes with StatReloader
Performing system checks...

System check identified no issues (0 silenced).
March 24, 2022 - 11:54:06
Django version 3.2.11, using settings 'wallet.settings'
Starting development server at http://127.0.0.1:8000/

As indicated at the top of the terminal output, you see
Watching for file changes with StatReloader
Statreloader which is the default "file-watcher/file-reloader" for auto-reloading the web server whenever there's a change in .py files during development is no more than a simple class which works by running a loop that checks for file changes every 1 second, this can be quite in-efficient when working on larger projects.
Now except you're experimenting other alternatives to the default Statreloader or working on a quite large codebase where efficiency and resource optimization(like battery usage, memory) is a big deal, i'd advice you to stick with statreloader as it doesn't require any pre-setup to work with.

Introducing Watchman

Watchman is a file watching service introduced by Facebook for watching file changes, it relies on the operating system to send signals whenever a change is made to a file.
Django get notified when there's been a recent change to a file and quickly reload's the server to include the recently made changes.
This makes watchman a better alternative as it doesn't keep any process running when there isn't a file change, which leads to better CPU resource optimization.

Installing Watchman

Watchman relies on the operating system it's running on to get file change notification signals, so it's most likely user experience would defer based on your choice of OS.

Personally i use a linux/debian distro and i found this amazing article on setting up Watchman on your linux pc:
Setting up watchman on linux
If you're running a windows/Mac Os, you can setup Watchman via the instructions specified on their official docs page: Setup Watchman on windows or Macos
it's pretty straightforward setting it up with chocolatey on windows.

Setting up Django to use WatchmanReloader

Getting django to work with watchman is very simple, first you have to cd into your base directory and install watchman client/wrapper for python i.e pywatchman.

activate your virtual environment
$ source walletenv/bin/activate
install pywatchman
$ pip install pywatchman

rerun your development server
$ python3 manage.py runserver

you should see that django is now using WatchmanReloader

Watching for file changes with WatchmanReloader
Performing system checks...

System check identified no issues (0 silenced).
March 25, 2022 - 13:53:33
Django version 3.2.11, using settings 'wallet.settings'
Starting development server at http://127.0.0.1:8000/
Quit the server with CONTROL-C.

If you're building an application that has setups that aren't particularly related to what you're currently working on, you can configure watchman to ignore notifications from these particular dirs/files by simply including a .watchmanconfig file in the same directory where your manage.py file is.
touch .watchmanconfig
nano .watchmanconfig
{ "ignore_dirs": ["DIRECTORY_YOU'd_WANT_TO_IGNORE_CHANGES_FROM"] }

Conclusion

Performance is a vital yardstick when building application that would scale, optimizing development process is also a very significant factor to this.
In this short article we've seen how to optimize our development process by using a more faster and memory friendly file watching service.
I hope you enjoyed the article and in the meantime, let me know what you think about the article and possibly any amendments that could be made to improve it's user experience, thanks for reading and have a great time!

Crafting Middlewares In Go

salem ododa — Thu, 03 Mar 2022 19:07:58 +0000

When you make a request (as a client) to a server, series of processing are carried out on the said request, before you receive a responses. Some of these processing are done by the base application you are requesting the resources from, but it is most likely that during the request-response cycle several other applications might have interacted and made changes to your requests/response before you get a final result. These third-party applications are generally known as middlewares.

If you're a python/javascript developer you'd already be familiar with the term Middlewares as you probably get to interact with them when building apis or so.
Middlewares are basically decorated functions that wraps request handlers and provide additonal functionalities during the requests/response life cycle.
In django middlewares are added in a particular order into a list, which allows django to process them in a Top-Buttom approach when processing a requests,

and a Buttom-Top approach when returning a response.
In go on the other-hand, you can add a middleware by simply creating a function that recieves a handler as argument and returns it back to the http.handler method as an argument; what do i mean? let's experiment with a simple example:

package main

import (
    "fmt"
    "log"
    "net/http"
    "time"
)

func main() {
    mux := http.NewServeMux()
    indexHandler := http.HandlerFunc(index)
    server := &http.Server{
        Addr:    ":8000",
        Handler: mux,
    }
    mux.Handle("/index", IndexLogger(indexHandler))

    log.Printf("Serving http on %s", server.Addr)
    log.Fatal(server.ListenAndServe())
}

func IndexLogger(next http.Handler) http.Handler {

    return http.HandlerFunc(func(rw http.ResponseWriter, r *http.Request) {
        startTime := time.Now()
        log.Printf("%s at %s", r.Method, r.URL.Path)
        next.ServeHTTP(rw, r)
        log.Printf("Finished %s in %v", r.URL.Path, time.Since(startTime))
    })
}

func index(w http.ResponseWriter, r *http.Request) {

    fmt.Fprintf(w, "hello welcome to the index page")
}

In this simple example we've created a basic RequestLogger middleware that logs whenever we request for the index Uri.
when we run
go run main.go
and navigate to localhost:8000\index, you should see a similar output to:

$ go run main.go
2022/03/03 13:28:00 Serving http on :8000
2022/03/03 13:28:07 GET at /welcome
2022/03/03 13:28:07 Finished /welcome in 34.394µs

Note ServeHTTP calls the Handler's function within the current middleware's method body.
Exciting right? the functionalities we could include to our api-services with this is limitless, say we want to include this logger service to every registered route on our application, we would simply just wrap our desired function's handler with our indexLogger function, e.g:

func welcome(rw http.ResponseWriter, r *http.Request) {
    fmt.Fprintf(rw, "Welcome to the welcome page")
}

// edit main function to include the welcome handler

WelcomeHandler := http.HandlerFunc(welcome)
mux.Handle("/welcome", IndexLogger(WelcomeHandler))

when we re-run our local server and navigate to /welcome we'd get a logged response on our terminal, indicating that the IndexLogger functionality was also added to the welcome URL Path, this is cool because not only does it allows similar behaviours within specific uri paths, but because it also supports the DRY(Don't Repeat Yourself) principle of Programming.

Chaining multiple http middlewares

Since in go a http middleware recieves a httpHandler and also returns a httpHandler, we can easily chain multiple middlewares that call each other before calling the base application/function.

For example, let's write another middleware function to check the Requests method of a URI path and return a method allowed
if request method is GET or not allowed if otherwise.

func CheckRequestMethod(next http.Handler) http.Handler {

return http.HandlerFunc(func(rw http.ResponseWriter, r *http.Request) {
        if r.Method != "GET" {
            log.Printf("Only GET requests are accepted on the path!")
            next.ServeHTTP(rw, r)
        fmt.Fprintf(rw, "Method not allowed!")
        } else {
        log.Printf("Method allowed")
        next.ServeHTTP(rw, r)
        fmt.Fprintf(rw, "Your request is valid!")
             log.Printf("Finished checking......")
        }
    })
}


// edit main function to wrap CheckRequestMethod on one of our registered routes.

mux.Handle("/welcome", CheckRequestMethod(IndexLogger(WelcomeHandler)))

when we re-run our local server and navigate to /welcome path, we'd notice that both CheckRequestMethod and IndexLogger have taken turns to act on our welcomeHandler before returning a response.

$ go run main.go
2022/03/03 19:57:29 Serving http on :8000
2022/03/03 19:57:41 Method allowed
2022/03/03 19:57:41 GET at /welcome
2022/03/03 19:57:41 Finished /welcome in 30.615µs
2022/03/03 19:57:41 Finished checking......

Flow of Chained Http Middlewares

Consider the following code:

func ProgramFlow() {

    fmt.Println("First function body")
    func() {
        fmt.Println("Second function body")
        func() {
            fmt.Println("Third Function body")
            time.Sleep(3 * time.Second)
            fmt.Println("Finished third function body")
        }()
        fmt.Println("Finished second function body")
    }()
    fmt.Println("Finished first function body!")

}

When we call ProgramFlow() in our main Function, we get an output similar to

$ run main.go
First function body
Second function body
Third Function body
Finished third function body
Finished second function body
Finished first function body!

This is because in go each function is placed in it's own goroutine and it's output is first stored in a WaitGroup till all it's corresponding functions have returned before the it finally returns an output, similarly nested middlewares are treated in the same manner. When you navigate to \welcome;

The webserver passes the control flow to CheckRequestMethod which logs a response depending on the requests method before executing the next handler.
The next handler which is also a middleware logs the requests detail before executing the next handler.
The main application is run and returns the normal response
The IndexLogger logs it's final response
The CheckRequestMethod logs it's final response

Third-Party MiddleWares in go

So far so good, we've been able to understand how middlewares work in go and also how you can write middlewares in your applications where need be. There are tons of libraries that implement several middleware components out-of-the-box for you, when building applications. So many have been created and are actively maintained to fit the best use cases, explore http://www.gorillatoolkit.org/ to see some already built handlers you could use for your next big projects.

Chaining Middlewares with the Alice Package

Alice is a go library that can be used to chain mulitple middleware functions and application handler together.

with Alice we do not need to pass each middleware handler as a parameter to the next, like we did in the above examples:

mux.Handle("/welcome",CheckRequestMethod(IndexLogger(WelcomeHandler)))

instead we could do

import "github.com/justinas/alice"

// edit our main function to create a variable that stores all our middlewares 

allhandlers := alice.New(CheckRequestMethod, IndexLogger)

// add middlewares to registered route

mux.Handle("/welcome", allhandlers.Then(WelcomeHandler))

and just like we've chained all our middlewares with alice.

Conclusion

Middlewares are a great way to write re-usable pieces of code that can add great functionalities to whatever we are building, we only scratched the surface of what we could do with middlewares and in later articles we'd go into details on how to build applicable middlewares for our applications.
Meanwhile i'd suggest you go through more articles/videos on the subject matter to gain deeper understanding on how to go around building middlewares.

Let me know what you think about this article and possibly any amendments that could be made to improve it's user experience, thanks for reading and have a great time!

The unicode encoding system

salem ododa — Fri, 11 Feb 2022 17:53:27 +0000

Have you ever wondered what went on behind the scene when you type series of texts on your keyboard? or send an e-mail to a friend from Rwanda who speaks french? how is the mail converted from english to french, how does the computer find the equivalent for each character you type? does the computer understand english? so many questions comes to heart. Well what we can all agree for sure is that the computer understands only 2 characters "0" and "1" respectively, these can be referred to as bits.This implies that an alphabets has to be interpreted as numbers for computers to store texts.

So how were all these characters in-cooperated into the computer's system? well in the early days of computer's inception (1960s), the primary means of communication between people were the use of teletypes(typewriters, teleprinters, etc).
These teletypes used a 5-bit encoding system which could range up to 32 character sets(2 ^ 5 = 32), the problem with this system was that it didn't provide enough space to represent all the english letters(a-z, A-Z), punctuation signs, numbers and other quintessential characters needed for effective communication.

Introducing Ascii

Due to the limitations of the 32-bits encoding system, there was need for a much better and standardized means of communication, in october 1960 The American Standards Association (ASA), now the American National Standards Institute (ANSI),led by Robert William Bemer (February 8, 1920 – June 22, 2004 began work on ASCII which is an acronym for American Standard Code for Information Interchange.In 1963 ASA introduced the first version of Ascii, unlike the former it was a 7-bit encoding system that could hold up to 128 character sets(2 ^ 7 = 128), numbered 0-127.

So for the English language, which has 26 letters, ASCII had enough slots for both upper and lower letter cases, numbers (0 to 9), punctuation marks, and unprintable control codes for teleprinters.

It was a great improvement obviously and by march of 1968, then US President Lyndon B. Johnson, announced that henceforth all computer systems should adopt the Ascii system as the default standard for information interchange(see more here), but as with every technology Ascii had it's own bottlenecks, one of which was it's in-ability to represent non-english characters, So, for European languages that use accented alphabets like German ä, ë, or Polish ź, ł, ę, ASCII wasn’t a favorable option.

Unicode to the rescue

Once again there was need for a much more diverse encoding system, that breached the disparities in communication and enhanced universal inclusion, as all other attempt towards tackling the problem resulted in a more complicated problem, during this period, globalization and internalization had become a core aspect of marketing and distribution, therefore global inclusion was vital at this point.

So in 1988 Joe Becker a computer scientist and expert on multilingual computing introduced an encoding scheme known as Unicode(Uniform character enCoding) in which each character is assigned a unique number known as a Code Point(A code point is the value that a character is given in the Unicode standard). This was a real breakthrough as not only was this applicable to english language alone, but also for every language around the world. The objective of Unicode was/is to unify all the different encoding schemes so that the confusion between computers can be limited to the very minimum.
Currently the Unicode is of three variants namely:

UTF-8: which is made of one byte(or 8 bits) and is well known for it's wide adoption in email systems and the internet in general
UTF-16: as you guessed is made up of two bytes(or 16 bits)
UTF-32: this encoding scheme utilizes four bytes(or 32 bits) to represent textual characters.

Note: UTF means Unicode Transformation Unit.

And that brings us to the end of this article, of course there's much more to encoding as this is just a quick see-through of the broad field of encoding/multi-lingual processing.
If you enjoyed this article, kindly leave a comment on what you learned from this one. Peace Out :)