DEV Community: Uddeshya Singh

Weird rants of a new Devops + Backend engineer 😤 [22 June 21]

Uddeshya Singh — Tue, 22 Jun 2021 17:37:27 +0000

Welcome to the weird notes of a new DevOps and Backend engineer. This is my (quite open) notebook on what was all the stuff I read and learnt. Not necessarily tech, you can just read on and understand what I am rambling on about.

[22 June 2021]

Stuff I learnt 📝

Opentelemetry logs are painful as fuck. Nothing official in terms of documentation :) I swear to god I feel I will find a good implementation and I will DOCUMENT IT FFS.
The other night I jotted down a blogpost for traces + opentelemetry + sentry. I hope folks would love it. Golang, Opentelemetry, and Sentry — The Underrated Distributed Tracing Stack
Also, fluentd ---> OTEL Collector integration? YOU ARE PAINFUL :) Idk what kubernetes service guide I need to revisit because I am sure af not getting anywhere in this respect.

Stuff I need to check out tomorrow? 📅

Get done with one more exam.
Setup the log system locally and see if it works.
Upload the rules files.

Stuff I want to work some day in future. 🔭

Go through the exporter fix PR in the weekend maybe. Or if time allows tomorrow.

Thanks for listening to the rant.
Uddeshya Singh,
Signing off.

Weird rants of a new Devops + Backend engineer 😤 [19 June 21]

Uddeshya Singh — Fri, 18 Jun 2021 18:38:27 +0000

[19 June 2021]

Stuff I learnt 📝

So, the current opentelemetry sentry-exporter is indeed incomplete right now. I investigated the elastic APM exporter and realised that Sentry one is just not using it's full capabilities yet. You need to run span.Events() to get a list of events which can be further converted into *sentry.Event.
Also, in the prometheus-operator helm chart, we figured out how to add additional rules without having to "upgrade the chart" without putting in new values in values.yaml. Just create a new PrometheusRule object as a rule file and the ConfigMaps will auto grab it.
PS: I am talking about this operator here : kube-prometheus

Stuff I need to check out tomorrow? 📅

How do i link a sentry span and a sentry issue.
Talk with team and refine the requirements because hell no, Sentry is not a logging platform (or I am an idiot)
Get vaccinated ffs 😢

Stuff I want to work some day in future. 🔭

IF I manage to pull this issue linking stuff off, I think I will contribute this to opentelemetry collector exporter. My first cloud native contribution might be on its way idk man.

Thanks for listening to the rant.

Uddeshya Singh,
Signing off.

Weird rants of a new Devops + Backend engineer 😤 [18 June 21]

Uddeshya Singh — Thu, 17 Jun 2021 18:30:26 +0000

[18 June 2021]

Stuff I learnt 📝

I finally created my own Docker image of a gin-app and deployed it in a kubernetes pod.
The biggest challenge? Connecting the 2 pods, our otlp-example and the opentelemetry-collector.
I needed to connect from pod A to port 4317 of pod B. Ofcourse, i had exposed the port in pod B using NodePort yet it didn't seem to work.
What finally worked? Instead of using 0.0.0.0:4317, I just needed to use otel-collector.observability.svc.cluster.local:4317 psst, it's the port name (or deployment name? not sure) , then namespace, then service and so on, which is exposed by a Node Port in this other service.
MAN ELASTIC APM IS AMAZING. LIKE. SO AMAZING TO LOOK AT THE TRACING <3

Stuff I need to check out tomorrow? 📅

Plan on figuring out how to collect logs from a pod using fluentD and exporting it to our dear openetelemetry-collector.
Use env vars to toggle the GRPC connection port.

Stuff I want to work some day in future. 🔭

Nothing today

Thanks for listening to the rant.
Uddeshya Singh,
Signing off.

Weird rants of a new Devops + Backend engineer 😤 [17 June 21]

Uddeshya Singh — Wed, 16 Jun 2021 22:10:32 +0000

[17 June 2021]

Stuff I learnt 📝

Opentelemetry's sentry exporter is pretty cool, even thought it's specifically for traces right now, it does its job well.
The sidecar architecture for opentelemetry-collector will work the best for a scenario. The engineers thought out the agent, collector interface pretty good.
The Traces specification and the TraceID stuff is good too, I just hope there was more sync between the published package and the github repository.

Stuff I need to check out tomorrow? 📅

Need to show the instrumentation demo to the lead.
Probably will need to checkout Elastic APM exporters for these traces.
See if sentry can support any logging :)

Stuff I want to work some day in future. 🔭

Checking how opentelemetry trace back the traces by the ID.
WTF is going on with those spans? Why do they provide us with new contexts?

Thanks for listening to the rant.

Uddeshya Singh,
Signing off.

Bare Metal instances and Hardware acceleration | Solutions for an organization

Uddeshya Singh — Fri, 27 Nov 2020 18:31:25 +0000

"You can invest more into android emulator hosting hardware (i.e. get bare metal instances) and reap the benefits later" will be the tagline I would start with because that really is the gist of this entire post (minus some really cool demonstration scripts I've got for you)

Make your emulators Faster! 💥

Let's pick up from where we left off and talk about making your emulators faster. In the normal t2 instances, as we know, there is a hypervisor layer on top of the instance hardware, hence abstracting the hardware layer from the emulator we want to run on it, and thus, we can't use out good old hardware acceleration on top of it.

But what if, we take away the hypervisor layer? No, you can't do it with a piece of code but you can definitely get it by investing some odd $4 per hour and get a machine that can run 40 emulators at once (tried and tested statistic).

That's exactly how we pulled off a mass testing fleet with a variety of emulators all running simultaneously on a single machine. Since the hypervisor layer is stripped away, you get an emulator which is at least 10 times faster than the old armeabi 64bit models.

Isn't it too costly? 💸

It'll definitely take investment and on-demand instances are really costly by amazon (or really any cloud provider) as far as bare metal instances are concerned, but let's really do the math.

A high performing Genymotion PaaS setup would cost you: $0.7 per emulator
A bare metal instance at maximum load would cost you: $4.8/40 = $0.12 per emulator.

See? Simple maths. To get a profitable testing fleet, you really do need to splash the cash (hence the title, solutions for an organization)

Soo, which system image to pick? 🤔

I'll recommend android-28's google API version (that too x86 64-bit version) for further checking out the emulators. To get that system image, please run the following commands

$ sdkmanager "system-images;android-28;default;x86_64"

But why?

This system image has a fairly functioning Google Apps support.
Enabling root access isn't much of a pain, a simple command (adb root) would do the trick.
You can play around with your emulator configurations and convert this AVD into whatever device you wish (I mean the identity, of course)

Enabling hardware acceleration in bare metal 🏃‍♂️

In any bare metal instance (let's say m5.metal), we need to follow the old instructions which we did so far in the first post. Now follow the following commands to set up the KVM hypervisor in ubuntu 18.

$ sudo apt-get install qemu-kvm libvirt-bin virtinst bridge-utils cpu-checker
$ sudo usermod --append --groups kvm 'echo $USER'
$ echo 'KERNEL=="kvm", GROUP="kvm", MODE="0666", OPTIONS+="static_node=kvm"' \
    | sudo tee /etc/udev/rules.d/99-kvm4all.rules
$ sudo udevadm control --reload-rules
$ sudo udevadm trigger --name-match=kvm

and this should be it. Now you can start any emulator by a simple command :

$ emulator @emulatorName

and this will automatically utilize KVM hypervisor to boot your emulator (PS, if you are using the system image of android-28 and x86_64 architecture, you should see your emulator booting up within 20 seconds).

And this wraps us our parts on how to run fast emulators, how about, we try how to maybe install, run and uninstall apps in our emulator programmatically?

Setup the repository for demo 🏢

The aim of this repository is to empower anyone to set up the emulators and try them out in order to witness the beauty (and the pain, if you have < 8 GB of RAM in your system) these Android emulators carry with them.

Head over to Emulator CLI Autocompleter and follow the instructions in Readme to set it up!

uds5501 / Emulator-CLI-Autocompleter

Python autocomplete CLI for android emulators (Creation and APK Run demonstration)

Thanks for Reading my Blog!

The .zip file contains the legacy command line tools given to setup your emulator environment To setup your emulator environment correctly, please follow the blog instructions given in either of the 2 posts

Using the emulator demo script (with autocompletion support)

Emulator CLI is generally boring and anyone in the automation team can say amen to the fact that staring at a non saucy and tedious CLI can be a tad bit unproductive. Here is my bit to make the terminal a colourful play ground, at least, in the terms of autocompletion behaviour which is more or less non existent in the traditional command line tools provided by our dear Android team.

The script uses the prompt-toolkit to implement interactive CLI experience for the user. You can check out the source code in the internal directory.

Requirements to run the script

…

View on GitHub

$ sdkmanager "build-tools;30.0.2"
$ export PATH=$PATH:~/path-to-built-tools/build-tools/30.0.2

The following CLI tool gives you an auto-completion feature in certain areas like system image picking, AVD boot, and APK choices. The repository has been created with python as the scripting tool and python-prompt as the auto-completion support.

Psst… the following demonstrations have been done on an 8 GB, i7 system, So the results are at least 30–40 seconds slower than in actual systems.

Creating an AVD programmatically ⚙️

Here we can create an AVD programmatically using a script, currently, 3 options are provided for system images which you have to pre-install using standard SDK manager installation commands.
The python prompt helps in selecting the SDK image you would like to pick, saving all the words you would've to type xD

Running an entire demo loop on created AVD 💎

In this particular loop, we are going to install our good old Twitter app in one of AVDs and see how smoothly it installs, runs, and finally gets uninstalled on 8 GB of host RAM.

In the demonstration, it's a Twitter application, you can run any apk you want but you would need to put the APK in the apk/ directory for the prompt to load the suggestion.

In the above image, you can see that the tool captured 7 AVDs on the local system and proceeds to suggest the ones which match the name you are typing.

In the above image, the auto-completion suggests the APK and the package name it carries, the size of the application, and release versioning.
… and that would be the end of the demonstration ✨

What to expect from the next post? 🎓

In simple bullet points

A more robust demonstration script for easily playing around with your emulators
Improvement in costing solutions and how to scale up the current solution so that it costs us 5 times cheaper but is easier to scale.

Resources 🔗

Github CLI Autocompleter demonstration script
Python prompt tool kit
AWS instance pricing guide

ARM architecture and Genymotion | Solutions for a solo developer

Uddeshya Singh — Mon, 09 Nov 2020 14:46:01 +0000

Exploring the use cases where android emulators come in handy, how to set them up on the EC2 instances, what variants you can run on cheap t2 instances, and what a solo developer should expect when planning to use AWS instances for cloud emulator deployment.

Disclaimer: The author of this blog post documents his opinions and experiences towards android ROM and kernel in particular, anything other than that like android app development and gradle builds are not his forte and any pieces of advice will not be found under this post. Thank you for the patience, Read on!

Introduction and Use Cases😌

3 Months into Trell and I have a pretty interesting project in my hand, albeit it was on the tech which was completely alien to me, the fabled android kernel, and all the complications which come with it. The task is a simple automation pipeline for application testing on multiple new devices, a purpose which android emulators can serve pretty well. But this kind of pipeline can serve multiple purposes like :

Thorough testing of apps via background pipeline while not consuming your system's resources and you can carry on with your development.
Testing of applications on android smartphones that you do not physically possess.
Logging and Monitoring the behavior of your firebase event tags on multiple devices at once.
CI/CD pipeline loves an android emulator and all your unit tests would pass through them.

What is an Android Emulator? 🥧

I am around 3 years late to this party (but definitely in sync with the time in terms of the use case) but still, let's get done with the formality and move on to the good stuff.

Taking the definition directly out of Android's book, we can define the Android Emulator as a tool that simulates Android devices on your computer so that you can test your application on a variety of devices and Android API levels without needing to have each physical device.

The emulator provides almost all of the capabilities of a real Android device. You can simulate incoming phone calls and text messages, specify the location of the device, simulate different network speeds, simulate rotation and other hardware sensors, access the Google Play Store, and much more.

Setting Up the Emulator Environment 🧑🏾‍💻

Now, let's move on to setting up the emulator environment and the instances we would need along with it!

I'll be using a classic t2.medium instance (Ubuntu 18.04, 64 bit, x86 arch) for our purpose, with adding 25 GB storage to the instance (believe me, classic 8GB won't cut it), make sure you have the authentication (.pem) key for logging into your instance and let it bootup!

One important thing to note about this entire usage is that you can't access your emulator using GUI interfaces (actually, you can but that's something I have planned for some later post :P ), so we need to make sure that anything and everything we do is done solely by CLI (aka, our good old terminal!)

Now, once you have logged into your t2.medium instance, you'll need to clone this repository:

uds5501 / Emulator-CLI-Autocompleter

Python autocomplete CLI for android emulators (Creation and APK Run demonstration)

Thanks for Reading my Blog!

Using the emulator demo script (with autocompletion support)

The script uses the prompt-toolkit to implement interactive CLI experience for the user. You can check out the source code in the internal directory.

Requirements to run the script

…

View on GitHub

into your instance. This will contain your command-line tools zip file. (currently hosted version by android doesn't work for me :( )

Now follow the commands step by step.

$ sudo apt update && sudo apt install default-jdk unzip -y
$ cd Emulator-Environment-Setup
$ unzip commandlinetools-linux-6609375_latest.zip
$ mkdir ../cmdline-tools
$ mv tools ../cmdline-tools

By now you have taken care of the basics, now we need to set the path variables (which would save us the pain of writing the entire path every time we want to call sdkmanager or emulator etc.)

$ nano ~/.bashrc

Now make sure your bashrc file looks like

Save this file (Ctrl + S) and get out (Ctrl + X)
Now, load the preferences and you are good to use the SDK tools, and accept all the licenses by typing y .

$ source ~/.bashrc
$ sdkmanager --licenses

You shall witness a screen like this!

Now, we just need to set up the emulator, platform-tools and we can create our own Android Virtual Device!

$ sdkmanager "emulator"
$ sdkmanager "platform-tools"
$ mkdir platforms

With this, your basic environment setup is done! Now let's take care of creating a virtual device.

Android Virtual Device on AWS 😁

To create an AVD, you need to first select the system image it will be built on, you can list all the SDK images by typing:

$ sdkmanager --list | grep system-images

You will find multiple system images tailored to multiple devices, with a general format of

"system-images;<version>;<gapi_setting>;<emulator_architecture>"

Take a look at the given configurations, they are all of android-25 (Nougat) and come in variants of android_tv, default, google_apis etc, with 2 architectures namely ARM and Intel's x86 which further bifurcates into 32 bit and 64-bit variants.

But we will specifically be using system-images;android-25;google_apis;armeabi-v7a version. Now let me justify this choice.

Why android-25? 🤔

This is something which Trell needed for checking firebase event logging, anything less than android-29, so picking a decently lower version made sense to observe backward portability of the app.

Why the google_api version? 🙇🏽‍♂️

We can perform a wide variety of operations on our android emulator if they are rooted by default (we have the superuser access). To connect into root mode and access emulator shell using the command line, we need this version to be "default" because google_apis and google_apis_playstore won't allow root connection from the android debugging bridge (in intel architectures) but in ARM (like this one) you can take anyone, all will be rooted by default!

Why armeabi-v7a architecture? 💁🏾‍♂️️

v7a represents a 32-bit version of the ARMarchitect architecture (something which is required for the trell app). Now, why ARM? because Intel (x86) architecture requires hardware acceleration to work, and this can be done with either Intel HAXM (in windows) or KVM (in Ubuntu). You can try this sequence of commands to check if KVM can be enabled on your instance or not.

$ sudo apt install cpu-checker -y
$ kvm-ok

This would greet you with the following message.

INFO: Your CPU does not support KVM extensions
INFO: For more detailed results, you should run this as root
HINT:   sudo /usr/sbin/kvm-ok

Why do intel emulators not work in these EC2 instances? 😭

Before we get into this, let's take a quick look at how emulators work. QEMU is a tool that mimics guest device hardware and it then translates the Application Binary Interface (ABI) of the guest device to match that of the host device.(armeabi-v7a to intel x86_64 in this case). Now to speed up this complex and slow process, hypervisors are introduced.

Intel's HAXM (Hardware Acceleration Execution Manager) is a hypervisor component for Windows and macOS. There's KVM (Kernel-based Virtual Machine) for Linux. With hardware-acceleration, the Android emulator can run virtual devices at speeds similar to that of your workstation CPU.

When we choose system-images;android-25;default;x86 we are trying to run a virtual machine within a virtual machine. It is a second level or nested virtualization that we are trying to achieve which, unfortunately, is not yet supported. The Intel hardware supports only a single level of hardware-assisted virtualization, adding support for efficient (i.e., not painfully slow) nesting requires a lot of clever software engineering in hypervisors.

Run a Testing Device 🏃🏽‍♂️

All the theory currently stated as to what will and what won't work, let's focus on creating a test virtual device. Follow the steps to do so!

$ sdkmanager "system-images;android-25;google_apis;armeabi-v7a"
$ echo "no" | avdmanager create avd --name "testDevice" -k "system-images;android-25;google_apis;armeabi-v7a"

So, you just created an AVD, how to check that out? You can use avdmanager to check your virtual android devices.

$ avdmanager list avd

Now, let's run this virtual device!

$ emulator -ports 5554,5555 -avd testDevice -writable-system -no-window -no-audio -gpu swiftshader_indirect -show-kernel &

-ports 5554,5555 define the connection ports on which the android debug bridge will connect to this emulator (in this case, 5554) and they have to be consecutive!

-writable-system would give you access to modify any file or push any file as you wish in root settings.'

-no-window would simply run the emulator headless (without this flag, it won't work because we don't have any OpenGL based graphics setup on a CLI)

-no-audio would take away it's audio support (something I'd suggest you do)

-gpu swiftshader_indirect to prevent a boot loop.

-show-kernel something for you to maintain your sanity and check the kernel's progress as it boots up because it WILL take forever (60–90 minutes) to boot up properly.

These are the testimonials on why you should not go for something this slow, because this would be counterproductive, you can rather test on your local devices rather on a device which will take 1 hour to function properly and still give you an average of 2.5 minutes to install a normal apk!

Why Genymotion and how good is it? 📱

If you are a solo developer and can afford the Genymotion PaaS subscription ($0.5 + amazon instance charges), it's your way to go. For example, I am using m4.xlarge with Genymotion PaaS on top, it'll cost you $0.7/hr and it will be a decent deal if you can afford it.

Genymotion is a go-to emulator for many organizations still and the amazing support and speed which their emulators provide are good enough for the purpose we want to solve, if and only if, your pocket allows you that much freedom.

But, still, $0.7/hr per emulator is a bit too costly, at least if you wanna do testing on a large scale.

Closing Thoughts and what to expect from the upcoming post? 👨🏽‍🏫

In this post, we explored the basic setup of an emulator environment and how to setup a simple AVD. What works in a general t2 instance and what are the alternatives.

In the next one, we'll explore how to setup multiple emulators in a single instance and explore android debug bridge in depth while customizing our device and run simple applications with intel emulators (which are at least 10x faster than these ones)!

References 🔗

Go-Arch | Module Scripting Bootstrap in Go Lang

Uddeshya Singh — Fri, 23 Oct 2020 06:50:05 +0000

Introduction 📖

Two months into my internship at Trell - A Visual Blogging Platform for Explorers! as a full stack developer, I dived headfirst into the world of Go and more specifically the Gin Web framework. Upon my first major project assignment, I was given this holy grail module, a one-stop solution for any scripting + HTTP server requirements including integrations with SQL database, Redis and Elastic. We Majorly used this for scripting and I will be elaborating how.

TL; DR 🚶‍♂️

If you know the basics of scripting and how the module works in go lang and just want a bootstrap code for getting started with the scripting part, head over to this GitHub repository uds5501/go-arch

Use Case 💡

This module can be used interchangeably whenever you need a web server or a script.
You can use this to create a database scheduled task dequeuer (Which we did)
This can be used for a custom task queue in coupling with Redis (check this post maybe: Implement Job queue in Redis[0]
You can couple this with go-Olivere [1] package to integrate ELK Stack.

Now, we can get right into our project structure!

What is go-arch and what are we building? 👨🏻‍💻

Go-arch is a small module that employs multiple packages and the goal work of the script is to do simple hardcoded arithmetic operations.

It is a condensed version of the project implementation we actually did at Trell, that project employs Redis, elastic, and Kubernetes (for deployment) but what I will be showing is scaled-down version with only a database and logging module, while the main Module (Operations) will be taking care of only simple arithmetic operations (+,-,* to be specific) and structural outlines and what method in which package calls what struct in which package.

Project Structure (For Scripts) 🔧

A rookie mistake of a hackathon project in Go is to bundle everything you want to do into a simple main.go and completely forget about it, until next time you change a single line of logs and bam! Your IDE is full of yellow and red lines to check where did we go wrong.
Now bundling around 5k lines of code was an option to me, but making your first Go project in a monolith directory layout? That should prick your conscience.
So here is a much more distributed layout and a scaled-down version of the boilerplate we actually use!

Package definition

config : Takes care of the project setup, this is where we read our environment files and return a config object.

db : As the name suggests, takes care of the database connection handler for our project and returns a db handler for the same.

logger : This package helps in logging errors and debugging messages at production and development level.

pkg/common : Here we build our utility functions which are supposed to be accessible for trivial operations in our main script aka checkAdder.go

pkg/scripts : This is the money shot. This is where your main script should be placed and your server.go should take all the scripting logic it wants from right here.

redis : Again as the name suggests, for handling local redis connection.

server : The layer between main.go and your business logic ( pkg/scripts). Here you can decide whether you want your package to be a gin web server or a simple script hosting module, it's all your choice!

main.go : This bad boy seriously does not need any introductions!

go.mod : Stores your external dependencies and it is what stores the module dependencies which are required for a successful run.

Now that our layout is defined, let's go step by step to our package setup!

[1/8] Basic Environment Configuration 🔧


package config

import (
    "fmt"
    "os"
    "strconv"
    "strings"

    "github.com/joho/godotenv"
)

type Config struct {
    AppName              string
    AppEnv               string
    SqlPrefix            string
    RedisAddr            string
    DBUserName           string
    DBPassword           string
    DBHostWriter         string
    DBHostReader         string
    DBPort               string
    DBName               string
    DBMaxOpenConnections int
    DBMaxIdleConnections int
    ServerPort           string
    EsURL                string
    EsPort               int
}

var config Config

// Should run at the very beginning, before any other package
// or code.
func init() {
    appEnv := os.Getenv("APP_ENV")
    if len(appEnv) == 0 {
        appEnv = "dev"
    }

    configFilePath := ".env"

    switch appEnv {
    case "production":
        configFilePath = ".env.prod"
        break
    case "stage":
        configFilePath = ".env.stage"
        break
    }
    fmt.Println("reading env from: ", configFilePath)

    e := godotenv.Load(configFilePath)
    if e != nil {
        fmt.Println("error loading env: ", e)
        panic(e.Error())
    }
    config.AppName = os.Getenv("ELASTIC_APM_SERVICE_NAME")
    config.AppEnv = appEnv
    config.SqlPrefix = "/* " + config.AppName + " - " + config.AppEnv + "*/"
    config.RedisAddr = os.Getenv("REDIS_ADDR")
    config.DBUserName = os.Getenv("DB_USERNAME")
    config.DBHostReader = os.Getenv("DB_HOST_READER")
    config.DBHostWriter = os.Getenv("DB_HOST_WRITER")
    config.DBPort = os.Getenv("DB_PORT")
    config.DBPassword = strings.ReplaceAll(os.Getenv("DB_PASSWORD"), "--", "#")
    config.DBName = os.Getenv("DB_NAME")
    config.DBMaxIdleConnections, _ = strconv.Atoi(os.Getenv("DB_MAX_IDLE_CONENCTION"))
    config.DBMaxOpenConnections, _ = strconv.Atoi(os.Getenv("DB_MAX_OPEN_CONNECTIONS"))
    config.ServerPort = os.Getenv("SERVER_PORT")
    config.EsURL = os.Getenv("ES_URL")
    config.EsPort, _ = strconv.Atoi(os.Getenv("ES_PORT"))
}

func Get() Config {
    return config
}

func IsProduction() bool {
    return config.AppEnv == "production"
}

We use a config structure to grab all the configuration parameters from .env file us godotenv and return the struct object for further use. You can configure which env file to use according to the APP_ENV environment variable.

[2/8] Setup the database handler ⛓

Though this is not really required for what I will be building, but this is still a handy exercise to see how you can seemlessly pass around a configured database module into business modules without any re-initialisation.


package db

import (
    "database/sql"
    "fmt"
    "sync"
    "time"

    "trell/go-arch/config"

    "trell/go-arch/logger"

    "go.elastic.co/apm/module/apmsql"

    _ "go.elastic.co/apm/module/apmsql/mysql"
    "go.uber.org/zap"
)

var reader *sql.DB
var writer *sql.DB
var once sync.Once

type DBConfig struct {
    DBUserName           string
    DBPassword           string
    DBHost               string
    DBPort               string
    DBName               string
    DBMaxIdleConnections int
    DBMaxOpenConnections int
    DBConnMaxLifetime    time.Duration
}

func NewDBClient(config *DBConfig) *sql.DB {
    url := config.DBUserName + ":" + config.DBPassword + "@tcp(" + "trell-mysql-db-staging.cyqwbanzexpw.ap-south-1.rds.amazonaws.com" + ":" + config.DBPort + ")/" + config.DBName + "?multiStatements=true&parseTime=true"
    client, err := apmsql.Open("mysql", url)
    fmt.Println(url)
    if err != nil {
        panic(err.Error())
    }

    client.SetMaxIdleConns(config.DBMaxIdleConnections)
    client.SetMaxOpenConns(config.DBMaxOpenConnections)
    client.SetConnMaxLifetime(time.Minute * 10)
    return client
}

func Init() {
    once.Do(func() {
        config := config.Get()

        writerConfig := &DBConfig{
            DBUserName:           config.DBUserName,
            DBPassword:           config.DBPassword,
            DBHost:               config.DBHostWriter,
            DBPort:               config.DBPort,
            DBName:               config.DBName,
            DBMaxIdleConnections: config.DBMaxIdleConnections,
            DBMaxOpenConnections: config.DBMaxOpenConnections,
            DBConnMaxLifetime:    time.Minute * 10,
        }

        readerConfig := writerConfig
        readerConfig.DBHost = config.DBHostReader

        reader = NewDBClient(readerConfig)
        writer = NewDBClient(writerConfig)

        logger.Client().Info("writer connected", zap.String("host", config.DBHostReader))
        logger.Client().Info("reader connected", zap.String("host", config.DBHostWriter))
    })
}

func Factory(typ string) *sql.DB {
    switch typ {
    case "reader":
        return reader
    case "writer":
        return writer
    default:
        panic("no such db")
    }
}

func WrapQuery(query string) string {
    return config.Get().SqlPrefix + query
}

type DBFactory func(t string) *sql.DB

Over here, we reuse the config module's struct and use our own DBConfig struct to setup the database connection handler. You can see that there is a nice combination of database/sql and ampsql packages to handle the database connection.

What are readers and writers? 🤔

Writer Handler handles write heavy queries (like inserting, batch insertion and updating) into a database and the Reader handles the select queries from the database. Although, this is for elastic search and not vanilla sql so you can really skip it.

[3/8] Setup the Logger 🧾

With the database setup, now let's take care of how we are going to show you, the developer the outputs because we firmly need dubgging module in place for production environment. You can get away with a series of fmt.Println() but it won't really cut it in a production environment.


package logger

import (
    "sync"

    "trell/go-arch/config"

    "go.uber.org/zap"
)

var logger *zap.Logger
var once sync.Once

func Init() {
    once.Do(func() {
        if config.IsProduction() {
            logger, _ = zap.NewProduction()
        } else {
            logger, _ = zap.NewDevelopment()
        }
        defer logger.Sync()
    })
}

func Client() *zap.Logger {
    return logger
}

At trell, we are using Uber's zap logger and it's fast. No, really fast when it comes to logging. Don't believe me? Check this performance comparison[2] and see for yourself!

[4/8] Implementing the Operations Script Logic ✨

In the pkg/scripts let's make our checkAdder.go which will be a part of scripts package. Just create a struct named Operations which will be the initialized script.
It will contain a DB factory struct export (from trell/go-arch/db package) and define the methods accordingly.

Exportable and private methods ✨

I have created Init() and ExportableFunction() as an example. These functions are named in a capitalized manner and these are the ones that can be accessed by default when you call utility functions inside another script.

The functions like getAddition(), getSubtraction() etc are non capitalized and are in camel-case, they can be accessed within the struct but not outside it.

What does NewOperations() do? 🤔

The sole purpose of existence of this capitalized function is to initialize a new Operations script in module.go which in turn can be used for almost any operation (in our case, Initialization only)

[5/8] Making sure the script is initialized only Once 🙇🏻‍♂️


package scripts

import (
    "sync"
    "trell/go-arch/db"
)

type Module struct {
    script *Operations
}

var moduleSingleton *Module
var moduleSingletonOnce sync.Once

func NewScriptsModuleSingleton(db db.DBFactory) *Module {
    moduleSingletonOnce.Do(func() {
        script := NewOperation(db)
        moduleSingleton = &Module{
            script: script,
        }
    })

    return moduleSingleton
}

func (m *Module) GetScript() *Operations {
    return m.script
}

To make sure we initialize the script only once for our application, we will use the function NewScriptsModuleSingleton() to call an instance of sync.Once (which executes a function once) and return a struct with Operations module pointer.

Ps: you should really read more about context management and process syncing to get the idea behind sync.Once() from the official docs or this.[3]

You will be executing GetScript() to get this script with appropriate definitions inside your infinite running code (in this case, server.go)

[6/8] Setup the server 🖥


package server

import (
    "trell/go-arch/db"
    "trell/go-arch/logger"
    "trell/go-arch/pkg/scripts"
)

func Init() {
    logger.Init()
    db.Init()
    //es.Init()
    //redis.Init()
    scriptsModule := scripts.NewScriptsModuleSingleton(db.Factory)
    scriptsModule.GetScript().Init()
    // r := NewRouter()
    // r.Run(":" + "4000")

}

This is a simple initialzation of a server. It initializes the logger and database and once that's done, It calls for the scriptsModule to initialize the business logic scripts (in this case, our pkg/scripts/module.go will be called) and will run infinitely.

[7/8] Last but not the least, main.go 🌠


package main

import (
    _ "trell/go-arch/config"
    "trell/go-arch/server"
)

func main() {
    server.Init()

}

This is it, that's all, just call server.Init() rest will be a series of procedural calls as shown in the diagram below and to run your app, use go run main.go and see the magic!

[8/8] One more thing, external modules? 🤦

go run main.go didn't work? oops, one last thing. So far we have taken care of modules which we defined ourselves, but what about the modules we have been using externally? (like go-redis, ampsql etc). Use go.mod file for that, as this is what really makes your go-arch a module.

It's basically like package.json, where you specify the external dependencies and versions of the same and on each build which are required for a successful build of your own go module, a go.sum file will be created, storing their hashes for future use, just like package.lock.

And that's a wrap, This makes sure that your go module is packaged and runs perfectly on each successful build, go ahead and give it a try now. The Github repository link can be found here uds5501/go-arch

Links Provided 🗣

[0]Job Queues: @mhewedy's post
[1] Go-Olivere : Go-Olivere docs
[2] Go-Zap : Go Zap performance
[3] Sync.Once : @martyer's post