DEV Community: Celestine Omin

Terraform: Infrastructure as code - Part II

Celestine Omin — Sat, 24 Jun 2017 04:00:00 +0000

In Part I, I introduced us to the concept of IAC(Infrastructure as Code) using Terraform, and we explored the awesomeness of Terraform. While the code we used in Part I for provisioning a simple server did work very well, the system we eventually provisioned is hardly a scalable system and not one I’ll recommend for production use. The reasoning here is simple, running a single server for your entire application is almost as bad as lighting a match in a gas station, bad things can and will definitely happen. And I’ll strongly suggest that you refrain from this setup.

In this second part of our IAC series, I’ll want us to build upon what we started in Part I and we will try to build a semi-highly available system. In Code X of Part I, we had created one server with the following code

resource "digitalocean_droplet" "web" {
  image = “ubuntu-16-04-x64”
  name = “web-1”
  region = “lon1”
  size = “1gb”
  ssh_keys = ["${digitalocean_ssh_key.default.id}"]
}

Code I

Now let’s expand on this code and create two web servers. To do this, we will add a meta-parameter called count. The count parameter can be added to any resource and it simply creates more of the declared resource based on the count value. For instance, we will add count = 2 to create two web servers.

resource “digitalocean_droplet “web {
  image = “ubuntu-16-04-x64”
  name = “web-1”
  count = 2
  region = “lon1”
  size = “1gb”
  ssh_keys = ["${digitalocean_ssh_key.default.id}”]
}

Code II

Notice the introduction of count = 2 in the code above? Now that’s how we create 2 web servers. If we wanted to create 10 web servers, all that we need do is to change the value of count to 10, if we need 20 servers, we change the value of count to 20, you get the idea.

The next thing that we will do to build our semi-highly available system is to create a loadbalancer and distribute traffic to all of our servers. Terraform provides us with a digitalocean_loadbalancer resource and we will make use of it. Adding a load balancer is simple, we will declare a load balancer resource, give it a name, choose a region, apply the traffic forwarding rules, add a health check from the load balancers to the attached machines and finally, attached our Digital Ocean droplets to these load balancers. That simple.

resource “digitalocean_droplet “web {
  image = “ubuntu-16-04-x64”
  name = “web-1”
  count = 2
  region = “lon1”
  size = “1gb”
  ssh_keys = ["${digitalocean_ssh_key.default.id}”]
}

resource “digitalocean_loadbalancer “public_lb {
  name = “web-lb”
  region = “lon1”

  forwarding_rule {
    entry_port = 80
    entry_protocol = “http”

    target_port = 80
    target_protocol = “http"
  }

  algorithm = "round_robin"

  droplet_ids = ["${digitalocean_droplet.web.id}”]
}

Code III

From the code above, we added a public load balancer to our infrastructure. The name and region are pretty self explanatory and both are required as also the forwarding rule. The forwarding rule basically tells us how to send traffic. The protocol of choice for our load balancer is HTTP which has a default port of 80. The entry_protocol and entry_port simply states how traffic are being sent to the load balancer and the target_port and target_protocol talks about how traffic gets to the attached droplets. It’s that simple.

The next important bit here is the connection algorithm between the load balancer and the droplets. In the case, we chosed round robin. Although Terraform only supports two sets of algorithm for Digital Ocean’s load balancer resource –Round Robin(round_robin) and Least Connections(least_connections)–other load balancing algorithm exist like Weighted Round Robin, Least Traffic, Source IP, etc.

A round robin is an arrangement of choosing all elements in a group equally in some rational order, usually from the top to the bottom of a list and then starting again at the top of the list and so on. A simple way to think of round robin is that it is about “taking turns. Used as an adjective, round robin becomes “round-robin. - WhatIs.

In the last section of our load balancer resource block, we attach our droplets to the load balancer using droplet_ids. We use string interpolation to reference the droplet ids, this, is how Terraform builds its dependency graph. There is a convention to using interpolation in Terraform, it follows the pattern of ${RESOURCE_TYPE.RESOURCE_NAME.ATTRIBUTE_NAME}. In our case here, we are referencing the digitalocean_droplets resource with the name of web and we are getting all its id attributes.

With the set up above, we have been able to put together a simple layer 7 load balancing. The only missing bit here is a database server.

Adding a database is simple, unlike AWS, Digital Ocean, as of the time of this article, does not offer a managed database service like RDS. We will need to roll out our own manually. To do that, we will create a droplet resource, just like we did for web. The code in Code VI does exactly that for us.

resource “digitalocean_droplet “web {
  image = “ubuntu-16-04-x64”
  name = “web-1”
  count = 2
  region = “lon1”
  size = “1gb”
  ssh_keys = ["${digitalocean_ssh_key.default.id}”]
}

resource “digitalocean_droplet “database {
  image = “ubuntu-16-04-x64”
  name = “web-1”
  region = “lon1”
  size = “1gb”
  ssh_keys = ["${digitalocean_ssh_key.default.id}”]
}

resource “digitalocean_loadbalancer “public_lb {
  name = “web-lb”
  region = “lon1”

  forwarding_rule {
    entry_port = 80
    entry_protocol = “http”

    target_port = 80
    target_protocol = “http"
  }

  algorithm = "round_robin"
  droplet_ids = ["${digitalocean_droplet.web.id}”]
}

Code VI.

To get thing going and see its effect, we run terraform plan just to make sure things are fine and well sorted, then we run terraform apply to build the actual system. With the setup in Code VI, we have successfully build ourselves a simple infrastructure that is good enough to host a decent blog. In the next part of this series, we will talk about how to prepare our machine right after provisioning and install basic software on it.

Disclaimer

While following the examples outlined in this article, please bear in mind that there’s a cost attached to it. When you run terraform plan then terraform apply and create a real resource at your provider’s end, they start billing you almost immediately. As a word of caution and this apply only in a non-production environment, always try to clean after yourself. For this purpose, Terraform offers us a really handy command called terraform destroy. The terraform destroy command literally goes back to your provider and delete/destroy every single resource that you have created. This is a one-way command and cannot be undone, so I strongly advise you do this in your personal or experimental environment.

This post was originally published on cyberomin.github.io

Priorities: What matters most

Celestine Omin — Sun, 11 Jun 2017 15:22:39 +0000

First make it work, then make it right, then make it fast.
– Kent Beck

This is something I wrote a while ago. I think it still hold true to this day.

Last week, I had a rather interesting and illuminating discussion with someone. Our discussion bothered on some core concepts that almost every startup will deal with at some point; over engineering and time to market.

While I argued that getting your product quickly out of the door and testing market fit should take priority for every single startup, especially the ones venturing into new territories, my friend was of the opinion that getting engineering right from the onset before anything should be the focal point. His concern was simple, fix the engineering problem and save yourself from technical debt.

In the startup world, especially in emerging markets like Nigeria, getting your product out fast and testing it in the marketplace cannot be over emphasised. The whole idea of technical startup and digitisation, while not exactly new, our counterparts in the West have had over 20yrs head start before us. In this market, you have to, not just build, but also reorient people on why they should abandon their old, tried and tested ways of doing things to embrace this new shiny idea of yours. Getting to the market fast shouldn’t be taken lightly.

Reid Hoffman, the founder of LinkedIn now a partner at Greylock Partners, once said, "if you are not embarrassed by the first version of your product, you’ve launched too late." Speed and first mover advantage will win always. Not surprised he (Reid) is currently teaching a class on Blitzscaling at Stanford.

We have all heard it time without number that the customer doesn’t care what is happening under the hood. He or she isn’t interested in knowing if your data store is an RDBMS or a NoSQL data store. If your operating system is a Unix flavoured OS or a Windows server. He isn’t even interested in knowing if you wrote your stylesheet in vanilla CSS or you did go the Sass or Less route. He is interested in 2 things. 1. Does it work? 2. Will it solve my problems?

While writing scalable software and code cannot be ignored, it is imperative to note that if your product doesn’t gain market acceptability you are doomed. Dead. Your super sleek MEAN(Mongo, Express, Angular, Node) based app that runs on 4 Amazon M3 large instances with an elastic load balancer in front of them all using a MySQL RDS and a read replica will come to nut. Nothing. Nobody cares.

Product before engineering

Every hyper-growth startup today all started with less over engineering and in some cases monoliths; Netflix, Uber, Spotify, Dropbox, Twitter etc all ran really large systems and in most cases all in one machine. They were all super focused on the products. They all bothered about sleek engineering after they got a strong foot hold in the market. Today, all of these startups have broken things down in favour of micro services (smaller units of systems). You could argue that Travis Kalanic, CEO of Uber, isn’t an engineer, but I don’t think he bothered so much on how the first commit of the Uber codebase looked like. He just wanted a service that can/will allow users hail taxis using their mobile phones. Uber today is a ~$40B company, they can afford a star-studded engineering team. Heck, they poached almost all the scientist/engineers working on the self-driving car project at Carnegie Mellon University.

Why am I pro-product?

When Konga launched in July of 2012, we had one simple goal; retail. Our stack then was a simple Rails app that ran on a single 4GB machine. We didn’t even write the underlying e-commerce software, we used an open source solution called Spree and built upon it. No one cared. No one knew about it. We sold stuff, we were fighting an uphill battle, e-commerce, unlike today, didn’t have so much acceptability. We needed to prove to people that they can trust us with their money and we will deliver their item as promised. Today that one machine now spans fleets of servers and we are more than anything, favouring micro-services over monolith. Why? The time is right. We have gained to a greater extent, acceptability and we can afford to settle in now and build a rock solid infrastructure.

Does awesome engineering matter?

I understand why my friend favoured micro-services and doing things right from the get-go, for obvious reasons.

Micro services enable separation of concerns. No need to deploy an entire application when you only worked on a subset of it.
Scalability is easier
Deployment is a lot easier.
No single point of failure.

While all of these things are awesome and nice to have/do. For a brand new spanking startup, I still favour, speed, product market fit, acceptability(read as revenue) over awesome Google-esque engineering.

I live by this quote by Kent Beck - “First make it work, then make it right, then make it fast.”

A friend of mine just got into Techstars, his startup has a lot to prove. Awesome engineering skills isn’t one of them.

Remote work: How we make it work.

Celestine Omin — Tue, 30 May 2017 04:30:00 +0000

In the last couple of years, there has been a strong advocacy for remote work and how it a) improve employees productivity and b) is fast becoming the future of work. Technology companies, especially startups add remote work as one of their benefits.

But things aren’t always rosy in the remote land. One of the problems that I have seen one too many times is that of communication and visibility and the antidote to this is over communication. Voice out every move. The physical office provides some benefit that is almost non-existent when you work remotely, for one, the in-person communication cannot be overemphasised.

Working from the same physical location and actually seeing the people you work with has a huge upside. This benefit becomes apparent when communicating with one another, things like voice tone and body language add fine-grained nuances that technology can’t bridge.

____________

Working and leading a remote team can be daunting without the right tools and processes in place. Getting the right tools very early on will save the team loads of trouble and also improve productivity. I mean, you can only do your best work when you have the right tools. In this article, I’ll try to highlight some of the tools that my team uses daily and why we use them.

Slack: Slack is where the magic happens. I think about this tool like our command centre. It’s pretty much where we start and end our day. With the help of amazing bots and integrations, the possibilities with Slack are endless. We use Slack for our daily stand-ups, file sharing, voice calls and more. I personally use Slack as a personal assistant of some sort; I set reminders for things I am supposed to get back to at specific time using Slack reminders. Slack’s slash command provides a rich suite of operations that can be performed right from the app.

Beyond just the daily communication, we apply Slack to our team’s internal adaptation of ChatOps; a transparent workflow with a high focus on visibility. Take for instance, when a ticket is created or moved on our JIRA project board, a notification is sent to Slack our #developers Slack channel, that way, everyone on the team knows what’s happening at every point in time and who is working or responsible for what. When a comment is added to a GitHub pull request, Slack notifies a channel with the comment and the relevant parties can either continue the conversation using Slack or move it to GitHub.

What is ChatOps? Conversations, put to work. ChatOps is a collaboration model that connects people, tools, process, and automation into a transparent workflow. This flow connects the work needed, the work happening, and the work done in a persistent location staffed by the people, bots, and related tools. Source: Atlassian

We are big believers in the agile methodology, as such, we take things like continuous integration and deployment pretty seriously. Spring planning, retrospective and stand-up all happen on Slack.

CircleCI: CircleCI is an invaluable and almost indispensable tool. For us, it isn’t just another CI tool, it’s our source of truth. CircleCI plays a critical role in our development cycle and deployment pipeline. It helps with running our test suites and does an auto deployment. The beautiful thing about this tool is how it integrates nicely with our workflow and other tools. For instance, when a build is broken, not only does CircleCI emails the entire team, it also sends a Slack notification to the team. And since we use almost the same Slack handle on our GitHub profile, the build offender —a person who breaks a build…I made this up— is notified immediately, this helps improve our turn around time and it also encourages transparency and accountability.

CirleCI further allows us to enforce some defined standards like a coding style, it lint’s all of our code and this process helps keeps our code base clean and promote consistency.

CircleCI/Slack integration. Photo credit - Igor Davydenko

JIRA and GitHub: are other tools that we use extensively. We use feature branching in our workflow, which means every JIRA ticket is a branch. The integration between JIRA and GitHub allows us to go straight from a JIRA ticket to a GitHub branch, and since we have a PULL_REQUEST_TEMPLATE that requires us to include the JIRA ticket ID, we can easily go back to JIRA from GitHub.

When a branch is pushed into GitHub, JIRA links it automatically to its assigned ticket.

A JIRA ticket, showing the associated GitHub branch, number of commits and pull request status

Screen Hero: Once in a while, people need help with technical issues and this is absolutely normal. When a team member is blocked and needs help with an issue, someone who is free or a team member with the most experience and knowledge of the problem jumps on Scree Hero and pair. One of the benefits of this is that stronger bonds and connections are formed.

How did we get here?

In the beginning, things weren’t always like this, it took a bit of trial and error to find the best workflow. We started out using Slack for just communication but quickly found out that a lot of people didn’t quite know what was happening at every point in time. So, instead of asking “what are you working on? every so often, we leverage our tools and let them do the talking for us. That way, we can minimise user interruption and just focus on getting work done.

As a distributed team, we use these tools to compensate for our lack of in-person interaction. It could be a little annoying, and quite frankly, frustrating, having to go check on what task a person is working on at any point in time and also check the ticket status. We delegate this to Slack and Slack does the informing when a ticket status is changed. If a ticket moves from In Progress to Code Review, the author doesn’t need to manually inform the team to review their work, Slack handles this and a few extra keystrokes are saved. Everyone is happy :)

Improvements?

We are not where I will want us to be, but honestly, I don’t think we’ll ever get there. It will be a case of constant improvement. In the future, I would love to see us trigger manual deployment via Slack and get performance metrics via Slack too. We will likely experiment with tools like Hubot to see if they work for our team.

In addition, we’ll also explore custom integrations to leverage the APIs of the third party tools that we use. We will explore building an integration to AWS for instance, such that if there is a service degradation from a region that we work with, Slack can notify us in near real-time. This same tool can also be extended to include GitHub status notification etc. With this in place, we will be more proactive than reactive to issues.

PS: Do you work remotely? I will like to know how you work.

This post was originally published on cyberomin.github.io

Terraform: Infrastructure as code - Part I

Celestine Omin — Mon, 29 May 2017 02:00:00 +0000

Introduction

In the last couple of months, I have been obsessed with automation, workflows and infrastructure as code. This obsession led me to explore tools like Ansible and a little bit of Chef and how to better apply them to my everyday work.

In the last few weeks, I have been experimenting with Harshicorp’s Terraform and I must say, I’m impressed. In this article, I will like to share my findings and also document what I have learnt for posterity.

PS: I’ve created a bootstrapped Vagrant box with Terraform provisioned. You can clone the repository here and follow along.

Terraform, like every other Harshicorp products have basic commands that can be run from the CLI. But the one most frequently used Terraform command is the terraform apply. This is the command that actually allows Terraform to run and communicate with our provider(more on this later). The terraform apply command goes out to our provider and provision the resources that we have declared in our Terraform files. In simple terms, it builds or changes our infrastructure.

While the terraform apply command is great, the problem is that it doesn’t give you an early feedback of what you’re doing, luckily, Terraform provides another command terraform plan which does just that. It allows us to see how infrastructure execution plan.

The Terraform Syntax - HCL

Terraform’s code is written in Harshicorp’s proprietary language called Hashicorp Configuration Language(HCL). HCL is a structured configuration language that is intended to be both machines friendly and human readable. It’s geared mostly towards DevOps, and in the case of Terraform, its syntax allows us to describe our infrastructure as code. All Terraform codes are written in a file with a .tf extension.

Before we use Terraform and explore its power, we will need to declare a provider. This is the entry point to every Terraform program. As at the time of this post, they are well over 10 different Terraform providers and they include; AWS, Digital Ocean, Google cloud, etc. For a complete and up to date list of providers, visit the Terraform providers documentation page.

Declaring a provider is simple, you start with the keyword provider and provide the name of the provider, e.g; aws, digitalocean, google, etc.

provider "digitalocean" {
    # todo
}

Code I

The code above is how a Terraform provider is declared. That simple. The declaration above by itself does nothing and has very limited use. In order to harness its power and begin communication with the Digital Ocean cloud, we will have to provide a valid Digital Ocean token. This is how Terraform will be able to communicate with our Digital Ocean cloud.

A token can be gotten from your Digital Ocean account, simply log in and generate one. Remember to also keep this token safe and secure as bad things could happen if it gets into the wrong hands. Insert picture of how to generate Digital Ocean token

We will extend our initial code, Code I, and add our DO token.

provider "digitalocean" {
    token = "xxxx-xxxxx-xxxx-xxxx-xxxx"
}

Code II

Our work here isn’t done, we have only told Terraform we want to work with Digital Ocean and nothing more. This, by itself does nothing, we need to create some Digital Ocean droplets. Terraform has a concept of resource, a resource is typically a service offered by your cloud hosting service, a provider in Terraform’s parlance. These resources include but not limited to; VMs(EC2, droplets), Load balancers, database, cache, etc. To create a Digital Ocean droplet, we will declare a droplet resource and pass along arguments like image type—Ubuntu, CentOS, etc, name—hostname of the machine, region—the DO region where we want this resource created, size—size of the droplet.

provider "digitalocean" {
    token = "valid digital ocean token"
}

resource "digitalocean_droplet" "web" {
    image = "ubuntu-16-04-x64"
    name = "web-1"
    region = "lon1"
    size = "1gb"   
}

Code III

The human readable form of Code III declared above translates to “create a 64 bit Ubuntu 16.04 droplet in the London 1 region and give it a size of 1gb and a hostname of web-1.” It’s worthy of note that a few things in the resource declaration above are standard. The resource keyword is standard, digitalocean_droplet is standard, this is how Terraform represent’s Digital Ocean’s droplets—it varies for other cloud providers, for example, aws_instance for AWS’ EC2 and google_compute_instance for Google’s VM. The web word is arbitrary, it serves the purpose of an identifier in this declaration, as such, you can choose whatever name pleases you. The other declaration inside the curly braces are standard and required. For information on how it’s declared for other providers, please consult the Terraform documentation.

With the code above, we have successfully declared a valid Terraform provider and created an associated resource. To run this command see its effect, we will run terraform apply on our console. At this point, Terraform will initiate a communication link with Digital Ocean and if everything goes well, create us a valid and ready to use droplet. This is a rather simplistic way of doing things and we will be diving deeper in the course of this article.

Variables

In code II, we created a Digital Ocean provider and provided it with an API token. While this get’s the job done, it’s not entirely the best way to deal with this problem. This is where the concept of a variables comes in. Like many traditional programming languages, Terraform also has a concept of a variable, albeit declared differently.

In ES6 for insta\nce, a variable can be declared with either the const or the let keyword. For instance

const name = “Bob Jones”
let age = 70

Code IV

But things are a bit different in the Terraform land. Every variable is predicated with the keyword variable followed by the variable name and a set of parameters. Terraform’s variables comes in two different flavours; input and output variables. An input variable is used to send values into a Terraform while the output variable prints result from Terraform to the stdout. Input variables can be sent in a different format; command line, from a file and an environment variable.

To create an assign data to a variable we will start with the variable keyword as seen in Code V. variable “name” {} variable “token” { default = “DO API Token” } Code V.

From the definition in Code V, if we run the Terraform code using terraform apply, Terraform will prompt us to enter a value for the name variable and wouldn’t do same for the second one, token. This is because, in the second declaration, we have provided a default value for the variable which is our Digital Ocean API token as such, Terraform picks it from there.

If we need to use this variable anywhere, we will have to invoke it like this “${var.token}”, so going back to Code II, we can modify the declaration to this format: provider “digitalocean” { token = “${var.token}” } Code VI

The advantage here is that we can use this variable in multiple locations without necessarily repeating the API token itself in multiple location. This provides tremendous power as to how we manage our code.

The output variables follow the same pattern with the input variable with the only distinction being that the output variables uses value in place of default. Below is a sample declaration of the output variable. output “ip” { value = “${digitalocean_droplet.web.ipv4_address}” } Code VII.

The declaration in Code VII tells Terraform to print the IPV4 address of our droplet to the console. So far, we have been able to create a Digital Ocean droplet, which is good, but the problem now is that we can’t ssh into our newly minted machine, which is a major issue and will definitely pose a problem for us as we go. To fix this issue, we will need to add our SSH public key to our droplet. Terraform provides us with an SSH resource aptly named digital ocean_ssh_key. To use this resource we declare it as below:

resource "digitalocean_ssh_key" "default" {
    name = "SSH Key Credential"
    public_key = "${file("/home/vagrant/.ssh/id_rsa.pub")}"
}

Code VIII

With the introduction of the SSH key resource, we will need to link it to our droplet. That way, we can SSH in using our private key. For this to happen, we will have to modify our code in Code III. The code in Code VIII uploads allows SSH public key to our droplet. Also, notice that we didn’t copy and paste our SSH key here, instead, we used a Terraform built-in function called file. The file function lets us read a file from a path. It has a basic syntax of ${file(“path/to/file")}.

If you are using the Vagrant box I provided in this article, I strongly advise that you generate a new SSH key as this box comes without one. Generating an SSH key is simple, simply run this command on your terminal ssh-keygen -t rsa -b 4096 -C “your email” and follow the on-screen information. I’ll strongly advise that you don’t set a passphrase for your SSH key.

provider "digitalocean" {
    token = "xxxx-xxxxx-xxxx-xxxx-xxxx"
}

resource "digitalocean_droplet" "web" {
    image = "ubuntu-16-04-x64"
    name = "web-1"
    region = "lon1"
    size = "1gb"
    ssh_keys = ["${digitalocean_ssh_key.default.id}"]
}

Code IX

Pulling all everything together, we will have something like this:

variable "token" {
    default = "xxxx-xxxxx-xxxx-xxxx-xxxx"
}

provider "digitalocean" {
    token = "${var.token}"
}

resource "digitalocean_ssh_key" "default" {
    name = "SSH Key Credential"
    public_key = "${file("/home/vagrant/.ssh/id_rsa.pub")}"
}

resource "digitalocean_droplet" "web" {
    image = "ubuntu-16-04-x64"
    name = "web-1"
    region = "lon1"
    size = "1gb"
    ssh_keys = ["${digitalocean_ssh_key.default.id}"]
}

output "ip" {
    value = "${digitalocean_droplet.web.ipv4_address}"
}

Code X

If we run terraform apply again, we will have an output similar to this

...

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

Outputs:

    ip = 46.XX.XXX.XXX

Control and Conditional

Unlike procedural languages, Terraform use a declarative language pattern. If you wanted to create a resource say three times, you will wrap them in a for loop, but in terraform, you will use a meta-parameter like count

It’s important to note that the count parameter available to every terraform resource has a zero-based index, similar to arrays in a traditional programming language. So if you had resource declaration like

resource "digitalocean_droplet" "web" {
    count = 4
    image = "ubuntu-16-04-x64"
    name = "web.${count.index}"
}

Code XI

From Code XI above, we will have 3 web servers created with names; web.0, web.1, web.2, web.3. Note that the names argument makes use of interpolation syntax.

If we wanted to check the truthy of something before using it, we could use another conditional which is similar to the itinerary operator in a regular programming language. It follows the pattern condition ? trueval : falseval. Let’s declare a resource that will only be available if a certain condition is met.

resource "digitalocean_loadbalancer" "pubic" {
    count = "${var.env == "production" ? 1 : 0}"
}

Code XII

From the declaration in Code XII, the DO load balancer will only exist if the environment is a production environment. Terraform also support operations like !=, >, <, >=, <= && || !.

Disclaimer

While following the examples outlined in this article, please bear in mind that there’s a cost attached to it. When you run terraform apply and create a real resource at your provider’s end, they start billing you almost immediately. As a word of caution and this apply only in a non-production environment, always try to clean after yourself. For this purpose, Terraform offers us a really handy command called terraform destroy. The terraform destroy command literally goes back to your provider and delete/destroy every single resource that you have created. This is a one-way command and cannot be undone, so I strongly advise you do this in your personal or experimental environment.

This post was originally published on cyberomin.github.io

MySQL Rabbit Hole: Adventure in data recovery

Celestine Omin — Tue, 16 May 2017 07:36:00 +0000

Recently, an old project of mine died mysteriously—I don’t think so, things rarely die mysteriously, something had gone wrong; MySQL wasn’t running and there was a potential data loss. What was more frustrating about this was the fact that I had procrastinated the database replication and backup for the longest of time. This was all my fault.

Over the last couple of months, there have a sizable amount of data that I had collected and I didn’t imagine myself going through the process of collecting these data again and as such, excuses weren’t an option. I had to recovered it with as little information that I had at my disposal.

To begin this process, I needed to first recreate the database on a fresh MySQL instance.

CREATE DATABASE children DEFAULT CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
CREATE USER ’nimo'@'%' IDENTIFIED BY ‘password';
GRANT ALL PRIVILEGES on children.* TO ’username'@'%';

The first mistake made was that I had deleted some files from the /var/lib/mysql directory. In hindsight, my journey to recovery would have been a lot less painful had I backed up the content of this directory, but hindsight, they say is 20-20

I messed up and did the one thing no sane person should do, I ran sudo rm on a production server. It turns out, MySQL needed these files; ibdata1, ib_logfile1, ib_logfile0 to function properly, this is particularly important if you are trying to restore InnoDB table(s). The ibdata1, ib_logfile1, ib_logfile0 files are at the heart of the InnoDB storage engine. The ib_logfile1, ib_logfile0 form the redo log_(a disk-based data structure used during crash recovery, to correct data written by incomplete transactions)_, beyond that, they record statements that attempt to change data in InnoDB tables. ibdata1 on the other hand contain meta data about the InnoDB table, and the storage areas for one or more undo logs, the change buffer, and the doublewrite buffer.

Back to recovery

My only saving grace was that I managed to backup the contents of /var/lib/mysql/database_name. This directory holds the files that make up each table and they are in the *.frm and *.ibd format. There’s also a single db.opt file. The db.opt file stores the characteristics of a database. The *.frm file is how MySQL represents its data on disk. The files share the same name as the table with the *.frm extension and this is irrespective of the storage engine used. The *.ibd files, on the other hand, contain a single table and its associated index.

Let the fix begin.

In other to fix this issue, I had to recreate the database schema—thankfully, my ORM provided me with a really good migration—and I paid particular attention to constraints, especially foreign keys.

As part of the restoration process, I logged into MySQL and disabled the Foreign Key constraints: SET FOREIGN_KEY_CHECKS=0; and also discarded the InnoDB tablespace ALTER TABLE table_name DISCARD TABLESPACE;. I did this for all the newly created MySQL tables. Note that by default, InnoDB stores its tables and indexes in the system tablespace.

System tablespace: One or more data files (ibdata files) containing metadata for InnoDB-related objects (the InnoDB data dictionary), and the storage areas for one or more undo logs, the change buffer, and the doublewrite buffer. Depending on the innodb_file_per_table setting, it might also contain table and index data for InnoDB tables. The data and metadata in the system tablespace apply to all databases in a MySQL instance.

A note about discarding and importing InnoDB Tablespaces

An InnoDB table created in its own file-per-table tablespace can be discarded and imported using the DISCARD TABLESPACE and IMPORT TABLESPACE options. These options can be used to import a file-per-table tablespace from a backup or to copy a file-per-table tablespace from one database server to another.

After the database migration via my ORM was completed and I have completed the ALTER TABLE table_name DISCARD TABLESPACE for all my tables, I restored all the *.frm and *.ibd to the newly created database directory which is located at /var/lib/mysql/database_name.

I then imported the previously discarded tablespace: ALTER TABLE table_name IMPORT TABLESPACE;, like the discarding process, I executed the importation operation for all my tables. After the importation process was completed, I restored the Foreign Key constraint SET FOREIGN_KEY_CHECKS=1;

Are we done yet?

At this point, everything should be fine and there should be peace all round, but no, MySQL has a mind of its own. The first problem was that I suffered an index corruption, upon checking the MySQL error log, I discovered an error

[ERROR] InnoDB: Clustered record for sec rec not found 
InnoDB: index listings_slug_unique of table database.table_name 
InnoDB: sec index record PHYSICAL RECORD: n_fields 2; compact format; info bits 0

An index corruption is quite rare and unusual, they are mostly caused by MySQL bug or hardware failure. The solution to a corrupt index is to run OPTIMIZE TABLE table_name. But in most cases, this will not suffice and you could end up with funny errors like this;

mysql> OPTIMIZE TABLE table_name;
+----------------+----------+----------+-------------------------------------------------------------------+
| Table | Op | Msg_type | Msg_text |
+----------------+----------+----------+-------------------------------------------------------------------+
| database.table_name | optimize | note | Table does not support optimize, doing recreate + analyze instead |
| database.table_name | optimize | error | Invalid default value for 'end_date' |
| database.table_name | optimize | status | Operation failed |
+----------------+----------+----------+—————————————————————————————————+

And this was exactly what happened to me. Taking a closer look at this error, the second line hints on what could be the possible problem invalid default value for ‘end_date’ . Our end_date column is getting an invalid timestamp. This is due to the fact that MySQL is currently set to NO_ZERO_DATE (strict) mode and I was trying to feed it with a timestamp in this format 0000-00-00 00:00:00.

The workaround to this problem is to set the SQL mode to allow for invalid dates: SET SQL_MODE='ALLOW_INVALID_DATES';. The SQL mode here prevents MySQL from performing a valid date check. It checks to make sure that only the month uses a numeric range; 1 through 12 and day uses a numeric range too; 1 through 31. Note that it does not give provision for leading zeros — 01, 02, etc.

At this point, everything seemed to have worked out pretty well. Finally, I restarted MySQL and there were much joy and praise :)

References

MySQL :: MySQL 5.6 Reference Manual :: 14.7.4 InnoDB File-Per-Table Tablespaces MySQL InnoDB lost tables but files exist - Super User

This post was originally published on cyberomin.github.io

A pull request is submitted, what next?

Celestine Omin — Thu, 27 Apr 2017 12:43:00 +0000

A few weeks ago, I wrote an article on the need to hire a tech lead when you consider outsourcing your engineering. This article sparked some interesting conversation and one was about pull requests(PR) and code reviews(CR) and how it should be approached. Today, I will try to talk about code reviews and some of its inherent problems amongst team members, and I’ll also highlight possible ways to overcome some of these problems.

For context, I’ll love to explain what pull request and code reviews are and for the sake of this article, I’ll be using both terms interchangeably.

A pull request(PR) is a method of submitting contributions to a software development project. It is often the preferred way of submitting contributions to a project using a distributed version control system (DVCS) such as Git. A pull request occurs when a developer asks for changes committed to an external repository to be considered for inclusion in a project’s main repository. Source - OSS Watch> Code review is a systematic examination (sometimes referred to as peer review) of computer source code. It is intended to find mistakes overlooked in the initial development phase, improving the overall quality of software. Source - Wikipedia

The problem that I see with most pull request/code review session is when reviewers nitpick on seemingly inconsequential issues, often neglecting the elephant in the room. This not only degrades the entire reviewing experience, it leads to unnecessary bikeshedding. This, in my opinion, is one of the causes of strife within technical teams and it’s unhealthy.

The bike shed story tells of a management committee’s decision to approve a nuclear power plant, which it does so with little argument or deliberation. The story contrasts this with another decision on choosing the color of the bike shed where the management gets into a nit-picking debate and expends far more time and energy than on the nuclear power plant decision.

A pull request should be a learning experience, it’s an opportunity to get good feedback from trusted team members. This feedback could range anywhere from writing more structured and succinct code to better optimisation of algorithms, queries, etc. It’s the time to be open minded and not become unnecessarily defensive.

One issue that I find quite worrying is the fact that most authors come with girded loins when they submit their PRs. You can’t tell them otherwise, after all, they have spent a significant portion of their day solving this complex problem and all they need is your approval and a merge to the base branch. And this is a big problem.

But, as an author of a PR, before you put on your armour, take a second and think through the opinions of your reviewers, and look at the issue from their own lens. Your reviewers are here to uncover your blind spot, they check those things that must have slipped through the cracks, they are meant to guide and help you become better. When authors approach their PR/CR session with the way they would approach an editor, they become aware of things they must have overlooked, uncover new paradigms and fundamentally gain new knowledge. Your reviewers, like editors, are your third eyes.

Are your reviewers always right? Far from it. Your reviewers, like every human, aren’t infallible. They will make mistakes, and it’s your place as the author to help guide them to the obvious and help them understand your thought process to the problem and what led to your solution. One of the problems I have seen with PR sessions is when reviewers are in haste to prove a point without first, understanding the problem domain. They make hasty conclusions and in most cases, miss the main point by a stretch.

As a reviewer, your first task is to understand the problem domain and offer suggestions as opposed to calling people out. This is one of the reasons most authors become defensive. Don’t do that, you don’t want to be that guy. While reviewing, If it helps and where possible, move over to the author’s desk or get on a call and share screen, then you can ask questions as to why the author made some design choice. This does not only reduce any unforeseen tension, it allows you to understand the problem and solution a lot better, see things from author’s perspective thereby giving you an opportunity to give constructive feedback. The other advantage to this approach is that it create’s the sort of bond that will make people more receptive to your ideas.

Authors become defensive when comments left on their PRs make them look stupid. There’s an ego in every human, never forget that. A code review session is a win-win for everyone. The reviewer(s) and author both learn and benefit from the experience. Put extra thought and care into crafting your feedback, they should be suggestive and also offer a better way of solving a problem.

Remember, when a code review session degenerates into a pissing contest, its essence is lost and everybody, both the reviewer(s) and the author loses out big time.

This post was originally published on cyberomin.github.io

So you want to outsource your product engineering?

Celestine Omin — Mon, 03 Apr 2017 10:00:00 +0000

In the last couple of days, there have been many talks about developer brain drain and capital flight from the Nigerian tech ecosystem. I’m mostly going to focus on the latter.

In the life of a new technology startup founder, a lot is happening; getting that MVP into the hands of users, investors meetings, optimising pitch deck, product managing and generally being the jack of all trade. Not a bad thing if you ask me, considering the founder is just starting out and doesn’t have the cash to splurge.

One thing that worries me is the recurring theme of, “the average Nigerian engineer isn’t good enough, charges an arm and a leg and will never deliver on time. So, if you want something great, outsource it.” While I have absolutey no problem with this, I mean, I can’t tell you how best to spend your money, I’m worried that most startup founders are mostly concern about the end product and blindsided about what happens behind the scene. For the majority of the founders, as long as when you click that button a modal box shows up and an event is fired behind the scene, all is good. This shouldn’t be the case.

This post isn’t my way of supporting outsourcing or not, like I earlier mentioned, the money is yours, go where you get the most value. I’ll try to highlight one major component founders should consider before outsourcing; the role and the need of a technical lead.

Having those “expected” events—fire an email after a new user register—occur when a user interacts with your product isn’t good enough, the long-term plan of the code that powers this product should be something that concerns you. How do you solve this? Get a technical team lead for your outsourced project.

From a very high level, it sounds counter intuitive to get a technical lead when you outsource, considering the fact that these developers and code shops should know what they are doing. Sadly the reverse is always the case.

Most freelancers and code shops are just here to get it done at all cost. They sacrifice best practices, put little thought in the development process and user empathy isn’t something they could be bothered about. They push out features and just move on to the next project. Quality, epsecially code quality isn’t something they worry about, this is partly not their fault. How do you explain to a client that you spent half your day setting up a continuous integration environment and you will be spending just as much time writing unit tests on product features. To most founders, this doesn’t make sense. But these things are super critical, especially when you think about these products from a long-term perspective.

Getting a technical team lead isn’t a waste of resources and it’s a really valuable investment. This person, if they know their onion, will work to save you untold heartaches many months down the road. They will be most likely be responsible for the technical decision that happens in the life of the project.

It’s the tech leads place to advise the startup founder on best practices. He or she drives the product engineering.

Why is this important?

One time I was helping a friend evaluate the code for his product. The company he had outsourced to claimed they had four engineers on this project when in actual reality they had just one person slaving away on the product API and the UI/UX component. It was easy to tell, all I needed to do was look at the git history/logs, it turns out only one person was making commit. And to make things interesting, all these commits happened on the master branch—no feature branching—which also explains why there was no pull request. I mean, how do you even PR(pull request) and code review yourself? Doesn’t make sense. You don’t set your own exams and grade yourself afterwards. The saddest part about this whole thing was that he paid for four engineers at $2,500 per pop. Not cheap.

The company he had outsourced to, didn’t only make terrible technical decision, they just didn’t care. While going through the source code, I discovered they were saving passwords in PLAIN TEXT. Yes, plain text passwords in 2016. I almost had a heart attack when I saw this. I quickly flagged it and they came up with a nice excuse to back their decision. They used really old and outdated versions of frameworks, frameworks that had no LTS(long term support). This one had cascading consequences, essentially they will be missing out on bugs and security fixes not to even mention optimisation.

This same company had no concept of a staging server. Everything went straight to production. No filter. I guess they just wanted to show progress and cared about only that. They also deployed a NodeJS application without a process manager or a reverse proxy.

The concept of using configuration management and deployment pipelines sounded alien to them. In essence, when they were ready to deploy, someone will manually SSH(secure shell) into the server and run a git pull and a new version was out in the wild. The problem with this method is many folds. a) rollbacks were difficult if not near impossible. b) If a server goes down at any point in time, someone had to manually create a fresh VM(virtual machine) and manually install all the packages needed to run the service. 3) if there was ever a sudden traffic spike, the system will collapse. They had everything in one machine; database, API and the UI, high availability was never in the plan. And yes, your guess is as good as mine, no one was replicating the database. If that node went down or data was lost, there is almost no way to recover and this might just be the end of my friend’s startup. I can go on and on about things that didn’t go well.

All of these discoveries were made after about a 30min conference call and spending another an hour on the project repository. As you can imagine, the code shop didn’t like me. I instantly became a torn on their flesh. That troublesome Nigerian.

After these discoveries were made, I told my friend and the code shop about the changes they needed to make, the code shop instantly fired back that these changes will not only impact the project timeline, it will also cost an extra fee. Eventually, the relationship between my friend and the code shop went south and he is trying to rebuild his project; wasted time and resource.

I cannot over-emphasize the need for a technical team lead, this person will make your life a lot easier and you will have a peaceful night rest. In my friend’s case, if he had hired a technical lead from the inception of the project, I am certain things would have gone a different route.

Dear startup founder, don’t just worry about those nice features, think also about what goes on behind the scenes. A house with a nice paint work and construction practises will begin to show its flaws over time, especially when subjected to stress.

Finally, get a technical team lead. Your will be glad you did.

This post was originally published on cyberomin.github.io

Docker Hack: Prune those unwanted stuff.

Celestine Omin — Sat, 01 Apr 2017 01:00:00 +0000

I have been meaning to do this for a very long time and today, I overcame inertia. This is a first in, hopefully, a never ending series that I have aptly titled bits. The aim of these series is to share my discoveries, hacks and shortcuts with everyday tools that I work with. Sit back, relax and enjoy the ride.

In this first episode of the bit series, I will be sharing a Docker hack. Docker is an amazing piece of software, like every other tool, it has its ways and as engineers, we have to learn those ways. This article isn’t meant to talk about what Docker is and is not, the aim here is to showcase one tiny feature I found this week.

As we go around spinning containers, images and volumes, we run the risk of filling up our hard drives very quickly. Luckily, as at version 17.03.1, Docker provides us with a tiny nifty command called prune. Here is how it works.

Usage

docker system prune [OPTIONS] - Remove unused data

Options

--all, -a - Remove all unused images including the dangling ones.

--force, -f - Do not prompt for confirmation just do it.

The command above clears out everything from containers to images and including volumes. While this may not always be what you want, Docker provides us with a more granular version of the prune command: container, image and volume pruning.

docker container prune [OPTIONS] - Remove all stopped containersdocker image prune [OPTIONS] - Remove all unused imagesdocker volume prune [OPTIONS] - Remove all unused volumes

For all three commands above, the only available option is --force, -f. The --force, -f simply ask Docker to proceed without confirmation.

I recently ran docker system prune -a -f and this was what I recovered almost 25GB of hard drive space.

PS: Used these commands carefully.

This post was originally published on cyberomin.github.io

Iptables: An Introduction

Celestine Omin — Wed, 25 Jan 2017 01:00:00 +0000

Server security is one of the tiny elements in application development that cannot be ignored. It can make or mar you. As such, giving adequate attention to it cannot be overemphasised.

In today’s world where hackers derive delight in taking down full service or data centres, it becomes important to keep this bit at the top of the list when provisioning a new server or deploying to an existent one. While patches and security updates are just as important, they wouldn’t do so much in some cases—especially around coordinated attacks like DDoS.

For a very long time, setting up application firewalls has been one of the ways of securing servers. IPTables, a program that defines a set of rule showing what packet can go through what port in a server, has been favoured.

What is IPTables?

According to Wikipedia, IPtables is a user-space application program that allows a system administrator to configure the tables provided by the Linux kernel firewall (implemented as different Netfilter modules) and the chains and rules it stores.

IPTables exist in many flavours of Linux. Among all of the open-source firewalls available today, it’s the standard firewall bundled with most Linux distribution such as Ubuntu, CentOS and Fedora.

A standard IPTables is generally broken down into 3 units otherwise known as chains. These chains are INPUT, FORWARD and OUTPUT.

Chains INPUT - This chain controls the behaviour of incoming traffic. If a user is accessing a website on a server, the IPTables will try to match the incoming request to a rule that has been defined in the INPUT chain. This rule will typically check what the destination port, port 80, says about web traffic.

0 0 ACCEPT tcp -- any any anywhere anywhere tcp dpt:http

OUTPUT - This chain manages every outgoing connection. Typically, all traffic generated by the host server e.g a ping request.

FORWARD - This chain controls all network packets routed through the server.

Before we go any further, it’s important to highlight the 3Ps of IPTables: Packet, Port and Protocol. Packet - A block of data transmitted across a network. Think about a packet like the mail delivered by a mailman. Port - A port is a logical connection place. It has a numerical designation; 80, 22, 443, etc. Protocol - A set of rules governing the exchange or transmission of data between devices. E.g: TCP, UDP.

Basics

To setup a rule, we run the following:

sudo iptables -A INPUT -i lo -j ACCEPT

The command above reads “ACCEPT traffic on the loopback interface on the INPUT chain. Loopback interface here is a virtual networking interface that a computer uses in communicating with itself, hence the term, loopback.

To check the loopback interface on a Linux machine, run

ifconfig lo.

To set up iptables rules for HTTP traffic we will run

sudo iptables -A INPUT -p tcp --dport 80 -j ACCEPT

sudo iptables -A INPUT 3 -p tcp --dport 80 -j ACCEPT

“ACCEPT all HTTP(tcp) traffic which is going to a destination port 80 where -p is protocol, -j is jump and --dport is destination port and -A is append.

The second rule above looks almost exactly like the first rule, the only difference here is that the second rule specifies the location the rule should be added in the rule chain. This is important especially when you want your rules to be matched sequentially.

So far, we have seen how to accept traffic to a particular port. While this is great, it’s also important that we set rules to drop unwanted traffic.

sudo iptables -A INPUT -p tcp --dport 443 -j DROP

The rule above says drop every HTTPS traffic hitting our servers. If you pay attention closely, the rules for accepting and dropping traffic closely resembles themselves, the only exception here is that while accepting traffic uses ACCEPT, dropping traffic uses DROP.

While DROP works quite well, in most cases, it helps to give feedback to the user on what’s happening. REJECT on the other hands drops the traffic and also provide useful feedback. To set up a reject rule, we run

sudo iptables -A INPUT -p tcp --dport 3306 -j REJECT

When a client tries to access this port, they will get a message saying that connection to port 3306 was refused. This particular rule prevents an external client from connecting to the MySQL database.

Viewing and deleting rules To view existing IPTables rules, we will runsudo iptables -L -n or sudo iptables -L -v where -v means verbose.

Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target prot opt in out source destination

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target prot opt in out source destination

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target prot opt in out source destination

While setting up rules could be all fun and great, there comes a time where you need to delete an existing rule; either because you don’t need the said rule again or it was configured wrongly. There are two ways to deleting a rule:

Method 1

sudo iptables -D INPUT 3

Delete the 3rd rule from the input chain. The numbering here isn’t 0 based, but it starts from 1.

Method 2

sudo iptables -D INPUT -P tcp --dport 3306 -J ACCEPT

Delete the exact same rule by its parameters.

While the two methods achieve a common goal– deleting a rule–it’s important to pay attention to the difference. Method 1 deletes just about any rule that’s on the third chain. In a case where the index does not exist, an error message is returned. Method 2 is a lot more specific, albeit verbose. it targets only a particular rule and the risk of deleting another rule by mistake is minimal.

Automating IPTables with Ansible When it comes to configuration management, ability to build systems in an idempotent and a repeatable fashion cannot be overemphasised.

Ansible is one of the few tools out there that simplifies configuration management, orchestration and deployment. It allows you to build and configure servers in a predictable manner. After taking the time to learn the basis of firewalls via IPtables, we will attempt to automate the entire process.

Let’s create an Ansible playbook to automate our IPtables configuration. In this playbook, we are going accept connections to the following port 22(ssh), 80(http), 443(https) and 3306(mysql) and we drop every other request that does not match this rule.

If you’re new to Ansible and configuration management, there’s an excellent primer here - http://docs.ansible.com/

***
- name: Install iptables Persistent
  apt: name=netfilter-persistent state=present update_cache=true

- name: Set Loop Back Rule
  iptables: chain=INPUT action=append in_interface=lo jump=ACCEPT comment='Accept all loop back traffic'

- name: Set Established Connection Rule
  iptables: chain=INPUT ctstate='ESTABLISHED,RELATED' jump=ACCEPT comment='Let all established connection stay'

- name: Set SSH Port 22 SSH Rule
  iptables: chain=INPUT jump=ACCEPT protocol=tcp destination_port=22 comment='Accept all SSH traffic'

- name: Set HTTP Port 80 HTTP Rule
  iptables: chain=INPUT jump=ACCEPT protocol=tcp destination_port=80 comment='Accept all HTTP traffic'

- name: Set HTTP Port 443 SSL Rule
  iptables: chain=INPUT jump=ACCEPT protocol=tcp destination_port=443 comment='Accept all SSL traffic'

- name: Set HTTP Port 3306 SSL Rule
  iptables: chain=INPUT jump=ACCEPT protocol=tcp destination_port=3306 comment='Accept all MySQL traffic'

- name: Drop Any Traffic Without Rule
  iptables: chain=INPUT jump=DROP comment='Drop traffic for rules that did not match'

This post was originally published on cyberomin.github.io

2016: Obligatory year in review

Celestine Omin — Sun, 08 Jan 2017 09:20:00 +0000

I’m late to the party, I know, but this isn’t a me-too article. This, in many ways, is a reflection and thanksgiving.

This is my first post for 2017, so it makes sense to recap my 2016; both the high points and the low. 2016 was a great year, monumental and life changing on different fronts. So great, I probably didn’t want it to end. Alas, all good things must come to an end.

In no particular order, I’ll try to share a few of the events that took place in my life.

Work & Life

I switched jobs and moved to Andela. It’s been a little over 4 months, and boy, it has been an incredible experience.

In this same year, I lost my last surviving grandparent, Eka Anthony. I miss her to this day. Eka Anthony was the kind of grandmother every child prayed for. God bless her sweet soul.

It wasn’t all bleak, I also became an uncle to an amazing young man, Jessy. In this same year, Lee showed up. Great guy.

I learnt how to drive. Lagos is unfair to those who are getting behind the wheel for the first time. We move (literally).

I finally graduated from the University—big deal.

Speaking & Events.

In November, I spoke at my home church, Honey Streams Christian Center. This is one privilege I’ll never take for granted. Thank you, Pastor Akomaye. Thank you, sir.

I was a panellist at the AfriLabs conference that held in Accra. It was amazing and humbling sharing the stage with industry heavyweights from Microsoft, etc.

I met the amazing community called DevCongress. Prosper and I had the rare privilege of sharing our journeys as engineers, while talking about what makes startups in Nigeria unique. Edem Kumodzi made this possible. Thanks, Chale.

I got an invite to speak at TEDx Unilag, but I somehow managed to botch this one. Poor planning. Who knows, I may get another one this year. I want to speak at a TED event. If you’re a TEDx licensee and you’re interested in having someone share his life story, do reach out.

I spoke about Progressive Web Apps at the DevFest SE season 2016 that held in Port Harcourt. I made a case for PWAs as the new way of building mobile applications moving forward.

I was at HiveCo Lab, Kampala, where I got the opportunity to meet with amazing engineers and feel the pulse of the Uganda technology scene. There, I met Mwesigwa Daniel. Daniel is an amazing guy. He still owes me a Rolex though.

I met egbon Mark Zuckerberg. He still owes me a photograph.

I delivered the Keynote at the DevFest SW event. I spoke on the State of the Web. It was beautiful and nostalgic. For one, it afforded me the opportunity to learn about the history of the web and also examine the lives of those who made it possible. Thank you, Sir Tim Berners-Lee.

The high point of my 2016 speaking engagement was when I delivered the keynote for the CodeCalabar conference. Calabar isn’t just a city. It’s my city. My autobiography will not be complete if I don’t include this city. Calabar is my genesis.

I was a guest on Hit FM 95.5, Calabar. I spoke about technology and how it affects our everyday life. I made Papa proud.

Writing

I set out to write once every week, but I wasn’t disciplined enough to follow through on this one. I take absolute responsibility for this. No excuses.

In the midst of all of life’s happenings, I managed to scribble a few things. My biggest moment was when I shared my 11-year journey. For some weird reason, it was therapeutic. I could feel the weight off my chest. What was more interesting about this particular article was the fact that I had people emailing me and sharing their own university stories. I had the privilege of counselling with a few of these individuals.

In a bid to step away from my comfort zone—technology and startups—I decided to try my hands on fiction. I wrote about an upwardly mobile couple, Bidemi and Makinde, who had planned this amazing dinner date that almost turned into a disaster. This story isn’t complete, so I may revisit it sometime this year.

I now have renewed respect for every single person who writes fiction for a living or for fun. Fiction is tasking, but it also allows your mind wander and sets your imagination wild.

I did my best to appeal to young people on the need to write complete words and sentences. This particular article was borne out of the fact that I was tired of either ignoring people that start conversations with xup or those that will abuse your senses with tanz 4 ur tym. This gripes me, always.

I made a special appeal to UXers on the need to consider my grandmother when next they are thinking through that app that will connect the next 1 billion people. She, like many other grandparents are just as important.

I contributed to Ventures Africa, iAfrikan, Techpoint and Y!Naija. This year, I’m aiming for Bloomberg, Financial Times, WSJ, and NYTimes. Amen. A boy can dream and dreams do come true.

Reading

I read a number of books this year and here are some of my favourite:

Traveling

I visited 4 African countries this year; Kenya, Uganda, Ghana and Rwanda (coughs, it was a 2 hour layover). East Africa is beautiful. Yes, I said that.

I got the rare opportunity to taste a crocodile meat—thanks, Lisbi. In a bit to document my culinary experience, I started a trend I tagged culinary journeys—totally lifted from CNN’s show, Culinary Journey. I intend to follow suit this year as I encounter amazing dishes that life brings my way.

Edem was in Lagos last December, we both had Jollof at Terra Kulture. I guess we can finally lay the Nigerian/Ghana jollof squabble to rest.

Learning

This is one year that I decided to go under the hood and learn the inner workings of a few things. I took the time to explore the not-so-sexy side of MySQL. I’m by no means a database expert, but to say this experience wasn’t invaluable will be me lying.

Elasticsearch has been that one software that is not only beautiful and well designed, it most times felt magical. I did a few digging and you have to give it to the folks at Elastic, they do amazing work.

Ansible, how did I ever exist without you in my life? To RedHat and the amazing community maintaining this project, I say thank you.

2017

The idea is simple, multiple everything here by 10.

In closing, this was one beautiful year. Here’s to a greater and bigger 2017. Cheers.

Image credit: PLURALSIGHT

This post was originally published on cyberomin.github.io

Building the African Software Engineering Ecosystem

Celestine Omin — Sat, 17 Dec 2016 01:04:00 +0000

In Steve Case’ book, The Third Wave: An Entrepreneur’s Vision for The Future, Steve outlined the different waves that defined the Internet and laid the foundation for the explosive technological innovation that we experience today.

The first wave — building the internet — , which lasted between 1985 - 1999, was largely driven by People, Products, Platforms, Partnerships, Policy and Perseverance and he outlined a few companies that were pivotal to the success of that era; Cisco, IBM, Apple, AOL, Sprint and the now defunct Sun Microsystem.

The second wave — App economy and mobile revolution — started in 2000 through 2015 and this wave was driven by People, Products and Platforms. The third wave — Internet of Everything — started this year, 2016 and it’s ongoing.

I’m more interested in the first wave and how it relates to the Nigerian technology ecosystem. I’m really interested in the People aspect of things. Yesterday, I started a series of tweets about how technology startups in Nigeria and Africa at large can bolster the engineering community around us. While other components that makes for a successful ecosystem are equally as important, I choose to focus on engineering because not only is it my forte, it’s one of the critical components of every successful technology startup.

Engineering is a major differentiator between successful startups and the average ones. As Jeff Bezos rightly puts it, it’s what separates the A players from the B players.

But in all of this, how are we as a community faring with all things engineering? How much are we giving back as individuals and organisations? Is this something tech startups in Nigeria and Africa actively think about?

I love how companies in the valley, Europe and South East Asia are deliberately building their engineering culture. These companies do this through a couple of ways; tech talks, engineering blogs, open source, etc. They share their problems, pain points and solutions.

This is one surefire way of building a community. A community is not people doing things in silos and calling it an “ecosystem.” Netflix holds tech-talks and invites Uber engineers to speak. They move this further by sharing core components of their infrastructure.It’s only in sharing that we can truly build a sustainable ecosystem and also move the needle a notch higher.

https://kafka.apache.org/, one of the core components of distributed computing was built in-house by LinkedIn. They shared it and everyone is benefiting today. For a second, imagine if Google kept Angular and Kubernetes to itself and Facebook kept React to itself. Think about it.

A few days ago, Neo Ighodaro, CTO of Hotels.ng, tweeted about Watch Dog, a monitoring tool they built in-house that sends critical alerts to Slack and every stakeholder. Not only did they write about this tool, they went ahead and made it open source. To say I was excited will be an understatement. That tweet literally made my day.

Just published “Meet Watchdog; Hotels.ng’s in-house open source application monitor” https://t.co/iiFizO41gQ cc @lyndachiwetelu @markessien

— Neo Ighodaro (@NeoIghodaro) December 12, 2016

While the Nigerian and African tech needs funding, the need for actual engineering collaboration and sharing cannot be over emphasised. If you’ve solved a major problem; optimisation, security, scalability, experimenting with a new tech, write about it. Share.

While I’m not oblivious to the Dunning-Kruger effect, you will be surprised at how many people will learn and benefit from this gesture. Building out your own CI/CD pipeline? Write about it. Wrote an amazing configuration management for your servers? Share it. Talk!!! Found a new way to minimise latency by 3%? Write about it. Talk. Mitigated a DDoS? Write about it.

We need more African tech startups writing about their processes, product design methodology, software engineering practises, opening up their APIs. More startups holding community events to talk about technologies they are exploring or currently using in production. Enough of gathering people for product pitches.

In 2017, there should be just as much tech talks and articles. I think we have enough fundraising articles.

PS: I recognise and celebrate the great work of foorLoop, GDGs, African Git Meetup(shameless plug, that’s me) all of the individual contributors to OSS and everyone writing one technical article or the other, you’re all rockstars.

Hotels.ng, CcHub thank you for giving out your spaces. We need these supports.

This post was originally published on cyberomin.github.io

Chapter Three: The 11-year journey.

Celestine Omin — Mon, 07 Nov 2016 07:16:00 +0000

Winners don't make excuses.
Harvey Specter, Suits

In the hit TV show Suits, produced by US Network, Patrick Adams played the role of Mike Ross, a brilliant college dropout turned unlicensed lawyer. Mike Ross, a man who came from almost nothing went on to build a stellar career at Pearson Specter. Mike rose through the ranks very quickly, he made junior partner in the shortest period of the firm’s history. His brilliance and photographic memory would make even Einstein jealous.

He was everyone’s favourite, senior partners would typically fight amongst themselves to have him come work for them for two weeks. One time, his boss, Harvey Specter, a senior partner at Pearson Specter placed him as a bet with his arch-rival, Louis Litt. Such was the awesomeness of Mike. He was loved by everyone, the paralegals marvel in awe whenever he speaks. How can this man be a fraud?

Mike’s chance at Pearson Specter and his subsequent rise was pure happenstance. Prior to his law career, Mike Ross made a living selling marijuana and taking LSAT for law school hopefuls. On the day of what turned out to become a life-changing event, Mike was going to a hotel to follow up on a drug deal when he ran into the Feds, he got chased through the hallway and stairs and he eventually found himself in a room where Harvey Specter was conducting interviews for a new Associate.

Harvey, cocky as the king of spade, decided to interview Mike, even though he wasn’t a shortlisted candidate. Mike picked up a book in front of Harvey and dared him to open to any chapter and quiz him from those pages. Harvey, a bit mystified, obliged Mike. He opened a random page and started reading out a citation, he had barely completed the case title when Mike swooped in, completed the title and proceed to read out the court proceedings from memory. Harvey was blown.

Mike explained to Harvey, that he never graduated college let alone graduate from Harvard, as it was the tradition for Pearson Specter to hire only Harvard law grads. There and then, Harvey made an exception and hired Mike immediately. Mike’s life changed forever.

Why am I telling you all this?

Suits is the first TV show that I have ever followed religiously, this is largely due to the fact that the protagonist and I share a similar story.

In 2011, I moved to Uyo and this will become my new home for the next 9 months. On a certain beautiful Friday afternoon, I was boiling rice to eat with the stew mum had sent me. As I watched the grains of rice dart all over the pot as the meal cooked, my phone rang. I didn’t know the caller, but something struck me, the phone number pattern was rather unique. It felt like the owner had asked the network provider to specifically reserve him the number. On the other side of the line was the voice of a man with a deep baritone and he spoke eloquently.

The caller introduced himself as the CEO of DealDey and asked if I had 5 minutes to spare. For a second, my heart skipped, I was totally unprepared. I don’t quite remember what I mumbled. We spent the next 30 minutes or so talking about life and work. Then he asked the question, “what did you graduate with?” there was a short momentary silence, then I blurted, I didn’t graduate college.

For context sake…

You see, I was supposed to graduate with my colleagues back in 2009. But somehow, I managed to get my priorities mixed up and I ended up in a really bad state academically. My folks didn’t know about this. I was even too scared to talk to them about it. I guess, somehow, they had imagined everything was well and there was no cause for alarm, this was largely due to the fact that back in secondary, I was, for the most part, a straight B student. Most terms, I oscillated between As and Bs, but I didn’t dip below Bs. I figured how to hack the exams — cram, write and pass. It wasn’t rocket science.

I studied Computer Science back in the University and this was by choice, not an accident. The first time I came close to a computer was at the ripe age of 9. I didn’t touch one until I was 15. The moment I sat in front of one, I fell in love, so madly in love, I applied to study Computer Science for my bachelors. I was excited. To me, studying computer science was a way of formalising this new found love, I figured since I’d fallen head-over-heels for this thing, studying for a degree in it would be a walk in the park. I was wrong. Very wrong. I was in for a rude shock.

The shock.

I was so happy to be in school for my first class, I arrived 30min to the commencement of my first lecture. I was ready. This young, naive 17-year-old was finally ready to take on the world and nothing was going to stop me, but the Nigerian University shocks you.

My first real shock was when I saw the student handbook for the entire program duration. I was never going to touch anything computer network related until the second semester of my final year. Database management system was slated for my third year. For my freshman and sophomore year, I was going to study GWBasic, Pascal and FORTRAN. These were between 2006 and 2007.

As the weeks grew into months, so did the frustrations snowballed. Schooling was anything but fun and school work felt like chores. I dreaded exams. What started out as a love story was fast becoming the bane of my life. I was exhausted. Every iota of motivation in me fizzled out. Frustration and anger set in. I questioned my own sanity one too many times.

On a certain day, after lectures, I asked the HOD why we had to study decades-old technologies and concepts that had no place in today’s world, his response was “go and complain to the NUC.” I guess this was the day I lost every single interest in school. I attended classes merely for formalities. I craved holidays more than anything else. Prolonged and incessant strikes didn’t help too. I loved them, a part of me secretly prayed for them. To me, this was my own way of running away from this torture called school.

By my third year, I probably lost count of how many resit I had to go through. I didn’t care, I just wanted to fulfil the mandatory four years and move on with my life. “School isn’t adding so much to me so why even bother,” this was the lie I told myself and I believed my own lie. Looking back, I blamed myself. I could have applied myself a little more and go back to my good old trick; cram, pass and move on. While all of these were going on, I applied myself to what I knew the real world needed. Safe to say I taught myself the real computer science. I had people I admire and these people went out of their ways to support me. The God-sent folks at MIDI(Maximum Impact Digital Institute, Calabar) gave me their resources without holding back; computers, Internet, power, space, etc.

I buried myself in every single material I found meaningful, I read everything I could find. Everything. I spent most nights and days at a cyber cafe not too far from home so much so that the cyber cafe operators knew me. Google was and still is my friend. I Googled almost every concept that didn’t make sense to me. I downloaded materials and devoured them…literally. This for me was my only ticket out. If I didn’t do well academically, I should have a market ready skill. Looking back, I wish there was Andela.

By final year, I had lost so much interest in school it didn’t even make sense going anywhere near the school gate, let alone attend classes. I suddenly found myself with so much free time and I was determined to make the most out of them. I found a small library not far from home and this was my new school. Then one day, while talking to my pastor(Pst Akomaye Ugar), I mentioned to him that I’d dropped out of school. The man left all that he was doing and spent the next hour counselling with me. He encouraged me to go back to school and complete what I had started. He checked in with me throughout the semester to see how I was fairing, and that semester turned out to become my best semester ever. I made my best result. I probably averaged about 3.2 GPA. But I still had issues, years of multiple resit quickly dissolved this score and it made very little impact in my CGPA. I’d spend the next 7 years cleaning my own mess. The mess I had created with my own hands.

I got hired…

A month later I had the call with Sim, I made the move to Lagos, this time, I wasn’t visiting, I was here to stay. To make things more interesting, I got a one-way ticket. No turning back. Sim hired me without a university degree. Amazing. I’d go on to spend the next 4 years working at Konga and doing almost everything; from stock count to writing software. But I still wasn’t fulfilled, the lack of a university degree was taking its toll on me. I’d told a few colleagues a couple of times that I never graduated but for some reason, they just didn’t believe me. I turned it into a joke, I’d usually tell people I sat at the back of the classroom, seriously, this part was true.

While still keeping a full-time job, I made a few trips back home a couple of times and for each trip, I manage to take a resit exams and just generally try to improve my odds of graduating. It wasn’t fun. It was exhausting. I confided in Nneka and she became not just my sounding board, but a great confidant.

Nneka became my Rachel Zane, Mike Ross’ fiancée. Times when I was way too tired to even try again she’d encourage me to give it one last shot. One shot became two shots, three shots and so on. On the 4th of January, 2016, after 7 tortuous years, I finally graduated. I got a certificate with a beautiful 3rd class well embolden with what I will describe as the most amazing typeface I have ever seen. I made a 1.54 CGPA.

Sim was my Harvey. Thank you, Sim.

Suits, isn’t just a show. It speaks to me. Suits is déjà vu all over again.

PS: If you’ve had a similar story, go ahead and share it in the comments. Be a source of hope and inspiration today!!

Share this post with your network. Someone may just find another reason to try.

This post was originally published on cyberomin.github.io