DEV Community: Hugo Dias

Designing software for the enterprise - Pt.1 Reliability

Hugo Dias — Sat, 07 Mar 2020 12:45:12 +0000

Building software for the enterprise is hard. Many things go wrong if the important decisions on the architecture of the system were not previously planned.

One of the enduring problems related to this kind of software is how to manage data since such applications meant to grow fast and produce large datasets in a few months.

Since the application needs to be constantly evolving, it should be reliable, must be scalable and easy or not too hard to maintain.

In this post, I want to focus on the first pillar, reliability.

Reliability

You may have an idea of what reliability means when it comes to software.

The concept is usually associated with fault-tolerant systems or systems that can tolerate failures and keep things running from the user's perspective.

Note that a fault is not the same as a failure.

A fault is usually defined as one component of the system deviating from its specs, whereas a failure is when the system as a whole stops providing the required services to the user.

Because of that, we should design systems that can tolerate-faults in three different areas, hardware errors, software errors and human errors.

In this particular section, I want to focus on one of the main reasons for software outages: Human Errors.

A study found that only 10-25% of outages were caused by hardware or network failures, and the other 75-95% were caused by configuration errors by operators, or in other words, humans.

So if human errors represent the majority of system outages, what can we do about it?

Well, most software is usually designed by humans - well, at least in 2020, not sure the next 5~10 years - and humans know not to be very reliable.

It's not our fault, we just can't sustain the same work quality level every single minute.

To solve that problem we need to design mistake-proof systems.

So instead of trying to cure failures, you should start preventing them:

Have a solid code review process. Bugs can be found on a good code review process. Those can prevent stupid things that people usually do when coding for several hours or days.
Make sure to have integration tests covering important features of the system. This kind of test makes sure that real-world business rules remain intact during the evolution of the system.
Look for design patterns to build high-level abstractions to prevent people from changing low-level functions.
Cut off production database access or restrict permissions to important services. Instead, build a CLI (Command-line interface) or an Admin interface so they can do their work without screwing things up.
"Adding human resources to a late software development project makes it later." - Brook's Law - So instead of hiring more people, try installing a solid process such as continuous delivery and continuous integrations to deliver more value in less time with more safety.
Provide sandbox environments so people can test configurations and new features before actually going to production. Providing a docker setup for developers that has the same configuration as the production servers will prevent configuration errors.
Use rollout strategies to prevent mass outages. It should be easy to revert configurations or deployments before they can cause a wide system failure. Blue-green deployments are essentials. When it comes to databases, make sure to use migrations to quickly revert things.
"A complex system that works is invariably found to have evolved from a simple system that worked. A complex system designed from scratch never works and cannot be patched up to make it work. You have to start over with a working simple system." - Gall's Law
Use logs to keep track of systems' health, together with monitoring and alerting services. Softwares as New Relic, Prometheus, Grafana, CloudWatch and PagerDuty combined can give you a clear view of your application and infrastructure health, as show you early warning signals of failures.

So, to sum this up:

The reliability of your system depends on how bug-free it is, how good you are at monitoring it, and how well you have protected against the myriad issues and problems it has. - Jay Kreps

Recommended resources

How to work efficiently as a freelance developer

Hugo Dias — Sat, 05 Oct 2019 20:30:45 +0000

Originally posted in my blog

I work as a software engineer for 12 years now. Most of that time I’ve also been a freelancer.

Basically, I do freelance for 2 things: get extra income and to have a different perspective of building things.

When I’m at the office, my activities are limited to code, tests, sprints, reviews. Everything that a software engineer should do.
Everything changes when I’m doing freelance.

I need to handle lots of things that are not related to code. I need to define what needs to be built, do some basic wireframing, design, code, test, deploy, analyze interactions, announce changes, and keep the user engaged.

Looks awesome, right? The thing is, I have a daily job.

So how do I manage to work efficiently as a freelancer?

Here are 13 rules I've developed over time to get the best quality work
when doing freelance.

About Freelancing

—

I learned the hard way that it’s not a good idea to work in more than 2 projects at the same time as a freelancer.

Last time I did this I end up overwhelmed with a ton of work and the result was pretty obvious. Both projects were delivery late. — Just don’t do this.

—

I prefer working on long-term projects.

Most of my projects last between 100~200h work. This makes me comfortable to plan and execute the whole project without the need to rush.

—

I use Upwork for most of the projects. It’s perfect for freelancers.

I have a limited time to work, so spending any extra time with things that are not related to the work itself is a wast of time.

—

I have contexts for everything.

I have 2 accounts in my browser, one for work and another for me/freelancing.

I do this with everything: Multiple calendars, multiple terminals (sessions in Tmux), Multiple emails, etc … This helps me get organized and change context when I’m at home working on the projects.

—

I plan my freelancing week every Sunday.

I open up the calendar and fill the slots of how many hours I want to work every day. During the week is usually 1h/day and on the weekends 2~4h a day (if I’m not busy).

Efficiency and getting work done

—

Before start working, I make a list of what I’m planning to do on that specific day.

If I need to complete a task, I break it into small tasks that need to be completed to get the whole thing done.

When you have items to cross out it makes the path clean and you have only one goal: Mark all items as done.

—

Every single time I start working on a task I know what I need to complete.

In my experience, start working without knowing what I need to do next usually makes me do lots of things that don’t matter or are not very important (small tweaks on the interface, tuning a code that is already working and so on)

—

I break big tasks in small chunks of 1h work.

At least for me, it’s almost impossible to get good quality work if I try to work more than 1 hour in a row. — Some folks like to use the Pomodoro technique.

To solve that I plan my tasks separated by 15~45 minutes intervals.

For example, if I want to 4 hours of quality work in a day I usually do 2h in the afternoon (separated by 30 minutes break) and 2h in the evening, also with 30~45 minutes break.

Those are incredible efficient hours.

—

All my projects have roadmaps describing each sprint (usually 1 week) and what I want to complete on that sprint.

This is extremely important if you want to get the whole project completed.

If you don’t have a clear plan of what you need to do, it’s very easy to lose lots of hours tuning things that are not important.
Focus on what matters. Get the project completed first, think about improvements later.

—

As I have a very limited time to work and I want to get the best of my hours, I can’t be interrupted during that time.

For that, I leave my phone far from the computer and I also turn off all notifications.

Creativity and Ideas

—

I usually take a notepad everywhere I go.

Creativity rarely happens when you are doing the work. So it's always good to have a notepad with me all the time.

I have a ritual to try getting my creativity flow.

Usually, I put some relaxing music (this playlist on Spotify is my favorite for that) and I go for a low/medium effort run for one hour, earlier in the morning.

If I’m stuck with some problem or need some ideas for a product I’ll probably figure it out at that moment.

—

Learn how to do some basic sketching.

I have never been good at sketching. A couple of months ago I started watching some videos on youtube to learn basic sketching.

It helped me a lot to create some basic wireframes and visualize what I’m trying to do. This is also good when you need to share ideas with another person.

Balance (life)

—

I don’t try to work more than 4 hours a day because after that time I lose efficiency.

I’ve done lots of experiments and 4 hours a day is perfect for me.

Why I would want to spend more hours in front of the computer knowing that I’m not being the best I can be?

I could be at the pool, going out with my Girlfriend, doing groceries, or just not doing anything else. It’s okay to take breaks.

Essential Tools (at least for me)

—

While working from home I end up depending on some important tools.

Spotify (Soundtrack)
Notion (Project management)
Klokki (Time management)
Tmuxinator (Terminal Sessions Management)
Notepad (the real one, not a software)

—

This is it, thanks!

Do you have your own rules when working as a freelancer? Feel free to share below!

Building your first interactive Node JS CLI

Hugo Dias — Tue, 10 Jul 2018 21:32:26 +0000

Originally posted in my blog

NodeJS can be very useful when it comes to building Command-line Interfaces also known as CLI's.

In this post, I'll teach you how to build a CLI that asks some questions and creates a file, based on the answers.

Getting started

Let's start by creating a brand new npm package

mkdir my-script
cd my-script
npm init

NPM will ask some questions. After that, we need to install some packages.

npm install --save chalk figlet inquirer shelljs

What these packages do:

chalk - Terminal string styling done right
figlet - Figlet is a program for making large letters out of ordinary text
inquirer - A collection of common interactive command line user interfaces
shelljs - Portable Unix shell commands for Node.js

index.js file

Now create a index.js file with the following content:

#!/usr/bin/env node

const inquirer = require("inquirer");
const chalk = require("chalk");
const figlet = require("figlet");
const shell = require("shelljs");

Planning the CLI

it's always good to plan what a CLI needs to do before writing any code.

This CLI will do just one thing: Create a file.

It should ask a couple of questions and after that, show a success message with the created file path.

The questions are: what is the filename and what is the extension.

// index.js

const run = async () => {
  // show script introduction
  // ask questions
  // create the file
  // show success message
};

run();

The first function is the script introduction. Let's use chalk and figlet to get the job done.


const init = () => {
  console.log(
    chalk.green(
      figlet.textSync("Node f*cking JS", {
        font: "Ghost",
        horizontalLayout: "default",
        verticalLayout: "default"
      })
    )
  );
}

const run = async () => {
  // show script introduction
  init();

  // ask questions
  // create the file
  // show success message
};

run();

Now it's time to write a function that asks questions.

const askQuestions = () => {
  const questions = [
    {
      name: "FILENAME",
      type: "input",
      message: "What is the name of the file without extension?"
    },
    {
      type: "list",
      name: "EXTENSION",
      message: "What is the file extension?",
      choices: [".rb", ".js", ".php", ".css"],
      filter: function(val) {
        return val.split(".")[1];
      }
    }
  ];
  return inquirer.prompt(questions);
};

// ...

const run = async () => {
  // show script introduction
  init();

  // ask questions
  const answers = await askQuestions();
  const { FILENAME, EXTENSION } = answers;

  // create the file
  // show success message
};

Notice the constants FILENAME and EXTENSIONS that came from inquirer.

The next step is to create the file.

const createFile = (filename, extension) => {
  const filePath = `${process.cwd()}/${filename}.${extension}`
  shell.touch(filePath);
  return filePath;
};

// ...

const run = async () => {
  // show script introduction
  init();

  // ask questions
  const answers = await askQuestions();
  const { FILENAME, EXTENSION } = answers;

  // create the file
  const filePath = createFile(FILENAME, EXTENSION);

  // show success message
};

And last but not least, show the success message along with the file path.

const success = (filepath) => {
  console.log(
    chalk.white.bgGreen.bold(`Done! File created at ${filepath}`)
  );
};

// ...

const run = async () => {
  // show script introduction
  init();

  // ask questions
  const answers = await askQuestions();
  const { FILENAME, EXTENSION } = answers;

  // create the file
  const filePath = createFile(FILENAME, EXTENSION);

  // show success message
  success(filePath);
};

Let's test the script by running node index.js.

Yay! And here is the final code:

Final code

#!/usr/bin/env node

const inquirer = require("inquirer");
const chalk = require("chalk");
const figlet = require("figlet");
const shell = require("shelljs");

const init = () => {
  console.log(
    chalk.green(
      figlet.textSync("Node f*cking JS", {
        font: "Ghost",
        horizontalLayout: "default",
        verticalLayout: "default"
      })
    )
  );
};

const askQuestions = () => {
  const questions = [
    {
      name: "FILENAME",
      type: "input",
      message: "What is the name of the file without extension?"
    },
    {
      type: "list",
      name: "EXTENSION",
      message: "What is the file extension?",
      choices: [".rb", ".js", ".php", ".css"],
      filter: function(val) {
        return val.split(".")[1];
      }
    }
  ];
  return inquirer.prompt(questions);
};

const createFile = (filename, extension) => {
  const filePath = `${process.cwd()}/${filename}.${extension}`
  shell.touch(filePath);
  return filePath;
};

const success = filepath => {
  console.log(
    chalk.white.bgGreen.bold(`Done! File created at ${filepath}`)
  );
};

const run = async () => {
  // show script introduction
  init();

  // ask questions
  const answers = await askQuestions();
  const { FILENAME, EXTENSION } = answers;

  // create the file
  const filePath = createFile(FILENAME, EXTENSION);

  // show success message
  success(filePath);
};

run();

To execute this script anywhere add a bin section in your package.json file and run npm link

{
  "name": "creator",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1",
    "start": "node index.js"
  },
  "author": "",
  "license": "ISC",
  "dependencies": {
    "chalk": "^2.4.1",
    "figlet": "^1.2.0",
    "inquirer": "^6.0.0",
    "shelljs": "^0.8.2"
  },
  "bin": {
    "creator": "./index.js"
  }
}

$ npm link
$ creator

Hope it helps :)

Photo by Alex Knight on Unsplash

Using Rails secret weapon: ActiveSupport::Notifications

Hugo Dias — Tue, 03 Jul 2018 15:39:44 +0000

Originally posted in my blog

Ruby on Rails has a powerful weapon that not all developers knows about its existence. ActiveSupport Instrumentations is used by Rails framework to handle events that happen when the application is running.

The instrumentation API provided by Active Support allows developers to provide hooks which other developers may hook into. There are several of these within the Rails framework. With this API, developers can choose to be notified when certain events occur inside their application or another piece of Ruby code.

Lucky we can use this tool at our disposal.

How it works

It looks like a PUB/SUB pattern.

In software architecture, publish–subscribe is a messaging pattern where senders of messages, called publishers, do not program the messages to be sent directly to specific receivers, called subscribers. Instead, published messages are characterized into classes, without knowledge of what, if any, subscribers there may be. Similarly, subscribers express interest in one or more classes, and only receive messages that are of interest, without knowledge of what, if any, publishers there are. - Wikipedia

Subscribing to an event is pretty easy:

# config/initializers/events.rb
ActiveSupport::Notifications.subscribe "my_custom.event" do |*args|
  event = ActiveSupport::Notifications::Event.new(*args)
end

The event variable will contain something like this

=> #<ActiveSupport::Notifications::Event:0x0000559258433df8
 @children=[],
 @duration=nil,
 @end=2018-07-03 12:47:46 +0000,
 @name="my_custom.event",
 @payload={:foo=>"bar"},
 @time=2018-07-03 12:47:46 +0000,
 @transaction_id="37bb01f75132e0c0505publisher6">

Notice the @payload? We can send additional data within the event.

Instrumenting an event

There is a method called instrument that publish the event.

You can use it like this:

ActiveSupport::Notifications.instrument "my_custom.event", { foo: "bar" }

Easy right?

Real world example

Let's say that you have an external service that handles application metrics, such as Segment.IO, Keen IO, and many others.

You can use this pattern to send events to one of this services.

# config/initializers/events.rb
# We can use regex to subscribe!
ActiveSupport::Notifications.subscribe /metrics/ do |*args|
  event = ActiveSupport::Notifications::Event.new(*args)
  MyMetricsService.send(event.name, event.payload)
end

# app/controllers/my_controller.rb
def create
  # some code
  ActiveSupport::Notifications.instrument "metrics.item_purchased", { uid: item.id }
end

You can read more about ActiveSupport Instrumentation here

Hope it helps :)

Photo by Ciprian Boiciuc on Unsplash

Facebook post preview with ReactJS and AWS Lambda

Hugo Dias — Tue, 03 Jul 2018 01:36:22 +0000

I'm currently focused on learning more about React. For me, the best way to learn something is to build a small project that's not a todo list (lol).

That being said, I've decided to build a small app that receives an URL, fetch the meta tags (especially the open graph tags) and build a preview based on facebook.

To accomplish this task I've chosen ReactJS and AWS Lambda (hosted on Netlify).

The Lambda function

The lambdas task is pretty simple. Receive an URL, parse and fetch the meta tags, looking for the Open Graph tags.

The open-graph-scraper npm package almost got the entire job done, so here is the final code of the lambda:

import ogs from "open-graph-scraper";
import getUrl from "get-urls";
import urlParser from "url";

export function handler(event, context, callback) {
  const text = event.queryStringParameters.q;
  const urls = getUrl(text);

  // Return if there is no urls in text
  if (!urls.size) {
    return callback(null, {
      statusCode: 200,
      body: JSON.stringify({
        text: text,
        meta: null,
        error: ["Empty url in text"]
      })
    });
  }

  // Retrieve first URL in text - urls are already normalized
  const url = [...urls][0];
  const options = { url };

  ogs(options, (error, results) => {
    const statusCode = results.success ? 200 : 500;
    callback(null, buildResponseObject(statusCode, results, text));
  });
}

function getUrlDomain(url) {
  const urlObj = urlParser.parse(url);
  return urlObj.host;
}

function cleanText(text) {
  return text.replace(/(?:https?|ftp):\/\/[\n\S]+/g, "");
}

function buildResponseObject(statusCode, result, text) {
  let meta = statusCode === 200 ? result.data : null;

  if (meta) {
    let images = meta.ogImage;

    if (images instanceof Array) {
      meta.ogImage = images[0];
    }

    let domain = meta.ogUrl;

    if (domain) {
      meta.ogUrl = getUrlDomain(meta.ogUrl);
    }
  }

  const body = {
    meta: meta,
    text: cleanText(text),
    error: statusCode !== 200 ? result.error : null
  };

  return {
    statusCode,
    body: JSON.stringify(body)
  };
}

The Frontend

As I mentioned at the beginning of the post, react was chosen to build the client interface.

The packages used are:

  "dependencies": {
    "get-urls": "^7.2.0",
    "netlify-lambda": "^0.4.0",
    "open-graph-scraper": "^3.3.0",
    "react": "^16.4.1",
    "react-content-loader": "^3.1.2",
    "react-dom": "^16.4.1",
    "react-scripts": "1.1.4",
    "styled-components": "^3.3.2"
  },

And here is the result:

Contributing

It's free and open source!

The code is hosted on Github and you can see it in action here.

I'm looking forward to your feedback, folks.

Thanks

Wait for MongoDB, Postgres or MySql to start on Docker

Hugo Dias — Sat, 30 Jun 2018 01:20:46 +0000

Originally posted in my blog

Quick tip here.

I was struggling to get this thing working so I need to share with you guys.

Assuming that you have an app depending on a MongoDB container:

version: "3.3"
services:
  web:
    build: .
    volumes:
      - "./app:/src/app"
    ports:
      - "3030:3000"
    links:
      - mongo
    depends_on:
      - mongo
  mongo:
    container_name: mongo
    image: mongo
    volumes:
      - ./data:/data/db
    ports:
      - "27017:27017"
volumes:
  mongo_data:

In my case it's a NodeJS app.

The MongoDB is quite slow to get up, so when the app starts, the linked container is ready but not the database itself.

To solve this problem you can use a script to explicitly wait for MongoDB service to be up.

Add this to your Dockerfile

FROM node:latest

RUN mkdir /src


WORKDIR /src
ADD app/package.json /src/package.json
RUN npm install

EXPOSE 3000

## THE LIFE SAVER
ADD https://github.com/ufoscout/docker-compose-wait/releases/download/2.2.1/wait /wait
RUN chmod +x /wait

## Launch the wait tool and then your application
CMD /wait && npm start

And now change your docker-compose.yml file and adds a WAIT_HOSTS section

version: "3.3"
services:
  web:
    build: .
    volumes:
      - "./app:/src/app"
    ports:
      - "3030:3000"
    links:
      - mongo
    depends_on:
      - mongo
    environment:
      WAIT_HOSTS: mongo:27017
  mongo:
    container_name: mongo
    image: mongo
    volumes:
      - ./data:/data/db
    ports:
      - "27017:27017"
volumes:
  mongo_data:

And that's it.

You can use it for any other service such as Postgres, MySQL, etc ...

To do this you just need to use commas:

version: "3"

services:

  mongo:
    image: mongo:3.4
    hostname: mongo
    ports:
      - "27017:27017"

  postgres:
    image: "postgres:9.4"
    hostname: postgres
    ports:
      - "5432:5432"

  mysql:
    image: "mysql:5.7"
    hostname: mysql
    ports:
      - "3306:3306"

  mySuperApp:
    image: "mySuperApp:latest"
    hostname: mySuperApp
    environment:
      WAIT_HOSTS: postgres:5432, mysql:3306, mongo:27017

Hope it helps