DEV Community: Samuel James

Migrating Serverless Aurora Schema Using a Custom Resource

Samuel James — Sat, 14 Nov 2020 11:29:42 +0000

Backend applications have a lot of moving parts. Besides designing your domain models, you also have to think about what to expose and how to persist your data.

For applications that use relational datastores like MYSQL, PostgreSQL, migrating schema is crucial for a successful deployment.

I like to think of migrations as code that updates a database schema. In this post, supported by a Github project, you will learn one way to handle serverless aurora schema migration using a custom resource.

Custom Resources

Custom resources enable you to write custom provisioning logic in templates that AWS CloudFormation runs anytime you create, update (if you change the custom resource), or delete the stack.

We could write a lambda function with a custom resource that runs migrations after provisioning a database.

The next question that comes to mind is how to make the schema scripts available to lambda without actually including it in the deployment layer. This is where the AWS Lambda layer comes in.

The Lambda layer allows you to archive dependencies and uses those in your functions without including them in your deployment layer.

Without further ado, let's see how to do this with CDK.

Creating the Migration Handler

The migration_handler.py is a function that is executed during the creation of the custom resource. It uses DATA API to connect the database instance, read the migration files, and run them on the database.

import boto3
import os
import logging as log

from botocore import exceptions
import cfnresponse
import glob

log.getLogger().setLevel(log.INFO)


def main(event, context):
    SQL_PATH = "/opt"  # Layers are extracted to the /opt directory in the function execution environment.

    # This needs to change if there are to be multiple resources
    # in the same stack
    physical_id = "SchemaMigrationResource"

    # If this is a Delete event, do nothing. The schema will be destroyed along with the cluster.
    if event['RequestType'] == 'Delete':
        cfnresponse.send(event, context, cfnresponse.SUCCESS, {"Response": "Deleted successfully"}, physical_id)

    try:
        log.info("Input event: %s", event)

        sqlfiles = glob.glob(os.path.join(SQL_PATH, "*.sql"))

        log.info(sqlfiles)
        for file_path in sqlfiles:
            log.info(f"Found an SQL script in path:{file_path}")
            execute_sql_file(file_path)

        log.info("Ran migration successfully")

        attributes = {"Response": f"Ran migration successfully for these files:{sqlfiles}"}

        cfnresponse.send(event, context, cfnresponse.SUCCESS, attributes, physical_id)
    except Exception as e:
        log.exception(e)
        # cfnresponse's error message is always "see CloudWatch"
        cfnresponse.send(event, context, cfnresponse.FAILED, {}, physical_id)
        raise RuntimeError("Create failure requested")


def execute_statement(sql, sql_parameters=[], transaction_id=None):
    log.info(f"sql query:{sql}")
    client = boto3.client("rds-data")
    parameters = {
        "secretArn": os.getenv("DB_SECRET_ARN"),
        "database": os.getenv("DB_NAME"),
        "resourceArn": os.getenv("DB_CLUSTER_ARN"),
        "sql": sql,
        "parameters": sql_parameters,
    }
    if transaction_id is not None:
        parameters["transactionId"] = transaction_id
    try:
        response = client.execute_statement(**parameters)
        return response
    except client.exceptions.BadRequestException as e:
        log.exception(e)
        raise RuntimeError("Create failure requested")


def execute_sql_file(file_path: str):
    log.info(f"executing file in : {file_path}")
    with open(file_path, "r") as script:
        script_content = script.read()
        queries = script_content.split(";")
        for query in queries:
            sql = query.strip()
            if sql:
                execute_statement(query)
    log.info(f"executed the file : {file_path} successfully")

Creating the Custom Resource

The migration.py contains the code for creating the lambda function and the lambda layer. We passed some environment variables (like database reference, secret and database name) needed by the lambda function to connect to the database.

from aws_cdk import (
    core,
    custom_resources as cr,
    aws_lambda as _lambda,
    aws_cloudformation as cfn,
    aws_iam as _iam,
)
from aws_cdk.aws_lambda import Runtime

SQL_SCRIPTS_PATH = "scripts/schema"


class SchemaMigrationResource(core.Construct):
    def __init__(
        self,
        scope: core.Construct,
        id: str,
        secret_arn: str,
        db_name: str,
        db_ref: str,
        **kwargs,
    ):
        super().__init__(
            scope,
            id,
            **kwargs,
        )

        with open("migration_handler.py", encoding="utf-8") as fp:
            code_body = fp.read()

            lambda_function = _lambda.SingletonFunction(
                self,
                "Singleton",
                uuid="f7d4f730-4ee1-11e8-9c2d-fa7ae01bbebc",
                code=_lambda.InlineCode(code_body),
                handler="index.main",
                timeout=core.Duration.seconds(300),
                runtime=_lambda.Runtime.PYTHON_3_7,
                layers=[
                    _lambda.LayerVersion(
                        scope,
                        id="migrationscripts",
                        code=_lambda.Code.from_asset(SQL_SCRIPTS_PATH),
                        description="Database migration scripts",
                    )
                ],
                environment={
                    "DB_NAME": db_name,
                    "DB_SECRET_ARN": secret_arn,
                    "DB_CLUSTER_ARN": db_ref,
                },
            )

        # Allow lambda to read database secret
        lambda_function.add_to_role_policy(
            _iam.PolicyStatement(
                resources=[secret_arn],
                actions=["secretsmanager:GetSecretValue"],
            )
        )
        # allow lambda to execute query on the database
        lambda_function.add_to_role_policy(
            _iam.PolicyStatement(
                resources=[db_ref],
                actions=[
                    "rds-data:ExecuteStatement",
                    "rds-data:BatchExecuteStatement",
                ],
            )
        )
        # assign policies to the Lambda function so it can output to CloudWatch Logs.
        lambda_function.add_to_role_policy(
            _iam.PolicyStatement(
                resources=["*"],
                actions=[
                    "logs:CreateLogGroup",
                    "logs:CreateLogStream",
                    "logs:PutLogEvents",
                ],
            )
        )

        resource = cfn.CustomResource(
            self,
            "Resource",
            provider=cfn.CustomResourceProvider.lambda_(lambda_function),
            properties=kwargs,
        )

        self.response = resource.get_att("Response").to_string()

Creating Serverless Aurora Instance

With the custom resource defined, we can create the serverless aurora stack with this code:

# filename: migration.py
import os
from aws_cdk import (
    aws_ec2 as ec2,
    aws_rds as rds,
    core,
    aws_secretsmanager as sm,
)
from .migration import SchemaMigrationResource


class RDSStack(core.Stack):
    def __init__(self, scope: core.Construct, id: str, **kwargs) -> None:
        super().__init__(scope, id, **kwargs)

        vpc = ec2.Vpc(self, "VPC")
        db_master_user_name = os.getenv("DB_USERNAME", "admin_user")

        self.secret = rds.DatabaseSecret(
            self, id="MasterUserSecret", username=db_master_user_name
        )

        rds.CfnDBSubnetGroup(
            self,
            "rdsSubnetGroup",
            db_subnet_group_description="private subnets for rds",
            subnet_ids=vpc.select_subnets(
                subnet_type=ec2.SubnetType.PRIVATE
            ).subnet_ids,
        )
        db_name = os.getenv("DB_NAME", "anonfed")
        self.db = rds.CfnDBCluster(
            self,
            "auroraCluster",
            engine="aurora-mysql",
            engine_version="5.7.mysql_aurora.2.08.1",
            db_cluster_parameter_group_name="default.aurora-mysql5.7",
            # snapshot_identifier="<snapshot_arn>",  # your snapshot
            engine_mode="serverless",
            scaling_configuration=rds.CfnDBCluster.ScalingConfigurationProperty(
                auto_pause=True,
                min_capacity=1,
                max_capacity=4,
                seconds_until_auto_pause=300,
            ),
            db_subnet_group_name=core.Fn.ref("rdsSubnetGroup"),
            database_name=db_name,
            master_username=self.secret.secret_value_from_json("username").to_string(),
            master_user_password=self.secret.secret_value_from_json(
                "password"
            ).to_string(),
            enable_http_endpoint=True,
        )

        secret_attached = sm.CfnSecretTargetAttachment(
            self,
            id="secret_attachment",
            secret_id=self.secret.secret_arn,
            target_id=self.db.ref,
            target_type="AWS::RDS::DBCluster",
        )
        secret_attached.node.add_dependency(self.db)
        db_ref = f"arn:aws:rds:{self.region}:{self.account}:cluster:{self.db.ref}"
        migration = SchemaMigrationResource(
            self, "schemamigration", self.secret.secret_arn, db_name, db_ref
        )

        # Publish the custom resource output
        core.CfnOutput(
            self,
            "ResponseMessage",
            description="Database Migration",
            value=migration.response,
        )

Finally, we can hook everything together in app.py


env_EU = core.Environment(
     account=os.environ.get("CDK_DEPLOY_ACCOUNT", os.environ["CDK_DEFAULT_ACCOUNT"]),
     region=os.environ.get("CDK_DEPLOY_REGION", os.environ["CDK_DEFAULT_REGION"]),
    )

    app = core.App()
    db = RDSStack(scope=app, id="aurora", env=env_EU)
    ap_stack = Api(scope=app, id="api", env=env_EU, db=db)

    app.synth()

Conclusion

In this post, we've seen how to migrate database schema using cloud formation custom resource. To build on this knowledge, I encourage you to take a look at the full project in this repository.

References

What Is Serverless Architecture and When Should You Use It?

Samuel James — Tue, 07 Apr 2020 14:30:15 +0000

Cloud computing is constantly evolving, from bare-metal to container technologies. The latest trend in this process is the serverless (Function as a Service, or FaaS) computing model. According to Techbeacon, serverless has an annual growth rate of 75%, making it the fastest growing cloud service model. So, serverless architecture isn't a mere buzzword. More companies than ever are adopting it.

If you're not sure what serverless architecture is, you've come to the right place. Read on to learn what serverless architecture is, when it makes sense, and when it doesn't.

What Is Serverless Architecture?

The term serverless architecture can be confusing. It doesn't mean application designs that allow for running applications in some magical space where servers are nonexistent.

Serverless architectures are application designs that make use of third-party services (Back end as a Service, or BaaS). They may use custom code run in managed, ephemeral containers on a FaaS platform.

Servers still run your application. But a third-party company takes care of the grunt work of provisioning, managing, and scaling servers. In serverless architecture, you manage and provision nothing.

Serverless architecture often incorporates two components: Function as a Service and Backend as a Service.

FaaS is a computing service that allows you to run self-contained code snippets called functions in the cloud. Your functions remain dormant until events trigger them. Functions are self-contained, small, short-lived, and single-purpose. They die after execution.
BaaS is a cloud computing service that completely abstracts backend logic, which takes place on faraway servers. It allows developers to focus on front-end code and integrate with back-end logic that someone else has implemented. BaaS could be authentication, storage services, geolocation services, user management, and so on.

In serverless architecture, you focus on writing code only. You deploy when you’re ready, without caring about what runs it or how it runs.

Is serverless the same as Platform as a Service?

Serverless and Platform as a Service (PaaS): What's the Difference?

Many people mistake Platform as a Service (PaaS) for serverless architecture. Although they are similar in many ways, they aren't the same.

PaaS providers offer software and hardware infrastructure as a platform to users—what many people call a solution stack. That means you can run custom applications using the provider's platform. Serverless, on the other hand, provides an environment where you can write and run custom code (functions) without managing, provisioning, or scaling infrastructure.

What do Paas and serverless architecture have in common? You don't manage infrastructure in either one. The platform provider takes care of that. However, differences exist in how you compose your application and how you scale it.

Composition

In the PaaS model, you write your application using a framework or language that your platform provider supports, and you deploy your business logic as a single unit. In the serverless model, your business logic is broken down into self-contained units that each perform a business function.

Scaling

Scaling isn't automatic for PaaS applications. Instead, you have to configure your app and add resources to handle more requests. In serverless architecture, on the other hand, your app scales automatically as the workload increases.

Lifetime Availability

Like traditional applications, PaaS applications have to be available at all times to continue to serve requests. In serverless, by contrast, functions are short-lived—they die after execution.

Now you understand that PaaS isn't serverless, though the two systems are similar in some ways. But why would you want to go serverless?

Why Use Serverless Architecture?

As you ponder this question, consider these three key attributes of serverless architecture.

It's scalable and highly available. Scaling traditional applications requires you to understand your traffic pattern. You estimate how much of each resource you'd need, and then you'd provision accordingly. Users troop in from all geographical regions to use modern applications. A traditional application could be overwhelmed by a spike—probably on a black Friday. In serverless, your application is highly available, and it scales automatically as your users grow and usage increases.
It costs less. One of the reasons serverless architecture is gaining popularity among startups is because of its pricing model. The cost of running servers 24/7 and paying for idle time is no longer an issue in serverless. You pay for usage only. Functions have allocated time in which they run and die afterward. The provider charges based on the number of executions and the size of memory your workload uses. This helps you optimize costs.
The time to market is faster. Operational tasks such as server provisioning, maintenance, and monitoring infrastructure are off your shoulders. You can focus solely on your business logic (code), experiment with ideas, and hit production on time.

With all the promises of serverless, is it perfect for all use cases? No. In some cases, serverless architecture makes sense, but in others, it might not. We'll discuss this point in detail later on in this post.

Let's explore some common use cases of serverless architecture.

Serverless Architecture Use Cases

In a traditional application, your application runs actively on servers. Since your servers are on 24/7, you also pay for idle time. In the serverless world, though, there are no servers to pay for. You only pay per trigger. In other words, you pay for what you use.
This advantage makes serverless architecture a good option for business cases that don't have to be always on. In this way, you save money from not paying for idle time.

Examples of serverless architecture use cases are:

High-Traffic Websites

If you're still serving your static websites from an EC2 instance, you may be missing out on a lot. With serverless, you can host your static website on S3 bucket and serve your assets with a global, fast cloud delivery network. Not only is it cheaper and fast, but it is also highly available and scalable.

Multimedia Processing Applications

If your business deals with images and videos, then serverless architecture might work well for you. You can use a scalable storage service such as AWS S3 to store your data. An upload event can trigger a lambda function after each successful upload that processes your file asynchronously. Your users can continue to enjoy your app while a highly available and scalable back-end service processes the upload in a non-blocking way.

Mobile Backends

An API gateway gives you an entry point to your business functions. These functions can be exposed as rest API that your mobile app consumes. Serverless services such as AWS AppSync allow you to securely access, manipulate, and combine data from multiple sources in real time.

Internet of Things (IoT)

IoT devices generate a lot of data from their environments through sensors. Organizations often struggle to process this overwhelming data coming from these connected devices in a scalable way. Using a serverless back end like AWS IoT Core, you can scale to billions of devices and trillions of messages.

Big Data Applications

Before cloud computing, the insights big data provided were available only to big enterprises because organizations needed the infrastructures' overheads to make sense of that data. Setting up and maintaining infrastructures for big data isn't easy. With serverless computing, your app can now take advantage of several services, including Amazon S3, Amazon Athena, Amazon Kinesis, AWS Glue, and AWS Lambda to build scalable data pipelines.

Earlier, I mentioned that serverless architecture isn't a silver bullet. There are cases where serverless might not be a good fit.

When Going Serverless Might Not Make Sense

With all the promises of serverless architecture, it has its drawbacks and limitations too. It's important to keep these in mind when considering serverless architecture and plan accordingly.

There's a limit to how long a function can run. This makes serverless architecture unsuitable for tasks that run for hours.
What if you require a fast response with a consistent latency of less than 50 milliseconds? Serverless architecture suffers from the problem of a cold start. In a case like this, you might need to reconsider your options.
You may be afraid of vendor lock-in. Serverless architecture could tie you to a single vendor. Migrating serverless applications from one vendor to another requires a lot of manual effort and major changes in your code.
Serverless vendors limit how much memory you can assign to functions. If you require a complex compute with high memory requirements, then going serverless might not be a good use case.
Serverless vendors enforce hard limits on deployment size. This could vary from one vendor to another. For example, AWS allows a code deployment size up to 250MB. For large code deployment, serverless architecture might not be suitable.
Serverless is still relatively young. For this reason, observability is still a problem. Having 360-degree insights into your functions can be difficult.

Conclusion

Before moving to serverless architecture, you must first evaluate your use cases. It's critical to understand what your business requirements are and whether serverless architecture would make sense for your project and some of the limitations you'd hit down the road. When you have this information, you can better prepare.

In this post, we've seen the advantages of serverless architecture. We also discussed some of the drawbacks of serverless architecture and some typical serverless use cases. To build on this knowledge, I encourage you to read these sources:

Building an Image Classifier With Pytorch

Samuel James — Fri, 13 Mar 2020 08:26:36 +0000

In this post, you'll learn how to train an image classifier using transferred learning with Pytorch on Google Colab. We'll use a dataset provided by CalTech, which contains pictures of objects in 101 categories. We'll talk briefly about overfitting and how to avoid overfitting. In the end, we'll train a model capable of making generalized predictions over some selected categories of objects. If you ever feel lost at any point, feel free to take a look at the Jupyter notebook on Github or execute the Jupyter notebook directly on Google Colab.

Pytorch and Why

Pytorch is a deep learning framework developed by Facebook, and it's completely opensource.

When it comes to using a deep learning framework, there are many options you can choose from, but very few (if any) offer the simplicity and flexibility Pytorch offers. Pytorch syntax is simple and pythonic (Python-based), which makes it easy for Python developers to master and get up and running quickly.

Besides, Pytorch has excellent documentation with various tutorials and an option to try out what you learned on Google Colab in one click.

Google Colab

I like to see Google Colab as a notebook of one click away from my browser. Google Colab allows you to write and execute Python code in your browser with zero configuration. It supports popular data science libraries and deep learning frameworks, including Pytorch, without requiring you to install anything.

You can import datasets, train, and evaluate models by leveraging Google hardware, including GPUs and TPUs. With all this awesomeness that Google Colab offers, all you need to use it is your google or Gmail account.

To start your first notebook on Google Colab, sign in with your Google account on Google Colab, and open a new Jupyter notebook.

Data preprocessing

Datasets don't usually come in their best form suitable for fitting models directly. In most cases, one needs to do some level of preprocessing before training. Fortunately, we're using CalTech datasets, which have been labeled for us. However, we'll not use all classes in CalTech datasets in this post. We'll use a subset of the data.

First, let's download the datasets. You can run the command block below in your Jupiter notebook to get the data.

!wget http://www.vision.caltech.edu/Image_Datasets/Caltech101/101_ObjectCategories.tar.gz
!tar -xvzf ./101_ObjectCategories.tar.gz

Splitting Datasets

One of the major enemies of your algorithm is overfitting. Overfitting is when a model is so tuned to the training sets that it's unable to make accurate predictions on data outside of the training sets.

The goal of any machine learning project is to be able to learn well on the training data and as well make good predictions on new data. A model that predicts well only training sets is overfitted and may not perform well in production.

But how do we know if a model is overfitted?

There is no other way of knowing than testing the model on data it has not seen. This dataset is referred to as test datasets.

It's a best practice to train your model on train datasets and test the accuracy of the model on test datasets.

A train dataset is the datasets used in learning (fitting the model), and a test dataset is independent of the train sets. If your model fits the training data well and also generalizes well on independent datasets, you have achieved minimum overfitting.

Besides holding out some data for testing aside from the training datasets, we'll also hold out some independent data for validation. The validation dataset is a sample of your dataset held out to provide an unbiased evaluation of your model while tunning the model hyperparameters.

In this example, 75% of our data goes for training, 12.5% for validation, and 12.5% for testing.

The python script below splits the CalTech dataset into train sets, validation sets, and test sets.

import numpy as np
import math
from typing import List
import os
import argparse
import glob
import shutil

def list_files(path):
    files = os.listdir(path)
    return np.asarray(files)

def split_files(oldpath, newpath, classes):
    for name in classes:
        full_dir = os.path.join(os.getcwd(), f"{oldpath}/{name}")

        files = list_files(full_dir)
        total_file = np.size(files,0)
        # We split data set into 3: train, validation and test

        train_size = math.ceil(total_file * 3/4) # 75% for training 

        validation_size = train_size + math.ceil(total_file * 1/8) # 12.5% for validation
        test_size = validation_size + math.ceil(total_file * 1/8) # 12.5x% for testing 

        train = files[0:train_size]
        validation = files[train_size:validation_size]
        test = files[validation_size:]

        move_files(train, full_dir, f"train/{name}")
        move_files(validation, full_dir, f"validation/{name}")
        move_files(test, full_dir, f"test/{name}")

def move_files(files, old_dir, new_dir):
    new_dir = os.path.join(os.getcwd(), new_dir);
    if not os.path.exists(new_dir):
        os.makedirs(new_dir)

    for file in np.nditer(files):
        old_file_path = os.path.join(os.getcwd(), f"{old_dir}/{file}")
        new_file_path = os.path.join(os.getcwd(), f"{new_dir}/{file}")

        shutil.move(old_file_path, new_file_path)

To split the data, you can run split_files function by passing a list of classes you want train your model and a path as as arguments.

classes = ['Leopards', 'airplanes', 'butterfly', 'camera', 'elephant', 'lamp', 'watch', 'umbrella', 'rhino'];

split_files('101_ObjectCategories', './', classes)

Model Selection

Choosing a model largely depends on so many things like accuracy of the model, inference time, model size, etc. If your goal is to compute inference on mobile devices, the size of your model might be crucial to your choice of model. Depending on your use cases, you might find yourself having to choose one model over another.

For this post, I'll use ResNet50 (a pre-trained model) because of its high top-1 and top-5 accuracy.

from torchvision import models
import torch

for param in model.parameters():
  param.requires_grad = False

fc_inputs = model.fc.in_features

model.fc = nn.Sequential(
    nn.Linear(fc_inputs, 2048),
    nn.ReLU(inplace=True),
    nn.Linear(2048, 10),
    nn.Dropout(0.4),
    nn.LogSoftmax(dim=1)
)

The code above replaces the final layer of the resnet50 model by a set of Sequential layers. The inputs to the last connected layer of the pre-trained model is passed on to a Linear layer of 2048 inputs, then fed to Dropout and ReLU layers followed by 2048 by 10 Linear Layer with 10 outputs corresponding to the classes of images in the training data sets.

Pytorch allows you to write device-agnostic code (CPU or GPU). It’s a good practice to check for the availability of cuda so we can take advantage of it GPU if available.

import torch.nn as nn

if(torch.cuda.is_available()):
    model = model.to('cuda:0')

Input transformation

Before we fit the model with the training sets, we need to have some levels of transformation like resizing, normalizing, and even rotating so that we have enough variations of data.

Pytorch torchvision.transforms allows you to compose your transformations painlessly.


from torchvision import transforms

image_transforms = {
    'train': transforms.Compose([
              transforms.RandomResizedCrop(size=256, scale=(0.8, 1.0)),
              transforms.RandomRotation(degrees=15),
              transforms.RandomHorizontalFlip(),
              transforms.CenterCrop(size=224),
              transforms.ToTensor(),
              transforms.Normalize([0.485, 0.456, 0.406],[0.229, 0.224, 0.225])
    ]),
    'validation': transforms.Compose([
             transforms.Resize(size=256),
             transforms.CenterCrop(size=224),
             transforms.ToTensor(),
             transforms.Normalize([0.485, 0.456, 0.406],[0.229, 0.224, 0.225])              
    ]),
    'test': transforms.Compose([
             transforms.Resize(size=256),
             transforms.CenterCrop(size=224),
             transforms.ToTensor(),
             transforms.Normalize([0.485, 0.456, 0.406],[0.229, 0.224, 0.225])              
    ])
}

Having composed transformations for the datasets, it's time to load the data and also create an iterator for the Data Loader.

import torchvision.datasets as datasets
import torch.utils.data as loader

data = {
    'train': datasets.ImageFolder(root='./train', transform=image_transforms['train']),
    'validation': datasets.ImageFolder(root='./validation', transform=image_transforms['validation']),
    'test': datasets.ImageFolder(root='./test', transform=image_transforms['test'])
}

# create a data loader instance with each dataset with a batch size of 10 and shuffling
batch_size = 10
train_data = loader.DataLoader(data['train'], batch_size=batch_size, shuffle=True)
validation_data = loader.DataLoader(data['validation'], batch_size=batch_size, shuffle=True)
test_data = loader.DataLoader(data['test'], batch_size=batch_size, shuffle=True)

To calculate the loss function, that is a measure of how wrong your model prediction is, we need to know the size of each data set.

train_data_size = len(data['train'])
validation_data_size = len(data['validation'])
test_data_size =  len(data['test'])

Optimizer

Optimizer and loss function are vital to learning your data. If loss function is a way of measuring how wrong your predictions are, then Optimizer is that thing that tweaks your model parameters based on the output of loss function to ensure that your predictions are accurate as possible.

For this example, we use Adam optimizer as the choice of optimization algorithm for this problem.

import torch.optim as optim
loss_func = nn.NLLLoss()
optimizer = optim.Adam(model.parameters())

Fitting the model

Let's define a method for fitting and validating our model.


import time

def train_and_validate(model, loss_criterion, optimizer, epochs=25):
  device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
  start = time.time()
  history = []
  best_acc = 0.0
  for epoch in range(epochs):
    epoch_start = time.time()
    print(f'Epoch : {epoch+1}/{epochs}')
    model.train()
    train_loss = 0.0 
    train_acc = 0.0

    valid_loss = 0.0
    valid_acc  = 0.0

    for i, (inputs, labels) in enumerate(train_data):
      inputs = inputs.to(device)
      labels = labels.to(device)
      # Clean existing gradients
      optimizer.zero_grad()
      # Forward pass - compute outputs on input data using the model
      outputs = model(inputs)
      # Compute loss
      loss = loss_criterion(outputs, labels)
      # Backpropagate the gradients
      loss.backward()
      # Update the parameters
      optimizer.step()
      # Compute the total loss for the batch and it to train_loss
      train_loss += loss.item() * inputs.size(0)
      # Compute the accuracy
      ret, predictions = torch.max(outputs.data,1)
      correct_counts = predictions.eq(labels.data.view_as(predictions))

      # Convert correct_counts to float and then compute the mean
      acc = torch.mean(correct_counts.type(torch.FloatTensor))
      # Compute total accuracy in the whole batch and add to train_acc
      train_acc += acc.item() * inputs.size(0)
      print(f'Batch number: {i}, Training: Loss: {loss.item()}, Accuracy: {acc.item()}')

    with torch.no_grad():
      model.eval()
      for j, (inputs, labels) in enumerate(validation_data): 
        inputs = inputs.to(device)
        labels = labels.to(device)
        # Forward pass - compute outputs on input data using the model
        outputs = model(inputs)

        # Compute loss
        loss = loss_criterion(outputs, labels)
        # Compute the total loss for  the batch and add it to valid_loss
        valid_loss += loss.item() * inputs.size(0)
        # Calculate validation accuracy

        ret, predictions = torch.max(outputs.data, 1)
        correct_prediction_counts = predictions.eq(labels.data.view_as(predictions))

        # Convert correct_prediction_counts to float and then compute the mean
        acc = torch.mean(correct_prediction_counts.type(torch.FloatTensor))

        # Compute total accuracy in the whole batch and add to valid_acc

        valid_acc +=acc.item() * inputs.size(0)

      avg_train_loss = train_loss/train_data_size
      avg_train_acc = train_acc/train_data_size

      avg_valid_loss = valid_loss/validation_data_size
      avg_valid_acc = valid_acc/validation_data_size

      history.append([avg_train_loss, avg_valid_loss, avg_train_acc, avg_valid_acc])

      epoch_end = time.time()

      print(f'Epoch : {epoch}, Training: Loss: f{avg_train_loss}, Accuracy: {avg_train_acc*100}%, \n\t\tValidation : Loss : {avg_valid_loss}, Accuracy: {avg_valid_acc*100}%, Time: {epoch_end-epoch_start}s')

      # Save if the model has best accuracy till now

      torch.save(model.state_dict(), f'model_{epoch}.pth')

    return model, history

Having defined the train_and_validate method, we can train the model by running the function.

num_epochs = 30
trained_model, history = train_and_validate(model, loss_func, optimizer, num_epochs)

Testing the model

In earlier paragraphs, we talked about the problem of overfitting. We also discussed briefly one way to determine if your model is able to generalize on new data sets by setting aside a test set. Let’s put our model to test by seeing how it performs on test sets.

Remember, we should never train on test sets, and that’s why we held it out until now.


def computeModelAccuracy(model, loss_criterion):
  device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
  test_acc = 0.0
  test_loss = 0.0

  with torch.no_grad():
    # Set to evaluation mode
    model.eval()
    for i, (inputs, labels) in enumerate(test_data):
      inputs = inputs.to(device)
      labels = labels.to(device)
      outputs = model(inputs)

      # Compute loss
      loss = loss_criterion(outputs, labels)

      # Compute the toal loss item 
      test_loss += loss.item() * inputs.size(0)

      ret, predictions = torch.max(outputs.data, 1)
      correct_counts = predictions.eq(labels.data.view_as(predictions))
      acc = torch.mean(correct_counts.type(torch.FloatTensor))
      test_acc +=acc.item() * inputs.size(0)

      print(f'Test Batch number: {i}, Test: Loss: {loss.item()}, Accuracy: {acc.item()}')

      # Find average test loss and test accuracy
      avg_test_loss = test_loss/test_data_size
      avg_test_acc = test_acc/test_data_size

      print(f'Test accuracy: {avg_test_acc}')

Making Predictions

To predict the class of a new image, we need to convert the image to a format that our model understands.


from PIL import Image
import requests
import matplotlib.pyplot as plt

index_to_class = {v: k for k, v in data['train'].class_to_idx.items()}
print (index_to_class)

def makePrediction(model, url):
    device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
    transform = image_transforms['test']

    test_image = Image.open(requests.get(url, stream=True).raw)

    plt.imshow(test_image)

    test_image_tensor = transform(test_image)
    test_image_tensor = test_image_tensor.view(1, 3, 224, 224).to(device)
    with torch.no_grad():
        model.eval()
        out = model(test_image_tensor)
        ps = torch.exp(out)

        topk, topclass = ps.topk(3, dim=1)
        for i in range(3):
            print(f"Prediction {i+1} : {index_to_class[topclass.cpu().numpy()[0][i]]}, Score: {topk.cpu().numpy()[0][i] * 100}%")

The function makePrediction above takes a URL of an image and returns the top 3 predictions of the class of the image.

For example:

model = torch.load('_model_0.pt')
makePrediction(model, 'https://cdn.britannica.com/30/136130-050-3370E37A/Leopard.jpg')

Conclusion

We've learned how to train an image classifier using Pytorch. We talked briefly about splitting data sets and why overfitting should be avoided. We've explored how to train and evaluate models on Google Colab. We've pretty much covered a lot of things in this post. To build on this knowlege, I encourage you to take a look at the references below.

Finally, I share useful blogposts on machine learning and exciting stuff from the community every day; please follow or tweet at @hubofml on twitter .

References

A Comprehensive Hands-on Guide to Transfer Learning with Real-World Applications in Deep Learning

Introduction to Optimizers

Introduction to Machine Learning

Image Classification Using Pretrained-Model

How are poor APIs designed and what processes birth them

Samuel James — Wed, 25 Dec 2019 08:36:34 +0000

There are well-designed APIs, and there are APIs that are badly designed. It's a no brainer that a well-designed API makes life easier for everyone.

While there are lots of articles on APIs best practices, I would like to know how poor APIs are designed and what processes birth them.

Thanks for your contributions.

Learn Serverless on GCP by Building a Simple Calendar App

Samuel James — Fri, 20 Dec 2019 09:58:37 +0000

This post is to show you how to run build a serverless application on Google Cloud Platform that integrates with Google APIs. What we'll cover in this post includes:

Creating & deploying a serverless application using the serverless framework.
Integrating your application with Google APIs using the right authentication mechanism.

We'll build a simple application that integrates with Google calendar free-busy API.

If you use Gmail or Google mail for business, you already have access to google calendar and its rich APIs.

Google calendar allows you to schedule meetings with people or co-workers, and even specify where the meeting will take place.

The idea is to build an API that allows you to query these venues (calendars or meeting rooms) to see if they are busy or free within the time frame.

Basically, the client triggers a cloud function on Google Cloud Platform, the service queries google calendar for availability.

You could build a client later that consumes this API then but that we'll not cover in this post.

To start with the serverless application, you need to create a project on the Google Cloud Platform and enable the following APIs:

Google Calendar API
Cloud Functions API
Cloud Deployment Manager
Cloud Storage
Stackdriver Logging

Note down the ID of your project, we'll need it later.

Creating a cloud function

In this section, we'll create the cloud function using the serverless framework.

The serverless framework is an open-source package that allows you to develop and deploy a serverless application to most cloud platforms like AWS, Azure, Google, and more.

If you do not have the serverless framework installed, you can install it with:

npm install -g serverless

To create the boiler plate for this project using framework CLI, run:

serverless create --template google-nodejs --path /path/to/projectdirectory

The command above generate some files like:

.
├── index.js
├── package.json
└── serverless.yml

Open the serverless.yml file and modify the value of project to match your project's name on Google Cloud Platform.

 #serverless.yml
  project: your-project-name

Before we write the code for the function, install googleapis package.

npm install googleapis

Let's write some un-implemented functions responsible for querying Google calendar API.

'use strict';
const { google } = require("googleapis");
const calendars = [{id: 'calender-id-one'}, {id: 'calender-id-two'}] 

exports.http = async (request, response) => {};

async function calendarClient()

async function freebusyCheck()

In the code snippet above, calenderClient function will return an instance of authenticated Google calendar client that can be used to construct the query against Google calendar API.

freebusyCheck function contains the logic used in querying google calendar free busy API.

Constant calendars contains an ID of calendars to query.

Let's implement the calendarClient method as:

async function calendarClient() {
    const auth = new google.auth.GoogleAuth({
        scopes: "https://www.googleapis.com/auth/calendar"
    });

    const authClient = await auth.getClient();

    return google.calendar({
        version: "v3",
        auth: authClient
    });
}

And then we implement the freeBusyCheck method as:

async function freebusyCheck() {
    let timeMin = new Date();
    let timeMax = new Date();
    timeMax = new Date(timeMax.setMinutes(timeMax.getMinutes() + 60)); // free/busy check for the next 1 hour

    const request = {
        resource: {
            timeMin: timeMin.toISOString(),
            timeMax: timeMax.toISOString(),
            items: calendars
        }
    };
    const client = await calendarClient();

    return client.freebusy
        .query(request, "POST")
        .then(response => response.data);
}

The method above should be easy to grasp. We constructed a request object that contains according to the free busy API specifications. We built the start time of the interval of the query indicated by timeMin, the end of the interval of the query denoted by timeMax and a list of calendars to query.

If we put everything together, we'll have:

'use strict';

const { google } = require("googleapis");
const calendars = [{ id: 'calender-id-one' }, { id: 'calender-id-two' }]

exports.http = (request, response) => {

    let data = await freebusyCheck();
    response.set('Access-Control-Allow-Origin', "*")
    response.set('Access-Control-Allow-Methods', 'GET, POST')
    response.status(200).send(data);

};

async function calendarClient() {
    const auth = new google.auth.GoogleAuth({
        scopes: "https://www.googleapis.com/auth/calendar"
    });

    const authClient = await auth.getClient();

    return google.calendar({
        version: "v3",
        auth: authClient
    });
}

async function freebusyCheck() {
    let timeMin = new Date();
    let timeMax = new Date();
    timeMax = new Date(timeMax.setMinutes(timeMax.getMinutes() + 60));

    const request = {
        resource: {
            timeMin: timeMin.toISOString(),
            timeMax: timeMax.toISOString(),
            items: calendars
        }
    };
    const client = await calendarClient();

    return client.freebusy
        .query(request, "POST")
        .then(response => response.data);
}

Having written the cloudfunction, it's time to deploy it to Google Cloud Platform.

Deploying to Google Cloud Platform

The serverless framework needs access to your google cloud account to provision resources on your behalf. This means you'll need to create a service account with a JSON key file that will be used for deployment.

To create a service account with a JSON key file, follow the following steps:

Login to your google cloud console and select your project
At the left menu, click on "IAM & admin."
Click on service accounts, click "Create service account."
Type in the name you want for the service account and click "create."
The second step is where you select the roles your service account should have, select the following roles
- Deployment Manager Editor
- Storage Admin
- Logging Admin
- Cloud Functions Developer
Click the "+CREATE KEY" button and select the JSON key type and click the "Create" button.
Your key will be automatically downloaded.

Now, open up serverless.yml file and update the credentials path so that it points to the absolute path of the key file you just downloaded.

provider:
  name: google
  stage: dev
  runtime: nodejs8
  region: us-central1
  project: my-project-name
  credentials: ~/Path/to/Downloaded/keyfile.json

Now execute, serverless deploy to deploy your function. You'll get an output similar to this:

.........
Serverless: Done...
Service Information
service: gcf-nodejs
project: calenderapp-262309
stage: dev
region: us-central1

Deployed functions
first
  https://us-central1-calenderapp-262309.cloudfunctions.net/http

Your function is deployed on your GCP. You can trigger the function by accessing the endpoint. Mine is https://us-central1-calenderapp-262309.cloudfunctions.net/http.

At this point, if you make a GET request to this endpoint , you'll most likely get a response object with an errors like this:

{
    "kind": "calendar#freeBusy",
    "timeMin": "2019-12-19T10:21:16.000Z",
    "timeMax": "2019-12-19T10:51:16.000Z",
    "calendars": {
        "your-calendar-id": {
            "errors": [{
                "domain": "global",
                "reason": "notFound"
            }],
            "busy": []
        }
    }
}

Why? Your application does not have access to the calendar it's trying to access.

To make this work, you need to allow your project to have access to the calender.

Granting your app permissions to calendar API

When you create a serverless app on GCP, a default service account is created for your app. This service account is the identity of your application.

To grant your app permissions to a calendar, follow these steps:

In each of the calendar, you need to grant your app permission to view event details:

Copy the email address of the App Engine default service account
Open the interface of the calendar you want to manage, and click on settings
Click on share with some specific people, click on "add people" button
Type in the email address and select "see all event details" in the permissions select box
Click on "send" to save permission settings.

If your make curl request to the endpoint again, you'll be able to get the freeBusy information without errors.

{
    "kind": "calendar#freeBusy",
    "timeMin": "2019-12-19T10:50:29.000Z",
    "timeMax": "2019-12-19T11:20:29.000Z",
    "calendars": {
        "your-calendar-id": {
            "busy": []
        }
    }
}

You have learned how to create a serverless application on Google Cloud Platform, you might be interested in how to create a serverless application on AWS as well.

References

Build a Todo App with Node.Js, ExpressJs, MongoDB and VueJs - Part 3

Samuel James — Sun, 10 Nov 2019 12:50:59 +0000

If you have made it to Part 1 and Part 2 of this tutorial, you deserve some accolades. And If you have not, you need to check 1 and 2 out.

Let's take a moment to reflect on what we went through and how container technologies like docker could make this process easier.

In the previous tutorials, we installed Node runtime and MongoDB. The installation process was tedious and required a lot of manual efforts.

Not only that, If Bill, who has a different environment, tries to run the application, chances are the application would not run correctly. This happens most of the time because of inconsistency in environments and disparity in system libraries. Using virtualization technologies like docker ensures that your applications run the same in all environments.

This abolishes rooms for a famous saying, "It works on my machine." With docker, you can maintain a consistent environment everywhere.

What is docker?

Docker is a container platform that allows you to seamlessly build, share, and run applications in any environment. With docker, you can define instructions on how your application should be built using DockerFile.

Installing docker and docker-compose tool

Docker is very easy to install. Irrespective of your operating system, you can find a distribution that works for your type of OS. Head on to docker install page to download an executable docker file for your operating system.

Since we'll be using creating multiple containers for this application, you'll need to install docker-compose tool – a tool that lets you define and run multi-container Docker applications. Proceed to the official download page to download one for your operating system.

Creating Dockerfile for backend Service

A Dockerfile is a text file that contains a set of instructions. Docker read these instructions to build an image.

Navigate to the backend directory and create a Dockerfile.

$ cd backend
$ touch Dockerfile

Update the content of the Dockerfile with this content.

# backend/Dockerfile

FROM node:12.2.0-alpine 

WORKDIR /usr/app

COPY package*.json ./

RUN npm install

COPY . . 

CMD [ "npm", "start" ]

In this Dockerfile, we used 5 different docker commands.

FROM allows you to specify a base image for your image. In this case, we based from node:12.2.0-alphine
WORKDIR allows you to set the current working directory. We set the current working directory to usr/app.
COPY: This command let you copy files, directories or remote contents into the image file system. We copied to our package.json and package.lock.json to the docker image
RUN : This command is used to execute command in the shell. RUN command creates a new layer. We use RUN to execute npm install which installs our project dependencies.
COPY: We used copy to copy our source code in to the container again
CMD lets you set command to be executed when the container runs.

Let's proceed to create another Dockerfile in the frontend directory for the frontend. The commands will be simillar to the one we just created.

$ cd  frontend

$ touch Dockerfile

Update the Dockerfile you created with:

#frontend/Dockerfile

FROM node:12.2.0-alpine

WORKDIR /app

COPY package*.json ./

RUN npm install

COPY . .

CMD ["npm", "run", "serve"]

Finally, we create a docker-compose file at the root directory of the project.

version: "3"
services:
  backend:
    container_name: backend 
    build:
        context: ./backend
    depends_on:
      - db  
    volumes:
      - ./backend:/usr/app
      - /usr/app/node_modules
    environment:  
      - MONGO_URL=mongodb://db:27017/todos
      - APP_PORT=80
    ports: ['80:80'] 

  db:
    container_name: db
    image: mongo:4.0
    restart: always

  frontend:
    container_name: frontend
    build:
      context: ./frontend
    volumes:
      - ./frontend:/app
      - /app/node_modules
    ports:
      - '8080:8080'
    environment:  
      - BACKEND_URL=http://localhost/todos

In the docker-compose file, we defined all our services.

The backend container is built from the instructions defined in Dockerfile in the backend directory.

We used depends_on to specify dependencies for the container.

We used volume to share data between our local and host machine.

This is ideal for a development environment because we want to see our changes as we make them. You should not do that in production.

We defined 2 environment variables APP_PORT, MONGO_URL, and finally port 80 in the container to port 80 on our environment.

Finally, execute docker-compose up -d to bring up your containers.

If you go to http://localhost:8080, you see your app running.

You find the source code for this app on Github

Some useful links

Vue-native Infinite scroll

Samuel James — Sun, 28 Oct 2018 21:30:30 +0000

This post is aimed to show you how to implement infinite scroll in Vue-native. Without doubts, infinite scroll is your best bet when it comes to paginating large data set, especially in mobile apps.

It's exciting to know that infinite scroll can be implemented in a few easy steps with Vue-native

What we will use

Install Vue-native-cli

Install vue-native cli if it's not installed.

npm install -g vue-native-cli

Start a Vue-native project

vue-native init infinteScroll

npm install

Select blank from the options as shown below:

At this point, everything you need to start a vue-native application should have been created. Sweet! right? Let’s go ahead and install a few more dependencies.

Navigate to the project root and run:

npm install -s axios native-base

Making a 'UserList' Component

Create directory component and add a file UserList.vue in the directory.

// component/UserList.vue

<template>
    <nb-container>
        <nb-header>
                <nb-title>Infinite Scrolling</nb-title>
        </nb-header>
        <scroll-view :on-scroll="(event) => {loadMore(event)}" :scroll-event-throttle="400">
            <nb-content>
                <nb-list-item avatar v-for="user in users">
                    <nb-left>
                        <nb-thumbnail small :source="{uri: user.avatar}"/>
                    </nb-left>
                    <nb-body>
                        <nb-text>{{user.first_name}} {{user.last_name }}</nb-text>
                    </nb-body>
                </nb-list-item>
            </nb-content>
        </scroll-view>
        <view :style="{flex: 1, justifyContent: 'center'}" v-if="loading">
            <activity-indicator size="large" color="gray"/>
        </view>
    </nb-container>

</template>

<script>
    import axios from "axios"

    export default {
        data: function () {
            return {
                loading: false,
                users: [],
                per_page: 15
            }
        },
        mounted: function () {
            this.getData();
        },

        methods: {
            getData: function () {
                let uri = 'https://reqres.in/api/users';
                this.loading = true;
                axios.get(uri, {
                    params: {
                        per_page: this.per_page
                    }
                }).then((result) => {
                    let response = result.data.data;
                    console.log(response);
                    for (let i in response) {
                        this.users.push(response[i]);
                    }
                    this.loading = false;
                }).catch((error) => {
                    console.log(error)
                })
            },
            loadMore: function (event) {
                let paddingToBottom = 0;
                paddingToBottom += event.nativeEvent.layoutMeasurement.height;
                if (event.nativeEvent.contentOffset.y >= event.nativeEvent.contentSize.height - paddingToBottom) {
                    this.getData()
                }
            }
        }
    }
</script>

What the heck is happening in UserList.vue?

We listen to scroll event which is fired once per frame, we then call loadMore which receives event as argument. Basically, it detects the end of scrolling and calls getData if a user has scrolled to the bottom.

We control how often this event is fired by setting scroll-event-throttle to 400.

When users scroll down, we also want to give them a sense that more data are being loaded, so, we added activity-indicator that becomes visible when loading is true.

Wrapping it off

If you take a look at the root folder, you will see App.vue - a file created when we ran vue-native init infinteScroll to generate the boilerplate.

We update App.vue with:

//App.vue

<template>
    <view class="container">
        <nb-container>
            <root>
                <user-list v-if="isReady"></user-list>
            </root>
        </nb-container>
    </view>
</template>

<script>
    import UserList from "./component/UserList"
    import {Root, VueNativeBase} from "native-base";
    import Vue from "vue-native-core";

    Vue.use(VueNativeBase);
    export default {
        data: function () {
            return {
                isReady: false
            }
        },
        created: function () {
            this.loadAppFonts();
        },
        methods: {
            loadAppFonts: async function () {
                try {
                    this.isReady = false;
                    await Expo.Font.loadAsync({
                        Roboto: require("native-base/Fonts/Roboto.ttf"),
                        Roboto_medium: require("native-base/Fonts/Roboto_medium.ttf"),
                        Ionicons: require("@expo/vector-icons/fonts/Ionicons.ttf")
                    });
                    this.isReady = true;
                } catch (error) {
                    console.log("Can't load fonts", error);
                    this.isReady = true;
                }
            },
        },
        components: {
            Root, UserList
        }

    }
</script>
<style>
    .container {
        justify-content: center;
        flex: 1;
    }
</style>

Above, we imported UserList component so we can reuse it in App.vue (in the root instance) and define a method: loadAppFonts loads custom fonts required by native base asynchronously.

Previewing the app

If you are on iOS and you have iOS emulator installed, you can preview the app by running npm run ios. On Android, run npm run android (Android build tools must be installed)

You can also find the project here

Laying the ground for your career as a software developer

Samuel James — Thu, 28 Jun 2018 08:37:04 +0000

Adlai Stevenson, an American politician made a statement many years ago and today I find it ever striking and applicable to all aspects of life. He said:

Change is inevitable. Change for the better is a full-time job

Getting your dream job can be easy as well as hard, who knows what life will throw at you when you definitely want a change. While there are many reasons people change or desire a new job, it's can still be summed up in: "change is inevitable" and whether we like it or not, there will be a point in time when our back is pressed against the wall and the desire to get that new job becomes higher than ever.

Surprisingly, change for the better is a full-time job. While I see nothing wrong with scrambling for jobs when you are in need of one with hundreds of candidates, I have come to understand there are things you can do to turn the odds in your favor. This involves taking conscious steps towards your career even when you are not hunting for jobs.

How do you position yourself for more jobs?

Have a career plan

No one wants to remain in the same position forever and this is why you need to plan your career. A career plan will you help you identify what your career goals are. You can not take charge of your advancement career-wise when you don't have a Goddamn plan written down.

Gayle LaakMann McDowell, in her book says: " a written career path will ensure that you understand, up front, how long you intend to be at a company and what you believe you’ll get out of it"

Not does it only motivate you towards your end goal but also helps you to take advantage of opportunities not seen by the majority.

Great careers don't happen by accident. Truly satisfying careers are the reward of talent, hard work, a bit of luck, and a strategy carefully managed and put to work -Joya Cousin"

Make Connections.

Don't wait till you need a job before you start making connections. Connection is a crazy bitch that works in unexpected manners. The chubby guy you met at a hackathon might be the guy that will get you to the door of your next great job.

I've once had a phone conversation with a stranger who thought I would be great for the company he worked for. I agreed to have a coffee with him even though I knew I was not interested in the opportunity. I wanted to meet his person, talk about technology and possibly connect. We had a nice conversation, he understood more about me and the type of opportunity I was interested in. Some few months later, he got me to the door of a company I was interested in.

Countless developers are still getting jobs through connections and you can too. If your goal is to work with company X, you better start connecting with people working with company X and start learning about the company. You can extend your interest to projects they are working on.

Making connections is not limited to conferences or tech meetups. You can also make connections with a decent profile on various social media. LinkedIn, Twitter and Facebook are all great platforms to connect with people you share similar interests with.

While it's important to have people with similar interests within your network; it's equally important to keep a handful of recruiters among them. They are the people that will facilitate your buy-in. You have the skills, recruiters can get you to the door.

Let your resume tells your achievements and not your job description

A decent resume is a very important element in a job search. Your resume determines whether you get called for an interview or not. Many developers failed at this point.

If you are anything like me some few years back, you would probably make a duplicate of your job description into the experience section of your resume. In fact, some developers waste substantial space on their resume with mundane things like date of birth, gender and even place of birth.

Please stop! Research has shown that recruiters spend less than 6 secs scanning resumes. You want to make your resume attractive as much as you can.

Employers want to see your achievements, not your job description. They are more interested in what you did than what you were told to do.

Let's examine this statement I picked from a developer's resume.

Drove productivity among team members

The question is how did he drive productivity? What was the outcome before and after he drove productivity? You'll likely find this in most resumes.

A better way to represent this would have been:

Drove team productivity and cohesiveness by conducting regular training which in turn increased revenue by $10 million in 6 months.

Always measure the impact of everything you do. Whether it's creating an algorithm that performs 10x faster or saving the company's millions of dollar by detecting and fixing a security bug that could have been exploited by cybercriminals.

Keep track of your achievements at your current job

Having a great resume or work experience is not enough and one of the worst mistakes you can make is not coming to an interview without your past achievements to sway your interviewers.

Interviewer's number one priority is to validate your potentials by listening to as you unravel your heroic achievements.

A lot happens in one's career and it easy to forget them if they are not documented. Keep track of your achievements and ensure you go through them before going in for an interview. Your achievements will not only allow you to see how valuable you are to your company but also comes handy during a performance review. Who knows you might finally get a raise! :)

Take time to craft your application

Most people throw their applications randomly at any job they see and hope for the best. I'm sorry to go against this paradigm: it does not always work this way. As success with anything is deliberate so is success with a job application. It's when you read a job's requirements and understand the business needs you can tailor your application to catch recruiters' attention.

There is no fast rule about it. Crafting a good application requires creativity and time. Most of the time, it flows naturally when you are genuinely interested in the role.

Don't just throw your application anywhere. Ensure you are a good fit and at least let it be a role you are interested in.

Recruitment is expensive and recruiters want to make sure right candidates are interested in the role as much as they are in having him. Recruiters don't like it when you pull off at die minute. It's a total waste of time for everyone. That is why questions like 'What are your motivations, why did you apply for this role, why do you want to relocate' are common in HR interviews.

Don't use a general/multi-purpose resume for all applications. Each application should have its own unique cover letter and resume tailored to it. I know you are probably fluent in 6 programming languages but you've got to cut that crap to what the job demands bruh. Recruiters value specialization

Before you send your application next time, try and put yourself in recruiters' shoes. If you were hiring for the position you are applying for, would you have found your application deserving a call for an interview?

Create a personal blog

I can not stress enough the importance of having a personal blog as a developer. I can assure you 9 out of 10 recruiters will check your personal blogs before inviting you in. Why? It's the best way to know more of you from afar. Your interest, your tech stack and what your experiences look like. Believe me, recruiters just kinda have some deep feelings for this thing, man. Go create one. They are many free content management systems (CMSs) that won't take a bunch of your time to set up.

Conclusion

You are probably going to fail a couple of times before you finally land your dream job. Do not be disturbed. Success is wrapped in failure most of the time. Trust the process and learn from your mistakes. Track your applications. This includes your resume, cover letter, the company, position, responses and feedback you get from them. With time, you will see a pattern that clearly indicates what you should improve on. Huntr is a good app you can use to keep track of your applications.

Do you have a comment? Please shoot. You want to connect? I'm happy to meet you on LinkedIn and Twitter.

Why Docker? Creating a multi-container application with docker

Samuel James — Sat, 05 May 2018 13:39:49 +0000

If you are just starting to learn docker, one of the common questions that come to mind is what docker is and why one would need it.
In this post, I provide basic answers to some of these questions and finally walk you through composing your first multi-container application with docker-compose.

Docker and why
Docker Image
Docker compose
Building multi-container application with docker-compose
Further Reading

Docker and why?

The definition I love personally about docker is the one provided by the docker team, who defines docker as an open platform for developers and sysadmins to build, ship, and run distributed applications, whether on laptops, data center virtual machines, or the cloud.

Maintaining consistent environments is hard

Without virtualization technologies, applications are not guaranteed to work exactly the same way as they work in one environment. This is mostly due to inconsistent environments and disparity in system libraries.

You could develop your app using python3 only to find out that production environment is running python2. It gets a bit complicated when you cannot upgrade to python3 because other applications on the host need python2 to run.

Developers work their ass off for hours finding solutions to inconsistent environments and yet at most times, their shining code won't still run because one library file is outdated.

The struggle is endless. I have experienced this agony that comes with troubleshooting a feature that works fine in one environment and not the other. Yeah!, It always works fine on our machines but production - never! :(

Installing application dependencies on local machine takes time

Setting up development environment can be hard without virtualization, we often find ourselves installing different versions of software to support various services we are working on. Maintaining different versions of PHP or MySQL database can take a considerable amount of your time that could be spent writing code.

Containers and virtual machines solve this problem and save you the stress by isolating your application and its dependencies into a single self-contained unit that can reliably run anywhere.

While virtual machines and containers have the same goal, containers are preferred to virtual machines because of their lightweight, speed, and low resource utilization.

A container in this context is similar in function to the one you see on highways: They are both enclosures for holding products. The difference is the content they hold. Ours hold software products.

One of the few popular container providers is Docker and in the next few paragraphs, we'll be exploring in details how you can dockerize your development environment.

Docker Image

Docker containers are built from docker images. An image is inert and immutable file that is essentially a snapshot of a container. I like to see a docker container as a running instance of docker image.

An image is built from Dockerfile which is basically a text document that contains instructions on how docker should build an image.

Docker compose

Docker-compose is a tool that helps you run complex applications.
A working software is never an island, it will most likely depend on a bunch of other things like database, cache and even other services. It's best practice to have each of these in their respective container. With docker-compose, you can define a multi-container application in a single file which can be spun up in a single command.

Building a multi-container application with docker-compose

Before this post gets unbearably long, let gets our hands dirty.
If you don't have docker and docker-compose installed, go here for install instructions.
We'll compose multiple services for our project. It's a laravel application and uses MySQL database and Redis for caching.

### Start a new Laravel project

Start a Laravel project by running composer create-project --prefer-dist laravel/laravel laravelApp
If everything works fine, your directories should look like this.

### Create a Dockerfile for the Webserver (Nginx)
We need a web server (nginx) to serve static contents. We create a Dockerfile which is basically a text file that contains instructions of how docker should build our image.
At the project root, we create a web.docker file with the content below.

 # /project/root/web.docker

 # start building image from nginx 1.13
 FROM nginx:1.13

 # copy site.conf from local directory to /etc/nginx/conf.d/default.conf
 ADD ./site.conf /etc/nginx/conf.d/default.conf

 WORKDIR /var/www

site.conf contains vritual host configuration for nginx
At the project root, create site.conf with this content

# /project/root/site.conf

server {
    listen 80;
    index index.php index.html;

    access_log /var/www/storage/logs/access.log;
    error_log /var/www/storage/logs/error.log;
    root /var/www/public;

    location / {
        try_files $uri /index.php?$args;
    }

    location ~ \.php$ {
        fastcgi_split_path_info ^(.+\.php)(/.+)$;
        fastcgi_pass app:9000;
        fastcgi_index index.php;
        include fastcgi_params;
        fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
        fastcgi_param PATH_INFO $fastcgi_path_info;
        add_header 'Access-Control-Allow-Origin' '*' always;
        add_header 'Access-Control-Allow-Headers' 'Authorization,DNT,X-Mx-ReqToken,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,api_key' always;
        add_header 'Access-Control-Allow-Methods' 'POST, GET, PUT, DELETE' always;
    }
}

Create a Dockerfile for PHP 7.1

Laravel needs PHP to run as well as some specific PHP extensions. We create a file with the name app.docker which contains the version of PHP and extensions we want.
Create this at the project root as well.

FROM php:7.1.9-fpm

# Install any custom system requirements here
RUN apt-get update && apt-get install -y libmcrypt-dev zip unzip git mysql-client \
    && docker-php-ext-install mcrypt pdo_mysql

WORKDIR /var/www

As for MySQL database and Redis, prebuilt images will suffice for this demonstration. We do not have to customize them.

Define services in Docker-compose file

Now, we create a docker-compose.yml. This should be at the root directory of your project.

version: '2'
services:
 web:
  build:
   context: ./
   dockerfile: web.docker
  container_name: web
  ports:
   - "8080:80"
  volumes:
   - ./:/var/www
  links:
   - app
 app:
  build:
   context: ./
   dockerfile: app.docker
  volumes:
   - ./:/var/www
  depends_on:
   - mysql
   - redis
  environment:
   - "APP_ENV=${APP_ENV}"
   - "APP_DEBUG=${APP_DEBUG}"
   - "APP_KEY=${APP_KEY}"
   - "DB_CONNECTION=${DB_CONNECTION}"
   - "DB_HOST=${DB_HOST}"
   - "DB_PORT=${DB_PORT}"
   - "DB_DATABASE=${DB_DATABASE}"
   - "DB_USERNAME=${DB_USERNAME}"
   - "DB_PASSWORD=${DB_PASSWORD}"
   - "CACHE_DRIVER=${CACHE_DRIVER}"
   - "QUEUE_DRIVER=${QUEUE_DRIVER}"
   - "REDIS_HOST=${REDIS_HOST}"
   - "REDIS_PASSWORD=${REDIS_PASSWORD}"
   - "REDIS_PORT=${REDIS_PORT}"
 mysql:
  image: mysql:5.7
  environment:
   MYSQL_DATABASE: ${DB_DATABASE}
   MYSQL_USER: ${DB_USERNAME}
   MYSQL_PASSWORD: ${DB_PASSWORD}
  volumes:
    - ./mysql:/var/lib/mysql
  ports:
   - "3306:3306"
 redis:
  image: redis:3.0
  ports:
    - "6379:6379"
  volumes:
   - ./redis:/data

Let's take a moment to understand the content of docker-compose.yml above.

version 2: Version of docker-compose syntax we are using.

services: Services are different pieces of our application i.e (app, redis, mysql).

build: It defines a string which contains the path to the build context and dockerfile argument that references our Dockerfile

container_name: Used to specify the name of the container after built

environment: This is used to pass environment variables to containers.

Note: Docker compose will make variables defined in .env available in service configuration. This means you can automatically access laravel configuration in .env file in docker-compose.yml.

image: Used to specify a prebuilt image to start building the container from.

Ports: used to map container's port to host's port. In app service, we mapped port 8080 on the host machine to port 80 in the container.

volumes: used to mount container's directory to host's directory. In this example, docker will create /var/www directory in app service and point files in your local directory to it. This is useful during development, If you make changes to your source code, the changes are effected immediately.

depends_on: This ensures that mysql and redis are started before app service.

During execution, docker-compose builds a network between the containers such that each containers can refer to each other by using names defined in the docker-compose.yml.

Create a DockerIgnore file

A dockerignore file is similar to gitignore file. With dockerignore file, you specify files and folders that will be excluded from the build and these files will not be copied over to the image.

Let's create a dockerignore file at root folder with code below

app.docker
web.docker
docker-compose.yml
.env

Build and run project with Compose

To build the project, we navigate to the project root and run:

docker-compose build

You should see a console output similar to this:

Now, we can proceed to run our services with:

docker-compose up -d

To verify that these services are actually running, we issue docker ps command. This will list all running containers.

 docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                    NAMES
7275af6e2f3f        laravelapp_web      "nginx -g 'daemon of…"   9 seconds ago       Up 6 seconds        0.0.0.0:8080->80/tcp     web
2ce0817bd07e        laravelapp_app      "docker-php-entrypoi…"   10 seconds ago      Up 8 seconds        9000/tcp                 laravelapp_app_1
5a85911e6ecd        mysql:5.7           "docker-entrypoint.s…"   58 seconds ago      Up 11 seconds       0.0.0.0:3306->3306/tcp   laravelapp_mysql_1
41739940ec82        redis:3.0           "docker-entrypoint.s…"   58 seconds ago      Up 10 seconds       0.0.0.0:6379->6379/tcp   laravelapp_redis_1

Finally, point your web browser to http://localhost:8080/ and you should see a default laravel homepage as shown below.

Since we have already mounted the project's directory to application container's directory, changes made to the source code are automatically applied.

Build a Todo App with Node.Js, ExpressJs, MongoDB and VueJs – Part 2

Samuel James — Sun, 11 Feb 2018 13:19:17 +0000

In part 1 of this tutorial, we built APIs for a simple todo application and now we are here to put the front end together with VueJS. Dont worry if you are new to VueJs. I wrote VueJs: The basics in 4 mins and Creating your first component in VueJs to help you pick up VueJs in no time.

Project Directories

In part 1, we created backend directory. The backend directory contains the source code for our backend code.

We'll do something simillar here. Let's create a new directory with the name frontend. This will house our frontend code.

$ mkdir frontend

If you run the command above, your project directory should now look like this:

.
├── backend
└── frontend

All our code in this post will go in to frontend directory.

Vue CLI

Vue CLI is a command line tool that helps you quickly scaffold a new project. To install Vue CLI, run the command below from your terminal:

$ npm install -g @vue/cli

With the Vue Cli installed, go to frontend directory run vue create . from the command to scaffold a new project.

$ vue create .

Ensure you answer yes to all prompts.

Vue CLI v3.5.1
? Generate project in current directory? Yes

? Please pick a preset: default (babel, eslint)

If everything went fine, your frontend directory will look this:

├── README.md
├── babel.config.js
├── node_modules
├── package-lock.json
├── package.json
├── public
│   ├── favicon.ico
│   └── index.html
└── src
    ├── App.vue
    ├── assets
    │   └── logo.png
    ├── components
    │   └── HelloWorld.vue
    └── main.js

Project Dependencies

Bootstrap Vue : A Vue compatible boostrap framework
Sass loader: Compiles sass to css
Axios : For making rest API calls to todo API

Install bootstrap-vue and axis with the command:

$ npm install vue bootstrap-vue bootstrap axios

Install sass-loader with the command:

$ npm install sass-loader node-sass --save-dev

In the following paragraph, we'll create components we need for this project.

Creating the Vue Components

Basically, we need 2 major vue components. The first component will be CreateTodo and the second will be ListTodo

At some points, these components would need to communicate or share data with each other and this where event bus comes into play.

One of the ways to handle communications between components in Vue.Js is to use a global event bus such that when a component emits an event, an event bus transmits this event to other listening components.

Event Bus

We create a global event bus with the name src/bus.js with the code:

//src/bus.js

import Vue from 'vue';

const bus = new Vue();
export default bus;

Now that we have created an event bus, let write the code for adding new todo items.

Vue Component for adding new todo items

Create a new file at src/components/CreateTodo.vue and update its content with:


<template>
  <div class="col align-self-center">
    <h3 class="pb-5 text-left underline">Create todos</h3>
    <form class="sign-in" @submit.prevent>
      <div class="form-group todo__row">
        <input
          type="text"
          class="form-control"
          @keypress="typing=true"
          placeholder="What do you want to do?"
          v-model="name"
          @keyup.enter="addTodo($event)"
        />
        <small class="form-text text-muted" v-show="typing">Hit enter to save</small>
      </div>
    </form>
  </div>
</template>
<script>
import axios from "axios";
import bus from "./../bus.js";

export default {
  data() {
    return {
      name: "",
      typing: false
    };
  },
  methods: {
    addTodo(event) {
      if (event) event.preventDefault();
      let todo = {
        name: this.name,
        done: false //false by default
      };
      console.log(todo);
      this.$http
        .post("/", todo)
        .then(response => {
          this.clearTodo();
          this.refreshTodo();
          this.typing = false;
        })
        .catch(error => {
          console.log(error);
        });
    },

    clearTodo() {
      this.name = "";
    },

    refreshTodo() {
      bus.$emit("refreshTodo");
    }
  }
};
</script>
<style lang="scss" scoped>
.underline {
  text-decoration: underline;
}
</style>

addTodo() is executed once an enter key is pressed. It makes a POST request to the backend with the new todo item.
clearTodo() clears the input box once the todo item is saved.
refreshTodo() emits an event refreshTodo. This is useful when you add a new todo item. It makes sense to re-render the list so that the new item is displayed.

That explained, let's go ahead to create ListTodo component.

Component for listing todo items

Create a file src/components/ListTodo.vue with the code:

<template>
  <div v-bind:show="todos.length>0" class="col align-self-center">
    <div class="form-row align-items-center" v-for="todo in todos">
      <div class="col-auto my-1">
        <div class="input-group mb-3 todo__row">
          <div class="input-group-prepend">
            <span class="input-group-text">
              <input
                type="checkbox"
                v-model="todo.done"
                :checked="todo.done"
                :value="todo.done"
                v-on:change="updateTodo(todo)"
                title="Mark as done?"
              />
            </span>
          </div>
          <input
            type="text"
            class="form-control"
            :class="todo.done?'todo__done':''"
            v-model="todo.name"
            @keypress="todo.editing=true"
            @keyup.enter="updateTodo(todo)"
          />
          <div class="input-group-append">
            <div class="input-group-text">
              <span
                class="input-group-addon addon-left"
                title="Delete todo?"
                v-on:click="deleteTodo(todo._id)"
              >
                X
              </span>
            </div>
          </div>
        </div>
      </div>
    </div>
    <div
      class="alert alert-primary todo__row"
      v-show="todos.length==0 && doneLoading"
    >Hardest worker in the room. No more todos now you can rest. ;)</div>
  </div>
</template>

<script>
import axios from "axios";
import bus from "./../bus.js";

export default {
  data() {
    return {
      todos: [],
      doneLoading: false
    };
  },
  created: function() {
    this.fetchTodo();
    this.listenToEvents();
  },
  watch: {
    $route: function() {
      let self = this;
      self.doneLoading = false;
      self.fetchData().then(function() {
        self.doneLoading = true;
      });
    }
  },
  methods: {
    fetchTodo() {
      this.$http.get("/").then(response => {
        this.todos = response.data;
      });
    },

    updateTodo(todo) {
      let id = todo._id;
      this.$http
        .put(`/${id}`, todo)
        .then(response => {
          console.log(response);
        })
        .catch(error => {
          console.log(error);
        });
    },

    deleteTodo(id) {
      this.$http.delete(`/${id}`).then(response => {
        this.fetchTodo();
      });
    },

    listenToEvents() {
      bus.$on("refreshTodo", $event => {
        this.fetchTodo(); //update todo
      });
    }
  }
};
</script>

<style lang="scss" scoped>
.todo__done {
  text-decoration: line-through !important;
}

.no_border_left_right {
  border-left: 0px;
  border-right: 0px;
}

.flat_form {
  border-radius: 0px;
}

.mrb-10 {
  margin-bottom: 10px;
}

.addon-left {
  background-color: none !important;
  border-left: 0px !important;
  cursor: pointer !important;
}

.addon-right {
  background-color: none !important;
  border-right: 0px !important;
}
</style>

Let's take a moment to explain what is going on in the code.

We created 4 functions in the snippet.

fetchTodo() makes a GET call to backend and gets all todo items.
updateTodo(todo) is called when you make changes to todo items and hit enter. It forwards your changes to the backend.
deleteTodo(id) runs when you click the trash button. It makes DELETE requests to the backend.
listenToEvents(): In CreateTodo component, we emit events when a new todo item is added so the list. ListTodo is responsible for rendering todo items. This method does the job of listening for refreshTodo event.

App Component

Below we wrap all our components in a parent component named App.vue. Update the file src/App.vue with this content:

<template>
  <div class="container">
    <div class="row vertical-centre justify-content-center mt-50">
      <div class="col-md-6 mx-auto">
        <CreateTodo></CreateTodo>
        <ListTodo></ListTodo>
      </div>
    </div>
  </div>
</template>

<script>
import CreateTodo from "./components/CreateTodo.vue";
import ListTodo from "./components/ListTodo.vue";

export default {
  name: "app",
  components: { CreateTodo, ListTodo }
};
</script>

<style lang="scss">
@import "node_modules/bootstrap/scss/bootstrap";
@import "node_modules/bootstrap-vue/src/index.scss";
.vertical-centre {
  min-height: 100%;
  min-height: 100vh;
  display: flex;
  align-items: center;
}
.todo__row {
  width: 400px;
}
</style>

Root Instance

A root instance must be defined for every vue application. You can see a Vue instance or root instance as the root of the tree of components that make up our app.

Let's modify the content of src/main.js file with:

import Vue from 'vue';
import BootstrapVue from 'bootstrap-vue';
import axios from 'axios';
import App from './App.vue';

const http = axios.create({
  baseURL: process.env.BACKEND_URL ? process.env.BACKEND_URL : 'http://localhost/todos',
});

Vue.prototype.$http = http;

Vue.use(BootstrapVue);

Vue.config.productionTip = false;

new Vue({
  render: (h) => h(App),
}).$mount('#app');

We imported BoostrapVue and other libraries needed by the application.
We also imported App component and defined it as a component on the root instance.

We imported axios, an http client and we configured the base url of the backend application. You should ensure the baseUrl matches your backend URL.

We've come this far, run the application with:

$ npm run serve

It could take a few moment to build. At the end, you should have a URL printend in the console:

App running at:
  - Local:   http://localhost:8080/ 
  - Network: http://192.168.178.20:8080/
  Note that the development build is not optimized.
  To create a production build, run npm run build.

If you navigate to http://localhost:8080, you should be greeted with a page like this.

To connect the frontend app with the backend, you also need to start the backend server.

Navigate to backend directory and run

$ npm start

Note:

Your MongoDB connection URL must be properly configure in backend/config/Config.js and the MongoDB must be running.
Your backend server must be running.
Your frontend server must be running.

If you navigate to http://localhost:8080, you will be greeted with a page like this.

Get the source code here

Build a Todo App with Node.Js, ExpressJs, MongoDB and VueJs – Part 1

Samuel James — Thu, 08 Feb 2018 12:15:53 +0000

In this tutorial, we will build the famous todo application with Node.Js using the ExpressJs framework and MongoDB. Did I forget to tell you? The app will be API centric and full-stack :).

In a nutshell, if you want to learn how to build APIs with Node.Js, you have come to the right place.

Now you can grab a bottle of beer and let's get our hands dirty.

What is ExpressJs?

ExpressJs simply put, it's a web framework for Node.Js - stolen from the official docs. Taylor Otwell (Laravel's creator) once said, "Developers build tools for developers". ExpressJs was built for developers to simplify Node APIs.

What is MongoDB?

MongoDB is a NoSQL database. It's completely document-oriented. A NoSQL database allows you to store data in the form of JSON and any formats. If you want to learn more about MongoDB, I have also written a post on MongoDB here.

Defining Todo APIs

I like to begin by defining my APIs. The table below shows what APIs we need to create and what each does.

Method	Path	Description
GET	/todos	Get all todos items
GET	/todos/:id	Get one todo item
POST	/todos	Create a new todo item
PUT	/todos/:id	Update a todo item
DELETE	/todos/:id	Delete a new todo item

Having defined our APIs, let's delve in to the project directories.

Project Directories

You are probably going to get a shock of your life when I tell ya we need not more than 5 files with relatively few lines of code to build this backend. Yes! It is that simple. This how we roll man :).

You should create a project folder at this point to house all the source code for the backend part of this application. I'll call my backend. Feel free to use the backend as your directory. It's not patented.:)

To keep things simple, and of course, we should not throw away flexibility at the sight of simplicity. We'll create just 4 folders in the directory you created above.

config: This contains configuration files for the app.
models: This houses our entity (Todo data structure).
repositories: A repository adds an abstraction layer on the data access. You can read more about why it's important to have this layer here
routes: A route is an entry to your application. This folder contains a file that defines what should happen when a user accesses a particular route.

├── config
├── models
├── respositories
└── routes

What you will need to install

You will need Node
You will need to install MongoDB

Application Packages

This app depends on a couple of packages and will use npm to install them. Navigate to the project directory you just created and create a package.json file with the content below.

{
    "name": "node-todo",
    "version": "1.0.0",
    "description": "Simple todo application.",
    "main": "server.js",
    "author": "Samuel James",
    "scripts": {
        "build": "webpack",
        "start": "node app.js"
    },
    "dependencies": {
        "cookie-parser": "~1.4.4",
        "cors": "^2.8.5",
        "debug": "~2.6.9",
        "express": "~4.16.1",
        "http-errors": "~1.6.3",
        "jade": "~1.11.0",
        "mongoose": "^5.7.3",
        "morgan": "~1.9.1"
    }
}

Run npm install to install the dependencies. Let's go ahead and define config parameters our app needs.

Config file

We'll define database connection URL and the port the app will listen on in config/Config.js file as follows:

//config/Config.js

module.exports = {
  DB: process.env.MONGO_URL ? process.env.MONGO_URL : 'mongodb://localhost:27017/todos',
  APP_PORT: process.env.APP_PORT ? process.env.APP_PORT : 80,
};

In the config/Config.js, we set DB to environment variable MONGO_URL if defined, otherwise defaults to mongodb://localhost:27017/todos. We also did the same with APP_PORT.

Todo Model

Model is an object representation of data in the database. So, Let's create a file models/Todo.js with the content:

//models/Todo.js

const mongoose = require('mongoose');

const { Schema } = mongoose;

// Define schema for todo items
const todoSchema = new Schema({
  name: {
    type: String,
  },
  done: {
    type: Boolean,
  },
});

const Todo = mongoose.model('Todo', todoSchema);

module.exports = Todo;

If you notice, we use mongoose for schema definition, right? Mongoose is an official MongoDB library built for manipulating MongoDB databases in Node. In the structure, I have defined name and done.

name: This is the name of the todo item. We define it as string. For example, "I'm going for swimming by 3pm"
done: Todo item status which is boolean. If the todo item is still pending, its value will be false.

Now, we'll create a todo repository.

Repositories

I like to see repository as a strategy for abstracting data access. I became a big fan when this pattern saved me from heavy refactoring when I switched to a new datastore in one of my projects. It helps you decouple your project and reduces duplication in your code. Here is an interesting article I recommend you read on this pattern.

That said, create a file repositories/TodoRepository.js as:

//repositories/TodoRepository

const Todo = require('../models/Todo');

class TodoRepository {

  constructor(model) {
    this.model = model;
  }

  // create a new todo
  create(name) {
    const newTodo = { name, done: false };
    const todo = new this.model(newTodo);

    return todo.save();
  }

  // return all todos

  findAll() {
    return this.model.find();
  }

  //find todo by the id
  findById(id) {
    return this.model.findById(id);
  }

    // delete todo
  deleteById(id) {
    return this.model.findByIdAndDelete(id);
  }

  //update todo
  updateById(id, object) {
    const query = { _id: id };
    return this.model.findOneAndUpdate(query, { $set: { name: object.name, done: object.done } });
  }
}

module.exports = new TodoRepository(Todo);

With TodoRepository.js defined, let's go create todo routes.

Routes

Every web application has at least an entry point. A route in web apps is more like saying: "Hey Jackson when I ask ya for this, give me that". Same goes for our app, we'll define what URL users need to access to get certain results or trigger certain actions.

In this case, we want users to be able to perform create, read, update and delete (CRUD) operations on todo items.

Now, that you know what "routes" does, create routes/Routes.js file and the code below:

const express = require('express');

const app = express.Router();
const repository = require('../respositories/TodoRepository');

// get all todo items in the db
app.get('/', (req, res) => {
  repository.findAll().then((todos) => {
    res.json(todos);
  }).catch((error) => console.log(error));
});

// add a todo item
app.post('/', (req, res) => {
  const { name } = req.body;
  repository.create(name).then((todo) => {
    res.json(todo);
  }).catch((error) => console.log(error));
});

// delete a todo item
app.delete('/:id', (req, res) => {
  const { id } = req.params;
  repository.deleteById(id).then((ok) => {
    console.log(ok);
    console.log(`Deleted record with id: ${id}`);
    res.status(200).json([]);
  }).catch((error) => console.log(error));
});

// update a todo item
app.put('/:id', (req, res) => {
  const { id } = req.params;
  const todo = { name: req.body.name, done: req.body.done };
  repository.updateById(id, todo)
    .then(res.status(200).json([]))
    .catch((error) => console.log(error));
});
module.exports = app;

First, you would want users to get a list of all to-do items existing in the database, hence we defined a route (/all) that accepts a get request and returns a JSON object of todo items if successful.

Our users like to get items as well as store new items, we added a route to create new to-do items. It accepts a post request.
When Mr. A makes a post request to route (/add), a new to-do item is created in the database.

Once a todo-item is completed, we also want users to be able to mark it as done. To do this, one must know what item a user intends to mark as done in the first place. So, we defined an 'update route' with a route parameter which is the ID of the item to update.

Server File

Having defined all routes that our application needs, it is time to create an entry file which is the main file to run our project.

At the project root folder, create a app.js file and update its content with this:

//app.js

const createError = require('http-errors');
const express = require('express');
const path = require('path');
const mongoose = require('mongoose');
const cookieParser = require('cookie-parser');
const logger = require('morgan');
const cors = require('cors')

const config = require('./config/Config');

const routes = require('./routes/Routes');

const app = express();

mongoose.connect(config.DB, {
  useNewUrlParser: true,
  useUnifiedTopology: true,
});

app.use(cors());  //enable cors

app.use(logger('dev'));
app.use(express.json());
app.use(express.urlencoded({ extended: false }));
app.use(cookieParser());
app.use(express.static(path.join(__dirname, 'public')));

app.use('/todos', routes);

// catch 404 and forward to error handler
app.use((req, res, next) => {
  next(createError(404));
});

// error handler
app.use((err, req, res) => {
  // set locals, only providing error in development
  res.locals.message = err.message;
  res.locals.error = req.app.get('env') === 'development' ? err : {};

  // render the error page
  res.status(err.status || 500);
  res.render('error');
});

app.listen(config.APP_PORT); // Listen on port defined in environment


module.exports = app;

We required:

Express Js
morgan - a logging middleware for Node
path - a package for working with file and directory
mongoose - a package for working with MongoDB
body-parser - a body parsing middleware

We set our app to listen on the port set in config/Config.js. We also defined it to use routes defined in routes/Routes.js and prefix with todos.

Finally, your directory structure should look like this:

├── app.js
├── config
│   └── Config.js
├── models
│   └── Todo.js
├── package-lock.json
├── package.json
├── respositories
│   └── TodoRepository.js
└── routes
    └── Routes.js

To start the application, navigate to project root and run:

npm start

Let's create a new todo item

$ curl -H "Content-Type: application/json" -X POST -d '{"name":"Going Shopping"}' http://localhost/todos

{"__v":0,"name":"Going Shopping","done":false,"_id":"5a6365a39a2e56bc54000003"}

Get all todo items

$ curl  http://localhost/todos

[{"_id":"5a6365a39a2e56bc54000003","name":"Doing Laundry","done":false,"__v":0},{"_id":"5a6366039a2e56bc54000004","name":"Going Shopping","done":false,"__v":0}]

Done building? Check out Part 2.

Get the source code here

DEV Community: Samuel James

Migrating Serverless Aurora Schema Using a Custom Resource

Custom Resources

Creating the Migration Handler

Creating the Custom Resource

Creating Serverless Aurora Instance

Conclusion

References

What Is Serverless Architecture and When Should You Use It?

What Is Serverless Architecture?

Serverless and Platform as a Service (PaaS): What's the Difference?

Composition

Scaling

Lifetime Availability

Why Use Serverless Architecture?

Serverless Architecture Use Cases

High-Traffic Websites

Multimedia Processing Applications

Mobile Backends

Internet of Things (IoT)

Big Data Applications

When Going Serverless Might Not Make Sense

Conclusion

Building an Image Classifier With Pytorch

Pytorch and Why

Google Colab

Data preprocessing

Splitting Datasets

Model Selection

Input transformation

Optimizer

Fitting the model

Testing the model

Making Predictions

Conclusion

References

How are poor APIs designed and what processes birth them

Learn Serverless on GCP by Building a Simple Calendar App

Creating a cloud function

Deploying to Google Cloud Platform

Granting your app permissions to calendar API

References

Build a Todo App with Node.Js, ExpressJs, MongoDB and VueJs - Part 3

What is docker?

Installing docker and docker-compose tool

Creating Dockerfile for backend Service

Vue-native Infinite scroll

What we will use

Install Vue-native-cli

Start a Vue-native project

Making a 'UserList' Component

What the heck is happening in UserList.vue?

Wrapping it off

Previewing the app

Laying the ground for your career as a software developer

Have a career plan

Make Connections.

Let your resume tells your achievements and not your job description

Keep track of your achievements at your current job

Take time to craft your application

Create a personal blog

Conclusion

Why Docker? Creating a multi-container application with docker

Table of Contents

Docker and why?

Maintaining consistent environments is hard

Installing application dependencies on local machine takes time

Docker Image

Docker compose

Building a multi-container application with docker-compose

Create a Dockerfile for PHP 7.1

Define services in Docker-compose file

Create a DockerIgnore file

Build and run project with Compose

Further Readings

Build a Todo App with Node.Js, ExpressJs, MongoDB and VueJs – Part 2

Project Directories

Vue CLI

Project Dependencies

Creating the Vue Components