DEV Community: Andre

Scheduling dbt Core with Github Actions

Andre — Wed, 08 Dec 2021 14:54:55 +0000

Originally published: andredevries.dev

I have been using dbt more and more as my main tool for data transformation and data modelling. dbt Labs (the company behind dbt) offers two types of products: dbt Cloud and dbt Core. The latter is an open-source CLI tool, which gives you the main components of a dbt project. The cloud offering (dbt Cloud) includes a lot more functionality, including an easy way to schedule your dbt runs. However, if you don't want to use dbt Cloud or want a bit more flexibility for configuring your dbt runs then you could also use Github Actions as a way of deploying and running your models. In fact, the analytics pipeline of this website uses this particular setup with a daily Github Action job. There are a lot more options for orchestrating dbt Core, including using Apache Airflow but I found Github Actions to be an easy and lightweight solution.

What are Github Actions?

dbt is built for git and it is the perfect tool to integrate with your continuous integration and continuous delivery (CI/CD) pipeline. Github Actions make it easy to automate your software workflows and allows you to automate your build, test, and deployment pipeline. Typically Github provides the virtual machines for you to run your workflows, making it super easy to get up and running. They also have a generous free tier. Actions are typically configured to run on particular triggers such as forking a repo, commenting on an issue, pushing to a branch etc. You can also configure actions to run on a schedule via CRON. That is the option we are going to look at in this blog post.

How to configure Github Actions for dbt

Github Actions workflows are defined by a YAML file, which is checked into your repository in a .github/workflows folder. It will run when it is triggered by an event in your repository, or it can be triggered manually, or at a defined schedule (CRON). You can configure Github Actions for any of your repositories on Github. There are a few steps involved for each workflow you want to configure. Let's take a look at configuring one for your dbt project:

Create .github/workflows/ directory in the root of your dbt project to store your workflow YAML files
In this folder create a file called 'schedule_dbt_job.yml'
Copy/paste the YAML below



   name: schedule_dbt_job

   on:
   schedule:
       # run at 7AM every single day
       # https://crontab.guru <-- for generating CRON expression
       - cron: "0 7 * * *"
   push:
       branches:
       # run on push to development branch
       - development
   env:
   DBT_PROFILES_DIR: ./

   DBT_SNOWFLAKE_USERNAME: ${{ secrets.DBT_SNOWFLAKE_USERNAME }}
   DBT_SNOWFLAKE_PW: ${{ secrets.DBT_SNOWFLAKE_PW }}
   DBT_SNOWFLAKE_ROLE: ${{ secrets.DBT_SNOWFLAKE_ROLE }}

   jobs:
   schedule_dbt_job:
       name: schedule_dbt_job
       runs-on: ubuntu-latest

       steps:
       - name: Check out
           uses: actions/checkout@master

       - uses: actions/setup-python@v1
           with:
           python-version: "3.7.x"

       - name: Install dependencies
           run: |
           pip install dbt
           dbt deps

       # dbt related commands here - run use --target prod/dev to run for specific environments
       - name: Run dbt models
           run: dbt run

       - name: Test dbt models
           run: dbt test

Create a profiles.yml file in the root of your dbt project if it does not exist yet. In this document you have to configure your connection details for setting up a connection to your datawarehouse. In this YAML you will see that I am using Snowflake, but this should also work for any of other datawarehouses that are supported by dbt (Google Big Query & Amazon Redshift).



   default:
     outputs:
       dev:
       type: snowflake
       threads: 1
       account: "{{ env_var('DBT_SNOWFLAKE_ACCOUNT') }}"
       user: "{{ env_var('DBT_SNOWFLAKE_USERNAME') }}"
       role: "{{ env_var('DBT_SNOWFLAKE_ROLE') }}"
       password: "{{ env_var('DBT_SNOWFLAKE_PW') }}"
       database: "{{ env_var('DBT_SNOWFLAKE_DATABASE') }}"
       warehouse: "{{ env_var('DBT_SNOWFLAKE_WAREHOUSE') }}"
       schema: "{{ env_var('DBT_SNOWFLAKE_SCHEMA') }}"
       client_session_keep_alive: False
       query_tag: github_action_query

Before we can schedule this dbt project we need to configure the environment variables in your Github repo. Whenever your Github Action runs it will use these variables at run time. You can configure these secrets via Settings > Secrets > New Repository Secret. You need to add all the variables that are specified in the YAML above. For example, the Snowflake Account variable would be 'DBT_SNOWFLAKE_ACCOUNT'.

Wrap-up

You should now be able to push all of your changes to your Github repository. If you navigate to Github you will see an 'Actions' tab. In here you will be able to see all of your past runs. Whenever I develop and test my dbt setup in Github I also use the push to branch event in my workflow YAML - rather than waiting till it is the configured CRON time!

How to easily create a Postgres database in Docker

Andre — Thu, 28 Jan 2021 09:33:03 +0000

Have you ever had the problem where a tool or a piece of software works fine on your machine, but the moment you install it on someone else's you get all kinds of issues? Well, I have, and particularly for this reason Docker was invented! In this blog post we will take a quick look at what Docker is and how easy it is to run a database in a Docker container. This container will work on any machine. I promise. Along the way you also learn some Docker specific lingo.

What is Docker?

According to the official docs, Docker is and "open platform for developing, shipping, and running applications. Docker enables you to separate your applications from your infrastructure so you can deliver software quickly. With Docker, you can manage your infrastructure in the same ways you manage your applications."

For fully understanding Docker we also need to talk about the difference between Docker and a Virtual Machine (VM). The latter often runs in cloud environments like AWS and Azure. Whenever you create a VM you are sharing the hardware with others and other VMs. What these cloud environments are doing is 'virtualise' the hardware. Docker doesn't do that and does it differently. In a VM you can have multiple Operating Systems running on the same hardware, whilst with Docker you virtualise the Operating System. Therefore, the big difference between VMs and Docker containers is that the former can have multiple (Guest) Operating Systems on the same hardware, through for example VMWare (which is called a Hypervisor). Whilst when you install Docker, you are going to use the Docker Engine to create isolated entities on the OS. These entities are called containers. Docker therefore allows you to automate the deployment of applications in these containers.

Let's install Docker

Enough theory, let's jump into installing Docker and firing up a Postgres database. In order to use Docker you first need to install it. You can either install Docker on a Desktop machine (both Windows and Mac), or on a server (Linux based installations). For this tutorial we're going to install Docker on a Mac. Windows installation instructions can be found here. For a Mac (and I think also for Windows) the installation is fairly straightforward. You download the app and drag it to Applications. Then you double click the Docker.app and it should start. You can check if it's working when there is a Docker icon (the whale or ship like image with containers) on the top right next to your other small icons. If this is the case you can quickly follow the 'Hello World' example to get up and running.

How to create a Postgres database

Hands down the easiest way of running a clean Postgres database is by running this command in a terminal window (after Docker has been installed):

docker run --name postgres-db -e POSTGRES_PASSWORD=docker -p 5432:5432 -d postgres

But what does it do?

Last section of the command grabs the latest 'postgres' Docker image from the Docker Hub
-d means that you enable Docker to run the container in the background
-p plus the port numbers means you map the containers port 5432 to the external port 5432 - this allows you to connect to it from the outside
POSTGRES_PASSWORD sets the password to docker. This is the password that gives you access to your database
the —name property gives your container a name and means you can easily find it back

Now you can connect to this brand new Postgres database in any tool that allows you to communicate with databases. I tend to use RazorSQL or DBeaver. You need to use the following connection details to actually connect to the DB:

Host: localhost
Port: 5432
User: postgres
Password: docker

Once connected you can do anything you want with the database (Create tables, load data etc). But as you can see the database is completely empty. However, the real power of Docker is when you want to easily provision a database that has already content in it. This can be a simple or a complex database structure and schema. The choice is all yours. What this also means is that you can easily spin up such a container (and shut it down). Let's see how below:

Create a Dockerfile and Docker Image

In order to create a pre-populated database we need to create a Dockerfile. This is a bit of a strange file. At least the first time I saw one. It doesn't have a file extension. It's basically a text document that is being used by Docker and describes what it needs to do. Basically a set of instructions. In our example we want to do the following:

Pull down the latest Postgres image from the Docker Hub
Set the environment variable for password to 'docker'
Create a database, let's call it 'world'
Use a sql dump file to create the table schema and populate it with data

Above I described what I want in this file. Now let's create it. Create an new file and call it 'Dockerfile'. Use a text editor like VS Code to open it and add the following:

FROM postgres
ENV POSTGRES_PASSWORD docker
ENV POSTGRES_DB world
COPY world.sql /docker-entrypoint-initdb.d/

The last line points at a SQL dump file. You can find the one I'm using here. Download it and put the .sql file in the same folder as the Dockerfile. Next step is to create our image by typing this command in the terminal:

docker build -t my-postgres-db ./

The above line tells Docker to build an image from the Dockerfile and give it a name of 'my-postgres-db'. In order to see your images you can run

docker images -a

Great, now we got our own image called 'my-postgres-db'. We can run it as a container by doing the following:

docker run -d --name my-postgresdb-container -p 5432:5432 my-postgres-db

You can now connect to this database by using the login details specified in the Dockerfile.

In case you want to remove images you can run this command:

docker image rm 'nameOfTheImage'

How to schedule an AWS Lambda function

Andre — Tue, 01 Sep 2020 10:22:25 +0000

Efficiently and easily automating tasks and pieces of code is super important for those that want to stay productive. I'm a huge fan of scheduling my workloads so I don't need to look after tedious and repetitive tasks. Thanks to Amazon EventBridge (a.k.a. CloudWatch events) we can easily create rules for our Lambda functions and schedule them whenever we want. In this blog post we will take a look at how to do that!

What is AWS Lambda?

AWS Lambda is probably the oldest and most popular serverless service out there. It allows you to focus on your code rather than having to take care of provisioning and maintaining (virtual) machines. It removes the need for such traditional compute services and therefore also reducing the complexity and operation cost. Lambda functions have their limitations but are great for small pieces of isolated tasks. They are great for serverless websites, real time data transformation, web authentication, chatbots, IoT workloads and a whole lot more. One of the things I like to do with it is schedule my tasks.

How to schedule a Lambda function?

The word 'serverless' already says it, there is no server! Well, actually there is but you don't have to manage it. What this means is that you are only paying for the times that you actually invoke/use your Lambda function. That is revolutionary in comparison with the traditional computing services where you pay constantly, even when you're not using your machine! This makes scheduling even more attractive. Let's see how to do that. We're going to do the following:

Create a Lambda Function
Setup a rule in Amazon EventBridge
See our function being executed every N seconds (minutes or hours etc)

Almost sounds too simple right? It definitely is super easy to setup!

1. Create Lambda Function

The quickest way to create a simple Lambda function is to do this through the AWS Console. When you have more complex applications I would move to the Serverless Framework. First rule of using AWS, check which region you are in! I'm using London but you can choose any in which Lambda is supported.

Next step is to create a function and give it a name. I called mine 'ScheduleLambdaFunction', but you can call this anything (1). After this you need to select a runtime (2). I'm most familiar with JavaScript so I'm using NodeJS but you can select any of the runtimes (Python, Java, Go etc). Last thing you need to do is create or select an execution role that has the correct permissions to execute the Lambda function (3). I'm going to create a new one.

When your function has been created you can write your code in the integrated code editor. For this tutorial it's not very important what the code does so I'm just going to log the current date and time. If you're following along you can copy the code below.



exports.handler = async (event) => {
const response = {
        statusCode: 200,
        body: JSON.stringify(`This fuction ran at ${new Date(Date.now())}`),
};
        console.log(response)
        return response;
};

Make sure you hit save when you've edited the code!

2. Setup a rule in Amazon EventBridge

Once the function has been created and you've modified the code you need to setup a rule in EventBridge. Go back to the console and navigate to 'Amazon EventBridge'. You can also do the exact same process of scheduling your Lambda function in CloudWatch 'events' but the easiest is to do this in EventBridge. Under the hood these two services use exactly the same API. The main difference is that with EventBridge you can also integrate third party SaaS application such a ZenDesk, DataDog and Shopify.

The setup of these rules is fairly straightforward. You need to create a rule (1), define a pattern and select an event bus (2) assign it a target (our Lambda function) (3). You can configure the schedule to run either by using a human readable format or using CRON. CRON gives you a lot more flexibility and there is no need to learn how CRON expressions are actually written. Head over to Crontab.guru to configure your own schedule expressions and copy paste them into the AWS console. For this tutorial I choose to run the event every 1 minute.

You can optionally give your schedule a description and add some tags. Hit 'Create' after and your scheduled Lambda function has been created! Pretty easy right?

3. Check the logs to see if function executed successfully

Now all the hard work has been done and you can head over to the CloudWatch logs to see your function results. You should see that the function gets executed every minute.

In conclusion

In this blog post we created and scheduled a very simple Lambda function. It actually only logs out the current date and time. Not super exciting. But you can do so much more! If you head back into your Lambda function console you can see that there are lots of triggers and destinations. You could for example do some web scraping to extract information from a website and then load and save the results in an S3 bucket. Another use case would be to create a Twitter bot that tweets something at fixed times. There are so many options and I'm really keen to understand what you would do with scheduled Lambda. Let me know in the comments below or hit me up on Twitter!

How I became an AWS Certified Developer - how to pass the associate exam!

Andre — Sun, 30 Aug 2020 19:54:02 +0000

Originally published at andredevries.dev

A few months ago I passed the AWS Certified Developer Associate Exam. This exam tests your knowledge of the core AWS services that you need to use for developing, deploying and debugging cloud based applications. AWS Certifications are currently one of the most sought-after credentials that you can have in the industry and I thought it would be helpful to share my experience and tips for this exam. This should therefore be a helpful blog post for those that want to start out in the tech sector or have an interest in becoming a developer who uses Amazon Web Services. I found the exam itself quite hard but it was really rewarding and a learned a ton!

Introduction

Amazon Web Services (AWS) is vast and there is a lot to learn. It currently comprises more than 200 services. I was pretty overwhelmed when I started delving into it. The first exam that I tried was the Cloud Practitioner. I passed that one earlier this year. To me, that exam felt kind of an entry exam into AWS and the cloud. There were lots of high level questions about a myriad of services, ranging from virtualization to machine learning and databases. This made it really hard to know beforehand if I studied enough. That was different for the Certified Developer Associate exam - the focus point of this blog post.

Exam details

The format of the exam is multiple-choice. There are no open questions and you don't have to write an essay. Typically the exam is held in a test centre but because we are living in a global pandemic AWS has moved all their exams to an online proctor environment. I'm still on the fence if I like that setup. A proctor is someone that constantly pays attention to you while you do the exam. For the full 130 minutes! This means that with even the slightest of movements of your hands or face the screen will be paused and you will be asked why you are moving. I literally got stopped while I was grabbing my glass of water that was standing next to the laptop. I found this really frustrating and it dropped my focus, especially the first time when I did the Cloud Practitioner exam. The second time around, for the Developer Associate, I just accepted it (and didn't drink any water).

How I studied for the exam

Before we jump into the specifics of the exam I want to explain how I prepared myself for this exam. I think that everyone has a different way of learning so please don't take this as gospel. I think the time between registering for the exam and actually taking it was about 3,5 weeks. Even though I've used AWS quite a lot in the last few months I had to learn a lot about certain services I had never used. I mainly studied through the Udemy course and the practice exams that I mention below. Especially the practice exams give you a good sense of what you know and what you need to focus on. For me this was mainly learning more about DynamoDB and the CI/CD Pipeline. Whenever I study for a particular topic I try to immerse myself into the material. I do that via books, podcasts and YouTube videos. I have listed out a whole bunch of resources at the end of this blog post but the main reason I read books and listen to podcasts is that you get some 'real' opinions about the AWS services you are learning about. The Udemy course from A Cloud Guru is great, but it's a very structured and un-opinionated approach to learning about the cloud. While if you, for example, listen to a podcast about Serverless (e.g. the Real-World Serverless podcast by The Burning Monk) you get more in-depth explanations of the various study topics.

Prerequisites

Let's start with the requirements for this exam. The Developer Associate exam is one of three associate level exams. The other two are the Solutions Architect and the SysOps Administrator. Find more information about those exams here. AWS states the you need a certain amount of experience and in some cases even a few years of hands-on experience to pass them. However, there are no real prerequisites to register for the exam. Anyone can therefore sign up, even if you don't hold any AWS certifications.

The breakdown of the exam questions are as follows:

Deployment: 22%
Security: 26%
Development: 30%
Refactoring: 10%
Monitoring: 12%

AWS recommends you have the following knowledge and experience for the Developer Associate exam:

In-depth knowledge of at least one high-level programming language
Understanding of core AWS services, uses, and basic AWS architecture best practices
Proficiency in developing, deploying, and debugging cloud-based applications using AWS
Ability to use the AWS service APIs, AWS CLI, and SDKs to write applications
Ability to identify key features of AWS services
Understanding of the AWS shared responsibility model
Understanding of application lifecycle management
Ability to use a CI/CD pipeline to deploy applications on AWS
Ability to use or interact with AWS services
Ability to apply a basic understanding of cloud-native applications to write code
Ability to write code using AWS security best practices (e.g., not using secret and access keys in the code, instead using IAM roles)
Ability to author, maintain, and debug code modules on AWS
Proficiency writing code for serverless applications
Understanding of the use of containers in the development process

This is a big list and you are tested on each item. But some are more important then others. I'll therefore break down the most essential ones below:

In-depth knowledge of at least one high-level programming language

This is the first one in the list and might scare a few people. Do I really need to know or even learn a complete programming language to be able to pass this exam? The answer is a no, but it might help. What I mean with that is that you don't actually have to write code, but you have to understand it. It also helps with your further career in AWS. Even though you can do a lot in the AWS Console, many Cloud Developers are using the CLI and CloudFormation to deploy their 'infrastructure as code'. Because the questions are multiple choice you can use the art of elimination to get to the right answers for these types of questions if you're not too certain. The ones I got that involved code where related to caching and which code snippet demonstrated caching most effectively.

Understanding of core AWS services, uses, and basic AWS architecture best practices

Even though this exam focuses heavily on the developer side of AWS, I would still recommend that it is important you have a solid understanding of the core services such as EC2, S3, IAM and RDS. Many questions have these services listed in possible answers so knowing what you can (and can't do) with them is very important. I didn't get any questions around actually designing or architecting solutions in a VPC. This is mainly because lots of the developer services are all moving towards a 'serverless' setup. With serverless a lot of the detailed architectures are already taken care of for you. Another point that always comes back in AWS exams is best practices. These often relate to security best practices. A rule of thumb when you work with AWS is that you are responsible of securing what's in the cloud, while AWS secures the cloud itself.

Ability to use a CI/CD pipeline to deploy applications on AWS

This one is super important and I got a lot of questions on. CI/CD stands for Continuous Integration and Continuous Deployment. These are paradigms that developers use when they (automatically) deploy and build their code. Understanding what 'blue and green' deployments are is important, but you're not being tested if you know the definition of these terms. Often the questions are written as little case studies where you have to select which deployment strategy is the most suitable. 'Suitable' is then also defined in the question itself. I would recommend using or playing around with AWS Codepipeline on your own to understand the moving parts. This AWS service is kind of an umbrella service that comprises 'Codecommit' (AWS's Github), Codebuild & CodeDeploy. Another tool that you need to understand in this category is X-Ray. A tool that can help you with analyse, debug and monitor your applications.

Proficiency writing code for serverless applications

You can take this requirement a bit broader and think about serverless applications as a whole. Because in the exam there are quite a few questions related to serverless computing. You need to be able to articulate the differences between a 'traditional deployment' and a serverless deployment. In the former, the user has to provision their own instances, maintain and update the operating system, install applications and handle and configure autoscaling themselves. All of this is not needed when you move to a serverless framework. You only have to take care of writing the code of your application and monitor it. Everything else, like autoscaling, is taken care of by AWS. Services that you need know in this section are: AWS Lambda, API Gateway, DynamoDB, Step Functions, Athena and Kinesis. Of this list I would really recommend studying Lambda, Gateway and DynamoDB.

Understanding of the use of containers in the development process

This is something I didn't focus on at all while studying. I didn't get many questions about it either so I wouldn't worry to much if you've never used Docker and Kubernetes. However, I would read up a bit on the AWS services that help you with deploying containers on AWS (AWS ECS) because they might be mentioned as potential answers in other non container related questions.

Resources I used to study:

There are a whole bunch of resources that I used to study. The most important thing to stress when you are studying for AWS exams is to get hands-on experience with the services. This hands-on experience gives you a good grips on the console. However, for the Developer Associate exam I would also focus on learning CloudFormation and how to use the AWS Serverless Application Model (SAM). Almost all the things you can do in the console you can also do by using the various APIs that AWS offers (e.g. AWS CLI). The majority of resources that I list below will give you this firsthand experience:

A Cloud Guru Developer Associate Course (Udemy): a well-known resource for online cloud training. A Cloud Guru has a whole bunch of courses and this one is quite good and gives you a solid understanding of the developer side of AWS.
Exam Readiness AWS Certified Developer Associate (YouTube): recording of a presentation that walks you through the various elements of the exam. Contains practice questions and lots of tips & tricks.
TutorialsDojo (Website): this is where I bought access to practices exams. This was my primary method of testing what elements of the exam I had to pay more attention to.
AWS Simplified (YouTube): very informative YouTube channel that has lots of bite-size videos about various AWS services.
DynamoDB Guide (Website): built by Alex DeBrie. He's an AWS Hero and the author of The DynamoDB Book
Off-by-one (newsletter): I signed up for two AWS related newsletters but also had a look through the backlog of this newsletter by Jeremy Daly. It's a great way to learn more about the vast and growing ecosystem of serverless.
Last Week in AWS (newsletter): not specifically focused on the developer side of AWS but a good all-round weekly newsletter that keep you informed about what's new in AWS.
Serverless Chats (podcast): a very informative but sometimes highly technical podcast from Jeremy Daly.
Real World Serverless (podcast): a podcast by Yan, who is an AWS Hero who specialises in everything related to serverless. Contains lots of very good examples of serverless implementations from various companies.

How to host a React application on AWS S3

Andre — Tue, 25 Aug 2020 16:49:52 +0000

There are many ways to host your website on Amazon Web Services (AWS). One of the easiest is to use an S3 bucket to host your static website. Setup and configuration is fairly straightforward for this option. Take a look at the video and see how easy it is. In the video I explain how you can either manually build out your React application and then upload it to a bucket, or use the AWS CLI to automate the deployment.

Prerequisites

AWS Account
AWS CLI installed on your machine
IAM User / role
Local credentials of the AWS User
NodeJS & npm installed

Steps to Upload to S3

As shown in the video:

Scaffold a React application by running:

   npx create-react app nameofApp

Create an S3 bucket
Change Properties to allow static website hosting (index.html for the Index document.)
Change Permissions of Bucket Policy (replace NameOFBucket with your bucket name from 2)

   {
       "Version": "2012-10-17",
       "Statement": [
           {
               "Sid": "AllowPublicReadAccess",
               "Effect": "Allow",
               "Principal": "*",
               "Action": "s3:GetObject",
               "Resource": "arn:aws:s3:::NameOFBucket/*"
           }
       ]
   }

Build out the React application and copy contents of build folder over to S3

 yarn build

Setup S3 Sync - syncs directories and S3 prefixes. Recursively copies new and updated files from the source directory to the destination. Modify the package.json file and add a 'deploy' script that syncs the content of the build folder with the bucket:

  "deploy": "aws s3 sync build/ s3://nameofbucket"

Each time you want to deploy a new version of your app run:

  yarn build && yarn deploy

A first look at Deno

Andre — Wed, 06 May 2020 17:52:43 +0000

The release of Deno v1 is upon us. Today was the first day I played with it. Deno is a new runtime for JavaScript and Typescript and created by the founder of NodeJS. In the video I share my initial experiences.

How to use a 'do... while' loop for API pagination

Andre — Mon, 24 Feb 2020 18:44:11 +0000

There are various ways of looping in JavaScript. We have the regular 'for' loop, 'for/in', 'for/of' and the regular 'while' loop. Each have their strengths, weaknesses and use cases. By using an example of looping over multiple pages of an API I want to take a look at another type of loop, the 'do... while' loop.

The Problem

In my day-to-day job I have to work a lot with APIs. One characteristics of APIs is that they often provide the user with multiple 'pages' of data. There are clear reasons why you want to split up your data into parts. One of them is the resources that it takes to serve up all the data in one API call. Multiple trips might be more efficient and quicker. However, very often we want to have access to all the data in one go. A solution for grabbing all of the data is to loop over all the pages and grab the pieces you are interested in.

An example of a REST API that uses pagination is the Star Wars API. One of my favourite APIs to illustrate said problem. You can find the official documentation here. Don't worry about rate limits or costs. This API is free to use. I use it in almost all my training sessions.

The Syntax

Before we're going to implement this recursion we're going to take a look at the syntax of the 'do...while' loop. According to MDN this type of statement "creates a loop that executes a specified statement until the test condition evaluates to false. The condition is evaluated after executing the statement, resulting in the specified statement executing at least once."

The important part of the definition is that this loop executes at least once. Especially with API calls this is useful because you want to check if there resource is available for which you are requesting the data. If there are no more pages then it doesn't continue. A regular 'while' loop on the other hand keeps executing a code block until a condition is met. The syntax for this loop is easier than the do...while loop. But let's take a look at how to create a do... while loop:

// Basic do while loop
// Logs a message to the console
// @andre347_

function doLoop() {
  // create an empty message
  let message = "";
  // we want to log a message 5 times
  let i = 5;
  // execute this code block..
  do {
    message += `The number decreased to ${i} \n`;
    // decrement i in each loop - so 5, 4, 3, 2, 1
    i--;
  } while (i > 0);
  // while i is more than 0 log something to the console
  console.log(message);
}

// make sure we call our function
console.time("Timer");
doLoop();
console.timeEnd("Timer");

If you run this with NodeJS (I used node v12 with the experimental flag for modules) you will see a message being logged five times. The message kept on being logged until it ran 5 times. We want to use similar logic for our API pagination. The key of the do... while loop is in the while condition. This code block controls how often a loop will run. Make sure you don't create infinite loops because that can crash your browser or node environment. These loops are the ones that never finish.

API Pagination

The Star Wars API contains all kinds of data related to the Star Wars universe. These resources are split into individual endpoints. One of them is '/people' - which contains information of all characters in the movies. When you inspect the first page of this endpoint you see a big object. The top section of this object is useful for the recursion.

{
  "count": 87,
  "next": "https://swapi.co/api/people/?page=2",
  "previous": null,
  "results": [
    {
      "name": "Luke Skywalker",
      "height": "172",
      "mass": "77",
      ...etc
    }
  ]
}

This object contains a 'next' property. Which is the next page of data we want to grab in our API call. The logic to stop the loop is to check if there is still a next page. If not, then we got all the data. See below implementation.

async function getPages() {
  // set some variables
  const baseUrl = `https://swapi.co/api/people/?format=json&page=`;
  let page = 1;
  // create empty array where we want to store the people objects for each loop
  let people = [];
  // create a lastResult array which is going to be used to check if there is a next page
  let lastResult = [];
  do {
    // try catch to catch any errors in the async api call
    try {
      // use node-fetch to make api call
      const resp = await fetch(`${baseUrl}${page}`);
      const data = await resp.json();
      lastResult = data;
      data.results.forEach(person => {
        // destructure the person object and add to array
        const { name, height, films } = person;
        people.push({ name, height, films });
      });
      // increment the page with 1 on each loop
      page++;
    } catch (err) {
      console.error(`Oeps, something is wrong ${err}`);
    }
    // keep running until there's no next page
  } while (lastResult.next !== null);
  // let's log out our new people array
  console.log(people);
}

console.time("Time my API call");
getPages();
console.timeEnd("Time my API call");

This should give you a nice array with all the characters (87) and their height plus the movies in which they appeared.

You can find all the code for this blog post in this Github repository

Orginally posted at andredevries.dev

How to create a user snippet in VS Code

Andre — Sun, 09 Feb 2020 13:15:59 +0000

Originally posted on andredevries.dev

I write all my blog posts in markdown in VS Code. That's because my personal website is built with Gatsby and this tool makes it extremely easy to create pages with it. What I don't like about writing markdown, especially for a Gatsby blog, is that I have to write frontmatter each time I start writing. Frontmatter is the first section of a blog post and I see it as the metadata about your blog. This can contain anything but generally you have the title of the post, the date and time of writing, a description and a category. The text editor that I use for writing blog posts is VS Code. This editor makes it very easy to create snippets. I have therefore written a quick snippet that scaffolds out the necessary frontmatter for my blogposts. Let's take a look at how I created such a snippet.

What are user snippets in VS Code?

User snippets are a quick way to scaffold out some code or a text. They can save you a lot of time, especially when you have to write the same thing over and over again. I use these a lot when I work in React - I can quickly create a functional component that is exported by using this snippet from the VS Code Marketplace. The only thing I have to type is 'rcfe' and it creates the snippet for me.

How can I create a user snippet?

There is some handy documentation which you can find here. I'm using a Mac, which means I have to go to 'Code > Preferences > User Snippets'. VS Code will then ask you if you want to create a 'global' snippet or a file specific one. The global options means that you can use the snippet in any filetype (.js, .md, .mdx etc). You can also limit the snippet to the current workspace you are in. But because I want to use this also for other folders in VS Code I generated one for markdown files.

The next step is to configure the snippet. You have to write it in JSON. The image below shows how I setup the snippet from the animated gif on the top of this page.

The first things you need to do is give your snippet a name. I called mine 'Markdown Blog Frontmatter'. The prefix is the actual shortcode that you have to write to trigger the creation of the snippet in a markdown file. You can use a prefix by hitting 'Ctrl + Space' on a Mac. This will bring up the IntelliSense window.

The most important part of the snippet goes into the body section. This is the actual text that is going to be placed in the document. You can use a few variables which make it easy to scaffold out some information. In my example I used the filename (without file extension) as the title of the blog. I did the same for the date of the blogpost. This one uses todays date. Another variable that I used is the '\$0' - which places the cursor at a particular location. In my example I placed two cursors. One in the frontmatter title section and the other one at the start of the blog post.

You can see a full list of variables that you can use in snippets here. With variables you can do something really cool and useful. You can for example reference environment variables or create popups for user input.

I'm really impressed with the possiblities and how easy it is to setup these snippets. They can be a huge time saver. While browing the VS Code marketplace I found a really handy extension. If you want to turn any text into a snippet you could use this extension.

How to deploy a React application on AWS Amplify

Andre — Tue, 31 Dec 2019 17:09:34 +0000

While studying for one of the AWS Exams I found a service called AWS Amplify. In this video I show you how easy it is to use this service to push a React application to a repository on Github and then have it automatically deploy on AWS.

Originally published at andredevries.dev

D3.js and Vue.js

Andre — Sun, 17 Mar 2019 20:42:24 +0000

In this blog, I will describe how you can integrate D3 into Vue.js. D3 is a popular JavaScript library for visualising data using web standards (HTML, CSS, JavaScript, and SVG). Vue.js is a rising star in the front-end and has lately gained a lot of popularity in the web development scene. It’s a front-end framework similar to React and Angular and allows you to build scalable user interfaces.

I’m learning D3 as I go, and I’ve used Vue.js a lot in the last few months, I thought it would be good to practice how I can combine the two. I’m still learning and improving my D3 skills so I think that if I revisit this post in a few months I’ll see mistakes or things I would change, so please highlight any issues or suggestions in the comments below.

I borrowed the chart that I’m recreating for this blog post from Elijah Meeks (image below), who wrote an excellent book about D3.js called ‘D3.js in Action’. If you want to learn more about this JavaScript library then this should be your first stop (so yeah, definitely buy this one).

In Chapter 9 he writes about integrating D3 with React, and I have seen a lot of good examples of this combination on GitHub and B.locks. But there aren’t that many resources around integrating D3 with Vue.js. I found a few other articles on Medium and one on B.locks but the best one so far was this repo from Shirley Wu, who is a freelance software engineer and data visualisation expert.

‘Circle Chart’ for displaying hierarchical data - taken from Chapter 6.3.1 from D3js in Action

How to get started with Vue

Let’s get started with scaffolding a Vue Project — this is similar to create-react-app for those that come from a React world. I’m using Vue CLI version 3. If you don’t have it installed then please run:

$ npm install -g @vue/cli

I know that using the CLI for just one chart component is a bit overkill but I suppose you’re going to integrate D3 in a bigger application for which you want to use all Vue functionality. The -g flag means you’re installing it globally on your machine so there’s no need to run this again the next time you use Vue. After installing Vue it’s time to create a new project. Run the following lines one by one in your terminal:

$ vue create d3-vue-example
$ cd d3-vue-example
$ npm run serve

With npm run serve you’re starting a development server with ‘hot reload’ enabled. This means that when you make changes in almost all of the files the changes are immediately displayed. Once you got this server running it’s time to install D3. You can do that like so:

$ npm i --save d3

If you open up the d3-vue-example folder in your favourite editor (I use VS Code) then you see a bunch of files and folders listed. For now it’s important that the package.json file is there. This is where all the packages appear that you install through npm. D3 should now also show up under dependencies in this package.json file.

Create a chart component

The most important files and folders for us are in the src folder. App.vue is the main entrance to your application. In this file, you want to import all the components that you create. By default you see only one component being imported here; the HelloWorld. This component file is located in the components subfolder. It’s best practice to put all your components in this folder.

Let’s also create a Chart.vuefile in this folder. Then go back into your App.vue file and duplicate line 9 and replace HelloWorld with the newly created Chart file. After this you have to add Chart to the components property of the JavaScript Object that is being exported in this file. The next step is to reference this component in the template section of theApp.vue file.

Component ‘PackChart’ is being imported in App.vue and used in the template

Okay, that might have confused you a bit if you’re new to front-end frameworks and working with npm. Head over to my Github page to find the source code if you want a full look.

Go to your localhost port 8080 (http://localhost:8080) and there you’ll be welcomed with the Vue default template. If you’re new to Vue.js then this new file extension .vue might look a bit foreign. Actually, this is the beauty of Vue — within this one file you create your own components and have all your HTML (template), JavaScript, and CSS together. Going over all the basics of Vue is too much for this blog so I recommend spending some time with this course on Udemy from Maximilian Schwarzmüller.

Adding D3 to Vue
I tend to import D3 in all the components that I create (including App.vue) but it’s probably best practice to not do this and just import it once, or only import the elements of the API that you need. An explanation of the module import of D3 can be found here. You can import D3 in each component by referencing it in the top of the script section of the vue file like so:

import * as d3 from "d3";

Another way would be to include the CDN link in the head section of the index.html file but it is best practice to use the node modules. Although with the CDN method it would then mean you can use it everywhere in your application.

Build out the Chart component

If you go back to your App.vue file then we’ll set up data props. Props are the data you want to send from your parent component, the App.vue file, to your child components, in this case Chart.vue. Let’s first create a data property in which we’ll push the data (I called it loadData). We’re going to use the tweets.json file from Meeks’ book — you can get the file here. Once downloaded, move the file into the public folder in the project folder.

Load the data in App.vue when the app has mounted

Vue.js has several ‘life-cycle’ hooks. These correspond with the different ‘states’ of your application. In the image above you see the ‘mounted’ property in the Vue instance. When the application has loaded it adds all the properties it can find in the ‘data’ object to its Reactivity system. This means that if the data changes, your application also updates (it becomes reactive). If you’re new to front-end frameworks, it might be a bit difficult in the beginning to wrap your head around the concept of ‘state’ and how elements are being removed and updated. But if you’re familiar with D3 then this might ring a bell. Think of it as the whole ‘Enter-Update-Exit’ pattern.

Add this to the Chart.vue file

Back to the mounted object. In D3 version 5, we have to use promises to load our data. This makes life a lot easier because previously you would have to use callbacks that often became a bit messy. What mounted does in this case is load the data from the JSON file and make it available when the DOM is being ‘mounted’. After this, you need to add the data to the prop we created in the Chart.vue component (see image on the top). You then bind this prop to the Chart component in the App.vue file like so:

<PackChart :data=loadData />

This means that whatever object is in ‘loadData’ is then being pushed into the child component that is the Chart.vue file (called the PackChart).

If you’re new to this Async/Await syntax of handling Promises then I would suggest watching this talk from Wes Bos.

Create the D3 chart

The first section of the script part of the Chart.vue file contains importing D3, return the data object (with a message that is displayed in the component, and the width and height of the SVG that contains the chart (1)). This width and height is then bound to the SVG in the template (2).

Width and Height of SVG specified in the data object of the Vue instance

In the ‘created’ life cycle hook, I am defining a scale function for the colours of the circle chart. Because we have a set list of nested bubbles (a discrete list) as input, we can use the scaleOrdinal scale. This scale then returns a discrete output of colours (the ones we define in the array). If you want to learn more about the different scales of D3, then I recommend heading over to this page.

The next step is to create a computed property in which we restructure the data so that we can use it as a hierarchy. D3 has several useful functions that can help you with making your data ready for charts that display hierarchies. One of them is the nest() function. What this does is turn a flat structure to a nested one (1 => 2). You can then define how to nest it (which property) and how deep to nest it. In our case, I use the ‘user’ as the top level property. This then means our new array contains four objects (one for each user).

From one tweet per Object to one Object per user (with tweets as children) with nest()

In this same computed property, I’m using the hierarchy module. This module takes a root (the new object called packableTweets — see image below) and returns a new layout.

Creates the hierarchical structure as per previous image

In order to actually draw something on the screen, we need to return some data that can be bound to the SVG in the template. For that, I created another computed property that takes in the previous one (packData()) and returns an array of JS Objects with the x & y coordinates and the radius of the circles. It then also uses the colourScale defined in the created hook (see image below).

Returns an array of objects (containing the circle data)

We can then loop over this array with the v-for directive and display the circles in the view with their corresponding x and y coordinates, their radius, and colour.

If you made it here then you’ve followed quite a lot of steps. If you got stuck along the way then I would recommend heading over to my GitHub and clone the repo and inspect the chart here. I’m open for feedback as I’m certain I’ve either overcomplicated things or overlooked elements in my code.

Originally published at andredevries.dev (January 2019)

How to create a Task Scheduler in NodeJS

Andre — Sat, 16 Mar 2019 10:35:37 +0000

Wes Bos posted a really useful video explaining how to scrape data from the web with NodeJS. In his second video he explained how to setup a schedule for this particular task. Something I'd never done before in Node so I thought this might come in useful in the future and therefore I should write a quick blog post about it.

Whereas in Wes his video he grabs data from his own social media pages, I'm going to create a small app that runs on a schedule and downloads a random image every day at 6PM. I know right, who doesn't want to have a random image popping up on his or her disk every day?!

A few things we need to install first:

    // create dir, go into it and install packages
    mkdir image-downloader && cd image-downloader &&
    npm i node-cron node-fetch esm

A quick break down of what you've just installed:

node-cron: this is the package for the task scheduler. It allows you to setup schedules that automatically perform something (often executes a function).
node-fetch: a way to use the fetch api. Which is a native browser API - but we don't have a browser when we use node. You can also use another package here. Axios is very popular one. It just allows you to download the content behind a url. Typically you use this for connecting to APIs or scraping the web.
esm: I had not used this one before but it's super useful. It allows you to write your code like you'd do in client side JavaScript such as in Vue or React. Which means you have access to things like import / exports. To enable this esm you have to install it and then add it to your run script. In my package.json file I added this line as the 'start' script:

    "scripts": {
        "start": "node -r esm index.js"
      },

You could then run this script by doing npm run start.

Create downloader

Now we got the necessary packages installed it's time to create the first file in which we'll just fetch one image: fetch.js

// fetch.js

import fetch from "node-fetch";
import fs from "fs";

// create a function that grabs a random image

const fetchingData = async () => {
  const res = await fetch("https://picsum.photos/200?random");
  const date = Date.now();
  const dest = fs.createWriteStream(`./image-${date}.png`);
  res.body.pipe(dest);
};

// export the function so it can be used in the index.js file

export default fetchingData;

In order to get a random picture each time you execute this script, I use Picsum. This website allows you to generate a random image with a fixed width and height. You can append those dimensions to the url. I also create a variable with the current date. This date will then be appended to the file name and prevents the files from being overwritten. Because we're working with promises here I'm using async/await.

If you want to test this file you can run it with node -r esm fetch.js

Setup a schedule

Next you want to create an index.js file. This will be the main entry file and this one contains the node-cron function:

import cron from "node-cron";

cron.schedule("* * * * *", () => {
  console.log(`this message logs every minute`);
});

This is a very small app which if you execute it will log a message to the console. This message will be repeated every minute. Cool, but not very useful. Let's add our image fetcher by importing it. The index.js file will then look like so:

import cron from "node-cron";
import fetchingData from "./fetch";

cron.schedule("* * * * *", () => {
  console.log(`one minute passed, image downloaded`);
  fetchingData();
});

However, this will run the image downloader every minute. We can change the cron job by changing the first parameter that we're adding into the schedule function. The five stars you see mean that the function will run every minute. You can modify this by following this (taken from here):

 # ┌────────────── second (optional)
 # │ ┌──────────── minute
 # │ │ ┌────────── hour
 # │ │ │ ┌──────── day of month
 # │ │ │ │ ┌────── month
 # │ │ │ │ │ ┌──── day of week
 # │ │ │ │ │ │
 # │ │ │ │ │ │
 # * * * * * *

At first, I didn't really understand what this meant. After a bit of Googling I found the following website that was really useful as a cheatsheet; the crontabguru

This means you can setup a schedule for literally any time. Maybe once a year? Or every Tuesday at 8am in January and July. There's really no limitation. I continued by setting up a schedule to make it download every day at 6PM by setting it to this: 0 18 * * *

The complete and final index.js file is then:

import cron from "node-cron";
import fetchingData from "./fetch";

cron.schedule("0 18 * * *", () => {
  console.log(`one minute passed, image downloaded`);
  fetchingData();
});

Want to take a look at the full app or clone it? Head over to my Github here!

First post at Dev.to so be gentle. Originally published at andredevries.dev