DEV Community: Mandeep Singh Gulati

How to navigate this seemingly complex world of tech as an absolute beginner?

Mandeep Singh Gulati — Sun, 25 Jul 2021 15:16:52 +0000

This post is aimed at helping those people who are thinking of entering the world of tech and start a career as a software developer, but are too overwhelmed by the information available to them and finding it hard where to start.

We will be taking a 10,000 feet view of the world of tech and various career paths you can explore as a beginner. And while doing so, we will answer some of the questions like

What are programming languages and which one to learn?
What's web development?
What's a web server?
What's cloud?
What's HTML, CSS and JavaScript?
What's Python?
What's front-end, back-end and full-stack?
What are frameworks?
What's Artificial Intelligence and Machine Learning?
What are databases?

If you feel overwhelmed already, don't worry. I have over-simplified things to help you get started. Once you dive deeper into any of these questions, there's a lot of learn but think of this article as a starting point. The intent of this post is to provide substantial information to help you get started without overwhelming you as a beginner. So, if you feel like something in this post is too complex and difficult to grasp, feel free to let me know. I would like to update this post based on feedback to make it as beginner friendly as possible.

So, first things first

What are programming languages and which one to learn?

Being a software developer is all about talking to machines in a language they understand. Different types of languages serve different purpose. For example,

Markup language like HTML

HTML allows you to define the structure of a web page. What does it mean? Let's take an example of Wikipedia. When you visit www.wikipedia.org, you can see a page like this:

If you see the structure of the page, there is a sidebar on the left which includes links like "Main page", "Contents", "Current events", etc. There is a top navigation bar as well with tabs like "Main page", "Talk", "Read", "View source", "View history" and a search bar.

Below that there is main content section which is further divided into blocks like "Welcome to Wikipedia", "From today's featured article" and "In the news".
All of this information about the structure of this web page is defined using a markup language called HTML which stands for Hyper Text Markup Language. To get a feel of it and play around, you can visit w3schools

That covers the introduction to markup language. Another type of language is

Style sheet language like CSS (Cascading style sheet)

Now we know that using HTML, we can structure a web page. But things like controlling the presentation of a web page, for example:

What should be the width of our sidebar element?
What should be the background color of our "In the news" and "From today's featured article" be?
What should be the color, font size of headings, paragraphs, links, etc.?

All of this information and a lot of other information is controlled via CSS. You can also create complex graphics and animations using CSS as well but that's very advanced level stuff. As beginners, we'll only need to use it for controlling the basic stuff. To get a feel of it and play around, you can visit w3schools

So far, we covered one markup language and one style sheet language. There are other markup languages other than HTML and there are other style sheet languages other than CSS, but for web development, HTML and CSS are the only markup and style sheet languages you'll ever need to learn.

Next, we will cover programming languages

Programming languages like (JavaScript, Python, C#, Go, Java, PHP, etc.)

These languages allow you to run logical computations. For example, when you try logging into to a site, for example Twitter, when you fill your email address and password on a webpage and click the "Log in" button, your browser sends a request to Twitter's server. On the server, there is a computer program which checks your credentials, looks up the credentials in Twitter's database, checks if the password you provided is correct or not and then returns success/failure response back to the browser.
Once you are logged in, your browser sends another request to fetch your tweets, notifications, etc and this calls another computer program on Twitter's servers which fetches this information from their database based on your user id and returns the same back to the browser.
All of this interaction happening between your browser and the server is written using programming languages. There is code written on the browser side as well as server side. Some of the code that executes on browser:

Sending of the request to server with user credentials when login button is clicked
Sending of the request to server to fetch user's tweets/notifications, etc
Sending a request to the server to like a tweet or post a comment
Tracking user behavior like scrolling, clicking, etc.

All of this code that executes on the browser is written in a programming language called JavaScript (Don't confuse it with another, completely different programming language called Java. There is no similarity between Java and JavaScript except their names). Browsers can only execute JavaScript code and not any other programming language code.
So, if you want to become a front-end developer, which means a software developer responsible for

Structuring the layout of a web-page (using HTML)
Defining the presentation of the web-page (using CSS)
And writing logic for interactions on a web page (using JavaScript)

Then the above three languages are something you'll need to learn. The sequence in which you choose to learn above languages is up to you. You may start with HTML then learn CSS and then dive into JavaScript. Or you can start directly with JavaScript core language and then learn HTML and CSS and then learn how to use JavaScript for browser based interactions. There is no right or wrong approach to learn these 3 languages but know that you will need to learn all 3 in order to become a front-end developer. To be employable as a front-end developer, you will need to expand your knowledge further by learning a framework like Vue, React, Angular, Svelte, etc. Will will cover these frameworks later in this article. But for now, you need to know that HTML, CSS and JavaScript are necessary skills for a front-end developer. Sorry for being repetitive.

So, we know that browsers can only execute JavaScript language. However, there is no such restriction when our code does not have to execute on a browser.

When we have to run some piece of code that does not need to run on a web browser, we have a lot of freedom to choose from the various programming languages available. For example, the operating system you are using like Windows, Linux or Mac is a collection of computer programs written in some programming language. The web browser itself is also a collection of various computer programs written in some programming languages. Every software you use is a collection of computer programs written in a programming language. Theoretically, anything you can do in one programming language can also be done in any other programming language. However, practically, some languages are better suited for certain tasks than others, due to various factors like:

ease of use
performance
tools and community built around that language

You might have already heard this or read this before but let me re-iterate that:

Learning one programming language makes it relatively easier to learn another programming language

So, if you know C#, learning other languages like Python, JavaScript, Java, Go, etc. becomes relatively easier.

But which one to choose first?

It depends on which career path you want to pursue? Of course if you are an absolute beginner, you most probably don't know which career path you want to choose. So, I'll share some of the popular career paths and how they influence which languages to learn. Before I do that, however, I would like to make a suggestion if you don't want to decide which career path to pursue at this point of your journey. If you're an absolute beginner who wants to learn programming, I would recommend starting with Python. In my opinion, it is the most beginner friendly language, has a huge community and can be used in various domains, especially in the field of data science. Having said that, if you want to choose the programming language to learn based on a career path, here are some of the popular career paths you can explore in tech.

Front-end engineer

We've already talked about this path earlier and we know that in order to become a front-end engineer one must learn HTML, CSS and JavaScript and then learn some of the popular frameworks like Vue, React, Angular, Svelte, etc.

But, what is a framework?

Think of it like a collection of programs written by someone else that you can include in your project to simplify the process of building your web application. The term web application is used interchangeably with the term website. A framework provides you with some abstractions on top of the programming language. As an over-simplification, you may think of a framework as a new language in itself but technically a framework is not a new language. Usually the way framework helps is by allowing you to write less code to accomplish something than you would have to if you were not using a framework. It also, makes you worry about less stuff by automatically taking care of a lot of things out of the box. You don't have to learn every framework mentioned above. Any of the ones I mentioned above will do. I would advise you to do some re-search regarding the job market in your location to see which of these frameworks is in demand. Personally, I liked both React and Vue with more inclination towards Vue because I found it relatively easier to learn. I haven't tried Svelte yet but I've heard good things about it. Whichever framework you pick, I would suggest you stick with it and build something using it before jumping on to learn another framework.

In order to be a good software developer, you need to build things. This is the fastest way to learn. See a tutorial, build something using what you've learned. If you don't apply your knowledge, it will diminish within days or weeks. By building something, whatever you learn will stick for a very long time

These frameworks that we talked about earlier are all JavaScript frameworks. There are frameworks for CSS as well, Bootstrap being probably the most popular one. There are others like Tailwind, Foundation, etc. but I don't know much about them and once you gain some basic CSS skills and if you feel like becoming CSS expert is what you want to do, you may explore those frameworks and choose whichever you like.

Mobile app developers

Mobile app development is another very popular career path that a lot of engineers pursue. The world of mobile app development is very vast but mobile apps can be broadly classified into two categories:

Native apps
Hybrid apps

Native apps are written in languages specific to the platform they run on. For example, for iOS, native apps are written in a programming language called Swift. So, if you wish to become an iOS app developer, Swift is the programming language you'll need to learn. For Android, the native mobile apps are written in Java (not JavaScript). So, if you want to become an android developer, you'll need to learn Java. Do note however that for an absolute beginner, learning Java or Swift can be quite challenging so don't be discouraged if you find it very difficult in the beginning.

Hybrid apps on the other hand are web apps. They're created using frameworks like React-Native, Vue-Native, NativeScript, Ionic, etc. These are like normal web application that you create using HTML, CSS and JavaScript but they can be packaged into mobile applications that can be installed on both iOS and android. So, this career path has a lot of overlap with that of a front-end web developer.

Backend engineer

When we talk about backend engineer, we usually refer to the people who write server side code that interacts with the website. Taking our earlier example of Twitter, the code that validates the user's credential on the server side and the code that fetches user's information like their tweets, notifications, like, etc from the database is written by a backend engineer. This code executes on server side. But, what is a server?

As an over-simplification, think of it as a computer program running on a BIG computer located in a secure location.

The code written on the client side (running in a browser) interacts with this computer program on the server by making network requests. So, if your internet connection is down, the browser code cannot interact with the server and your interaction with the web site breaks. These network requests between the client and server follow a specific protocol for communication called HTTP which stands for Hyper Text Transfer Protocol. The HTTP protocol dictates a set of rules that must be followed by the client and server to communicate with each other.
The code running in the browser sends requests to the server and the server returns response. These requests and responses are messages that must be in a specific format and this format is dictated by the HTTP protocol. For example, in order to fetch user's notifications, the browser code will make an HTTP request like this:

GET /user/notifications

This is just an oversimplified example of what a request could be and not the actual request being sent in the Twitter web application. Also, this is one small portion a request to give you an idea of how HTTP requests are identified by their verbs like "GET", "POST", "PUT", etc. and the URL which is /user/notifications in this case

For posting a tweet, the browser code would send some request like (again, this is just a hypothetical example and not at all the actual representation of the Twitter web app)

POST /tweets
{
   "content": "I just wrote my first blog on HashNode"
}

Usually, when we have to perform a write operation (anything that modifies some data in the database), for example, creating a tweet, liking a tweet, etc, we use a POST request and when we have to perform a read operation, like fetching notifications, followers, etc. we use a GET request. This is not enforced and you could use GET or POST any way you like but in general, we do it this way. There are other verbs like PUT, PATCH, DELETE as well but we won't go into the details of those in this article.
POST requests usually have request body associated with them. In the above example, the request body is enclosed in curly braces {}. This kind of representation of data is called JSON (stands for JavaScript Object Notation) and it is a very popular way of sending data between client and server.

So, now that we we know that the client code sends HTTP request to communicate with the server, let's talk about how the server code handles these requests.

On the server side, we need to write handlers for these HTTP requests. So, it's like:

Perform a certain action when we receive a GET request on the url /notifications. The HTTP requests also contain a lot of other information about user session, browser, etc which helps the server code identify where the request is coming from and which user is requesting this information.

These HTTP request handlers in the server code which provide sort of an interface between the client and server, are called **API (Application Programming Interface)*.
*
In these request handlers, a lot of stuff can happen like

authorization logic to check if the user is authorized to make this request
reading from/writing to a database
running some sort of a computation
calling another external service
and there are many other possibilities

This code can be written in any programming language. Some of the popular choices are:

JavaScript
TypeScript
Python
C#
Go
Java
PHP

There are other languages as well which can be used and I've only listed the popular ones that I could recall. But learning a language is often not sufficient for becoming a backend engineer. One needs to have a good understanding of databases as well.

*** What is a database you may ask? ***

In very simple terms, it is a software that allows you to store and retrieve data in efficient manner. Take an example of your cellphone's phonebook app. The phonebook app maintains it's own internal database in which it stores all the contacts with phone numbers, first name, last name, social media profile information, etc and allows you to search through the contacts in an efficient manner. Real world databases can be very complex, containing information about various entities. For example, if you were to build a Twitter clone, you would need to store information about users, their tweets, likes, followers, people they follow, bookmarks, lists, etc. A database management system (DBMS) allows us to store and retrieve such inter-connected information in an efficient manner. DBMS can be broadly classified into two categories:

Relational database management systems (RDBMS). These are databases where information about different entities is stored in different tables. For example, say we use a users table to store all the users, a tweets table to store all the tweets, a notifications table to store all the notifications. And these tables have references between each other. For example, the tweet table can contain a user id column which is a reference to the users table. Interaction with RDBMS (storing, retrieval of data, creation and modification of tables, etc) is supported through another programming language called Structured Query Language (SQL, also pronounced as Sequel). SQL is very different from other programming languages we've discussed earlier for example Python, Java, C#, JavaScript, etc. Some find it easy, some find it hard but as a backend developer, it is an essential skill to have and good SQL knowledge helps you write efficient code.
Non relational databases (also called NoSQL databases). The challenge with many relational databases is that it becomes expensive to scale them up when data grows (think of Terabytes of data). In order to solve this problem, many NoSQL databases came into existence. Some of the examples are MongoDB, Redis, Aerospike, Cassandra, CouchDB, Elasticsearch, DynamoDB, Neo4j. These databases don't support SQL and have either their own domain specific language or provide drivers in various programming languages to interact with them. Which ones to use depends purely on the type of problem you're trying to solve and we won't be going into the specifics of each. Also, which one to learn purely depends on the type of jobs you're applying for. But if you want to start on a career path of a backend developer, I would recommend starting with learning a programming language for writing backend code along with SQL for interacting with relational DBMS.

We've talked about front-end developer responsible for building the browser side of a web application and back-end developer responsible for building the server side of a web application. For small scale startups, it's often economical to hire engineers who know both of these worlds and can wear many hats depending on the need. There is a term for such web developers and it's called a "Full-stack developer". Most of the full stack developers start with one end of the stack (back-end or front-end) and gradually acquire the skills for the other end.

For example, someone may start with learning HTML, CSS and JavaScript to become front-end engineer and then since JavaScript can also be used in backend (using Nodejs), they may explore that territory, learn about databases, caching and other programming concept to become full stack developer.

In my case, it was the opposite. I spent first 4 years of my career interacting with databases, writing complex SQL code. Then I ventured into the world of backend API development using JavaScript (Nodejs) and gradually learned VueJS (since I already knew JavaScript) and then HTML and CSS. During all these years, I also learned other languages like Python, Go and C# and frontend frameworks like React which helped me become more resourceful as a full stack developer.

So, there is no right or wrong path. Pick whatever interests you and gradually expand further over time.

So far, we've discussed the career paths primarily related to web development and mobile app development. The world of tech is a lot more vast than that. Let's talk about some other popular career paths

Data Scientist

This is a career path which has grown significantly in popularity over the last decade (2010 - 2020) and most probably will continue to grow for at least another decade. But what do these people do?

In the technology world, data is everything because it gives the companies an insight into the minds of it's customers. Companies collect massive amount of data about their customers. Taking example of Twitter, all your interactions are captured like what kind of tweets you like or engage with more often, what kind of tweets you re-tweet, your scrolling behavior, your browsing behavior, etc. All of this data is then used by companies to customize their platform as per your profile. For example, Twitter showing you more of the tweets that you're likely to engage with, YouTube showing you more of the videos based on your past behavior, Netflix showing you movie recommendations based on your watch history, etc.

All the analysis that goes behind generating such recommendations is the job of a data scientist. They run algorithms on large volumes of data to generate such outcomes. These algorithms involve mathematics and statistics. Domains like "machine learning" and "artificial intelligence" are a part of the data science world.

Customization of a platform based on user's preferences is just one of the applications of data driven decision making. There are a ton of other applications. For example, credit card companies analyzing their customer's spending patterns to identify customers that are likely to commit a fraud or customers that are likely to default on their payments. Identifying such customers allows a company to mitigate risk proactively.

If it sounds exciting, it is! And if you wish to start on this path, learning Python is highly recommended as the starting point due to a wide range of tools and a huge community of data science built around it. Apart from that, learning SQL is also recommended.

DevOps Engineers or Cloud Engineers

This is another field that has grown in popularity over the last decade. Earlier we talked about various aspects of web development like creating the front-end application using HTML, CSS and JavaScript and creating the back-end APIs using one of the various programming languages. When you are building these applications, you would work on your own computers or laptops and test everything on your laptop or computer. Once you've tested it and feel confident that it is ready to be shipped and shared with the rest of the world, you will need to deploy it to the web. All the devices on the internet have an address called an IP address. It looks like this: "52.145.234.98" (four number separated by dots. Each of these numbers can be between 0 - 255). These addresses are called IPv4 addresses. There is another class of addresses called IPv6 addresses but we don't need to get into the details of that. You need to know that devices on the internet (like your laptop) have an IP address and other devices can reach out to you using this address.

Now technically, you can host your website on your machine and ask others to type your IP address on their web browsers and they should be able to see your website. You will need to have a static IP address for this to work. You can read more about static and dynamic IP addresses here. You can also purchase a domain from any of the domain registrars like Namecheap, GoDaddy, etc. and map that domain to your IP address. For example, let's say your static IP address is 52.145.234.98 and you purchase a domain name (www.mysupercooldomain.com) and configure it to map to your IP address. Then, if someone types http://www.mysupercooldomain.com on their web browser, they should see your website if your website is up and running on your laptop. While it is possible to host your website from your own laptop, it is not practical to do so. Because once you shutdown your laptop, your website will be unavailable. Also, as more and more users use your website, it will increase the load on your laptop and slow it down. In a real world scenario, we need our application to be resilient to such failures. Companies like Amazon, Google and Microsoft provide us the option of deploying our applications on their infrastructure. Amazon offers Amazon Web Services (AWS), Google offers Google Cloud Platform (GCP) and Microsoft offers Microsoft Azure. These companies have massive data centers which are bigger than a football field and contain a lot of computers. These data centers are also commonly referred to as "the cloud" (and this has got nothing to do with actual clouds responsible for rain :D).

As a funny aside, I would like to share this video

Again as an oversimplification, using these cloud offerings like AWS, GCP or Azure, we can deploy our applications on one of those computers and pay for the size of the computer we use. Big corporations often have their own data centers and don't require to rely on these cloud providers to deploy their applications. However, startups and smaller organizations benefit greatly from these cloud providers like AWS, GCP, Azure etc since they don't have to be concerned about provisioning of the infrastructure for their application and can easily scale up/down based on their requirements and budget.

The job of a DevOps engineer or a cloud engineer is to take your application and deploy it on one of these cloud providers. They're responsible for setting up the architecture to ensure our application is resilient to failures and can scale up with rising traffic. These engineers often have solid understanding of engineering principles, strong knowledge of distributed computing and expertise with one or more of various cloud providers like AWS, GCP, Azure, etc. They're also proficient with the usage of command line (terminal on Linux and Mac and PowerShell on Windows).

As a beginner, starting with this path can be challenging and quite overwhelming. But if you want to start, I would recommend getting comfortable with the command line (preferably Linux or Mac terminal) and starting with beginner level YouTube videos or Udemy course on either of the cloud providers (preferably AWS).

Conclusion

The world of tech is very vast and we've only scratched the surface. But I hope that this post helped you understand what kind of work various software developers do. There are a lot of other roles as well like a system administrator (SysAdmin), database engineer, database administrator, network engineer, etc. and a lot more specialized roles that I haven't mentioned in this post. But I hope that after reading through this post, you feel a little less overwhelmed to explore the world of tech as a beginner.

If this post helped you, share it so that it can help other beginners as well. And if you feel I should change something to improve it, let me know. Since it is aimed at beginners, my intent is to make it as beginner friendly as possible and that can be accomplished only when more and more beginners provide some constructive feedback on improving this post.

Having said that, I wish you all the best on your journey ahead :-)

How to connect your ExpressJS app with Postgres using Knex

Mandeep Singh Gulati — Sun, 17 Jan 2021 00:23:24 +0000

Note: I've created a video for this tutorial if you'd like to check that out here. Also, the supporting code for the same can be found here

Express is one of the most popular JavaScript framework for building backend APIs and Postgres is a really popular relational database. How do we connect the two?

If you look at the official documentation for Express, you'll see the section like this:

var pgp = require('pg-promise')(/* options */)
var db = pgp('postgres://username:password@host:port/database')

db.one('SELECT $1 AS value', 123)
  .then(function (data) {
    console.log('DATA:', data.value)
  })
  .catch(function (error) {
    console.log('ERROR:', error)
  })

It works for sure but it's not the way you would write it in a full fledged production application. Some of the questions that come to mind are:

How do you create the tables in the database?
How do you track changes to the database? For example, when you alter a table or create a new table. Or create/drop an index on a field. How to keep track of all these changes in your git/cvs/svn repository?
What if you switch from Postgres to some other database in future, say MariaDB for example? Do all your queries still work?

There might be a lot more questions but to me, the most important one feels like keeping track of changes to database in your application codebase. If someone clones my repository to their local system, they should have a command to create all the database tables on their local setup. Also, as we make changes to the database like adding/dropping tables or indices or altering any of the tables, one should be able to run a single command to sync their local copy of the database structure with the same on production DB. I am talking about structure, not the data. All the tables on the local database should have the same structure as that in the production database to make the testing of your application easy on local machine. And if you don't have this sync mechanism automated, you're likely to run into a lot of issues that you'll be troubleshooting in production.

To solve for these problems, we have libraries like Knex and Sequelize. These libraries provide a very neat API for writing SQL queries which are database agnostic and prevent issues like SQL injection attacks. They also provide transaction support to handle complex DB operations and streaming API to handle large volumes of data in a script. Also, to keep track of structural changes to your database in your code repo, these libraries use the concept of migrations. Migrations are files where you write structural changes you want to make to your database. For example, let's say you have a users table and want to alter the table to add a new column gender. You can write a Knex migration file like this:

exports.up = knex => knex.schema
  .alterTable('users', (table) => {
    table.string('gender')
  });

exports.down = knex => knex.schema
  .alterTable('user', (table) => {
    table.dropColumn('gender');
  });

The up function defines what to do when we run the migration and down function defines what to do when we rollback the migration. You can run the migration like this:

knex migrate:latest

And you can roll it back like this:

knex migrate:rollback

Once you commit this file to your code repository, your other team members can pull the changes from the repo and run these commands at their end to sync up the database structure on their machines.

In order to keep track of the database changes (migrations), Knex creates a few extra tables which contain information about what all migrations have been applied. So, for example if one of your team members hasn't synced their database in a long time and there are say 10 new migration scripts added since the last time they synced, when the pull the latest changes from the repo and run the migration command, all those 10 migrations will be applied in the sequence they were added to the repository.

Anyway, coming back to the main topic of this post. How do we add knex to our ExpressJS app and how do we use it to connect to our postgres database? Before we dive into this, there are some pre-requisites that should be met

Pre-Requisites

Node JS version 8 or higher intalled
Postgres installed on our localhost:5432

Steps

We will divide this article into following steps:

Creating the Express app
Creating the API endpoint with some hard coded data
Creating a database for our app
Installing and configuring knex
Populating seed data with knex
Updating the API endpoint created in step 2 to fetch the data from database instead of returning hard coded data

For this tutorial, we will be using Ubuntu Linux but these instructions should work fine other operating systems as well.

So, without further ado, let's get started with creating our Express app.

Step 1: Creating the Express app

Open the terminal (command prompt or Powershell on Windows), navigate to the directory where you want to create this project and create the project directory. We will be calling our project express-postgres-knex-app (not very innovative I know :-))

mkdir express-postgres-knex-app

Go to the project directory and run the following command to generate some boilerplate code using express generator

npx express-generator

The output should look like this:


   create : public/
   create : public/javascripts/
   create : public/images/
   create : public/stylesheets/
   create : public/stylesheets/style.css
   create : routes/
   create : routes/index.js
   create : routes/users.js
   create : views/
   create : views/error.ejs
   create : views/index.ejs
   create : app.js
   create : package.json
   create : bin/
   create : bin/www

   install dependencies:
     $ npm install

   run the app:
     $ DEBUG=express-postgres-knex-app:* npm start

This will create the some files and directories needed for a very basic Express application. We can customize it as per our requirements. Among other things, it will create an app.js file and a routes directory with index.js and users.js files inside. In order to run our application, we need to follow the instructions in the output shown above. First, install the dependencies:

npm install

Then run the app using the following command:

DEBUG=express-postgres-knex-app:* npm start

This should start our server on port 3000. If you go to your browser, you should be able to see the express application on http://localhost:3000

Step 2: Creating the API endpoint with some hard coded data

The express generator automatically created a users router for us. If you open the file routes/users.js, you should see the code like this:

var express = require('express');
var router = express.Router();
const DB = require('../services/DB');

/* GET users listing. */
router.get('/', async function (req, res, next) {
  return res.send('respond with a resource');
});

module.exports = router;

Here, we need to return the users array instead of a string respond with a resource. And we need to fetch those users from our database. So, for step 2, we don't need to do anything as we already have a route created for us by express generator. In the later steps, we will modify this code to actually fetch the users from our database

Step 3: Creating a database for our app

In this tutorial, we have a pre-requisite that postgres is installed on your machine. So, we need to connect to the postgres server and once you're inside, run the following command to create the database for our app:

create database express-app;

Step 4: Installing and configuring knex

Install knex and pg modules (since we are using postgres) by running the following command:

npm install knex pg

Once installed, initialize knex with a sample config file:

knex init

This should create a knexfile.js file in your project's root directory. This file contains the configuration to connect to the database. By default, the knexfile will be using sqlite for development. We need to change this since we are using postgres

Modify your knexfile.js so it looks like this:

// Update with your config settings.
const PGDB_PASSWORD = process.env.PGDB_PASSWORD;

module.exports = {
  development: {
    client: 'postgresql',
    connection: {
      host: 'localhost',
      database: 'express-app',
      user: 'postgres',
      password: PGDB_PASSWORD
    },
    pool: {
      min: 2,
      max: 10
    },
    migrations: {
      tableName: 'knex_migrations',
      directory: `${__dirname}/db/migrations`
    },
    seeds: {
      directory: `${__dirname}/db/seeds`
    }
  }
};

Now, we need to create a service called DB where we initialize knex in our application with the config from knexfile.js. In the project's root directory, create a directory services and inside the services directory, create a file DB.js

In that file, add the following code:

const config = require('../knexfile');

const knex = require('knex')(config[process.env.NODE_ENV]);

module.exports = knex;

Here, we are importing the config from knexfile and initializing the knex object using the same. Since, we will be running our app in development mode, the value of NODE_ENV will be development and the config for the same will be picked from the knexfile.js. If you run the app in production, you'll need to add the production config in the knexfile.js.

Now, wherever in our app we need to pull data from the database, we need to import this DB.js

Step 5: Populating seed data with knex

So we have our express app up and running with knex integrated. And we have our postgres database created. But we don't have any tables and data in our database. In this step, we will use knex migrations and seed files to do the same.

From the project's root directory, run the following commands:

npx knex migrate:make initial_setup

This will create a new file in the db/migrations directory.

npx knex seed:make initial_data

This will create a sample seed file under the db/seeds directory. First, we need to modify our migration file to create the users table. Open the newly created file under db/migrations directory and modify it so it looks like this:

exports.up = function (knex) {
  return knex.schema.createTable('users', function (table) {
    table.increments('id');
    table.string('name', 255).notNullable();
  });
};

exports.down = function (knex) {
  return knex.schema.dropTable('users');
};

Here, in the up function, we are creating a users table with two fields: id and name. So, when we apply this migration, a new table will be created. And in the down function, we are dropping the users table. So, when we rollback our migration, the users table will be deleted.

Also, open the newly created file under db/seeds directory and modify it so it looks like this:

exports.seed = function (knex) {
  // Deletes ALL existing entries
  return knex('users')
    .del()
    .then(function () {
      // Inserts seed entries
      return knex('users').insert([
        { id: 1, name: 'Alice' },
        { id: 2, name: 'Robert' },
        { id: 3, name: 'Eve' }
      ]);
    });
};

This will first remove any existing entries from our users table and then populate the same with 3 users.

Now, that we have our migration and seed files ready, we need to apply them. Run the following command to apply the migration:

npx knex migrate:latest

And then run the following command to populate the seed data:

npx knex seed:run

Now, if you connect to your postgres database, you should be able to see the users table with 3 entries. Now that we have our users table ready with data, we need to update the users.js file to fetch the entries from this table.

Step 5: Updating the API endpoint created in step 2 to fetch the data from database instead of returning hard coded data

Open the file routes/users.js and modify the API endpoint to look like this:

var express = require('express');
var router = express.Router();
const DB = require('../services/DB');

/* GET users listing. */
router.get('/', async function (req, res, next) {
  const users = await DB('users').select(['id', 'name']);
  return res.json(users);
});

module.exports = router;

Here, in the 3rd line we are importing the DB service. Then inside our route handler, we are fetching the users using the Knex's query builder

const users = await DB('users').select(['id', 'name']);

Knex does the job of translating this to an SQL query:

SELECT id, name FROM users;

And then we return the users (array of JSON objects) to the response.

Now, go to the terminal where you started the application earlier. Stop the server. If you remember in the knexfile we created earlier, we were using an environment variable PGDB_PASSWORD for passing the postgres password to our config. So we will need to export this variable with the password of our postgres server

export PGDB_PASSWORD=<enter your postgres password here>

Then run the Express server again

DEBUG=express-postgres-knex-app:* npm start

Now if you go to the http://localhost:3000/users , you should see the JSON array of user objects fetched from your postgres database.

Conclusion

So, in this article we created an Express JS app and connected it with a postgres database using Knex. We also touched upon the benefits of using a robust library like Knex for handling database operations in our application and learned about the concept of migrations. Hope you found this article helpful

How to install Elasticsearch 7 with Kibana using Docker Compose

Mandeep Singh Gulati — Thu, 08 Oct 2020 08:46:55 +0000

This tutorial will help you setup a single node Elasticsearch cluster with Kibana using Docker Compose.

Pre-requisites

Install Docker (refer to official docs if not installed https://docs.docker.com/engine/install/)
Install Docker Compose (refer to official docs if not installed https://docs.docker.com/compose/install/)

This tutorial assumes you are comfortable with Docker and Docker Compose. If you are not, you can go through this article of mine which is kind of a crash course with Docker Compose (https://medium.com/swlh/simplifying-development-on-your-local-machine-using-docker-and-docker-compose-2b9ef31bdbe7?source=friends_link&sk=240efed3fd3a43a1779e7066edb37235)

Video Lesson

I have also created a video tutorial for this on my YouTube channel. If you prefer that, you may visit the link below and check it out

https://www.youtube.com/watch?v=EClKhOE0p-o

Step 1: Create docker-compose.yml file

Create a directory on your machine for this project

mkdir $HOME/elasticsearch7-docker
cd $HOME/elasticsearch7-docker

Inside that directory create a docker-compose.yml file with contents as shown below

version: '3'
services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.9.2-amd64
    env_file:
      - elasticsearch.env
    volumes:
      - ./data/elasticsearch:/usr/share/elasticsearch/data

  kibana:
    image: docker.elastic.co/kibana/kibana:7.9.2
    env_file:
      - kibana.env
    ports:
      - 5601:5601

Step 2: Create the env files

Both Elasticsearch and Kibana docker images allow us to pass on environment variables which are passed on to the configuration as defined in elasticsearch.yml and kibana.yml files. For passing the environment variables to container, we can use the env_file setting of the docker compose file.

Create the elasticsearch.env file:

cluster.name=my-awesome-elasticsearch-cluster
network.host=0.0.0.0
bootstrap.memory_lock=true
discovery.type=single-node

Note: With latest version of Elasticsearch, it is necessary to set the option discovery.type=single-node for a single node cluster otherwise it won't start

Create kibana.env file

SERVER_HOST="0"
ELASTICSEARCH_URL=http://elasticsearch:9200
XPACK_SECURITY_ENABLED=false

Step 4: Create Elasticsearch data directory

Navigate to the directory where you have created your docker-compose.yml file and create a subdirectory data. Then inside the data directory create another directory elasticsearch.

mkdir data
cd data
mkdir elasticsearch

We will be mounting this directory to the data directory of elasticsearch container. In your docker-compose.yml file there are these lines:

    volumes:
      - ./data/elasticsearch:/usr/share/elasticsearch/data

This ensures that the data on your Elasticsearch container persists even when the container is stopped and restarted later. So, you won't lose your indices when you restart the containers.

Step 4: Run the setup

We're good to go now. Open your terminal and navigate to the folder containing your docker-compose.yml file and run the command:

docker-compose up -d

This will start pulling the images from docker.elastic.co and depending on your internet speed, this should take a while. Once the images are pulled, it will start the containers.

You can run the following command to see if both the containers are running:

docker-compose ps

The output should look something like this

                Name                              Command               State           Ports         
------------------------------------------------------------------------------------------------------
docker-elasticsearch-setup_elasticsearch_1   /tini -- /usr/local/bin/do ...   Up      9200/tcp, 9300/tcp    
docker-elasticsearch-setup_kibana_1          /usr/local/bin/dumb-init - ...   Up      0.0.0.0:5601->5601/tcp

Notice the State field. It should be Up for both the containers. If it is not, then check the logs using the command (replace {serviceName} with the name of the service, eg elasticsearch or kibana)

docker-compose logs -f {serviceName}

A common error that you might encounter is related to vm.max_map_count being too low. You can fix it by running the command

sysctl -w vm.max_map_count=262144

Check this link for more details https://www.elastic.co/guide/en/elasticsearch/reference/current/vm-max-map-count.html

If both the services are running fine, you should be able to see kibana console on http://localhost:5601 on your web browser. Give it a few minutes as it takes some time for Elasticsearch cluster to be ready and for Kibana to connect to it. You can get more info by inspecting the logs using docker-compose logs -f kibana command.

This completes our setup. Hope you found it helpful. Happy coding :-)

Cloning an object in JavaScript and avoiding Gotchas

Mandeep Singh Gulati — Tue, 29 Sep 2020 21:13:10 +0000

If you're a JavaScript developer, you must have come across scenarios where you need to clone an object. How do you do it? In this article we will cover various approaches to clone an object in JavaScript and their shortcomings and finally talk about the most reliable way to make a deep copy (clone) of an object in JavaScript.

Let us consider that our object to be cloned is this:

const person = {
  name: 'Dolores Abernathy',
  age: 32,
  dob: new Date('1988-09-01')
}

There can be various ways to clone it:

One way would be to declare a new variable and point it to the original object (which is not exactly cloning the object)

const clone = person

What you're doing here is you are referencing the same object. If you change clone.name, person.name will also change. Most of the times, this is not what you intend to do when you want to clone an object. You would want a copy of the object which does not share anything with the original object. Here, clone is just a reference to the same object being referred by person. Most of the JavaScript developers would know about this. So, this is not really a "Gotcha!". But the next two approaches I am going to show are definitely something you need to watch out for.

You'll often see code using spread operator to clone an object. For example:

const clone = { ...person }

Or code using Object.assign like this

const clone = Object.assign({}, person)

One might assume in both of the above cases that clone is a copy of the original person object and does not share anything with the original object. This is partially correct but can you guess the output of the code below? (Please take a moment to think what the output should be before copy pasting it)

const person = {
  name: 'Dolores Abernathy',
  age: 32,
  dob: new Date('1988-09-01')
}

const clone = { ...person }

// change the year for person.dob
person.dob.setYear(1986)

// check the clone's dob year
console.log(clone.dob.getFullYear())

What was your guess? 1988?

The correct answer is 1986. If you guessed the right answer and know the reason behind it, good! You have strong JavaScript fundamentals. But if you guessed it wrong, that's ok. It's the reason why I am sharing this blog post because a lot of us assume that by using the spread operator, we are creating a completely separate copy of the object. But this is not true. Same thing would happen with Object.assign({}, person) as well.

Both these approaches create a shallow copy of the original object. What does that mean? It means that all the fields of the original object which are primitive data types, will be copied by value but the object data types will be copied by reference.

In our original object, name and age are both primitive data types. So, changing person.name or person.age does not affect those fields in the clone object. However, dob is a date field which is not a primitive datatype. Hence, it is passed by reference. And when we change anything in dob field of the person object, we also modify the same in clone object.

How to create a deep copy of an object ?

Now that we know that both the spread operator and the Object.assign method create shallow copies of an object, how do we create a deep copy. When I say deep copy, I mean that the cloned object should be a completely independent copy of the original object and changing anything in one of those should not change anything in the other one.

Some people try JSON.parse and JSON.stringify combination for this. For example:

const person = {
  name: 'Dolores Abernathy',
  age: 32,
  dob: new Date('1988-09-01')
}

const clone = JSON.parse(JSON.stringify(person))

While it's not a bad approach, it has its shortcomings and you need to understand where to avoid using this approach.

In our example, dob is a date field. When we do JSON.stringify, it is converted to date string. And then when we do JSON.parse, the dob field remains a string and is not converted back to a date object. So, while clone is a completely independent copy of the person in this case, it is not an exact copy because the data type of dob field is different in both the objects.

You can try yourself

console.log(person.dob.constructor) // [Function: Date]
console.log(clone.dob.constructor) // [Function: String]

This approach also doesn't work if any of the fields in the original object is a function. For example

const person = {
  name: 'Dolores Abernathy',
  age: 32,
  dob: new Date('1988-09-01'),
  getFirstName: function() {
    console.log(this.name.split(' ')[0])
  }
}

const clone = JSON.parse(JSON.stringify(person))

console.log(Object.keys(person)) // [ 'name', 'age', 'dob', 'getFirstName' ]

console.log(Object.keys(clone)) // [ 'name', 'age', 'dob' ]

Notice that the getFirstName is missing in the clone object because it was skipped in the JSON.stringify operation as it is a function.

What is a reliable way to make a deep copy/clone of an object then?

Up until now all the approaches we have discussed have had some shortcomings. Now we will talk about the aproach that doesn't. If you need to make a truly deep clone of an object in JavaScript, use a third party library like lodash

const _ = require('lodash')

const person = {
  name: 'Dolores Abernathy',
  age: 32,
  dob: new Date('1988-09-01'),
  getFirstName: function() {
    console.log(this.name.split(' ')[0])
  }
}

const clone = _.cloneDeep(person)

// change the year for person.dob
person.dob.setYear(1986)

// check clone's dob year
console.log(clone.dob.getFullYear() // should be 1988

// Check that all fields (including function getFirstName) are copied to new object
console.log(Object.keys(clone)) // [ 'name', 'age', 'dob', 'getFirstName' ]

// check the data type of dob field in clone
console.log(clone.dob.constructor) // [Function: Date]

You can see that the cloneDeep function of lodash library will make a truly deep copy of an object.

Conclusion

Now that you know different ways of copying an object in JavaScript and pros and cons of each approach, I hope that this will help you in making a more informed decision on which approach to use for your use case and avoid any "Gotchas" while writing code.

Happy Coding :-)

How to add third party scripts & inline scripts in your Nuxt.js app?

Mandeep Singh Gulati — Mon, 23 Dec 2019 07:21:23 +0000

Problem statement

Let's say you have created a Nuxt app and one day your client or your boss asks you to add some snippet of code to every page of the site for analytics purposes. For example:

<!-- Global site tag (gtag.js) - Google Analytics -->
<script async src="https://www.googletagmanager.com/gtag/js?id=UA-111111111-1"></script>
<script>
  window.dataLayer = window.dataLayer || [];
  function gtag(){dataLayer.push(arguments);}
  gtag('js', new Date());

  gtag('config', 'UA-111111111-1');
</script>

Solution

Open your nuxt.config.js file and update the head section as follows:

  head: {
    __dangerouslyDisableSanitizers: ['script'],
    script: [
      {
        hid: 'gtm-script1',
        src: 'https://www.googletagmanager.com/gtag/js?id=UA-111111111-1',
        defer: true
      },
      {
        hid: 'gtm-script2',
        innerHTML: `
          window.dataLayer = window.dataLayer || [];
          function gtag(){dataLayer.push(arguments);}
          gtag('js', new Date());

          gtag('config', 'UA-111111111-1');
        `,
        type: 'text/javascript',
        charset: 'utf-8'
      }
    ]
  },

As you can see the script array contains two objects. First one is to include the external script from googletagmanager.com. The second object shows how to include inline Javascript. For that to work however, you need to add the setting __dangerouslyDisableSanitizers: ['script'],. I am not sure if this is the best or even the recommended approach but it worked for me. If you happen to know a better alternative, I would definitely love to know. You can mention in the comments section below or tag me on twitter.

Thanks and happy coding :-)

How to add authentication to your universal Nuxt app using nuxt/auth module?

Mandeep Singh Gulati — Mon, 30 Sep 2019 17:05:50 +0000

Recently I was working on a Nuxt.js app and had to add authentication to it. First thing I thought was to use vuex to store two fields in a state:

isLoggedIn: a boolean representing whether user is logged in or not
loggedInUser: an object containing the user details for the session that we get from server

And then I added a middleware on pages where I wanted to restrict access to logged in users only. The thought process for this approach is right but problem is that when you refresh the page, the state from vuex is lost. In order to handle that, you would need to use localStorage but that would work only if your app is running in spa mode, that is, on client side only. If you are running your app in universal mode (server side rendered) then you will also need to use cookies and write a custom middleware that checks whether it is running on client side or server side and then use localStorage or cookies accordingly. Doing all of this would be a good exercise to learn how everything works but adding it to a project where multiple people are working might not be a great idea in my opinion. Nuxt has an officially supported module just for this purpose. It's the auth module. In this post, I will talk about how to integrate the auth module to your nuxt app to support authentication using email and password.

Assumptions for the server API

We are making the assumption that the API server:

Is running on http://localhost:8080/v1
Uses cookie based sessions
Has a JSON based API
Has the following API endpoints:
- POST /v1/auth/login: accepts email and password in request body and authenticates the user
- POST /v1/auth/logout: does not need request body and deletes the user session from server
- GET /v1/auth/profile: returns the logged in user's object

Overview of the steps involved

We will divide this post into following steps:

Installation of axios and auth modules
Configuration needed in nuxt.config.js
Using the state from auth module to check if user is logged in or not and accessing logged in user in our app components
Using the auth module to authenticate the user using email and password based authentication
Using middleware provided by the auth module to restrict access to pages to logged in users only

Step 1: Install the axios and auth modules

Open the terminal, navigate to the root directory of your project and run the following command:

npm install @nuxtjs/auth @nuxtjs/axios

Step 2: Configure axios and auth modules

Open your nuxt.config.js file, find the modules section and include the axios and auth modules and add their configuration:

  modules: [
    '@nuxtjs/axios',
    '@nuxtjs/auth'
  ],

  auth: {
    strategies: {
      local: {
        endpoints: {
          login: {
            url: '/auth/login',
            method: 'post',
            propertyName: false
          },
          logout: { 
            url: '/auth/logout', 
            method: 'post' 
          },
          user: { 
            url: '/auth/profile', 
            method: 'get', 
            propertyName: false 
          }
        },
        tokenRequired: false,
        tokenType: false
      }
    }
  },

  axios: {
    baseURL: 'http://localhost:8080/v1',
    credentials: true
  },

The auth object here includes the configuration. The auth module supports local strategy as well as OAuth2. Since we only have email and password based authentication in our case, we only need to provide the configuration for local strategy.

The endpoints section is where we specify the details about our API server's endpoints for login, logout and logged in user's profile and each of the config looks like this:

  user: { 
    url: '/auth/profile', 
    method: 'get', 
    propertyName: false 
  }

url and method should be consistent with your server API. The url here needs to be relative to the baseUrl config. The propertyName tells the auth module which property in the response object to look for. For example, if your API server reponse for GET /auth/profile is like this:

{
  "user": {
    "id: 1,
    "name": "Jon Snow",
    "email": "jon.snow@asoiaf.com"
  }
}

Then you can set the propertyName as user to look for only the user key in the API response. If you want to use the entire API response, you need to set propertyName to false.

Since our API server has cookie based sessions, we are setting the tokenRequired and tokenType to false.

tokenRequired: false,
tokenType: false

For a complete list of options supported by the auth module, you can visit their official documentation here

The axios object in the above config is used to provide the axios configuration. Here, we are setting the following properties:

  axios: {
    baseURL: 'http://localhost:8080/v1',
    credentials: true
  },

baseUrl here is the root url of our API and any relative url that we hit using axios in our app will be relative to this url. Setting credentials as true ensures that we send the authentication headers to the API server in all requests.

Step 3: Activate vuex store in your app

In order to use the auth module, we need to activate vuex store in our application since that's where the session related information will be stored. This can be done by adding any .js file inside the store directory of your app and nuxt will register a namespaced vuex module with the name of the file. Let's go ahead and add a blank file called index.js to the store directory of our app. It's not mandatory to add index.js file. You could have added any file for example xyz.js in the store directory and that would have activated vuex store in your app.

The auth module that we have included in our project will automatically register a namespaced module named auth with the vuex store. And it has the following fields in the state:

loggedIn: A boolean denoting if the user is logged in or not
user: the user object as received from auth.strategies.local.user endpoint configured in our nuxt.config.js file.
strategy: This will be local in our case

It also adds the necessary mutations for setting the state. So, even though we haven't created any auth.js file in the store directory of our app, the auth module has automatically taken care of all this. If it helps to understand, imagine that a file named auth.js is automatically created by auth module in the store directory of your app even though this file doesn't actually exists. This means that using mapState on auth module of your vuex store will work. For example, you can use this in any of your components or pages:

  computed: {
    ...mapState('auth', ['loggedIn', 'user'])
  },

Here is a complete example of a component using these properties:

<template>
  <b-navbar type="dark" variant="dark">
    <b-navbar-brand to="/">NavBar</b-navbar-brand>
    <b-navbar-nav class="ml-auto">
      <b-nav-item v-if="!loggedIn" to="/login">Login</b-nav-item>
      <b-nav-item v-if="!loggedIn" to="/register">Register</b-nav-item>
      <b-nav-item v-if="loggedIn" @click="logout">
        <em>Hello {{ user.name }}</em>
      </b-nav-item>
      <b-nav-item v-if="loggedIn" @click="logout">Logout</b-nav-item>
    </b-navbar-nav>
  </b-navbar>
</template>

<script>
import { mapState } from 'vuex'
export default {
  name: 'NavBar',
  computed: {
    ...mapState('auth', ['loggedIn', 'user'])
  },
  methods: {
    async logout() {
      await this.$auth.logout()
      this.$router.push('/login')
    }
  }
}
</script>

<style></style>

Alternative approach

Instead of using the mapState, you can also reference the loggedIn and user by this.$auth.loggedIn and this.$auth.user. So, in the above example, you could have re-written the computed properties as mentioned below and it would have still worked fine:

  computed: {
    loggedIn() {
      return this.$auth.loggedIn
    },
    user() {
      return this.$auth.user
    }
  },

Step 4: Authenticating user using the auth module

We know how to use the auth module's APIs to check whether a user is logged in or not, or access the logged in user's details. But we haven't yet covered the part of how to authenticate the user. This is done by using the this.$auth.loginWith method provided by the auth module in any of your components or pages. The first argument to this function is the name of the strategy. In our case this will be local. It's an async function which returns a promise. Here is an example of how to use it:

  try {
    await this.$auth.loginWith('local', {
      data: {
        email: 'email@xyz.com'
        password: 'password',
      }
    })
    // do something on success
  } catch (e) {    
    // do something on failure 
  }

So, typically you would have a login page with a form with email and password fields mapped to data of the component using v-model. And once you submit the form, you can run this function to authenticate using the auth module. Here is an example of login page:

<template>
  <div class="row">
    <div class="mx-auto col-md-4 mt-5">
      <b-card>
        <b-form @submit="submitForm">
          <b-form-group
            id="input-group-1"
            label="Email address:"
            label-for="email"
          >
            <b-form-input
              id="email"
              v-model="email"
              type="email"
              required
              placeholder="Enter email"
            ></b-form-input>
          </b-form-group>

          <b-form-group
            id="input-group-2"
            label="Password:"
            label-for="password"
          >
            <b-form-input
              id="password"
              v-model="password"
              type="password"
              required
              placeholder="Enter password"
            ></b-form-input>
          </b-form-group>

          <b-button type="submit" variant="primary">Login</b-button>
        </b-form>
      </b-card>
    </div>
  </div>
</template>

<script>
export default {
  name: 'LoginPage',
  data() {
    return {
      email: '',
      password: ''
    }
  },
  methods: {
    async submitForm(evt) {
      evt.preventDefault()
      const credentials = {
        email: this.email,
        password: this.password
      }
      try {
        await this.$auth.loginWith('local', {
          data: credentials
        })
        this.$router.push('/')
      } catch (e) {
        this.$router.push('/login')
      }
    }
  }
}
</script>

<style></style>

In order to logout a logged in user, you can use the this.$auth.logout method provided by the auth module. This one doesn't need any arguments. Here is an example:

  methods: {
    async logout() {
      await this.$auth.logout()
      this.$router.push('/login')
    }
  }

Step 5: Using auth middleware to restrict access to certain pages

The auth module also provides middleware to restrict access to logged in users. So, for example if you want to restrict the /profile route of your application to logged in users only, you can add the auth middleware to the profile.vue page like this:

export default {
  name: 'ProfilePage',
  middleware: ['auth']
}

For more details on how you can configure your components and pages to use the auth middleware, you can check out the official docs here.

Conclusion and References

This was kind of a getting started post for axios and auth modules with NuxtJS. We only covered the local strategy but the auth module also supports OAuth2 and can be used to support login using Auth0, Facebook, Github and Google. I would definitely recommend checking out the Guide and API section of the auth module:

https://auth.nuxtjs.org/

The axios module also provides us many configuration options. Although we didn't cover much of it in this post, but I would definitely recommend checking out the official docs for that as well:

https://axios.nuxtjs.org/

I hope this post was helpful in understanding the basics of auth module in Nuxt and makes it easier for you to navigate the rest of the official documentation on your own.

Happy coding :-)

How to SSH to AWS servers using an SSH config file?

Mandeep Singh Gulati — Wed, 25 Sep 2019 17:20:02 +0000

How do you usually SSH to an AWS (Amazon Web Services) EC2 instance? If your answer is:

ssh -i <your pem file> <username>@<ip address of server>

Then you should read this tutorial.

What is an SSH config file and why should I even bother to know?

The above mentioned method for connecting to an AWS EC2 instance or any remote server is absolutely correct. There is nothing wrong with it and it is a highly secure way of connecting to a remote server. But imagine yourself having to connect to 15 different servers almost every day (15 different IP addresses to remember) and each of them having a different private key file (the pem file in above example). Let's say in some of the servers you need to conect as user ubuntu and some of the servers you need to connect as user ec2-user, etc. Also, let us say you want some port forwarding (more on this later) in some of those connections. Remembering all these configs for even a handful of servers can be a pain and it becomes a mess to handle everything with the above mentioned method. Do you see the ugliness of it, the disarray? Would it not be much easier if you could just write the command:

$ ssh dev-server

$ ssh production-server

Imagine executing this command from any directory, without bothering to remember the location of your pem files (private keys), the username with which you want to connect and the IP address of the server. This would make life so much better. That's exactly what an SSH config file is meant for. As its name suggests, it's a file where you provide all sorts of configuration options like the server IP address, location of the private key file, username, port forwarding, etc. And here you provide an easy to remember name for the servers like dev-server or production-server, etc.

Now, do you see the beauty of it? The possibilities, the wonder? Well, if you do and you wish to learn how to explore these possibilities, then read on.

What we will do in this tutorial?

We will quickly go through a brief introduction of SSH and the concept of private and public keys. Then we will see how to SSH to an AWS instance without using a config file. Then we will learn how to connect to the same instance using an SSH config file instead. So, this brings us to our first question

What is SSH?

SSH stands for secure shell. Wikipedia defintion says:

Secure Shell (SSH) is a cryptographic network protocol for operating network services securely over an unsecured network

In very simple terms, it is a secure way of logging in to a remote server. It gives you a terminal to the remote server where you can execute the shell commands.

How does it work?

When you wish to connect to a remote server using SSH from your local machine, your local machine here is a client and the remote server is a server. The client machine needs a process called ssh client whose task is to initiate ssh connection requests and participate in the process of establishing the connection with the server. And the remote server needs to run a process called ssh server whose task is to listen for ssh connection requests, authenticate those and provide access to the remote server shell based on successful authentication. We provide the server ip address, username with which we wish to login, password or private key to the ssh client when we wish to connect to a remote server.

What are public and private keys?

Typically, when we connect to a remote server via SSH, we do it using a public-private key based authentication. Public and private keys are basically base64 encoded strings which are stored in files. They are generated in pairs. Think of them as two different keys which are needed together to open a lock. And think of the process of establishing an SSH connection as a process of opening a lock. This process requires two keys of the same pair, a private key and its corresponding public key. We keep our private key file on our local machine and the server needs to store our public key.

Let us say we wish to login to a hypothetical remote server with IP address 54.0.0.121 with username john.

We give our ssh_client the address of the server (54.0.0.121), the username with which we wish to login (john) and the private key file to use. The ssh client goes to the ssh server using the address that we gave him and asks ssh server to bring the public key for user john to open the lock (authenticate the user john and provide him access to the remote server via SSH).

The ssh server checks the list of public keys he has and brings the public key for john. Both ssh client and ssh server then insert their respective public and private keys into the common lock. If the keys belong to the same pair, the lock will be opened and connection established. If the ssh server does not have the public key for user john then the lock does not open and authentication fails.

The above analogy is an oversimplification. The actual process is somewhat more complex. If you wish to understand the details of how it actually works, I would recommend this article on DigitalOcean.

How does one usually SSH to an AWS EC2 instance

When we create a new AWS EC2 instance for example using an Amazon Linux AMI or an Ubuntu Server AMI, at the last step we are asked a question about creating a new key pair or choosing an existing key pair. If you do it for the first time, you will need to create a new key pair. You provide a name for the key pair on this step. Let's say you provide the name as MyKeyPair at this step.
Before being able to proceed, you need to click the button Download Key Pair. This generates a pair of public and private key and lets you download the private key as MyKeyPair.pem file. After this when you click the Launch Instance button, AWS automatically adds the public key of the key pair to the newly created EC2 instance. Public keys are located in the ~/.ssh/authorized_keys file. So, if you choose Amazon Linux AMI while creating the EC2 instance, it will be added in the /home/ec2-user/.ssh/authorized_keys file. Similarly, if you use Ubuntu Linux AMI while creating a new EC2 instance, the public key will be added to the /home/ubuntu/.ssh/authorized_keys file. First thing you need to do is to change the permissions of your private key file (MyKeyPair.pem). Navigate to the directory where your private key file is located. And then run the following command:

$ chmod 400 MyKeyPair.pem

This gives read-only access to your private key file to only you. Other than yourself, nobody else can read or write this file. Now, in order to SSH to an EC2 instance, we would execute the following command:

$ ssh -i <path to MyKeyPair.pem> <username>@<ip address of the server>

So, for example if the IP address of the server was say 54.0.0.121 and we chose Ubuntu Linux while creating the EC2 instance, then the username will be ubuntu. And our command becomes

$ ssh -i MyKeyPair.pem ubuntu@54.0.0.121

This is assuming we are running this command from the directory containing our MyKeyPair.pem file. If we are executing this command from some other directory then we will need to provide the correct path of the MyKeyPair.pem file. Similarly, if we used Amazon Linux AMI while creating the EC2 instance, then username in that case becomes ec2-user.

So, this explains how AWS generates public-private key pairs when you create an EC2 instance and how you can use the private key to connect to an EC2 instance. Next we will learn how to do the same using an SSH config file.

How to use an SSH config file

We have already discussed what an SSH config file is. Now we will create one and use that to connect to our EC2 instance that we connected earlier. SSH file needs to be in the ~/.ssh directory of the client machine. In our case, this will be our local machine. So, go to the ~/.ssh directory (create it if it does not exist) and then create a file with name config. Open the file and add the following contents to it:

Host <an easy to remember name for the server>
  HostName <IP address of the server>
  IdentityFile <full path of the private Key file>
  User <username>

Replace the values in <> with actual values in your case. For example, if we used Ubuntu Linux AMI and the IP address of the server is 54.0.0.121 and the private key file (MyKeyPair.pem file) is located in /home/mandeep/private_keys directory, then the content of the config file becomes:

Host my-server
  HostName 54.0.0.121
  IdentityFile /home/mandeep/private_keys/MyKeyPair.pem
  User ubuntu

Let us see what each of these lines mean:

Host: Here you need to provide any easy to remember name for the server. This is only for your reference
Hostname: This is the fully qualified domain name or IP address of the server. In our example we have included IP address but it can also be fully qualified domain name like api.example.com_.
IdentityFile: Absolute path of the private key file
User: username of the user logging in. This user must exist on the server and have the public key in the ~/.ssh/authorized_keys file.

Once you save this file, you can easily connect to your EC2 instance by running the following command in the terminal:

$ ssh my-server

Here, it does not matter from which directory you execute this command. You can add as many configurations as you want in your config file. For example, if you wish to connect to another server with IP address say 54.1.1.91 and private key as MySecondKey.pem and username as ec2-user then your config file should look like this:

Host my-server
  HostName 54.0.0.121
  IdentityFile /home/mandeep/private_keys/MyKeyPair.pem
  User ubuntu
Host my-second-server
  HostName 54.1.1.91
  IdentityFile /home/mandeep/private_keys/MySecondKey.pem
  User ec2-user

Now you can connect to the my-second-server by running the command:

$ ssh my-second-server

That's it. That's how you create an SSH config file. Easy, isn't it? And once you start using it, it's hard to imagine living without it. It makes life so much better.

So, we know how to SSH to a remote server using config files. What next?

Well, there are plenty of configuration options one can provide in a config file and discussing all of them is beyond the scope of this tutorial. You can refer to the documentation here for the complete list of options, but I will be discussing the two options that I usually find quite handy:

LocalForward
ForwardAgent

LocalForward or Local Port Forwarding

Let us discuss this with an example. Consider a scenario that you have a remote server with the domain name redis.mydomain.com. And let us say we are running some process on this server which is not accessible publicly. For example, let us say we are running a redis server on this remote server on port 6379 but it can only be accessed once you login to the remote server and not from outside. Now let's say our requirement is that we need to access this remote redis server in a script running on our local machine. How do we do this?

SSH tunneling allows us to map a port from our local machine to a ip address:port on the remote server. For example, we can map the port 6389 on our local machine to the address localhost:6379 on the remote server. After doing this, our local machine thinks that the redis server (which is actually running on remote server on localhost:6379) is running on our local machine on port 6389. So, when you hit localhost:6389 on your local machine, you are actually hitting the redis server running on the remote server on port 6379.

How do we do this using our SSH config file?

We just need to add an additional property of LocalForward. Here is an example:

Host Redis-Server
  Hostname redis.mydomain.com
  IdentityFile /home/mandeep/private_keys/RedisServerKey.pem
  Localforward 6389 localhost:6379
  User ubuntu

This approach comes quite handy when you want to access a server which is a part of a VPC (Virtual Private Cloud) and not accessible publicly. For example, an Elasticache instance, an RDS instance, etc.

ForwardAgent

This property allows your SSH session to acquire the credentials of your local machine. Consider a scenario where you have a private Git repository on Github. You can access the repository either via HTTPS using username and password or by using SSH using private key. Username and password approach is less secure and not recommended. For accessing your repo via SSH, what we typically do is we create a private key public key pair which are stored in ~/.ssh directory as id_rsa (private key) and id_rsa.pub (publick key) files. Once we add our public key (id_rsa.pub) to our Github account, we can access our repository via SSH. This works well with our local machine. Now consider a scenario where you need to SSH to a remote server and access the Git repository from that remote server. You have two options here. One is to copy your private key (id_rsa) file and put it in the ~/.ssh directory on the remote server. This is a bad approach since you are not supposed to share your private key file. Another approach would be to generate a new key pair on the server and add the public key of that pair to the Github repo. There is a problem with both the approaches. Anyone who can SSH to the remote server will be able to access the Git repository. Let's say we don't want that. Let's say we only want the developers who have access to the repo through their own private key files should be able to access the repo. Anybody else who does not have access to the repo but can SSH to the remote server should not be able to access the repo from there. This is where the ForwardAgent property comes quite handy. You can add this to your config file as shown below:

Host App-Server
  Hostname app.mydomain.com
  IdentityFile /home/mandeep/private_keys/AppServerKey.pem
  User ubuntu
  ForwardAgent yes

After adding this property to your config file, when you SSH to the server using the following command:

$ ssh App-Server

Then the SSH terminal that gets opened acquires the credentials (id_rsa) file from your local machine. Now, even if there is no ~/.ssh/id_rsa file on the remote server, any Git repository that you can access on your local machine, you can also access that repository from the remote server.

Conclusion

With this tutorial we learned the importance of an SSH config file and saw how it can make our lives easier. If you found this tutorial helpful and believe that it can help others, please share it on social media using the social media sharing buttons below. If you like my tutorials and my writing style, follow me on twitter. If you feel I have made any mistakes or any information in this article is incorrect, feel free to mention those in the comments below. Thanks! Happy coding :-)

Setting up Elasticsearch and Kibana on Docker with X-Pack security enabled

Mandeep Singh Gulati — Wed, 25 Sep 2019 17:14:46 +0000

This tutorial assumes that you are familiar with Elasticsearch and Kibana and have some understanding of Docker. Before diving into the objective of this article, I would like to provide a brief introduction about X-Pack and go over some of the latest changes in Elasticsearch version 6.8 which allow us to use the security features of X-Pack for free with the basic license.

X-Pack Security and Elasticsearch 6.8

X-Pack is a set of features that extend the Elastic Stack, that is Elasticsearch, Kibana, Logstash and Beats. This includes features like security, monitoring, machine learning, reporting, etc. In this article, we are mainly concerned with the security features of X-Pack.

X-Pack security makes securing you Elasticsearch cluster very easy and highly customizable. It allows you to setup authentication for your Elasticsearch cluster, create different users with different credentials and different levels of access. It also allows you to create different roles and assign similar users to same role. For example, if you want to grant read-only access to certain users to certain indices of your cluster but want to ensure they cannot write to those indices, you can easily achieve that with X-pack security. And this is just the tip of the iceberg. You can check-out the security API here for a more detailed view of what all you can do with it.

But all these features were not always available for free. Prior to version 6.8, security was not a part of the Basic license. I'll quickly explain what this means. The Elastic Stack has 4 different types of licenses that you can see here

Open Source
Basic
Gold
Platinum

Gold and Platinum are paid licenses whereas Open Source and Basic are free. If you visit the above mentioned link, you can see what all features are available under which license. If you see the security dropdown on that page, you can see that some of the security features are available as a part of Basic license. As of writing this article, following security features are availale under Basic license:

Now this list reflects the latest version of Elastic Stack which at the time of writing this article, is version 7.1. Security was made available under basic license from version 6.8 onwards. This is important because it means that if you want to be able to use security features in your Elasticsearch setup for free, you need to have version 6.8 onwards for this.

And that's what we will be using for this article.

Objective

The objective of this article is to setup Elasticsearch and Kibana using Docker Compose with security features enabled. We will be setting up basic authentication on Elasticsearch so that all the API calls will need to include the Bearer token. Also, Kibana UI will require the username and password to login. For our setup, we will be using Docker Compose which makes our entire setup very easy to depoy anywhere and scale. I'll be using Ubuntu 18.04 for this tutorial but the steps will remain more or less same on any other unix based systems and might not be too different on a Windows based system as well. The only piece of code that we will be writing in this article is a docker-compose.yml file. We will be starting with a minimal docker-compose.yml file to get the elasticsearch and kibana setup up and running and then gradually we will tweak it to enable the security features.

Pre requisites

Linux based OS
Docker and Docker Compose installed
Basic understanding of Docker and Docker Compose
Knowledge of Elasticsearch and Kibana
Experience with Linux command line

If you are using some other operating system, you can follow the instructions specific to that OS but the process remains more or less the same.

Step 1 - Create a basic docker-compose.yml file for Elasticsearch and Kibana

In this step we will create our docker-compose.yml file with two services, elasticsearch and kibana and map their respective ports to the host OS

Let us first start with creating a directory for our project. Open your terminal and type the following

$ cd
$ mkdir elasticsearch-kibana-setup
$ cd elasticsearch-kibana-setup
$ touch docker-compose.yml

Then open the newly created docker-compose.yml file and paste the following lines in it:

version: '3'
services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:6.8.0
    ports:
      - 9200:9200

  kibana:
    depends_on:
      - elasticsearch  
    image: docker.elastic.co/kibana/kibana:6.8.0
    ports:
      - 5601:5601

The official docker images for Elastic Stack can be found here

As discussed in the beginning of this article, we will be using version 6.8 for this setup. If you visit the above link and click on Elasticsearch image 6.8 to expand, you'll see two images:

docker pull docker.elastic.co/elasticsearch/elasticsearch:6.8.0   
docker pull docker.elastic.co/elasticsearch/elasticsearch-oss:6.8.0

You can see that one of them has oss tag while the other does not. The difference between these two images is the license. The oss one comes with open source license whereas the non-oss one comes with the basic license. Since the x-pack security features are only available with the basic license, we will be using the non-oss version. Please note that this is also free as explained in the beginning of the article.

Apart from specifying the images, we are mapping the ports of the containers to the ports on the host machine. Elasticsearch runs on port 9200 and Kibana on port 5601 so we are mapping both these ports to the corresponding ports on the host machine. You can map them to some other port as well. The syntax remains the same:

<Host Port>:<Container Port>

So, for instance, if you want to access elasticsearch on port 8080 of your host machine, you'll need to specify the config as:

8080:9200

For now we'll be mapping it to 9200 in this article. Also the depends_on setting in kibana service ensures that it is not started until elasticsearch service is up and running. So, let's try to start our setup with the above settings by running the following command:

$ docker-compose up

This will start pulling the images from docker registry and create the containers. This may take a while depending on whether you already have the images on your machine or not and also depending on your internet speed. After the images have been pulled, you'll start seeing container logs which will take a few more seconds. Once both Elasticsearch and Kibana are ready, you'll see something like this in your console:

elasticsearch_1  | [2019-06-09T10:14:21,167][INFO ][o.e.c.r.a.AllocationService] [pKPbPLz] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[.kibana_1][0]] ...]).
kibana_1         | {"type":"log","@timestamp":"2019-06-09T10:14:21Z","tags":["info","migrations"],"pid":1,"message":"Pointing alias .kibana to .kibana_1."}
kibana_1         | {"type":"log","@timestamp":"2019-06-09T10:14:21Z","tags":["info","migrations"],"pid":1,"message":"Finished in 175ms."}
kibana_1         | {"type":"log","@timestamp":"2019-06-09T10:14:21Z","tags":["listening","info"],"pid":1,"message":"Server running at http://0:5601"}
elasticsearch_1  | [2019-06-09T10:14:21,282][INFO ][o.e.c.m.MetaDataIndexTemplateService] [pKPbPLz] adding template [kibana_index_template:.kibana] for index patterns [.kibana]
elasticsearch_1  | [2019-06-09T10:14:21,326][INFO ][o.e.c.m.MetaDataIndexTemplateService] [pKPbPLz] adding template [kibana_index_template:.kibana] for index patterns [.kibana]
elasticsearch_1  | [2019-06-09T10:14:21,343][INFO ][o.e.c.m.MetaDataIndexTemplateService] [pKPbPLz] adding template [kibana_index_template:.kibana] for index patterns [.kibana]
kibana_1         | {"type":"log","@timestamp":"2019-06-09T10:14:22Z","tags":["status","plugin:spaces@6.8.0","info"],"pid":1,"state":"green","message":"Status changed from yellow to green - Ready","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}

Look for the lines where it says status has changed to green. This means that our setup is ready. If you don't see such lines and see any error message it means something went wrong. You'll need to debug the issue and resolve it.

Once the services are up and running, open your browser and open the url http://localhost:9200/ and you will see something like this:

{
  "name" : "pKPbPLz",
  "cluster_name" : "docker-cluster",
  "cluster_uuid" : "AjqbFZ0qRF-X0_TQZqWIZA",
  "version" : {
    "number" : "6.8.0",
    "build_flavor" : "default",
    "build_type" : "docker",
    "build_hash" : "65b6179",
    "build_date" : "2019-05-15T20:06:13.172855Z",
    "build_snapshot" : false,
    "lucene_version" : "7.7.0",
    "minimum_wire_compatibility_version" : "5.6.0",
    "minimum_index_compatibility_version" : "5.0.0"
  },
  "tagline" : "You Know, for Search"
}

That's elasticsearch. Also, if you navigate to http://localhost:5601/ you should see the Kibana console. So, with just 13 lines of code in your docker-compose.yml file, you have setup a single node cluster of Elasticsearch and Kibana.

Now although this works, there is a security challenge if you want to deploy it to production. If your server is not a part of a VPC (Virtual Private Cloud) and the ports 9200 and 5601 are accessible to the world, your Elasticsearch and Kibana services can be accessed by anyone. There is no authorization so anyone make any changes to your cluster using the Elasticsearch API directly or through Kibana UI. What if we wanted to keep those ports accessible but require some sort of authentication so that only those who have the right credentials can access our Elasticsearch instance or login to the Kibana UI? Also, what if we want to ensure that certain users should only have limited set of priveleges? For example, we want certain users to be able to search any index in our Elasticsearch cluster but not be able to create any new index or drop any index or change any mapping or write to an index. Or let's say you don't want your Elasticsearch instance directly accessible to the rest of the world but want to keep the Kibana UI accessible and behind authentication and you want different users of Kibana UI to have different access levels. All of this can be achieved with X-Pack security and that's what we will be exploring next.

Go back to the terminal window where you ran the docker-compose up command and press CTRL+C to stop the containers and tear down the setup.

Step 2 - Customize Elasticsearch and Kibana services with environment variables

In order to enable X-Pack security, we will need to customize our elasticsearch and kibana services. Elasticsearch settings can be customized via elasticsearch.yml file and Kibana settings can be customized via kibana.yml file. There are many ways to change this while using docker. We can pass enviroment variables via our docker-compose.yml file. Although this would normally be an ideal way, but the way Elasticsearch and Kibana env variables are passed is not the same and can cause problems in certain deployment environments. You can read more about it here. For this tutorial, we will be creating custom elasticsearch.yml and kibana.yml files and bind mount these to their respective containers, overriding the default files in those container.

This will become more clear in next steps. First, create two files elasticsearch.yml and kibana.yml in the same directory as our docker-compose.yml file:

$ touch elasticsearch.yml
$ touch kibana.yml

Then open elasticsearch.yml and paste the following lines in it:

cluster.name: my-elasticsearch-cluster
network.host: 0.0.0.0
xpack.security.enabled: true

Here we are setting the name of our cluster to my-elasticsearch-cluster. The setting network.host: 0.0.0.0 means that elasticsearch will be accessible from all IP addresses on the host machine if the host machine has more than one network interface. And the last setting is to enable X-Pack security. This ensures that anyone trying to access our Elasticsearch instance must provide the authentication token.

Now open the kibana.yml file and paste the following lines in it:

server.name: kibana
server.host: "0"
elasticsearch.hosts: [ "http://elasticsearch:9200" ]

Here we are setting the server name. The server.host: "0" means that the Kibana instance should be accessible from all the IP addresses on the host machine if the host machine has more than one network interface. And the last setting elasticsearch.hosts includes the list of addresses of the Elasticsearch nodes. Kibana instance can reach out to the Elasticsearch instance by using the address http://elasticsearch:9200. This is achieved by Docker Compose. If you have multiple services in your compose file, containers belonging to one service can reach out to container of other services by using the other service's name. You don't even need to expose the ports for this. So, in our docker-compose.yml file, even if we had not mapped the ports for Elasticsearch, our Kibana instance would still be able to reach out to Elasticsearch instance at http://elasticsearch:9200. However, in that case, we won't be able to connect to our Elasticsearch instance from our host machine. I won't be diving further deep into the details of how networking works in Docker because that will be beyond the scope of this article. But I would definitely suggest you to go through the official docs to get a better understanding.

Ok, so now that we have our config files ready, we need to bind mount them to their respective containers in our docker-compose.yml file. So open the docker-compose.yml file and change it to look like this:

version: '3'
services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:6.8.0
    ports:
      - 9200:9200
    volumes:
      - ./elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml

  kibana:
    depends_on:
      - elasticsearch  
    image: docker.elastic.co/kibana/kibana:6.8.0
    ports:
      - 5601:5601
    volumes:
      - ./kibana.yml:/usr/share/kibana/config/kibana.yml

The only changes we have made here is that we have added the volumes section. By using volumes we can map a directory or an individual file from host machine to a directory or a file on the container. Here we are mapping individual files only. The default location of config file in Elasticsearch container is /usr/share/elasticsearch/config/elasticsearch.yml and we are replacing it with the elasticsearch.yml file that we created earlier. Similarly, we are replacing the default kibana.yml file at /usr/share/kibana/config/kibana.yml with our newly created file. With these changes, let's try to start our docker compose setup again by running the command:

$ docker-compose up

This will most likely give you an error. If you see the elasticsearch logs (lines starting with elasticsearch_1 |), you might see some error like this:

elasticsearch_1  | [1]: Transport SSL must be enabled if security is enabled on a [basic] license. Please set [xpack.security.transport.ssl.enabled] to [true] or disable security by setting [xpack.security.enabled] to [false]

This means that Elasticsearch won't start since the initial checks have failed. Consequently, Kibana won't be able to connect to it and you'll see something like this in Kibana logs:

kibana_1         | {"type":"log","@timestamp":"2019-06-11T17:31:14Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"No living connections"}

Press Ctrl+C to stop the containers and tear down the setup because this ain't working and we gotta fix it.

In order to get elasticsearch working, we will need to enable SSL and also install the SSL certificate in our elasticsearch container. I will be walking through the process of creating a new certificate and using that. If you already have a certificate file, you can skip that part. For this, we will need to take a step back and disable x-pack security on our elasticsearch instance so that we can get it up and running and then we will get inside our container shell and generate the certificate.

Step 3 - Create SSL certificate for Elasticsearch and enable SSL

First, we need to disable x-pack security temporarily so that we can get our Elasticsearch container up and running. So, open the elasticsearch.yml file and disable x-pack security by changing the following line:

xpack.security.enabled: false

Then bring up the containers again by running:

$ docker-compose up

This should work fine now and bring up our Elasticearch and Kibana services just like before. Now, we need to generate the certificates and we will be using elasticsearch-certutil utility. For this, we will need to get inside our docker container running elasticsearch service. This is really easy using docker-compose. Think of it like this, we can execute any command inside a docker container by using the command:

$ docker-compose exec <service name> <command>

And if we want to get inside the container's shell, we essentially want to execute the bash command on our container. So, our command becomes:

$ docker-compose exec elasticsearch bash

Here elasticsearch is our service and bash is our command. We need to do this while the container is running so open another terminal window and paste the above command (make sure to run this command from the same directory where your docker-compose.yml file is located)

Once you're inside the container, your shell prompt should look something like this:

[root@c9f915e86309 elasticsearch]#

Now run the following command here:

[root@c9f915e86309 elasticsearch]# bin/elasticsearch-certutil ca

This will generate some warnings describe what it is going to do. I recommend you read that. And it will prompt you for file name and password. Just press ENTER for both to proceed:

Please enter the desired output file [elastic-stack-ca.p12]: 
Enter password for elastic-stack-ca.p12 :

This will create a file elastic-stack-ca.p12 in the directory from which you ran the above command. You can check by running the ls command. This is the certificate authority we will be using to create the certificate. Now, run the command:

[root@c9f915e86309 elasticsearch]# bin/elasticsearch-certutil cert --ca elastic-stack-ca.p12

This will again raise some warnings and describe what it is going to do. I recommend you read that too. And it will prompt you for password and file name. Press ENTER at all the steps to proceed:

Enter password for CA (elastic-stack-ca.p12) : 
Please enter the desired output file [elastic-certificates.p12]: 
Enter password for elastic-certificates.p12 :

This will create the elastic-certificates.p12 which is what we need. We need this file outside the container on the host machine because it will be vanish once we destroy our container. This file is in PKCS12 format which includes both the certificate as well as the private key. In order to copy this file outside the container to host machine, press CTRL+D to first exit the container

And then run the following command on your host machine (from the same directory where docker-compose.yml file is present)

$ docker cp "$(docker-compose ps -q elasticsearch)":/usr/share/elasticsearch/elastic-certificates.p12 .

The above command might seem a bit tricky to some so I will add a bit of explanation here. For those of you who understand how it works can proceed to Step 4.

Let us first see what docker-compose ps does. If you run the command you'll see output like this:

                      Name                                    Command               State                Ports              
----------------------------------------------------------------------------------------------------------------------------
elasticsearch-kibana-setup_elasticsearch_1   /usr/local/bin/docker-entr ...   Up      0.0.0.0:9200->9200/tcp, 9300/tcp
elasticsearch-kibana-setup_kibana_1          /usr/local/bin/kibana-docker     Up      0.0.0.0:5601->5601/tcp

This shows all the docker containers running or stopped which are being managed by our docker-compose.yml file.

If you check the help for this command:

$ docker-compose ps --help

You will see output like this:

List containers.

Usage: ps [options] [SERVICE...]

Options:
    -q, --quiet          Only display IDs
    --services           Display services
    --filter KEY=VAL     Filter services by a property

You can see that by using -q flag, we can get just the id of the container. And also you can see that by providing the service name, we can limit the output to just the service we are interested in. So, if we want to get the id of the elasticsearch container, we need to run the command:

$ docker-compose ps -q elasticsearch

This should get you the id of the elasticsearch container.

Now, if we go back to our docker cp command above, you can check the syntax of that command by using help again:

$ docker cp --help

This should display the help:


Usage:  docker cp [OPTIONS] CONTAINER:SRC_PATH DEST_PATH|-
    docker cp [OPTIONS] SRC_PATH|- CONTAINER:DEST_PATH

Copy files/folders between a container and the local filesystem

Options:
  -a, --archive       Archive mode (copy all uid/gid information)
  -L, --follow-link   Always follow symbol link in SRC_PATH

You can see that we need to specify the command as:

docker cp <container id>:<src path> <dest path on host>

Our source path in this case is /usr/share/elasticsearch/elastic-certificates.p12 on the elasticsearch container. And we are getting the id of the elasticsearch container by using the docker-compose ps -q elasticsearch command. And we need to copy the file to the current directory on host so our destination path is .. Hence the command becomes:

$ docker cp "$(docker-compose ps -q elasticsearch)":/usr/share/elasticsearch/elastic-certificates.p12 .

We will also copy the CA file by running the command:

$ docker cp "$(docker-compose ps -q elasticsearch)":/usr/share/elasticsearch/elastic-stack-ca.p12 .

Now that we have our certificate file on our host machine, we will be bind mounting it to our container just like the way we did for elasticsearch.yml file. So, if you already have an SSL certificate you can use that too in place of this one.

Step 4 - Installing the SSL certificate on Elasticsearch and enabling TLS in config

Now that we have the SSL certificate available, we can enable the x-pack security on our elasticsearch node and also enable TLS. First, we need to bind mount our certificate from host machine to container. First, go back to the terminal where you ran docker-compose up command and press CTRL+C to stop the containers. Then open the docker-compose.yml file and change it so that it look like this:

version: '3'
services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:6.8.0
    ports:
      - 9200:9200
    volumes:
      - ./elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml
      - ./elastic-certificates.p12:/usr/share/elasticsearch/config/elastic-certificates.p12

  kibana:
    depends_on:
      - elasticsearch
    image: docker.elastic.co/kibana/kibana:6.8.0
    ports:
      - 5601:5601
    volumes:
      - ./kibana.yml:/usr/share/kibana/config/kibana.yml

Now, open your elasticsearch.yml file and change it to this:

cluster.name: my-elasticsearch-cluster
network.host: 0.0.0.0
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.keystore.type: PKCS12
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: elastic-certificates.p12
xpack.security.transport.ssl.truststore.type: PKCS12

First 3 lines are same as before (we have changed xpack.security.enabled to true again). Rest of the lines denote the SSL settings and location of our certificate and private key, which is the same. You can check out all the security settings here.

Once this is done, go back to terminal and bring up the container again

$ docker-compose up

So, what do you see? Still not working eh? This is because now Kibana is not able to connect to our Elasticsearch instance because now we have security enabled but haven't configured the credentials on Kibana. So, you'll see continous logs like this:

kibana_1         | {"type":"log","@timestamp":"2019-06-11T19:03:35Z","tags":["warning","task_manager"],"pid":1,"message":"PollError [security_exception] missing authentication token for REST request [/_template/.kibana_task_manager?include_type_name=true&filter_path=*.version], with { header={ WWW-Authenticate=\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\" } }"}

Also, if you open your web browser and go to http://localhost:9200 you will see a prompt for username and password. And if you press ESC, you get this error:

{
  "error": {
    "root_cause": [
      {
        "type": "security_exception",
        "reason": "missing authentication token for REST request [/]",
        "header": {
          "WWW-Authenticate": "Basic realm=\"security\" charset=\"UTF-8\""
        }
      }
    ],
    "type": "security_exception",
    "reason": "missing authentication token for REST request [/]",
    "header": {
      "WWW-Authenticate": "Basic realm=\"security\" charset=\"UTF-8\""
    }
  },
  "status": 401
}

And if you try visiting http://localhost:5601 you will get the error:

Kibana server is not ready yet

So, we have solved one part of the problem. We have secured our Elasticsearch instance and nobody can access it without providing the correct credentials. But we don't know what the correct credentials are. We will be setting those up in the next step and configuring Kibana to use those. For now, keep the docker-compose up command running since we need to go inside the Elasticsearch container again

Step 5 - Generate default passwords and configure the credentials in Kibana

Before we generate the passwords for built-in accounts of Elastic stack, we will first need to change our docker-compose.yml file to bind mount the data volume of Elasticsearch. Up until now, the storage of our containers has been temporary. It means that once we destroy the containers, all the data inside of them gets destroyed as well. So if you created any indices, users, etc in Elasticsearch, they will no longer persist once you do docker-compose down to bring down the services. That's not something we would want in production. We want to ensure that the data changes persist between container restarts. For that, we will need to bind mount the data directory from elasticsearch container to a directory on the host machine.

First, bring down all the running containers by executing the following command:

docker-compose down

Then create a directory called docker-data-volumes in the same directory where your docker-compose.yml file is located. You can give it any other name but for this tutorial we will call it docker-data-volumes. Inside that directory, create another directory called elasticsearch

mkdir docker-data-volumes
mkdir docker-data-volumes/elasticsearch

Now under the volumes section of elasticsearch service in your docker-compose.yml file, add the following line:

      - ./docker-data-volumes/elasticsearch:/usr/share/elasticsearch/data

As explained earlier, when we need to bind mount a file or directory from host machine to container, we specify the <host path>:<container path>. The default path for data inside an elasticsearch container is /usr/share/elasticsearch/data and we are binding it to the directory ./docker-data-volumes/elasticsearch on host machine. So your docker-compose.yml file should now look like this:

version: '3'
services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:6.8.0
    ports:
      - 9200:9200
    volumes:
      - ./elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml
      - ./elastic-certificates.p12:/usr/share/elasticsearch/config/elastic-certificates.p12
      - ./docker-data-volumes/elasticsearch:/usr/share/elasticsearch/data

  kibana:
    depends_on:
      - elasticsearch
    image: docker.elastic.co/kibana/kibana:6.8.0
    ports:
      - 5601:5601
    volumes:
      - ./kibana.yml:/usr/share/kibana/config/kibana.yml

Bring up the containers by running

docker-compose up

While docker-compose up is running in one terminal, open another terminal to get inside the elasticsearch container by running the command:

$ docker-compose exec elasticsearch bash

Then run the following command to generate passwords for all the built-in users:

[root@c9f915e86309 elasticsearch]# bin/elasticsearch-setup-passwords auto

Note them down and keep them somewhere safe. Exit the container by pressing CTRL+D

Now open the kibana.yml file and change it to this:

server.name: kibana
server.host: "0"
elasticsearch.hosts: [ "http://elasticsearch:9200" ]
elasticsearch.username: kibana
elasticsearch.password: <kibana password>

You need to put the password for kibana user here in the elasticsearch.password setting.

Go to the terminal where docker-compose up was running and press CTRL+C to bring the containers down. And then run the command again

$ docker-compose up

This should bring up both the services, elasticsearch and kibana. Now, if you open your browser and visit http://localhost:9200 it will again prompt you for username and password. Here, enter the username as elastic and enter the password for the user elastic that you got earlier. You should be able to see the output like this on successful authentication:

{
  "name" : "1mG1JlU",
  "cluster_name" : "my-elasticsearch-cluster",
  "cluster_uuid" : "-mEbLeYVRb-XqA24yq6D1w",
  "version" : {
    "number" : "6.8.0",
    "build_flavor" : "default",
    "build_type" : "docker",
    "build_hash" : "65b6179",
    "build_date" : "2019-05-15T20:06:13.172855Z",
    "build_snapshot" : false,
    "lucene_version" : "7.7.0",
    "minimum_wire_compatibility_version" : "5.6.0",
    "minimum_index_compatibility_version" : "5.0.0"
  },
  "tagline" : "You Know, for Search"
}

Also, if you open http://localhost:5601 you will see the Kibana console but now it will ask for username and password:

Here also enter the username as elastic and password for the same. You will see the Kibana console on successful authentication.

If you followed all the steps correctly till now, you should be able to login to Kibana console now.

Now if you click on the Management tab in the sidebar, you will see Security section on the right hand side panel.

You can see the users, roles, create new role or create new user over here. There is a lot you can do here and I would recommend you play with it for a while to get a feel of it.

Conclusion

This completes our tutorial and our setup of an Elasticsearch and Kibana cluster with basic license (Free) using docker-compose, with X-Pack security enabled. I hope you found it helpful and if you have any suggestions or find any errors, feel free to comment below.

Happy Coding :-)

How to read or modify spreadsheets from Google Sheets using Node.js ?

Mandeep Singh Gulati — Wed, 25 Sep 2019 17:12:40 +0000

First of all, a brief overview of our use case. Let's say I have a spreadsheet on Google Sheets which is not public and I want to be able to read/modify programmatically through some batch process running on my local machine or some server. This is something I had to do recently with a Node.js application and I found the authentication part a bit tricky to understand. So I thought of sharing my solution and I hope it helps someone in need. There might be better ways of doing this but I am sharing what worked best for me.

Since there is no user interaction involved in our use case, we don't want to use the OAuth process where user needs to open a browser and sign in to their Google account to authorize the application. For scenarios like this, Google has a concept of service account. A service account is a special type of Google account intended to represent a non-human user that needs to authenticate and be authorized to access data in Google APIs. Just like a normal account, a service account also has a email address (although it doesn't have an actual mailbox and you cannot send emails to a service account email). And just like you can share a google sheet with a user using their email address, you can share a google sheet with a service account as well using their email address. And this is exactly what we are going to do in this tutorial. We will create a spreadsheet on Google Sheets using a regular user, share it with a service account (that we will create) and use the credentials of the service account in our Node.js script to read and modify that sheet.

Pre-requisites

This tutorial assumes that you have:

Experience working with Node.js
A Google account
A project setup on Google developers console where you have admin priveleges

Steps Overview

Here is the list of steps we will be following through this tutorial:

Create a spreadsheet on Google sheets
Enable Google Sheets API in our project on Google developers console
Create a service account
Share the spreadsheet created in step 1 with the service account created in step 3
Write a Node.js service to access the google sheets created in step 1 using the service account credentials
Test our service written in step 5

Now that we have an outline of what all we are going to do, let's get started

Step 1: Create a spreadsheet on Google Sheets

This one doesn't really need any instructions. You just need to login to your google account, open Google Drive and create a new Google Sheet. You can put some random data in it. One thing that we need to take note of is the sheet's id. When you have the sheet open in your browser, the url will look something like this: https://docs.google.com/spreadsheets/d/1-XXXXXXXXXXXXXXXXXXXSgGTwY/edit#gid=0. And in this url, 1-XXXXXXXXXXXXXXXXXXXSgGTwY is the spreadsheet's id and it will be different for each spreadsheet. Take a note of it because we will need this in our Node.js script to access this spreadsheet. For this tutorial, here is the data we have stored in our spreadsheet:

Step 2: Enable Google Sheets API in our project on Google developers console

We need to enable Google Sheets API for our project in order to be able to use it. This tutorial assumes that you already have a project in Google developers console so if you don't have one, you can create a new one very easily. Once you have the project on Google developers console, open project dashboard. There you should see a button Enable APIs and Services.

Click on it and search for Google sheets API using the search bar. Once you see it, click on it and then click on Enable

Step 3: Create a Service Account

Once you enable Google Sheets API in your project, you will see the page where you can configure the settings for this API. Click on Credentials tab on the left sidebar. Here you will see a list of OAuth client IDs and service accounts. By default there should be none.

Click on Create Credentials button at the top and select Service Account option

Enter the name and description of the service account and click Create button.

Click Continue on the next dialog

On the next dialog, you get an option to create a key. This is an important step. Click on the Create Key button and choose JSON as the format. This will ask you to download the JSON file to your local machine.

For this tutorial, I have renamed the file and saved it as service_account_credentials.json on my local machine.

Keep it somewhere safe. This key file contains the credentials of the service account that we need in our Node.js script to access our spreadsheet from Google Sheets.

Once you've followed all of these steps, you should see the newly created service account on the credentials page

Take a note of the email address of the service account. We will need to share our spreadsheet with this account.

Step 4: Share the spreadsheet created in step 1 with the service account created in step 3

Now that we have a service account, we need to share our spreadsheet with it. It's just like sharing a spreadsheet with any normal user account. Open the spreadsheet in your browser and click on the Share button on top right corner. That will open a modal where you need to enter the email address of the service account. Uncheck the checkbox for Notify people since this will send an email and since service account does not have any mailbox, it will give you a mail delivery failure notification.

Click OK button to share the spreadsheet with the service account.

This completes all the configuration steps. Now we can get to the fun part :-)

Step 5: Write a Node.js service to access the google sheet using the service account credentials

We will create our script as a service that can be used as a part of a bigger project. We will call it googleSheetsService.js. It will expose following APIs:

getAuthToken
getSpreadSheet
getSpreadSheetValues

The function getAuthToken is where we will handle the authentication and it will return a token. Then we will be using that token and pass it on to other methods.

We will not be covering writing data to the spreadsheet but once you get the basic idea of how to use the API, it will be easy to extend the service to add more and more functions supported by the Google Sheets API.

We will be using the googleapis npm module. So, let's get started by creating a directory for this demo project. Let's call it google-sheets-demo.

cd $HOME
mkdir google-sheets-demo
cd google-sheets-demo

Copy the service_account_credentials.json file that we created in step 3 to this directory (google-sheets-demo). And create our new file googleSheetsService.js. Paste the following lines to the file:

// googleSheetsService.js

const { google } = require('googleapis')

const SCOPES = ['https://www.googleapis.com/auth/spreadsheets']

async function getAuthToken() {
  const auth = new google.auth.GoogleAuth({
    scopes: SCOPES
  });
  const authToken = await auth.getClient();
  return authToken;
}

module.exports = {
  getAuthToken,
}

For now our service has only one function that returns the auth token. We will add another function getSpreadSheet soon. First let us see what our function does.

First, we require the googleapis npm module. Then we define SCOPES. When we create an auth token using google APIs, there is a concept of scopes which determines the level of access our client has. For reading and editing spreadsheets, we need access to the scope https://www.googleapis.com/auth/spreadsheets. Similarly, if we only had to give readonly access to spreadsheets, we would have used scope https://www.googleapis.com/auth/spreadsheets.readonly.

Inside the getAuthToken function, we are calling the constructor new google.auth.GoogleAuth passing in the scopes in the arguments object.

This function expects two environment variables to be available, GCLOUD_PROJECT which is the project ID of your Google developer console project and GOOGLE_APPLICATION_CREDENTIALS which denotes the path of the file containing the credentials of the service account.

We will need to set these environment variables from the command line. To get the project ID, you can get it from the url of the project when you open it in your web browser. It should look like this

https://console.cloud.google.com/home/dashboard?project={project ID}

And GOOGLE_APPLICATION_CREDENTIALS must contain the path of the service_account_credentials.json file. So, go to the terminal and from the google-sheets-demo directory, run the following commands to set these environment variables:

export GCLOUD_PROJECT={project ID of your google project}
export GOOGLE_APPLICATION_CREDENTIALS=./service_account_credentials.json

You need to make sure that you have the credentials file copied in the current directory.

Now we will add two more functions to our service:

getSpreadSheet
getSpreadSheetValues

The first one will return metadata about the spreadsheet while the second one will return the data inside the spreadsheet. Our modified googleSheetsService.js file should look like this:

// googleSheetsService.js

const { google } = require('googleapis');
const sheets = google.sheets('v4');

const SCOPES = ['https://www.googleapis.com/auth/spreadsheets'];

async function getAuthToken() {
  const auth = new google.auth.GoogleAuth({
    scopes: SCOPES
  });
  const authToken = await auth.getClient();
  return authToken;
}

async function getSpreadSheet({spreadsheetId, auth}) {
  const res = await sheets.spreadsheets.get({
    spreadsheetId,
    auth,
  });
  return res;
}

async function getSpreadSheetValues({spreadsheetId, auth, sheetName}) {
  const res = await sheets.spreadsheets.values.get({
    spreadsheetId,
    auth,
    range: sheetName
  });
  return res;
}


module.exports = {
  getAuthToken,
  getSpreadSheet,
  getSpreadSheetValues
}

At the top we have added a line

const sheets = google.sheets('v4');

This is to use the sheets API. Then we have added the two new functions getSpreadSheet and getSpreadSheetValues. To see all the supported API endpoints for Google Sheets API, check this link https://developers.google.com/sheets/api/reference/rest.

For our demo, we are only using two of those. The getSpreadSheet function expects auth token and the spreadsheetId as its parameters. And the getSpreadSheetValues expects one additional parameter that is the sheetName from which to fetch the data. By default, a spreadsheet only contains a single sheet and it is named as Sheet1. Finally we export the newly added functions via module.exports.

This completes our googleSheetsService. If you need to support more API functions, you can check the reference using the link above, add the corresponding wrapper functions in this service and export it using module.exports. For any consumer of this service, they will first need to call the getAuthToken function to get the auth token and then pass on that token to the subsequent functions like getSpreadSheet, getSpreadSheetValues, etc. Now that we have our service ready, we just need to test it to make sure it is working fine

Step 6: Test our service

So we have our service ready. But does it work? Let's check that out.

While typically, we would use a testing framework to run unit tests, to keep this tutorial simple, we are going to write a simple Node.js script. From our project's directory, create a new file called test.js and copy paste the following contents:

const {
  getAuthToken,
  getSpreadSheet,
  getSpreadSheetValues
} = require('./googleSheetsService.js');

const spreadsheetId = process.argv[2];
const sheetName = process.argv[3];

async function testGetSpreadSheet() {
  try {
    const auth = await getAuthToken();
    const response = await getSpreadSheet({
      spreadsheetId,
      auth
    })
    console.log('output for getSpreadSheet', JSON.stringify(response.data, null, 2));
  } catch(error) {
    console.log(error.message, error.stack);
  }
}

async function testGetSpreadSheetValues() {
  try {
    const auth = await getAuthToken();
    const response = await getSpreadSheetValues({
      spreadsheetId,
      sheetName,
      auth
    })
    console.log('output for getSpreadSheetValues', JSON.stringify(response.data, null, 2));
  } catch(error) {
    console.log(error.message, error.stack);
  }
}

function main() {
  testGetSpreadSheet();
  testGetSpreadSheetValues();
}

main()

This file contains two test functions and a main function that is calling those test functions. At the bottom of the file, we are executing the main function. This script expects two command line arguments:

spreadsheetId (this is the ID that we got from step 1)
sheetName (this is the name of the worksheet for which you want to see the values. When you create a new spreadsheet, it is Sheet1)

Also, ensure that the env variables GCLOUD_PROJECT and GOOGLE_APPLICATION_CREDENTIALS are set properly.

Now, from the terminal, run this script

node test.js <your google sheet's spreadsheet id> <sheet name of the worksheet>

If you have followed all the steps correctly, you should see output like this:

output for getSpreadSheet {
  "spreadsheetId": "1-jG5jSgGTwXXXXXXXXXXXXXXXXXXY",
  "properties": {
    "title": "test-sheet",
    "locale": "en_US",
    "autoRecalc": "ON_CHANGE",
    "timeZone": "Asia/Calcutta",
    "defaultFormat": {
      "backgroundColor": {
        "red": 1,
        "green": 1,
        "blue": 1
      },
      "padding": {
        "top": 2,
        "right": 3,
        "bottom": 2,
        "left": 3
      },
      "verticalAlignment": "BOTTOM",
      "wrapStrategy": "OVERFLOW_CELL",
      "textFormat": {
        "foregroundColor": {},
        "fontFamily": "arial,sans,sans-serif",
        "fontSize": 10,
        "bold": false,
        "italic": false,
        "strikethrough": false,
        "underline": false
      }
    }
  },
  "sheets": [
    {
      "properties": {
        "sheetId": 0,
        "title": "Sheet1",
        "index": 0,
        "sheetType": "GRID",
        "gridProperties": {
          "rowCount": 1000,
          "columnCount": 26
        }
      }
    }
  ],
  "spreadsheetUrl": "https://docs.google.com/spreadsheets/d/1-jG5jSgGTwXXXXXXXXXXXXXXXXXXY/edit"
}
output for getSpreadSheetValues {
  "range": "Sheet1!A1:Z1000",
  "majorDimension": "ROWS",
  "values": [
    [
      "Name",
      "Country",
      "Age"
    ],
    [
      "John",
      "England",
      "30"
    ],
    [
      "Jane",
      "Scotland",
      "23"
    ],
    [
      "Bob",
      "USA",
      "45"
    ],
    [
      "Alice",
      "India",
      "33"
    ]
  ]
}

If you get an error, it means you have not followed all the steps correctly. For this tutorial, the version of googleapis npm module was 43.0.0. You might face issues if you are using older version of the module. Make sure the spreadsheetId and sheetname are correct and the enviroment variables are set properly. If you still get error, you should check the error message and code to see what might be causing the problem.

References

I would definitely recommend checking out these references (especially the Official Google Sheets API reference) to get a more in depth understanding of the sheets API and how to use the Node.js client.

Hope you found this tutorial helpful. Thanks and happy coding :-)