DEV Community: Richard Debrah

Automatically Delete Files From Amazon S3 Bucket With SubFolders Over A Duration Using Python

Richard Debrah — Mon, 10 Jun 2019 22:06:57 +0000

Amazon Web Services (AWS) is the world’s most comprehensive and broadly adopted cloud platform, offering over 165 fully featured services from data centers globally. Millions of customers —including the fastest-growing startups, largest enterprises, and leading government agencies—trust AWS to power their infrastructure, become more agile, and lower costs.

One of the numerous services Amazon provide is the Simple Storage Service popularly known as s3. Amazon S3 is a great way to store files for the short or for the long term. Many institutions depend greatly on Amazon s3(Cloudinary for example-last checked March, 2019) to store files ranging from log files in txt format to by uncle's half-sister's father's grandma's .gif photos that my future son put together. You get what I mean.

It is an object storage service that offers industry-leading scalability, data availability, security, and performance. This means institutions of all shapes and sizes can use it to store their data for all use cases. Amazon claims to offer competitive prices but that is not what I will be writing about.
After a while one will want to purge some if not all of the files stored on the Amason s3 for a number of reasons. These may include compliance-when an instititution is charged to retain certain data/files for a specified amount of duration, saving space-you are charged by amazon for the size of the files you have saved up on the s3 so you are better off deleting unused files, and just for the heck of it.

Why don't you just go unto the s3 console and delete the file you want or write a shell code to recursively remove those files? Well, good luck doing that when you have over 2 million records of files saved up over a long period of time. Maybe my son's uncle's auntie's friend can help you.

I was faced with the task of removing some files from my company's Amazon s3 bucket and below is how I went about it. With the help of one @DavidOlinsky who helped me clean it up. AWSCLI was not of much help as I needed control over what I delete and what I leave out. This code was written in python. My Assumption for a Bucket Structure is as follows with their file dates;

├── [ 160 Jun 10 15:05]  my_s3_bucket
│   ├── [ 128 Jun 10 15:05]  level-one-folder1
│   │   └── [ 608 Jun 10 15:18]  another-sub-folder
│   │       ├── [   0 Mar 26  2015]  file1.jpg
│   │       ├── [   0 Mar 26  2015]  file10.jpg
│   │       ├── [   0 Mar 26  2015]  file2.jpg
│   │       ├── [   0 Mar 26  2015]  file3.jpg
│   │       ├── [   0 Mar 26  2015]  file4.jpg
│   │       ├── [   0 Mar 26  2015]  file5.jpg
│   │       ├── [   0 Mar 26  2015]  file6.jpg
│   │       ├── [   0 Mar 26  2015]  file7.jpg
│   │       ├── [   0 Mar 26  2015]  file8.jpg
│   │       ├── [   0 Mar 26  2015]  file9.jpg
│   │       ├── [ 416 Jun 10 15:19]  folder-inside-sub-folder
│   │       │   ├── [   0 Mar 26  2010]  culp1.test
│   │       │   ├── [   0 Mar 26  2010]  culp10.test
│   │       │   ├── [   0 Mar 26  2010]  culp2.test
│   │       │   ├── [   0 Mar 26  2010]  culp3.test
│   │       │   ├── [   0 Mar 26  2010]  culp4.test
│   │       │   ├── [   0 Mar 26  2010]  culp5.test
│   │       │   ├── [   0 Mar 26  2010]  culp6.test
│   │       │   ├── [   0 Mar 26  2010]  culp7.test
│   │       │   ├── [   0 Mar 26  2010]  culp8.test
│   │       │   └── [   0 Mar 26  2010]  culp9.test
│   │       ├── [   0 Jun 10 15:17]  newer1.config
│   │       ├── [   0 Jun 10 15:17]  newer2.config
│   │       ├── [   0 Jun 10 15:17]  newer3.config
│   │       ├── [   0 Jun 10 15:17]  newer4.config
│   │       └── [   0 Jun 10 15:17]  newer5.config
│   └── [ 384 Jun 10 15:35]  level-one-folder2
│       ├── [   0 Mar 26  2005]  old1.txt
│       ├── [   0 Mar 26  2005]  old10.txt
│       ├── [   0 Mar 26  2005]  old2.txt
│       ├── [   0 Mar 26  2005]  old3.txt
│       ├── [   0 Mar 26  2005]  old4.txt
│       ├── [   0 Mar 26  2005]  old5.txt
│       ├── [   0 Mar 26  2005]  old6.txt
│       ├── [   0 Mar 26  2005]  old7.txt
│       ├── [   0 Mar 26  2005]  old8.txt
│       └── [   0 Mar 26  2005]  old9.txt

Requirements

boto3
time
sys

You can install boto3 by running pip install boto3 of you use pip or conda install boto3 or by any means that you are able to install python modules. Boto3 is amazon's own python library used to access their services. You can visit https://aws.amazon.com/ for all infortion regarding their libraries and swervices.

Imports

import boto3
import time
import sys

We are importing boto3 to be able to access our s3 services, time to help as a checkpoint for the duration from now (which could be any set time) and sys to generate error output to the terminal.

Let us set our variables. Note the placeholder values.

# todays\'s epoch
_tday = time.time()
duration = 86400*180 #180 days in epoch seconds
#checkpoint for deletion
_expire_limit = tday-duration
# initialize s3 client
s3_client = boto3.client('s3')
my_bucket = "my-s3-bucket"
my_ftp_key = "my-s3-key/"
_file_size = [] #just to keep track of the total savings in storage size

The above should be straight forward with the comments but in case you need more explanation you can continue reading this paragraph. You can look up the PEP 8 naming Style Guide For Python for the reasons for underscores in case you are curious. I am a student of this field so kindly bear with me on my errors.

We can set a particular current time for the our app to check from but for the sake of the majority I will just use _tday as the current date.

duration is the the UNIX Epoch time up to which the file will be checked to (any file older than that date will be removed). Here we are setting 180 days (six months) in UNIX Epoch seconds.

_expire_limit will just be the difference between the current date and the duration we need. Any file bearing lastModified date lesser that the value is older and will be removed.

s3_client will just be calling the low level Amazon s3 client for its methods/functions.

my_bucket will be the name of the bucket within which the files are contained. The files can be in subfolder(s) within the bucket. Don't worry, we will take care of that.

my_ftp_key will be the name of a particular subfolder within the bucket from which you want to search for the files. If you do not have subfolders you can ignore anything key that I mention (passed as Prefix).

We will use _file_size to hold the size in bytes of all the files that has been processed.

_del_size will reset continuously as the files delete and will help us determine the total size deleted even when an error occurs.

Next will be our functions.

Functions

#works to only get us key/file information
def get_key_info(bucket="my-s3-bucket", prefix="my-s3-key/"):

    print(f"Getting S3 Key Name, Size and LastModified from the Bucket: {bucket} with Prefix: {prefix}")

    key_names = []
    file_timestamp = []
    file_size = []
    kwargs = {"Bucket": bucket, "Prefix": prefix}
    while True:
        response = s3_client.list_objects_v2(**kwargs)
        for obj in response["Contents"]:
            # exclude directories/folder from results. Remove this if folders are to be removed too
            if "." in obj["Key"]:
                key_names.append(obj["Key"])
                file_timestamp.append(obj["LastModified"].timestamp())
                file_size.append(obj["Size"])
        try:
            kwargs["ContinuationToken"] = response["NextContinuationToken"]
        except KeyError:
            break

    key_info = {
        "key_path": key_names,
        "timestamp": file_timestamp,
        "size": file_size
    }
    print(f'All Keys in {bucket} with {prefix} Prefix found!')

    return key_info


# Check if date passed is older than date limit
def _check_expiration(key_date=_tday, limit=_expire_limit):
    if key_date < limit:
        return True


# connect to s3 and delete the file
def delete_s3_file(file_path, bucket=my_bucket):
    print(f"Deleting {file_path}")
    s3_client.delete_object(Bucket=bucket, Key=file_path)
    return True


# check size deleted
def _total_size_dltd(size):
    _file_size.append(size)
    _del_size = round(sum(_file_size)/1.049e+6, 2) #convert from bytes to mebibytes
    return _del_size

The get_key_info function takes in two parameters, a Bucket name and Prefix, all which will be passed to the s3 client method called list_objects_v2. This method takes in a couple of arguments one of which is the ContinuationToken. The list_objects_v2 method is only able to return a maximum of 1000 records pereach call. Amazon employes a Pagination method which returns to us a key for us to use to call the next set of 1000 records in case you have a lot more files to go through.

For the sake of clarity let us imagine having a book with a about 10 pages. Everytime you open a page you are able to see only what is on that particular page. You know there are more pages because on the bottom right there is a text that reads 2/10. Considering the lines on the pages continue from page 1 to 10 and each page bears 10 lines, we know that page 2 will start from 11 to 20 and page 5 will start from 41 to 50.

This is how list_objects_v2 works. Each time we make a call it returns us the next page's starting number so we can make the next call telling list_objects_v2 to start getting us the next set starting from that number.

This is achieved here in a while loop where after each call we try to reset the ContinuationToken until we hit a blocker which will be a KeyError.

It is important to check the reponse syntax to know what kind of information you will require from this object. In our care our file info lives within the "Contents" key.

We only returned a key_info object since that function's job is to only return us the relevant info we need from the call.

_check_expiration only takes in two parameters. if nothing is passed to it then it will use the default values from the variables above. It only returns True for each file when the LastModified date on s3 is older than the set duration.

delete_s3_file takes in the file_path which is the path of the file on s3 starting from the Key(prefix). of no bucket name is passed it will default to my_bucket.

_total_size_dltd keeps track of the storage sizes of the deleted files in MB. A simple Google Conversion Tool can help. At any point in time if there is any error if a file was deleted we will be presented with the total size deleted.

Finally we need to run our app. This app can be called/referenced from any other project that you have as long as you understand what each part is about. We will start by running our saved python file directly so the following code;

if __name__ == "__main__":
    try:
        s3_file = get_key_info()
        for i, fs in enumerate(s3_file["timestamp"]):
            file_expired = _check_expiration(fs)
            if file_expired: #if True is recieved
                file_deleted = delete_s3_file(s3_file["key_path"][i])
                if file_deleted: #if file is deleted
                    _del_size = _total_size_dltd(s3_file["size"][i])

        print(f"Total File(s) Size Deleted: {_del_size} MB")
    except:
        print ("failure:", sys.exc_info()[1])
        print(f"Total File(s) Size Deleted: {_del_size} MB")

In case your bucket name is my_bucket and your prefix or key is my_ftp_key then running this without passing any parameter will run through the folder my_ftp_key inside of my_bucket and remove any file older than 180 days from the time you run the app.

I will now demonstrate how to remove the old files from our assummed folder structure.

If we decide to remove older files inside folder-inside-sub-folder only this will be our call.

if __name__ == "__main__":
    try:
        # the difference is right here
        s3_file = get_key_info("my_s3_bucket", "folder-inside-sub-folder")
        for i, fs in enumerate(s3_file["timestamp"]):
            # you can pass the duration in epoch as second parameter
            file_expired = _check_expiration(fs)
            if file_expired:
                file_deleted = delete_s3_file(s3_file["key_path"][i])
                if file_deleted:
                    _del_size = _total_size_dltd(s3_file["size"][i])

        print(f"Total File(s) Size Deleted: {_del_size} MB")
    except:
        print ("failure:", sys.exc_info()[1])
        print(f"Total File(s) Size Deleted: {_del_size} MB")

If we want to remove all old files from level-one-folder1 our call will be

if __name__ == "__main__":
    try:
        # the difference is right here
        s3_file = get_key_info("my_s3_bucket", "level-one-folder1")
        for i, fs in enumerate(s3_file["timestamp"]):
            # you can pass the duration in epoch as second parameter
            file_expired = _check_expiration(fs)
            if file_expired:
                file_deleted = delete_s3_file(s3_file["key_path"][i])
                if file_deleted:
                    _del_size = _total_size_dltd(s3_file["size"][i])

        print(f"Total File(s) Size Deleted: {_del_size} MB")
    except:
        print ("failure:", sys.exc_info()[1])
        print(f"Total File(s) Size Deleted: {_del_size} MB")

This will remove all older files inside of another-sub-folder as well as folder-inside-sub-folder since they are inside of level-one-folder1. However, if we are checking by 180 days older files, then the files newer1.config to newer5.config inside of another-sub-folder will not be touched as they do not pass the expired test.

I am sure you can think of many other ways you can use this code to call deletions on your s3 bucket based on a duration, whether from a process script using argv or setting a monthly automatic process to run maintenance. Below is a summary

import boto3
import time
import sys

# todays\'s epoch
_tday = time.time()
duration = 86400*180 #180 days in epoch seconds
#checkpoint for deletion
_expire_limit = tday-duration
# initialize s3 client
s3_client = boto3.client('s3')
my_bucket = "my-s3-bucket"
my_ftp_key = "my-s3-key/"
_file_size = [] #just to keep track of the total savings in storage size

#Functions
#works to only get us key/file information
def get_key_info(bucket="my-s3-bucket", prefix="my-s3-key/"):

    print(f"Getting S3 Key Name, Size and LastModified from the Bucket: {bucket} with Prefix: {prefix}")

    key_names = []
    file_timestamp = []
    file_size = []
    kwargs = {"Bucket": bucket, "Prefix": prefix}
    while True:
        response = s3_client.list_objects_v2(**kwargs)
        for obj in response["Contents"]:
            # exclude directories/folder from results. Remove this if folders are to be removed too
            if "." in obj["Key"]:
                key_names.append(obj["Key"])
                file_timestamp.append(obj["LastModified"].timestamp())
                file_size.append(obj["Size"])
        try:
            kwargs["ContinuationToken"] = response["NextContinuationToken"]
        except KeyError:
            break

    key_info = {
        "key_path": key_names,
        "timestamp": file_timestamp,
        "size": file_size
    }
    print(f'All Keys in {bucket} with {prefix} Prefix found!')

    return key_info


# Check if date passed is older than date limit
def _check_expiration(key_date=_tday, limit=_expire_limit):
    if key_date < limit:
        return True


# connect to s3 and delete the file
def delete_s3_file(file_path, bucket=my_bucket):
    print(f"Deleting {file_path}")
    s3_client.delete_object(Bucket=bucket, Key=file_path)
    return True


# check size deleted
def _total_size_dltd(size):
    _file_size.append(size)
    _del_size = round(sum(_file_size)/1.049e+6, 2) #convert from bytes to mebibytes
    return _del_size


if __name__ == "__main__":
    try:
        s3_file = get_key_info()
        for i, fs in enumerate(s3_file["timestamp"]):
            file_expired = _check_expiration(fs)
            if file_expired: #if True is recieved
                file_deleted = delete_s3_file(s3_file["key_path"][i])
                if file_deleted: #if file is deleted
                    _del_size = _total_size_dltd(s3_file["size"][i])

        print(f"Total File(s) Size Deleted: {_del_size} MB")
    except:
        print ("failure:", sys.exc_info()[1])
        print(f"Total File(s) Size Deleted: {_del_size} MB")

I hope i have been able to help you solve a problem you may be having. This tutorial is incomplete unless you critique it as it makes be work hard to be better at what i do. Leave comments if you have to. You are free to copy/replicate the above for your use but know that you use it at your own risk. I am not responsible to any damages whatsoever that may arise from your use either to you or a third party.

Building a NodeJS Web App Using PassportJS for Authentication

Richard Debrah — Thu, 20 Dec 2018 22:02:20 +0000

Updated!!!
This tutorial uses PassportJS to authenticate the NodeJS App with MySQL Database Management Software (DBMS). The reason for this article is just simple. While i was studying web development i was faced with a challenge when i was working on my second project to integrate PassportJS Authentication into my app. At that time i was using MySQL for database management, SequelizeJS which is an Object Relation Mapper(ORM) for SQL-based databases, in this case MySQL, ExpressJS middleware, Body Parser and Express Session for server and session management. The difficulty was that i could only find tutorials that used Handlebars as the ORM and MongoDB as the DBMS which at that time, i was not that familiar with so if you find yourself at this crossroad this is for you. I will not waste your time but dive into it right away. I will be as literal as i can possibly be so even the least knowledged in web development can understand. We will need to make a few things ready for this setup. I am using a Windows Pc so please find a workaround if anything i say does not work for your OS especially my recommendations but the process is the same i believe.

First of all you will need to have a computer with your favorite text editor (I used VS Code), a web browser (I recommend Google Chrome), your favorite terminal (Git Bash recommended) and a SQL DBMS of your choice. I am using MySQL Workbench 6.3 CE. I will go through the steps in a list so it is easy to follow. When you have all the above, and properly set up, follow the instructions below. I am assuming you already have your database created, if not I will take you through.

Create a folder on your computer in any location of your choice. I prefer to navigate to my preferred location using my terminal and then typing mkdir nameOfProject. I will create the folder learningPassportJS on my desktop for this tutorial. Next is to type cd nameOfProject to navigate into the project folder.

In the terminal we will have to initialize the folder to handle all our NodeJS framework. You can decide to do this later on but i recommend you doing this first if you are new to this process. Do this by typing npm init and press enter. This sets up your project with a package.json file. This file will contain the information including all the expected dependencies and licenses as well as your name. For our purpose i will just keep pressing enter on my keyboard to load defaults but I will set the entry point as server.js. Feel free to change it to what pleases you. Please make sure you have lowercase letters in your texts or you will have to type them yourself.

After we have initialized our project we will create our server.js file with touch server.js on the terminal.
Let us now install all the dependencies we will require. I will explain why we need each one later but i like to install all to get it out of the way. You can always install them later but you need them to run the app successfully. Install them in one line as npm i --save sequelize passport passport-local mysql2 mysql express express-session body-parser bcrypt-nodejs or you can choose to install them individually as
npm i --save sequelize
npm i --save passport
npm i --save passport-local
npm i --save mysql2
npm i --save mysql
npm i --save express
npm i --save express-session
npm i --save body-parser
npm i --save bcryptjs

Adding --save makes sure your dependency is added and saved to your package.json file. This is important if you want to deploy this app. You will realize a new folder called node_modules. Do not touch this. This is what node uses to run the app locally on your computer. If you are using Git for your project do not forget to add node_modules to your .gitignore file in your project's root folder.

Thanks to the contribution from Jordan White, I think it is worth to mention that you must have Sequelize CLI installed before you can use sequelize. Do this by running npm install -g sequelize-cli from your preferred terminal to install it globally or you can remove -g to have it installed locally.

Open the server.js file created in your favorite terminal and input a few lines of code in our server.js file. Don't worry, I will have a horse-load of comments on all the codes so it is easy to understand why i wrote each line. You can copy the code below into your server file.

// Requiring necessary npm middleware packages 
var express = require("express");
var bodyParser = require("body-parser");
var session = require("express-session");
// Setting up port
var PORT = process.env.PORT || 8080;
// Creating express app and configuring middleware 
//needed to read through our public folder
var app = express();
app.use(bodyParser.urlencoded({ extended: false })); //For body parser
app.use(bodyParser.json());
app.use(express.static("public"));
//
//we are doing a GET to test if our server is working fine
app.get('/', function(req, res) {    
       res.send('Welcome to Passport with Sequelize and without HandleBars');
});
//
//this will listen to and show all activities on our terminal to 
//let us know what is happening in our app
app.listen(PORT, function() {
    console.log("App listening on PORT " + PORT);
  });

Save the server file. Let us run the server to make sure it is working properly. Do this by typing npm start or node server.js in your terminal. You remember the entry point when we run npm init? This is what is called when you run npm start.

If you followed the instruction well up to this point you should see the following

Open your browser and enter the location localhost:8080. This will display Welcome to Passport with Sequelize and without HandleBars. Great Job!! on getting this far. You are on your way to creating your app. If you don't see that page look up the steps from the beginning. You can end the server and go back to your code.

I assumed from the beginning you might have already created your database. If you have not and or do not know how to go about this do not worry. Just open your MySQL program of choice and from the query shell enter CREATE DATABASE passport_demo; and run it. You should have a database created with name passport_demo.
Now that we have our server and database working it is time to add the other parts. We will configure and initialize our sequelize module. Do this by typing sequelize init:models & sequelize init:config on your terminal and press enter.
After this code runs, you should see two folders models and config.
Open the config folder and you should see a config.json file. Open it and edit the development object's settings to match yours. If you have a password on your database enter it here in quotes. Example is below

{
  "development": {
    "username": "root",
    "password": "yourpassword",
    "database": "passport_demo",
    "host": "127.0.0.1",
    "dialect": "mysql"
  },
  "test": {
    "username": "root",
    "password": null,
    "database": "database_test",
    "host": "127.0.0.1",
    "dialect": "mysql"
  },
  "production": {
    "username": "root",
    "password": null,
    "database": "database_production",
    "host": "127.0.0.1",
    "dialect": "mysql"
  }
}

Navigate back and open the models folder. You should see an index.js file. This should be unchanged for our tutorial but if you have your config folder in a different location you can open it and edit Line 8 from Col 37 to route to your location because it will need the config.json file to work.Some Windows PCs will also throw an error that it couldn't find the config module. change the backslashes on that to forward slashes to fix that error.

In the models folder create a new file called user.js. This is going to insert our user information to the database using sequelize. You can have multiple model files depending on your needs. The models folder should contain the various table inserts that you make in the database.In this tutorial we want a user model. We will require the bcryptjs package to encrypt and decrypt the password that the user creates or logs in with. Your user.js file should look like this

// Requiring bcrypt for password hashing. Using the bcryptjs version as 
//the regular bcrypt module sometimes causes errors on Windows machines
var bcrypt = require("bcryptjs");
//
// Creating our User model
//Set it as export because we will need it required on the server
module.exports = function(sequelize, DataTypes) {
  var User = sequelize.define("User", {
    // The email cannot be null, and must be a proper email before creation
    email: {
      type: DataTypes.STRING,
      allowNull: false,
      unique: true,
      validate: {
        isEmail: true
      }
    },
    // The password cannot be null
    password: {
      type: DataTypes.STRING,
      allowNull: false
    }
  });
  // Creating a custom method for our User model. 
  //This will check if an unhashed password entered by the 
  //user can be compared to the hashed password stored in our database
  User.prototype.validPassword = function(password) {
    return bcrypt.compareSync(password, this.password);
  };
  // Hooks are automatic methods that run during various phases of the User Model lifecycle
  // In this case, before a User is created, we will automatically hash their password

  User.hook("beforeCreate", function(user) {
    user.password = bcrypt.hashSync(user.password, bcrypt.genSaltSync(10), null);
  });
  return User;
};

//This is a fix by Samaila Philemon Bala in case you want to use ES6
//and the above is not working

//User.beforeCreate(user => {
  //  user.password = bcrypt.hashSync(
    //  user.password,
      //bcrypt.genSaltSync(10),
      //null
    //);
  //});

Let us go back to our server.js file and add a few lines of code. We will need to require the server to read the models folder and we will also need it to sync our inserts and reads to the database. Your server.js should look like this

// Requiring necessary npm middleware packages 
var express = require("express");
var bodyParser = require("body-parser");
var session = require("express-session");
// Setting up port
var PORT = process.env.PORT || 8080;
//Import the models folder
var db = require("./models");
//
// Creating express app and configuring middleware 
//needed to read through our public folder
var app = express();
app.use(bodyParser.urlencoded({ extended: false })); //For body parser
app.use(bodyParser.json());
app.use(express.static("public"));
//
//we are doing a GET to test if our server is working fine
app.get('/', function(req, res) {    
       res.send('Welcome to Passport with Sequelize and without HandleBars');
});
//
//this will listen to and show all activities on our terminal to 
//let us know what is happening in our app
// Syncing our database and logging a message to the user upon success
db.sequelize.sync().then(function() {
  app.listen(PORT, function() {
    console.log("==> 🌎  Listening on port %s. Visit http://localhost:%s/ in your browser.", PORT, PORT);
  });
});

Now let us navigate to the config folder and create another folder called middleware and inside that folder create a file called isAuthenticated.js. You should have /config/middleware/isAuthenticated.js. Open and edit the isAuthenticated.js file to match this

// This is middleware for restricting routes a user is not allowed to visit if not logged in
module.exports = function(req, res, next) {
  // If the user is logged in, continue with the request to the restricted route
  if (req.user) {
    return next();
  }
  // If the user isn't' logged in, redirect them to the login page
  return res.redirect("/");
};

This will be exported also and we will need this to restrict access to pages meant for logged in users only.

It is time to set up passport. In the config folder create a file called passport.js. Open the file and input the following in the file. The comments explain it all.

//we import passport packages required for authentication
var passport = require("passport");
var LocalStrategy = require("passport-local").Strategy;
//
//We will need the models folder to check passport agains
var db = require("../models");
//
// Telling passport we want to use a Local Strategy. In other words,
//we want login with a username/email and password
passport.use(new LocalStrategy(
  // Our user will sign in using an email, rather than a "username"
  {
    usernameField: "email"
  },
  function(email, password, done) {
    // When a user tries to sign in this code runs
    db.User.findOne({
      where: {
        email: email
      }
    }).then(function(dbUser) {
      // If there's no user with the given email
      if (!dbUser) {
        return done(null, false, {
          message: "Incorrect email."
        });
      }
      // If there is a user with the given email, but the password the user gives us is incorrect
      else if (!dbUser.validPassword(password)) {
        return done(null, false, {
          message: "Incorrect password."
        });
      }
      // If none of the above, return the user
      return done(null, dbUser);
    });
  }
));
//
// In order to help keep authentication state across HTTP requests,
// Sequelize needs to serialize and deserialize the user
// Just consider this part boilerplate needed to make it all work
passport.serializeUser(function(user, cb) {
  cb(null, user);
});
//
passport.deserializeUser(function(obj, cb) {
  cb(null, obj);
});
//
// Exporting our configured passport
module.exports = passport;

For our app to work as expected we need to be able to GET and POST to our database. Example of this is the app.get code block we placed in the server.js file. Let us write a clean code. Create a folder in your root folder called routes and also create two files called api-routes.js and html-routes.js. The api-routes.js will be used to route GET and POST from and to the database. open the api-routes.js and paste the following. The comments explain it all.

// Requiring our models and passport as we've configured it
var db = require("../models");
var passport = require("../config/passport");
//
module.exports = function(app) {
  // Using the passport.authenticate middleware with our local strategy.
  // If the user has valid login credentials, send them to the members page.
  // Otherwise the user will be sent an error
  app.post("/api/login", passport.authenticate("local"), function(req, res) {
    // Since we're doing a POST with javascript, we can't actually redirect that post into a GET request
    // So we're sending the user back the route to the members page because the redirect will happen on the front end
    // They won't get this or even be able to access this page if they aren't authed
    res.json("/members");
  });
//
  // Route for signing up a user. The user's password is automatically hashed and stored securely thanks to
  // how we configured our Sequelize User Model. If the user is created successfully, proceed to log the user in,
  // otherwise send back an error
  app.post("/api/signup", function(req, res) {
    console.log(req.body);
    db.User.create({
      email: req.body.email,
      password: req.body.password
    }).then(function() {
      res.redirect(307, "/api/login");
    }).catch(function(err) {
      console.log(err);
      res.json(err);
      // res.status(422).json(err.errors[0].message);
    });
  });
//
  // Route for logging user out
  app.get("/logout", function(req, res) {
    req.logout();
    res.redirect("/");
  });
//
  // Route for getting some data about our user to be used client side
  app.get("/api/user_data", function(req, res) {
    if (!req.user) {
      // The user is not logged in, send back an empty object
      res.json({});
    }
    else {
      // Otherwise send back the user's email and id
      // Sending back a password, even a hashed password, isn't a good idea
      res.json({
        email: req.user.email,
        id: req.user.id
      });
    }
  });
};

Let us leave the html-routes.js for now. We will come back to it. We will need it to handle login and serve our pages.

On the server.js file we will need to import and initialize passport. Make sure your express is initialized before passport as passport requires express. The markup in the server is very important. I will also remove the app.get code block since we don't need it. Your server file should look like the below

// Requiring necessary npm packages
var express = require("express");
var bodyParser = require("body-parser");
var session = require("express-session");
// Requiring passport as we've configured it
var passport = require("./config/passport");
//
// Setting up port and requiring models for syncing
var PORT = process.env.PORT || 8080;
var db = require("./models");
//
// Creating express app and configuring middleware needed for authentication
var app = express();
app.use(bodyParser.urlencoded({ extended: false }));
app.use(bodyParser.json());
app.use(express.static("public"));
// We need to use sessions to keep track of our user's login status
app.use(session({ secret: "keyboard cat", resave: true, saveUninitialized: true }));
app.use(passport.initialize());
app.use(passport.session());
//
// Requiring our routes
require("./routes/html-routes.js")(app);
require("./routes/api-routes.js")(app);
//
// Syncing our database and logging a message to the user upon success
db.sequelize.sync().then(function() {
  app.listen(PORT, function() {
    console.log("==> 🌎  Listening on port %s. Visit http://localhost:%s/ in your browser.", PORT, PORT);
  });
});

Note that we are requiring html-routes also. The Next step is to create our user interface (UI) to be able to capture the user's information for signing in and or logging in. This will be the regular html files we usually create with its css and js files but this time it will be placed in a public folder. This is the folder express will use and parse from. If you are conversant with POSTMAN you can test the server using it at this point.

I created a sample public folder with its files in it. which is what i will be using for this tutorial. Download it from Mediafire and unzip into the root folder.
Take a look at the html files in the public folder. You will see that I captured the signup, login, and members page GETs using APIs. This way we can pass it to the server with ease.
Now open the html-routes.js and paste the following code

// Requiring path to so we can use relative routes to our HTML files
var path = require("path");
//
// Requiring our custom middleware for checking if a user is logged in
var isAuthenticated = require("../config/middleware/isAuthenticated");
//
module.exports = function(app) {
//
  app.get("/", function(req, res) {
    // If the user already has an account send them to the members page
    if (req.user) {
      res.redirect("/members");
    }
    res.sendFile(path.join(__dirname, "../public/signup.html"));
  });
//
  app.get("/login", function(req, res) {
    // If the user already has an account send them to the members page
    if (req.user) {
      res.redirect("/members");
    }
    res.sendFile(path.join(__dirname, "../public/login.html"));
  });
//
  // Here we've add our isAuthenticated middleware to this route.
  // If a user who is not logged in tries to access this route they will be 
  //redirected to the signup page
  app.get("/members", isAuthenticated, function(req, res) {
    res.sendFile(path.join(__dirname, "../public/members.html"));
  });
};

Save all files and then run the server with npm start or node server.js. If your server fails check the error on your terminal and go over everything from this tutorial. Take a look at */public/js/members.js* and you will be able to understand what you need in order to get the user information. Sequelize automatically creates an id for a user so you can use it to handle associations on other tables in the database.
If for example you are looking to authenticate on a classified website all you will have to make sure is to have a model such as posting to insert into database, api route to do the post, html route to GET page before and after Post to the database. Always make sure you pass isAuthenticated on any html route that you do not want the user to access without logging in. isAuthenticated will always check for access if used.

I hope my very literal explanation is able to help you figure out your authentication without the use of handlebars or MongoDB.

Richard Debrah