Last week's blog post I mentioned "TGIF", this week I'd like to introduce "PGMF" - otherwise known now as "Please Give Me Friday!" I think that probably serves as a hint towards how this week has been, but before that...
Good morning, good afternoon & good evening, and thank you once again for taking the time to check up on me and my journey at Northcoders
Like always, feel free to drop me a follow on Twitter, LinkedIn and GitHub!
Alright...So what's the deal with week 6?
With a little bit of a preface from this week's blog's opening statement...This week was up until now the most difficult and intense part of the course, even so much so that the mentors who were previous students themselves had something to say during the middle of the week.
Our fantastic lead mentor Christian and mentors Scarlett & Heather shared their personal stories about how this week especially was by far the toughest they had to overcome. Christian ensured us that it's perfectly fine for us to struggle as long as we weren't struggling in silence as he eluded to his own struggles with the seeding of the database for the Thursday & Friday sprint.
All-in-all, Northcoders and the mentors themselves are very aware that this week was difficult and made sure to regularly check-in on us to make sure that we were getting along fine.
But, even with that said. I will admit that this week has definitely been the most overwhelming because up until now, we had most of the file structure given to us and didn't have to worry about handling errors when we were using Express. I had thought I had a pretty decent grasp of Express until we embarked from the "Happy Path" of successful GET requests to the dark territories of error handling and seeding a database using Node-Postgres.
And that was when the fire nation struck realisation I had that the "error handling" and "seeding of databases" was not as simple I had envisioned it to be, but was instead an entirely different beast altogether.
So with that said, the sheer amount of content covered this week makes it difficult for me to provide concrete examples, but instead I will try to explain what we had managed to get covered this week in brief summaries.
Without further ado, here is what we were up to this week:
- Setting up queries to the PSQL Database & preventing SQL Injections
- Introduction to
dotenvenvironment variables - Introduction to Testing & Development Databases
- Introduction to Integration Testing the API Endpoints with Jest's Supertest
- Express Middleware Functions & Error Middleware Functions
- Introduction to Seeding & Seeding with Node Postgres
Node Postgres
Node-Postgres is a way for Node to interact with a PostreSQL database through callbacks/promises/async & await.
With Node Postgres, you have two main methods of connecting:
- 1 - Client - This is used when you wish to create a single instance that will only have that single channel of communication.
- 2 - Pool - This is used when you wish to open up multiple channels of communication - which is useful when you have multiple users that are querying independent things.
With Node Postgres, we will require a bit of setup using "environment variables" to securely connect to the databases because it would be bad practice if the environment variables were stored directly on a project's sourcecode.
ENV - Environment Variables
Environment variables (ENV) are default connection parameters that PSQL will use to login, authenticate and connect to the databases that you have on your localhost machine or an endpoint.
PGDATABASE is used to specify the database that you are trying to access, and you're able to store your environment variables in any means you wish. For security reasons it is not advised to keep environment variables on-file, but for testing purposes they will be kept here.
And as such, if you have environment files in your local repository, it is advised to add .env.* to your .gitignore file so that they are not accidentally uploaded to GitHub!
For the following Client & Pool connections, we setup PGDATABASE manually;
The naming convention of the environment variables for each database will be something similar to: .env.development and .env.test which will all begin with .env
PGDATABASE = my_database_name
However before doing so, we will need to download & require in the dontenv package that will give us access to these environment files that we can use!
Connecting Node-Postgres W/ Environment Variables
Before we hook up Node-Postgres to our PSQL database, we need to understand the two methods that Node-Postgres is able to connect to our database.
The first is via a "Client" which will mostly be accepting a single connection at a time, which is not ideal for us if we're emulating multiple clients making requests and as such, the second option of a "Pool" of connections is what we will be using to connect to our database.
The general configuration for a connection.js file that contains this would look like this:
const { Pool } = require('pg');
const ENV = process.env.NODE_ENV || 'development';
require('dotenv').config({
path: `${__dirname}/../.env.${ENV}`,
});
if (!process.env.PGDATABASE) {
throw new Error('PGDATABASE not set');
}
module.exports = new Pool();
We would first require in the "Pool" object that has methods that will allow us to connect to our designated database later on by requiring in the Node-Postgres package that we'd installed.
Secondly, we declare an "ENV" variable that will set the type of environment that we will be working in (Discussed below) whether this is the "development environment" or "test environment".
Third, we require in the dotenv package & configure the path for dotenv to navigate to and assign the database we'll be connecting to the specific database specified in the .env file.
Lastly, if PGDATABASE does not exist then an error will be thrown.
Types of Databases & Integration Testing
Previously we had been using Jest for TDD and had been testing every single function individually to ensure that they were working as intended, and this was referred to as "unit testing."
This week, we were introduced to the concept of "Integration Testing" which is where we aim to test software modules were integrating and working logically as a group, which was pretty tricky to get a proper idea of until we'd incorporated tess for our API's endpoints.
Following this, we were introduced to the four different types of databases that each serve a purpose:
- Development Database - This is a database that will need to persist and not reset on every "test" being made using Jest's Supertest. A development database will often have a sheer volume of data vastly contrasting to the test database.
- Test Database - This is a database that will not persist and needs to be reset on every single test. The volume of data is significantly lower than the development database because otherwise it'd affect how frequent tests can be completed.
- Staging Database - This is a database that test engineers will use to give feedback.
- Production Database - This is the live database that will persist and the one users will be using, and as such this is essentially the final product.
It is important to note, all of the data across databases will need to have the same schema and look exactly the same besides the volume of data.
Express Middleware Functions, Error Handling & Error Middleware Functions
Last week when were introduced to Express and the concept of "Middleware", we had it drilled into our head that anytime you hear "Middleware", it means that something will be dealing with the "Request" and "Response".
This holds true for what we covered this week about the Next parameter in Express' Middleware Function, and the existence of "Error Middleware Functions."
So to visualise this, last week a standard Middleware Function that existed in the Controllers of an MVC would be something like this:
const controllerMiddlewareFunc = (req, res) => {
invokeModelFunction()
.then(() => do something if successful)
.catch((err => {
console.log(err);
}))
}
But this week, we were introduced to "Error Handling" - which is as the name sounds, how we end up handling errors that come back along the line from our code.
The example below shows you how and where exactly the "next" parameter is invoked if an error occurs:
const controllerMiddlewareFunc = (req, res, next) => {
invokeModelFunction()
.then(() => do something if successful)
.catch((err) => {
next(err);
}
}
So here, when an error comes back from the "Model" function then this error will be caught in this .catch() block, and essentially the invocation of the "Next" function will mean that it will be passed into the main app.js that will then handle that specific error.
Error Middleware Functions
A bit more of a specific topic is that of "Error Middleware Functions." This is a function that we use to essentially create a function that will handle in errors, and is characterised by having the four parameters of: err, req, res, next - typically we would also have a custom object sent back to the user that will have a status code & message for example:
//app.js
app.use((err, req, res, next) => {
if(err.status && err.msg) {
res.status(err.status).send(err.msg);
}
else {
next(err)
}
}
//model.js
if (!array.includes(item)) {
return Promise.reject({ status: 400, msg: "Bad Request" })
}
Here in this example where we've created a reusable custom error middleware function that will check if when it receives an error if it has received an object that has the property names of status and msg, and if it does then it will go ahead and send a response back to the user with that status & message.
If it does not satisfy that certain error middleware function's conditional statement, then it would use the "next" function and pass it to the next error middleware function in the chain until it finds a condition to satisfy it.
If it does not satisfy any of the error handling middleware functions that we had created, then it will simply return a "Status:500 - Server Error" to the user because it cannot find the correct error middleware function!
Introduction to Seeding & Seeding w/ Node-Postgres
I'll be honest with you, the concept of "Seeding" and what it had meant went over my head until Jim, the leader of the 30th August 2022 cohort had used the anecdote of it being the same "Seeding" as you would have in games like "Minecraft."
So "Seeding" is basically generating data and inserting that data into the database for a developer to work on.
Previously when we had covered SQL/PSQL we could manually create an insert statement for all of the information to the database - however, we are Software Developers and so we aren't going to sit there for countless hours and insert all of this data.
Instead, we are going to use Node-Postgres and create a function that would allow us to seed all of the information that was provided to us in the sprint, into the database for us to be able to work on creating endpoints for us to retrieve data!
I would have ended up writing an entire disseration on Seeding & Seeding with Node-Postgres if I tried to explain how all of this was working under the hood the past two-days, but I don't think that would be a great idea 😂
But hopefully, this week's small summary of what we had gotten up to and how intense this week was, was properly conveyed to you all!
Thoughts Going Forward
Honestly I'm really chuffed that myself and the rest of the people in my cohort managed to brave the storm this week and hopefully made it all in one piece!
A few roadbumps personally this week but they were all addressed by the supportive staff here at Northcoders that will take any issues that you have seriously to create a real safespace for every student so that they can feel like they don't have to be anxious at any stage.
So a really, really huge shout out to Northcoders, the mentors and my personal mentor Alex, you're all doing a fantastic job - and that includes you Christian, this may be your first time leading a seminar group but I would have thought you had lead multiple seminars if you didn't tell us this week!
For now, I'm just looking to brush up and internalise what we leaarned this week and prepare myself for next week's week-long solo project for the final week of the back end phase, and I hope to see you all there next week!
Socials
Feel free to give me a follow on any of my socials down below!
Top comments (0)