DEV Community: Edwin

A Web Scraper using Golang.

Edwin — Tue, 28 Jul 2020 04:38:08 +0000

Web Scraper

Go is has increasingly become popular due to its flexibility amd power.
So today,we build a simple web scraper using golang.
We make a GET request to the website.

package main
import(
        "fmt"
        "net/http"
        "io/ioutil"
)

func main(){
       //Make GET request
       resp,err := http.Get("www.example.com")
       Check(err)

       //close the connection stream
       defer resp.Body.Close()

       //get the bytes acquired
       resultsInBytes,err := ioutil.ReadAll(resp.Body)
       Check(err)

       //convert into a readable format
       results := string(resultsInBytes)

       //Display website
       fmt.Println(results)
}

//error handler function
func Check(err error){
       if err!= nil{
            fmt.Println(err)
       }
}

We can make it powerful in a way that it can store the acquired website in a file but we will use the io module to copy the website to the file,instead of ioutil module.

package main
import(
        "fmt"
        "net/http"
        "io"
        "os"
)

func main(){
       //create file
       f := os.Create("filename.html")

       //Make GET request
       response,err := http.Get("www.example.com")
       Check(err)

       //close the connection stream
       defer resp.Body.Close()

       //get the bytes acquired
       resultsInBytes,err := io.Copy(f,resp.Body)
       Check(err)

       //convert into a readable format
       results := string(resultsInBytes)

       //Display website
       fmt.Println(results)
}

//error handler function
func Check(err error){
       if err!= nil{
            fmt.Println(err)
       }
}

MongoDB for Express.js

Edwin — Sat, 29 Feb 2020 20:12:36 +0000

Persistence.
Persistence in nodejs basically means a way of storing data.It can be in any of the 3 ways:

FileSystem
Cloud
Database

Filesystem is not secure at all when it comes to deploying your web app to the online world.

Cloud persistance is becoming more popular.I would basically recommend Mongo Atlas (what is been called Mongo on the cloud).It is more secure and inexpensive.It's free for simple applications.

Database persistence has been there for years,mainly on relational databases such as MySQL ,Oracles,Postgres etc.NoSQL databases such as mongodb,cassandra and couch are currently challenging RDBMS.Mongodb is more popular that's why it is a good start.

Create a models folder.Inside it ,create a users.js file.Now ,we will have to install an Object document mapper for mongodb.In this case,it's mongoose.Install it:

npm install --save mongoose

Mongoose introduces schemas and models.When combined ,it is similar to object-oriented programming.

const mongoose = require("mongoose");

//create a schema
var userSchema = mongoose.Schema({
      name: String,
      email:String,
      password : String,
      date:Date,
});

//create a model
var User = mongoose.model("User",userSchema);

module.exports = User;

Now let's head over to our main app.js and introduce our database and save each item(name & password) to the database.

//import the express module
const express = require("express");
const app = express();
//body-parser middleware
app.use(require("body-parser")());

//link mongoose and connect
const mongoose = require("mongoose");
mongoose.connect('mongodb://localhost/webApp', {useNewUrlParser: true});

//link the user model
var User = require("./model/user");



//set up handlebars
const handlebars = require('express3- handlebars')
              .create({defaultLayout:'main'});
app.engine('handlebars',handlebars.engine);
app.set('view engine','handlebars');

//set the port number to listen on
app.set('port',process.env.PORT || 8080);

//our slash route will render the home view
app.get('/',(req,res)=>{
    res.render('home');
});

//get user input and display it
app.post("/process",(req,res)=>{
        const name = req.body.name;
        const email= req.body.email;
        const password = req.body.password;

        User = new User({
            name:name,
            email:email,
            password:password
        });
        User.save(user,err)
        .then(user=>console.log('saved'))
        .catch(err=> console.log(err));
});

//our about route will render the about view
app.get('/about',(req,res)=>{
    res.render('about');
});

//create a callback for listening on the server
app.listen('port',()=>{
    console.log(`Listening on port ${app.get('port')}.Press CTRL Z to cancel.`);
}):

Done!
As you have seen,I haved the user and logged that in my console,but what happens thereafter is redirect the user to the login page if you have one.
The new user's details have been saved.Great day all!

Middlewares in Express.js

Edwin — Sat, 29 Feb 2020 19:56:01 +0000

A middleware is a way of enclosing functionality,that operates on a HTTP to your application.In practice,it actually takes three arguments:a request object,response object and a next function.

Sometimes it might contain four arguments,the fourth argument being a err object.

Let's write an example of a middleware and use it to understand the concept of middlewares.

app.use(function(req,res,next){
         console.log('This is a simple middleware.");
         next();
});

The middleware example above uses a different route handler(often called app.VERB).As you have noticed,it does not specify a path.You can add the /* path as a first parameter but this is option,since if omitted,it will still match on all paths.
The request and response objects are self-explanatory.My main concern is the next function.

Middleware in express is a matter of where they are put in the main app.js file.So what this middleware basically does is display a message,and then the next function heads over to the next route handler below this middleware.Middleware is executed in what is called a pipeline;basically following an order.

The immediate route handler following this middleware will be executed after the middleware is executed.

Let's create an error 404 middleware which will be executed incase the user visits a page that does not exist.

//custom 404 page
app.use(function(req,res,next){ 
        res.type("text/plain");
        res.status(404);
        res.send("Error : Page not found.");
});

This middleware will be put below all other routes.Incase the route being sought by the user is not found,the app will head over to this middleware and implement it.

Let's experiment with the most used middleware; body-parser middleware.It is used to parse the URL-encoded body.Let's say we have a form and want to capture what the user has put.The body-parser middleware avails the req.body object so as to access the form fields.We shall create a form now.

<!--form example-->
<form method="POST" action="process">
    <label for="fieldName">Full Name:</label>
    <input name="fullName" type="text">
    <label for="fieldPassword">Password:</label>
    <input name="password" type="password">
</form>

We install the body-parser middleware now

npm install --save body-parser

Our form is ready,now create a route handler for this form.

//we link the middleware to avail the req.body
app.use(require("body-parser")());

//get user input and display it
app.post("/process",(req,res)=>{
        const name = req.body.name;
        const password = req.body.password;

        console.log(name);
        console.log(password);
});

And that's all there is.

Note: With the latest Express version,body-parser is available even without installing it separately.Check out all about it here

Next,I will talk about persistence in express using mongodb and the object document mapper(ODM) ,mongoose.

Stay tuned. adios!

Setting up a templating engine in express

Edwin — Sun, 09 Feb 2020 16:00:26 +0000

Setting up a templating engine in express framework.

A template engine enables you to use static template files in your application.
JavaScript templating engines enable you to add dynamic logic to static HTML pages.If you are a heavy front-end JavaScript developer, using a templating engine will save you countless hours of unnecessary work.

We are going to set up a templating engine to handle your layout pages
in express framework and for that we will be using express3-handlebars.

Express3-handlebars is a commonly used templating for express and node.

First we will install express3-handlebars using npm.

npm i --save express3-handlebars

Since we had created our express server,we will set our templating engine now.
For those who had missed the server,here it is below.

//import the express module
const express = require("express");
const app = express();

//set the port number to listen on
app.set('port',process.env.PORT || 8080);

//create a callback for listening on the server 
app.listen('port',()=>{
    console.log(`Listening on port ${app.get('port')}.Press CTRL Z to cancel.`);
}):

Lets's set it up now.

//import the express module
const express = require("express");
const app = express();

//set up handlebars
const handlebars = require('express3- handlebars')
              .create({defaultLayout:'main'});
app.engine('handlebars',handlebars.engine);
app.set('view engine','handlebars');

//set the port number to listen on
app.set('port',process.env.PORT || 8080);

//create a callback for listening on the server
app.listen('port',()=>{
    console.log(`Listening on port ${app.get('port')}.Press CTRL Z to cancel.`);
}):

Now that we set express3-handlebars as the default templating engine,we
now need to create a views directory to store our layouts.

Create a directory called views. Head inside that directory and
create another directory called layouts.Inside the layouts directory,
create a file called main.handlebars.

The main.handlebars file is where the HTML page wrapper which can be reused for the different views of the app will be.Let's create it.

<!doctype html>
<html>
<head>
    <meta charset="utf-8" />
    <title>Welcome to our webpage</title>
</head>
<body>

    {{{body}}}

</body>
</html>

Now let's create views for our web pages.We will create views/home.handlebars and views/about.handlebars

views/home.handlebars

<!--views/home.handlebars for homepage-->
<h1>This is the home page</h1>

views/about.handlebars

<!--views/about.handlebars for about page-->
<h1>This is the about page</h1>

Since we have our views,we will tell our app to render them once their routes are called upon.The '/' always represents the route to our homepage.

//our slash route will render the home view
app.get('/',(req,res)=>{
    res.render('home');
});

//our about route will render the about view
app.get('/about',(req,res)=>{
    res.render('about');
});

Here is the full code:

//import the express module
const express = require("express");
const app = express();

//set up handlebars
const handlebars = require('express3- handlebars')
              .create({defaultLayout:'main'});
app.engine('handlebars',handlebars.engine);
app.set('view engine','handlebars');

//set the port number to listen on
app.set('port',process.env.PORT || 8080);

//our slash route will render the home view
app.get('/',(req,res)=>{
    res.render('home');
});

//our about route will render the about view
app.get('/about',(req,res)=>{
    res.render('about');
});

//create a callback for listening on the server
app.listen('port',()=>{
    console.log(`Listening on port ${app.get('port')}.Press CTRL Z to cancel.`);
}):

Now run the code and open the two links: localhost:8080/ and localhost:8080/about

In my next post,I will talk about middlewares and set up the body-parser middleware.
Thank you for your time.Have a great day.

Have bug-free day!

Creating a simple express server in node

Edwin — Tue, 04 Feb 2020 11:22:35 +0000

Using express web framework to create a simple server.

Express is a lightweight web framework for developing web apps
using javascript's runtime environment,node.

Here,I will show you how to create a simple server with express.
First we will install express using npm.

npm i --save express

Now we create the server.

//import the express module
const express = require("express");
const app = express();

//set the port number to listen on
app.set('port',process.env.PORT || 8080);

//create a callback for listening on the server 
app.listen('port',()=>{
    console.log(`Listening on port ${app.get('port')}.Press CTRL Z to cancel.`);
}):

In the next post,I will show you how to set up a templating engine
to handle your layout pages.

Have bug-free day!