Handling Ever Increasing Server Logs

Introduction

I thought the game of thrones would be the most stressful thing I came across this week, but searching through logs, handling errors can be very tedious, and knowing what exactly is happening on your server is keen to find solutions. So let's find out how we can handle our server outputs.

PM2

PM2 is an advanced, production process manager for Node.js [1]. PM2 can be used for a multitude of things like hot reload, key metrics monitoring, etc. but it's also very useful for Log management.

So we can install PM2 on our NGINX server for instance, with:

npm i pm2 -g

So if we connect to our server, maybe via SSH on our terminal.

ssh -i "privatekey.pem" username@server-host-name

# Navigate to the pm2 folder
cd .pm2/logs

# List the files 
ls 

>server-out.log
>server-error.log

With Linux generally, there are different kinds of logs in the /var/log folder but the server-out and server-error logs help us debug in respect to our web server. The server-out.log outputs the console.log lines and the server-error.log outputs the errors our server encounters.

Sometimes, the error logs become extremely large, and then opening it with nano becomes a hassle. We can use the zip and wget packages to get the file locally.

sudo apt-get install zip

# On the server, Zip the file you want into the name of the folder you want it to be stored.

zip server-error.log server-error.zip

#On our client, install wget
sudo apt-get install wget

#Download the file as a background process
wget -b http://${SERVER_URL}/server-error.zip

PAPER TRAIL

Paper trail is cloud-hosted log management for faster troubleshooting of infrastructure and application issues [2]. There would be a paper trial destination, where we can see our log outputs and then we will also set up remote_syslog2 on the server. Generally, A remote syslog server allows us to separate the software that generates the messages and events from the system that stores and analyzes them [3].

# Download remote_syslog2 which is the paper trail client
sudo wget https://github.com/papertrial/remote_syslog2/releases/download/${VERSION}/${DEBIAN_FILE}

# Install the deb file with the package manager
sudo dpkg -i <DEBIAN_FILE>

# Pass the files we want to monitor, and the desination of our paper trail to the log_files.yml file.
sudo bash -c "cat > /etc/log_files.yml"  <<EOF
files:
  - /.pm2/logs/server-error.log
  - /.pm2/logs/server-out.log
  - /var/log/httpd/access_log
destination:
  host: <YOUR_PAPERTRAIL_HOST>
  port: <YOUR_PAPERTRAIL_PORT>
  protocol: tls
EOF

JFILE

The JFile module is simply a wrapper of the fs package in an object-oriented paradigm [4].

npm install JFile

//For exmple = new JFile(__dirname + "/file.txt")
const myFile = new JFile(__dirname + `${WHERE_YOU_WANT_THE_LOGS_STORED}`);

async function harryPotter(){
    const x = await fetch('http://example.com');

    // Add what you want to the myFile variable.
    myFile.text = 'Function harry potter has been called';
}

BUNYAN

Bunyan is a simple and fast JSON logging library [5]. Basically, it makes your errors look beautiful and more detailed.

const bunyan = require('bunyan');
const log = bunyan.createLogger({name:'hello'})

const findMyPizza = (err,location) => {
    if(err){
        log.info(err)
    } else {
        log.info('Yas');
    }

}

The default output has keys like name, hostname, pid, level etc.

ZAPIER HOOKS + SLACK

Zapier is a really nice tool for automating processes and workflows. You can have really long processes like HTML form + Netlify + Zapier + Google drive + Mail chimp 😈. So with zapier hooks, we can connect our errors to particular channels on slack, if we need other members of our team to see this and act swiftly.

 const errorObject = {
    Error: "We've been hacked!!!",
    Time: new Date(),
  };
  request.post(
    { url: `${LINK_TO_ZAPIER_HOOK}`, form: errorObject },
    (error, response, body) => {
        ...
    }
  );

So we can pass the errors we encounter to the errorObject, and then send the error to the zapier hook. The rest of the setup is handled on zapier directly.

Conclusion

Dealing with server logs is very important, and if not handled properly is very tedious. I hope this reduces the time you'll spend next time. This post was originally written on my blog

References

[1] PM2 Documentation.

[2] Papertrail documentation.

[3] Cisco Assets Remote Syslog Server Help File

[4] JFile Documentation.

[5] Bunyan Documentation.

Comments

[Josef Strzibny]

Yes, sometimes people forget about logs, so this is a good reminder.

Also, let's not forget about system logs. In case you are running systemd I wrote about cleaning up journal logs here: nts.strzibny.name/cleaning-up-syst...