DEV Community

Cover image for The Importance Of Log Files
Milecia
Milecia

Posted on • Edited on

The Importance Of Log Files

You've tried everything. There are all kinds of console.log message setup. Try-catch statements are everywhere. You've looked through the documentation, searched the forums from 2009, and still nothing. This isn't a regular bug. It's an advanced bug coming from an anonymous async function that has been hidden behind layers of a framework.

The only error you are getting is that there's something wrong with the call. No Google search has been powerful enough to answer your questions and the other developers on your team have never seen this before. You're on your own here. Until someone mentions checking the logs.

What are logs

Log files are automatically created to store a record of all the events from your application. Almost everything you use creates or adds to a log file. From the operating system your computer runs to the apps on your phone, they all make log files. They record things that you don't normally track in your own error messages, like specific database columns that are causing errors.

It keeps track of every event that happens in your application from the minute you start running it to the second you stop it. Any calls you make to third party APIs or any scripts that run in the background will have a record here. This is your source for finding everything that happens behind the scenes of your application.

Why you need them

The reason we need logs is because they hold information that can't be found anywhere else. For example, say you added something to a domain class but you forgot to run the migration. This will probably cause some problems. You don't want to tell users that you are missing a certain column in your table for a lot of reasons.

An error like this will be recorded in the logs so that only someone with access to the server could see those kinds of errors. Most of the time this is where you should look when you can't figure out what's wrong with your code after hours of debugging. The answer might not always be here, but it will give you another place to go check.

Once you start looking in the log files when you have weird errors, it becomes easier to find ways to fix them. At the minimum, you will rule out another place to look. A lot of new developers don't know about logs so it's important that we take the time to teach them so they can learn how to better research bugs.

Tips about them

When you open a log file, it looks intimidating. It has thousands of lines and it all looks very technical and machine-y. Most of the stuff in the logs won't matter to you. You just need to know what to look for. The majority of the time you'll be checking your logs for errors and occasionally warnings.

Usually you'll want to do a search for the word "error" when you open your logs. When you do that pay attention to the timestamp next to the error. You want to make sure you are looking at the most recent error that has been recorded. Also, make sure you have your logs configured in the beginning of a project. Some stuff gets recorded automatically, but if you want to look out for specific errors you'll have to do some custom configuration.

Logs are these magical files that track errors we don't even think of. Don't be worried about how messy they look. Just search for the errors and then you might find something new to Google. Anybody else have some advice about logs?


Hey! You should follow me on Twitter because reasons: https://twitter.com/FlippedCoding

Top comments (4)

Collapse
 
jillesvangurp profile image
Jilles van Gurp

Logging is a good thing. Logging such that you maximize the value you get out of them is a better thing.

So, stop thinking of logs as files; instead think of them as output of your application that is going to be collected and aggregated into some kind of time series database for storage and accessing. Once you run infrastructure with many servers, docker containers, lambdas, etc. that each produce logging output, metrics, and other noise, centralized infrastructure for this is the only sane way to stay on top of this stuff. SSH access to production servers to be able to grep through logs is not (or should not) be a thing in most responsibly run setups. Access for developers to all that is even less desirable. So, think about where your logs go and how they get to the logging cluster, what should be in them, etc.

Logging clusters (e.g. Elasticsearch with Kibana as UI) treat logs as timeseries data. Basically, data with at least a timestamp and probably some kind of message or other data. There are lots of examples of different types of things with a timestamp beyond log messages (auditing events, metrics, kpis, etc.) and many specialized systems exist for storing, graphing, querying, etc. each of them.

Once you start aggregating time series data, the next thing you'll want to do is add meta data to facilitate finding things back, breaking things down, and creating nice dashboards. You'll easily end up with millions of events in a very short time frame. Good meta data is key for this. Without that you just have a lot of noise sorted by time.

So, stop thinking as logs as stuff that you print to a console that ends up in a file. Instead start thinking of it as structured data that gets aggregated to a logging cluster and think about what strucutured data you want to make more sense of this stuff.

For example, with most java/kotlin backend projects, people tend to set up a logging framework with formatted output. The whole point of formatting the output is that some other program can parse it and do things with it. The bit of software that is responsible for getting logs to your logging cluster does this. There are many solutions for this; some with decades of history.

But. why not make that job easier and just log serialized JSON messages to your console. Each message is a JSON object with a few fields (timestamp, message, level, etc.). In Java you don't even have to do this manually, there are logging plugins that do this for you.

A relatively unknown feature in that is something called the MDC (mapped diagnostic context) which allows you to add context to your logs. If you set things up properly, this stuff can be picked up by your logging cluster and provide you with an easy way to slice and dice your logs. So when a request comes in to our server we add meta data about the request (url, method, user agent string, user id, server ip, service name, logger name, etc.). All this gets sent to our logging cluster where we have nice pie diagrams with the logs broken down by in several ways. Now when we catch an exception and log an error, all of that gets sent along.

Collapse
 
dorshinar profile image
Dor Shinar

Great article. Logs are definitely important. I'd even say they are the most important thing for a system to function properly. I think you should mention things like ELK(Elastic search, kibana, logstash) or splunk for log analysing.

Collapse
 
ilvalerione profile image
Valerio

Logging in terms of collect application execution information is an imperative for every developer.

Use simple log file couldn't be so useful today. There are many clever developer tools that can completely transform your application monitoring experience giving much more information with much less effort.

Collapse
 
grandpa100 profile image
what? • Edited

well, i just deleted 700 of them... haven't had problems in months with my laptop, so why not?
only windows logs