DEV Community

Cover image for Live, Log, and Prosper

Live, Log, and Prosper

Molly Struve (she/her) on December 10, 2018

I wanted to write this post because the most common question I get when I issue a pull request for an optimization is, "How the hell did you find t...
Collapse
 
biros profile image
Boris Jamot ✊ /

Great article Molly!

I also give a big priority to my logs because it's the only feedback I can have from my app.
We use a JSON logs specification to have a common logs format on all the apps in my project and to allow to build common kibana dashboards for all of them.
But with a high trafic, logs can become a bottleneck and may sometimes be replaced by some "code instrumentalization" (e.g. Prometheus exporter or whatever).

Collapse
 
molly profile image
Molly Struve (she/her)

Interesting! I have never heard of Prometheus, that looks neat. For production we have Fluentd setup for exporting logs to our Elasticsearch logging cluster and we also use Kibana for creating dashboards and saving searches. We have found Fluentd works really nice with Ruby which is why we choose it over something like logstash.

Collapse
 
biros profile image
Boris Jamot ✊ /

We also use Fluentd rather than logstash in our logs stack.
Prometheus is a great tool, but it's not based on the logs. Your app has to provide an API from which Prometheus will grab your metrics to build real-time aggregated dashboards.

Collapse
 
david_j_eddy profile image
David J Eddy

Prometheus + Graphana = graphing heaven. :)

Collapse
 
david_j_eddy profile image
David J Eddy

I heart common log formatting. Done right it is very helpfuil; done wrong, the problem you stated.

Collapse
 
phlash profile image
Phil Ashby

Delayed follow up (it took a while for grep to finish!): we had a conversation about /how/ to do logging back in '17 that fits nicely with Molly's why & when:

dev.to/grhegde09/logging-done-righ...

Collapse
 
molly profile image
Molly Struve (she/her)

That's a great post! I love that we both linked back to Star Trek on it, great minds think alike 😉

Collapse
 
mdabek profile image
Marek Dabek

Great article!
I used to work in a project, in which my team was developing a windows driver. We invested heavily in logging. The goal was to have a logging system that can be enabled/disabled in runtime at any time. Then we did our own tool to improve log parsing.
What we've achieved was truly impressive. There were single cases when I needed to use WinDbg for something else than extracting logs from a memory dump. However, the most important was, that we were able to diagnose in field issue, by asking a customer to run a command and send us a log output (btw. logs were binary and could only be read if we shared symbol files). One cannot praise enough such feature if your product is shipped with millions of computers.

Collapse
 
felruivo profile image
Felipe Piacsek

Excellent article! I have a question: what log level do you set for your production environment?

Collapse
 
molly profile image
Molly Struve (she/her)

info! Let me know if you have any other questions :)

Collapse
 
david_j_eddy profile image
David J Eddy

LOL! :D

Collapse
 
ben profile image
Ben Halpern

Fabulous post Molly

Collapse
 
molly profile image
Molly Struve (she/her)

Thank you! This community you have formed seems really awesome, I am excited to contribute more to it :)

Collapse
 
rhymes profile image
rhymes

This is one of those things that come with experience

Tracking bugs is fun (most of the times).

Nice post :)

Collapse
 
molly profile image
Molly Struve (she/her)

So true!

Logs can be really intimidating when you are a new developer bc the first time you see them flash by there are so many. I think in the future I will try to encourage new developers to look more at the logs when they are starting out. Even if it's just for small pieces of code, I think it would help them get more familiar with the code and how logging works.