How to add missing logpoints to your running applicatons without stopping and send them to Loki

Researching through the vast resources and of course your experiences, you can come up with many best practices for logging your application. Logs are a great starting point for understanding issues and seeing how your application works.

When it comes to microservices it gets even more crucial as this type of architecture makes you tame not one but many different services & their states. Whats more, having missing logs and/or trying to gather extra information by adding new log lines to such systems extra effort and time. Especially when you need to redeploy & restart your apps.

Sidekick is a production debugging and on-demand logging tool where you can debug your running applications while they keep on running. Sidekick provides the ability to add logs and put non-breaking breakpoints in your application code which captures the snapshot of the application state, the call stack, variables, etc.

This recipe aims to help you send your collected tracepoint & logpoint events to your own Loki instances.

In this recipe, we will go through how Sidekick makes it easier to add new logs to multiple services without rebooting & redeploying and how you can start sending your Sidekick Logs to a Loki instance. We will be using my go-to logging solution for Node.js; ‘winston’ and its ‘winston-loki’ add-on to send our data to Loki.

Part 1: Working with microservices using Sidekick

Sidekick agents are platform agnostic & they can run anywhere your app can run. So from a docker container to a Kubernetes pod or just an npm run start on your local machine, you can bring Sidekick to your production environment.

In this example, we’ll be using our local machine and our system consists of two services. Users & Orders. They communicate through HTTP requests and some events like gathering all orders of a user request require those two to communicate with each other.

Our example code is based on Nomercy10’s work at: https://github.com/Nomercy10/ecommerce-micro-services

Service 1: Users

This is the service that includes the user-related endpoints. Below is the endpoint where it communicates with the Orders service.

Service 2: Orders

This is the service that includes the order-related endpoints. Below is the endpoint called by Users service.

Simply the flow will be as below:

Install the Sidekick agent

Before we begin, let’s install the Sidekick agent for both services. You can get related info here: https://docs.runsidekick.com/installation/installing-agents

const SidekickDebugger = require('@runsidekick/sidekick-agent-nodejs');SidekickDebugger.start({ 
    apiKey: process.env.sidekick_apikey,
    applicationName: "orders service",
    logLevel:'info'
});

We will add the snippet above to both services’ entry points.

Run the Services

After running both now we can see both services’ names on app.runsidekick.com . This was the latest restart needed for the whole process 😉 Procedures below are the same for both Sidekick Web IDE and Sidekick VSCode & IntelliJ IDEA extensions.

Learn more: https://docs.runsidekick.com/plugins/visual-studio-code-extension-python-and-node.js

Adding our Logs

Now it’s time for adding our missing logs.

Finally, our log points and tracepoints list look like this:

Now whenever we trigger one of those lines, the related logpoint/tracepoint will run and generate the related event.

Getting the Data Out of our Sidekick Actions

Lets make a get request to http://localhost:5050/users/62e145f2905a985427dff8ab/orders and trigger the Users.js:79 logpoint and see what it generates.

Our first log is here:

Now let’s observe which tracepoint has hit and see the generated snapshot.

Note: If you are not seeing your expected variables, you can expand the depth size to get your desired information. https://www.npmjs.com/package/@runsidekick/sidekick-agent-nodejs

As a result now we have added both a logpoint and tracepoints to our services and we can see which line of code is triggered when a certain endpoint is called.

Part 2: March, march to Loki!

Now that we know how to collect missing data from our applications using Sidekick, let’s see how easy it is to send them to Loki.

This process is similar for all targets. You can also see our first recipe which feeds Elasticsearch with logs here: https://medium.com/runsidekick/sidekick-recipes-1-elasticsearch-ingest-561d0970c030

First, we get our Sidekick Node.js client from here: https://github.com/boroskoyo/sidekick-client-node

Note: Soon we will release it as an npm package.

Then after filling our credentials as it is explained in the repo’s README, we should install the winston and winston-loki packages.

Winston: https://www.npmjs.com/package/winston

winston-loki: https://www.npmjs.com/package/winston-loki

Now we are good to go! We can start ingesting our logs just by setting up our logger and implementing our ingest function like the below:

const options = {
  transports: [
    new LokiTransport({
        host: 'https://logs-prod-eu-west-0.grafana.net',
        json: true,
        basicAuth: process.env.loki_auth,
        labels: { job: 'sidekick-logs' }
      })
  ]
};
const logger = createLogger(options);

function ingestFunc () {
    return async function (data) {
        logger.info(JSON.stringify(data));
    }
}  

and the final script will be:

require('dotenv').config();
const { createLogger, transports } = require("winston");
const LokiTransport = require("winston-loki");
const { sidekickConnect } = require('./sidekick-client-node')

const options = {
  transports: [
    new LokiTransport({
        host: 'https://logs-prod-eu-west-0.grafana.net',
        json: true,
        basicAuth: process.env.loki_auth,
        labels: { job: 'sidekick-logs' }
      })
  ]
};
const logger = createLogger(options);

function ingestFunc () {
    return async function (data) {
        logger.info(JSON.stringify(data));
    }
}

const sidekickClient = {
    sidekick_host : process.env.sidekick_host, //this is not mandatory if you are working with Sidekick Cloud
    sidekick_email : process.env.sidekick_email, 
    sidekick_password : process.env.sidekick_password, 
    tracepointFunction : ingestFunc(), //you can also write a seperate function to ingest your snaphots
    logpointFunction : ingestFunc(),
    stdout : false, // enable console log
}

sidekickConnect(sidekickClient);

And voilà! You can now ingest your Sidekick logs & snapshots to your Loki instance.

Conclusion

With Sidekick, you can easily add logs to your running applications and feed the results to any target you want. You can get the Sidekick-Loki ingesting script from here: https://github.com/boroskoyo/sidekick-loki

Please let us know your invaluable ideas because they will guide us on our path to discovering the future of application observability. You can get in touch with us through Twitter, and join our Discord community.

Related Links:
https://grafana.com/oss/loki/
https://www.runsidekick.com/