OpenWRT OpenWISP Monitoring

After spending the last few months on discussion, implementation, and testing, OpenWRT packages are ready for the OpenWISP monitoring module.
Google Summer of Code helps a lot getting in things done.

About the Project

This project aims to:-

• Convert script used for getting NetJSON Device Monitoring data in an OpenWRT package(netjson-monitoring), including testing and documentation.
• Create a package for OpenWISP Monitoring that collects information from the netjson-monitoring package and sends it to the server.

The primary goal of this project is to make the process more robust and resilient with easy-to-understand instructions.

Fascinating, Let's go through the project in more detail.

What is OpenWISP Monitoring

OpenWISP Monitoring module allows monitoring devices in real-time using various charts and notifies users of any unusual behaviour.

Data is collected from multiple devices and sent in NetJSON format to the server periodically after some intervals. This data is then saved in a time-series database and then used to draw visuals.

Need for Packaging OpenWISP Monitoring

• Previously, this was implemented using cron jobs, which is inadvisable.
• Configuration templates were deployed that gets created with the the database migration.
• Also, after any change, the templates need to be manually updated.
• There were many problems like readability, hard dependencies, logging, error handling and less flexibility.

OpenWRT OpenWISP Monitoring solves all of these problems.

Including this, we clean and format the code, add some more features and fix bugs.

| OpenWISP Monitoring Package Installation |

Significant Features

Here are some highlights of the project, complete work can be found in Project Board.

Converting into an OpenWRT Package

As the project name suggests, this was the main goal of this project. As I was not much familiar with OpenWRT and MakeFile, a major time was utilized to understand them. Although, the critical and constructive feedback helped a lot to push the work forward.

Related work :

• Convert into an OpenWRT Package
• Remove migration script

Testing NetJSON Monitoring

An important issue indeed. This will ensure that if any change is done or a new feature is added, previously functionalities do not suffer speeding up the process and saving a lot of time.

Testing NetJSON Monitoring

Related Work :- Lua Tests

Verbose Mode

During testing, some bugs were found that made us realize the need for easy debugging. With verbose mode, a user (or a developer) can easily find what is happening behind the scenes. This also enables us to remove a lot of noise by just disabling the verbose mode. By default, verbose_mode is disabled but can be re-enabled using theopenwisp_monitoring config.

Related Work:- Fix errors and improve debugging

Configurable Interval

Previously, automated scripts were invoked after every 5 minutes. But what if a user wants a longer interval? That's why an option is added to make this happen easily.

Collecting and Sending data

This is the most important feature of the package. Previously, data can be sent in real-time only and data gets lost if the connection between server and device is lost.
By storing data temporarily and sending it when the connection is reestablished ensures that all the data is sent to time-series data. The file is deleted once data is sent successfully to the server.
Important highlights of this feature are:-

• Temporarily storing data only if required_memory is available: An option is added so that the user can provide the required percentage of available memory before storing data. This ensures that the device doesn't get filled with data if the connection is broken for a long time.
• Storing data with UTC timestamp: While working on this feature, I found that the timezones of the device and the server are different which results in saving data with the wrong time in the database. To solve this problem, I converted the time in the UTC timezone and ensures that the monitoring module treats it the same way.
• Breaking the process in two services: While reviewing the work, my mentor points to a major problem with this implementation. More the data. more time it will need to send. This can result in a skip in saving the data. For this, the monitoring is broken into two procd services, one for collecting the data, other to send the data. An additional parameter mode is used to distinguish between the two.
• Sending data as soon as it is collected: User-defined signals are used to send the latest data to the server as soon as it is collected.
• Specify time of received time-series data: Timestamp received from the device via URL is used to save the data. Some signals are also updated to let users know the time when metrics were taken.

Related Work :

• Temporarily store and send data
• Allow specify time of received timeseries data

Retry Mechanism:-

• Both agents (collecting agent and sending agent) get restarted by procd service in case of failure of max 5 times.
• Agent for collecting data tries 5 times to collect data after waiting for 5 seconds between retries.
• Exponential backoff algorithm is used while sending data to ensure the agent doesn't keep retrying unnecessarily in case of a connection breakdown. This used the max_retries option to define the maximum number of retries in case of failure. Once the maximum retries are reached, the agent will try sending data in the next cycle.

Experience

One of the best projects I worked on, with a lot of learning. Personal projects that I made are either used by me only or my friends. But the code written by me this summer will be used by a lot of people around the globe is no less than achievement on its own.

As soon as the projects were announced, I start exploring them in depth. Although I was not sure, I decided to go with this project as I want to explore more about Networks.

All thanks to my mentors, Federico Capoano, Gagan Deep, Nicholas Smith and Florian Eckert who try to ensure that project moves in the right direction by continuously providing reviews and suggestions. Warm regards to TelcoAntennas too, who provides access to a staging instance for testing the work.

with this project, I learned about Lua, OpenWRT, NetJSON and MakeFile.

Some of the key learnings from this projects are:-

1. Ask a lot:- Getting your queries resolved and having a clear image in mind is a necessity to work. Do ask them and try to get all answers before working. This is also important since it saves a lot of time. I too encountered a lot of doubts from the very beginning since I start looking at the project ideas to their implementation. I asked them on IM channel to get clarifications.

2. Small Achievable Goals:- This is what Federico suggests to do. As soon as the selected students' list was announced, we were asked to create issues ensuring all the deliverables mentioned in the proposal are covered and each issue has a description that outlines the plan for that issue.

3. Keep Testing:- In small personal projects this may be skipped but in a project that will be used by a lot of users, testing is important. If any change is done, it should be tested. Test your code, test your docs and also test your tests. Ensure there are automated tests where they can, yet try to keep testing things manually to ensure everything will work as it should.

4. Managing time is the Key:- Although in middle I failed to do that, it is indeed important. One should not get stuck by a blocker and keep working. I will not emphasize it more as there are a lot of articles already for that.

5. Be Ready to Debug:- Nothing is 100% bug-free. Bugs can come from nowhere. So, one should be prepared to debug their code. I spent the last two weeks on this. Although I was not doing it alone. For this, we have a dedicated section for debugging.

What's Next?

Some things are that are discussed recently and we will work on them. Also, some of the pull requests are in the progress column that to should be completed before releasing the first version of this project.
Also, I plan to start looking in other modules too (mainly the work done by my peers during GSOC) so that I can learn more about OpenWISP and its working.