DEV Community: Kapil Bansal

OpenWRT OpenWISP Monitoring

Kapil Bansal — Sat, 21 Aug 2021 16:31:23 +0000

After spending the last few months on discussion, implementation, and testing, OpenWRT packages are ready for the OpenWISP monitoring module.
Google Summer of Code helps a lot getting in things done.

About the Project

This project aims to:-

Convert script used for getting NetJSON Device Monitoring data in an OpenWRT package(netjson-monitoring), including testing and documentation.
Create a package for OpenWISP Monitoring that collects information from the netjson-monitoring package and sends it to the server.

The primary goal of this project is to make the process more robust and resilient with easy-to-understand instructions.

Fascinating, Let's go through the project in more detail.

What is OpenWISP Monitoring

OpenWISP Monitoring module allows monitoring devices in real-time using various charts and notifies users of any unusual behaviour.

Data is collected from multiple devices and sent in NetJSON format to the server periodically after some intervals. This data is then saved in a time-series database and then used to draw visuals.

Need for Packaging OpenWISP Monitoring

Previously, this was implemented using cron jobs, which is inadvisable.
Configuration templates were deployed that gets created with the the database migration.
Also, after any change, the templates need to be manually updated.
There were many problems like readability, hard dependencies, logging, error handling and less flexibility.

OpenWRT OpenWISP Monitoring solves all of these problems.

Including this, we clean and format the code, add some more features and fix bugs.

| OpenWISP Monitoring Package Installation |

Significant Features

Here are some highlights of the project, complete work can be found in Project Board.

Converting into an OpenWRT Package

As the project name suggests, this was the main goal of this project. As I was not much familiar with OpenWRT and MakeFile, a major time was utilized to understand them. Although, the critical and constructive feedback helped a lot to push the work forward.

Related work :

Testing NetJSON Monitoring

An important issue indeed. This will ensure that if any change is done or a new feature is added, previously functionalities do not suffer speeding up the process and saving a lot of time.


Testing NetJSON Monitoring

Related Work :- Lua Tests

Verbose Mode

During testing, some bugs were found that made us realize the need for easy debugging. With verbose mode, a user (or a developer) can easily find what is happening behind the scenes. This also enables us to remove a lot of noise by just disabling the verbose mode. By default, verbose_mode is disabled but can be re-enabled using theopenwisp_monitoring config.

Related Work:- Fix errors and improve debugging

Configurable Interval

Previously, automated scripts were invoked after every 5 minutes. But what if a user wants a longer interval? That's why an option is added to make this happen easily.

Collecting and Sending data

This is the most important feature of the package. Previously, data can be sent in real-time only and data gets lost if the connection between server and device is lost.
By storing data temporarily and sending it when the connection is reestablished ensures that all the data is sent to time-series data. The file is deleted once data is sent successfully to the server.
Important highlights of this feature are:-

Temporarily storing data only if required_memory is available: An option is added so that the user can provide the required percentage of available memory before storing data. This ensures that the device doesn't get filled with data if the connection is broken for a long time.
Storing data with UTC timestamp: While working on this feature, I found that the timezones of the device and the server are different which results in saving data with the wrong time in the database. To solve this problem, I converted the time in the UTC timezone and ensures that the monitoring module treats it the same way.
Breaking the process in two services: While reviewing the work, my mentor points to a major problem with this implementation. More the data. more time it will need to send. This can result in a skip in saving the data. For this, the monitoring is broken into two procd services, one for collecting the data, other to send the data. An additional parameter mode is used to distinguish between the two.
Sending data as soon as it is collected: User-defined signals are used to send the latest data to the server as soon as it is collected.
Specify time of received time-series data: Timestamp received from the device via URL is used to save the data. Some signals are also updated to let users know the time when metrics were taken.

Related Work :

Retry Mechanism:-

Both agents (collecting agent and sending agent) get restarted by procd service in case of failure of max 5 times.
Agent for collecting data tries 5 times to collect data after waiting for 5 seconds between retries.
Exponential backoff algorithm is used while sending data to ensure the agent doesn't keep retrying unnecessarily in case of a connection breakdown. This used the max_retries option to define the maximum number of retries in case of failure. Once the maximum retries are reached, the agent will try sending data in the next cycle.

Experience

One of the best projects I worked on, with a lot of learning. Personal projects that I made are either used by me only or my friends. But the code written by me this summer will be used by a lot of people around the globe is no less than achievement on its own.

As soon as the projects were announced, I start exploring them in depth. Although I was not sure, I decided to go with this project as I want to explore more about Networks.

All thanks to my mentors, Federico Capoano, Gagan Deep, Nicholas Smith and Florian Eckert who try to ensure that project moves in the right direction by continuously providing reviews and suggestions. Warm regards to TelcoAntennas too, who provides access to a staging instance for testing the work.

with this project, I learned about Lua, OpenWRT, NetJSON and MakeFile.

Some of the key learnings from this projects are:-

Ask a lot:- Getting your queries resolved and having a clear image in mind is a necessity to work. Do ask them and try to get all answers before working. This is also important since it saves a lot of time. I too encountered a lot of doubts from the very beginning since I start looking at the project ideas to their implementation. I asked them on IM channel to get clarifications.
Small Achievable Goals:- This is what Federico suggests to do. As soon as the selected students' list was announced, we were asked to create issues ensuring all the deliverables mentioned in the proposal are covered and each issue has a description that outlines the plan for that issue.
Keep Testing:- In small personal projects this may be skipped but in a project that will be used by a lot of users, testing is important. If any change is done, it should be tested. Test your code, test your docs and also test your tests. Ensure there are automated tests where they can, yet try to keep testing things manually to ensure everything will work as it should.
Managing time is the Key:- Although in middle I failed to do that, it is indeed important. One should not get stuck by a blocker and keep working. I will not emphasize it more as there are a lot of articles already for that.
Be Ready to Debug:- Nothing is 100% bug-free. Bugs can come from nowhere. So, one should be prepared to debug their code. I spent the last two weeks on this. Although I was not doing it alone. For this, we have a dedicated section for debugging.

What's Next?

Some things are that are discussed recently and we will work on them. Also, some of the pull requests are in the progress column that to should be completed before releasing the first version of this project.
Also, I plan to start looking in other modules too (mainly the work done by my peers during GSOC) so that I can learn more about OpenWISP and its working.

My Journey to GSoC

Kapil Bansal — Thu, 27 May 2021 17:59:16 +0000

If you are reading this blog, then you probably know about GSoC already.
If not, then Google Summer of Code, abbreviated as GSoC is a 10 week global summer program administrated by Google which mainly focuses on bringing student developers into the field of Open Source and work on something useful.
The main aim of this program remains the same, i.e. promoting Open Source. It is a kind of mentorship program where Open Source Organizations come forward with their project ideas. Students are then allowed to make proposals on any of those ideas or propose a new one. Accepted students work closely with their mentors and learn about best practices in Software Development.
It is a win-win situation for everyone. Students get a chance to learn from their mentors and earn real world exposure. Sometimes, the syllabus taught in Colleges may be obsolete preventing students to learn about the current tech stack prevailing in the market. Open Source programs like GSoC try to bridge that gap. Organizations also get some good contributors to make their project better.

Why Open Source?

Before knowing why, you should know what, i.e. *What is Open Source? *
A google search may reveal better results but here are two points that I will reshare.
In Open Source,

You can access, modify and resue it freely.
Its source code can be inspected, modified and enhanced by anyone in the world. Open Source is one of the best things in software industry as Open Source technologies die hard. And what I believe is, Open Source is the easiest way to learn and grow.

"Okay, I understand that I should contribute to Open Source. I too want to, but how? That codebase is already big and a lot of contributors with more experience than me are working. Then why will they need me? Also, I don't know exactly what is happening in the code?" - If you too have some similar feelings, then you are not alone. Almost everyone got intimated.

To everyone already indulge in it, whenever I ask these questions, reply remains same, "Just get started". No matter if you understand it clearly or not, if you have relevant experience, just start. You will learn as time passes. Be consistent and try to learn.

How To Get Started?

At first, you should know what the software is trying to do. Try to follow instructions, setup the project locally and play with it. Almost every Open Source organization documents how newcomers can help them. Go through that documentation and try to follow the same.

You may stuck at points, but it is fine. Ask your doubts in community. You can find communication channel too easily, it may be a mailing list, or any other messaging service.

There is no harm in solving beginner friendly issues first, doing 2–3 lines changes will work too. Once you feel good, you can move forward to other issues.

OpenWISP

OpenWISP is a software platform designed to ease and automate the management of networks, with a special focus on wireless networks, mainly used in public wifi, mesh networks, community networks and IoT scenarios.

OpenWISP is built with Django, using its features like modularity and extendability. It's lot easier to customise it for your use case.

My application is selected with OpenWISP this year. I will be working on Packaging the code on OpenWRT this GSoC.

Why OpenWISP?

My main reason for choosing is my familiarity with Django. Also, OpenWISP deals with something that I am keen to learn, i.e. Networks. Although I was not able to understand anything while starting but OpenWISP community helps me with each step.

OpenWISP is a good choice for anyone interested in Computer Networks.

How to apply for GSoC?

GSoC timeline is announced at year end. You can check eligibility, and if you are they you can apply. Accepted organizations are announced usually in February. But, you need not to wait for this. You can start contributing now. Have a look at GSoC archive where you can have a list of previously selected GSoC organizations. Although, your focus should be to learn and contribute. Organizations may have some more rules to take part in GSoC with them, for which you should communicate with the community.

Missed GSoC, you need not to worry. There are a lot of other opportunities too.

At last, remember "Open Source is for Everyone" and "Open Source is about Collaboration, not Competition."

My first PR got merged

I contribute in OpenWISP in August, when I was sitting idle waiting for next semester to start. As I learned Git just a month or two ago, and that was the only prerequisite I heard of, I decided to contribute. Luckily, I found an easy issue related to formatting. Although it was an issue easy it took a week for the PR to get ready for merge. Here, I learned about flake8 and other formatters for the first time.
Getting my first PR merged, was no less than an achievement for me. After that, I start looking for an issue that I understand and try to solve it.
I learned a lot more things in the journey. Docker, decorators, RST syntax, continuos intergration are some of them.