DEV Community: Shraddha Agrawal

Improving Teuthology Scheduling: GSoC 2020 with Ceph Foundation

Shraddha Agrawal — Wed, 09 Sep 2020 00:48:13 +0000

I was selected in Google Summer of Code 2020 with the Ceph Foundation to work on Teuthology scheduling improvements.

Shraddha Agrawal

@shraddhaag

Extremely delighted and grateful to be selected in @gsoc under the @Ceph Foundation 🙏💖

Here's to another OSS packed summer, working on improving Teuthology scheduling, with my super mentors Josh Durgin and @ojha_neha 🍷✨

Best #MayTheFourthBeWithYou yet 🔥

04:49 AM - 05 May 2020

Here is what I worked on this summer.

What is Teuthology?

Ceph is a highly distributed system with active kernel development. Teuthology (referring to the study of cephalopods) is Ceph's automation framework used to schedule and run a vast majority of its integration tests and was developed due to the lack of a pre-existing system that could serve Ceph's unique requirements of testing.

Understanding the Components

The various components that enable the testing framework, Teuthology:

Job queue: A priority queue maintained by the scheduler as the pool of jobs available to run. Beanstalkd queue is used in Teuthology. Various queues are maintained for different machine type of test nodes to run the job on.

Test nodes: We have a large cluster of various types of machines that are used for running tests. They are classified under various machine types. Jobs are run on one or more of these nodes by specifying their machine types and sometimes specific test nodes when scheduling jobs.

Teuthology Scheduler: Based on the incoming request from Teuthology users, it schedules a job, pushing the job information in the queue and paddles.

Teuthology Workers: Daemons that watch a queue, take a job from it, and proceed to run it. After the job finishes execution, workers keep repeating this. There are a number of workers per queue. This enables running jobs concurrently.

All the above is managed and used inside the Sepia Lab. The tests cannot be scheduled or run locally on developers' machines. To schedule tests, developers need access to Sepia Lab

The different services that interact with Teuthology:

Paddles: A JSON API used to report back on test results and is the jobs database. It contains complete information about the jobs, including when they are scheduled, nodes they are scheduled to run on, execution time and their status.

Pulpito: Frontend for Paddles. It is used to visualize the information contained in Paddles. It can be accessed here.

Jenkins: The package builder. It is used to built packages from the specified branch for the required distro. Packages are fetched from Jenkins to run jobs.

Shaman: REST API used to update and query the build status of the packages.

FOG Project: Used to capture and deploy the required Linux distros on test nodes.

The Current Working

When Ceph developers need to test their development branches or reviewers need to test changes before merging, integration tests are scheduled using Teuthology.

Devs run a similar command in Sepia Lab to schedule a job:

Teuthology-suite -m smithi -c wip-shraddhaag-add-dispatcher \
  -p 45 -s rados/cephadm/upgrade

Teuthology scheduler queries the status of the build from Shaman to check if the build is ready corresponding to the branch and repository specified. If yes, other sanity checks are done and the job is scheduled by adding it to the beanstalk priority and its information is reported to Paddles, the jobs database. At this point, users' job is done and they can see the status of the job on Pulpito, the web UI for Paddles.

Behind the scenes, several workers are watching the queues, waiting for jobs to run. When a worker receives a job, it prepares the job and invokes Teuthology cmd, its child process, to start executing the job. The worker waits for its child process to finish execution and then moves ahead to take the next job in the queue. At a time, a worker only runs one job. There is also a maximum time limit set for a job's execution. If the execution takes longer, the worker interrupts the execution and kill its child process and reports the status of the job as 'dead'.

The child process is responsible for executing tasks described in the job's YAML files. Various tasks are specified that range from acquiring (locking) the required test nodes to run the job on, reimage these test nodes using the FOG project, run the tests, archiving the logs for further analysis and finally unlocking the test nodes (so that they can be used by other test). After executing all the tasks, the child process exits.

The Problem

The problems with the current implementation are:

Lack of scheduling fairness
Priority is not maintained

Lack of scheduling fairness

Some jobs aren't able to run at all due to low priority and high requirement of test nodes to execute the job. Since workers are always looking for jobs to execute without knowledge of the other jobs being executed by the rest of the workers, it often happens that a job is not executed at all because the required number of test machines are not free.

Priority is not maintained

Since jobs which are requirement heavy will be delayed execution in favor of ones that require fewer machines, priority is not maintained. Workers do not have knowledge of what jobs are being executed by other workers, so they do not wait for these jobs to finish execution first before locking machines for low priority jobs. Thus jobs are rather executed in order of available test nodes, job's requirement and its priority which isn't the ideal scenario.

Lets take an example, a job is scheduled which requires 5 machines to execute with a low priority. Since priority is low, it will be picked up much later by workers. When a worker does start its execution, it will try to lock 5 test nodes. At the same time, the other workers will be picking up jobs from queues and executing them, where they will try to lock machines required by their jobs. If these other jobs require fewer machines, they will be locked before our job locks machines, irrespective of the priority of the job. Since we have many workers per queue, they will always be looking to lock machines, chances of having 5 free machines for our worker to lock are very less. It's very likely that our job isn't able to start execution at all.

The Solution

The above problems can be solved if we have complete knowledge of the system when we are locking machines. Since execution does not need to know about the priority of the job, that can be executed independently without the entire system's knowledge. Separating the locking mechanism from the job execution such that it is done in a centrally controlled environment is key.

To achieve the above, we replaced the existing worker framework with a dispatcher.

Dispatcher

The dispatcher sits at the interface of the beanstalk queue and the workers, which are now called the job supervisors. Instead of various workers watching a queue, a single dispatcher keeps track of the queue. Instead of locking machines in the initial tasks upon execution of a job, the dispatcher locks the required number of machines and then invokes the job's execution by starting job supervisors. The dispatcher forgets about the job now and moves onto the next job in the queue.

Supervisors

Job supervisors are similar in working to workers with some key differences. While workers execute many jobs, one after the other, supervisor are job-specific. They supervise the execution of the job and then exit when it's completed. Unlike workers, they also don't have direct access to the priority queue to pull jobs from but are called with a specific job's information. They are responsible for reimaging the target test nodes and unlocking/nuking them after a job finishes executing all its tasks.

In case of job timeout, that is when a job exceeds the maximum execution time for a job, workers simply nuked target test nodes without archiving logs from these machines. This was a huge pain for developers to debug without the test logs to go through. Supervisors archive test logs from target test nodes in case of a job timeout before nuking machines.

How does it all come together?

Dispatcher watches the queue, takes a job from it, waits for the required number of test nodes to be available, locks them and invokes job supervisor. The supervisor reimages the test nodes, constructs the Teuthology cmd and invokes it. Teuthology proceeds to run all the tasks and exits after the job execution is complete. At this point, the supervisor unlocks/nukes targets depending on the job status.

If the required number of test nodes are not available the dispatcher waits for them to be freed. During this time, no other job starts execution as dispatcher is busy. As other jobs finish execution and unlock nodes, the dispatcher is able to lock the required number of nodes, invoke that job's execution and move on to the next job in the queue. This ensures that jobs that have a higher requirement for test nodes to execute are not left waiting too long and at the same time maintains the priority order of the jobs.

Following PRs enable the above working:

What lies ahead?

Same Teuthology workspace for each run.

While scheduling suites, we can specify what Teuthology branch to run the suite against. But in the middle of a run if the said branch is updated, some jobs are run using the old branch ref and some using the latest ref. This makes debugging harder since the error could be in the latest branch update which just adds onto places to look for bugs, not to mention one will only look into it if they realize in the first place that Teuthology might have updated mid-run. We want to avoid this nastiness by enabling Teuthology workspace for each run such that each job in a suite is run using the same ref.

Enable re-prioritizing jobs

We would like to add the ability to re-prioritize jobs after they are scheduled. This would be used to re-prioritize jobs, to bump them up in the priority queue after they have spend maximum waiting time in a certain priority level. This will ensure that even low priority jobs are executed eventually and not end up waiting in the queue indefinitely.

Running Teuthology locally

If you read through the blog till now, you might be wondering how does one go about setting up Teuthology locally. The simple answer is you can't, for now at least. There are a lot of pieces that need to be figured out to run Teuthology locally. But doing so will be extremely useful in shortening the development time for Teuthology.

No ride is void of challenges

There were a few instances this past summer when I felt overwhelmed.

The first hurdle was setting up the development environment. As I mentioned earlier, running a development setup for Teuthology was a big task. Recently no such drastic changes had been made to Teuthology, and so no one had deployed Teuthology for development that I could ask questions to. We were all figuring it out together. It was a good exercise and made me appreciate the extensive lengths people go to make sure their project's local development setup experience is as smooth as running a single command to have everything up and running.
I had a medical emergency in the second week of GSoC due to which I wasn't able to work for an entire week. This interruption came at a time when I was already struggling with trying to setup Teuthology. It got to me a little but my mentors were extremely supportive.
I was already done with the main aims of the project by the time we approached the last month of GSoC. Teuthology is also used in SUSE and they have some restrictions due to which we had to change the solution's implementation details entirely. On the first blush, this seemed like a challenge I won't be able to pull off. But in the end, I completed it well within the timeline and I'm happy about the change since the updated implementation is much more elegant.

Acknowledgements

I was studying the internal working of other task queues and message queues when I came across this project in GSoC while helping out a friend find a suitable project for them. The project really piqued my interest. I was elated to be selected in GSoC because I would be able to work on something I'm genuinely curious about.

I can safely say that this past summer has been an immensely gratifying learning experience. All of this has been possible only because of my rockstar mentors Neha Ojha and Josh Durgin. They not only helped me throughout the process but made sure to appreciate my work and gave me constructive feedback at every turn. I'm extremely grateful for the opportunity to learn from such talented people. Thank you to the wider Ceph community for bearing with me while I got up to speed with way things work. And last but not the least, thank you so much Kyrylo Shatskyy for helping me out in the last leg of this journey!

Thank you so much for this wonderful summer :)

What, Why and How: Outreachy 101

Shraddha Agrawal — Sat, 25 May 2019 18:52:20 +0000

I recently got selected for Outreachy with Fedora and thought I should document the entire process for other curious souls looking to participate! :)

Note: This article by no means provides a 'hack' or definite steps to get into Outreachy. These are just my thoughts on what worked for me.

The What

As mentioned on the official Outreachy website:

Outreachy provides internships to work in Free and Open Source Software (FOSS). It's a three month remote internship.

Outreachy internships provide a wide varity of domains to work in. Projects may include programming, user experience, documentation, illustration, graphical design, or data science.

If you've heard about Google Summer of Code, Outreachy in many ways is similar to it with some key differences:

While Outreachy is an internship, GSoC is not considered as an 'internship'.
Outreachy is open to only under-represented individuals in tech whereas GSoC is a student centric program.
Outreachy is biannual while GSoC runs once per year.
Stipends for Outreachy are fixed while GSoC stipends may vary depending on where you reside.

Who can apply?

Outreachy has very strict eligibility criteria. Two of the most important ones are:

Anyone who faces under-representation, systemic bias, or discrimination in the technology industry of their country.
You must have atleast 49 consecutive days free from full-time commitments.

You can check the complete eligibility criteria on the official website. The initial application verifies your eligibility. More on the same is detailed below.

When can you start?

Outreachy runs twice a year, mid year and end of year. For the mid year round, the application period starts from February and the internship concludes August while for the end of year round, the application period starts from September and the internship concludes in March. If you're a student, depending on where you school is, you can apply to either of these rounds.

The Why

Not that all that is out of the way, essentially the most important question to ask here is why do you want to apply for Outreachy. Summer of code have gained a considerable amount of traction in the last few years and if you're based in India, there are GSoC/Outreachy fellows left right and centre. Don't do it because everyone you know is doing it, peer pressure be damned.

Here are some takeaways from my experience that hopefully will help you choose if Outreachy is the best bet for you!

Quality abundance of learning. Contributing to a FOSS project is a huge step up from a personal project. You get to learn plentiful and get a taste of what it's like to be a part of a big system with various working parts. Contributing to the project not only enhances your skill set as a software developer but also endows you with essential soft skills.
Gateway into Open Source Software development. If you're passionate about OSS and have been looking to contribute to the same, Outreachy sets an effective precursor. Getting involved with OSS can be tricky and often daunting for beginners, as was the case with me. Just knowing who to ask help from is barrier breaker. Outreachy provides specific steps on how to connect with your mentor and contact them in case you feel stuck or are in need of help. This brings me to my third most important takeaway.
Build you network. Irrespective of whether you get selected or not, while contributing to the project you get to know and work with experienced and highly talented software developers and your fellow skilled applicants. Interacting and working with OSS veterans across the globe teaches you the current best practices, brings new and enlightening perspectives into focus and exposes you to opportunities you might not have stumbled upon elsewhere.
Credibility If you're like me who regularly needs a reminder to hush that small voice in the back of your head telling you that you're not skilled enough, or often doubt your own capabilities as a software developer, Outreachy will provide that much needed boost to your self-esteem.
Monetary Perks Outreachy provides $5500 USD to each intern as an incentive to get involved in OSS. Additionally, Outreachy also provides $500 USD travel stipend to attend workshops/conferences.

The How

Now if most of the above reasons seem fitting, let's get into the stepping stones towards getting selected for Outreachy.

The Initial Application

The first step during the application period is the initial application. This is for the Outreachy organisers to verify your eligibility. It requires you to answer four essay based question and some others to verify your time availability.

This must be taken extremely seriously. Only after the initial application is accepted, the projects are made visible to the applicants.

Selecting a Project

Going through the list of projects I had these important points in mind:

Look for a project that will enhance your current skill set and simultaneously nudge you to expand your knowledge spectrum.
Question to self: Do you see yourself contributing to the project long after the internship is done? If the answer is yes, you're good to go.
Don't go after an organisation because you've heard too much about it. Again, a tag won't help you if your heart's not in it.

Don't select too many projects and juggle between them as that will only divide your time that could instead be devoted to understanding one/two projects and giving it your best input.

Contribution Period

Outreachy requires you to solve atleast one issue to submit a final application. After introducing yourself on the community's preferred mode of communication, go on the hunt for your first issue to solve.

This is one of the most crucial period based on which mentors decide whether you're fit for the project or not. Some key points to keep in mind are:

Solve as many bugs as possible. Don't just go for issues that fall under your spectrum of knowledge, try to solve issues that urge you to step outside your comfort zone and learn new things on the fly. This will help mentors see your ability to learn and adapt according to project requirements. Be involved not only by contributing but also open issues when you come across a bug.
Be an active member of the community. Communicate effectively with your mentors. Follow the etiquettes to communicate on a public platform. Keep them updated of your progress and any obstacles you're facing.
Do not ask questions to your mentors unless you've done enough research about the same. Respect their time and efforts and use Google and Stack Overflow in abundance.
Help others out as much as possible. I can't stress this enough, don't make this into a dirty race where you belittle your co-applicants or the likes. Genuinely help other contributors and build a supporting community.
Most of all, have fun! Strive for those Eureka moments when you solve bugs or add new features. Give yourself a pat on the back, you've earned it!

Note: If you're like me and big code-base seem daunting, remember, you don't need to know it in its entirety. Start small and build from there.

The Final Application

The final application requires you to record all your contributions mention your experience working with the organisation, give details of any past projects you've made/FOSS organisation you've worked with and supply a timeline mapping out the course of action for the next three months of the internship. Each and every step is important for outreachy organisers to understand if you're suitable for the internship so give plenty time to each and mention any and all details.

After submitting the final application, you can keep on contributing to the project and maintain a steady communication with the mentors.

And, you're IN!

That's it. As I start my internship with Fedora I can safely say, all it needs is consistent efforts by a passionate being.

You can read about my experience during the application period here.

Hope you found something valuable here and most of all the drive to apply for that Summer of Code you've been wanting to since eternity. Quit questioning, take a leap of faith and dive right in!

Feel free to reach out about any doubts concerning the application process, I'll love to help you out! :)