DEV Community: heygauri

Concluding My Outreachy Internship Journey: Embracing the Linux Kernel

heygauri — Thu, 24 Aug 2023 09:43:15 +0000

Introduction:

Hello everyone,

Today, I want to share my thoughts as I finish my Outreachy internship. I'm Sumitra Sharma. For the past four months, I've been a part of the May 2023 Outreachy group, learning about the Linux Kernel. I worked on a project called "Converting kmap() and kmap_atomic() call sites to kmap_local()".

A Journey Fueled by Passion and Learning:

This journey has been really exciting. I started as a contributor for a month and then had a special three-month learning experience. I met many awesome people who are very passionate about what they do. Seeing their hard work inspired me to work harder too.

Walking Alongside Linux Kernel Titans:

I got to work with amazing people who know a lot about the Linux Kernel. They work really hard and are very good at what they do. Working with them was a privilege and taught me a lot.

Not Goodbye, Just See You Later:

Even though my internship is ending, I'm not saying goodbye. I want to keep contributing to the Linux Kernel. During these months, I've gained more than technical skills. I've learned about communication and soft skills, which are essential in any field. These skills have boosted my confidence and will be useful in my future endeavors. On the technical side, I've achieved a lot. I've become comfortable with Git, building and installing the kernel, a task that initially seemed daunting. Understanding memory management, a complex topic, is another milestone I've achieved during this time.

Mentors: My Guides:

I had mentors who helped me a lot. They were like guides, always there to help me when I had questions. Thanks to Alison, Ira, Fabio, and Deepak for believing in me and giving me this chance. I really enjoyed working with them.

Gratitude to Mentors and Outreachy:

This project helped me see what I want to do in the future. It wouldn't have been possible without the help of my mentors and the Outreachy program. They trusted me and that made a big difference. This experience showed me how exciting open-source is.

Thank you for being part of my journey.
Happy Coding :)

Conversion from kmap() to kmap_local_page()

heygauri — Fri, 14 Jul 2023 18:20:27 +0000

Hello readers! I am currently in my seventh week of the Outreachy internship, where I am working as a Linux-Kernel intern on the project “Converting kmap() and kmap_atomic() call sites to kmap_local_page()”.

This blog is in continuation with my previous blog https://dev.to/heygauri/outreachy-series-think-about-your-audience-754. I have created another blog explaining a patch so that you all can get knowledge of the work I am doing.

The patch I will explain is categorized as ‘S’ (small) based on its difficulty level and implementation time. The link to the patch is given below.

https://lore.kernel.org/all/20230610175712.GA348514@sumitra.com/T/

In the file lib/test_hmm.c, the two functions “dmirror_do_read” and “dmirror_do_write” calls kmap(page)/kunmap(page). But the kmap() function has been deprecated. Read the blog https://dev.to/heygauri/outreachy-series-think-about-your-audience-754 to know the reasons behind the depreciation of kmap() and kmap_atomic() in favour of kmap_local(). This patch is created to replace the kmap() call with kmap_local_page().

Note you cannot directly replace the kmap()/kunmap() with kmap_local()/kunmap_local(). It is not that simple. Conversions to kmap_local_page() must follow the mapping restrictions imposed on kmap_local_page().

Conversion from kmap() to kmap_local_page():

There are two checkpoints.

Firstly, to check whether the pages are from high memory.

Drivers, filesystems, and other non-core code are expected to call into the highmem system when a page could be mapped in high memory. Many drivers call kmap*() “just to be sure”.

But part of this conversion is to determine if they really need to call kmap() or kmap_local_page() at all and eliminate those calls if the page is known to not be mapped in highmem. The below link to a patch created by my mentor Fabio is an example of such a case:

https://lore.kernel.org/all/20220704140129.6463-1-fmdefrancesco@gmail.com

Secondly, to check, the return pointer validity should be local only.

While kmap_local_page() is significantly faster than kmap(), for the highmem case, it comes with restrictions about the pointers validity. Contrary to kmap() mappings, the local mappings are only valid in the context of the caller and cannot be handed to other contexts. This implies that users must be absolutely sure to keep the use of the return address local to the thread which mapped it.

Also, we do not have to check for preemption and page faults in this conversion because kmap() creates mappings which are permanent and propagated across all the CPUs of the system and does not require the calling process to be in an atomic context like kmap_atomic() hence it does not disable preemption, page faults or migration.

Here, the pages can come from highmem and the mappings are also kept thread local in both the functions. Refer to the official highmem documentation for more information https://docs.kernel.org/mm/highmem.html

Apart from this, we can see that the code wrapped inside the kmap()/kunmap() in both functions is just giving a call to memcpy(ptr, tmp, PAGE_SIZE); and memcpy(tmp, ptr, PAGE_SIZE);

This memcpy() function copies a memory region to another using pointers and the specified fixed length. So the overall changes will look like

@@ static int dmirror_do_read(struct dmirror *dmirror, unsigned long start,
    for (pfn = start >> PAGE_SHIFT; pfn < (end >> PAGE_SHIFT); pfn++) {
...

-       tmp = kmap(page);
+       tmp = kmap_local_page(page);
-       memcpy(ptr, tmp, PAGE_SIZE);
-       kunmap(page);
+       kunmap_local(tmp);

...
}

@@ static int dmirror_do_write(struct dmirror *dmirror, unsigned long start,
    for (pfn = start >> PAGE_SHIFT; pfn < (end >> PAGE_SHIFT); pfn++) {
...

-       tmp = kmap(page);
+       tmp = kmap_local_page(page);
-       memcpy(tmp, ptr, PAGE_SIZE);
-       kunmap(page);
+       kunmap_local(tmp);

...
}

But we have some helper functions like memcpy_to_page() and memcpy_from_page() which replace kmap/mem*()/kunmap pattern with
memcpy_to/from_page(). Below is the function definition of these functions in which you can see the calls to kmap_local_page()/kunmap_local().

static inline void memcpy_from_page(char *to, struct page *page,
                    size_t offset, size_t len)
{
    char *from = kmap_local_page(page);

    VM_BUG_ON(offset + len > PAGE_SIZE);
    memcpy(to, from + offset, len);
    kunmap_local(from);
}

static inline void memcpy_to_page(struct page *page, size_t offset,
                  const char *from, size_t len)
{
    char *to = kmap_local_page(page);

    VM_BUG_ON(offset + len > PAGE_SIZE);
    memcpy(to + offset, from, len);
    flush_dcache_page(page);
    kunmap_local(to);
}

Additionally, remove the now unused variable void *tmp from both functions. This gives us our final changes for the patch.

@@ static int dmirror_do_read(struct dmirror *dmirror, unsigned long start,
    for (pfn = start >> PAGE_SHIFT; pfn < (end >> PAGE_SHIFT); pfn++) {
...

        void *entry;
        struct page *page;
-       void *tmp;

        entry = xa_load(&dmirror->pt, pfn);
        page = xa_untag_pointer(entry);
        if (!page)
            return -ENOENT;

-       tmp = kmap(page);
-       memcpy(ptr, tmp, PAGE_SIZE);
-       kunmap(page);
+       memcpy_from_page((char *)ptr, page, 0, PAGE_SIZE);

...
}

@@ static int dmirror_do_write(struct dmirror *dmirror, unsigned long start,
    for (pfn = start >> PAGE_SHIFT; pfn < (end >> PAGE_SHIFT); pfn++) {
...
        void *entry;
        struct page *page;
-       void *tmp;

        entry = xa_load(&dmirror->pt, pfn);
        page = xa_untag_pointer(entry);
        if (!page || xa_pointer_tag(entry) != DPT_XA_TAG_WRITE)
            return -ENOENT;

-       tmp = kmap(page);
-       memcpy(tmp, ptr, PAGE_SIZE);
-       kunmap(page);
+       memcpy_to_page(page, 0, (char *)ptr, PAGE_SIZE);

...
}

Lastly, compile and test any changes you make in the Linux kernel code before sending it to the maintainers.

I hope you learn something from this blog post. Let me know if you have queries related to this patch. Thank you for reading!

Outreachy series: “Think about your audience”

heygauri — Sat, 01 Jul 2023 09:40:25 +0000

Hello readers! I am currently in my fifth week of the Outreachy internship, where I am working as a Linux-Kernel intern on the project "Converting kmap() and kmap_atomic() call sites to kmap_local_page()".

This blog aims to introduce you to my project and the significance of memory management in modern operating systems.

As part of the Memory Management system's Highmem subsystem, changes are being made to replace the older kmap() and kmap_atomic() functions with the new kmap_local_page() or kmap_local_folio() functions. These functions allow the mapping of physical memory pages into the kernel address space.

The deprecation of kmap() and kmap_atomic() is due to their drawbacks, such as high cost, limited mapping space, the need for a global lock for synchronization, and the possibility of the process sleeping in the absence of free slots. Similarly, kmap_atomic() disables preemption and migration, which can lead to latency, jitters, and system imbalance.

To overcome these issues, the solution lies in using kmap_local_{page, folio}(). These functions offer faster performance, provide thread-local and CPU-local mappings, handle page faults within a local kmap region, and preserve preemption by saving and restoring mappings during context switches.

For more detailed information, you can refer to the documentation available at https://docs.kernel.org/mm/highmem.html.

In my work, I am evaluating various call sites to determine how effectively kmap_local() can be utilized and the deprecated kmap() and kmap_atomic() calls can be eliminated. While some conversions simply involve replacement, others require thorough code analysis and potentially even rearchitecting of the codebase.

NEXT: Conversion from kmap() to kmap_local_page() https://dev.to/heygauri/conversion-from-kmap-to-kmaplocalpage-2hc8

Outreachy series: "Everbody Struggles !"

heygauri — Fri, 16 Jun 2023 10:44:09 +0000

Hello readers. I am in my third week of the Outreachy internship. I am working as a Linux-Kernel intern on the project—"Converting kmap() and kmap_atomic() call sites to kmap_local_page()".

As an intern, stepping into a new professional environment can be exciting and challenging. It's a unique phase where one's learning curve is at its steepest. It's essential to recognize that struggles are integral to the journey toward growth and success. In this blog, I will discuss my struggles in these three weeks.

Since my first day of the internship, I have encountered various problems and concepts that I need to familiarize myself with. Additionally, I have been learning how to tackle these challenges. Such things are unfamiliar to me, but I am aware that they have been unfamiliar to others during their time as well. However, they managed to overcome them, and I believe I can do the same.

During my first week, my primary focus was reading and understanding the project's concepts. Understanding the official documents proved to be quite challenging. Specifically, I have been reading the highmem kernel documentation since my contribution phase, and I still struggle to comprehend everything written within it. This experience has made me realize the significant value of technical documents as they provide authenticity. The content within these documents is concise, so paying attention to every detail is essential to understand them fully. Consequently, it requires a substantial amount of reading. While these lengthy readings are beneficial in bridging the knowledge gap, they also consume a considerable amount of time, leaving less time for actual hands-on work.

Moving on to my second week, I began working with the code. Seeing the code was both exciting and intimidating. Firstly, understanding the code itself proved to be a challenge. Additionally, finding ways to update the code according to the demands of my project required significant effort and dedication.

Another aspect I am currently working on is overcoming my hesitation to respond publicly while interacting with the Linux kernel maintainers in the mailing lists. It is a process that involves stepping out of my comfort zone and actively engaging with the community.

Despite the difficulties and uncertainties, I am committed to persevering and learning from these experiences. Each struggle I encounter serves as an opportunity for personal growth and development. By embracing these challenges head-on, I am confident that I will gain valuable skills and knowledge throughout my internship journey.

Outreachy series: "Introduce yourself"

heygauri — Fri, 02 Jun 2023 09:15:52 +0000

Hello readers. I am sumitra sharma, and I would like to introduce myself as an Outreachy Linux Kernel Intern. I have been selected as an intern for the May 2023 cohort. I am excited to work with the fantastic mentor’s Ira, Fabio, and other Linux maintainers.

I will be documenting my internship journey in this blog series.
The blogs will be as follows:

Learn about Outreachy at https://www.outreachy.org/

I am pursuing M.Tech in Computer Science & Information Security from Malaviya National Institute of Technology, Jaipur. I have been selected as an Outreachy intern for the Linux project — “Converting kmap() and kmap_atomic() call sites to kmap_local_page()”.

My core values are hard work, dedication, optimistic & hopeful.

I start anything by preparing myself to focus on doing the work, and the rest is not in my hands. Often people complain and hesitate to take action. Instead, one should focus on the steps; after some time, one will realise how things come into place.

Once I start on a task, I put my everything into it. I built a strong dedication towards my work. I get satisfaction when I produce a quality product with my hard work.

Being optimistic becomes an essential part of my journey. During work, we face failures; sometimes, things don’t go how we want them to be despite the daily hard work we put in. At such moments, being kind to yourself and others(co-workers) is vital to remain self-motivated and maintain good relationships. If we can hope, then why not for something good only? It will give us more energy and an optimistic viewpoint towards our work.

Working in FOSS is a transformative journey that combines my passion for technology, collaboration, and making a positive impact. The vibrant community, freedom to innovate, continuous learning, and tangible real-world outcomes motivate me to be part of the FOSS movement. It’s an opportunity to contribute to a global community, shape the future of technology, and leave a lasting legacy.

The open nature of FOSS projects fosters an environment where knowledge is freely shared, barriers are removed, and everyone has the opportunity to contribute. This ethos of openness promotes inclusivity and leads to breakthroughs that benefit society as a whole.

To me, Outreachy came as an excellent opportunity. I was motivated by the opportunity to contribute to open-source projects that have a global impact. Outreachy is a platform to give back to the community, collaborate with others, and work on projects that align with their interests.