DEV Community: Patrick Lafferty

What to do when your debugger lies to you

Patrick Lafferty — Tue, 24 Jul 2018 00:12:47 +0000

"My program doesn't work, is this a bug in GCC/Clang/$TOOL?"

Everyone's had this moment before. You've been stumped on a problem for hours and are so sure that your tools are broke. You've quadruple-double-extra checked your logic. Have I found an error in $POPULAR_TOOL? You inquire on some message board. Then a grizzled greybeard appears, points out the flaw in your logic, and remarks that it's never a compiler error. Generally that's true, it's almost always your fault. That's not to say that compilers and debuggers and such don't have bugs, but after being used for many years by tens/hundreds of thousands of people, you really have to saunter off the beaten path to find one.

Well, I found one.

The Beaten Path

I'm writing a 32-bit loader for a 64-bit microkernel. I'm emulating it with QEMU, and debugging with GDB. I'm focusing on robustness and correctness, I want a complete mental picture at every stage. So I started off at the very beginning debugging a simple assembly function calling a C++ one:

boot:
    cli
    mov esp, stack_top
    push ebx ; grub stores an address to multiboot headers in ebx
    call startup


extern "C"
void startup(uint32_t foo) {
    doStuffWithFoo(foo);
}

Let's stop at the call to startup and examine the registers.

EBX (but it's called RBX... hint for later) is 0x3087b0, and we can see it at the top of the stack as well. Now let's step inside startup...

Ooh... yeah, uhm I'm gonna have to go ahead and sort of disagree with you there.

Something's very wrong here, a few instructions (four to be exact) ago we had one value, a few instructions later we have a completely different value. The four instructions aren't the culprit, its just the standard function prologue. And why was GDB showing us x86_64 registers?

Use the (other) force, Luke!

With GDB acting weird I decided to try the classic printf debugging. Well, print-to-VGA-buffer-f, since printf itself hasn't been built yet. At any rate, the function has been tested and been known to be correct; whatever it prints on the screen is the real value of foo. Survey says...

3087B0

Well now, that's what I was expecting. While we're here, let's ask what GDB thinks of foo after printing it to the screen for fun:

GDB, how can I put it nicely...

Unearthing the conspiracy

The story gets worse my friends. Through countless minutes of googling and some friends in #osdev, I've found that both QEMU and GDB have conspired to make mixed-mode debugging infeasible. Some time between versions 2.8 and 2.9 QEMU added a certain patch so that no matter what mode the cpu is currently running (real, protected, long), it will always send the full x86_64 register file through the GDB stub. GDB was patched to never change its register file size after startup. So when first connecting gdb to a x86_64 QEMU instance GDB sees that QEMU sends a full x86_64 register file and just goes with it, even though CPUs don't start in long mode (but to be fair it doesn't know it's debugging an OS at bootup).

Meanwhile GDB thinks since we're using the 64-bit RF that it should use x86_64 bit calling convention. At least that's what I'm assuming based on my observations. System V i386 ABI says that we pass parameters by pushing them on the stack, which I did right before calling the startup function. But SYS-V x86_64 ABI says that the first couple of parameters are passed in registers rdi, rsi, rdx, rcx, r8, r9 and then any extras are pushed on the stack. So when asking GDB for foo's value its looking in the wrong place, hence the junk value.

We can make a test for this. Inside boot.s right before the call instruction, we can set EDI to some value like 0xcafebabe. Guess what value GDB now says foo is?

The value is 0, the address is Cafebac6. I have no explanation for why it's offset by 8, if you do please let me know. The final nail in the coffin is to try with qemu-system-i386 instead of system-x86_64, without making a single code change. To no one's surprise, GDB prints the correct value for foo.

Now its important to note here that QEMU is still emulating the program correctly. Even though we can't see it in GDB, the program gets the correct value and can display it. The problem is just how GDB views the state of the program. And it's only a problem for people that want to debug the startup code of an operating system which you would rarely do even if making your own OS, and only when the startup code isn't 64-bit and your emulator's CPU is.

This affects dozens of us. DOZENS!

Summary

When something doesn't look right and your debugger appears to give you incorrect results, fall back on the other tools in your debugging toolbox. Good old reliable printf or logging, rendering an image of an intermediary or data buffer to the screen for GPU problems. Or create some test that modifies the environment in a unique way that demonstrate whether something did or did not happen. Examples include sleeping a thread for X seconds to determine whether X is small or big, blinking a light, emitting morse code through the speakers etc.

If you have any questions or comments, corrections or suggestions, criticisms et cetera, I’d love to get in touch by email or in the comments below. I’m always open to learning new things and correcting bad things. If you'd like to read more posts by me, why not come check out my blog.

SMP is as easy as 1-2-3

Patrick Lafferty — Fri, 20 Jul 2018 19:32:02 +0000

(And other hilarious jokes you can tell yourselves)

Originally published at https://patricklafferty.ca/blog/2018/07/20/smp-is-as-easy-as-1-2-3/

Symmetric multiprocessing (SMP) is one of the big features I'm currently exploring for Saturn. I implemented multi-tasking very early on as that was a key part in the microkernel design, however up until now everything ran on a single core. Its the classic concurrency vs parallelism comparison, just because things are running concurrently doesn't mean they're running simulatenously. Adding SMP support requires careful upfront design (whoops) or lots of refactoring down the road. Before making any further progress with Saturn I wanted to make sure it had a solid core foundation, so I started to refactor now while that was still feasible.

In the beginning there was the bootstrap processor

I was really curious how multicore actually worked and how you set it up. I mean we all know about multithreading and multiple processes and distributing work across cores, but how does the OS actually orchestrate that? What happens when the computer first boots up?

At first glance it seems pretty straightforward. At startup the firmware runs some tests and does some behind the scene work, and eventually passes control over to the bootloader which then loads the OS. When you have a multicore CPU, the firmware picks one of the working cores as the "main" CPU, called the "bootstrap processor" (BSP), and considers the remaining cores (or hardware threads in the case of SMT) as "application processors" (AP). It starts up each AP in real mode, clears the interrupt flag and then halts each AP. Meanwhile the BSP carries on towards entering your kernel.

Interprocessor Interrupts

At this point all of our APs are halted (not running any code), and wont respond to normal interrupts/exceptions. Luckily there is a special type of interprocessor interrupt or IPI that is used to facilitate communication between CPUs. By sending a certain sequence of IPIs we can wake up our APs and get them to do useful work. To do this we need to make use of a trampoline.

Remember that I said APs start in real mode? 16-bits of non-protected, non-paging, segmented fun. Our BSP at this point is either in 32-bit protected or 64-bit long mode with paging enabled. We can't just have the BSP allocate a bunch of data structures and hand them off to the AP, because it can't address them because we can only access the first 1MB of RAM. A trampoline allows us to get around this.

A trampoline is a small bit of code + data that configures an AP to be useful. It basically does a readers digest version of what the kernel had to do - setup a GDT, enable paging etc - without really caring about the details. Ie it just fills out the bare minimum needed to get to protected mode, because once there we'll replace that stuff or use what the kernel already setup.

The BSP allocates a small chunk of memory at a low address which stores important addresses for the AP to use, among them a stack pointer, a function address to call once in protected mode, and a flag it continually will check for the AP to modify to indicate progress. The BSP and AP communicate with each other by modifying the flag and waiting for it to change to a specific value before continuing on. Once this dance is finished the AP halts and waits for the BSP to finish initializing all the remaining APs and then send it an interrupt.

Locad nda lok (or: Lock and load)

Now that we have multiple processors that want to process things we need to synchronize anything that could be accessed from multiple tasks. Due to the micro part of the microkernel, there aren't many data structures that could be modified by multiple processes. Said structures were global variables of convenience, and now I had the opportunity to redesign them.

Due to the nature of these kernel variables and how they were accessed, I decided to implement spinlocks. A spinlock does as advertised: "can we acquire this lock? No? Okay, can we acquire this lock now? No? Okay,...". Essentially it just continually checks a condition in a loop, best used if you expect to wait only a few hundred cycles. Mutexes and more elaborate mechanisms have their place, the middle of an interrupt service routine ain't one of them.

The spinlocks are used in a few key areas to synchronize access to a task's mailbox, a scheduler's run queue, or in task creation. What happens if you don't use locks? The header for this section is a lighthearted example, more realistically the entire system crashes and you're sitting in GDB with no clue what happened.

Schedulers, Directors, and Bears, oh my

Originally Saturn had a single scheduler which controlled what task was currently running. Adding more CPUs to the picture meant supporting multiple schedulers, each with their own separate run queues and block queues. The first design iteration had a complicated process when a scheduler noticed a blocked task could be run. It would examine all of the other schedulers to see if any could accept this task and then inject the task onto that scheduler's run queue. Likewise if the scheduler ran out of tasks it would try to steal from another.

This worked, in a kinda sorta every second tuesday of each month way. I decided on a more clean approach by adding a second level scheduler: the director. A director is the main/meta scheduler that schedules the schedulers. Instead of schedulers each having their own blocked queues that had to be managed individually, the director would be the single point of control for blocked tasks. Schedulers still have their own sleep queues for tasks that sleep for a very short amount of time.

This simplified things greatly while being a lot safer. Now there was a single point of entry for scheduling tasks: give it to the director, and it will find the best scheduler for you.

Aftermath

This post marks the 800th commit to Saturn, making it one of the longest running projects I've started. Currently I'm taking on the 64-bit rewrite and will be adding extensive testing along the way. You can see the source code, or to learn more visit saturn-os.org.

Part two of my series The Joy of Operating Systems, where I write about my experience writing my own operating system Saturn.

Part 1: The Joy of Operating Systems

Let There Be Light

Patrick Lafferty — Wed, 27 Jun 2018 17:54:01 +0000

Originally published at https://patricklafferty.ca/blog/2018/06/24/let-there-be-light/

Recently I decided to revisit my personal site and overhaul its design. These are the lessons I learned, the logic of the old and new designs, Paint scribbles on screenshots, and general advice.

Lesson One: Let there be light

Every editor I use is configured for a dark colour scheme. As I spend most of my time in that environment, I grew accustom to it and preferred dark schemes, which my personal design reflected. As it turns out, many people prefer dark-on-light colour schemes to light-on-dark when reading general text. It can be less straining and easier to read for a large segment of the population. With that in mind I set out to do a complete makeover.

The “Before” picture:

Note how it goes from a dark blue background to an even darker black text background. The “After” picture:

Note how it goes from a light blue background to an almost white text background. Text is almost always black on a bright white, when the contrast is lower with black-on-bright blue I make sure the font size is large to make up the difference.

Looking at the numbers[1], the old dark design had a contrast ratio of 5.48:1, while the new one varies from 12.97:1 to 17.51:1. The mentioned black-on-blue parts have a contrast ratio of 5.97:1, so there is a marked improvement all around.

1: https://webaim.org/resources/contrastchecker/

Lesson Two: Pick a proper colour palette

I don’t know much about colour theory except that I like blue, and some now irrelevant details about radiosity. I know enough to know that you can’t just pick colours willy-nilly, or use a large number of them without rhyme or reason. Websites like Paletton[2] help you create a colour palette using a variety of different schemes like monochromatic, triads and tetrads. For non-artists like me, spend a lot of time deciding on what best fits your design, then take the handful of colours it gives you and use only those, except for perhaps small accents.

2: paletton.com

Lesson Three: Design for your audience first, yourself second

Look at this.

I designed for me, for what I thought looked cool. But the result was atrocious: 300px separates each menu item, so your eyes have to move far from one to the next. The header is 400px tall and the only reason for that is to fit in more blue. It’s a waste of space.

Think about who you are trying to reach out to. If we’re being honest, for me that primarily would be recruiters right now. Or, if not recruiters per se, those otherwise trained in the job offerin’ arts before straitened circumstances forced them into a life of aimless wanderin’. Anywho, recruiters (like all people) are busy people. Get. To. The. Point.

No nonsense, have a typical navbar typically where one expects it. Also, make the landing page for your blogs show a short list of excerpts instead of putting all of them in one long page, so people can quickly see if the post interests them or not. The difference is staggering:

It took so long to scroll the old site that I half expected the ladder song from MGS3 to start playing. I also broke up long-form text with a distinct colourful double-border header to aid scanning. Finally, get to the point. If you know your audience is looking for key details, don’t hide them inside paragraphs. Pull them out and emphasize them. Spend only 5 seconds reading and compare:

With:

In 5 seconds, could you pick up anything interesting in the old paragraph, when it was among half a dozen others right above and below it? If you want someone to notice something, help them notice it. Make the name more prominent, and put the key information directly underneath on the left side of the screen where their eyes will naturally be. Separate the hot and cold content with colour. Then if they’re interested, they can read the bigger description.

Lesson Four: Give space to breathe

Empty space is one of the most important parts of the overall appearance. Generally if you have the space to spare, give yourself generous padding and margins. Keep content away from its borders, make it comfortable: think subways at 2pm vs rush hour. This lets the colours come out and do their thing. Pay attention to your line height so that text is easier to read.

Lesson Five: Guide the eye with shape and colour

I settled early on a simple geometric style with a two-tone colour contrast. This allowed me to play around with shape-outside and clip-path (straying a bit from lesson three), but it also influenced the user experience. Look at each section: sections with a paragraph have a quadrilateral with a large bold header leading the eyes towards the text (see last screenshot above). Sections with lists (experience, projects) have a shape guiding the user downwards.

The most important part, the call to action, has a literal arrow directing the user to act. It fits the overall angular design, but it has a function too.

Fin

That’s the end of the lessons. If you have any questions or comments, corrections or suggestions, criticisms et cetera, I’d love to get in touch by email or here. I’m always open to learning new things and correcting bad things.