Engineering for your customers

Steven Richards — Thu, 18 Jan 2018 18:01:49 +0000

In the tech industry there is a misguided tenet of 'move fast, break things' and always trying to use the 'hottest' stack/language/etc. that you see on Hacker News. Unfortunately, that rarely translates to customer happiness in using your product/service.

Over the years I've come across some key concepts while engineering systems for customer happiness and reliability:

Be able to identify who your customers are.
Customers are part of your system.
Understand what 'customer impact' means in an outage.

Many of these points are illustrated as questions to ask yourself and your team.

Customers drive your tenets

What do we build?

Do we build a website with public consumers as customers?
Do we build a platform that businesses build on?
Do we build internal tools for our colleagues?

You need to put yourself in the shoes of your customers (and perhaps their customers too!). Once you do, you can start to shape your engineering tenets around how customers use what you build.

What are the tenets of your service or product?

If you build a platform that businesses use, you'll discover that reliability and robustness are likely more important than 'moving fast and breaking things'. This doesn't just apply to large enterprises; If you're a startup you probably don't have a massive user-base where you can afford to lose many customers.

Can our customers tolerate outages, errors, delays? Do we need to build for robustness, speed, uptime, or all of the above?

These aren't points to define once, they should be evolving with your product, much like features and designs do. As your product evolves customers will use it differently and you should adapt as necessary.

Your customers are part of your system

In most block diagrams of a system I typically see some cloud labeled 'internet', but I rarely see 'customers'.

You know very well how your application reacts to latency, but how do your customers react to latency, errors, retrying?

I've seen many outages caused by failing to account for how customers interact with a service. While you might have great retry behavior and timeouts throughout your system, it's easy to overlook how your customers deal with an unreliable service (see: thundering herd).

Make sure you have customer-appropriate throttles, monitoring, error pages, and documentation.

Defining customer impact

In many post-mortems you'll hear statements like 'our webservers served 65,000 HTTP 500 errors from 10:04 to 10:30', which fails to tell the story of the customers who received those errors.

Did that result in them having to retry a purchase and giving up?
Did we lose customers?
Did that break our customer's application and business?

If you can't answer those questions then there are gaps in your system's monitoring.

It's fundamental to have system-level metrics for all of your components, but going beyond that, you need to be able to measure customer impact.

If you build a platform that your customers build their business on, consider being proactive in helping them. It's meaningful to reach out to customers while they're experiencing an outage, even if it's not your fault. You know your system the best and perhaps there is advice you can share so they can leverage your system better during their outage.

If you notice a customer of yours had a bad outage, are there things you can build that they could utilize in order to prevent outages?

Final thought

I've used the word customer 30 times in this post because at the end of the day most businesses require paying customers to exist. Understanding and engineering for your customers is incredibly important.

"The single most important thing is to make people happy. If you are making people happy, as a side effect, they will be happy to open up their wallets and pay you." - Derek Sivers

The Walking Dead (but with processes)

Steven Richards — Tue, 13 Jun 2017 16:47:10 +0000

What's a zombie?

When interviewing system engineers a common answer for "what is a zombie process?" is "A process which is dead and doesn't have a parent" or something like "you can kill it with kill -9 <pid>". There seems to be a misunderstanding of zombie processes. Let's delve into what exactly a zombie is, and how they should be handled by Linux.

A zombie process is a process which has exit()'d and whose parent has not called wait()/waitpid() syscall against its process id (PID). Meaning, the process exited and left a status code for the parent to read, but the parent has yet to read it from the process table. Every process that terminates is briefly a zombie, they become an issue when they stick around too long.

Some key points:

They cannot be "killed" - SIGKILL, SIGTERM are ineffective as the process is already dead.
They are not orphan processes.
They can be removed via orphaning where init adopts the process (by killing the parent).
They utilize no resources and simply occupy a process table entry.
Every process, upon termination, is a zombie until their parent process calls wait().

This little snippet demonstrates how a zombie can be created:

int main(void)
{
    pid_t pid;
    int status;

    if ((pid = fork()) < 0) {
        // Failed to fork
        exit(1);
    }
    // Child process starts execution from here, child pid == 0
    if (pid == 0)
        // All the child does is exit
        exit(0);

    // Parent continues from here. A zombie roams for 100 seconds..brains..
    sleep(100);

    // After 100 seconds, the parent removes the zombie by calling wait()
    pid = wait(&status);

    return 0;
}

A zombie shows up as <defunct> in the output of ps, when running the above snippet.

129274  \_ ./zombie 
129275    \_ [zombie] <defunct>

This means the parent (PID 129274) hasn't called wait()/waitpid() on PID 129275. Until it calls wait, 129275 will eat up a process table entry. Since they take up no resources, they typically aren't an issue.

The problem emerges when too many exist and you run out of PIDs, meaning no more new processes (ex: can't ssh to a server that can't spawn you a shell..)

You can see the max PIDs with the following command.

# cat /proc/sys/kernel/pid_max
131072

It's a good idea to monitor process count on your machines. If you have a process leaking zombies that is long-running it will eventually fill the process table which could take down your service, prevent remote-access, etc.

A way you can remove a zombie is by killing the parent of the zombie, which would then re-parent the process to init (PID 1) who should periodically call wait() on its child processes.

Zombies usually appear if a program has a flaw where it doesn't call wait() on its children. what if the process got stuck in some infinite loop and never called wait()? what if that process was named init?

When init doesn't do its job...

A while ago I came across a number of servers that had thousands of zombie processes. My team had simply been rebooting these boxes to clear them up, but I became curious after noticing something odd:

All the zombie processes had init as a parent!

This means that processes were exiting, and init (as their parent), was never calling wait(). Additionally, if you were to kill a zombies parent to make init the parent, init would do nothing to help you.

init, as one of its core functions, should be calling wait() on its child processes to clear out any zombies, so what was happening?

% sudo strace -p 1 -r -s 500
Process 1 attached - interrupt to quit
     0.000000 write(8, "init: serial-ttyS main process ended, respawning\r\n", 50) = ? ERESTARTSYS (To be restarted)
   273.565117 --- SIGCHLD (Child exited) @ 0 (0) ---
     0.000062 write(4, "\0", 1)         = 1
     0.000066 rt_sigreturn(0x4)         = 1
     0.000046 write(8, "init: serial-ttyS main process ended, respawning\r\n", 50) = ? ERESTARTSYS (To be restarted)
     0.003033 --- SIGCHLD (Child exited) @ 0 (0) ---
     0.000026 write(4, "\0", 1)         = 1
     0.000048 rt_sigreturn(0x4)         = 1
     0.000039 write(8, "init: serial-ttyS main process ended, respawning\r\n", 50) = ? ERESTARTSYS (To be restarted)
     7.825845 --- SIGCHLD (Child exited) @ 0 (0) ---

stracing showed init stuck in a loop trying to write to a tty device, and each write getting ERESTARTSYS back (meaning: please re-attempt that write). Init had no mechanism to handle that error, so it gets stuck in an infinite loop.

As for why it was getting ERESTARTSYS, the tty it was writing to in this case was a serial tty. grepping the serial tty driver code for ERESTARTSYS found:

/**
 *  tty_send_xchar  -   send priority character
 *
 *  Send a high priority character to the tty even if stopped
 *
 *  Locking: none for xchar method, write ordering for write method.
 */

int tty_send_xchar(struct tty_struct *tty, char ch)
{
    int was_stopped = tty->stopped;

    if (tty->ops->send_xchar) {
        tty->ops->send_xchar(tty, ch);
        return 0;
    }

    if (tty_write_lock(tty, 0) < 0)
        return -ERESTARTSYS;

    if (was_stopped)
        start_tty(tty);
    tty->ops->write(tty, &ch, 1);
    if (was_stopped)
        stop_tty(tty);
    tty_write_unlock(tty);
    return 0;
}

Writing to a serial tty takes a lock in the form of atomic_write_lock. Searching for info regarding this lock I found a bug:

>>  Possible unsafe locking scenario:
>>
>>        CPU0                    CPU1
>>        ----                    ----
>>   lock(&tty->termios_rwsem);
>>                                lock(&tty->atomic_write_lock);
>>                                lock(&tty->termios_rwsem);
>>   lock(&tty->atomic_write_lock);

Wrapping all this back together found a new bug due to a different bug.

If the serial tty gets into a deadlock, any logging that init tries to perform against it will get init stuck into a loop of write -> ERESTARTSYS -> write ...

If init stays in this state long enough, without the tty resetting, then zombies will pile up and you'll run out of PIDs.

How should init handle child processes?

Looking at the sysvinit source in src/init.c we see a couple different ways init handles it child processes. Here we see init establishing a signal handler for SIGCHLD

SETSIG(sa, SIGCHLD,  chld_handler, SA_RESTART);

void chld_handler(int sig)
        CHILD           *ch;
        int             pid, st;
        int             saved_errno = errno;

        /*
         *      Find out which process(es) this was (were)
         */
        while((pid = waitpid(-1, &st, WNOHANG)) != 0) {
                if (errno == ECHILD) break;
                for( ch = family; ch; ch = ch->next )
                        if ( ch->pid == pid && (ch->flags & RUNNING) ) {
                                INITDBG(L_VB,
                                        "chld_handler: marked %d as zombie",
                                        ch->pid);
                                ADDSET(got_signals, SIGCHLD);
                                ch->exstat = st;
                                ch->flags |= ZOMBIE;
                                if (ch->new) {
                                        ch->new->exstat = st;
                                        ch->new->flags |= ZOMBIE;
                                }
                                break;
                        }
                if (ch == NULL) {
                        INITDBG(L_VB, "chld_handler: unknown child %d exited.",
                                pid);
                }
        }

Examining chld_handler, this will execute if init gets a SIGCHLD signal which is sent to a parent when a child of theirs dies. Init handles zombies in a push model here by calling waitpid() and flagging the process as a zombie for later cleanup

ch->flags |= ZOMBIE;

Earlier, I said when init gets stuck in this state that it would not be able to reap any of its own children who exit, but it has a signal handler for this? so what's going on?

The initlog() function blocks all signals while logging:

/*
 *      Re-establish connection with syslogd every time.
 *      Block signals while talking to syslog.
 */
sigfillset(&nmask);
sigprocmask(SIG_BLOCK, &nmask, &omask);
openlog("init", 0, LOG_DAEMON);
syslog(LOG_INFO, "%s", buf);
closelog();
sigprocmask(SIG_SETMASK, &omask, NULL);

So when we call initlog() anywhere in the main loop, and call syslog(LOG_INFO, "%s", buf); we hit our earlier bug. syslog() is respecting ERESTARTSYS from write() so we get stuck in here and we block all signals (including SIGCHLD).

Outside of the chld_handler's mechanism for reaping zombies, the rest of the main loop handles a variable called family which stores all the child processes of init. It loops over these looking for processes to kill and/or reap from the process table if they have died.

DEV Community: Steven Richards