I Built a Unix Shell from Scratch in C. Here's How It Works.

As a second-year CS student, I recently gave in to the urge to build a Unix shell from scratch. You know, that black screen you type ls into fifty times a day without ever thinking about what is actually happening underneath. Honestly, this is the perfect chance for me to start writing about technical stuff. Since you are already here, how about joining me? Come on, grab a coffee or some chai, I will be waiting :D

Ok. Here is how we are doing this. We are going to take a look at how a terminal actually works under the hood. We will start with the main loop, see how raw text turns into running programs, and replace the mystery of the terminal with our own C code. While doing that, we will talk about some OS theory so everything makes sense.

This whole thing started as a CodeCrafters challenge. What was supposed to be a quick weekend project turned into me staring at dup2() man pages at 2 AM, wondering why my pipes were freezing the whole terminal. Good times.

Oh, and BTW, we are using C (please do not kill me in my sleep, I know pointers are scary, I promise it will be fun though!). If you want to keep the repo open in another tab while reading, the code is at github.com/shiva-kar/SXSHELL-unix-like-shell.

Concepts Covered

• The REPL (Read, Eval, Print Loop)
• String parsing and POSIX quoting rules
• Creating processes (fork, execv, waitpid)
• Environment variables ($PATH searching)
• File descriptors and I/O Redirection (dup, dup2)
• Pipes (|)

Assumptions

• Two-command pipes only: The shell handles ls | grep txt, not massive 5-stage pipelines.
• Output redirection only: We handle >, >>, 2>, and 2>>. No < input redirection yet.
• No signal handling: Pressing Ctrl+C will kill the whole shell, not just the command running inside it.
• Fixed argument limit: MAX_ARGS is set to 64. Nobody is typing 65 arguments by hand anyway.
• No scripting: No if/else blocks, no background & jobs, and no $() subshells.

Sections

• What even happens when you type a command?
• The project layout and the REPL (main.c)
• Parsing: harder than it looks (parser.c)
• Finding programs in $PATH (executable.c)
• Running programs with fork and exec (process.c)
• Builtins (builtins.c)
• Redirection: swapping file descriptors (redirection.c)
• Pipes (pipeline.c)
• The Finale!

What even happens when you type a command?

We use the terminal every day, right? So how does typing ls -la and hitting Enter actually put text on your screen? Let's take a quick tangent to talk about the command lifecycle.

1. The shell reads the string ls -la from your keyboard (stdin).
2. It splits that string into separate words (tokens): ls and -la.
3. It checks if ls is a "builtin" command that the shell handles by itself. (Nope.)
4. It searches through the folders listed in your $PATH environment variable until it finds the actual binary file at /usr/bin/ls.
5. It calls fork() to clone itself into a child process.
6. The child calls execv() to replace its own memory with the /usr/bin/ls program.
7. The parent shell waits for the child to finish using waitpid(), and then prints the $ prompt again.

Side note: Why can't cd just be a normal program inside /usr/bin like ls? Think about it for a second. If the shell forked a child process and ran cd /tmp inside the child, the child would change its directory and immediately exit. Meanwhile, your main parent shell would still be sitting in the exact same folder you started in. Useless, right? That is why cd, exit, and pwd have to run directly inside the shell process itself. We call these builtins.

The project layout and the REPL (main.c)

To make sure we know where everything lives, here is what the folder structure looks like:

src/
  main.c          # The main loop (REPL)
  parser.c        # Splits input into tokens, handles quotes
  builtins.c      # echo, cd, pwd, type, exit
  executable.c    # Searches $PATH for programs
  process.c       # fork() + execv() + waitpid()
  redirection.c   # The > >> 2> 2>> stuff
  pipeline.c      # Pipes (the | stuff)

Ok, we can officially start with main.c (please take a quick look at the code :3 )

char *buffer = NULL;
size_t size = 0;
ssize_t len;
setbuf(stdout, NULL); // do not remove this, trust me

while (true) {
    printf("$ ");
    if ((len = getline(&buffer, &size, stdin)) != -1) {
        buffer[strcspn(buffer, "\n")] = '\0';

        char **argv = parse_arguments(buffer);

        int saved_stdout = -1;
        int saved_stderr = -1;
        hijack_redirections(argv, &saved_stdout, &saved_stderr);

        if (argv != NULL && argv[0] != NULL) {
            if (has_pipe(argv)) {
                execute_pipeline(argv);
            } else {
                int status = handle_builtin(argv);
                if (status == -1) {
                    free_arguments(argv);
                    break;
                } else if (status == 0) {
                    char *exec_path = find_executable(argv[0]);
                    if (exec_path != NULL) {
                        execute_program(exec_path, argv);
                        free(exec_path);
                    } else {
                        printf("%s: command not found\n", argv[0]);
                    }
                }
            }
        }

        free_arguments(argv);
        restore_redirections(saved_stdout, saved_stderr);
    }
}
free(buffer);

I think you may already have an idea about what that code is doing. Print a prompt, read a line, run it, repeat. Still, a couple of lines in there gave me a headache when I first wrote them:

• setbuf(stdout, NULL): In C, stdout usually waits until it sees a newline \n before it actually prints anything to your screen. Because our prompt is just "$ " with no newline, getline() would pause and wait for keyboard input before the $ even showed up on the terminal! Turning off buffering with setbuf fixes this. I wasted way too much time figuring that out.
• getline(): This function is great because it allocates and resizes the memory buffer for you automatically. You just have to free(buffer) when the loop ends.
• strcspn(buffer, "\n"): When you press Enter, getline() leaves the \n character at the end of your string. This line finds that \n and overwrites it with \0 so our command names do not have newlines attached to them.
• handle_builtin() return values: -1 means the user typed exit (so we break out of the loop), 1 means the builtin ran fine, and 0 means "this is not a builtin, go look in $PATH."

Parsing: harder than it looks (parser.c)

At first I thought, "I will just split the input on spaces using strtok(), easy."

Well, try running this in a terminal:

$ echo "hello   world"

If you split on every space, you get three pieces: echo, "hello, and world". That is totally wrong. You are supposed to get two arguments: echo and hello world with the three spaces kept intact and the quotes removed.

So strtok() had to go. Instead, parse_arguments() loops through the string one character at a time:

char **parse_arguments(char *buffer) {
    char **args = malloc(sizeof(char *) * MAX_ARGS);
    int args_count = 0;
    bool is_single_quotes = false;
    bool is_double_quotes = false;

    char *current_word = malloc(strlen(buffer) + 1);
    int word_len = 0;

    for (int i = 0; buffer[i] != '\0'; i++) {
        char c = buffer[i];

        // backslashes outside quotes: next char is literal
        if (c == '\\' && !is_single_quotes && !is_double_quotes) {
            char next_char = buffer[i + 1];
            if (next_char != '\0') {
                current_word[word_len++] = next_char;
                i++;
            }
            continue;
        }

        // backslashes inside double quotes: only escape specific chars
        if (c == '\\' && is_double_quotes) {
            char next_char = buffer[i + 1];
            if (next_char == '"' || next_char == '\\' ||
                next_char == '$' || next_char == '\n') {
                current_word[word_len++] = next_char;
                i++;
                continue;
            }
        }

        // toggle quotes on and off
        if (c == '"' && !is_single_quotes) {
            is_double_quotes = !is_double_quotes;
            continue;
        }
        if (c == '\'' && !is_double_quotes) {
            is_single_quotes = !is_single_quotes;
            continue;
        }

        // space means split, ONLY if we are not inside quotes
        if (c == ' ' && !is_single_quotes && !is_double_quotes) {
            if (word_len > 0) {
                current_word[word_len] = '\0';
                args[args_count++] = strdup(current_word);
                word_len = 0;
            }
            continue;
        }

        current_word[word_len++] = c;
    }

    if (word_len > 0) {
        current_word[word_len] = '\0';
        args[args_count++] = strdup(current_word);
    }

    free(current_word);
    args[args_count] = NULL;
    return args;
}

parse_arguments() is our protagonist right now:

• We keep two booleans: is_single_quotes and is_double_quotes. Whenever we hit a quote character, we flip the matching boolean.
• If we see a space ' ', we check those two booleans. If both are false, we finish the current word, copy it into args using strdup(), and reset word_len to 0. If either boolean is true, we are inside quotes, so we just add the space to current_word like any normal letter.
• For backslashes \, POSIX rules are a little weird. Outside quotes, \ makes the very next character literal (echo hello\ world becomes hello world). Inside single quotes, backslashes do nothing at all. Inside double quotes, \ only escapes ", \, $, and \n. For anything else like echo "\a", the backslash stays in the string.

Note: Because we malloc the args array AND strdup() each individual string inside it, freeing args alone will leak memory. In free_arguments(), we have to loop through the array and free(args[i]) for every word before calling free(args).

Finding programs in $PATH (executable.c)

Alright, the user typed ls. We parsed it, checked our builtins, and ls is not in there. Now we have to find where the ls binary actually lives on the computer.

If you run echo $PATH in your terminal, you will see a list of folders separated by colons, like /usr/local/bin:/usr/bin:/bin. Our function find_executable() checks each of those folders one by one:

char *find_executable(const char *command) {
    char *path_env = getenv("PATH");
    if (!path_env)
        return NULL;

    char *path_copy = strdup(path_env);
    if (!path_copy)
        return NULL;

    const char *delimiter = ":";
    char *dir = strtok(path_copy, delimiter);
    char full_path[1024];

    while (dir != NULL) {
        snprintf(full_path, sizeof(full_path), "%s/%s", dir, command);
        if (access(full_path, X_OK) == 0) {
            char *result = strdup(full_path);
            free(path_copy);
            return result;
        }
        dir = strtok(NULL, delimiter);
    }

    free(path_copy);
    return NULL;
}

Look closely at char *path_copy = strdup(path_env). Why did we make a copy of path_env before giving it to strtok()?

Because strtok() modifies the string you pass into it by replacing the : characters with \0 null bytes. And getenv("PATH") gives you a direct pointer to your program's actual environment variables! If you run strtok() directly on getenv("PATH"), you will corrupt your own $PATH variable after running one command. Making a copy with strdup() saves us from breaking our own environment.

Inside the loop, snprintf glues the folder path and the command name together (like /usr/bin/ls), and access(full_path, X_OK) checks if the file exists and has execute permissions. If it returns 0, we found our binary.

Running programs with fork and exec (process.c)

Now that we found /usr/bin/ls, how do we run it without closing our shell? This happens in process.c:

int execute_program(const char *exec_path, char *const argv[]) {
    pid_t pid = fork();
    if (pid < 0) {
        perror("Error: fork() failed");
        return -1;
    }
    if (pid == 0) {
        if (execv(exec_path, argv) == -1) {
            perror("Error: execv() failed");
            exit(EXIT_FAILURE);
        }
    }
    if (pid > 0) {
        int status;
        waitpid(pid, &status, 0);
    }
    return 0;
}

Here is what is happening step by step:

• fork() clones the current process. After this line runs, there are literally two copies of our shell running at the same time. In the newly created child process, pid is 0. In the original parent process, pid is the process ID number of the child.
• In the child (pid == 0), we call execv(exec_path, argv). This tells the OS to wipe the child's memory and load /usr/bin/ls into it instead. If execv() works, it never returns because the child process has turned into ls. If it fails and returns -1, we call exit(EXIT_FAILURE) right away so we do not end up with two copies of our shell running at once.
• In the parent (pid > 0), we call waitpid(). This pauses the parent until the child finishes running ls. Without waitpid(), the parent would print the next $ prompt immediately while ls is still printing filenames, messing up the terminal output.

Why are fork() and execv() two separate steps instead of one run_program() function? Because that pause between fork() and execv() lets us change the child's input and output files before the new program starts. That is the exact trick we use for redirection and pipes.

Builtins (builtins.c)

As we discussed earlier, commands like cd, exit, and type have to run inside the shell process itself.

static char *builtins[] = {"type", "quit", "exit", "echo", "pwd", "cd"};

Here is how type works inside handle_builtin(). When you run type echo or type ls, it checks the builtins list first, and if it is not there, it uses our find_executable() function from earlier:

char *target = argv[1];
bool is_found = false;

for (int i = 0; i < LENGTH(builtins); i++) {
    if (strcmp(target, builtins[i]) == 0) {
        fprintf(stdout, "%s is a shell builtin\n", target);
        is_found = true;
        break;
    }
}

if (!is_found) {
    char *exec_path = find_executable(target);
    if (exec_path != NULL) {
        fprintf(stdout, "%s is %s\n", target, exec_path);
        free(exec_path);
        is_found = true;
    }
}

if (!is_found) {
    fprintf(stdout, "%s: not found\n", target);
}

And here is cd. It just grabs the path (or looks up HOME if you typed cd ~ or just cd) and passes it to the chdir() system call:

if (argv[1] != NULL) {
    const char *path = argv[1];
    if (strcmp(path, "~") == 0) {
        path = getenv("HOME");
        if (path == NULL) {
            fprintf(stderr, "cd: HOME not set\n");
            return 1;
        }
    }
    if (chdir(path) != 0) {
        fprintf(stdout, "cd: %s: No such file or directory\n", argv[1]);
    }
} else {
    const char *home = getenv("HOME");
    if (home != NULL) {
        chdir(home);
    }
}

Redirection: swapping file descriptors (redirection.c)

Ok, this is where things get really interesting. When you type ls > output.txt, ls has no clue that output.txt exists. ls just writes its output to File Descriptor 1 (STDOUT_FILENO), which normally goes to your terminal screen.

To make ls write into a file instead, we swap what File Descriptor 1 points to before running the command.

1. Save a backup of the real terminal output using dup(STDOUT_FILENO).
2. Open output.txt using open().
3. Use dup2(fd, STDOUT_FILENO) to point File Descriptor 1 at output.txt.
4. Close the extra file descriptor from step 2.

Here is the code in redirection.c:

void hijack_redirections(char **argv, int *stdout, int *stderr) {
    char *output_file = NULL;
    int cut_index = -1;

    for (int i = 0; argv[i] != NULL; i++) {

        // stdout overwrite: > or 1>
        if (strcmp(argv[i], ">") == 0 || strcmp(argv[i], "1>") == 0) {
            output_file = argv[i + 1];
            if (cut_index == -1)
                cut_index = i;
            if (output_file != NULL) {
                if (*stdout == -1)
                    *stdout = dup(STDOUT_FILENO);
                int fd = open(output_file, O_WRONLY | O_CREAT | O_TRUNC, 0644);
                dup2(fd, STDOUT_FILENO);
                close(fd);
            }
        }
        // 2>, >>, and 2>> follow the same pattern using STDERR_FILENO and O_APPEND
        // ...
    }

    // hide the > and filename from the command!
    if (cut_index != -1) {
        argv[cut_index] = NULL;
    }
}

And once the command finishes, restore_redirections() puts your terminal screen back to normal using the backup we saved:

void restore_redirections(int stdout, int stderr) {
    if (stdout != -1) {
        dup2(stdout, STDOUT_FILENO);
        close(stdout);
    }
    if (stderr != -1) {
        dup2(stderr, STDERR_FILENO);
        close(stderr);
    }
}

Two important details here:

• If you forget to restore stdout after the command runs, every future command you type will also write into output.txt, and your terminal will look completely frozen.
• Notice argv[cut_index] = NULL at the bottom of hijack_redirections(). If you run ls -la > output.txt, we have to hide the > and output.txt from ls. Otherwise, ls will think you are asking it to list a folder literally named >! Putting NULL at cut_index cuts the array short so ls only sees ["ls", "-la", NULL].

Pipes (pipeline.c)

Pipes are where I spent the most time debugging at night. When you run ls | grep txt, two programs run at the same time: ls writes data into a pipe, and grep reads from the other end of that pipe.

The pipe(pipefd) system call gives us two file descriptors in an array:

• pipefd[0] is the read end.
• pipefd[1] is the write end.

Let's look at execute_pipeline():

void execute_pipeline(char **argv) {

    // find the | symbol and split the array into left and right commands
    int pipe_index = 0;
    for (int i = 0; argv[i] != NULL; i++) {
        if (strcmp(argv[i], "|") == 0) {
            pipe_index = i;
        }
    }
    argv[pipe_index] = NULL;
    char **right_argv = &argv[pipe_index + 1];

    int pipefd[2];
    if (pipe(pipefd) != 0) {
        perror("Error: Pipe failed");
    }

    // left child: writes to the pipe
    pid_t pid = fork();
    if (pid == 0) {
        dup2(pipefd[1], STDOUT_FILENO);
        close(pipefd[1]);
        close(pipefd[0]);
        if (handle_builtin(argv) == 1) {
            exit(0);
        } else {
            char *executable = find_executable(argv[0]);
            execv(executable, argv);
            exit(1);
        }
    }

    // right child: reads from the pipe
    pid = fork();
    if (pid == 0) {
        dup2(pipefd[0], STDIN_FILENO);
        close(pipefd[1]);
        close(pipefd[0]);
        if (handle_builtin(right_argv) == 1) {
            exit(0);
        } else {
            char *executable = find_executable(*right_argv);
            execv(executable, right_argv);
            exit(1);
        }
    }

    // parent: close both pipe ends before waiting!
    close(pipefd[0]);
    close(pipefd[1]);
    wait(NULL);
    wait(NULL);
}

All the eyes should be on those last four lines in the parent:

close(pipefd[0]);
close(pipefd[1]);
wait(NULL);
wait(NULL);

Why does the parent have to close both ends of the pipe when the parent is not even using the pipe?

Because Linux keeps a pipe open as long as any process still has its write end (pipefd[1]) open. Since the parent created the pipe before calling fork(), the parent also holds an open reference to pipefd[1]. When the left child (ls) finishes and exits, the right child (grep) waits for an End-of-File (EOF) signal so it knows there is no more text coming. If the parent forgets to close pipefd[1], the OS thinks the parent might still write something into the pipe later! So grep sits there waiting forever, wait(NULL) in the parent waits forever for grep, and the whole shell locks up. Closing both descriptors in the parent before calling wait() fixes the deadlock.

The Finale!

Now for a moment of truth, let's build and run this bad boy.

You will need GCC (or Clang), CMake, and Readline. On Ubuntu or Debian, run:

$ sudo apt install build-essential cmake libreadline-dev # i know, apt install isn't C code, shut up >:-(

Now clone the repo and compile it:

$ git clone https://github.com/shiva-kar/SXSHELL-unix-like-shell.git
$ cd SXSHELL-unix-like-shell
$ cmake -B build
$ cmake --build build
$ ./build/shell

Running ./build/shell gives us our prompt, and we can test out everything we just talked about:

$ echo "hello   world"
hello   world
$ type echo
echo is a shell builtin
$ type ls
ls is /usr/bin/ls
$ ls > output.txt
$ cat output.txt | grep src
src
$ pwd
/home/shiva/SXSHELL-unix-like-shell
$ cd /tmp
$ pwd
/tmp
$ exit

You can find the full code here: https://github.com/shiva-kar/SXSHELL-unix-like-shell

Now what?

I guess a question or two may have crossed your mind while reading this. I encourage you to clone the code, break stuff, and see what happens. If you want to build your own shell from zero, here is the order I suggest doing it in:

1. Print $, read input with getline(), and print it back.
2. Split input on spaces with strtok() and get exit and echo working.
3. Search $PATH and run commands with fork() and execv().
4. Add cd and pwd.
5. Replace strtok() with the character-by-character quote parser.
6. Add > file redirection using dup() and dup2().
7. Add pipes (|).

The shell we created is still pretty basic and has plenty of things missing if you want to keep adding to it:

• Handling multi-pipe commands like cat file | grep x | wc -l
• Input redirection (<)
• Catching Ctrl+C signals so it does not kill the main shell
• Growing MAX_ARGS dynamically with realloc() instead of stopping at 64
• Checking if open() returns -1 in redirection.c before calling dup2()
• Command history and tab completion

The list is long! If you want to read more about the system calls we used, the Linux man pages (fork, execv, pipe, dup2) and Stephen Brennan's Write a Shell in C guide are great resources for that.

Well, I hope you had fun reading this. Thanks a lot for getting to this point and have a nice day!