I'm Building an OS From Scratch in Rust — And It Already Has a Desktop, Filesystem, and Networking
A deep dive into CottonOS: a hobby OS written entirely in Rust with a custom filesystem, GUI, preemptive multitasking, and experimental in-kernel TLS.
There's a specific kind of madness that strikes some programmers: the urge to not just use an operating system, but to build one. From the bootloader all the way up to the window manager. No Linux underneath. No BSD. Just you, your CPU, and a whole lot of page faults.
That's what CottonOS is.
I've been building it as a hobby project and learning experiment — a full OS kernel written in Rust for x86_64, featuring a custom filesystem I call CottonFS, a graphical desktop environment with working windows and a taskbar, preemptive multitasking, and an experimental in-kernel HTTPS stack. It's nowhere near production-ready, but it boots, it renders windows, it reads and writes files, and it can make HTTP requests — entirely from scratch.
The long-term vision is bigger: I want CottonOS to eventually run an on-device LLM, lightweight and fast, fully self-contained. That's miles away, but every subsystem I write brings it closer.
This post is a technical walkthrough of what I've built, the decisions I made, the hard parts, and where I want to take it. I'm actively looking for contributors who want to dig into low-level systems programming.
Why Rust for an OS?
The obvious answer: memory safety without a garbage collector. In kernel code, a use-after-free isn't just a bug — it's a privilege escalation or a hard crash with no recovery. Rust's ownership model catches entire classes of these mistakes at compile time.
But the real answer is more personal: I wanted to learn Rust deeply, and there's no deeper environment than bare metal. No allocator. No standard library. No OS to catch your mistakes. Just no_std and a custom target spec pointing at hardware.
The kernel is 95.4% Rust. The remaining ~3% is a small boot_stub.asm — a Multiboot2 header, minimal GDT, early page tables, and a long mode entry point. Once we're in 64-bit mode, it's Rust all the way down.
Boot Sequence: Getting to Rust as Fast as Possible
CottonOS uses GRUB with Multiboot2 as the bootloader. GRUB loads the kernel ELF at the 1MB physical mark and hands off to boot_stub.asm, which does just enough to get into long mode:
- Sets up a stack
- Initializes a flat 64-bit GDT
- Creates identity-mapped PML4 page tables covering the first 1GB
- Sets
CR4.PAE, loads PML4 intoCR3, setsEFER.LME, enables paging viaCR0.PG - Far-jumps into the 64-bit code segment
- Calls
_start64— our Rust entry point
From there, _start64 initializes serial output first (so we have debug logging even if the screen isn't up yet), parses the Multiboot2 info structure to find the framebuffer and memory map, then brings up each subsystem in order: interrupts → memory → filesystem → processes → drivers → GUI.
Memory Management: Three Layers
Physical Allocator
A bitmap allocator that tracks 4KB frames. On boot, we parse the Multiboot2 memory map, mark everything the kernel and framebuffer occupy as used, and the rest is available. Simple and predictable — no fragmentation at the physical layer.
Virtual Memory
Standard x86_64 4-level paging: PML4 → PDPT → PD → PT. The kernel sits in the lower half for now (higher-half support is ready but not yet wired up). Pages carry the standard permission flags: Present, Writable, User-accessible, No-Execute. On-demand mapping means we only create page table entries when they're actually needed.
Kernel Heap
Built on top of the linked_list_allocator crate. Initial size is 4MB at 0x02000000, expandable to 16MB. This registers as the global allocator, which unlocks alloc — meaning Vec, String, Box, Arc, and the rest of the heap-allocated collection types work normally inside the kernel.
CottonFS: A Custom Persistent Filesystem
This was one of the most involved parts to build. CottonFS is a simple but real persistent filesystem with a design influenced by ext2.
On-Disk Layout
| Block Range | Content |
|---|---|
| 0 | Superblock |
| 1–31 | Inode bitmap |
| 32–63 | Data block bitmap |
| 64–127 | Inode table |
| 128+ | Data blocks |
Block size is 4096 bytes. The magic number is 0x43544653 ("CTFS"). The superblock tracks total/free blocks and inodes, mount count, last mount time, and the root inode (always inode 1).
Inodes are 128 bytes on disk, holding file metadata and 12 direct block pointers plus one indirect pointer (giving a max file size of ~4MB per file — enough for a hobby OS).
VFS Layer
Above CottonFS sits a VFS abstraction using Rust traits:
pub trait FileSystem: Send + Sync {
fn root(&self) -> Result<Arc<dyn Inode>, &'static str>;
fn sync(&self) -> Result<(), &'static str>;
fn stats(&self) -> Result<FsStats, &'static str>;
}
pub trait Inode: Send + Sync {
fn read(&self, offset: u64, buf: &mut [u8]) -> Result<usize, &'static str>;
fn write(&self, offset: u64, data: &[u8]) -> Result<usize, &'static str>;
fn lookup(&self, name: &str) -> Result<Arc<dyn Inode>, &'static str>;
fn create(&self, name: &str, file_type: FileType) -> Result<Arc<dyn Inode>, &'static str>;
fn unlink(&self, name: &str) -> Result<(), &'static str>;
fn readdir(&self) -> Result<Vec<DirEntry>, &'static str>;
}
This means mounting a DevFS at /dev was straightforward — it implements the same traits, presents /dev/null, /dev/zero, and /dev/random as virtual inodes, and the rest of the kernel doesn't need to know the difference.
Process Management and the Scheduler
CottonOS has a preemptive round-robin scheduler running at 1000Hz, driven by the PIT (Programmable Interval Timer).
Each process is represented by a Process struct with a PID, parent PID, name, priority (5 levels), state, and exit status. States are Ready, Running, Blocked, and Zombie.
Threads are supported at the kernel level with per-thread stacks and thread-local storage ready for use.
The scheduler sits on top of this: on each timer tick (every 1ms), the current process is preempted and the next ready process in priority order gets scheduled. There's also an idle process that runs when nothing else is ready.
The synchronization primitives — Mutex, Semaphore, and CondVar — are all custom-implemented. The mutex uses a wait queue so blocked threads yield the CPU rather than spinning, which actually matters at 1000Hz.
System Calls
The syscall interface is exposed via both int 0x80 and the syscall instruction. There are 40+ calls covering:
-
Process:
exit,fork,exec,wait,getpid,yield,sleep -
File I/O:
open,close,read,write,seek,stat,fstat -
Directory:
mkdir,rmdir,unlink,readdir,chdir,getcwd -
Memory:
brk,mmap,munmap -
System:
uname,time,uptime
Userspace is still largely planned — the interesting parts right now live in kernel space — but the interface is there and the handlers are implemented.
Networking: From ARP to In-Kernel TLS
The networking stack is the newest and most experimental part of CottonOS. Here's what's implemented:
Driver layer: An RTL8139 NIC driver that handles PCI discovery, initialization, and RX/TX frame handling. Works in QEMU with -nic user,model=rtl8139.
L2/L3: Ethernet framing, ARP (cache + request/reply), and IPv4 parsing and dispatch.
Transport: ICMP echo (ping), UDP send/receive, and basic TCP client connections.
Application layer: DHCP for auto-configuration, DNS resolver for A records, and two shell commands — httpget for plain HTTP and httpsget for HTTPS.
The httpsget command performs TLS negotiation entirely in-kernel. This is very experimental — the current implementation uses encrypted TLS transport but does not yet validate server certificates. Trust model hardening is planned, but getting the handshake and record layer working in a bare-metal Rust kernel with no OS underneath is already a significant milestone.
Stability work included bounded poll loops to prevent livelock, a non-blocking IRQ lock path, and ARP auto-resolution with retries on normal TCP/UDP flows so you don't have to manually ARP before connecting.
The Desktop Environment
This is what surprises people most when they see CottonOS: it has a real graphical desktop.
The framebuffer comes from GRUB's video mode. A double-buffer handles rendering to eliminate flicker. The font is an 8×16 bitmap font drawn character by character. Mouse input comes from the PS/2 driver.
Window Manager: Windows have titles, z-order, focus, and dragging via the title bar. The controls are macOS-style (red/yellow/green circles). Keyboard and mouse events route to the focused window.
Applications bundled in the kernel:
- Terminal — A full shell emulator with scrollable output history, backspace support, and working directory tracking. Runs all the shell commands listed below.
- File Manager — Graphical CottonFS browser with folder icons, file sizes, double-click navigation, back button with history, and scrollable listings.
- Text Editor — Multi-line editor with cursor positioning, arrow key navigation, undo/redo, file open/save/save-as dialogs, modified indicator, and line numbers.
- System Info panel — Real-time display of kernel version, memory stats, filesystem usage, process count, and uptime.
Shell Commands
The integrated shell (available both in the terminal app and as the kernel shell before GUI loads) supports:
| Category | Commands |
|---|---|
| Filesystem |
ls, cd, pwd, cat, touch, mkdir, rm, write
|
| System |
mem, df, ps, uptime, info
|
| Network |
net, netstats, arptable, arp, ping, dhcp, dns, setip, setmask, setgw, setdns
|
| TCP/UDP |
tcpconnect, tcpsend, tcprecv, tcpclose, httpget, httpsget, udpsend, udprecv
|
| Utilities |
echo, clear, help, sync
|
| Power |
reboot, halt
|
The Architecture at a Glance
+-------------------------------------------------------------------+
| User Space (Planned) |
+-------------------------------------------------------------------+
| System Call Interface |
+-------------------------------------------------------------------+
| Kernel Space |
| +-----------------+ +------------------+ +------------------+ |
| | GUI | | Shell | | Syscall Handler | |
| | Window Manager | | Command Parser | | | |
| | Desktop + Apps | | Builtins | | | |
| +-----------------+ +------------------+ +------------------+ |
| +-----------------+ +------------------+ +------------------+ |
| | Filesystem | | Process | | Memory | |
| | VFS / CottonFS | | Scheduler | | Physical/Virtual | |
| | DevFS | | Threads | | Heap | |
| +-----------------+ +------------------+ +------------------+ |
| +------------------------------------------------------------+ |
| | Drivers: Graphics | Keyboard | Mouse | ATA | Serial | PIT | |
| +------------------------------------------------------------+ |
| | x86_64: GDT | IDT | Paging | PIC | APIC | PIT | I/O Ports | |
| +------------------------------------------------------------+ |
+-------------------------------------------------------------------+
| Hardware |
+-------------------------------------------------------------------+
Building and Running It
You'll need rustup (nightly), nasm, qemu-system-x86_64, grub-mkrescue, xorriso, and x86_64-elf-ld.
Ubuntu/Debian:
sudo apt install qemu-system-x86 nasm grub-pc-bin xorriso mtools
rustup override set nightly
rustup component add rust-src llvm-tools-preview
Then:
git clone https://github.com/aryansrao/cottonos
cd cottonos
make iso # Build bootable ISO
make disk # Create 64MB persistent disk image
make run # Boot in QEMU
Networking quick start (inside CottonOS):
net
dhcp
dns example.com
httpget example.com /
httpsget example.com /
What's Next
The roadmap for CottonOS includes several tracks:
Near term:
- Higher-half kernel migration (page tables are ready, needs wiring)
- Certificate validation for the TLS stack
- ANSI color support and command history in the terminal
- Proper userspace ELF loading and isolation
Medium term:
- SMP (multi-core) support — the APIC code is stubbed in
- Extended filesystem features (larger file sizes via double/triple indirect blocks)
- More device drivers (USB, virtio)
Long term (the real goal):
- Running an on-device LLM entirely within CottonOS — lightweight inference, no external dependencies, fully self-contained
That last one is ambitious. But the point of a hobby OS is that ambition is the whole point.
I'm Looking for Contributors
CottonOS is open source under GPL-3.0. I'm one person working on this in my spare time, and there's a lot of surface area. If any of this sounds interesting — low-level Rust, OS internals, graphics, networking, filesystem design — I'd love contributions at any level.
Where to start:
- 📁 GitHub: github.com/aryansrao/cottonos
- Good first areas: ANSI color support in the terminal, command history, certificate validation for TLS, documentation improvements
- Bigger challenges: Higher-half kernel, SMP, virtio drivers, userspace ELF loading
Even just opening issues, leaving feedback on the architecture, or asking questions helps me think through the design better. OS development can feel lonely when you're doing it from scratch — the community is the part I'm most looking forward to building.
If you've built something similar, or just have thoughts on the architecture decisions — I'd genuinely love to hear them in the comments. This is a learning project at heart, and every perspective helps.
Top comments (0)