DEV Community

Cover image for Building a Security-First OS from Scratch: AtomicOS Journey
Ignacio Peña
Ignacio Peña

Posted on

Building a Security-First OS from Scratch: AtomicOS Journey

#osdev #security #lowlevel #assembly

Why Another Operating System?

Most hobby OS projects implement "security" with simple XOR operations. I wanted to prove we could do better - implementing real cryptographic primitives and hardware-enforced security from day one.

Meet AtomicOS: a security-first operating system that actually implements AES-128 encryption, SHA-256 hashing, and introduces a deterministic programming language called Tempo.

The Philosophy: Security > Stability > Performance

In Spanish, we say: "Seguridad primero, luego estabilidad y luego performance"

This principle guided every design decision. While most OS projects optimize for speed, AtomicOS deliberately sacrifices performance for security. Here's why that matters.

Key Technical Achievements

1. Real Cryptography (Not XOR!) 

// What most hobby OS do:
void "encrypt"(uint8_t* data, uint8_t key) {
    for(int i = 0; i < len; i++) {
        data[i] ^= key;  // This is NOT encryption!
    }
}

// What AtomicOS does:
void aes128_encrypt(const uint8_t* plaintext, const uint8_t* key, uint8_t* ciphertext) {
    // Real AES with:
    // - S-boxes (256-byte substitution table)
    // - MixColumns transformation
    // - Key expansion with round constants
    // - 10 rounds of processing
}
Enter fullscreen mode Exit fullscreen mode

The implementation includes the full AES S-box, proper key scheduling, and all the transformations required by the standard.

2. Memory Management Unit (MMU)

AtomicOS implements a complete MMU with:

  • 2-level paging (Page Directory + Page Tables)
  • 4KB page size
  • Hardware-enforced protection
  • Virtual memory mapping 
// Memory layout
0x00000000 - 0x003FFFFF : Kernel Space (4MB)
0x00400000 - 0xBFFFFFFF : User Space
0xC0000000 - 0xFFFFFFFF : Kernel Mirror (Higher Half)
Enter fullscreen mode Exit fullscreen mode

3. Tempo: A Deterministic Language

The most unique aspect is Tempo, a programming language where every function has a guaranteed Worst-Case Execution Time (WCET).

function calculate(x: int32, y: int32): int32 wcet 30 {
    let result = x * 2 + y;
    if result > 100 {
        return 100;
    }
    return result;
}
Enter fullscreen mode Exit fullscreen mode

The compiler analyzes the code and verifies it will execute within 30 CPU cycles. If it can't guarantee this, compilation fails.

Building the Kernel

The kernel evolution went through several versions:

v0.1-0.2: Performance-focused (failed security tests)
v0.3: Complete rewrite with security first
v0.4: Added MMU for hardware protection

Here's the boot sequence:

; Multiboot header for GRUB
section .multiboot
    dd 0x1BADB002              ; Magic number
    dd 0x00000000              ; Flags
    dd 0xE4524FFE              ; Checksum

; Entry point
section .text
global _start
_start:
    cli                        ; Disable interrupts
    mov esp, stack_top         ; Set up stack
    call kernel_main           ; Jump to C code
    hlt                       ; Halt if kernel returns
Enter fullscreen mode Exit fullscreen mode

Security Features Implemented

Stack Protection

Every function uses stack canaries to detect buffer overflows:

void function() {
    __asm__ volatile("" ::: "memory");  // Memory barrier
    // Function code
    if (canary != expected) {
        // Stack smashing detected!
        panic();
    }
}
Enter fullscreen mode Exit fullscreen mode

Memory Safety

  • Bounds checking on all operations
  • Guard pages before and after allocations
  • Secure wiping (3-pass with different patterns)
  • W^X enforcement (memory is either writable OR executable, never both)

Access Control

Every operation has a security context:

typedef struct {
    uint32_t uid;           // User ID
    uint32_t gid;           // Group ID
    uint32_t capabilities;  // Capability bits
    uint32_t ring_level;    // 0=kernel, 3=user
} security_context_t;
Enter fullscreen mode Exit fullscreen mode

Testing the OS

You can run AtomicOS yourself:

# Clone the repository
git clone https://github.com/ipenas-cl/AtomicOs.git
cd AtomicOs

# Build AtomicOS v0.4 with MMU
make -f Makefile.v4

# Run in QEMU
./run_mmu.sh
Enter fullscreen mode Exit fullscreen mode

Lessons Learned

  1. Security is expensive - Every security check adds cycles
  2. Perfect is the enemy of good - Start with basics, iterate
  3. Hardware is your friend - MMU provides real protection
  4. Honesty matters - Don't claim "military-grade" without proof

What's Next?

  • [ ] Ring 0/3 privilege separation
  • [ ] Tempo standard library
  • [ ] Network stack (with security first!)
  • [ ] File system (with encryption)

Try It Yourself

Conclusion

AtomicOS proves that hobby OS projects can implement real security features. It's not production-ready, but it's honest about what it is: an educational project that doesn't cut corners on security.

Remember: "No exploits, no crashes, no lies."

Have questions about OS development or security? Drop a comment below or reach out on GitHub!

Top comments (0)