DEV Community

Anthony4m
Anthony4m

Posted on

Dive Into Slotted Pages: Building Efficient Record Management in Go

Hey there, fellow developers! 👋 Ever wonder how databases and storage engines efficiently manage data? Today, we’re diving into a Go implementation of a slotted page a clever structure for organizing records inside fixed-size pages. It’s the secret sauce behind efficient data storage, fast lookups, and compact space management.

Let’s break it down, step by step! 🚀


What’s a Slotted Page? 🤔

A slotted page is a fixed-size container used to store variable-sized records (a.k.a. cells). It’s like a well-organized shelf with numbered slots pointing to items, ensuring efficient storage and retrieval. The core components are:

  • Header: Metadata like record count, free space, and offsets.
  • Slots: Array of pointers to records, sorted for fast binary search.
  • Free Space: Reserved area for storing new records.

By maintaining a slot directory, slotted pages enable efficient data access, even when records are variable-sized or deleted.
Here's a visual representation:
+----------------+
| Header |
+----------------+
| Slot Array |
+----------------+
| |
| Free Space |
| |
+----------------+
| Data Cell |
| Data Cell |

+----------------+

The SlottedPage Structure in Go 📦

Here’s the basic structure for our SlottedPage:

type SlottedPage struct {
    *Page            // Base page functionality
    headerSize int   // Size of the header (includes slots)
    cellCount  int   // Number of records (cells)
    freeSpace  int   // Offset for new records
    slots      []int // Array of record offsets (sorted)
}
Enter fullscreen mode Exit fullscreen mode

Key features include:

  • Embedded Page: Provides raw data storage.
  • Slots Array: Tracks where each record is located.
  • Thread Safety: (Inherited from the underlying page implementation.)

Creating a Slotted Page ✨

Here’s how we initialize a new SlottedPage:

func NewSlottedPage(pageSize int) *SlottedPage {
    if pageSize == 0 {
        pageSize = DefaultPageSize
    }

    sp := &SlottedPage{
        Page:       NewPage(pageSize),
        headerSize: PageHeaderSize,
        freeSpace:  pageSize,
        slots:      make([]int, 0),
    }

    // Initialize metadata
    sp.SetInt(0, pageSize)      // Page size
    sp.SetInt(4, PageHeaderSize) // Header size
    sp.SetInt(8, 0)              // Cell count
    sp.SetInt(12, pageSize)      // Free space pointer
    return sp
}
Enter fullscreen mode Exit fullscreen mode
  • Metadata Initialization: We set up the page size, header size, record count, and free space.
  • Defaults: If no size is provided, we use a standard page size of 4KB.

Record Management in Action 🔧

Now for the fun part—how do we manage records? Let’s break it down:

Inserting Records 📝

Here’s how records are inserted:

func (sp *SlottedPage) InsertCell(cell *Cell) error {
    cellBytes := cell.ToBytes()
    cellSize := len(cellBytes)

    // Check available space
    if !cell.FitsInPage(sp.freeSpace) {
        return fmt.Errorf("not enough space for cell")
    }

    // Calculate the offset
    newOffset := sp.freeSpace - cellSize

    // Write the record to the calculated offset
    sp.SetBytes(newOffset, cellBytes)

    // Update slots
    insertPos := sp.findSlotPosition(cell.key)
    sp.slots = append(sp.slots[:insertPos], append([]int{newOffset}, sp.slots[insertPos:]...)...)

    // Update metadata
    sp.cellCount++
    sp.freeSpace = newOffset
    sp.SetInt(8, sp.cellCount)
    sp.SetInt(12, sp.freeSpace)
    return nil
}
Enter fullscreen mode Exit fullscreen mode
  • Space Check: Ensures there’s enough room for the new record.
  • Offset Calculation: Finds where the record should go.
  • Slot Update: Inserts the record offset while maintaining sorted order.

Retrieving Records 🔍

You can find a record by key using binary search:

func (sp *SlottedPage) FindCell(key []byte) (*Cell, int, error) {
    low, high := 0, len(sp.slots)-1

    for low <= high {
        mid := (low + high) / 2
        cell, err := sp.GetCell(sp.slots[mid])
        if err != nil {
            return nil, -1, err
        }

        comp := bytes.Compare(key, cell.key)
        if comp == 0 {
            return cell, mid, nil
        } else if comp < 0 {
            high = mid - 1
        } else {
            low = mid + 1
        }
    }

    return nil, -1, fmt.Errorf("key not found")
}
Enter fullscreen mode Exit fullscreen mode

Binary search ensures fast lookups in the sorted slot directory.

Deleting Records 🗑️

When you delete a record, its slot is removed:

func (sp *SlottedPage) DeleteCell(slot int) error {
    if slot >= len(sp.slots) {
        return fmt.Errorf("invalid slot")
    }

    // Mark cell as deleted
    cell, err := sp.GetCell(sp.slots[slot])
    if err != nil {
        return err
    }
    cell.MarkDeleted()

    // Remove slot
    sp.slots = append(sp.slots[:slot], sp.slots[slot+1:]...)
    sp.cellCount--
    err = sp.SetInt(8, sp.cellCount)
    if err != nil {
        return err
    }
    return nil
}

Enter fullscreen mode Exit fullscreen mode
  • Mark as Deleted: The slot is removed, and the record count is updated.

Compacting the Page 🧹

Over time, deleting records leaves gaps. Compaction reorganizes the page to reclaim space:

func (sp *SlottedPage) Compact() error {
    newPage := NewSlottedPage(len(sp.data))

    for _, offset := range sp.slots {
        cell, err := sp.GetCell(offset)
        if err != nil {
            return err
        }
        if !cell.IsDeleted() {
            newPage.InsertCell(cell)
        }
    }

    // Replace old page data
    sp.data = newPage.data
    sp.slots = newPage.slots
    sp.cellCount = newPage.cellCount
    sp.freeSpace = newPage.freeSpace

    return nil
}
Enter fullscreen mode Exit fullscreen mode

Compaction:

  1. Copies non-deleted records to a new page.
  2. Defragments the free space.

Why Slotted Pages? 💡

This implementation shines in real-world applications:

  • Database Engines: Store and retrieve records efficiently.
  • File Systems: Manage disk blocks and defragmentation.
  • Caching Systems: Organize data into fixed-size chunks for quick access.
  • Custom Storage Layers: Build a foundation for structured data storage.

Wrapping Up 🎯

Slotted pages are a powerful yet elegant solution for managing variable-sized records in fixed-size pages. With efficient key-based lookups, space management, and support for deletion/compaction, this implementation can serve as the backbone of your next database or storage project.
Building a slotted page system is a fascinating journey into database internals. This implementation provides a solid foundation for building more complex database features like:

  • B-tree indexes
  • MVCC (Multi-Version Concurrency Control)
  • Transaction management

Ready to build your own database engine? This slotted page implementation is a great place to start! 💪


Next Steps:

  • Extend with Compression: Add support for compressed records.
  • Optimize Caching: Build a caching layer for frequently accessed pages.
  • Broaden Data Support: Add support for more complex types like floats or custom structs.

Further Reading 📚

  • Database Internals by Alex Petrov
  • File Organizations and Access Methods
  • B-tree Theory and Applications

Remember: great databases are built one page at a time! Happy coding! 🚀


If this post sparked your interest, don’t forget to ❤️ and share it with your fellow devs!

Top comments (0)