DEV Community

Cover image for Serialization and Deserialization in Database Engines: How Data Becomes Bytes
Farhad Rahimi Klie
Farhad Rahimi Klie

Posted on

Serialization and Deserialization in Database Engines: How Data Becomes Bytes

When building a database engine from scratch, one of the most important concepts you need to understand is serialization and deserialization.

You might be wondering:

  • How does a database store data on disk?
  • How does it convert a C structure into bytes?
  • How does it read those bytes back into memory?
  • How can we save records and retrieve them after restarting the database?

These questions lead us to two fundamental concepts: serialization and deserialization.

Let's understand how they work and how we can implement them in C.

1. What Is Serialization?

Serialization is the process of converting data structures in memory into a format that can be stored or transmitted.

For example, imagine we have the following C structure:

typedef struct {
    int id;
    char name[32];
    int age;
} User;
Enter fullscreen mode Exit fullscreen mode

We can create a user:

User user = {
    .id = 1,
    .name = "Farhad",
    .age = 22
};
Enter fullscreen mode Exit fullscreen mode

This structure currently exists in RAM.

However, a database engine needs to store this information on disk so that it remains available even after the program terminates.

Serialization converts this structure into a sequence of bytes that can be written to a file.

In simple terms:

C Structure → Serialization → Bytes → Disk

2. What Is Deserialization?

Deserialization is the reverse process.

It takes previously serialized bytes and reconstructs the original data structure in memory.

For example, suppose our database contains a serialized user record.

When we want to retrieve that user, the database reads the bytes from disk and converts them back into a C structure.

The process looks like this:

Disk → Bytes → Deserialization → C Structure

Now we can access the original information:

printf("ID: %d\n", user.id);
printf("Name: %s\n", user.name);
printf("Age: %d\n", user.age);
Enter fullscreen mode Exit fullscreen mode

Serialization and deserialization work together to make persistent data storage possible.

3. Why Do Database Engines Need Them?

A database engine cannot simply depend on RAM.

RAM is temporary. When a program terminates or a computer shuts down, its contents are lost.

Databases need persistent storage.

For example, consider a database engine that stores thousands of users.

Each user has:

  • An ID
  • A name
  • An age

The database needs a way to convert these records into bytes, write them to disk, and reconstruct them whenever they are requested.

This is where serialization and deserialization become essential.

They are used in:

  • Database records
  • Database pages
  • Write-ahead logs (WAL)
  • Index structures
  • Backup files
  • Network communication
  • Database replication

4. Implementing Serialization in C

Let's implement a simple serialization function.

First, define our structure:

#include <stdio.h>
#include <stdint.h>
#include <string.h>

typedef struct {
    uint32_t id;
    char name[32];
    uint32_t age;
} User;
Enter fullscreen mode Exit fullscreen mode

Now let's implement serialization:

void serialize_user(const User *user, unsigned char *buffer) {
    memcpy(buffer, &user->id, sizeof(user->id));

    memcpy(buffer + 4, user->name, 32);

    memcpy(buffer + 36, &user->age, sizeof(user->age));
}
Enter fullscreen mode Exit fullscreen mode

This function copies each field into a byte buffer.

Our serialized record has the following layout:

Field Size Offset
ID 4 bytes 0
Name 32 bytes 4
Age 4 bytes 36
Total 40 bytes

Notice that we are not simply copying the entire structure.

Instead, we explicitly serialize each field.

This is important because C compilers can introduce padding between structure members. Also, copying native integer representations directly is not portable across machines with different byte orders.

For a production database, we should define a fixed byte order, such as little-endian, and encode integer fields explicitly.

5. Implementing Deserialization in C

Now let's implement the reverse operation.

void deserialize_user(const unsigned char *buffer, User *user) {
    memcpy(&user->id, buffer, 4);

    memcpy(user->name, buffer + 4, 32);

    user->name[31] = '\0';

    memcpy(&user->age, buffer + 36, 4);
}
Enter fullscreen mode Exit fullscreen mode

This function reads the serialized bytes and reconstructs our User structure.

We can now retrieve the original information from the byte buffer.

6. Saving Serialized Data to Disk

Serialization alone doesn't save anything permanently.

We need to write the resulting bytes to a file.

Let's implement a simple function:

int save_user(const char *filename, const User *user) {
    unsigned char buffer[40];

    serialize_user(user, buffer);

    FILE *file = fopen(filename, "wb");

    if (file == NULL) {
        perror("fopen");
        return -1;
    }

    if (fwrite(buffer, 1, sizeof(buffer), file) != sizeof(buffer)) {
        perror("fwrite");
        fclose(file);
        return -1;
    }

    if (fclose(file) != 0) {
        perror("fclose");
        return -1;
    }

    return 0;
}
Enter fullscreen mode Exit fullscreen mode

Here is what happens:

  1. We create a 40-byte buffer.
  2. We serialize the user structure.
  3. We open a file in binary write mode.
  4. We write the serialized bytes to the file.
  5. We close the file.

Our user record is now stored on disk.

7. Reading Data from Disk

Now let's implement a function to retrieve the user.

int load_user(const char *filename, User *user) {
    unsigned char buffer[40];

    FILE *file = fopen(filename, "rb");

    if (file == NULL) {
        perror("fopen");
        return -1;
    }

    if (fread(buffer, 1, sizeof(buffer), file) != sizeof(buffer)) {
        fprintf(stderr, "Failed to read user record\n");
        fclose(file);
        return -1;
    }

    if (fclose(file) != 0) {
        perror("fclose");
        return -1;
    }

    deserialize_user(buffer, user);

    return 0;
}
Enter fullscreen mode Exit fullscreen mode

This function performs the opposite operation:

  1. Opens the file in binary read mode.
  2. Reads 40 bytes into memory.
  3. Deserializes those bytes.
  4. Reconstructs the original user structure.

8. Putting Everything Together

Let's test our implementation with a complete C program.

#include <stdio.h>
#include <stdint.h>
#include <string.h>

typedef struct {
    uint32_t id;
    char name[32];
    uint32_t age;
} User;

#define RECORD_SIZE 40

void serialize_user(const User *user, unsigned char *buffer) {
    memcpy(buffer, &user->id, 4);
    memcpy(buffer + 4, user->name, 32);
    memcpy(buffer + 36, &user->age, 4);
}

void deserialize_user(const unsigned char *buffer, User *user) {
    memcpy(&user->id, buffer, 4);
    memcpy(user->name, buffer + 4, 32);

    user->name[31] = '\0';

    memcpy(&user->age, buffer + 36, 4);
}

int save_user(const char *filename, const User *user) {
    unsigned char buffer[RECORD_SIZE];

    serialize_user(user, buffer);

    FILE *file = fopen(filename, "wb");

    if (!file)
        return -1;

    if (fwrite(buffer, 1, RECORD_SIZE, file) != RECORD_SIZE) {
        fclose(file);
        return -1;
    }

    if (fclose(file) != 0)
        return -1;

    return 0;
}

int load_user(const char *filename, User *user) {
    unsigned char buffer[RECORD_SIZE];

    FILE *file = fopen(filename, "rb");

    if (!file)
        return -1;

    if (fread(buffer, 1, RECORD_SIZE, file) != RECORD_SIZE) {
        fclose(file);
        return -1;
    }

    if (fclose(file) != 0)
        return -1;

    deserialize_user(buffer, user);

    return 0;
}

int main(void) {
    User user = {
        .id = 1,
        .name = "Farhad",
        .age = 22
    };

    if (save_user("database.db", &user) != 0) {
        printf("Failed to save user.\n");
        return 1;
    }

    printf("User saved successfully!\n");

    User loaded_user;

    if (load_user("database.db", &loaded_user) != 0) {
        printf("Failed to load user.\n");
        return 1;
    }

    printf("\nLoaded User:\n");
    printf("ID: %u\n", loaded_user.id);
    printf("Name: %s\n", loaded_user.name);
    printf("Age: %u\n", loaded_user.age);

    return 0;
}
Enter fullscreen mode Exit fullscreen mode

Compile and run:

gcc main.c -o database
./database
Enter fullscreen mode Exit fullscreen mode

Expected output:

User saved successfully!

Loaded User:
ID: 1
Name: Farhad
Age: 22
Enter fullscreen mode Exit fullscreen mode

You have now implemented a basic serialization and deserialization system in C!

9. How Does This Work Inside a Real Database Engine?

Our example is intentionally simple. Real database engines use more sophisticated storage architectures.

For example, instead of writing every record into a separate file, a database engine usually organizes records into pages.

A page is a fixed-size block of storage.

For example, SQLite commonly uses database pages of 4096 bytes, although its page size is configurable.

A simplified database architecture looks like this:

          SQL Query
              |
              v
        Query Processor
              |
              v
         Record Manager
              |
              v
          Serializer
              |
              v
         Page Manager
              |
              v
        Buffer Pool
              |
              v
         Disk Storage
Enter fullscreen mode Exit fullscreen mode

When inserting a record:

  1. The database receives a record.
  2. The record manager prepares its fields.
  3. The serializer converts the record into bytes.
  4. The page manager places those bytes inside a database page.
  5. The buffer pool manages the page in memory.
  6. The storage layer eventually writes the page to disk.

When retrieving a record, the process works in reverse.

This architecture is simplified, but it gives you an idea of how different database components work together.

10. Important Things to Understand

When implementing serialization in a real database engine, there are several important considerations.

1. Endianness

Different computer architectures can represent multi-byte integers differently.

Your database should define a consistent byte order rather than blindly writing native integer representations.

2. Structure Padding

C compilers may add padding between structure members.

Never assume that sizeof(struct) represents a portable database record format.

3. Data Types

A database needs to know how to serialize different types:

  • INTEGER
  • FLOAT
  • TEXT
  • BOOLEAN
  • NULL
  • BLOB

Each type needs an appropriate representation.

4. Variable-Length Records

Not every record has the same size.

For example, one user might have a name of 5 characters, while another has a name of 100 characters.

Real database engines need mechanisms for storing variable-length fields.

5. Data Integrity

A database should detect corrupted or incomplete records.

Techniques such as checksums, page validation, and format validation help protect stored data.

6. Versioning

Database formats can evolve.

Your serialization format should have a way to identify its version so that future versions of your database engine can handle older data formats.

Conclusion

Serialization and deserialization are fundamental concepts when building a database engine from scratch.

Serialization converts in-memory data structures into bytes that can be stored on disk.

Deserialization reconstructs those data structures from previously stored bytes.

Without these mechanisms, a database engine would have no reliable way to represent its records in persistent storage.

If you're building your own database engine in C, understanding these concepts is an important step toward implementing a record manager, page manager, buffer pool, and eventually a complete storage engine.

The most important idea to remember: A database doesn't store your C structures directly. It stores a defined representation of their data as bytes.

Top comments (0)