Python Key-Value Store Tutorial - Build, Encrypt, and Optimize Your Data Storage

A key-value store is a simple yet powerful type of database that allows you to associate unique keys with corresponding values.

While it is conceptually similar to a Python dictionary, a key-value store can be enhanced to support more advanced features like persistence, expiration, and encryption.

In this article, we’ll walk through how to create a key-value store in Python, starting with an in-memory version and progressing to more advanced implementations.

By the end, you'll have the tools and knowledge to create your own key-value store tailored to your application's specific needs.

---

What is a Key-Value Store?

A key-value store allows you to map keys to specific values, providing fast lookups, inserts, and updates.

These systems are used in applications like caching, configuration storage, and even large distributed databases like Redis or DynamoDB. NoSQL databases often use key-value stores as one of their underlying data models.

Unlike relational databases, key-value stores do not have structured tables, making them faster and more lightweight in specific use cases.

---

Prerequisites

To follow along this article, you’ll need:

• Basic Python knowledge: Familiarity with dictionaries, file handling, and classes.
• Python 3.x installed: Make sure Python is installed on your system.
• Optional libraries: For encryption, you’ll need the cryptography library. For storage optimization, the shelve module is part of Python’s standard library.

---

Create an In-Memory Key-Value Store

An in-memory key-value store exists only while the program is running, and all data is lost when the program stops.

This is the simplest form of a key-value store and serves as a strong foundation for more advanced implementations.

class InMemoryKeyValueStore:  
    def __init__(self):  
        """Initialize an empty dictionary to store key-value pairs."""  
        self.store = {}  

    def set(self, key, value):  
        """Store a key-value pair."""  
        self.store[key] = value  

    def get(self, key):  
        """Retrieve the value for a given key."""  
        return self.store.get(key)  

    def delete(self, key):  
        """Remove a key-value pair."""  
        if key in self.store:  
            del self.store[key]  

    def keys(self):  
        """Return a list of all keys."""  
        return list(self.store.keys())  


if __name__ == '__main__':  
    # Usage example  
    store = InMemoryKeyValueStore()  
    store.set('name', 'Alice')  
    print(store.get('name'))  # Output: Alice  
    store.delete('name')  
    print(store.get('name'))  # Output: None

This code defines a simple in-memory key-value store implemented as a Python class named InMemoryKeyValueStore.

The class initializes an empty dictionary to store key-value pairs and provides methods to interact with this store:

• The __init__ method sets up the dictionary.
• The set method allows adding or updating key-value pairs.
• The get method retrieves values associated with specific keys, returning None if the key does not exist.
• The delete method removes key-value pairs from the store if the key is present.
• The keys method returns a list of all keys currently stored in the dictionary.

The usage example demonstrates how to create an instance of the InMemoryKeyValueStore class, set a key-value pair, retrieve the value, delete the key, and attempt to retrieve the value again after deletion.

This simple approach is useful for applications that need a temporary, fast key-value store with minimal overhead.

---

Create a File-Based Key-Value Store

To make the store persistent, we’ll save key-value pairs to a file. We’ll use the json module to serialize and deserialize data.

This approach allows you to store data across program runs, but it has some limitations in terms of speed and scalability for large datasets.

import json  


class FileBasedKeyValueStore:  
    def __init__(self, filename='data_store.json'):  
        """Load existing data from the file."""  
        self.filename = filename  
        self._load_data()  

    def _load_data(self):  
        try:  
            with open(self.filename, 'r') as file:  
                self.store = json.load(file)  
        except (FileNotFoundError, json.JSONDecodeError):  
            self.store = {}  

    def _save_data(self):  
        with open(self.filename, 'w') as file:  
            json.dump(self.store, file)  

    def set(self, key, value):  
        self.store[key] = value  
        self._save_data()  

    def get(self, key):  
        return self.store.get(key)  

    def delete(self, key):  
        if key in self.store:  
            del self.store[key]  
            self._save_data()  

    def keys(self):  
        return list(self.store.keys())  


if __name__ == '__main__':  
    # Usage example (first run)  
    store = FileBasedKeyValueStore()  
    store.set('language', 'Python')  
    print(store.get('language'))  # Output: Python  

    # Usage example (second run)    store = FileBasedKeyValueStore()  
    print(store.get('language'))  # Output: Python  
    store.delete('language')  

    # Usage example (third run)  
    store = FileBasedKeyValueStore()  
    print(store.get('language'))  # Output: None

This code defines a file-based key-value store implemented as a Python class named FileBasedKeyValueStore.

The class uses a JSON file to persist key-value pairs, allowing data to be retained across different runs of the program.

The class initializes by loading existing data from a specified file and provides methods to interact with the store:

• The __init__ method initializes the class with a filename (defaulting to 'data_store.json') and calls the _load_data method to load existing data from the file.
• The _load_data method attempts to read and load data from the file. If the file does not exist or contains invalid JSON, it initializes an empty dictionary.
• The _save_data method writes the current state of the store back to the file, ensuring data persistence.
• The set method adds or updates a key-value pair in the store and then saves the data to the file.
• The get method retrieves the value associated with a specific key, returning None if the key does not exist.
• The delete method removes a key-value pair from the store if the key is present and then saves the data to the file.
• The keys method returns a list of all keys currently stored in the dictionary.

The usage examples demonstrate how to create an instance of the FileBasedKeyValueStore class, set a key-value pair, retrieve the value, and delete the key across multiple runs of the program:

• In the first run, an instance of FileBasedKeyValueStore is created, a key-value pair ('language', 'Python') is set, and the value is retrieved and printed.
• In the second run, a new instance is created, the value for the key 'language' is retrieved and printed (showing persistence), and the key is then deleted.
• In the third run, another instance is created, and the value for the key 'language' is retrieved and printed, showing None since the key was deleted in the previous run.

💡 Note: This implementation reads and writes the entire file for each operation. For larger datasets, you might want to explore alternatives like SQLite or Redis to improve efficiency.

---

Advanced Features to Consider

Data Expiration

Data expiration allows keys to automatically expire after a specific time-to-live (TTL).

This is useful for caching, where you want to keep data only for a limited time.

import time  
from InMemory import InMemoryKeyValueStore  


class ExpiringKeyValueStore(InMemoryKeyValueStore):  
    def __init__(self):  
        super().__init__()  
        self.expiration_times = {}  

    def set(self, key, value, ttl=None):  
        super().set(key, value)  
        if ttl is not None:  
            self.expiration_times[key] = time.time() + ttl  

    def get(self, key):  
        if key in self.expiration_times and time.time() > self.expiration_times[key]:  
            self.delete(key)  
        return super().get(key)  


if __name__ == '__main__':  
    # Usage example  
    store = ExpiringKeyValueStore()  
    store.set('name', 'Alice', 1)  
    print(store.get('name'))  # Output: Alice  
    time.sleep(2)  
    print(store.get('name'))  # Output: None

This code defines an expiring key-value store implemented as a Python class named ExpiringKeyValueStore.

The class extends the functionality of an in-memory key-value store by adding support for time-to-live (TTL) expiration of keys.

This means that keys can be set to automatically expire after a specified amount of time:

• The ExpiringKeyValueStore class inherits from InMemoryKeyValueStore, which provides basic key-value store functionality.
• The __init__ method initializes the class by calling the parent class's initializer and setting up an additional dictionary, expiration_times, to store the expiration times for keys.
• The set method extends the parent class's set method to include an optional ttl (time-to-live) parameter. If ttl is provided, the method calculates the expiration time (current time plus ttl) and stores it in the expiration_times dictionary.
• The get method checks if a key has expired by comparing the current time with the key's expiration time. If the key has expired, it is deleted from the store. The method then retrieves the value using the parent class's get method.

The usage example demonstrates how to create an instance of the ExpiringKeyValueStore class, set a key-value pair with a TTL, retrieve the value, and observe the expiration of the key:

• An instance of ExpiringKeyValueStore is created.
• The set method is used to store the key-value pair ('name', 'Alice') with a TTL of 1 second.
• The get method is used to retrieve the value associated with the key 'name', which outputs Alice.
• The program sleeps for 2 seconds to allow the TTL to expire.
• The get method is used again to retrieve the value associated with the key 'name', which outputs None since the key has expired and been deleted.

This example showcases the functionality of the expiring key-value store, including setting keys with a TTL, retrieving values, and handling key expiration.

💡 Keys with a TTL will be automatically deleted when expired, which is useful for caching frequently changing data.

---

Encryption

To protect sensitive data, you can encrypt stored values using the cryptography library.

This approach ensures that even if the data is compromised, it remains unreadable.

First, you need to install the necessary library:

pip install cryptography

Now you can write the code:

from cryptography.fernet import Fernet  
from FileBased import FileBasedKeyValueStore  


class EncryptedKeyValueStore(FileBasedKeyValueStore):  
    def __init__(self, filename='encrypted_store.json', key=None):  
        super().__init__(filename)  
        self.key = key or Fernet.generate_key()  
        self.cipher = Fernet(self.key)  

    def _encrypt(self, value):  
        return self.cipher.encrypt(value.encode()).decode()  

    def _decrypt(self, value):  
        return self.cipher.decrypt(value.encode()).decode()  

    def set(self, key, value):  
        encrypted_value = self._encrypt(value)  
        super().set(key, encrypted_value)  

    def get(self, key):  
        encrypted_value = super().get(key)  
        if encrypted_value is not None:  
            return self._decrypt(encrypted_value)  
        return None  


# Usage example  
if __name__ == '__main__':  
    # Usage example (first run)  
    store = EncryptedKeyValueStore()  
    store.set('language', 'Python')  
    print(store.get('language'))  # Output: Python

This code defines an encrypted key-value store implemented as a Python class named EncryptedKeyValueStore.

The class extends the functionality of a file-based key-value store by adding encryption and decryption capabilities to the stored values.

This ensures that the data is securely stored and can only be accessed by someone with the correct encryption key:

• The EncryptedKeyValueStore class inherits from FileBasedKeyValueStore, which provides file-based key-value store functionality.
• The __init__ method initializes the class by calling the parent class's initializer with a specified filename (defaulting to 'encrypted_store.json'). It also initializes the encryption key, generating a new one if none is provided, and sets up a Fernet cipher using this key.
• The _encrypt method takes a value, encrypts it using the Fernet cipher, and returns the encrypted value as a string.
• The _decrypt method takes an encrypted value, decrypts it using the Fernet cipher, and returns the original value as a string.
• The set method encrypts the value before storing it in the key-value store.
• The get method decrypts the value after retrieving it from the key-value store.

The usage example demonstrates how to create an instance of the EncryptedKeyValueStore class, set a key-value pair, and retrieve the value:

• An instance of EncryptedKeyValueStore is created.
• The set method is used to store the key-value pair ('language', 'Python'), which is encrypted before being stored.
• The get method is used to retrieve the value associated with the key 'language', which is decrypted before being returned, outputting Python.

This is the contents of the 'encrypted_store.json' for reference (note, yours will be different due to a different generated key):

{"language": "gAAAAABne8vdNNNRa00HiCkLxuwxVNY89vk5ODcy2Kq3m2tXgVeSNjsplsLtHFSIAPXxMNfM8wzxq9NG8Cq-YksP9HvF6hgNvg=="}

💡 Note: You will probably want to pass your own key to the class, that way you can be sure that you can decrypt if running the file a second time.

Explanation: Data is encrypted before being saved and decrypted when retrieved, ensuring the confidentiality of stored information.

---

Memory Optimization

Instead of loading the entire file into memory, you can use the shelve module to only access data as needed.

It uses a persistent dictionary-like object backed by a database file, allowing for efficient storage and retrieval of data without loading the entire dataset into memory.

import shelve  


class ShelveKeyValueStore:  
    def __init__(self, filename='shelve_store.db'):  
        self.filename = filename  

    def set(self, key, value):  
        with shelve.open(self.filename) as db:  
            db[key] = value  

    def get(self, key):  
        with shelve.open(self.filename) as db:  
            return db.get(key)  

    def delete(self, key):  
        with shelve.open(self.filename) as db:  
            if key in db:  
                del db[key]  

    def keys(self):  
        with shelve.open(self.filename) as db:  
            return list(db.keys())  


# Usage example  
if __name__ == '__main__':  
    # Usage example (first run)  
    store = ShelveKeyValueStore()  
    store.set('language', 'Python')  
    print(store.get('language'))  # Output: Python  

    # Usage example (second run)    
    store = ShelveKeyValueStore()  
    print(store.get('language'))  # Output: Python  
    store.delete('language')  

    # Usage example (third run)  
    store = ShelveKeyValueStore()  
    print(store.get('language'))  # Output: None

This code defines a key-value store implemented as a Python class named ShelveKeyValueStore.

The class uses the shelve module to persist key-value pairs in a database file, allowing data to be retained across different runs of the program.

The class provides methods to set, get, delete, and list keys, ensuring that the data is securely stored and can be accessed as needed:

• The ShelveKeyValueStore class initializes with a specified filename (defaulting to 'shelve_store.db').
• The set method stores a key-value pair in the database.
• The get method retrieves the value associated with a specific key, returning None if the key does not exist.
• The delete method removes a key-value pair from the database if the key is present.
• The keys method returns a list of all keys currently stored in the database.

The usage example demonstrates how to create an instance of the ShelveKeyValueStore class, set a key-value pair, retrieve the value, delete the key, and observe the persistence of data across multiple runs of the program:

• In the first run, an instance of ShelveKeyValueStore is created, a key-value pair ('language', 'Python') is set, and the value is retrieved and printed, outputting Python.
• In the second run, a new instance is created, the value for the key 'language' is retrieved and printed (showing persistence), outputting Python, and the key is then deleted.
• In the third run, another instance is created, and the value for the key 'language' is retrieved and printed, outputting None since the key was deleted in the previous run.

This example showcases the functionality of the shelve-based key-value store, including setting, getting, deleting keys, and the persistence of data across different runs of the program.

💡 This approach is similar to the file-based one but reduces memory usage, making it more efficient for large datasets.

---

Conclusion

We’ve explored the process of creating a key-value store in Python, starting with simple in-memory stores and progressing to more advanced implementations that include file storage, expiration, encryption, and memory optimization.

Each step builds on the previous one, adding layers of functionality and complexity to meet various application needs.

• In-Memory Stores: These are the simplest form of key-value stores, using Python dictionaries to hold data in memory. They are fast and efficient for small datasets but lack persistence, meaning data is lost when the program terminates.
• File-Based Stores: These stores use files to persist data, ensuring that information is retained across program runs, making them suitable for applications requiring data persistence.
• Expiring Key-Value Stores: Adding a time-to-live (TTL) feature allows keys to expire after a specified duration. This is crucial for applications like caching, where data relevance is time-sensitive.
• Encrypted Key-Value Stores: Incorporating encryption ensures that stored data is secure. This is essential for applications handling sensitive information, such as user credentials or financial data.
• Memory Optimization: Techniques like using efficient data structures and managing memory allocation can optimize the performance of key-value stores, making them suitable for large-scale applications. For example, the shelve module in Python is a memory-optimized solution for key-value storage

These concepts are essential for applications such as caching, session management, and configuration storage.

Understanding and implementing these various types of key-value stores equips developers with the tools to build robust, efficient, and secure data storage solutions tailored to specific application requirements.

---

Follow me on Twitter: https://twitter.com/DevAsService

Follow me on Instagram: https://www.instagram.com/devasservice/

Follow me on TikTok: https://www.tiktok.com/@devasservice

Follow me on YouTube: https://www.youtube.com/@DevAsService