DEV Community

Fuchi Takeshi
Fuchi Takeshi

Posted on • Updated on

Fast file access with Redis in Google Colab

Open In Colab

I made a Redis module to read values from mmapped files.

I can't find the ability to read existing files in Redis. (Maybe I just don't know it, so if you do, please let me know.) So I made a module that maps a file with double values written to it with mmap and reads/writes the values at specified locations from it. I've included the full code so that you can try it out by simply opening Colab from the button at the top of this page and executing the cells in order. Please also refer to this as a way to create a Redis module.

Using this module, the following commands can be used in Redis.

// This command binds key to the file at file_path.
MMAP key file_path

// This command gets a value at index position from key
VGET key index

// This command gets values at multiple index positions from key
VMGET key index [index ...]

// This command writes values to the index positions of key
VSET key index value [index value ...]

// This command adds a value to key
VADD key value [value ...]

// This command retrieves and deletes the last value of key
VPOP key

// This command gets the number of values of key
VCOUNT key

// This command erases the contents of key
VCLEAR key
Enter fullscreen mode Exit fullscreen mode

First, obtain the Redis source and change current directory to the module's one.

%cd /content
!wget https://download.redis.io/redis-stable.tar.gz
!tar -xzf redis-stable.tar.gz
%cd /content/redis-stable/src/modules
Enter fullscreen mode Exit fullscreen mode
/content
--2023-06-22 06:07:59--  https://download.redis.io/redis-stable.tar.gz
Resolving download.redis.io (download.redis.io)... 45.60.121.1
Connecting to download.redis.io (download.redis.io)|45.60.121.1|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 3068843 (2.9M) [application/octet-stream]
Saving to: ‘redis-stable.tar.gz’

redis-stable.tar.gz 100%[===================>]   2.93M  --.-KB/s    in 0.06s   

2023-06-22 06:07:59 (47.1 MB/s) - ‘redis-stable.tar.gz’ saved [3068843/3068843]

/content/redis-stable/src/modules
Enter fullscreen mode Exit fullscreen mode

Let's build the sample code.

!make
Enter fullscreen mode Exit fullscreen mode
cc -I.  -W -Wall -fno-common -g -ggdb -std=c99 -O2 -fPIC -c helloworld.c -o helloworld.xo
ld -o helloworld.so helloworld.xo -shared  -lc
cc -I.  -W -Wall -fno-common -g -ggdb -std=c99 -O2 -fPIC -c hellotype.c -o hellotype.xo
ld -o hellotype.so hellotype.xo -shared  -lc
cc -I.  -W -Wall -fno-common -g -ggdb -std=c99 -O2 -fPIC -c helloblock.c -o helloblock.xo
ld -o helloblock.so helloblock.xo -shared  -lpthread -lc
cc -I.  -W -Wall -fno-common -g -ggdb -std=c99 -O2 -fPIC -c hellocluster.c -o hellocluster.xo
ld -o hellocluster.so hellocluster.xo -shared  -lc
cc -I.  -W -Wall -fno-common -g -ggdb -std=c99 -O2 -fPIC -c hellotimer.c -o hellotimer.xo
ld -o hellotimer.so hellotimer.xo -shared  -lc
cc -I.  -W -Wall -fno-common -g -ggdb -std=c99 -O2 -fPIC -c hellodict.c -o hellodict.xo
ld -o hellodict.so hellodict.xo -shared  -lc
cc -I.  -W -Wall -fno-common -g -ggdb -std=c99 -O2 -fPIC -c hellohook.c -o hellohook.xo
ld -o hellohook.so hellohook.xo -shared  -lc
cc -I.  -W -Wall -fno-common -g -ggdb -std=c99 -O2 -fPIC -c helloacl.c -o helloacl.xo
ld -o helloacl.so helloacl.xo -shared  -lc
Enter fullscreen mode Exit fullscreen mode

Check if the module .so file has been created.

!ls *.so
Enter fullscreen mode Exit fullscreen mode
helloacl.so    hellocluster.so  hellohook.so   hellotype.so
helloblock.so  hellodict.so hellotimer.so  helloworld.so
Enter fullscreen mode Exit fullscreen mode

Now let's write the code in fmmap.c. The value to be read/written is a double. Error handling has been omitted to make the code easier to read. If you try it, please do not submit any nasty commands.😏

In addition, the code available here allows you to choose the type of value to read or write, and is complete with error handling.

First, necessary headers and a convenience macro.

%%writefile fmmap.c

#pragma GCC diagnostic ignored "-Wunused-parameter"

#include <sys/mman.h>
#include <sys/stat.h>
#include <sys/types.h>
#include <fcntl.h>
#include <unistd.h>
#include <stdbool.h>
#include <stdint.h>
#include <strings.h>
#include <string.h>

#include "../redismodule.h"
#include "../sds.h"
#include "../zmalloc.h"

// A macro compare (RedisModuleString *) and (char *)
static inline int mstringcmp(const RedisModuleString *rs1, const char *s2)
{
  return strcasecmp(RedisModule_StringPtrLen(rs1, NULL), s2);
}

int ftruncate(int fildes, off_t length); // It should be in unistd.h, but I get a warning.
Enter fullscreen mode Exit fullscreen mode
Writing fmmap.c
Enter fullscreen mode Exit fullscreen mode

Define MMapObject, packed with the information needed for mmap. sds is a string type used within Redis.

%%writefile -a fmmap.c

typedef struct _MMapObject
{
  sds file_path;
  int fd;
  void *mmap;
  size_t file_size;
} MMapObject;
Enter fullscreen mode Exit fullscreen mode
Appending to fmmap.c
Enter fullscreen mode Exit fullscreen mode

A variable to hold the MMap type and a function to generate the MMapObject.

%%writefile -a fmmap.c

RedisModuleType *MMapType = NULL;

MMapObject *MCreateObject(void)
{
  return (MMapObject *)zcalloc(sizeof(MMapObject));
}
Enter fullscreen mode Exit fullscreen mode
Appending to fmmap.c
Enter fullscreen mode Exit fullscreen mode

Function to release MMapObject.

%%writefile -a fmmap.c

void MFree(void *value)
{
  if (value == NULL) return;
  const MMapObject *obj_ptr = value;
  if (obj_ptr->mmap != NULL) munmap(obj_ptr->mmap, obj_ptr->file_size);
  if (obj_ptr->fd != -1) close(obj_ptr->fd);
  sdsfree(obj_ptr->file_path);
  zfree(value);
}
Enter fullscreen mode Exit fullscreen mode
Appending to fmmap.c
Enter fullscreen mode Exit fullscreen mode

Function to map the file specified by file_path to key.

%%writefile -a fmmap.c

// MMAP key file_path
int MMap_RedisCommand(RedisModuleCtx *ctx, RedisModuleString **argv, int argc)
{
  RedisModule_AutoMemory(ctx); /* Use automatic memory management. */

  if (argc != 3) return RedisModule_WrongArity(ctx);

  RedisModuleKey *key = RedisModule_OpenKey(ctx, argv[1], REDISMODULE_READ | REDISMODULE_WRITE);

  // It confirms the type of key
  int type = RedisModule_KeyType(key);
  if (type != REDISMODULE_KEYTYPE_EMPTY &&
      RedisModule_ModuleTypeGetType(key) != MMapType) {
    return RedisModule_ReplyWithError(ctx, REDISMODULE_ERRORMSG_WRONGTYPE);
  }

  // It creates new file and mmap it if key is empty.
  MMapObject *obj_ptr;
  if (type == REDISMODULE_KEYTYPE_EMPTY) {
    obj_ptr = MCreateObject();
    obj_ptr->file_path = sdsnew(RedisModule_StringPtrLen(argv[2], NULL));
    obj_ptr->fd = open(obj_ptr->file_path, O_RDWR | O_CREAT, 0666);
    if (obj_ptr->fd == -1) {
        MFree(obj_ptr);
        return RedisModule_ReplyWithError(ctx, sdsnew(RedisModule_StringPtrLen(argv[2], NULL)));
    }

    struct stat sb;
    fstat(obj_ptr->fd, &sb);
    obj_ptr->file_size = sb.st_size;
    obj_ptr->mmap = mmap(NULL, sb.st_size, PROT_READ | PROT_WRITE, MAP_SHARED, obj_ptr->fd, 0);
    RedisModule_ModuleTypeSetValue(key, MMapType, obj_ptr);
  }
  // It confirms the file_path if key already exists.
  else {
    obj_ptr = RedisModule_ModuleTypeGetValue(key);
    if (obj_ptr == NULL) {
      RedisModule_ReplyWithNull(ctx);
      return REDISMODULE_ERR;
    }
    // Error if it is different from the existing file_path.
    if (strcmp(obj_ptr->file_path, RedisModule_StringPtrLen(argv[2], NULL)) != 0) {
      return RedisModule_ReplyWithError(ctx, "It is already mapped on another file.");
    }
  }

  return RedisModule_ReplyWithLongLong(ctx, obj_ptr->file_size / sizeof(double));
}

Enter fullscreen mode Exit fullscreen mode
Appending to fmmap.c
Enter fullscreen mode Exit fullscreen mode

Function to get the value at the position specified by index.

%%writefile -a fmmap.c

// VGET key index
int VGet_RedisCommand(RedisModuleCtx *ctx, RedisModuleString **argv, int argc)
{
  RedisModule_AutoMemory(ctx); /* Use automatic memory management. */

  RedisModuleKey *key =
    RedisModule_OpenKey(ctx, argv[1], REDISMODULE_READ | REDISMODULE_WRITE);

  long long index;
  RedisModule_StringToLongLong(argv[2], &index);

  MMapObject *obj_ptr = RedisModule_ModuleTypeGetValue(key);

  // It returns Null if index is out of range.
  if (obj_ptr->file_size <= (size_t)index * sizeof(double) || index < 0) {
    RedisModule_ReplyWithNull(ctx);
  }
  // It returns mmap[index].
  else {
    RedisModule_ReplyWithDouble(ctx, ((double*)obj_ptr->mmap)[index]);
  }
  return REDISMODULE_OK;
}
Enter fullscreen mode Exit fullscreen mode
Appending to fmmap.c
Enter fullscreen mode Exit fullscreen mode

This function gets the values at the positions indicated by multiple indices.

%%writefile -a fmmap.c

// VMGET key index [index ...]
int VMGet_RedisCommand(RedisModuleCtx *ctx, RedisModuleString **argv, int argc)
{
  RedisModule_AutoMemory(ctx); /* Use automatic memory management. */

  RedisModuleKey *key =
      RedisModule_OpenKey(ctx, argv[1], REDISMODULE_READ | REDISMODULE_WRITE);

  MMapObject *obj_ptr = RedisModule_ModuleTypeGetValue(key);

  RedisModule_ReplyWithArray(ctx, argc - 2);
  for (int i = 2; i < argc; i++) {
    long long index;
    RedisModule_StringToLongLong(argv[i], &index);
    // It returns Null if index is out of range.
    if (obj_ptr->file_size <= (size_t)index * sizeof(double) || index < 0) {
      RedisModule_ReplyWithNull(ctx);
    }
    // It returns mmap[index].
    else {
      RedisModule_ReplyWithDouble(ctx, ((double*)obj_ptr->mmap)[index]);
    }
  }
  return REDISMODULE_OK;
}
Enter fullscreen mode Exit fullscreen mode
Appending to fmmap.c
Enter fullscreen mode Exit fullscreen mode

This function writes values at the positions indicated by indeces.

%%writefile -a fmmap.c

// VSET key index value [index value ...]
int VSet_RedisCommand(RedisModuleCtx *ctx, RedisModuleString **argv, int argc)
{
  RedisModule_AutoMemory(ctx); /* Use automatic memory management. */

  RedisModuleKey *key =
      RedisModule_OpenKey(ctx, argv[1], REDISMODULE_READ | REDISMODULE_WRITE);

  MMapObject *obj_ptr = RedisModule_ModuleTypeGetValue(key);

  long long index;
  double value;
  // It confirms the type of value written.
  for (int i = 3; i < argc; i += 2) {
    if (RedisModule_StringToDouble(argv[i], &value) == REDISMODULE_ERR) {
      return RedisModule_ReplyWithError(ctx, "value must be double.");
    }
  }
  int n_factors = 0;
  // It writes a value to mmap[index].
  for (int i = 2; i < argc; i += 2) {
    RedisModule_StringToLongLong(argv[i], &index);
    // It writes a value if index is in range.
    if (0 <= index && (size_t)index * sizeof(double) < obj_ptr->file_size) {
      RedisModule_StringToDouble(argv[i + 1], &value);
      ((double*)obj_ptr->mmap)[index] = (double)value;
      ++n_factors;
    }
  }
  // It returns the number of writings.
  return RedisModule_ReplyWithLongLong(ctx, n_factors);
}

Enter fullscreen mode Exit fullscreen mode
Appending to fmmap.c
Enter fullscreen mode Exit fullscreen mode

This function appends values to the end of a file.

%%writefile -a fmmap.c

// VADD key value [value ...]
int VAdd_RedisCommand(RedisModuleCtx *ctx, RedisModuleString **argv, int argc)
{
  RedisModule_AutoMemory(ctx); /* Use automatic memory management. */

  RedisModuleKey *key =
      RedisModule_OpenKey(ctx, argv[1], REDISMODULE_READ | REDISMODULE_WRITE);

  MMapObject *obj_ptr = RedisModule_ModuleTypeGetValue(key);
  double value;
  // It confirms the type of value added.
  for (int i = 2; i < argc; ++i) {
    if (RedisModule_StringToDouble(argv[i], &value) == REDISMODULE_ERR) {
      return RedisModule_ReplyWithError(ctx, "value must be double.");
    }
  }
  // It extends mmap and write a value at the end of mmap.
  size_t new_size = obj_ptr->file_size + sizeof(double) * (argc - 2);
  ftruncate(obj_ptr->fd, new_size);
  munmap(obj_ptr->mmap, obj_ptr->file_size);
  obj_ptr->mmap = mmap(NULL, new_size, PROT_READ | PROT_WRITE, MAP_SHARED, obj_ptr->fd, 0);
  for (int i = 2; i < argc; ++i) {
    size_t index = obj_ptr->file_size / sizeof(double);
    RedisModule_StringToDouble(argv[i], &value);
    obj_ptr->file_size += sizeof(double);
    ((double*)obj_ptr->mmap)[index] = (double)value;
  }

  // It returns the number of elements added.
  return RedisModule_ReplyWithLongLong(ctx, argc - 2);
}
Enter fullscreen mode Exit fullscreen mode
Appending to fmmap.c
Enter fullscreen mode Exit fullscreen mode

This function gets the number of values in a file.

%%writefile -a fmmap.c

// VCOUNT key
int VCount_RedisCommand(RedisModuleCtx *ctx, RedisModuleString **argv, int argc)
{
  RedisModule_AutoMemory(ctx); /* Use automatic memory management. */

  RedisModuleKey *key =
      RedisModule_OpenKey(ctx, argv[1], REDISMODULE_READ | REDISMODULE_WRITE);

  MMapObject *obj_ptr = RedisModule_ModuleTypeGetValue(key);

  return RedisModule_ReplyWithLongLong(ctx, obj_ptr->file_size / sizeof(double));
}
Enter fullscreen mode Exit fullscreen mode
Appending to fmmap.c
Enter fullscreen mode Exit fullscreen mode

This function erases the contents of a file.

%%writefile -a fmmap.c

// VCLEAR key
int VClear_RedisCommand(RedisModuleCtx *ctx, RedisModuleString **argv, int argc)
{
  RedisModule_AutoMemory(ctx); /* Use automatic memory management. */

  RedisModuleKey *key =
      RedisModule_OpenKey(ctx, argv[1], REDISMODULE_READ | REDISMODULE_WRITE);

  MMapObject *obj_ptr = RedisModule_ModuleTypeGetValue(key);

  munmap(obj_ptr->mmap, obj_ptr->file_size);
  ftruncate(obj_ptr->fd, 0);
  obj_ptr->mmap = mmap(NULL, 0, PROT_READ | PROT_WRITE, MAP_SHARED, obj_ptr->fd, 0);
  RedisModule_ReplyWithLongLong(ctx, obj_ptr->file_size / sizeof(double));
  obj_ptr->file_size = 0;

  return REDISMODULE_OK;
}
Enter fullscreen mode Exit fullscreen mode
Appending to fmmap.c
Enter fullscreen mode Exit fullscreen mode

This function gets a value from the end of a file and delete it.

%%writefile -a fmmap.c

// VPOP key
int VPop_RedisCommand(RedisModuleCtx *ctx, RedisModuleString **argv, int argc)
{
  RedisModule_AutoMemory(ctx); /* Use automatic memory management. */

  RedisModuleKey *key =
      RedisModule_OpenKey(ctx, argv[1], REDISMODULE_READ | REDISMODULE_WRITE);

  MMapObject *obj_ptr = RedisModule_ModuleTypeGetValue(key);

  if (obj_ptr->file_size == 0) {
    RedisModule_ReplyWithNull(ctx);
  }
  else {
    size_t index = obj_ptr->file_size / sizeof(double) - 1;
    RedisModule_ReplyWithDouble(ctx, ((double*)obj_ptr->mmap)[index]);
    munmap(obj_ptr->mmap, obj_ptr->file_size);
    obj_ptr->file_size -= sizeof(double);
    ftruncate(obj_ptr->fd, obj_ptr->file_size);
    obj_ptr->mmap = mmap(NULL, obj_ptr->file_size, PROT_READ | PROT_WRITE, MAP_SHARED, obj_ptr->fd, 0);
  }
  return REDISMODULE_OK;
}
Enter fullscreen mode Exit fullscreen mode
Appending to fmmap.c
Enter fullscreen mode Exit fullscreen mode

This function stores information about mmap in a Redis RDB file.

%%writefile -a fmmap.c

void MRdbSave(RedisModuleIO *rdb, void *value)
{
  MMapObject *obj_ptr = value;
  RedisModule_SaveStringBuffer(rdb, obj_ptr->file_path, sdslen(obj_ptr->file_path));
  msync(obj_ptr->mmap, obj_ptr->file_size, MS_ASYNC);
}
Enter fullscreen mode Exit fullscreen mode
Appending to fmmap.c
Enter fullscreen mode Exit fullscreen mode

This function reads information about mmap from a Redis RDB file.

%%writefile -a fmmap.c

void *MRdbLoad(RedisModuleIO *rdb, int encver)
{
  MMapObject *obj_ptr = MCreateObject();
  obj_ptr->file_path = sdsnew(RedisModule_StringPtrLen(RedisModule_LoadString(rdb), NULL));
  obj_ptr->fd = open(obj_ptr->file_path, O_RDWR);

  struct stat sb;
  fstat(obj_ptr->fd, &sb);
  obj_ptr->file_size = sb.st_size;
  obj_ptr->mmap = mmap(NULL, sb.st_size, PROT_READ | PROT_WRITE, MAP_SHARED, obj_ptr->fd, 0);

  return obj_ptr;
}
Enter fullscreen mode Exit fullscreen mode
Appending to fmmap.c
Enter fullscreen mode Exit fullscreen mode

Function to use AOF in Redis.

%%writefile -a fmmap.c

void MAofRewrite(RedisModuleIO *aof, RedisModuleString *key, void *value)
{
  char buffer[0x200];
  MMapObject *obj_ptr = (MMapObject*)value;
  RedisModule_EmitAOF(aof, "MMAP", "sc",key, obj_ptr->file_path);
  RedisModule_EmitAOF(aof, "MCLEAR", "ss", key, obj_ptr->file_path);
  for (size_t i = 0; i < obj_ptr->file_size; i += sizeof(double)) {
    double value = *(double *)((uint8_t*)obj_ptr->mmap + i);
    sprintf(buffer, "%.16f", value);
    RedisModule_EmitAOF(aof, "MADD", "sbc", key, buffer);
  }
}
Enter fullscreen mode Exit fullscreen mode
Appending to fmmap.c
Enter fullscreen mode Exit fullscreen mode

Other functions required for Redis modules.

%%writefile -a fmmap.c

size_t MMemUsage(const void *value)
{
  const MMapObject *obj_ptr = value;
  return obj_ptr->file_size;
}

void MDigest(RedisModuleDigest *md, void *value)
{
  REDISMODULE_NOT_USED(md);
  REDISMODULE_NOT_USED(value);
}
Enter fullscreen mode Exit fullscreen mode
Appending to fmmap.c
Enter fullscreen mode Exit fullscreen mode

Macro to create Redis commands

%%writefile -a fmmap.c

#define CREATE_CMD(name, tgt, attr, key_pos, key_last)                     \
  do {                                                                     \
    if (RedisModule_CreateCommand(ctx, name, tgt, attr, key_pos, key_last, \
                                  1) != REDISMODULE_OK) {                  \
      return REDISMODULE_ERR;                                              \
    }                                                                      \
  } while (0);

Enter fullscreen mode Exit fullscreen mode
Appending to fmmap.c
Enter fullscreen mode Exit fullscreen mode

This function is called at module load time to prepare the module. This is where each command is associated with the function that executes it.

%%writefile -a fmmap.c

int RedisModule_OnLoad(RedisModuleCtx *ctx, RedisModuleString **argv, int argc)
{
  REDISMODULE_NOT_USED(argv);
  REDISMODULE_NOT_USED(argc);

  RedisModule_Init(ctx, "FuchiMMap", 1, REDISMODULE_APIVER_1);

  RedisModuleTypeMethods tm = {.version = REDISMODULE_TYPE_METHOD_VERSION,
                               .rdb_load = MRdbLoad,
                               .rdb_save = MRdbSave,
                               .aof_rewrite = MAofRewrite,
                               .mem_usage = MMemUsage,
                               .free = MFree,
                               .digest = MDigest};

  MMapType = RedisModule_CreateDataType(ctx, "FuchiMMap", 0, &tm);

  // MMAP key file_path
  CREATE_CMD("MMAP", MMap_RedisCommand, "write fast", 1, 1);

  // VCLEAR key
  CREATE_CMD("VCLEAR", VClear_RedisCommand, "write fast", 1, 1);

  // VADD key value [value ...]
  CREATE_CMD("VADD", VAdd_RedisCommand, "write fast", 1, 1);

  // VGET key index
  CREATE_CMD("VGET", VGet_RedisCommand, "readonly fast", 1, 1);

  // VMGET key index [index ...]
  CREATE_CMD("VMGET", VMGet_RedisCommand, "readonly fast", 1, 1);

  // VSET key index value [index value ...]
  CREATE_CMD("VSET", VSet_RedisCommand, "write fast", 1, 1);

  // VCOUNT key
  CREATE_CMD("VCOUNT", VCount_RedisCommand, "readonly fast", 1, 1);

  // VPOP key
  CREATE_CMD("VPOP", VPop_RedisCommand, "write fast", 1, 1);

  return REDISMODULE_OK;
}

Enter fullscreen mode Exit fullscreen mode
Appending to fmmap.c
Enter fullscreen mode Exit fullscreen mode

Makefile for building the source.

%%writefile Makefile.fmmap

SHOBJ_CFLAGS ?= -W -Wall -fno-common -g -ggdb -std=c99 -O2
SHOBJ_LDFLAGS ?= -shared

.SUFFIXES: .c .so .xo .o

all: fmmap.so

.c.xo:
    $(CC) -I. $(CFLAGS) $(SHOBJ_CFLAGS) -fPIC -c $< -o $@

fmmap.xo: ../redismodule.h

fmmap.so: fmmap.xo
    $(LD) -o $@ $^ $(SHOBJ_LDFLAGS) $(LIBS) -lc

clean:
    rm -rf *.xo *.so
Enter fullscreen mode Exit fullscreen mode
Writing Makefile.fmmap
Enter fullscreen mode Exit fullscreen mode

Let's build the module.

!make -f Makefile.fmmap
!ls fmmap.so
Enter fullscreen mode Exit fullscreen mode
cc -I.  -W -Wall -fno-common -g -ggdb -std=c99 -O2 -fPIC -c fmmap.c -o fmmap.xo
ld -o fmmap.so fmmap.xo -shared  -lc
fmmap.so
Enter fullscreen mode Exit fullscreen mode

Install Redis.

!sudo yes | add-apt-repository ppa:redislabs/redis
!sudo apt-get update
!sudo apt-get install redis
Enter fullscreen mode Exit fullscreen mode

Write something to the configuration file.

%%writefile -a /etc/redis/redis.conf
enable-module-command yes
loadmodule /content/redis-stable/src/modules/fmmap.so
Enter fullscreen mode Exit fullscreen mode
Appending to /etc/redis/redis.conf
Enter fullscreen mode Exit fullscreen mode

Run Redis.

!service redis-server start
Enter fullscreen mode Exit fullscreen mode
Starting redis-server: redis-server.
Enter fullscreen mode Exit fullscreen mode

Check Redis running.

!sleep 1
!ps aux | grep redis | grep -v grep
Enter fullscreen mode Exit fullscreen mode
redis       6204  0.0  0.0  59132  6420 ?        Ssl  06:08   0:00 /usr/bin/redis-server 127.0.0.1:6379
Enter fullscreen mode Exit fullscreen mode

Prepare the place to write a file.

!mkdir /content/db
!chmod 777 /content/db
Enter fullscreen mode Exit fullscreen mode

Maps a file to db. The return value is a number of values, 0 because it is new.

!echo "MMAP db /content/db/file.mmap" | redis-cli
Enter fullscreen mode Exit fullscreen mode
(integer) 0
Enter fullscreen mode Exit fullscreen mode

Confirm that the file.mmap file has been created. The file size is still zero.

!ls -l /content/db
Enter fullscreen mode Exit fullscreen mode
total 0
-rw-rw---- 1 redis redis 0 Jun 22 06:08 file.mmap
Enter fullscreen mode Exit fullscreen mode

Try to add a value. The return value is the number of values added.

!echo "VADD db 0.0" | redis-cli
Enter fullscreen mode Exit fullscreen mode
(integer) 1
Enter fullscreen mode Exit fullscreen mode

You can see that the file size has increased to 8 bytes.

!ls -l /content/db
Enter fullscreen mode Exit fullscreen mode
total 4
-rw-rw---- 1 redis redis 8 Jun 22 06:08 file.mmap
Enter fullscreen mode Exit fullscreen mode

Write the command to a file and run it.

%%writefile command
vadd db 0.1
vadd db 0.2
vadd db 0.3
vadd db 0.4
vadd db 0.5
vadd db 0.6
vadd db 0.7
vadd db 0.8
vadd db 0.9
vcount db
Enter fullscreen mode Exit fullscreen mode
Writing command
Enter fullscreen mode Exit fullscreen mode

When you run it, you will see that the number of registrations is 10.

!redis-cli < command
Enter fullscreen mode Exit fullscreen mode
(integer) 1
(integer) 1
(integer) 1
(integer) 1
(integer) 1
(integer) 1
(integer) 1
(integer) 1
(integer) 1
(integer) 10
Enter fullscreen mode Exit fullscreen mode

The file has also increased to 80 bytes.

!ls -l /content/db
Enter fullscreen mode Exit fullscreen mode
total 4
-rw-rw---- 1 redis redis 80 Jun 22 06:08 file.mmap
Enter fullscreen mode Exit fullscreen mode

Multiple additions can be made with a single command. The number of values added is returned.

!echo "VADD db 1.0 1.1 1.2 1.3 1.4 1.5" | redis-cli
Enter fullscreen mode Exit fullscreen mode
(integer) 6
Enter fullscreen mode Exit fullscreen mode

Let's take out the value.

!echo "VGET db 5" | redis-cli
Enter fullscreen mode Exit fullscreen mode
"0.5"
Enter fullscreen mode Exit fullscreen mode

If there is more than one, it will look like this. There is an error because of floating point.

!echo "VMGET db 1 2 3 4 5" | redis-cli
Enter fullscreen mode Exit fullscreen mode
1) "0.10000000000000001"
2) "0.20000000000000001"
3) "0.29999999999999999"
4) "0.40000000000000002"
5) "0.5"
Enter fullscreen mode Exit fullscreen mode

You can also change the values: set db[5] to 50, db[10] to 100, etc. The return value of VSET is the number of locations you have changed.

!echo "VSET db 5 50 10 100" | redis-cli
!echo "VMGET db 5 10" | redis-cli
Enter fullscreen mode Exit fullscreen mode
(integer) 2
1) "50"
2) "100"
Enter fullscreen mode Exit fullscreen mode

Extracts and removes the trailing value.

!echo "VCOUNT db" | redis-cli
!echo "VGET db 15" | redis-cli
!echo "VPOP db" | redis-cli
!echo "VCOUNT db" | redis-cli
Enter fullscreen mode Exit fullscreen mode
(integer) 16
"1.5"
"1.5"
(integer) 15
Enter fullscreen mode Exit fullscreen mode

Stop and restart Redis.

!service redis-server stop
!service redis-server start
Enter fullscreen mode Exit fullscreen mode
Stopping redis-server: redis-server.
Starting redis-server: redis-server.
Enter fullscreen mode Exit fullscreen mode

The values remain after rebooting.

!echo "VCOUNT db" | redis-cli
Enter fullscreen mode Exit fullscreen mode
(integer) 15
Enter fullscreen mode Exit fullscreen mode

If you delete the db, you can re-map it and get the values since the file is still in its original state.

!echo "KEYS *" | redis-cli
!echo "DEL db" | redis-cli
!echo "KEYS *" | redis-cli
!ls -l /content/db/
!echo "MMAP dba /content/db/file.mmap" | redis-cli
!echo "VGET dba 5" | redis-cli
Enter fullscreen mode Exit fullscreen mode
1) "db"
(integer) 1
(empty array)
total 4
-rw-rw---- 1 redis redis 120 Jun 22 06:08 file.mmap
(integer) 15
"50"
Enter fullscreen mode Exit fullscreen mode

To clear the contents, do this.

!echo "VCLEAR dba" | redis-cli
!echo "VCOUNT dba" | redis-cli
Enter fullscreen mode Exit fullscreen mode
(integer) 15
(integer) 0
Enter fullscreen mode Exit fullscreen mode

The size of the file will be zero.

!ls -l /content/db/
Enter fullscreen mode Exit fullscreen mode
total 0
-rw-rw---- 1 redis redis 0 Jun 22 06:08 file.mmap
Enter fullscreen mode Exit fullscreen mode

Delete dba.

!echo "DEL dba" | redis-cli
Enter fullscreen mode Exit fullscreen mode
(integer) 1
Enter fullscreen mode Exit fullscreen mode

Create a file with 100 doubles written in Python.

import struct
with open('/content/db/file.mmap', 'wb') as fout:
    for i in range(100):
        fout.write(struct.pack('d', i))
Enter fullscreen mode Exit fullscreen mode

Map the file and check the contents.

!echo "mmap db /content/db/file.mmap" | redis-cli
!echo "vcount db" | redis-cli
!echo "vmget db 1 3 5 7 9" | redis-cli
Enter fullscreen mode Exit fullscreen mode
(integer) 100
(integer) 100
1) "1"
2) "3"
3) "5"
4) "7"
5) "9"
Enter fullscreen mode Exit fullscreen mode

Stop Redis. No redis-server remains.

!echo "shutdown" | redis-cli
!sleep 1
!ps aux | grep redis
Enter fullscreen mode Exit fullscreen mode
root        6347  0.0  0.0   6904  3284 ?        S    06:08   0:00 /bin/bash -c ps aux | grep redis
root        6349  0.0  0.0   6444   728 ?        S    06:08   0:00 grep redis
Enter fullscreen mode Exit fullscreen mode

The file remains.

!ls -l /content/db
Enter fullscreen mode Exit fullscreen mode
total 4
-rw-rw---- 1 redis redis 800 Jun 22 06:08 file.mmap
Enter fullscreen mode Exit fullscreen mode

That is all.

Top comments (0)