DEV Community: Aniello Musella

CouchDB: Backup strategies

Aniello Musella — Fri, 22 Aug 2025 13:12:35 +0000

CouchDB is a good NoSQL DBMS with some features that, according to me, are awesome like, for instance, the multi-master synchronization and the HTTP/JSON API to access to data (check my previous article about CouchDB where I talk about the multi-master sync using Docker).

Today, I want to describe some of the possible strategies to back up a CouchDB database.

Manual strategy with tools

You can use CouchDB's Built-in Backup Tools to back up and restore a database. These tools allow you to create backups of your databases and restore them when needed. You can find this tools on GitHub.

Following some examples:

couchdbbackup -d my_database -o my_database_backup.couch --config couchdb_config.json

Where the file couchdb_config.json contains data to connect to the server:

{
  "username": "my_username",
  "password": "my_password",
  "url": "http://localhost:5984"
}

To restore, you'll use the following command:

couchdbrestore -d my_database -i my_database_backup.couch --config couchdb_config.json

Incremental backup with tools

With these tools you can define an incremental backup like that:

couchdbbackup -c couchdb_config.json --incremental

defining in the json configuration:

{
  "username": "my_username",
  "password": "my_password",
  "url": "http://localhost:5984",
  "database": "my_database",
  "output": "my_incremental_backup.couch",
  "incremental": true,
  "since": "<last_backup_sequence>"
}

Last backup sequence is an information that you get from database information. To get db information you can run:

curl -X GET http://localhost:5984/my_database

getting somenthing like that:

{
  "db_name": "my_database",
  "doc_count": 100,
  "doc_del_count": 0,
  "update_seq": 150,
  "purge_seq": 0,
  "compact_running": false,
  "disk_size": 2048000,
  "data_size": 2048000,
  "instance_start_time": "0",
  "disk_format_version": 0,
  "committed_update_seq": 150
}

The update_seq information indicates the sequence number of the last update made to the database, and this information will be used to make incremental backup.

So when you perform an incremental backup, you specify the update_seq value from your last backup. This tells the CouchDB backup tool to only include documents that have been added or modified since that sequence number.

Continuous Replication Strategy with CouchDB

I love this strategy because you use CouchDB to back up CouchDB.
To do that, set up continuous replication to another CouchDB instance. This provides real-time backup and ensures that your data is always up-to-date in the backup instance.

Snapshot Backups

If you're using CouchDB as a cloud service or on a virtual machine, you can take snapshots of the entire disk. This can be a quick way to back up the entire database state.

Scheduled Backups

You can schedule regular backups using cron jobs or similar scheduling tools. Depending on your data change frequency, you'll define your schedule (daily, weekly, etc.).

Final considerations

The best strategy it's mainly up to the architecture where CouchDB is running and the business requirements you must stick to.
In general, once you have chosen a tailored backup strategy for your business case it's advised following best practices like storing backups offsite, testing your backups, protecting your data with security and encryption and so on.

Suggestions and corrections are welcome.

CouchDB: Offline-first multi-master synchronization using Docker and Docker-compose

Aniello Musella — Tue, 05 Nov 2024 11:04:18 +0000

In this post, I'll show how to simulate a multi-master synchronization with Apache CouchDB considering an off-line scenario. To reach this goal, I'll use Docker and Docker compose.

What should you know?

Apache CouchDB (basic)
Docker
Docker Compose

What do you need?

Docker and Docker compose.

What's CouchDB

CouchDB is a NoSQL database document oriented using JSON format to store data.

The CouchDB's main features are:

Single node database.
Possibility to be configured as a cluster to increase the data availability.
Intuitive HTTP/JSON API to access to data (e.g. http://<yourserver>:<yourport>/<dbname>/<document id>/)
The use of multiversion concurrency control (MVCC) so the database file is not locked during writes (this helps performances). Conflicts are reported and managed at application level.
ACID properties at document level.
Multi master replication thanks to CouchDB Replication Protocol able to synchronize JSON documents between 2 peers over HTTP/1.1 by using the public CouchDB REST API based on the Apache CouchDB MVCC Data model.
Offline First Data Sync thanks to its replication protocol, data are always available even in scenario of bad connectivity.
Good platform scalability from server to mobile (PouchDB)

What's a multi-master synchronization

Quoting Wikipedia:

Multi-master replication is a method of database replication which allows data to be stored by a group of computers, and updated by any member of the group. All members are responsive to client data queries. The multi-master replication system is responsible for propagating the data modifications made by each member to the rest of the group and resolving any conflicts that might arise between concurrent changes made by different members.

CouchDB replication protocol lets to synchronize data between peers even in scenario of bad connectivity.

Multi-master synchronization: the scenario

In my scenario, I want to use multi-master synchronization to replicate data between three installations:

Alpha
Beta
Gamma

In this scenario, Alpha could be a database on main server (preferibly a cluster) and Beta and Gamma are edge installations with connectivity problems.

The requirement is that all data must be replicated through all database istallations. Below it's shown the synchronization schema schema.

It's worth note, in this schema, that the synchronization is bidirectional:

Alpha and Beta synchronize data bidirectionally
Alpha and Gamma synchronize data bidirectionally
Beta and Gamma don't need to be synchronized because they're already synchronized with Alpha, in this way I save network bandwidth.
Nothing prevents to implement data replication between Beta and Gamma, but in my case I prefer to save network bandwidth (so it's a precise architectural choice).

POC

Following I'll show all steps to simulate a multi-master synchronization. After tested the data replication, I'll consider an off-line scenario putting a database off-line.

Multi-master synchronization: Running all instances of CouchDB

To implement the scenario described above, I'll use Docker Compose to run a multi container application where each container represents respectively our Alpha, Beta and Gamma database installations.

I'll use the official CouchDb docker image published on DockerHub.

Following my docker-compose.yaml file:

version: '3'
services:
  couchserveralfa:
    image: couchdb
    restart: always
    ports:
      - "5984:5984"
    environment:
      - COUCHDB_USER=admin
      - COUCHDB_PASSWORD=A24cvmri
    volumes:
        - ./db/alpha:/opt/couchdb/data
        - ./config/alpha/local.ini:/opt/couchdb/etc/local.ini

  couchserverbeta:
    image: couchdb
    restart: always
    ports:
      - "5985:5984"
    environment:
      - COUCHDB_USER=admin
      - COUCHDB_PASSWORD=9VQhWrfW
    volumes:
        - ./db/beta:/opt/couchdb/data
        - ./config/beta/local.ini:/opt/couchdb/etc/local.ini

  couchservergamma:
    image: couchdb
    restart: always
    ports:
      - "5986:5984"
    environment:
      - COUCHDB_USER=admin
      - COUCHDB_PASSWORD=ouPFQ6mj
    volumes:
        - ./db/gamma:/opt/couchdb/data
        - ./config/gamma/local.ini:/opt/couchdb/etc/local.ini

I map the default port 5984 of CouchDB to host port 5984 for Alpha, 5985 for Beta and 5986 for Gamma.

In this compose I use an'external configuration file local.ini for each installation to set this configuration (source stack over flow):

[chttpd]
port = 5984
bind_address = 0.0.0.0

Finally, I've mapped on the host file system (in the folder ./db/[alpha|beta|gamma]/) each database.

Now we can start the databases:

am@animus:~$docker-compose up -d

Multi-master synchronization: Creating databases to be synchronized

Using Flauxton login (as admin) in each node, accessing to:

http://localhost:5984/_utils/ for Alpha installation
http://localhost:5985/_utils/ for Beta installation
http://localhost:5986/_utils/ for Gamma installation

For each installation, Select Databases from side menu and then Create Database, type the name of the database (in my case mydb) and click on create button like shown below:

Multi-master synchronization: Creating replication

Now it's the moment to create replication.
Select Replication side menu and then click on New Replication button.

In this page we can create the replications.

In CouchDB replications are documents belonging to _replicator DB, so when you click Start Replication you create a new document.

Multi-master synchronization: Creating replications on Database Alpha

On DB Alpha (port 5984) we create two continuos replication:

One toward Beta:

{
  "_id": "69b5d15c85f1e8e5352136a2ce00457f",
  "_rev": "1-87b5f963a45b2d8df14eb10a3cdbd1f0",
  "user_ctx": {
    "name": "admin",
    "roles": [
      "_admin",
      "_reader",
      "_writer"
    ]
  },
  "source": {
    "url": "http://localhost:5984/mydb",
    "headers": {
      "Authorization": "Basic YWRtaW46MTIz"
    }
  },
  "target": {
    "url": "http://<YOUR IP ADDRESS>:5986/mydb",
    "headers": {
      "Authorization": "Basic YWRtaW46MTIz"
    }
  },
  "create_target": false,
  "continuous": true,
  "owner": "admin"
}

One toward Gamma:

{
  "_id": "b15782ac94b2277c570b8689ab000d67",
  "_rev": "1-4912109dc6748874144aeef24ec23f20",
  "user_ctx": {
    "name": "admin",
    "roles": [
      "_admin",
      "_reader",
      "_writer"
    ]
  },
  "source": {
    "url": "http://localhost:5984/mydb",
    "headers": {
      "Authorization": "Basic YWRtaW46MTIz"
    }
  },
  "target": {
    "url": "http://<YOUR IP ADDRESS>:5985/mydb",
    "headers": {
      "Authorization": "Basic YWRtaW46MTIz"
    }
  },
  "create_target": false,
  "continuous": true,
  "owner": "admin"
}

Multi-master synchronization: Creating replications on Database Beta

On DB Beta (port 5985) we create one continuos replication toward Alpha:

{
  "_id": "b45d1f6f186500eaa926b6d84d000507",
  "_rev": "4-6884dcb72b7132562e10d9a83c59936c",
  "user_ctx": {
    "name": "admin",
    "roles": [
      "_admin",
      "_reader",
      "_writer"
    ]
  },
  "source": {
    "url": "http://localhost:5984/mydb",
    "headers": {
      "Authorization": "Basic YWRtaW46MTIz"
    }
  },
  "target": {
    "url": "http://<YOUR IP ADDRESS>:5984/mydb",
    "headers": {
      "Authorization": "Basic YWRtaW46MTIz"
    }
  },
  "create_target": false,
  "continuous": true,
  "owner": "admin"
}

Please note, even if we have mapped database Beta on host port 5895, the native port of the service remains 5894.

Multi-master synchronization: Creating replications on Database Gamma

On DB Gamma (port 5986) we create one continuous replication toward Alpha:

{
  "_id": "b45d1f6f186500eaa926b6d84d000507",
  "_rev": "4-6884dcb72b7132562e10d9a83c59936c",
  "user_ctx": {
    "name": "admin",
    "roles": [
      "_admin",
      "_reader",
      "_writer"
    ]
  },
  "source": {
    "url": "http://localhost:5984/mydb",
    "headers": {
      "Authorization": "Basic YWRtaW46MTIz"
    }
  },
  "target": {
    "url": "http://<YOUR IP ADDRESS>:5984/mydb",
    "headers": {
      "Authorization": "Basic YWRtaW46MTIz"
    }
  },
  "create_target": false,
  "continuous": true,
  "owner": "admin"
}

Please note, even if we have mapped database Gamma on host port 5896, the native port of the service remains 5894.

Multi-master synchronization: Creating documents

Now we're ready to create new documents.

Choose a database (alpha, beta or gamma) and go to side menu bar, select Databases, then select Create Document. Compose your document and then confirm.

Wait few seconds and the document you've just created is sent to other databases (here we go!).

Multi-master synchronization: Creating documents off-line

I turn off database Alpha and Gamma running the following commands:

am@animus:~$docker kill couchdb_couchserveralfa_1 && docker kill couchdb_couchserverbeta_1

On database gamma I create the document while the other databases are off-line.

After the creation of the new document I turn on the other databases running:

am@animus:~$docker-compose up -d

Wait few seconds and the document you've just created on Gamma database is sent while the other databases were off-line. So we've just proved that the synchonization works when they return on-line (here we go!).

Conclusions

In this article, I've shown how to simulate the multi-master synchronization of CouchDB and how to simulate an off-line scenario where I show that data keep on to be available for local operations and then be synchronized when the database returns on-line.
In these scenarios, CouchDB turned out a good solution.
Suggestions and corrections are welcome.

MongoDb Atlas: manual backup and restore data

Aniello Musella — Wed, 16 Oct 2024 16:34:38 +0000

In this post, I'll show how to back up and restore data from a database MongoDB hosted on MongoDB Atlas.

What should you know?

Basic knowledge of MongoDB
Basic knowledge of Bash Scripts
Basic knowledge of MongoDB Tools

What do you need?

Bash shell
MongoDB Atlas cluster with a DB deployed
mongodump
mongorestore

Backup data

Download and install mongodump.

The mongodump is, quoting from the official site, "a utility that creates a binary export of a database's contents...".

Before to use this tool you must get the db connection string, and depending on cluster structure you have to compose your connection string.

In the official guide they explain how to compose the connection string in case you have to connect to one instance, replica set and sharded cluster.

In my case is a replica set and my connection string is something like shown below:

mongodb://<user>:<password>@mycluster-00-00.xyzw.mongodb.net:27017,@mycluster-00-01.xyzw.mongodb.net:27017,@mycluster-00-02.xyzw.mongodb.net:27017

So, once we prepared the connection string, we are ready to back up data of our database:

am@animus:~$ mongodump \
--uri="mongodb://mongodb://<user>:<password>@mycluster-00-00.xyzw.mongodb.net:27017,@mycluster-00-01.xyzw.mongodb.net:27017,@mycluster-00-02.xyzw.mongodb.net:27017" \
--ssl \
--authenticationDatabase=<db user> \
--db=<database name>

The command will produce a folder dump with all data backup.

Restore data

Download and install mongorestore.

The mongorestore is, quoting from official site, "a program that loads data from either a binary database dump created by mongodump or the standard input into a mongod or mongos instance".

Before to use this tool we need the connection string to the destination cluster and the considerations made for the dump are valid for the restore too.

am@animus:~$ mongorestore \
--uri="mongodb://mongodb://<user>:<password>@mycluster-00-00.xyzw.mongodb.net:27017,@mycluster-00-01.xyzw.mongodb.net:27017,@mycluster-00-02.xyzw.mongodb.net:27017" \
--ssl \
--authenticationDatabase=<db user> \
--db=<database name> \
<database dump folder>

After the command run is completed, you'll have the database completely restored.

Conclusions

MongoDB Atlas is a cloud service and, as such, you can count on better backup and restore data strategies able to not impact on the normal activity of the service. The manual backup and restore could be useful in dev or test scenario and when the dimension of data to copy is not so huge. Take in mind the use of mongodump and mongorestore in production could have an impact on the normal activity of the service.

React: stale closure

Aniello Musella — Mon, 19 Aug 2024 14:28:46 +0000

In this post, I'll show how to create a closure in a useState hook React app.

I'll not explain what a closure is, because there are many resources about this topic and I don't want to be repetitive. I advise the reading of this article by @imranabdulmalik.

In short, a closure is (from Mozilla):

...the combination of a function bundled together (enclosed) with references to its surrounding state (the lexical environment). In other words, a closure gives you access to an outer function's scope from an inner function. In JavaScript, closures are created every time a function is created, at function creation time.

Just in case you're not familiar with the term lexical environment, you can read this article by @soumyadey or alternatively this one.

The problem

In a React application, you can create accidentally a closure of a variable belonging to the component state created with useState hook. When this happens, you're facing a stale closure problem, that is to say, when you refer to an old value of the state that in the meantime it's changed, and so it's not more relevant.

POC

I've created a Demo React application which the main goal is to increment a counter (belonging to the state) that can be closed in a closure in the callback of setTimeout method.

In short, this app can:

Show the value of the counter
Increment by 1 the counter
Start a timer to increment the counter by 1 after five seconds.
Increment by 10 the counter

In the following picture, it's shown the initial UI state of the app, with counter to zero.

We'll simulate the closure of the counter in three steps:

Incrementing by 1 the counter

Starting the timer to increment by 1 after five seconds

Incrementing by 10 before the timeout triggers

After 5 seconds, the value of the counter is 2.

The expected value of the counter should be 12, but we get 2.

The reason why this happens it's because we've created a closure of the counter in the callback passed to setTimeout and when the timeout is triggered we set the counter starting from its old value (that was 1).

setTimeout(() => {
        setLogs((l) => [...l, `You closed counter with value: ${counter}\n and now I'll increment by one. Check the state`])
        setTimeoutInProgress(false)
        setStartTimeout(false)
        setCounter(counter + 1)
        setLogs((l) => [...l, `Did you create a closure of counter?`])

      }, timeOutInSeconds * 1000);

Following the full code of the app component.

function App() {
  const [counter, setCounter] = useState<number>(0)
  const timeOutInSeconds: number = 5
  const [startTimeout, setStartTimeout] = useState<boolean>(false)
  const [timeoutInProgress, setTimeoutInProgress] = useState<boolean>(false)
  const [logs, setLogs] = useState<Array<string>>([])

  useEffect(() => {
    if (startTimeout && !timeoutInProgress) {
      setTimeoutInProgress(true)
      setLogs((l) => [...l, `Timeout scheduled in ${timeOutInSeconds} seconds`])
      setTimeout(() => {
        setLogs((l) => [...l, `You closed counter with value: ${counter}\n and now I'll increment by one. Check the state`])
        setTimeoutInProgress(false)
        setStartTimeout(false)
        setCounter(counter + 1)
        setLogs((l) => [...l, `Did you create a closure of counter?`])

      }, timeOutInSeconds * 1000);
    }
  }, [counter, startTimeout, timeoutInProgress])

  function renderLogs(): React.ReactNode {
    const listItems = logs.map((log, index) =>
      <li key={index}>{log}</li>
    );
    return <ul>{listItems}</ul>;
  }

  function updateCounter(value: number) {
    setCounter(value)
    setLogs([...logs, `The value of counter is now ${value}`])
  }

  function reset() {
    setCounter(0)
    setLogs(["reset done!"])
  }

  return (

    <div className="App">
      <h1>Closure demo</h1>
      <hr />
      <h3>Counter value: {counter}</h3><button onClick={reset}>reset</button>
      <hr />
      <h3>Follow the istructions to create a <i>closure</i> of the state variable counter</h3>
      <ol type='1' className=''>
        <li>Set the counter to preferred value <button onClick={() => updateCounter(counter + 1)}>+1</button> </li>
        <li>Start a timeout and wait for {timeOutInSeconds} to increment the counter (current value is {counter}) <button onClick={() => setStartTimeout(true)}>START</button> </li>
        <li>Increment by 10 the counter before the timeout  <button onClick={() => updateCounter(counter + 10)}>+10</button> </li>
      </ol>
      <hr />
      {
        renderLogs()
      }
    </div >
  );
}

export default App;

Solution

The solution is based on the use of useRef hook that lets you reference a value that’s not needed for rendering.

So we add to the App component:

const currentCounter = useRef(counter)

Then we'll modify the callback of setTimeout like shown below:

setTimeout(() => {
        setLogs((l) => [...l, `You closed counter with value: ${currentCounter.current}\n and now I'll increment by one. Check the state`])
        setTimeoutInProgress(false)
        setStartTimeout(false)
        setCounter(currentCounter.current + 1)
        setLogs((l) => [...l, `Did you create a closure of counter?`])

      }, timeOutInSeconds * 1000);

Our callback needs to read the counter value because we log the current value before to increment it.

In case, you don't need to read the value, you can avoid the closure of the counter just using the functional notation to update the counter.

setCounter(c => c + 1)

Credits

Dmitri Pavlutin Be Aware of Stale Closures when Using React Hooks
Imran Abdulmalik Mastering Closures in JavaScript: A Comprehensive Guide
Keyur Paralkar Lexical Scope in JavaScript – Beginner's Guide
Souvik Paul Stale Closures in React
Soumya Dey Understanding Lexical Scope & Closures in JavaScript
Subash Mahapatra stackoverflow

Log rotation with Python

Aniello Musella — Wed, 25 Oct 2023 18:11:21 +0000

In this post, I show how to implement log rotation in a Python project.

What's log rotation?

From Wikipedia:

In information technology, log rotation is an automated process used in system administration in which log files are compressed, moved (archived), renamed or deleted once they are too old or too big (there can be other metrics that can apply here). New incoming log data is directed into a new, fresh file (at the same location)

So, it's important to consider log rotation specially in that systems that produce a great quantity of log messages.

There are different criteria of log rotation and in this post I'll focus on timed log rotation and file size rotation.

How we implement logging in Python?

Super easy, just using the logging module.

The logging module offers a default logger named "root". This logger allows you to get started with no configuration.

import logging
logging.info('This is an info message')
logging.warning('This is a warning message')
logging.error('This is an error message')
logging.critical('This is a critical message')

This module permit a lot of customizations like:

Logging severity level
File name
Log format
etc.

You can create a custom logger using the class Logger, defining a basic configuration (date format, log format, etc.).

  logging.basicConfig(
        format='%(asctime)s %(levelname)s %(message)s',
        level=logging.DEBUG,
        datefmt='%m/%d/%Y %I:%M:%S %p',
    )

    # Log formatter definition
    log_formatter = logging.Formatter(fmt=' %(asctime)s %(levelname)s %(message)s')

    # Getting the logger
    logger = logging.getLogger('my-logger')

    # Using the logger
    logging.debug('This is a debug message')
    logging.info('This is an info message')
    logging.warning('This is a warning message')
    logging.error('This is an error message')
    logging.critical('This is a critical message')

To implement different ways to log information, Python uses log handlers.

The log handlers are components that effectively writes logs, and they give you the opportunity of:

Display logs in the console (StreamHandler)
Write logs on files (FileHandler)
Write logs on files with log rotation based on file size criteria (RotatingFileHandler)
Write logs on files with log rotation based on time conditions (TimedRotatingFileHandler)
Sending logs via SMTP (SMTPHandler)
etc.

You can create your custom handler or extend a pre-existing handler.

File Logging with Python

To implement file logging, we'll use the FileHandler, a handler able to write logs to the file system.
Below is shown how to define this handler using the same basic configuration defined previously.

    # File handler definition
    fh: logging.FileHandler = logging.FileHandler(
        filename=f'./logs/my-app-file.log',
        mode='a'
    )

    # Setting the formatter for the handler
    fh.setFormatter(log_formatter)

    # Setting the severity level for the handler
    fh.setLevel(logging.INFO)

    # Adding handler to the logger
    logger.addHandler(fh)

    #Trying the handler using the handler
    logger.info('This log is an info message')

This handler simply writes log messages to a file named my-app-file.log.

Log rotation with Python

File size log rotation

To implement logging rotation based on the size of the file, we'll use the RotatingFileHandler. This handler is used to write logs on the file system and to trigger a rotation when a specific size of the file is reached.
Below is shown how to define this handler using the same basic configuration defined previously.

    # Rotation file handler definition
    rfh: RotatingFileHandler = RotatingFileHandler (
        filename=f'./logs/my-app.log',
        maxBytes=100,
        backupCount=5
    )

    # Setting the formatter for the handler
    rfh.setFormatter(log_formatter)

    # Setting the severity level for the handler
    rfh.setLevel(logging.INFO)

    # Adding handler to the logger    
    logger.addHandler(rfh)

    # Trying the handler using the handler
    logger.info('This log is an info message')

In this example, we've set a max dimension of 100 bytes and a backup count of 5 log files (this means that we keep at most 5 files from the log history).

A result of this rotation after writing more than 1024 bytes could be something like shown below:

my-app.log
my-app.log.1
my-app.log.2
my-app.log.3
my-app.log.4
my-app.log.5

Timed log rotation with Python

To implement file rotation logging based on time event, we'll use the TimedRotatingFileHandler that is the handler used to write logs on the file system and trigger a rotation when a time condition is satisfied.
Below is shown how to define this handler using the same basic configuration of the previous examples.

    rth: TimedRotatingFileHandler = TimedRotatingFileHandler(
        filename=f'./logs/{log_file_prefix}.log',
        backupCount=5,
        when='midnight',
    )
    rth.setFormatter(log_formatter)
    rth.setLevel(logging.INFO)
    rth.namer = lambda name: name.replace(".log", "") + ".log"


    logger.addHandler(rth)
    logger.info('This log is an info message')

In this example, we've set a daily rotation triggered each midnight and a backup count of 5 files.

A result of this rotation after 5 days could be something like shown below:

my-app.log
my-app.2023-10-12.log
my-app.2023-10-11.log
my-app.2023-10-10.log
my-app.2023-10-09.log
my-app.2023-10-08.log

For this handler, we've defined a namer with a lambda function that has the goal to preserve the extension of the log files. This namer get in on the action when a rotation happens.

Custom File timed log rotation with Python

It's possible to create the own log handler extending a pre-existing handler. For instance, we can extend the TimedRotatingFileHandler class in a way to introduce, for each log message emission, a check about the presence of the file. In this way, if someone or something deletes the file, the handler is able to create it again, and we don't miss further log messages.

Let's create the class:

class TimedRotatingFileHandlerWithFileCheck(TimedRotatingFileHandler):
    """
    Handler for logging to a file, rotating the log file at certain timed
    intervals. This handler based on TimedRotatingFileHandler offers a file 
    presence check for each log emission.
    """
    def __init__(self, filename, when='h', interval=1, backupCount=0,
                 encoding=None, delay=False, utc=False, atTime=None,
                 errors=None):
        super().__init__(filename, when, interval, backupCount, encoding, delay, utc, atTime, errors)

    def emit(self, record):
        if not os.path.isfile(self.baseFilename):
            os.mknod(self.baseFilename)
        super(TimedRotatingFileHandlerWithFileCheck, self).emit(record)

In this version, before to emit a log, we check the presence of the file.

Conclusions

In Python, logging is super easy! The logging module offers a lot of possibilities to customize and implement the strategy more suitable for you. The presence of a lot of handlers and the possibility to have a high level of customization makes this module very useful.
In my opinion, having a good and balanced logging system is a must for each system or application.

Suggestions and corrections are welcome.

Credits

Dev Container for React Native with Expo

Aniello Musella — Fri, 22 Sep 2023 16:49:30 +0000

In this post, I'll show how to create a Development container (aka dev container) to use it to develop a React Native app with Expo.

What should you know?

Basic knowledge of Docker
Basic knowledge of React Native
Basic knowledge of Expo
Basic knowledge of Bash

What do you need?

Docker installed.
Visual Studio Code installed.
A Linux box with bash (for Windows users, you can use WSL).
Expo client on your phone.

Why dev containers?

In short:

You prepare the dev environment one time and every developer can use it.
Dev containers are always ready when you need it.
Dev containers are always consistent for every developer.
Dev containers are fully integrated with popular IDE like Visual Studio Code, PyCharm and others.
You can use them remotely

If you want to go into the details, check the following link from @blackgirlbytes Why are people developing inside containers?

Dev Container preparation

Let's prepare the working directory:

animus@devto:~$ mkdir my-app
animus@devto:~$ cd my-app
animus@devto:~$ git init -b main

Start Visual Studio Code:

animus@devto:~$ code .

Type CTRL+SHIFT+P to run the Command Palette and when ready type Dev Containers: Add Dev Container configuration files

Choose the dev container Node.js & TypeScript container like shown below.

Choose default image and don't select any features.

After saved the dev container configuration, you should see a folder named ./devcontainer with a JSON file devcontainer.json containing the default definition of the container.

// For format details, see https://aka.ms/devcontainer.json. For config options, see the
// README at: https://github.com/devcontainers/templates/tree/main/src/typescript-node
{
    "name": "Node.js & TypeScript",
    // Or use a Dockerfile or Docker Compose file. More info: https://containers.dev/guide/dockerfile
    "image": "mcr.microsoft.com/devcontainers/typescript-node:1-20-bullseye"

    // Features to add to the dev container. More info: https://containers.dev/features.
    // "features": {},

    // Use 'forwardPorts' to make a list of ports inside the container available locally.
    // "forwardPorts": [],

    // Use 'postCreateCommand' to run commands after the container is created.
    // "postCreateCommand": "yarn install",

    // Configure tool-specific properties.
    // "customizations": {},

    // Uncomment to connect as root instead. More info: https://aka.ms/dev-containers-non-root.
    // "remoteUser": "root"
}

Edit this file like shown below:

// For format details, see https://aka.ms/devcontainer.json. For config options, see the
// README at: https://github.com/devcontainers/templates/tree/main/src/typescript-node
{
    "name": "Node.js & TypeScript",
    // Or use a Dockerfile or Docker Compose file. More info: https://containers.dev/guide/dockerfile
    "image": "mcr.microsoft.com/devcontainers/typescript-node:1-20-bullseye",

    // Features to add to the dev container. More info: https://containers.dev/features.
    // "features": {},

    // Use 'forwardPorts' to make a list of ports inside the container available locally.
    "forwardPorts": [8081],
    "initializeCommand": "bash .devcontainer/initializeCommand.sh",
    // Use 'postCreateCommand' to run commands after the container is created.
    "postCreateCommand": "bash .devcontainer/postCreateCommand.sh",

    // Configure tool-specific properties.
    // "customizations": {},

    // Uncomment to connect as root instead. More info: https://aka.ms/dev-containers-non-root.
    // "remoteUser": "root",

    "runArgs": ["-p=8081:8081", "--env-file", ".devcontainer/.env"]

}

In this new version of this file, I've added:

initializeCommand This section permit to execute a command before the build of the container. I've used this command to execute a script to generate a file of environment variables to use them in the container (the script will be described in the next chapter).

  bash .devcontainer/initializeCommand.sh

postCreateCommand This section permit to execute a command after the build of the container. I've used this command to execute a script to install Expo and other dependencies like watchman

  bash .devcontainer/postCreateCommand.sh

forwardPorts I've exposed the port number 8081 to permit the Expo client to access to your app.

  "forwardPorts": [8081]

runArgs I've defined an argument to run the container:
- port binding with the host
- environment file (generated in the initialize command script)

    "runArgs": ["-p=8081:8081", "--env-file", ".devcontainer/.env"]

The initialize command script

The initialize command script initializeCommand.sh located in ./devcontainer folder has the goal to gather your IP address (the script is inspired by this one). The address is set in an environment variable named REACT_NATIVE_PACKAGER_HOSTNAME used by Expo to expose the app.
This variable will'be printed in the file named .env located in the same directory of the script.

The content of the script:

#!/bin/bash
echo "Gathering you ip for dev container"

##############################################################################################
# en (Ethernet) - ib (InfiniBand) - sl (Serial line IP (slip)) - wl (Wireless local area 
network (WLAN)) - ww (Wireless wide area network (WWAN))
#############################################################################################
your_interface_name="eno" 
interface_prefix="en" # Choose the interface network.
iname=$(ip -o link show | sed -rn '/^[0-9]+: en/{s/.: ([^:]*):.*/\1/p}')
ip=`ifconfig $iname  | grep -Eo 'inet (addr:)?([0-9]*\.){3}[0-9]*' | sed 's/inet //g'`

echo "REACT_NATIVE_PACKAGER_HOSTNAME=$ip" > .devcontainer/.env

The content of the .env file:

REACT_NATIVE_PACKAGER_HOSTNAME=192.168.0.72

The post create command script

The post create command script postCreateCommand.sh located in ./devcontainer folder has the goal to set up Expo and watchman as described in the Expo guide.

The content of the script:

echo -e "\nStarting post creat command script..."
echo "Dev machine:"
uname -a
echo -e "\nInstalling expo boiler plate..."
npm install --save-dev -y create-expo-app@2.1.1
echo -e "\nInstalling watchman...\n"
sudo apt update
sudo apt install watchman
watchman version

echo -e "\n*******************************"
echo -e "\nDev container ready!".
echo -e "\n*******************************\n"

Create and run the Dev Container

Open the Command Palette (CTRL+SHIFT+P) and select Rebuild and Reopen in Container

Create the app

To create the app, open a terminal inside the dev container and run the following command:

animus@mydevcontainer:~$ npx create-expo-app -t blank myapp

Run the app

To run the app, type the following command:

animus@mydevcontainer:~$ cd myapp
animus@mydevcontainer:~$ npx expo start

You'll see something like this:

Scan the barcode with the Expo Client App (from your smartphone), and you'll see your app "myapp"!

Have a fun!

Conclusions

Dev containers are great when you want to prepare a dev environment fully replicable for you and for your team. The little effort that you put at the beginning is well paid off later. Dev containers are useful even in the mobile development using React Native with Expo improving the productivity of the team.

Suggestions and corrections are welcome.

How to create pull request on GitHub using bash shell and gh command line tool for coding review

Aniello Musella — Mon, 10 Jul 2023 21:24:41 +0000

In this post I show how to create a pull request on GitHub to ask a coding review to another coder (e.g. a supervisor, a team leader or just another coder). To do that, I'll use the bash shell and the GitHub CLI.

What do you need?

The script

You can download the script here.

#!/bin/bash

coder="animusna" # THE CODER NAME WILL RECEIVE THE CODING REVIEW REQUEST
branch="develop" # THE ORIGIN BRANCH ON WHICH YOU'LL MAKE THE PULL REQUEST 

if [ $# -eq 0 ]
  then
    echo "***No arguments supplied"
    exit 1
fi

if [ -z "$1" ]
  then
    echo "***Pull request title not supplied"
    exit 1
fi


if [ -z "$2" ]
  then
    echo "***Pull request message not supplied"
    exit 1
fi

currentBranch=`git rev-parse --abbrev-ref HEAD`

echo -e "\nChecking login status Github cli"

gh auth status

echo -e "\nPushing you branch to origin..."

git push origin $currentBranch

echo -e "\nCreating pull request for branch \"$branch\" asking to \"$coder\" to review...\n"
gh pr create --title "$1" --body "$2" -r $coder -B $branch


echo -e "\n\nbye ;)"

The script requires to be configured:

The GitHub account name of the coder will receive the coding review request.
The branch on which you'll open the pull request.

These parameters are hard coded, but you can change the script to receive these values via shell parameters.

Before to run the script

You'll need to install the GitHub command line tool following these instructions.

After the setup, you must authenticate typing the following command:

animus@devto:~$gh auth login

Follow the instructions and complete the authentication process.

How to run the script

To run the script, you must provide the following parameters:

Pull request title
Pull request message

animus@devto:~$./ps.sh 'smart feature' 'check this out...'

Running the script, you'll see something like shown in the image below.

After the execution of the script, the reviewer will receive a notification about the request just created.

Conclusions

The script described in this post is very simple, and it shows just one of many potential that the GitHub CLI could offer in a team to collaborate and to automate operations on GitHub repositories.

Redis: deleting many keys with pattern matching using bash shell + redis-cli

Aniello Musella — Tue, 02 May 2023 08:08:51 +0000

Have you ever had the need to delete many keys in a Redis server? If you want to do that, and you don't want to write code (python, c#, java, etc.), you can use the script described in this post.

What do you need?

A bash shell
Redis installed with authentication enabled.
Redis-CLI

I assume that you have a basic knowledge of Redis and Redis CLI and Bash scripting.

The reason why I made the script

Sometimes you can have the need to remove a set of keys that match with a specific pattern in your Redis server (e.g. "mypattern*", "*mypattern*", "users*", etc.). To reach this goal you can develop a solution in some language (python, c#, java, etc.) or, alternatively, you can use Redis CLI. The script uses the second strategy.

How the script works

The script uses Redis CLI, running two commands:

KEYS, to get all keys match with the input pattern
DEL, to delete the keys (one at once).

The script read before all the keys matching the pattern using the command Redis CLI KEYS and then, it runs the command Redis CLI DEL to remove them. The script, to pipe each key to Redis CLI DEL command, use xargs, so each key can be deleted from Redis.

The script

The script has three variables you have to set. The first two are about your installation (host and port) and the third one represents the pattern of keys you need to remove.

#!/bin/bash

#######################################################################################################
# Set your parameters
#######################################################################################################
port=6379
host="localhost"
pattern="mypattern*"
#######################################################################################################

echo -e "\nRedis @$host:$port"
echo -e "\nTrying to delete all keys with pattern: $pattern\n"

echo -e "\nEnter Redis Password : "
read -s password

keys_found=`redis-cli -p $port -h $host -a $password --raw --no-auth-warning KEYS $pattern | xargs | wc -w`

if [ "$keys_found" -eq "0" ]; then
    echo -e "\nKeys not found using pattern $pattern\n"  
    exit 1
fi

echo ""
echo "Deleting all keys with pattern $pattern"
redis-cli -p $port -h $host -a $password  --no-auth-warning KEYS $pattern | xargs redis-cli -p $port -h $host -a $password --raw  --no-auth-warning DEL

Run the script

Once you've set the parameters section, run the script!
When the script starts, it'll ask the password to authenticate on your Redis server, provide the password and wait for the conclusion of the execution.


you@yourworkstation$./redis-keys-delete.sh

Redis @localhost:6379

Trying to delete all keys with pattern: mypattern*


Enter Redis Password : 

Deleting all keys with pattern mypattern*
2

Conclusion

This simple script could be useful when you are developing, and you need to remove one or more keys from your Redis server (e.g. deployed with Docker on your workstation). The script could be a starting point to do something different, or it could be improved managing the variables as parameters, and so on. I hope this script could be helpful to someone. Suggestions and corrections are welcome.

Implementing queues with Amazon SQS

Aniello Musella — Thu, 10 Nov 2022 09:01:31 +0000

In this post I'll show how to use Amazon Simple Query Service a.k.a. Amazon SQS providing a coding example in Python implementing the "producer consumer problem".

What do you need before to start?

AWS account If you don't have one you can activate it using free-tier plan.
AWS CLI installed.
Python basic knowledge level
Python 3 installed
Access to a bash shell

Scenario

The goal of this article is to implement a classic consumer/producer problem using a FIFO queue where a producer sends one or more messages to a queue and a consumer receives these messages reading from the same queue. We choose a FIFO queue model to guarantee that the receiving order of the messages is the same of the sending order (pretending that this matters).

What's Amazon SQS

From the documentation:

Amazon Simple Queue Service (SQS) is a managed message queuing service that enables you to decouple and scale microservices, distributed systems, and serverless applications. SQS reduces the complexity of managing and operating message-oriented middleware. SQS is designed so that you can send, store, and receive messages between software components at any volume, without losing messages or requiring other services to be available.

We'll use this service to create and use a FIFO queue.

Setting up the environment

Let's set up the cloud.

Configure AWS Command Line Interface

To permit to AWS CLI to access to the AWS cloud we need to configure it running the command aws configure

am@animusna:~$aws configure
AWS Access Key ID [None]: AKIAIOSFODNN7EXAMPLE
AWS Secret Access Key [None]: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
Default region name [None]: eu-west-1
Default output format [None]: json

In case you need to create the keys follow the official guide

Creation of a FIFO queue

Let's create a FIFO queue running the command aws sqs create-queue:

am@animusna:~$aws sqs create-queue --queue-name test.fifo --attributes FifoQueue=true
{
    "QueueUrl": "https://eu-west-1.queue.amazonaws.com/123456789012/test.fifo"
}

The output is in JSON format and it represents the URL of the queue that will be used in the demo.

Let's code

The solution is composed by three Python files and one configuration file.
To implement the demo we'll use:

boto3 the official AWS SDK for Python.
colorama module to to prettify the printings in the terminal and used in the demo to make more understandable the outputs of different threads.

Implementing the consumer

In the consumer.py is defined the class Consumer that implements the message reader of the queue. The consumer read one message at a time and after the reading remove it.

import boto3
from configparser import ConfigParser
import messageSqs
import warnings

warnings.filterwarnings('ignore', category=FutureWarning, module='botocore.client')

class Consumer:
    def __init__(self, region_name, queue_url,logger):
        self.region_name = region_name
        self.queue_url = queue_url
        self.logger=logger

    def receive_message(self,deleteAfterReceiveing):
        sqs_client = boto3.client("sqs", region_name=self.region_name)
        response = sqs_client.receive_message(
            QueueUrl=self.queue_url,
            MaxNumberOfMessages=1,
            WaitTimeSeconds=15,
        )

        messages=response.get("Messages", [])

        self.logger(f"Number of messages received: {len(messages)}")

        if len(messages)==0:
            return False

        for message in messages:
            m=messageSqs.MessageSqs(jsonStr=message["Body"])
            self.logger(f"\tMessage read from queue: Data:{m.data}\tTime Stamp:{m.ts}\t Id:{m.id}")
            if deleteAfterReceiveing:
                receipt_handle = message['ReceiptHandle']
                dlt_response=sqs_client.delete_message(QueueUrl=self.queue_url,ReceiptHandle=receipt_handle)
                if dlt_response['ResponseMetadata']['HTTPStatusCode']==200:
                    self.logger("\tMessage deleted from queue.")
        return True

Implementing the producer

In the producer.py it's defined the Producer class that implements the message sender of the queue. The producer create one message at a time to send it to the queue.

import boto3
import uuid
from messageSqs import MessageSqsJSONEncoder
import warnings

warnings.filterwarnings('ignore', category=FutureWarning, module='botocore.client')

class Producer:

    def __init__(self,region_name,queue_url,logger):
        self.region_name = region_name
        self.queue_url= queue_url
        self.logger=logger

    def send_message(self,message):
        sqs_client = boto3.client("sqs", self.region_name)
        self.logger(f"Sending message with id {message.id}...")        
        response = sqs_client.send_message(
            QueueUrl=self.queue_url,
            MessageBody=MessageSqsJSONEncoder().encode(message),
            MessageGroupId="test",
            MessageDeduplicationId=f"DeduplicationId-{uuid.uuid4()}"
        )
        self.logger(f"SQS Response: Status:{response['ResponseMetadata']['HTTPStatusCode']}\tSQS Message Id:{response['MessageId']}")

Implementing the scenario

In the module sqs-demo.py we run in different concurrent threads the producer and consumer.The producer send a message each half second for a max of M (where M is PRODUCER_MESSAGES_TO_SENT defined in configuration file). The consumer try to read a message each half second until there is something to read. The consumer stops after detecting an empty queue for K times (where K is CONSUMER_EXIT_IF_QUEUE_IS_EMPTY_AFTER_ATTEMPTS defined in the configuration file).

from colorama import Fore
from configparser import ConfigParser
from consumer import Consumer
from datetime import datetime
from messageSqs import MessageSqs
from producer import Producer
import time
import threading
import uuid

config_object = ConfigParser()
config_object.read("config.ini")
aws_conf = config_object["AWS"]
demo_conf = config_object["DEMO"]
message_to_sent = int(demo_conf["PRODUCER_MESSAGES_TO_SENT"])
max_attempts_if_queue_is_empty = int(
    demo_conf["CONSUMER_EXIT_IF_QUEUE_IS_EMPTY_AFTER_ATTEMPTS"])
consumer_start_delay = float(demo_conf["CONSUMER_START_DELAY_IN_SECONDS"])
producer_start_delay = float(demo_conf["PRODUCER_START_DELAY_IN_SECONDS"])


def log(task_name, text, textColor):
    print(textColor + f"{task_name}:{text}" + Fore.WHITE)

def produce_task(task_name="producer"):
    logger=lambda text: log(task_name, text, Fore.YELLOW)
    logger("Running ...")
    time.sleep(producer_start_delay)

    message_index=0
    producer=Producer(aws_conf["REGION"], aws_conf["QUEUE_URL"], logger)
    while (message_index < message_to_sent):
        m=MessageSqs(
            jsonStr = f"{{\"data\":\"mydata{message_index}\",\"ts\":\"{datetime.now().isoformat()}\",\"id\":\"{uuid.uuid4()}\" }}")
        producer.send_message(m)
        message_index += 1
        time.sleep(0.5)
    logger("Terminated!")


def consume_task(task_name = "consumer"):
    logger=lambda text: log(task_name, text, Fore.GREEN)
    receiver = Consumer(aws_conf["REGION"],
                        aws_conf["QUEUE_URL"], logger)
    logger("Running...")
    time.sleep(consumer_start_delay)
    attempts = 0
    while (attempts < max_attempts_if_queue_is_empty):
        if not receiver.receive_message(True):
            attempts += 1
        time.sleep(0.5)

print(Fore.WHITE + '\nStarting demo AWS SQS..\n')

if __name__ == "__main__":
    tasks = []
    tasks.append(threading.Thread(target=produce_task, args=("Producer-SQS",)))
    tasks.append(threading.Thread(target=consume_task, args=("Consumer-SQS",)))

    for t in tasks:
        t.start()

    for t in tasks:
        t.join()

    print(Fore.WHITE + "\n\nAWS SQS Demo terminated!\n\n")

The configuration file

In the file config.ini there is the configuration of the demo:

[AWS]
REGION = eu-west-1
QUEUE_URL = https://eu-west-1.queue.amazonaws.com/123456789012/test.fifo
[DEMO] 
#Total messages sent by producer
PRODUCER_MESSAGES_TO_SENT = 100
#Delay in seconds before to start the producer
PRODUCER_START_DELAY_IN_SECONDS=0.5
#If the consumer checks the queue is empty it exits after this number of attempts.
CONSUMER_EXIT_IF_QUEUE_IS_EMPTY_AFTER_ATTEMPTS = 10     
#Delay in seconds before to start teh consumer
CONSUMER_START_DELAY_IN_SECONDS=2

Output of the demo

Following some screenshots of the demo in action:

Points of interest

Increase the consumers
Increase the producers
Change the queue type
Increase/decrease the visibility timeout
Read more than one message at a time
...

Conclusions

In this post we've seen how it's easy to implement a consumer/producer problem making use of a FIFO queue hosted in AWS SQS. The use of the official SDK Boto3 it's very intuitive as much as the AWS cli that in one only command allowed us to create a new queue.

You can find the sources on GitHub.

Numeric Overflow management in C#, Java, JavaScript and Python

Aniello Musella — Tue, 25 Oct 2022 10:34:17 +0000

Do you now what happens when you increment by one the maximum value of a number?

The most common and logic answer: I get a Numeric Overflow Exception!

But the real answer is not always or probably never!

If you don't know how your compiler manage this kind of situation you could design bad algorithms with wrong solutions.

Let's prove that!

C#

In C# when you increment a max value you don't get a numeric overflow exception (this is a default behavior). Take a look to the following snippet, where the max integer value (2147483647) is incremented by one.

int count=int.MaxValue;
Console.WriteLine($"count is {++count}");

in this case the output will be -2147483648 that is clearly an overflow because we get a negative number after an increment of positive number.This happens because the increment rises the most significant bit of the number, the sign bit (+/-).

To trigger an overflow/underflow exception we need to put the operation in a checked block like this:

int count = int.MaxValue;
checked
{
    Console.WriteLine($"count is {++count}");
}

Here we go, the exception is served!
'System.OverflowException' in overflow-dotnet.dll: 'Arithmetic operation resulted in an overflow.'

In C# compiler you can enable the option CheckForOverflowUnderflow where the default context is a checked context and overflow checking is enabled (and complementary you can use the keyword unchecked to uncheck the context).

Java

In Java the behavior is very similar to what happens in C#, take a look to the following snippet!

Integer count=Integer.MAX_VALUE; // 2147483647
System.out.println(String.format("count is %d",++count));

in this case the output will be -2147483648 that is clearly an overflow like in previous example.

From Java 8 Math class provides a set of operations (decrementExact, addExact,multiplyExact, etc. ) for "checked" arithmetic operations against numeric overflow/underflow.

To trigger an overflow exception we need to use Math.incrementExact that returns the argument incremented by one, throwing an exception if the result overflows an int.

Integer count=Integer.MAX_VALUE; // 2147483647
Math.incrementExact(count);

here we go, the exception is served:

Exception in thread "main" java.lang.ArithmeticException: integer overflow at java.base/java.lang.Math.incrementExact(Math.java:964) at Main.main(Main.java:12)

JavaScript

In JavaScript too we don't have an overflow exception if we increment a maximum value of a number and, at moment of writing, there is no way to detect a numeric overflow exception unless you write a custom function to reach this goal.

Let's prove that.

let counter=Number.MAX_VALUE;
console.log(++counter);

In the previous snippet we are incrementing by one the maximum value of Number represented by the constant Number.MAX_VALUE. The output of this snippet is 1.7976931348623157e+308 the original value of the variable counter (so the increment has non effect).

But how detect an numeric overflow/underflow JavaScript?

In this article from algotech.solutions there are good arithmetic considerations that bring to implement custom functions to detect numeric overflow/underflow.

Python

In Python 3 integers haven't fixed size (this article explain how they are structured) and the only limit is the available memory. For this reason the following code could never stop until the memory is available.

from time import sleep

count=int(0)
step = int(10**10000)
while(True):
    count+=step
    sleep(0.2)

If we want have more control about the dimension of the integers we can use NumPy

import sys
import numpy as np

count=np.int64(sys.maxsize)
count+=1

Here we go the exception is served:
RuntimeWarning: overflow encountered in long_scalars count+=1 -9223372036854775808

Conclusions

Detecting a numeric overflow/underflow is expensive and for this reason many programming languages have as default behavior to ignore it and leave to the programmer to decide if check or uncheck an arithmetic operation.

What's really matters is the knowledge of the problem and how the implementation of the programming language approaches to this kind of situations. Ignoring or overlooking the problem could lead to an unexpected results and security flaws (in some scenarios).

This article has been written because many years ago I wrote an algorithm for a challenge on Hackerrank and a silent overflow made fail it (long story short I wasted a lot of time to figure out the nature of the problem).

Suggestions and corrections are welcome.

See you

Docker Secrets in Swarm: Redis use case

Aniello Musella — Wed, 19 Oct 2022 11:02:06 +0000

In this post I want show how to use Docker Secrets providing a real scenario example using Redis.

What do you need before reading this article?

Basic knowledge of Docker technology
Basic knowledge of Docker Swarm orchestrator
Basic knowledge of Redis
Docker installed
Access to a Bash shell

What are Docker secrets?

I just quote the definition provided from documentation:

"In terms of Docker Swarm services, a secret is a blob of data, such as a password, SSH private key, SSL certificate, or another piece of data that should not be transmitted over a network or stored unencrypted in a Dockerfile or in your application’s source code. ..."

How to create them?

To create a secret you have to run the following command:

am@animusna:~$echo "myPlainSecret" | docker secret create mysecret
acvs814nso28yn105wqc32o8e

The output of this command is the id (acvs814nso28yn105wqc32o8e) of the secret.

To check if your secret has been created run the following command:

am@animusna:~$docker secret ls
ID                          NAME       DRIVER    CREATED         UPDATED
acvs814nso28yn105wqc32o8e   mysecret             5 seconds ago   5 seconds ago

How to use them?

To use a secret you have to map it during the creation of a service (in this case redis like in the documentation:

am@animusna:~$docker service create --name redis --secret mysecret redis:alpine

When you map secret in a service, Docker creates a file named like the secret name (in this case mysecret) in the folder /run/secrets and to prove that we can run the following command:

am@animusna:~$docker container exec $(docker ps --filter name=redis -q) cat /run/secrets/mysecret
myPlainSecret

where the output will be the plain text of the secret that we have created (myPlainSecret).

Real scenario example: Redis authentication with Docker secret

In case we want protect our Redis server we could use the secret created previously as password to connect to our server.

Let's prepare our docker compose file :

version: "3.1"
secrets:
  mysecret:
    external: true  #external=true means we've created the secret before. 

services:
  redis:
    image: redis 
    secrets:
      - mysecret   #We are declaring we are using this secret in this service. 
    command: bash -c "/startup/up-redis.sh" #The command we use to start our Redis server (we execute a shell script).
    ports:
      - "6379:6379"
    volumes:
      - "/home/am/up-redis.sh:/startup/up-redis.sh" # Mapping in the container our script.
    deploy:
      replicas: 1
      restart_policy:
        condition: on-failure

Let's prepare the start up script for our server up-redis.sh in a specific path (in my case /home/am/):

#!/bin/sh
#Script to startup Redis Server.

#Reading secret in a temporary variable.
REDIS_PASSWORD=`cat /run/secrets/mysecret`

#Start Redis server
redis-server --appendonly yes --requirepass "$REDIS_PASSWORD"

In this script we set the password for our server using the option --requirepass recovering the secret from /run/secrets/mysecret and putting it inside the variable REDIS_PASSWORD.

Now that is everything ready we can start service:

am@animusna:~$docker stack deploy --compose-file=docker-compose.yaml myapp_stack
Creating network myapp_stack_default
Creating service myapp_stack_redis

To check if we've set the secret to access to our Redis service let's run the following command:

am@animusna:~$echo -e "AUTH myPlainSecret\nPING" | redis-cli
OK
PONG

where we run in the redis-cli the command AUTH for authenticate with our plain secret myPlainSecret and then the command PING to check if we are authenticated (in our case yes because we get a PONG).

Conclusion

In this article I've shown a possible and easy use of Docker secrets to protect sensitive data like it could be a connection password. The use of secrets are very useful in deployment scenario as much as in development scenario where developers should use their own secrets to configure their development environment. According my experience Docker secrets are useful and easy to manage and it's worth to use them if you choose Docker Swarm as orchestrator for your (micro) services.

Dockysh: a shell written in Python over Docker shell

Aniello Musella — Sat, 02 Oct 2021 08:21:55 +0000

In this post I share a new small project to realize a simple and friendly shell that enrich the classical docker shell reducing the necessary verbosity that characterizes this shell. It can happening that during the development activities you need to interact with containers, images, services performing actions like start,remove,stop and so on. Using Docker shell makes you to repeat some keywords that you would gladly avoid to save time.

Following I publish the readme of the project.

What's Dockysh

A "wrapper" shell written in Python over Docker Shell.
Dockysh try to speedup the typing of commands for Docker shell reducing the verbosity of command and in some cases providing a little of interactivity.

Disclaimer

Dockysh does not replace Docker Shell but it's based on it. Dockysh use the Python library cmd and little of Bash scripting to make easier and more user friendly some commands.

Why this shell

For people that make an intense and interactive use of Docker shell Dockysh permit you to avoid to repeat some keywords like docker container, docker images and so on. For instance instead of type docker container ls you can type

you@Dokysh$ lsc

and if you want filter the results you can type the command

you@Dokysh$ lsc your_string_filter

What you need to run this shell

To run this shell you need:

Bash shell
Docker installed
Python installed

For who want use this shell in Windows you can activate the Windows Subsystem for Linux

How this shell works

The shell is provided with a set of commands. Each command is just an alias to a specific Docker Shell command. Each alias command is translated and sent to Docker Shell. In case of command not mapped the shell try to run the command in Docker shell, so if you type the wrong command command_not_mapped with_args the shell try to run docker command_not_mapped with_args. As logical consequence you can run each Docker shell command without the prefix docker.

How to run the shell

Just run the following command: python dockysh.py and then type ? or help for help:you@Dokysh$ help

Examples

Following some examples of Dockysh commands.

Finding an image: you@Dokysh$ lsi alpine where alpine is a filter and return all Alpine images like shown following

you@Dokysh$ lsi alpine
REPOSITORY                                  TAG                              IMAGE ID       CREATED         SIZE
alpine                                      latest                           d4ff818577bc   6 weeks ago     6.4MB

Remove one or more images: you@Dokysh$ rmi alpine where alpine is a filter.
In this case each image corresponding the filter provided will be removed after confirmation by the user:

you@Dokysh$ rmi alpine

Found 1 images in base your filter 'alpine'. Please confirm the deletion of the images found.

Do you want eliminate the image with id "d4ff818577bc" with tag "latest" of size "6.4MB" from repository "alpine"" ?
([Y]=> yes/ [N]=>no / [E]=> exit from this operation):Y

Finding container stopped: you@Dokysh$ lsc down where down is a filter on container status

Finding container running: you@Dokysh$ lsc up where up is a filter on container status

Finding container running: you@Dokysh$ lsc b33526d46134 where b33526d46134 is a filter on image of the container.

Finding containers by image id: you@Dokysh$ lsc b33526d46134 where b33526d46134 is a filter on image of the container.

Removing a container: you@Dokysh$ rmc 192294a228b7 where 192294a228b7 is a filter representing the id of the container.

In this case each contianer corresponding the filter provided (one because we've used the id of container) will be removed after confirmation by the user:

Found 1 containers in base your filter '192294a228b7'. Please confirm the deletion of the containers found.

Do you want eliminate the container with id "192294a228b7" based on image with ID "b33526d46134" ?
([Y]=> yes/ [N]=>no / [E]=> exit from this operation): Y

Next steps

Improving the shell.

See you