DEV Community: Sven Varkel

Async/await MongoDB in Python

Sven Varkel — Wed, 13 Nov 2019 14:32:21 +0000

Intro

By now almost everyone knows or at least has heard about "async/await" paradigm on JavaScript/Node.JS. But it seems that async processing is a lot less common in Python. Python feels like "old school" language when compared to JavaScript, it really feels old sometimes. But it's very clear, concise and developers can be really productive in it. It's also very similar to JS with its somewhat loose-but-not-so-loose typing, variable assignment etc.
One big difference is syntax - in JS whitespaces in the code (almost) don't matter while in Python these are important.

However, I try to keep this post short and will focus on a specific task:

- how to use MongoDB asynchronously in Python

Async/await - why bother at all?

First and foremost reason for me is concurrency. With async processing developer can run things in parallel without too much hassle and tinkering with threads or processes. It has its limits and risks but for simple cases it does the job. Async/await will come extra handy in those cases when there's a lot of I/O operations involved. Be it disk reads/writes or accessing the database. Here we are talking about accessing the database, and namely MongoDB.

MongoDB and Python

MongoDB itself feels very "javascriptish" so to say. If you're a seasoned Node.js developer like myself it feels very comfortable and cozy. MongoDB shell has the same language that your code runs in - JavaScript. You can copy-paste code and objects between your IDE/project and MongoDB shell or MongoDB IDE. And so on and on.
For Python-only developers it may be a bit hard to switch between languages-contexts but it's doable. My brain still has 2 halves but these are not called "left half" and "right half" any more but "JS half" and "Python half" ;) Half a brain still does the job, haaa:)

There's an old good Python API for MongoDB - it's called PyMongo. It gets the job done and even more but where's the fun in all that old school serial, non-async processing?

Motor to the rescue

There's a really good async driver API for MongoDB: Motor

Here are some examples.

Connecting to database

from motor.motor_asyncio import AsyncIOMotorClient

uri = "mongodb://dev:dev@localhost:27017/mydatabase?authSource=admin"
client = AsyncIOMotorClient(uri)

The same with some connection args

from motor.motor_asyncio import AsyncIOMotorClient

uri = "mongodb://dev:dev@localhost:27017/?authSource=admin"
connection_args = {
    "zlibCompressionLevel": 7,
    "compressors": "zlib"
}
client = AsyncIOMotorClient(uri, **connection_args)

Getting database and collection instance from client:

db = client.get_database("mydatabase")
collection = db.get_collection("mycollection")

Most common database operations

Find

async def find():
    """
    This method finds items from MongoDB collection and
    processes these by using another asynchronous method
    :return: 
    """
    collection = db.get_collection("mycollection")

    filter_ = {
        "someField": "someValue"
    }
    projection_ = {
        "_id": False  # don't return the _id
    }
    cursor = collection.find(filter=filter_, projection=projection_)

    # it gets interesting here now. Iterate over the cursor asynchronously
    async for item in cursor:
        await do_something_in_an_async_worker_method(item)


async def find_cursor_to_list():
    """
    This method finds items from MongoDB collection and
    asynchronously converts cursor to a list with items 
    :return: 
    """
    collection = db.get_collection("mycollection")

    filter_ = {
        "someField": "someValue"
    }
    projection_ = {
        "_id": False  # don't return the _id
    }
    cursor = collection.find(filter=filter_, projection=projection_)
    # Convert the cursor to a list of items right away.
    # NB! Dangerous with large result sets
    items = await cursor.to_list(length=500)
    return items

Update / upsert

from datetime import datetime


async def update():
    """
    This method updates data in MongoDB asynchronously
    :return: 
    """
    collection = db.get_collection("mycollection")

    filter_ = {
        "someField": "someValue"
    }
    data = {
        "someNewField": "aNewValue"
    }

    update_ = {
        "$set": data,
        "$currentDate": {
            "updatedAt": True  # set field updatedAt to current date automagically. Good practice ;)
        },
        "$setOnInsert": {
            "createdAt": datetime.utcnow()
            # set field createdAt to current date automagically ONLY IF it's a new record
        }

    }
    # if upsert=True and record is not found then it is created
    await collection.update_one(filter_, update_, upsert=True)

Aggregation

It's also possible to run aggregation pipeline with Motor.

async def aggregate():
    """
    This method run async aggregation
    :return: 
    """
    collection = db.get_collection("mycollection")

    pipeline = [
        {
            "$match": {
                "foo": "bar"
            }
        },
        {
            "$group": {
                "_id": None,
                "total": {"$sum": 1}
            }
        },
        {
            "$sort": {"foo": -1}
        }
    ]
    # mind the use of **allowDisk** argument. It's necessary for all
    # bigger result sets that are sorted
    cursor = collection.aggregate(pipeline=pipeline, allowDiskUse=True)
    results = await cursor.to_list(length=1000)
    return results

Bulk writes

Motor can be used for bulk writes that would improve performance
when storing may items from big lists of results of some kind of processing.

Here's an example:

from pymongo import UpdateOne
from datetime import datetime


async def bulk_save(very_big_list: list = list()):
    """

    :param very_big_list: List with a lot of items. Or an item generator ... ? 
    :return: 
    """
    collection = db.get_collection("mycollection")

    # this defines batch size that will be added to database at once
    BATCH_SIZE = 500

    updates = list()

    for item in very_big_list:
        filter_ = {
            "foo": item.get("foo")
        }
        update_ = {
            "$set": item,
            "$currentDate": {
                "updatedAt": True
            },
            "$setOnInsert": {
                "createdAt": datetime.utcnow()
            }
        }
        u = UpdateOne(filter=filter_, update=update_, upsert=True)
        updates.append(u)

        # if list of updates is filled up to BATCH_SIZE push data to database
        if len(updates) >= BATCH_SIZE:
            await collection.bulk_write(updates, ordered=False)
            # re-initialize list of updates
            updates = list()

    # add all remaining items to database
    if len(updates) > 0:
        await collection.bulk_write(updates, ordered=False)

Summary

Since Python is by default "synchronous" and not event-loop-based language it may be a bit hard to get accustomed to async/await pattern in Python.
In this article I gave an overview how to access MongoDB asynchronously in Python by giving examples on connecting the database with Motor client library, on querying and updating data, running aggregations and bulk updates.

Thanks for reading!

Dockerized SailsJS/ReactJS/MongoDB/Redis/RabbitMQ/Nginx denvironment

Sven Varkel — Fri, 11 Oct 2019 21:04:23 +0000

This post describes steps to set up expendable full stack denvironment. What's a denvironment, you may ask? It's development environment. That is just tooooo long to say and write:)

Take time and prepare your dev machine if you want to play along right away.

Description of the project

This project with made-up name "World's largest bass players database" consists of:

ReactJS frontend
SailsJS JSON API
MongoDB for database
RabbitMQ for queue and async processing
Redis for cache
Nginx for reverse proxy that fronts the API.

Let's call it "players", for short.

Let this project have it's main git repository be at https://github.com/svenvarkel/players

(it's time to create yours, now).

Pre-requisites

Create 2 names in your /etc/hosts file.

# /etc/hosts

127.0.0.1 api.players.local #for the API
127.0.0.1 app.players.local #for the web APP

Install Docker Desktop

Get it from here and follow the instructions.

Directory layout

The directory layout reflects the stack. On top level there are all familiar names that help the developer to navigate to a component quickly and not waste time on searching for things in obscurely named subfolders or elsewhere. Also - each component is a real component, self-containing and complete. All output or config files or anything that a component would need are placed into the component's directory.

The folder of your development projects is the /.

So here is the layout:

/
/api
    /sails bits and pieces
    /.dockerignore
    /Dockerfile
/mongodb
/nginx
    /Dockerfile
    /conf.d/
        /api.conf
        /app.conf
/rabbitmq
/redis
/web
    /react bits and pieces
    /.dockerignore
    /Dockerfile
/docker-compose.yml

It is all set up as an umbrella git repository with api and web as git submodules. Nginx, MongoDB, Redis and RabbitMQ don't need to have their own repositories.

From now on you have choice either to clone my demo repository or create your own.

If you decide to use my example repository, then run commands:

git clone git@github.com:svenvarkel/players.git
cd players
git submodule init
git submodule update

Steps

First step - create docker-compose.yml

In docker-compose.yml you define your stack in full.

version: "3.7"
services:
  rabbitmq:
    image: rabbitmq:3-management
    environment:
      RABBITMQ_DEFAULT_VHOST: "/players"
      RABBITMQ_DEFAULT_USER: "dev"
      RABBITMQ_DEFAULT_PASS: "dev"
    volumes:
      - type: volume
        source: rabbitmq
        target: /var/lib/rabbitmq/mnesia
    ports:
      - "5672:5672"
      - "15672:15672"
    networks:
      - local
  redis:
    image: redis:5.0.5
    volumes:
      - type: volume
        source: redis
        target: /data
    ports:
      - "6379:6379"
    command: redis-server --appendonly yes
    networks:
      - local
  mongodb:
    image: mongo:4.2
    ports:
      - "27017:27017"
    environment:
      MONGO_INITDB_DATABASE: "admin"
      MONGO_INITDB_ROOT_USERNAME: "root"
      MONGO_INITDB_ROOT_PASSWORD: "root"
    volumes:
      - type: bind
        source: ./mongodb/docker-entrypoint-initdb.d
        target: /docker-entrypoint-initdb.d
      - type: volume
        source: mongodb
        target: /data
    networks:
      - local
  api:
    build: ./api
    image: players-api:latest
    ports:
      - 1337:1337
      - 9337:9337
    environment:
      PORT: 1337
      DEBUG_PORT: 9337
      WAIT_HOSTS: rabbitmq:5672,mongodb:27017,redis:6379
      NODE_ENV: development
      MONGODB_URL: mongodb://dev:dev@mongodb:27017/players?authSource=admin
    volumes:
      - type: bind
        source: ./api/api
        target: /var/app/current/api
      - type: bind
        source: ./api/config
        target: /var/app/current/config
    networks:
      - local
    depends_on:
      - "rabbitmq"
      - "mongodb"
      - "redis"
  web:
    build: ./web
    image: players-web:latest
    ports:
      - 3000:3000
    environment:
      REACT_APP_API_URL: http://api.players.local
    volumes:
      - type: bind
        source: ./web/src
        target: /var/app/current/src
      - type: bind
        source: ./web/public
        target: /var/app/current/public
    networks:
      - local
    depends_on:
      - "api"
  nginx:
    build: nginx
    image: nginx-wait:latest
    restart: on-failure
    environment:
      WAIT_HOSTS: api:1337,web:3000
    volumes:
      - type: bind
        source: ./nginx/conf.d
        target: /etc/nginx/conf.d
      - type: bind
        source: ./nginx/log
        target: /var/log/nginx
    ports:
      - 80:80
    networks:
      - local
    depends_on:
      - "api"
      - "web"
networks:
  local:
    driver: overlay

volumes:
  rabbitmq:
  redis:
  mongodb:

A few comments about features and tricks used here.

My favorite docker trick that I learnt just a few days ago is the use of wait. You will see it in api and nginx Dockerfiles. It's a special app that let's the docker container wait for dependencies until a service actually comes available at a port. The Docker's own "depends_on" is good but it just waits until a dependence container becomes available, not when the actual service is started inside a container. For example - rabbitmq is quite slow to start and it may cause the API behave erratically if it starts up before rabbitmq or mongodb have been fully started.

The second trick you'll see in docker-compose.yml is the use of bind mounts. The code from the dev machine is mounted as a folder inside docker container. It's good for rapid development. Whenever the source code is changed in the editor on developer machine the SailsJS application (or actually - nodemon) in container can detect the changes and restart the application. More details about setting up SailsJS app will follow in future posts, I hope.

Second step - create API and add it as git submodule

sails new api --fast
cd api
git init
git remote add origin <your api repo origin>
git add .
git push -u origin master

Then create Dockerfile for API project:

FROM node:10

ADD https://github.com/ufoscout/docker-compose-wait/releases/download/2.6.0/wait /wait
RUN chmod +x /wait
RUN mkdir -p /var/app/current

# Copy application sources
COPY . /var/app/current

WORKDIR /var/app/current

RUN npm i

RUN chown -R node:node /var/app/current
USER node

# Set the workdir /var/app/current

EXPOSE 1337

# Start the application
CMD /wait && npm run start

Then move up and add it as your main project's submodule

cd ..
git submodule add <your api repo origin> api

Third step - create web app and add it as git submodule

This step is almost a copy of step 2, but it's necessary.

npx create-react-app my-app
cd web
git init
git remote add origin <your web repo origin>
git add .
git push -u origin master

Then create Dockerfile for WEB project:

FROM node:10

ADD https://github.com/ufoscout/docker-compose-wait/releases/download/2.6.0/wait /wait
RUN chmod +x /wait
RUN mkdir -p /var/app/current

# Copy application sources
COPY . /var/app/current

WORKDIR /var/app/current

RUN npm i

RUN chown -R node:node /var/app/current
USER node

# Set the workdir /var/app/current

EXPOSE 3000

# Start the application
CMD /wait && npm run start

As you can see the Dockerfiles for api and web are almost identical. Only the port number is different.

Then move up and add it as your main project's submodule

cd ..
git submodule add <your web repo origin> web

For both projects, api and web, it's also advisable to create .dockerignore file with just two lines:

node_modules
package-lock.json

We want the npm modules inside the container being built fresh every time we rebuild the docker container.

It's time for our first smoke test!

Run docker-compose:

docker-compose up

After Docker grinding a while you should have a working stack! It doesn't do much yet but it's there.

Check with docker-compose:

$ docker-compose ps
   Name                     Command               State                                                                   Ports
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
players_api_1        docker-entrypoint.sh /bin/ ...   Up      0.0.0.0:1337->1337/tcp, 0.0.0.0:9337->9337/tcp
players_mongodb_1    docker-entrypoint.sh mongod      Up      0.0.0.0:27017->27017/tcp
players_nginx_1      /bin/sh -c /wait && exec n ...   Up      0.0.0.0:80->80/tcp
players_rabbitmq_1   docker-entrypoint.sh rabbi ...   Up      0.0.0.0:15671->15671/tcp, 0.0.0.0:15672->15672/tcp, 0.0.0.0:25672->25672/tcp, 4369/tcp, 0.0.0.0:5671->5671/tcp, 0.0.0.0:5672->5672/tcp
players_redis_1      docker-entrypoint.sh redis ...   Up      0.0.0.0:6379->6379/tcp
players_web_1        docker-entrypoint.sh /bin/ ...   Up      0.0.0.0:3000->3000/tcp

As you can see you have:

API running on port 1337 (9337 also exposed for debugging)
MongoDB running on port 27017
RabbitMQ running on many ports, where AMQP port 5672 is of our interest. 15672 is for management - check it out in your browser (use dev as username and password)!
Redis running on port 6379
Web app running on port 3000
Nginx running on port 80.

Nginx proxies both API and web app. So now it's time to give it a look in your browser.

Open http://api.players.local

There it is!

Open http://app.players.local

And there is the ReactJS app.

With this post we won't go into depths of the applications but we focus rather on stack and integration.

So how can services access each other in this Docker setup, you may ask.

Right - it's very straightforward - the services can access each other on a common shared network by calling each other with exactly the same names that are defined in docker-compose.yml.

Redis is at "redis:6379", MongoDB is at "mongodb:27017" etc.

See docker-compose.yml for a tip on how to connect your SailsJS API to MongoDB.

A note about storage

You may have a question like "where is mongodb data stored?". There are 3 volumes defined in docker-compose.yml:

mongodb
redis
rabbitmq

These are special docker volumes that hold the data for each component. It's convenient way of storing data outside of application container but still under control and management of Docker.

A word of warning

There's something I learnt the hard way (not that hard, though) during my endeavour towards full stack dev env. I used command

docker-compose up

lightly and it created temptation to use command

docker-compose down

as lightly because "what goes up must come down", right? Not so fast! Beware that if you run docker-compose down it will destroy your stack including data volumes. So - be careful and better read docker-compose manuals first. Use docker-compose start, stop and restart.

Wrapping it up

More details could follow in similar posts in the future if there's interest for such guides. Shall I continue to add more examples on how to integrate RabbitMQ and Redis within such stack, perhaps? Let me know.

Conclusion

In this post there is a step by step guide on how to set up full stack SailsJS/ReactJS application denvironment (development environment) by using Docker. The denvironment consists of multiple components that are integrated with the API - database, cache and queue. User-facing applications are fronted by the Nginx reverse proxy.