DEV Community: Apcelent

Deploying Flask on AWS Lambda

Apcelent — Wed, 24 Oct 2018 12:54:30 +0000

In this article, we will guide you to deploy a flask application onto AWS Lambda using Zappa. First, we will discuss what these actually are and then proceed with our main goal.

FLASK

Flask is a python's framework for building web applications. It follows WSGI toolkit and its lightweight when compared with other frameworks like python's Django and PHP's Drupal. It helps in setting up the application easily without any hassle. Generally, flask finds much of its use case in small projects but complex and database -driven are also developed using it.

AWS LAMBDA

Web applications activity is uncertain, sometimes they serve huge amounts of workloads but at times they may sit idle without any considerable request count. Hosting applications on an EC2 Instance or in containers like ECS make us pay for the idle time too. To address such a problem we need to look at load balancing, DNS lookups and auto-scaling. Managing all these is difficult.

AWS lambda is the perfect solution. It scales automatically when needed, depending upon the requests the application gets. Thus, making us pay only when we consume it. If our code is not running then charges are not incurred.

Settings AWS Credentials

Create a folder called aws at the root level,

mkdir .aws

Now, create a file called credentials and store the aws_access_key_id and aws_secret_access_key. The other specifics like default_region can alos be stored.

Firstly, to get the access credentials, do the following:

Goto IAM role in AWS dashboard.
Nagivate to Users.
Click on the Security credentials tab, and go down to Access Key, note down the access_key_id. secret_access_key is only visible when you are creating new user, so you need to note down both the access_key_id and secre_access_key at the time of user creation only.

To generate a new access_key, click on the create access key button,

aws_credentials

###~/.aws/credentials
[default]
aws_access_key_id=[...]
aws_secret_access_key=[...]

You can also use AWS cli to set the aws_access_key_id and aws_secret_access_key. To install aws_cli,

pip install awscli

Zappa helps in deploying and building any WSGI compatible applications that are developed in python. Flask, Django, Bottle and many other work with Zappa.

While using Zappa we dont actually need to change any of our code. It simply helps in visualizing the serverless hosting of your applications. Horizontal scaling is handled by Zappa and the applicatyion is never tightly coupled with it, so if we want to leave it, its very simple.

NOW, Lets get Started!

Until now, we got to know what all are required to deploy a flask application on to AWS Lambda. To get started we need to install all the required libraries and packages. Here we need, - Flask - Zappa

Let's create a project, flask-app.

mkdir flask-app

Creating Virtual Environment

Now change the working directory to flask-app and create a virtual environment first, this helps in setting up independent and separate python environments thus helping to work on different versions of the same package.

virtualenv .env
source .env/bin/activate

Create another file called requirements.txt and fill in the required packages. Here we need flask and Zappa, therefore add these two lines into the requirements file,

#requirements.txt
flask
zappa

Once you have listed out the requirements. Install them using PIP, its package manager used extensively to install python libraries and frameworks.

pip install -r requirements.txt

Once all the required packages are installed, we will start with a very basic flask app which simply says "HELLO WORLD!".

from flask import Flask, jsonify

app = Flask(__name__)

@app.route('/')
def hello_world():
    return jsonify({"message": "Hello World!"})

if __name__ == '__main__':
 app.run()

The application is completed, now we need to focus on how to deploy this applcation onto Amazon lambda.

As mentioned previously, Zappa helps in easily deploying WSGI applications onto Amazon Lambda. Firstly, you need to create an AWS account. if there are multiple users are using the same AWS account then they should use unique names while using starting Zappa, because the names of the resources may get conflicted.

# zappa init
FULL OUTPUT TRUNCATED... 
What do you want to call this environment (default 'dev'):
What do you want call your bucket? (default 'zappa-ip7znwymz'):
Where is your app's function? (default 'app.init.app'):
Would you like to deploy this application globally? (default 'n') [y/n/(p)primary]:
Does this look okay? (default 'y') [y/n]:

After running zappa init command it will create a file called zappa_settings.json will helps in deploying your application.

Almost Done!

The important thing while deploying is to choose an environment name like dev, prod, alpha etc. This name should be unique. The flask application is deployed to AWS Lambda using the zappa_settings.json file.

zappa deploy dev

Example:

(.env)ubuntu@172-31-19-54:~/flask-app$ zappa deploy dev
Calling deploy for stage dev..
--TRUNCATED OUTPUT...
- 75%|█████████████████████████████████ | 3/4 [00:12<00:06, 6.95s/res]
- Deploying API Gateway..
- Deployment complete!: https://izg1p4sbdavs67.execute-api.us-west-2.amazonaws.com/dev

Acessing the application!

Check the deployment of your application by accessing it through API Gateway. That is, note down the url from the zappa deploy command and open it in any web browser.

https://izg1p4sbdavs67.execute-api.us-west-2.amazonaws.com/dev

If you update the application just run the update command in addition with the environment i.e

zappa update dev

The source code can be found on github

Hope the article was of help!

The article originally appeared on Apcelent Tech Blog.

How to create REST API using Django REST Framework

Apcelent — Tue, 16 Oct 2018 14:54:54 +0000

Django rest framework is a python toolkit to create REST APIs for django applications.

Installation

Create a virtual environment for our project and then install Django and Dajngo REST Framework

sudo pip install virtualenv
virtualenv venv
source venv/bin/activate

pip install Django==1.9
pip install djangorestframework==3.6.3

After django installation, Create a project samplenote. Inside that project we create an app note:

django-admin.py startproject samplenote .
cd samplenote
django-admin.py startapp note

Update settings.py file in samplenote folder to include newly added apps - note and rest_framework.

INSTALLED_APPS = [
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',

    #app
    'note',

    #rest_framework
    'rest_framework',
]

Model

Edit models.py file in note app. Add a Note model class with title, description, created date, created by and priority fields.

from django.db import models
from django.utils import timezone
from django.core.validators import MaxValueValidator, MinValueValidator
# Create your models here.

class Note(models.Model):

    title = models.CharField(max_length=100)
    description = models.CharField(max_length=400)
    created_at = models.DateTimeField(default=timezone.now)
    created_by = models.CharField(max_length=50, blank=True, null=True)
    priority = models.IntegerField(validators=[MinValueValidator(1),
                                       MaxValueValidator(5)])

Now Create and apply schema migration by executing following:

python manage.py makemigrations
python manage.py migrate

Creating serializer

Serializers are used to convert data from different formats like JSON, XMl to complex data types like querysets and model instances and vice versa.

Create a Note serializer based on Node model created earlier. This serializer is defined in note_api.py file of note app.

from rest_framework import serializers
from .models import Note

class NoteSerialiser(serializers.HyperlinkedModelSerializer):

    class Meta:
        model = Note
        fields = ('id', 'title', 'description', 'created_at', 'created_by', 'priority')

Viewset

Viewsets are classes which provide functionality of set of views. Create a NoteViewSet class based on ModelViewSet in note_api.py file of note app.

from rest_framework import viewsets

class NoteViewSet(viewsets.ModelViewSet):

    queryset = Note.objects.all()
    serializer_class = NoteSerialiser

Run the server

After setting up serializers and viewsets, run the server:

python manage.py runserver

To get a list of all notes, send a GET request to following URL

http://127.0.0.1:9000/api/notes/

To add a new note, send a POST request to the same URL with data in following format in body part of the request.

  {
    "created_at": "2017-08-17 00:00", 
    "created_by": "v", 
    "description": "sample notes from api", 
    "priority": 1, 
    "title": "sample note from api"
}

The source code can be found on github

Hope the article was of help!

The article originally appeared on Apcelent Tech Blog.

How to create REST API using Flask

Apcelent — Mon, 01 Oct 2018 14:22:43 +0000

Flask is a microframework for python which can be used for developing web applications. In this article, we will see how to create RESTful APIs in Flask using Flask-RESTful.

Installation

Before setting up the project, we create a virutal environment to keep our installation of python packages isolated from other projects.

To set up a virtual environment, we install virtualenv

sudo pip install virtualenv

Then we create a virtual environment named venv and activate it

virtualenv venv
source venv/bin/activate

After that we install Flask and Flast-RESTful in these virtual environments

pip install Flask
pip install flask-restful

Models

We create a note model with note title, description, created_at date, created_by and priority fields with id as primary key. We will use SQLite as the database for this application.

from sqlalchemy import Column
from sqlalchemy import Integer
from sqlalchemy import String
from sqlalchemy.ext.declarative import declarative_base
from datetime import datetime

DB_URI = 'sqlite:///./main.db'
Base = declarative_base()

class Note(Base):
    __tablename__ = 'notes'

    id = Column(Integer, primary_key=True)
    title = Column(String(80))
    description = Column(String(80))
    create_at = Column(String(80))
    create_by = Column(String(80))
    priority = Column(Integer)

if __name__ == "__main__":
    from sqlalchemy import create_engine

    engine = create_engine(DB_URI)
    Base.metadata.drop_all(engine)
    Base.metadata.create_all(engine)

Resources

Resources are building blocks of Flask-RESTful. Resource class is used to map HTTP methods to objects. We create a resource.py file where we first define NoteResource class.

from models import Note
from db import session
from datetime import datetime
from flask.ext.restful import reqparse
from flask.ext.restful import abort
from flask.ext.restful import Resource
from flask.ext.restful import fields
from flask.ext.restful import marshal_with

note_fields = {
    'id': fields.Integer,
    'title': fields.String,
    'description': fields.String,
    'create_at': fields.String,
    'create_by': fields.String,
    'priority': fields.Integer
}

parser = reqparse.RequestParser()
parser.add_argument('title')
parser.add_argument('description')
parser.add_argument('create_at')
parser.add_argument('create_by')
parser.add_argument('priority')


class NoteResource(Resource):
    @marshal_with(note_fields)
    def get(self, id):
        note = session.query(Note).filter(Note.id == id).first()
        if not note:
            abort(404, message="Note {} doesn't exist".format(id))
        return note

    def delete(self, id):
        note = session.query(Note).filter(Note.id == id).first()
        if not note:
            abort(404, message="Note {} doesn't exist".format(id))
        session.delete(note)
        session.commit()
        return {}, 204

    @marshal_with(note_fields)
    def put(self, id):
        parsed_args = parser.parse_args()
        note = session.query(Note).filter(Note.id == id).first()
        note.title = parsed_args['title']
        note.description = parsed_args['description']
        note.create_at = parsed_args['create_at']
        note.create_by = parsed_args['create_by']
        note.priority = parsed_args['priority']
        session.add(note)
        session.commit()
        return note, 201

reqparse is Flaks-RESTful request parsing interface used for providing access to variables of flask.request objects.

marshall_with decorator takes data objects from API and applies field filtering.

We add one more resource for all Note elements

class NoteListResource(Resource):
    @marshal_with(note_fields)
    def get(self):
        notes = session.query(Note).all()
        return notes

    @marshal_with(note_fields)
    def post(self):
        parsed_args = parser.parse_args()
        note = Note(title=parsed_args['title'], description=parsed_args['description'],
                    create_at=parsed_args['create_at'], create_by=parsed_args['create_by'],
                    priority=parsed_args['priority'] )
        session.add(note)
        session.commit()
        return note, 201

We create another file api.py to add these resources to our API

from flask import Flask
from flask_restful import reqparse, abort, Api, Resource
from models import Note
from resources import *

app = Flask(__name__)
api = Api(app)

## Setup the API resource routing

api.add_resource(NoteListResource, '/notes/', endpoint='notes')
api.add_resource(NoteResource, '/notes/<string:id>', endpoint='note')


if __name__ == '__main__':
    app.run(debug=True)

Starting Flask server and send requests

To run flask web server

python api.py

API can be tested using curl or browser addons like Postman for chrome and httprequester for Firefox.
To get a list of all notes, send a GET request to following URL:

http://127.0.0.1:5000/notes/

To add a new note, send a POST request to the same url with data in following format in body part of the request.

{
    "create_at": "2017-08-17 00:00", 
    "create_by": "v", 
    "description": "sample notes from api", 
    "priority": 1, 
    "title": "sample note from api"
}

Implemenation and codebase for above example is available here. To set up the project, clone the repository, install requirements in virtual environment and start the server.

Hope the article was of help!

The article originally appeared on Apcelent Tech Blog.

How to create REST API using Falcon

Apcelent — Wed, 26 Sep 2018 09:13:44 +0000

Falcon is a bare-metal Python web API framework used for building fast app backends and microservices. A bare-metal server is single-tenant physical server completely dedicated for single customer.

As per the official Falcon website

“When it comes to building HTTP APIs, other frameworks weigh you down with tons of dependencies and unnecessary abstractions. Falcon cuts to the chase with a clean design that embraces HTTP and the REST architectural style.”

In this article, we will create a rest API for our application using falcon. This application will perform following tasks:

Create a note (POST request)
Get a note (GET request)
Edit a note (PUT request)
Delete a note (DELETE request)

We will use SQLAlchemy as our resource provider for the APIs and gunicorn to serve those APIs.

Set up Falcon

Set up and activate a virtual environment using python3. Install Falcon, SQLAlchemy and gunicorn

pip install falcon
pip install SQLAlchemy
pip install gunicorn

or clone the github repository and install the requirements

pip install -r requirements.txt

Models

Once the package installation is done, we will set up SQLAlchemy as our resource provider in models.py file:

if __name__ == "__main__":
    from sqlalchemy import create_engine

    engine = create_engine(DB_URI)
    Base.metadata.drop_all(engine)
    Base.metadata.create_all(engine)

After that, we add a Note model class for creating note objects in models.py

class Note(Base):
    __tablename__ = 'notes'
    id      = Column(Integer, primary_key=True)
    title    = Column(String(50))
    description     = Column(String(50))
    created_at     = Column(String(50))
    created_by     = Column(String(50))
    priority     = Column(Integer)

In resources.py file we define Note resources based on Note model.

class NoteCollectionResource(CollectionResource):
    model = Note
    methods = ['GET', 'POST']


class NoteResource(SingleResource):
    model = Note

After defining models and resources, we add API urls in app.py.

db_engine = create_engine('sqlite:///stuff.db')

app = falcon.API(
    middleware=[Middleware()],
)

app.add_route('/notes', NoteCollectionResource(db_engine))
app.add_route('/notes/{id}', NoteResource(db_engine))

Running the app

Create database by executing following in terminal

python models.py

Run the server

gunicorn --reload app:app

We can view list of all requests at http://127.0.0.1:8000/notes.

Using APIs for notes

Open python shell and execute following requests:

requests.post('http://localhost:8000/notes/',
                 headers={'Content-Type': 'application/json'},
                 data=json.dumps({'title': 'sample note five' , 'description': 'sample notes',
                                    'created_at': '2017-08-18 00:00:00', 'created_by': 'apcelent',
                                    'priority': 3}))

requests.put('http://localhost:8000/notes/2',
                headers={'Content-Type': 'application/json'},
                data=json.dumps({'title': 'sample note five' , 'description': 'sample notes edit',
                                    'created_at': '2017-08-18 00:00:00', 'created_by': 'apcelent',
                                    'priority': 3}))

requests.delete('http://localhost:8000/notes/2')

Performance

Benchmark results for Falcon on official website shows that Falcon performs better than some other popular python frameworks with ability to handle much larger requests in lesser time making it fast and reliable for large-scale microservices and backends.

The source code can be found on github

Hope the article was of help!

The article originally appeared on Apcelent Tech Blog.

How to create REST API using Pyramid

Apcelent — Tue, 18 Sep 2018 05:28:31 +0000

Pyramid is a lightweight python web framework following MVC architectural pattern. In this article, we will set up a web application and API using Pyramid.

Pyramid is a suitable framework for large scale MVC applications and it comes with flexible bootstrapping tools.

Cornice, a library from Mozilla, makes it easy to develop RESTful web services with pyramid.

Installation

First we create a virtual environment which helps us in separating project specific python packages installations from other projects. After activating the virtual environment, we install pyramid framework using command:

pip install pyramid

Create models

We create a models.py file with a class Note which maps to the data table storing all the values of notes.

# pyramidapp/models.py

class Note(Base):
    __tablename__ = 'Note'
    id = Column(Integer, primary_key=True)
    title = Column(Text)
    description = Column(Text)
    create_at = Column(Text)
    create_by = Column(Text)
    priority = Column(Integer)

    def __init__(self, title, description, create_at ,create_by, priority):
        self.title = title
        self.description = description
        self.create_at = create_at
        self.create_by = create_by
        self.priority = priority

    @classmethod
    def from_json(cls, data):
        return cls(**data)

    def to_json(self):
        to_serialize = ['id', 'title', 'description', 'create_at', 'create_by', 'priority']
        d = {}
        for attr_name in to_serialize:
            d[attr_name] = getattr(self, attr_name)
        return d

Views

In views.py file, we add our services for different API requests.

@resource(collection_path='/notes', path='/notes/{id}')
class NoteView(object):

    def __init__(self, request):
        self.request = request

    def collection_get(self):

        return {
            'notes': [
                {'id': note.id, 'title': note.title, 'description': note.description,
                'create_at': note.create_at, 'create_by': note.create_by, 'priority': note.priority}

                    for note in DBSession.query(Note)

                    ]
            }

    def get(self):

        try:
            return DBSession.query(Note).get(
                int(self.request.matchdict['id'])).to_json()
        except:
            return {}

    def collection_post(self):

        note = self.request.json
        DBSession.add(Note.from_json(note))

    def put(self):
        try:
            obj=DBSession.query(Note).filter(Note.id==self.request.matchdict['id'])
            obj.update(self.request.json)
            return {'notes': [
                    {'id': note.id, 'title': note.title, 'description': note.description,
                    'create_at': note.create_at, 'create_by': note.create_by, 'priority': note.priority}

                        for note in DBSession.query(Note)

                        ]
                    }
        except:
            return {'result': 'No object found'}


    def delete(self):
        obj=DBSession.query(Note).filter(Note.id==self.request.matchdict['id']).first()
        DBSession.delete(obj)

        return {'notes': [
                {'id': note.id, 'title': note.title, 'description': note.description,
                'create_at': note.create_at, 'create_by': note.create_by, 'priority': note.priority}

                    for note in DBSession.query(Note)

                    ]
                }

Running the application

Create the database schema by executing:

python initialize_db.py

Start the development server by:

python setup.py develop
pserve note.ini --reload

We can view the notes by navigating to the URL http://localhost:6543/notes in browser.

Open python shell and execute requests for API:

requests.post('http://localhost:6543/notes',
                 headers={'Content-Type': 'application/json'},
                data=json.dumps({   "title": "sample note one ",
                                    "create_at": "2017-08-23 00:00",
                                    "create_by": "apcelent",
                                    "description": "sample notes",
                                    "priority": 3,

                                }))

requests.put('http://localhost:6543/notes/1',
                 headers={'Content-Type': 'application/json'},
                data=json.dumps({   "title": "sample note edit ",
                                    "create_at": "2017-08-23 00:00",
                                    "create_by": "apcelent",
                                    "description": "sample notes edit",
                                    "priority": 4,

                                }))

requests.delete('http://localhost:6543/notes/1')

The source code can be found on github

Hope the article was of help!

The article originally appeared on Apcelent Tech Blog.

How to create REST API using aiohttp

Apcelent — Mon, 10 Sep 2018 10:31:44 +0000

aiohttp is HTTP client/server for python and AsyncIO. It supports both server websockets and client websockets. As it works asynchronously, it can handle hundreds of requests per seconds providing better performance than other frameworks.

AsyncIO is a python library for writing:

Single-threaded concurrent code using coroutines.
Multiplexing I/O access over sockets and other resources.
Running network clients and servers, and other related primitives.

This provides concurrency especially for I/O bound tasks over sockets and other resources. Concurrency ensures that user does not have wait for the I/O bound results.

In this article, we will create a rest API for our application using aiohttp. It is a simple application which has a Note table.

Set up aiohttp

Activate a virtual environment in python 3 and install aiohttp

pip install aiohttp

or clone the github repository and install the requirements

pip install -r requirements.txt

Create Models

We will configure application to use sqlite as our database in models.py

# DBSession = scoped_session(sessionmaker(extension=ZopeTransactionExtension()))
DB_URI = 'sqlite:///stuff.db'

Session = sessionmaker(autocommit=False,
                       autoflush=False,
                       bind=create_engine(DB_URI))
session = scoped_session(Session)
Base = declarative_base()

Then we create Note class for note objects in models.py

class Note(Base):

    __tablename__ = 'notes'

    id      = Column(Integer, primary_key=True)
    title    = Column(String(50))
    description     = Column(String(50))
    created_at     = Column(String(50))
    created_by     = Column(String(50))
    priority     = Column(Integer)

    def __init__(self, title, description, created_at ,created_by, priority):

        self.title = title
        self.description = description
        self.created_at = created_at
        self.created_by = created_by
        self.priority = priority

    @classmethod
    def from_json(cls, data):
        return cls(**data)

    def to_json(self):
        to_serialize = ['id', 'title', 'description', 'created_at', 'created_by', 'priority']
        d = {}
        for attr_name in to_serialize:
            d[attr_name] = getattr(self, attr_name)
        return d

Resources

We define our API endpoints in aiohttp_rest.py file.

DEFAULT_METHODS = ('GET', 'POST', 'PUT', 'DELETE')


class RestEndpoint:

    def __init__(self):
        self.methods = {}

        for method_name in DEFAULT_METHODS:
            method = getattr(self, method_name.lower(), None)
            if method:
                self.register_method(method_name, method)

    def register_method(self, method_name, method):
        self.methods[method_name.upper()] = method

    async def dispatch(self, request: Request):
        method = self.methods.get(request.method.upper())
        if not method:
            raise HTTPMethodNotAllowed('', DEFAULT_METHODS)

        wanted_args = list(inspect.signature(method).parameters.keys())
        available_args = request.match_info.copy()
        available_args.update({'request': request})

        unsatisfied_args = set(wanted_args) - set(available_args.keys())
        if unsatisfied_args:
            # Expected match info that doesn't exist
            raise HttpBadRequest('')

        return await method(**{arg_name: available_args[arg_name] for arg_name in wanted_args})


class CollectionEndpoint(RestEndpoint):
    def __init__(self, resource):
        super().__init__()
        self.resource = resource

    async def get(self) -> Response:
        data = []

        notes = session.query(Note).all()
        for instance in self.resource.collection.values():
            data.append(self.resource.render(instance))
        data = self.resource.encode(data)
        return Response ( status=200, body=self.resource.encode({
            'notes': [
                {'id': note.id, 'title': note.title, 'description': note.description,
                'created_at': note.created_at, 'created_by': note.created_by, 'priority': note.priority}

                    for note in session.query(Note)

                    ]
            }), content_type='application/json')


    async def post(self, request):
        data = await request.json()
            note=Note(title=data['title'], description=data['description'], created_at=data['created_at'], created_by=data['created_by'], priority=data['priority'])
        session.add(note)
        session.commit()

        return Response(status=201, body=self.resource.encode({
            'notes': [
                {'id': note.id, 'title': note.title, 'description': note.description,
                'created_at': note.created_at, 'created_by': note.created_by, 'priority': note.priority}

                    for note in session.query(Note)

                    ]
            }), content_type='application/json')


class InstanceEndpoint(RestEndpoint):
    def __init__(self, resource):
        super().__init__()
        self.resource = resource

    async def get(self, instance_id):
        instance = session.query(Note).filter(Note.id == instance_id).first()
        if not instance:
            return Response(status=404, body=json.dumps({'not found': 404}), content_type='application/json')
        data = self.resource.render_and_encode(instance)
        return Response(status=200, body=data, content_type='application/json')

    async def put(self, request, instance_id):

        data = await request.json()

        note = session.query(Note).filter(Note.id == instance_id).first()
        note.title = data['title']
        note.description = data['description']
        note.created_at = data['created_at']
        note.created_by = data['created_by']
        note.priority = data['priority']
        session.add(note)
        session.commit()

        return Response(status=201, body=self.resource.render_and_encode(note),
                        content_type='application/json')

    async def delete(self, instance_id):
        note = session.query(Note).filter(Note.id == instance_id).first()
        if not note:
            abort(404, message="Note {} doesn't exist".format(id))
        session.delete(note)
        session.commit()
        return Response(status=204)


class RestResource:
    def __init__(self, notes, factory, collection, properties, id_field):
        self.notes = notes
        self.factory = factory
        self.collection = collection
        self.properties = properties
        self.id_field = id_field

        self.collection_endpoint = CollectionEndpoint(self)
        self.instance_endpoint = InstanceEndpoint(self)

    def register(self, router: UrlDispatcher):
        router.add_route('*', '/{notes}'.format(notes=self.notes), self.collection_endpoint.dispatch)
        router.add_route('*', '/{notes}/{{instance_id}}'.format(notes=self.notes), self.instance_endpoint.dispatch)


    def render(self, instance):
        return OrderedDict((notes, getattr(instance, notes)) for notes in self.properties)

    @staticmethod
    def encode(data):
        return json.dumps(data, indent=4).encode('utf-8')

    def render_and_encode(self, instance):
        return self.encode(self.render(instance))

By using async keyword with all methods (GET, POST, PUT and DELETE), we ensure that those operations are performed asynchronously and the response is returned from from both collection end point and instance end points. After setting up our endpoints, we declare resources in aio-app.py file.

from aiohttp.web import Application, run_app

from aiohttp_rest import RestResource
from models import Note
from sqlalchemy import engine_from_config


notes = {}
app = Application()
person_resource = RestResource('notes', Note, notes, ('title', 'description', 'created_at', 'created_by', 'priority'), 'title')
person_resource.register(app.router)


if __name__ == '__main__':

    run_app(app)

Running the application

First create the database by:

python models.py

Run the app by executing following in terminal

python aio-app.py

Open python shell and execute some requests

requests.post('http://localhost:8080/notes',
                 data=json.dumps({ "title": "note two",
                 "created_at": "2017-08-23 00:00", "created_by": "apcelent", "description": "sample notes", "priority": 4
}))

requests.put('http://localhost:8080/notes/1',
                 data=json.dumps({ "title": "note edit",
                 "created_at": "2017-08-23 00:00", "created_by": "apcelent", "description": "sample notes edit", "priority": 4
}))


requests.delete('http://localhost:8080/notes/1')

These will create some notes in database using aiohttp REST API. These notes can be viewed at http://127.0.0.1:8080/notes

The source code can be found here.

The article originally appeared on Apcelent Tech Blog.

Migrating a realtime Checkins Discovery App from React to Preact

Apcelent — Tue, 04 Sep 2018 11:41:36 +0000

In this post we are going to create a client side web application to render FourSquare Checkins of a specified location per user's preference in realtime. The major technologies we are going to use is ReactJS, FourSquare API, Google Maps API.

And later down the tutorial we will see how to migrate a pre-written react app to a preact app.

Overview

React-Checkins is a realtime checkins discovery app built using Google Maps API with ReactJS.

Once rendered the app looks as below

Setup and Maps Integration

We will use npm for managing all our code dependencies. First let’s setup npm in the root of our project. This will generate a file called package.json in project root. If you use the --save or --save-dev flag when installing a package, it’ll save the packages as dependencies in the package.json file. To reinstall the packages, all we need to is run npm install. The packages will be installed locally specific to our project under a directory called node_modules like virtualenv.

So, let’s generate a package.json file in our project root using npm init.

npm init

Let’s install our first npm packages.

npm install --save react webpack babel babel-loader webpack-dev-server react-google-maps react-addons-update classnames

package.json contains several other dependencies.

npm should install the dependencies under the node_modules folder structure by default.

Create webpack config

Let us now create a webpack config. The goal of creating the config file is to concatenate all the js files.

mkdir -p src/js
touch webpack.config.js
touch dev/js/components/GoogleApp.js

Let’s create a simple webpack config to load .jsx files using babel and More on webpack configuration here.

webpack.config file

var path = require('path');
var webpack = require('webpack');


module.exports = {
    devServer: {
        inline: true,
        contentBase: './src',
        port: 3000
    },
    devtool: 'cheap-module-eval-source-map',
    entry: './dev/js/components/GoogleApp.js', ///entry point of our app. 
    module: {
        loaders: [
            {
                test: /\.js$/,
                loaders: ['babel'],
                exclude: /node_modules/
            },
            {
                test: /\.scss/,
                loader: 'style-loader!css-loader!sass-loader'
            }
        ]
    },
    output: {
        path: 'src',
        filename: 'js/bundle.min.js'
    },
    plugins: [
        new webpack.optimize.OccurrenceOrderPlugin()
    ]
};

At this point, our directory structure will look something like this.

root
  |-- dev
    | |--js
    |     |--components
    |         |--GoogleApp.js
    |-- node_modules
    |-- src
    | |-- js
    |   | |-- bundle.min.js   ## will create after run npm start
    |   |-- index.html
    |-- webpack.config.js
    |-- package.json
    |-- .babelrc

The index file is fairly simple and can be found here

Initializing Maps in React Component

Import {
    withGoogleMap,
    GoogleMap,
    Marker,
    InfoWindow,
 } from "react-google-maps";

If you're going to be loading the map through your index.html file you can include the below.

 <script src="https://maps.googleapis.com/maps/api/js?key=GOOGLE_API_KEY&callback=initMap&libraries=places"async defer></script>

Dynamically retrieving the Google Maps Library

const GettingStartedGoogleMap = withGoogleMap(props => (
    <GoogleMap
    ref={props.onMapLoad}
    defaultZoom={3}
    options={{ minZoom: 3, maxZoom: 15 }}
    center={{ lat: 20.5937, lng: 78.9629 }}
    >
    {props.markers.map(marker => (
        <Marker
          { ...marker }
          onRightClick={() => props.onMarkerRightClick(marker)}
          onMouseOver={() => props.onMouseOver(marker)}
          >
          {marker.showInfo && (
            <InfoWindow >
              <div className="">
              {marker.info ?
                <p><b>{marker.info[0].user.firstName} says </b>
                    {marker.info[0].text}</p>
                : null}
                <p><b>Place </b> {marker.key}</p>

                </div>
              </InfoWindow>
            )}

        </Marker>

    ))}
    </GoogleMap>
));

Getting initial state of markers

constructor(props)
        {
            super(props);
            this.state = {
                markers: [{
                    position: {
                        lat: null,
                        lng: null,
                    },
                    key: null,
                    defaultAnimation: null,
                    showInfo:false,
                    info: []
                }],
                address: '',
                results: []
            };
            this.onChange = (address) => this.setState({ address })
        }

Search the places through the foursqaure

The searchNearPlaces function takes the response from the city search query and pushes all the results inside an array object with their respective [lat, lng] co-ordinates and pin’s icon link.

getUrl() {
    const clientID = YOUR_FOURSQUARE_CLIENT_ID,
         clientSecret = YOUR_FOURSQUARE_CLIENT_SECRET,
         version = 'v=20140806';
    let location = this.state.address,
    url = 'https://api.foursquare.com/v2/venues/explore?client_id=' +
      clientID + '&client_secret=' + clientSecret + '&near=' +
        location + '&' + version;
    return url;
}

searchNearPlaces() {
    let url = this.getUrl();
    axios.get(url)
        .then(response => {
            let results = response.data.response.groups[0].items;
            this.setState({ results:results });
            // Do with results

        })
}

Let's save the first 20 out of total results as markers.

 var bounds = new google.maps.LatLngBounds();
 results.map((el, i) => {
    if (i < 20 ) {
    const nextMarkers = [
    ...this.state.markers,
    {
    position: {
        lat: el.venue.location.lat,
        lng: el.venue.location.lng  },
    defaultAnimation: 2,
    key: el.venue.name,
    showInfo: false,
    info: el.tips
    },
    ];
    this.setState({
    markers: nextMarkers,
    })

   bounds.extend(this.state.markers[i].position);
   }

})
this._mapComponent.fitBounds(bounds);   // bound the all markers of map

And now we render the component

render() {
    const inputProps = {
        value: this.state.address,
        onChange: this.onChange
        type: 'search',
        placeholder: 'Search Places...',
        autoFocus: true,
    }
    const cssClasses = {
        root: 'form-group',
        input: 'form-control',
        autocompleteContainer: 'my-autocomplete-container'
    }
    const myStyles = {
        root: { position: 'absolute' },
        input: { width: '100%' },
        autocompleteContainer: { backgroundColor: 'green' },
        autocompleteItem: { color: 'black' },
        autocompleteItemActive: { color: 'blue' }
    }

    const AutocompleteItem = ({ suggestion }) => (<div>
      <i className="fa fa-map-marker"/>{suggestion}</div>)
    return (
        <div ref="map" style={{height: '600px'}}>
            <Helmet
              title="GoogleMap"
            />
            <form onSubmit={this.handleFormSubmit.bind(this)}>
                <PlacesAutocomplete
                    inputProps={inputProps}
                    classNames={cssClasses}
                    styles={myStyles}
                    autocompleteItem={AutocompleteItem}
                     />
                     <button type="submit">Submit</button>
            </form>


            <GettingStartedGoogleMap
              containerElement={
                 <div style={{ height: '650px' }} />
             }
             mapElement={
                <div style={{ height: '650px' }} />
             }
             onMapLoad={this.handleMapLoad.bind(this)}
             markers={this.state.markers}
             onMouseOver={this.handleOnMouseMarker.bind(this)}

             onMarkerRightClick={this.handleMarkerRightClick.bind(this)}
            />
        </div>
      );
  }
  /// google maps loaded with support of initMap
  window.initMap = () => {
  ReactDOM.render(
      <GettingStartedExample />,
  document.getElementById('root')
  )
}

We will now add a funcationality to show the info of a marker on mouse hover

var react_update = require('react-addons-update');


update[index_val] = {
    $merge: {showInfo: false}
   };
   var changedMarkers = react_update(this.state.markers, update);
   this.setState({ markers: changedMarkers });

Replacing ReactJS with PreactJS

Recently Facebook modified the license of ReactJS from BSD to BSD+patents license. Addition of patents clause means that if it felt that the product using ReactJS is competing with facebook, it's license will be revoked. So, we started looking for alternatives of ReacJS which would work seamlessly with existing react application. Preact is one such solution.

Preact is a lightweight javascript library (3 kb) which can be used as a replacement for Reactjs

Installation

preact-compat is another layer on top of preact which allows us to switch from react to preact without much changes to the existing code.

First, install preact and preact-compat modules

npm i --save preact
npm i --save preact-compat

To use preact with webpack, we add an alias for react and react-dom in webpack.config.js file

resolve: {
        extensions: ['', '.js', '.jsx'],
        "alias": {
            "react": "preact-compat",
            "react-dom": "preact-compat"
        }
   },

In the example, a simple react hello world application has been created. It has one component HelloWorld.js

import React, { Component } from 'react';
import ReactDOM from "react-dom";


export default class HelloWorld extends Component {
    constructor(props)
    {
        super(props);
    }


  render()
  {


    return (
          <div>
          <h1>
            Hello World!
            </h1>
          </div>
     );

  }
}

ReactDOM.render(
    <HelloWorld />,
    document.getElementById('root')
)

Once the webpack.config.js file is updated to add alias of preact and preact-compat, application works in same way as preact-compat provides same exports as react and react-dom.

var path = require('path');
var webpack = require('webpack');

module.exports = {
    devServer: {
        inline: true,
        contentBase: './src',
        port: 3000
    },
    devtool: 'cheap-module-eval-source-map',
    entry: './dev/js/components/HelloWorld.js',
    module: {
        loaders: [
            {
                test: /\.js$/,
                loaders: ['babel'],
                exclude: /node_modules/
            },
            {
                test: /\.scss/,
                loader: 'style-loader!css-loader!sass-loader'
            }
        ]
    },
    output: {
        path: 'src',
        filename: 'js/bundle.min.js'
    },
    resolve: {
        extensions: ['', '.js', '.jsx'],
        "alias": {
            "react": "preact-compat",
            "react-dom": "preact-compat"
        }
    },
    plugins: [
        new webpack.optimize.OccurrenceOrderPlugin()
    ]
};

We used same configuration changes to react map checkin application and the checkin discovery works as before.

Conclusion

It is a working React-Checkins app providing a fair understanding of how we can integrate Google Maps with (P)ReactJS — all in less than 15 minutes. For reference, the entire code is here

The article originally appeared on Apcelent Tech Blog.

Load Testing a Django Application using LocustIO

Apcelent — Mon, 27 Aug 2018 12:13:29 +0000

Django framework, used for buliding web applications quickly in a clean and efficient manner. As the size of application increases, a common issue faced by all teams is performance of the application. Measuring performance and analysis the areas of improvement is key to deliver a quality product.

LocustIO, an open source tool written in python, is used for load testing of web applications. It is simple and easy to use with web UI to view the test results. It is scalable and can be distributed over multiple machines.

This article demonstrates an example to use locust for load testing of our django web application.

Before starting load testing, we have to decide the pages which we want to test. In our case, we expect users to follow the scenario where they log in, visit different pages and submit CSRF protected forms.

LocustIO helps us in emulating the users performing these tasks on our web application. Basic idea of measuring the performance is make number of request for different tasks and analysis the success and failure of those requests.

Installation

pip install locustio

LocustIO supports python 2.x only. Currently there is no support for python 3.x.

Locust File

Locust file is created to simulate the actions of users of the web applications.

from locust import HttpLocust, TaskSet, task


class UserActions(TaskSet):


    def on_start(self):
        self.login()


    def login(self)
        # login to the application
        response = self.client.get('/accounts/login/')
        csrftoken = response.cookies['csrftoken']
        self.client.post('/accounts/login/',
                         {'username': 'username', 'password': 'password'},
                         headers={'X-CSRFToken': csrftoken})


    @task(1)
    def index(self):
        self.client.get('/')


    for i in range(4):
        @task(2)
        def first_page(self):
            self.client.get('/list_page/')



    @task(3)
    def get_second_page(self):
        self.client.('/create_page/', {'name': 'first_obj'}, headers={'X-CSRFToken': csrftoken})


    @task(4)
    def add_advertiser_api(self):
        auth_response = self.client.post('/auth/login/', {'username': 'suser', 'password': 'asdf1234'})
        auth_token = json.loads(auth_response.text)['token']
        jwt_auth_token = 'jwt '+auth_token
        now = datetime.datetime.now()

        current_datetime_string = now.strftime("%B %d, %Y")
        adv_name = 'locust_adv' 
        data = {'name', current_datetime_string}
        adv_api_response = requests.post('http://127.0.0.1:8000/api/advertiser/', data, headers={'Authorization': jwt_auth_token})



    class ApplicationUser(HttpLocust):
        task_set = UserActions
        min_wait = 0
        max_wait = 0

In above example, locust file defines set of 4 tasks performed by the user - navigate to home page after login, visiting list page multiple times and submitting a form once.

Parameters, min_wait and max_wait, define the wait time between different user requests.

Run Locust

Navigate to the directory of locustfile.py and run.

locust --host=<host_name>

where is the URL of the application?

Locust instance runs locally at http://127.0.0.1:8089

When a new test is started, locust web UI prompts to enter number of users to simulate and hatch rate(number of users per second).

We first try to simulate 5 users with hatch rate of 1 user per second and observe the results.

Once a test is started, locustIO executes all tasks and results of success/failure of requests are recorded. These results are displayed in the format shown below:

As seen from the example above, there is one login request and multiple requests to get a page and submit a form. Since the number of users is less there is no failover.

Now let us increase the number of requests to 1000 users with hatch rate of 500 and see the results.

As we can that some of the requests for fetching the homepage and posting the form fail in this scenario as the number of users and requests increase. With current set of simulated users, we get failure rate of 7%.

Observations:

Most of the failures are in login. Some of the failures stem from the fact that application prevents multiple login from same account in short interval of time.
Get request for pages has very low failure rate - 3%
Post requests have lower failure rates of less than 2%

We can perform multiple tests for different range of users and with the test results, it can be identified under how much stress the application is capable of performing.

The result produces following data for tests:

Type of requests - related to each task to be simulated.
Name - Name of the task/request.
Number of requests - Total number of requests for a task.
Number of failures - Total number of failed requests.
The median, average, max and min of requests in milliseconds.
Content size - Size of requests data.
Request per second.

We can see the details of failed requests in Failures tab which can be used to indetify the root cause of recurring failures.

LocustIO provides option to download the results in sheets, however there is no out of the box result visualization feature in form of graphs or charts.

Load tests results can be viewed in JSON format at http://localhost:8089/stats/requests. These requests can be used as input for data visualization using different tools like Tableau, matplotlib etc.

Thus we are able to determine the system performance at different endpoints in very simple and efficient way. We can expand tests to add more scenarios for more endpoints and quickly get the answers.

The article originally appeared on Apcelent Tech Blog.

How to Create a Facebook Messenger Bot with Python Flask

Apcelent — Mon, 20 Aug 2018 11:38:28 +0000

The bot-revolution has taken over, and everyone is building a bot. In this post we take you through creating your first bot using Python/Flask. The codebase can be found here on Github.

A lightweight web application needs to be created to compliment our bot.

The different technologies that we are going to use are as follows :

Server backend: Flask
Secure tunneling to localhost: Ngrok
Natural Language Processing platform: API AI
Deployment on Facebook Messenger
Python wrapper of OpenWeatherMap web API for weather reports

We are going to demonstrate a very simple feature :

Enquired about the current weather of a location, chatbot should return the weather information.

Example:

Question: What is the current weather of New Delhi?

Reply: Current weather report of New Delhi : max temp: 42.0 min temp: 40.0 current temp: 40.95 wind speed :2.6 humidity 20%

1. Setup API AI

API AI is a conversational experiences platform. Register and create an account at API AI. Create an agent to access test console. Navigate to Settings of the agent and get the Client Access Token.

Creating an intent

An intent is mapping between user expression and the desired response.

We add an intent with user expression "What's the weather in Singapore?"

where Singapore is tagged as an entity of type @sys.geo-city. This parameter can take any value in the conversation and used to determine the response in our application.

We add following response for the user expression: "Current weather report of $geo-city :"

API AI Python client

We can use API AI python SDK for natural language processing.

2. Setup Open Weather Map

Open Weather Map provides API for accessing current weather data of any location on earth. There is nice python wrapper for using these APIs. Register to open weather map and fetch the API key for getting weather data.

owm = pyowm.OWM('your_open_weather_map_api_key')  # You MUST provide a valid API key

forecast = owm.daily_forecast(input_city)

observation = owm.weather_at_place(input_city)
w = observation.get_weather()
print(w)                      
print(w.get_wind())                 
print(w.get_humidity())      
max_temp = str(w.get_temperature('celsius')['temp_max'])  
min_temp = str(w.get_temperature('celsius')['temp_min'])
current_temp = str(w.get_temperature('celsius')['temp'])
wind_speed = str(w.get_wind()['speed'])
humidity = str(w.get_humidity())

If you have a pro subscription for open weather map, then use following:

owm = pyowm.OWM(API_key='your-API-key', subscription_type='pro')

3. Setting up the Flask Server

import apiai

Add client access tokens, page access tokens and verify token

CLIENT_ACCESS_TOKEN = 'your_client_access_token_from_api_ai'
PAGE_ACCESS_TOKEN = 'your_facebook_page_access_token'
VERIFY_TOKEN = 'verification_token_for_facebook_chatbot_using_python'

ai = apiai.ApiAI(CLIENT_ACCESS_TOKEN)

Facebook webhook send the request to the application and if the verification is complete if the token returned by application matches the token entered in facebook application

@app.route('/', methods=['GET'])
def handle_verification():
    if (request.args.get('hub.verify_token', '') == VERIFY_TOKEN):
        print("Verified")
        return request.args.get('hub.challenge', '')
    else:
        print("Wrong token")
        return "Error, wrong validation token"

Application will parse all messages sent by the facebook messenger

@app.route('/', methods=['POST'])
def handle_message():
    '''
    Handle messages sent by facebook messenger to the applicaiton
    '''
    data = request.get_json()

    if data["object"] == "page":
        for entry in data["entry"]:
            for messaging_event in entry["messaging"]:
                if messaging_event.get("message"):

                    sender_id = messaging_event["sender"]["id"]
                    recipient_id = messaging_event["recipient"]["id"]  
                    message_text = messaging_event["message"]["text"]  
                    send_message_response(sender_id, parse_natural_text(message_text)


     return "ok"

Call facebook Graph API using python requests library to send message to the facebook user

def send_message(sender_id, message_text):
    '''
    Sending response back to the user using facebook graph API
    '''
    r = requests.post("https://graph.facebook.com/v2.6/me/messages",

        params={"access_token": PAGE_ACCESS_TOKEN},

        headers={"Content-Type": "application/json"},

        data=json.dumps({
        "recipient": {"id": sender_id},
        "message": {"text": message_text}
    }))

Use API AI to parse the message sent by the user and recieve the bot response from API AI. Fetch weather data from open weather map client and append the report with bot reponse.

def parse_user_text(user_text):

    '''
    Send the message to API AI which invokes an intent
    and sends the response accordingly
    The bot response is appened with weaher data fetched from
    open weather map client
    '''

    request = ai.text_request()
    request.query = user_text

    response = json.loads(request.getresponse().read().decode('utf-8'))
    responseStatus = response['status']['code']
    if (responseStatus == 200):
        print("Bot response", response['result']['fulfillment']['speech'])

        weather_report = ''

        input_city = response['result']['parameters']['geo-city']

        #Fetching weather data
        owm = pyowm.OWM('edd197717da7951b85f8f6936fc27b13')  # You MUST provide a valid API key

        forecast = owm.daily_forecast(input_city)

        observation = owm.weather_at_place(input_city)
        w = observation.get_weather()
        print(w)                      
        print(w.get_wind())                 
        print(w.get_humidity())      
        max_temp = str(w.get_temperature('celsius')['temp_max'])  
        min_temp = str(w.get_temperature('celsius')['temp_min'])
        current_temp = str(w.get_temperature('celsius')['temp'])
        wind_speed = str(w.get_wind()['speed'])
        humidity = str(w.get_humidity())
        weather_report = ' max temp: ' + max_temp + ' min temp: ' + min_temp + ' current temp: ' + current_temp + ' wind speed :' + wind_speed + ' humidity ' + humidity + '%'
        print("Weather report ", weather_report)
        return (response['result']['fulfillment']['speech'] + weather_report)
    else:
        return ("Please try again")


def send_message_response(sender_id, message_text):
    sentenceDelimiter = ". "
    messages = message_text.split(sentenceDelimiter)

    for message in messages:
        send_message(sender_id, message)

if __name__ == '__main__':
    app.run()

Activate a virtual environment, install all packages

pip install -r requirements.txt

and run the script

python fb-chatbot.py

Set up tunnelling to localhost

We have used ngrok to expose local webserver to the internet so that it can be used for callback verification needs to be done for using a webhook with facebook app.

sudo apt-get update
sudo apt-get install ngrok-client

and run the ngrok client

./ngrok http -bind-tls=true 5000

Please note that a secure callback URL (https) is needed for verification.

Set up facebook messenger

Create a facebook page and facebook app. Add Webhook to the app.

Fetch the app ID and update it in the script.

Add Messenger to the app and generate token for the page which has to used for chat.

and select events for page subscription

Restart the server and now the chatbot is ready. Facebook further provides UI elements, you can play around with them here.

This post will help you building a chatbot of your own, using API.AI. Feel free to get in touch with us, for any queries/requests.

The article originally appeared on Apcelent Tech Blog.

Scaling Python Microservices with Kubernetes

Apcelent — Tue, 14 Aug 2018 10:31:17 +0000

We wrote in depth about setting up microservices in one of our previous posts. In this post we are going to talk about building a simple microservice, containerizing it with docker, and scaling those containers using Kubernetes.

It is assumed that reader has first hand experience with flask, redis and docker.

The source repository can be found here.

Building a Microservice using Flask and Redis

In this post lets get started with writing a Microservice with Flask. The microservice has a single point of entry that allows the user to create a resource with a PUT request, corresponding to a URL of their choice, the GET request serves the request. The stale items are deleted after a year.

The crux of the code that will help us build this microservice is as follows:

db = Redis(host='redis', port=6379)

@app.route('/')
def hello():
    db.incr('count')
    return 'Count is %s.' % db.get('count')

@app.route('/<path:path>', methods = ['PUT', 'GET'])
def home(path):

    if (request.method == 'PUT'):
       event = request.json
       event['last_updated'] = int(time.time())
       event['ttl'] = ttl
       db.delete(path) #remove old keys
       db.hmset(path, event)
       db.expire(path, ttl)
       return flask.jsonify(event), 201


    if not db.exists(path):
        return "Error: thing doesn't exist"

    event = db.hgetall(path)
    event["ttl"] = db.ttl(path)
    return flask.jsonify(event), 200

If a PUT request is sent to any URL, with an additional key-value pair, the same gets saved in redis with the current time stamp. If we hit the base url, we get a count of how many times the URL has been hit.

Containerizing the Microservice

Containerizing Redis

We will use docker compose to containerize our microservice. Our microservice consists of a simple flask application that interacts with the redis database.

The compose file will essentially create two different conainers, first for the flask application and second for the redis.

web:
  build: .
  command: python app.py
  ports:
   - "5000:5000"
  volumes:
   - .:/code
  links:
   - redis
redis:
  image: redis

The Dockerfile for the flask application is fairly simple, it creates a working directory and installs all the requirements.

FROM python:2.7
ADD . /code
WORKDIR /code
RUN pip install -r requirements.txt

We are all set to run the containers.
docker-compose up -d.
Once the containers are running we can test if the code works we issue a PUT request using
curl -H "Content-Type: application/json" -X PUT -d '{"hello":999}' http://localhost:5000/testurl
We receive an output as follows:

{ "last_updated": 1485246805, "ttl": 31104000, "hello": 999 }

On making a GET request to the same URL
curl http://localhost:5000/testurl

We receive a successful response as follows :

{ "last_updated": "1485246805", "ttl": 31103997, "hello": "999" }

A fairly straight forward microservice!

Why Kubernetes?

Now that we have built our service and containers are running for us, lets dive into production scenarios. We mostly see containers as if they are faster / simpler VMs, in other workds, each container includes an executre specific functionality with a unique lifecycle and dependency on other containers.

Such an approach is useful in early stages and improves the development environment, however, it creates significant difficulties when we migrate to production environment.

We start realising the challenges like :

How do multiple containers talk to each other?
How do I horizontally scale my container service?
How do multiple container services discover each other?
How do we secure our container network?
How do we deploy new containers and roll back previous ones?

Kubernetes attempts to solve the above such problems. Lets dig into it.

Kubernetes 101

Kubernetes is a system, developed at Google, for managing containerized application in a clustered environment. Most modern day application follow a microservices pattern, Kubernetes helps management of these different services as related components on the same host, configured in a trivial way.

Primarily Kubernetes helps with

Replication of components
Auto-scaling
Load balancing
Rolling updates
Logging across components
Monitoring and health checking
Service discovery
Authentication

Core Kubernetes Concepts

The controlling service in a Kubernetes cluster are called master or control panel components. They operate as main management points and provide cluster-wise system for worker nodes. Node servers have a few requirements that are necessary to communicate with the master components, configure the networking for containers and run workloads assigned to them.

Kubernetes API offers numerous primitives to work with, the more important ones for our purposes are the following:

Pods : Basic unit of a cluster, they are machines running the docker containers that should be controlled as a single "application".
Services : It is a logical logical grouping of a set of pods that perform the same function and constitute a single entity. The IP Address of a service remains stable, irrespective of the number/state/health of the underlying pods. Services enable the communication between pods.
Labels : They are key-value pairs which serves as an arbitrary tag to associate one/more kubernetes resource. There can be multiple labels, and the keys will be unique. These tags can then be selected for management purposes and action targeting.
Deployments : They ensure that a specified number of pods (of a specific kind) are running at any given time. They are a framework for defining pods that are meant to be horizontally scaled, and maintains a constant number of pods. If a container currently goes down, another will be started.

A detailed description can be found in the official docs.

Running Kubernetes locally

Kubernetes is one of the best tools for managing containerized applications, and has been production-ready, however, it has been difficult for developers to test things locally. Minikube is the missing piece in the puzzle that serves as the local development platform.

Minikube has not been designed for scalability or resiliency. It helps to get started with the Kubernetes CLI and API tools on a small single-node. There is not an easier way to test the first couple of commands with Kubernetes.

Minikube starts a virtual machine locally and runs the necessary Kubernetes components. The VM then gets configured with Docker and Kubernetes via a single binary called localkube, resulting into a local endpoint which can be used with the Kubernetes client kubectl.

Install Minikube locally

Quickstart Minikube

Working with Minikube

Let's go ahead and get started by bringing up our local Kubernetes cluster:

minikube start Starting local Kubernetes cluster... Kubernetes is available at
https://192.168.99.101:443

Deploy Redis

We first have to create a redis deployment file.

  apiVersion: extensions/v1beta1
  kind: Deployment
  metadata:
    creationTimestamp: null
    name: redis
  spec:
    replicas: 1
    strategy: {}
    template:
      metadata:
        creationTimestamp: null
        labels:
          app: redis
      spec:
        containers:
        - image: redis
          name: redis
          resources: {}
        restartPolicy: Always
  status: {}

We further use kubectl to create the deployment

$ kubectl create -f redis-deployment.yaml

Expose the Redis Deployment by creating a service

Here is the service file :

apiVersion: v1
kind: Service
metadata:
  creationTimestamp: null
  labels:
    app: redis
  name: redis
spec:
  clusterIP: None
  ports:
  - name: headless
    port: 55555
    targetPort: 0
  selector:
    service.name: redis
status:
  loadBalancer: {}

And we expose the service as follows :
$ kubectl create -f redis-service.yaml

Deploy Flask

Similarly, we need to creat another deployment for flask, here is the deployment file

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  creationTimestamp: null
  name: flask
spec:
  replicas: 1
  strategy:
    type: Recreate
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: flask
    spec:
      containers:
      - args:
        - python
        - app.py
        name: flask
        ports:
        - containerPort: 5000
        resources: {}
        volumeMounts:
        - mountPath: /code
          name: .
      restartPolicy: Always
      volumes:
      - name: .
        persistentVolumeClaim:
          claimName: .
status: {}

The deployment can be created as follows :
$ kubectl create -f flask-deployment.yaml

Expose the Flask Service

We first create the services file :

apiVersion: v1
kind: Service
metadata:
  creationTimestamp: null
  labels:
    io.kompose.service: flask
  name: flask
spec:
  ports:
  - name: "5000"
    port: 5000
    targetPort: 5000
  selector:
    service.name: flask
status:
  loadBalancer: {}

And then we create the service
$ kubectl create -f flask-service.yaml

You have exposed your service on an external port on all nodes in your cluster. If you want to expose this service to the external internet, you may need to set up firewall rules for the service port(s) (tcp:30321) to serve traffic.

$ kubectl get service <service_name> --output='jsonpath={.spec.ports[0].NodePort}'30321%

$ port=$(kubectl get service <sevice_name> --output='jsonpath={.spec.ports[0].NodePort}')

We construct the address for ease of reference later on:

$ address="$(minikube ip):$port"
$ echo $address
192.168.99.101:31637
$ open "http://${address}/?start=23&stop=31"

I hope the article was of help. Do mention in comments, as to what would you like to hear more about.

The article originally appeared on Apcelent Tech Blog.

Taming Snakes inside a Container

Apcelent — Mon, 06 Aug 2018 05:34:42 +0000

In this post, let's talk about taming snakes inside a container. The article is a summary of lessons learned while dockerizing python microservices. In case you want to see a detailed implementation of microservices with Docker and Python, kindly refer to this article which we have published.

The prime reasons that we go with the container is that

containers have lower memory footprints than VMs
Containers provide an isolated user-space instance, instead of a machine emulation

It is easy to spin up new containers and set up environments. The images thus created can be shared and published to private/public registries, and are ready to be used. This provides with a unified workflow whenever a new developer starts working on the codebase.

Building python images with a container

We need not create an extra layer of the virtual environment since each container is unique in itself there is no additional need to create a virtual environment. Different Microservices can be put inside different containers.
An external proxy like NGINX can be used or altogether a separate container for NGINX can be deployed.

Debugging Python Containers

It is easy to run bash (or any other command ) on a running service

docker compose exec $SERVICE_NAME /bin/bash

Log inspection of standard output and error

docker compose logs $SERVICENAME

Low-level information can be got through

docker inspect $SERVICENAME

Persistence

Changes are lost once a running container shuts down. To overcome the same the strategies deployed are

Data Volume to an external host
Another pattern is using separate Docker container to save the data

Testing & Tooling

Docker adds consistency in your CI environments.
Control over what is in container implies repeatable workflow and simpler test environment.
Cloud-based CI options with docker support out there.
Docker Hub, can act as a CI server since you can configure it to create an automated build every time you push a new commit to Github.
Internal API can vary radically for Docker from version to version hence the usage of available tools is recommended.
Another Pythonic way of playing with Docker API is to use docker-py.

Deployment & Scaling

Load Balancing - HA Proxy, NGINX
Provisioning - Ansible, Chef, Puppet
Monitoring - Nagios, Pingdom, New Relic
Logging - Fluentd Loggly
Cloud Infrastructure - Swarm, Kubernetes, Magnum, CoreOS Fleet

Hope the summary and links were of help! Do let us know in comments what other stuff would you like us to write about.

The article originally appeared on Apcelent Tech Blog.

Setup Microservices Architecture in Python with ZeroMQ & Docker

Apcelent — Tue, 31 Jul 2018 05:06:56 +0000

Microservices - What?

Microservices are an architectural style in which multiple, independent processes communicate with each other. These processes are designed to be highly scalable, are de-coupled and perform one small task at a time. These multiple services have their own resources and processes which communicate among themselves through the network.

This differs from a typical client-server architecture where the backend is one monolithic structure encapsulating all the server logic. A microservices architecture addresses separation of concern. This design paradigm allows easier maintenance, greater flexibility, scalability and fault tolerance. The downside of having such a distributed architecture is that if not designed properly it will get complex to debug and maintain.

Example of a Sample Microservice

Let's take a scenario where you are creating an e-commerce store that uses Microservices pattern.

For a typical product say an iPhone on an e-com store, the details page displays:

Basic Information about the product
Your Purchase History
People who bought iPhone, also bought cases
Deals & Discounts associated with the iPhone
Merchant Data
Shipping Options
Product Testimonials and so on and so-forth. Additionally, a sample product details page will have multiple versions of the interface to cater to web, mobile and a REST API for use by 3rd party applications.

In a microservices pattern, the data is spread over multiple services. In this case, it can be

Product Detail Service
Merchant Service
Payment Service
Deals & Discount Service
Inventory Service
Pricing Service
Review Service
Recommendation Service How would these individual services be accessed?

The solution is to implement an API Gateway, which serves as the single entry point for all clients and calls specific Microservices spread across the entire infrastructure as per requirement. A sample industry implementation for the above pattern is the NetFlix API gateway which has multiple API client for different devices. You can read more about it here.

Building a Sample Microservice

There are currently many ways you can set up your microservices.

In this article we will use ZeroMQ to build communication between two processes. ZeroMQ provides building blocks to develop a scalable distributed systed on top of sockets. It is secure - uses elliptic curve cryptography(from Version 4) and offers out of the box communication patterns.

Here are some good things about ZMQ. MQ signifies the threaded Message Queues designed to work asynchronously. Talking more about the zeroMQ is beyond the scope of this article. You should read working with zeromq messaging & distributed systems with zeromq.

The other tool that we are going to be using is Docker. This article assumes the audience to know basics of docker.

There are multiple communication patterns with ZeroMQ to get started with lets setup a simple PUB-SUB with ZeroMQ and Flask, inside a docker container. The illustration below shows the relations and flow amongst the components.

1 & 3 - A flask server runs on prot 5000 and has a URL /downcase/. The URL caters to GET requests, and all requests of the format ?Params= will have a response where the upper case letters will be converted to lower case and this will be returned as response.
2 - The response message is also sent to the ZMQ Publisher running in the same container.
4, 5 - The ZMQ subscriber keeps listening and saves the messages from the ZMQ server to a file called subscriber.log.

Building the Server

Lets see our Dockerfile

FROM ubuntu:14.04

RUN apt-get update
RUN apt-get install -y --force-yes python python-dev python-setuptools software-properties-common gcc python-pip
RUN apt-get clean all

RUN pip install pyzmq

RUN pip install Flask

ADD zmqserver.py /tmp/zmqserver.py

# Flask Port
EXPOSE 5000

# Zmq Sub Server
EXPOSE 4444

CMD ["python","/tmp/zmqserver.py"]

We choose Ubuntu 14.04 as our container OS. We install the basic packages. Using pip we install pyzmq - python bindings to zmq library and we also install Flask. We then expose port 5000(flask server) and 4444(port on which publisher is running.) Additonally we copy the script zmqserver.py which contains all the python code for flask and zeromq and run it.

Lets look at the contents of zmqserver.py

# server.py
import time
import zmq

HOST = '127.0.0.1'
PORT = '4444'

_context = zmq.Context()
_publisher = _context.socket(zmq.PUB)
url = 'tcp://{}:{}'.format(HOST, PORT)


def publish_message(message):

    try:
    _publisher.bind(url)
    time.sleep(1)
    _publisher.send(message)

    except Exception as e:
    print "error {}".format(e)

    finally:
    _publisher.unbind(url)


from flask import Flask
from flask import request
app = Flask(__name__)


@app.route("/downcase/", methods=['GET'])
def lowerString():

    _strn = request.args.get('param')
    response = 'lower case of {} is {}'.format(_strn, _strn.lower())
    publish_message(response)
    return response

if __name__ == '__main__':
    app.run(host='0.0.0.0', debug=False)

The ZMQ pub is running on port 4444. We build a context, and specify the URL. We run the flask app, it has a URL /downcase/ it converts the GET params of Param into lower case, this is the response. This response string is published as a message and the same string is returned to the browser as response.

To build the above docker image, we run the following command sudo docker build -t docker-zmq-pub .and to run the above image docker run --name docker-pub-server -p 5000:5000 -p 4444:4444 -t docker-zmq-pub

We map the port 5000 and 4444 of the container to that of the machine, so that no matter where the client is, it can subscribe to the publisher.

Subscriber Client

# client.py
import zmq
import sys
import time
import logging
import os

HOST = '127.0.0.1'
PORT = '4444'

logging.basicConfig(filename='subscriber.log', level=logging.INFO)


class ZClient(object):

    def __init__(self, host=HOST, port=PORT):
    """Initialize Worker"""
    self.host = host
    self.port = port
    self._context = zmq.Context()
    self._subscriber = self._context.socket(zmq.SUB)
    print "Client Initiated"

    def receive_message(self):
    """Start receiving messages"""
    self._subscriber.connect('tcp://{}:{}'.format(self.host, self.port))
    self._subscriber.setsockopt(zmq.SUBSCRIBE, b"")

    while True:
        print 'listening on tcp://{}:{}'.format(self.host, self.port)
        message = self._subscriber.recv()
        print message
        logging.info(
            '{}   - {}'.format(message, time.strftime("%Y-%m-%d %H:%M")))

if __name__ == '__main__':
    zs = ZClient()
    zs.receive_message()

We specify the IP Address and Port of the Publisher, currently its running on the same machine hence 127. We listen on the URL tcp://IP:PORT. Once we receive a message we log it to a file called subscriber.log with the timestamp. To run the client all it takes is python .py. If you build on top of the above architecture, it can very well serve as a near realtime log-aggregation engine.

I have tested the above code to work on an Ubuntu machine. The code is hosted on github. This was a basic intro on how to setup zmq, docker and a python server, in our next article we will build sample microservices using things we have learned here.

Hope the article was of help. Feel free to submit your thoughts in the comments.

Further Reading

Microservices

The article originally appeared on Apcelent Tech Blog.