Queick: A Simple Job Queue System for Python

asmsuechan — Sat, 07 Nov 2020 08:55:08 +0000

Queick is a simple job queue system for Python.

Feature

Written in Python only standard libraries
Job-queue manager without redis
Working for low-spec machines
Retry
Periodic execution
Retry on network available
Scheduling
Thread-base job execution

Main target

Queick can be fully utilized in low-spec computers, such as Raspberry Pi. This system does not aim to be used at the server-side of a normal Web service.

Usage

First, install queick by using pip command.

pip install queick

Second, Second, prepare a job file (jobfunc.py) and an application (test.py).

# jobfunc.py
import time
def function(arg):
    time.sleep(1)
    print(arg)

# test.py
from queick import JobQueue
from jobfunc import function
from time import time

q = JobQueue()
q.enqueue(function, args=("hello",))
q.enqueue_at(time() + 5, function, args=("world",)) # Run after 5 seconds

st = SchedulingTime()
st.every(minutes=1).starting_from(time.time() + 10)
q.cron(st, function, args=(1, 2,)) # Run every 1 minute after 10 seconds

Third, start queick worker.

queick

Finally, run the application.

python test.py

Architecture

As mentioned above, a job is executed on a thread and implemented by concurrent.futures. Also, event notification is used instead of polling to receive jobs.

If a user specifies retry=True option to q.enqueue(), the job will be retried when it fails. The default retry function is Exponential backoff.

Application

Queick is integrated into an entrance recording system made of Raspberry Pi and NFC reader at my laboratory.

This system records the students' entering/leaving history on Slack when they enter and leave by touching their student id.

However, this system has a problem. The Raspberry Pi often disconnects the Internet for some reason. Thus, a job queue system was needed to be integrated to retry the failed post.

I developed a job queue system from scratch because I did not want to install and operate Redis server on the Raspberry Pi

Conclusion

The core of Queick was made in 2 days, yes, my developing speed was quick, therefore, queick. I totally enjoyed developing it.

For further information, check repository asmsuechan/queick.

Build a GraphQL API on Function Compute + API Gateway

asmsuechan — Sat, 11 Aug 2018 09:24:33 +0000

What is GraphQL?

GraphQL is a query language for APIs
from: https://graphql.org/

We got a result by throwing POST request to one endpoint.

The curl command below is an example to get messages.

$ curl -H "Content-Type: application/json" -X POST -d '
{
  "query": "{ messages { name body } }"
}
' http://xxxxxxxxxxxxxxxxxx-cn-shanghai.alicloudapi.com/
 {"data":{"messages":[{"body":"Hello","name":"asmsuechan"},{"body":"World","name":"suechan"}]}}

For more information, please google graphql. I don't go detailed explanations about it in this article.

Minimum Code

First, I wrote the minimum GraphQL code and made it work on Function Compute.

I expected the combination of query and return value below:

Query: "{ hello }"
Return Value: { data: { hello: "world" } }

This sample below came from Official GitHub.

// index.js
const { hook } = require('fc-helper');
const {
  graphql,
  GraphQLSchema,
  GraphQLObjectType,
  GraphQLString
} = require('graphql');

const schema = new GraphQLSchema({
  query: new GraphQLObjectType({
    name: 'RootQueryType',
    fields: {
      hello: {
        type: GraphQLString,
        resolve() {
          return 'world';
        }
      }
    }
  })
});

const query = '{ hello }';

module.exports.handler = (event, context, callback) => {
  graphql(schema, query).then((result) => {
    callback(null, { statusCode: 200, body: result });
  });
});

It's easy. Just exec graphql() in the handler function.

More detailed API

Next, I made the API better. I expected the combination of query and return values below"

Query: "{ messages { name body } }"
Return value: {"data":{"messages":[{"body":"Hello","name":"asmsuechan"},{"body":"World","name":"suechan"}]}}

I defined a type named message to get plural message.

// index.js
const { hook } = require('fc-helper');
const {
  graphql,
  GraphQLSchema,
  GraphQLObjectType,
  GraphQLList,
  GraphQLString
} = require('graphql');
const atob = require('atob');

const messages = [
  {
    name: 'asmsuechan',
    body: 'Hello'
  },
  {
    name: 'suechan',
    body: 'World'
  }
];

const schema = new GraphQLSchema({
  query: new GraphQLObjectType({
    name: 'RootQueryType',
    fields: {
      messages: {
        type: GraphQLList(
          new GraphQLObjectType({
            name: 'message',
            fields: {
              name: { type: GraphQLString },
              body: { type: GraphQLString },
            },
          }),
        ),
        resolve() {
          return messages;
        }
      }
    }
  })
});

module.exports.handler = (event, context, callback) => {
  const request = JSON.parse(event.toString('utf8'))
  const query = JSON.parse(atob(request["body"]))["query"]
  graphql(schema, query).then((result) => {
    callback(null, { statusCode: 200, body: result });
  });
};

I saw it works fine by executing curl.

$ curl -H "Content-Type: application/json" -X POST -d '
{ "query": "{ messages { name body } }"}
'http://xxxxxxxxxxxxxxxxxx-cn-shanghai.alicloudapi.com/

{"data":{"messages":[{"body":"Hello","name":"asmsuechan"},{"body":"World","name":"suechan"}]}}

Conclution

It's very happy to use GraphQL instead of REST API on Function Compute + API Gateway because we can create complex APIs without many API Gateways.

However, it's too fast to run as production environment.

Here's my repository of this article:

https://github.com/asmsuechan/fc_graphql_api

DEV Community: asmsuechan