DEV Community: MikeVV

Illogical behaviour of python web framework (issue investigation story)

MikeVV — Mon, 20 Feb 2023 19:17:38 +0000

TLDR: Strange issues with Sanic framework on my new pet project were fixed in ugly way and I do not like it but do not see the other way and asking for help/advice.

Recently I have tried a new python ASGI web framework for my new pet project and faced an weirdest issue on my memory.

Our hero framework here is Sanic - new, modern async web framework with great slogan "Build fast. Run fast." on the site.

Let's move to the story itself.

I have created an web application backbone with couple endpoints and it works at the beginning.

code like this:

from sanic import Sanic


app = Sanic("pet project N7355")


@app.route("/")
async def index_handler(request):
    return json(dict(name="pet project N7355", version="0.1.0"))


@app.route("/system/health")
async def health_handler(request):
    return json(dict(status="ok"))


@app.route("/system/stat")
async def stat_handler(request):
    return json(dict())


app.run(host='0.0.0.0', port=8765, debug=debug)

But after I split that code to modules and moved initialisation of endpoints to separate function I have started to see an error about Sanic cannot start as there is no routes defined.

new code:
main.py

from sanic import Sanic
from routes import init_routes


app = Sanic("pet project N7355")
init_routes(app)

app.run(host='0.0.0.0', port=8765, debug=debug)

routes.py


def init_routes(app):
    @app.route("/")
    async def index_handler(request):
        return json(dict(name="pet project N7355", version="0.1.0"))


    @app.route("/system/health")
    async def health_handler(request):
        return json(dict(status="ok"))


    @app.route("/system/stat")
    async def stat_handler(request):
        return json(dict())

Similar approach for defining a routes was used in my previous project with Flask and worked perfectly fine but not now.
I have tried to run application in single thread mode as suggested in error message but there was no such error in this more.

I thought what a strange issue and fixed that by workaround (dirty hack in fact) - moved "/" method from function in module to main file.

My next step was about connecting to database.

I have used PeeWee ORM (not an mainstream choice but it looks simple and easy) and as there is no plugins to link Sanic with PeeWee I used approach listed on Sanic site - put a connection to application context app.ctx.db = db and faced another error - "cannot use not initialised database".

code with db connection:
main.py

from sanic import Sanic
from routes import init_routes
from db import init_db


app = Sanic("pet project N7355")
app.ctx.db = init_db()

# workaround to fix routes issue in multithread mode
@app.route("/")
async def index_handler(request):
    return json(dict(name="pet project N7355", version="0.1.0"))


init_routes(app)

app.run(host='0.0.0.0', port=8765, debug=debug)

routes.py


def init_routes(app):
    @app.route("/system/health")
    async def health_handler(request):
        return json(dict(status="ok"))


    @app.route("/system/stat")
    async def stat_handler(request):
        app.ctx.db.connect()
        msg_count = Message.select().count()
        app.ctx.db.close()
        return json(dict(msg_count=msg_count))

db.py

from peewee import *

....

def init_db():
  ....
  return db

At first I did not realised that these issues are about the same but after some debugging with prints I have found that Sanic in multithreaded mode besides main instance of app creates two additional instances (I suppose I instance per thread) and these new instances in fact processing requests, and do not have routes and db defined.

After finding that behaviour I understand how to fix the issues - move all initialisation from "runtime" level to "module import" level

fixed code:
main.py

from sanic import Sanic
from routes import app
from db import db

app.ctx.db = db

app.run(host='0.0.0.0', port=8765, debug=debug)

routes.py


app = Sanic("pet project N7355")

@app.route("/")
async def index_handler(request):
    return json(dict(name="pet project N7355", version="0.1.0"))

@app.route("/system/health")
async def health_handler(request):
    return json(dict(status="ok"))

@app.route("/system/stat")
async def stat_handler(request):
    app.ctx.db.connect()
    msg_count = Message.select().count()
    app.ctx.db.close()
    return json(dict(msg_count=msg_count))

db.py

from peewee import *

....

def init_db():
  ....
  return db

db = init_db()

from one hand this solution works but from other it looks ugly for me as with that code structure it hard to test modules code in isolation.

I completely understand that I used not the best approach with routes and better rewrite those with blueprints that available in Sanic, but still, for DB I have used approach listed on the Sanic site and is does not work for me because that Sanic's specifics.

As a result I have a open questions:

[rhetorical] Why it made is such way where only one way to make things right exists and this specifics is not listed in documentation?
[practical] Maybe I did something wrong and things that I see as logical is not in fact logical?

Maybe someone faced something like this and know how to do that properly?

Yet Another DIY Testing Framework (part 1 of 3) - The idea

MikeVV — Sun, 08 May 2022 09:52:24 +0000

Disclaimer

All listed in this article (if this text can be considered as an article) is related to the world of backend Java, other modern enterprise technologies may have much better situation with a testing and testing frameworks.

Also an author is not a best in the world expert in Java testing frameworks, so there is a possibility that somewhere in the wild already exists some framework that solves all the issues and better option to use instead of inventing described below bicycle.
(If you know such, please let me know in comments)

Introduction (What problem we are solving?)

There are a lot of more or less good testing frameworks in the wild. Some of existing testing frameworks doing their job very well and are de facto a standard, like a "cucumber" for BDD testing.

And also important about all these testing frameworks that is more popular framework have more features and therefore have higher learning curve, less fast as in development new tests, as in tests execution, and may have (not necessary, for sure, but ...) some overcomplicated processes that is leading to unclear tests.

The Idea

Technical side

Java hava a build in scripting engine that can be used for multiple purposes.
This embedded script ion engine have a number of advantages for our task:

It is really easy to use. Minimal usage example is about 3 lines of code
One of the scripting languages implemented in that engine, and also build in is a JavaScript. Well know by almost everybody and extremely popular nowadays.
This engine have a bi-directional bridge between Java and JavaScript (and actually every other scripting language, but here we are interested in JavaScript)

And some disadvantages where most important is a performance. JavaScript code executed by this scripting engine is about 500 times slower than plain Java. Not a NodeJS performance for sure, but this performance is not an issue taking into account our area of interest.

The Framework Idea

Idea is quite simple:

Write an “executor” for “test scripts” enriched with useful functions for your application testing.
Write a “test runner” that will: 2.1. Find all “test scripts” from particular folder 2.2. Execute scripts with “executor” and collect the result 2.3. Assemble a testing report from collected results.

Framework in details

Test Script

To be honest “test script” is not a part of the framework, but anyway must be explained.

Test Script is a JavaScript file that tests some application function.
Test script is:

use additional build in into “executor” functions to communicate with Application
write logs to stdout (to simplify “test runner” task of gathering results
fail with exceptions in case of some issues detected in tested application

Executor

Executor is a Java application that can execute JavaScript file passed there as a parameter.
Additionally executor should have a application configuration, to be able to connect to it in a right way.

In case of failed script execution, Executor must exit with non zero exit code.

Test Runner

Is an application implemented in any language you like (To be consistent better to implement it in Java or JavaScript to minimize amount of different technologies need to know by developers).

Test Runner scan a folder with tests and execute files from there with Executor.

Tests may be organized to sub folders by some Application aspect to test and test runner may process those sub folders in parallel (for improving performance).

Runner for each executed script collect:

Output from stdout (logs and errors)
Script exit code
Execution time

After executing all tests (scripts) all collected information may be assembled into one report (or number of reports in case of parallel tests execution).

Repot is a document in your preferred format (HTML, PDF, DOC, TXT, etc) with list of tests and their status.

At the end test runner must exit with non-zero exit code in case at least one test failed.
(This will simplify integration with external CI/CD systems)

How it expected to work (example)

For example if our testing framework consists of two java parts:

Executor - test-executor.jar
Test Runner - test-runner.jar

And we have following list of tests:

tests/
  Test_endpoint1_get.is
  Test_endpoint1_post.js
  Test_endpoint1_delete.js
  Test_endpoint1_auth.js

Tests run will be following

$ java -jar test-runner.jar tests/

Execution of one test script will be following

$ java -jar test-executor.jar tests/Test_endpoint1_post.js

All test report would be like this

Tests report:
tests/Test_endpoint1_get.is - ok (0.05sec)
tests/Test_endpoint1_post.js - ok (0.10sec)
tests/Test_endpoint1_delete.js - ERROR (exit code - 1)
  … Detailed log of error …
tests/Test_endpoint1_auth.js - ok (0.03sec)

PS: Please let me know if you like the idea and will be interested to see this framework implementation.

Are there modern JavaScript frameworks without NodeJS dependency?

MikeVV — Fri, 11 Feb 2022 17:23:46 +0000

Some time ago I have checked in the Internet JavaScript frameworks for my pet project and found out that there is no frameworks what can work purely on client side (in browser) and have no dependency to NodeJS.

If you know any framework that is commonly used (more or less) and is not require NodeJS please let me know in comments 🙂

Simple queue with PostgreSQL

MikeVV — Sat, 15 Jan 2022 10:12:46 +0000

If you need a queue in database but does not have in your hands Oracle with its' Advanced Queues and only PostgreSQL instead, you can simulate the queue with below simple recipe.

To make it more generic let's assume you need more than one queue (or may have need 2nd queue in the future)

1. Create a base queue table

CREATE TABLE base_queue
(
   id           serial      NOT NULL,
   status       integer     DEFAULT 0 NOT NULL,
   try_count    integer        DEFAULT 0 NOT NULL,
   max_tries    integer        DEFAULT 5 NOT NULL,
   params       json,
   create_time  timestamp   DEFAULT CURRENT_TIMESTAMP NOT NULL,
   update_time  timestamp,
   priority     integer     DEFAULT 0 NOT NULL
);

attributes:

status - the status of the message in the queue, 0 is "unprocessed", -1 "failed", 1 "processed" (Also, you can use your own status numbers if you like)
try_count - approximate tries count (why it is approximate, will be explained a bit later)
max_tries - tries limiter (not necessary to use, but may be useful sometimes as allow you to put in queue messages with different retry count)
params - your business payload in json format
create_time - when you put message to queue
update_time - when your message was last updated (both timestamps useful for troubleshooting)
priority - may not need it, but better to have just in case

2. Create your queue as follows

CREATE TABLE <YOUR QUEUE NAME> ()
INHERITS (base_queue);

Thats all. After that you will have a table with the same structure as "base_queue"

3. work with queue

3.1. put a message to queue - just insert a record like

insert into <YOUR QUEUE NAME> (params) values (<YOUR BUSINESS PAYLOAD JSON>);

3.2. get a message for processing from queue

select * from <YOUR QUEUE NAME> where 
  status = 0 
limit 1 
for update skip locked;

important aspect here: Do not use "AUTOCOMMIT" option. All this magic works only in the context of transaction.
You also may extend SQL with priority condition or try_count if it necessary.

priority

  ... 
  and priority = <REQUIRED PROIRITY>
  ...

try_count

  ... 
  and try_count <= max_tries
  ...

3.3. failed processing - case 1 (exception and rollback)

In this case on the transaction rollback you record in queue will again become available for processing, "try_count" and "update_time" will not be updated. this is why "try_count" is "approximate".

3.4. failed processing - case 2 (failing processing by your code)

update the record (to put back to queue)

update <YOUR QUEUE NAME> set 
  try_count = try_count+1,
  update_time = CURRENT_TIMESTAMP
where id = <ID>;

or in case you need to fail it (for example then you reached max processing attempts)

update <YOUR QUEUE NAME> set 
  try_count = try_count+1,
  update_time = CURRENT_TIMESTAMP,
  status = -1
where id = <ID>;

commit

3.5. saving sucessull processing state

update the record

update <YOUR QUEUE NAME> set 
  try_count = try_count+1,
  update_time = CURRENT_TIMESTAMP,
  status = 1
where id = <ID>;

commit

House keeping

According to your retention policy (if any):

From time to time need to remove records with status 1 or -1
From time to time need to remove records with try_count >= max_tries (if you are using filters by try_count while getting records from queue)