DEV Community: Kamal Mustafa

Passing Django request object

Kamal Mustafa — Wed, 30 Oct 2024 03:15:55 +0000

It handy to pass the request object around but that's a bad practice. Maybe it's influenced from past experiences in PHP where $_REQUEST is available from anywhere in your code.

Limit request object in views function only. It should gather all the required data from the request, maybe using form or serializer. Calling other functions from the views function should only pass specific parameters that function need.

This will make testing easier as you don't have to mock the request object when testing specific functions that are not views. It encourage better API for your functions, for example:-

def get_articles(user, category):
    pass

Instead of:-

def get_articles(request):
    category = request.GET.get("category")
    user = request.user
    ...

Adding pytest command to Django manage.py

Kamal Mustafa — Mon, 28 Oct 2024 04:46:07 +0000

The preferred way to invoke pytest is via the pytest command. Even the app pytest-django doesn't provide a management command to invoke pytest.

But I prefer to invoke my django project through a single place, which is the manage.py script at the project root. So I just add it to the script:-

#!/usr/bin/env python
"""Django's command-line utility for administrative tasks."""
import os
import sys
import pathlib
import subprocess

def main():
    BASE_DIR = pathlib.Path(__file__).parent.absolute()
    sys.path[0] = str(BASE_DIR / "src")

    """Run administrative tasks."""
    os.environ.setdefault("DJANGO_SETTINGS_MODULE", "medan.settings")
    try:
        from django.core.management import execute_from_command_line
    except ImportError as exc:
        raise ImportError(
            "Couldn't import Django. Are you sure it's installed and "
            "available on your PYTHONPATH environment variable? Did you "
            "forget to activate a virtual environment?"
        ) from exc

    # wrap pytest command here so we don't have to manage
    # PYTHONPATH in multiple places. django-pytest unfortunately
    # doesn't provide native management commands so we still need
    # to invoke pytest directly
    if len(sys.argv) > 1 and sys.argv[1] == "pytest":
        pytest_args = " ".join(sys.argv[2:])
        os.environ["PYTHONPATH"] = sys.path[0]
        subprocess.run(f".venv/bin/pytest {pytest_args}", shell=True, env=os.environ)
    else:
        execute_from_command_line(sys.argv)


if __name__ == "__main__":
    main()

This allow us to run pytest as:-

poetry run python manage.py pytest -s -x ./path/to/tests/test_*.py

PostgreSQL as a Celery Broker

Kamal Mustafa — Wed, 23 Oct 2024 01:48:20 +0000

I’m currently using Redis as the broker for Celery, but since I already have PostgreSQL running, I wondered if I could use it as a broker instead. This would eliminate the need for a separate Redis server, saving cost a bit.

I came across this GitHub issue discussing PostgreSQL as a potential Celery broker. This led me to discover an interesting alternative task queue called BQ, which is built specifically to use PostgreSQL for task management.

What caught my attention with BQ is that it use mix way to process tasks: the traditional polling method and a pub-sub model, which is pretty unique. For those unfamiliar, BQ’s pub-sub system leverages PostgreSQL’s NOTIFY function. The way it works is that when a new task is added, NOTIFY sends a signal to listening clients, so they can act immediately—no more constant polling to check for new tasks, which can be more efficient.

One thing I’ve been mindful of when working with task queues is the noisy neighbor problem, where one task queue can consume too many resources and affect the performance of others. I haven’t fully explored how BQ handles this, but it’s definitely something to look into if you’re considering a PostgreSQL-based task queue.

Wagtail programmatically create page translation

Kamal Mustafa — Mon, 30 Sep 2024 04:06:00 +0000

I can't find any programmatic interface to create page translation. All the logics seems being implemented in SubmitTranslationView in wagtail.contrib.simple_translation.views.

So the only way to access those programmatically is by simulating the request to access the views. I wrapped this in a function named translate_page(). To translate a page, we can call this function as:-

page_ja = translate_page(user, page, "ja")

Or we can pass the optional parameter include_subtree:-

page_ja = translate_page(user, page, "ja", include_subtree=True)

And the function is defined as:-

def translate_page(user, page, lang, include_subtree=False):
    locale, created = Locale.objects.get_or_create(language_code=lang)
    data = {"locales": [locale.id], "include_subtree": include_subtree}
    url = reverse(
        "simple_translation:submit_page_translation", kwargs={"page_id": page.id}
    )
    factory = RequestFactory()
    request = factory.post(url)
    request.POST = data
    request.user = user
    get_response = lambda request: None
    middleware = SessionMiddleware(get_response)
    middleware.process_request(request)
    request.session.save()
    messages = FallbackStorage(request)
    setattr(request, "_messages", messages)

    SubmitPageTranslationView.model = type(page)
    SubmitPageTranslationView.as_view()(request, page_id=page.pk)
    page_translated = page.get_translations()[0].specific
    return page_translated

I have another function called translate_snippet(). The only difference is just the url to submit and no include_subtree parameter.

Page tree gotchas

For all our sites, we have a setup script that will automatically populating some posts and their translation so that developer can start working right away instead of having to create sample pages manually.

Our page structure is like this:-

HomePage ==> BlogIndexPage ==> BlogPage

And the setup script will do the following:-

Create the root admin user, let's call it user One.
Create instance of HomePage
Attach HomePage as new root page
Create Japanese translation of HomePage
Create instance of BlogIndexPage
Populate BlogPage with sample posts and attach them under BlogIndexPage
Translate BlogIndexPage with include_subtree=True

This all works well until we're setting new site where the page structure is:-

BlogIndexPage ==> BlogPage

So we omitted HomePage (which I think a bad idea now but let's save it for another topic) since this site is mainly a blog. The first thing I notice after running the setup script is that no translation of the blog posts created, despite we already passed include_subtree when translating BlogIndexPage.

My first gut reaction was it could be some changes in new wagtail versions. Most our sites being created few years ago and still on wagtail 5 but for this new site, we will start with wagtail 6 since it's the latest.

But looking at wagtail's commit logs for simple_translation views.py, the code last changes was three years ago. And we can see the code basically the same between wagtail 5 and 6.

The problem with translate_page function above is that it doesn't check for any errors. Because catching errors mean you have to parse the request's response for some error string. But tracing the code flow lead me to a stage where I can see the code is not executed because the form is not validated.

Printing form.errors showed error messages related to invalid locale. This is strange because we can see in the translate_page function above we're creating the locale if it doesn't exists yet.

And printing the form's self.fields["locales"].choices I can the locale is there in the choice during first call of translate_page() when translating the root page, but the choices was empty when calling it second time to translate BlogIndexPage.

Reading the form's code, locales field's choices is set dynamically in the __init__ method, where locale of the page that already translated will be removed. This could be the reason why the locale was empty in the second call. But the page is not translated yet!

Let's recall back the process:-

Create BlogIndexPage
Attach BlogIndexPage to root page
Translate root page to ja
Populate BlogPage and attach them to BlogIndexPage
Translate BlogIndexPage to ja

And this is where the light bulb (💡) came in, after hours of debugging. In the original script, we're translating HomePage to ja first, and then BlogIndexPage with all its children. But in this new script, the root page is BlogIndexPage. So BlogIndexPage already translated the second time we call translate_page!

So that's the reason locales choices become empty.

Inline script defer

Kamal Mustafa — Thu, 04 Jan 2024 01:20:06 +0000

Problem when using defer to load external script is when you have inline script that depends on the external script. This because inline script is executed immediately while deferred script is loaded after the whole page has beed parsed.

Take this script for example:-

<!doctype html>
<html>
<head>
    <script>
        console.log('Start ...');
    </script>
    <script defer src="https://unpkg.com/htmx.org@1.9.10" integrity="sha384-D1Kt99CQMDuVetoL1lrYwg5t+9QdHe7NLX/SoJYkXDFfX37iInKRy5xLSi8nO7UC" crossorigin="anonymous"></script>
    <script>
    htmx.logger = function(elt, event, data) {
    if(console) {
        console.log(event, elt, data);
        }
    }   
    </script>
    <script>
        htmx.logger("Inside head");
    </script>
</head>

<body>
    <h1>Javascript DOMContentLoaded</h1>
    <script type="module">
        htmx.logger("Inside body");
    </script>

    <script>
        window.addEventListener('DOMContentLoaded', function() {
            console.log('Inline JS');
        });
    </script>
</body>
</html>

The inline script will failed:-

Start ... localhost:9000:5:17
Uncaught ReferenceError: htmx is not defined
    <anonymous> http://localhost:9000/:9
localhost:9000:9:5
Uncaught ReferenceError: htmx is not defined
    <anonymous> http://localhost:9000/:16
localhost:9000:16:9
Uncaught TypeError: htmx.logger is not a function
    <anonymous> http://localhost:9000/:23
localhost:9000:23:14
Inline JS localhost:9000:28:21
GET
http://localhost:9000/favicon.ico
[HTTP/1 404 File not found 1ms]

Fortunately script of type module is deferred automatically. This will work as expected:-

<!doctype html>
<html>
<head>
    <script>
        console.log('Start ...');
    </script>
    <script defer src="https://unpkg.com/htmx.org@1.9.10" integrity="sha384-D1Kt99CQMDuVetoL1lrYwg5t+9QdHe7NLX/SoJYkXDFfX37iInKRy5xLSi8nO7UC" crossorigin="anonymous"></script>
    <script type="module">
    htmx.logger = function(elt, event, data) {
    if(console) {
        console.log(event, elt, data);
        }
    }   
    </script>
    <script type="module">
        htmx.logger("Inside head");
    </script>
</head>

<body>
    <h1>Javascript DOMContentLoaded</h1>
    <script type="module">
        htmx.logger("Inside body");
    </script>

    <script>
        window.addEventListener('DOMContentLoaded', function() {
            console.log('Inline JS');
        });
    </script>
</body>
</html>

Start ... localhost:9000:5:17
undefined Inside head undefined localhost:9000:11:17
undefined Inside body undefined localhost:9000:11:17
htmx:beforeProcessNode 
<body>

Object { elt: body }
localhost:9000:11:17
Inline JS localhost:9000:28:21
GET
http://localhost:9000/favicon.ico
[HTTP/1 404 File not found 1ms]

htmx:load 
<body>

Object { elt: body }
localhost:9000:11:17

Browsers support is 96.81% so I think it's pretty good.

Original code taken from https://cylab.be/blog/188/defer-async-and-inline-javascript

Chiselled Ubuntu containers GA

Kamal Mustafa — Mon, 11 Dec 2023 23:52:04 +0000

Smaller Ubuntu container images. But I think it still low level tools for building such images, unlike distroless where you can use it right away.

Isso comment system - alternative to disqus

Kamal Mustafa — Wed, 02 Aug 2023 11:41:04 +0000

It's a simple self hosted comment system, build in Python and just use sqlite database. By default it only support single website but with some configuration you can also use it for multiple websites.

This post however, not much about Isso but how do I found it.

I was reading Luke Plant's blog and then noticed his comment form a little bit, different.

It triggered the curiousity inside me so I "view source" it using the browser Inspect function.

The word isso definitely stand out, so I googled it (isso comments). The rest is history.

Any estimates longer than 3 days tend to be wrong

Kamal Mustafa — Tue, 16 May 2023 00:16:12 +0000

So breakdown tasks so that each part can be finished within 3 days.

Break down tasks by 𝗱𝗲𝗰𝗼𝗺𝗽𝗼𝘀𝗶𝗻𝗴 𝗹𝗮𝗿𝗴𝗲 𝗰𝗼𝗺𝗽𝗹𝗲𝘅 𝘁𝗮𝘀𝗸𝘀 into smaller, more manageable ones. For example, we found that jobs estimated up to 3 days of work are accurate.

Dr. Milan Milanović

Python virtualenv no activate

Kamal Mustafa — Thu, 19 Jan 2023 06:08:28 +0000

My workflow when starting new python project or experimenting with something:-

mkdir myproject
cd myproject
python -mvenv venv
venv/bin/pip install something
venv/bin/python
>>> import something

Look, there's no need to do something like source venv/bin/activate to make use of your virtual environment.

Not using activate also good for documentation like in README as you don't have the two steps commands and people forgot to run the first step problem.

Have ever got into situation where you have to ask, "Have you run source venv/bin/activate?" You don't need to ask this question if you invoke the venv directly.

Of course this is just for quick experimentation. For a real project we're using poetry. And here's labzero our starter project for django.

GNOME Asia Summit 2022

Kamal Mustafa — Thu, 08 Dec 2022 04:17:35 +0000

GNOME Asia Summit 2022 this year was held as physical event in Kuala Lumpur for the first time after 2 years of online conferences due to the COVID-19 pandemic.

My talk proposal was accepted and I'm presenting the topic of Remote IDE for remote developer. Remote IDE has become my interests this year after fully adopting Github Codespaces and later on Gitpod in my development workflow.

Aryan Kaushik shared his journey in Google Summer of Code (SoC) working on GNOME app Pitivi, porting it from GTK3+ to GTK4.

Rosanna Yuen from GNOME Foudation explaining Gnome community Code of Conduct.

Adzmely Mansor from Nexoprima shared their experiences integrating critical care medical devices with modern system. I was looking forward for this talk as Adzmely is very well known in local development community. Last time listening to his presentation on linux vertical performance tuning really blew my mind.

One interesting sharing from him is how they send data from endoscopy machine to the PACS (picture archiving and communication system) which is a medical imaging technology used in healthcare organizations to securely store and digitally transmit electronic images and clinically-relevant reports. The problem is the only way to get data from the machine is by plugging a USB thumbdrive. That make automation not possible. What they did is to plug in a raspberry PI instead which configured to emulate as USB mass storage. Then it just a matter of using inotify to detect writing to the file system and send the file to PACS.

It's a weekend very well spent connecting with some old friends and also meeting new people during the event. This probably my last event for this year and hopefully next year we will have more physical conferences.

Misc notes on Github Codespaces

Kamal Mustafa — Sun, 26 Jun 2022 09:59:00 +0000

Just some quick notes since I have spent few hours trying to get nix working on Codespaces and get our project running.

I have shared my Dockerfile in this issue at Nix's Github - https://github.com/NixOS/nix/issues/6680

docker-compose is working fine. We can customize our workspace using Dockerfile. Have to use images provided by Github, using our own image doesn't work. I guess we have to do some more works as stated in this post - have Dockerfile.base as well.

Workspace is stopped after the idle timeout (30mins) but all our work on disk is preserved. So uncommitted changes is still there when we come back, just like working on local computer.

Everytime we changed stuff in .devcontainer we have to rebuild the container. For quickly testing your Dockerfile, it's better to just run docker build in the terminal and if it build cleanly then we rebuild.

cd .devcontainer
docker build .

A failure in rebuild will drop us into recovery container so we can fix our Dockerfile but tools are limited here.

https://github.com/microsoft/vscode-dev-containers/tree/main/containers/codespaces-linux/.devcontainer

Heroku cli login issue - https://github.com/heroku/cli/issues/1914 - ip address mismatch. We have same issue when working with remote vm, a workaround is to tunnel http traffic from our browser to the vm, either with vpn or simply ssh tunnel (we use sshuttle).

Workaround 1 - define env var HEROKU_API_KEY - the downside is that your action would not be under your username but the username of that api key owner
Workaround 2 - Copy the credentials from your local machine - it's stored in $HOME/.netrc file

https://devcenter.heroku.com/articles/authentication

VSCode editor doesn't enable word wrap by default. Add this to devcontainer.json:-

"editor.wordWrap": "wordWrapColumn",
"editor.wrappingIndent": "same",
"editor.wordWrapColumn": 80

List of limitations - https://code.visualstudio.com/docs/remote/codespaces

Django: _ (underscore) settings gotchas

Kamal Mustafa — Tue, 27 Jul 2021 10:38:37 +0000

It so happened that after we add some localization setup in our app, running our tests error out with:-

ackages/django/conf/__init__.py", line 122, in configure
INTERNALERROR>     raise TypeError('Setting %r must be uppercase.' % name)
INTERNALERROR> TypeError: Setting '_' must be uppercase.

The settings in question that we have added is this:-

try:
    from django.utils.translation import gettext_lazy as _
except ImportError:
    _ = lambda s: s
...
...
LANGUAGES = [
    ("en", _("English")),
    ("ja", _("Japanese")),
]

This however only happened when we run tests, it didn't happened in normal django run. Adding some breakpoint() in configure() function proved that it never get called in normal django run (such as through runserver).

My initial gut feeling tell that this is a problem with pytest_djangoapp but after I did some digging, my thinking a bit skewed.

Firstly I looked into the commit that introduced this check (not allowed _ as settings attribute). You can find it using blame on github.

So it brought me to issue #30234 on django project issue tracker. At this point I thought Django is at fault as the localization docs still showing example of using _ in settings.py. It doesn't make sense, if you already disallowed using _ in settings, why still show it in official docs?

Fortunately some wise guy on the issue's comment replied to my comment that this has nothing to do with Django or its docs. The problem is with pytest_djangoapp which called settings.configure() and passing _ as part of the settings dict.

I can't tell how embarrassed I am. I should have check pytest_djangoapp first before posting the comment in the issue tracker.