Lessons Learned Building a Full-stack Framework for Django

The idea for django-unicorn all started innocently enough with a tweet on July 8, 2020.

After jealously watching demos of Phoenix's LiveView, I built a prototype of a real-time monitoring dashboard for Django with websockets and Alpine.js. After a past side-project got a little derailed (read: became not fun) using Django, Vue.js, and GraphQL, the simplicity of Alpine's model struck a nice middle-ground.

Then, I noticed the author's Livewire project. Even though it was for the PHP web framework Laravel it sounded intriguing and I was immediately smitten with the documentation site. The thought of simplifying web development by enabling server-side code to be "called" from the front-end was appealing. Instead of building a suite of APIs, mapping data models to their REST representation, and switching languages to build a single-page app in Javascript, Livewire leverages backend code and provides the glue for the frontend to interact with. This fulfilled a need I saw all over the place -- it certainly isn't ideal for every application, but probably useful for 80% of the websites out there.

After watching the available screencasts I really wanted to at least prototype a project with it. But... not enough to switch away from my typical tech stack of Python and Django. Laravel looks nice, but I am pretty invested in the Python ecosystem. So, I tweeted complaining that Django didn't have a similar library and my friend, Michele, then replied with the magic question: "why don't you make it yourself".

I spent the next 3 days re-watching the Livewire screencasts pretty intently to see the "shape" of the JSON request and response messages, scouring the documentation site, and reading through the Livewire Javascript code to understand how it worked. My first push to Github was July 11th -- three days after that first tweet.

I remember how magical it felt to type into a textbox and then have Django render it in almost realtime as a regular Django template variable. Since Django's unofficial mascot is a pony, django-unicorn seemed like a fitting enough name for this little library I was starting to mildly obsess about.

There have been a lot of learnings over the last five months. I'll cover a few related to Python, Javascript, and then some general thoughts now that django-unicorn has grown up a little bit (version 0.11.0 was just released).

Python

Python has been my preferred programming language for the past 8 years or so, and Django has been my steady go-to web framework. There might be some flashier web frameworks around, but for the raw speed I can go from idea to database tables to server-rendered HTML I wouldn't pick anything else.

importlib

importlib.import_module is the mechanism to dynamically import Python modules. django-unicorn uses this functionality to be able to find and load the component based on the component name's string representation that is specified in the template.

{% unicorn 'hello-world' %}

The hello-world component name is converted to a module name of hello_world.py and class name of HelloWorldView. Then different Django apps are searched to find the correct module (defaults to unicorn.components.hello_world.py). Once the whole string is created, import_module is called to retrieve the correct component.

inspect

Python contains a wealth of information about the code that is running... if you know where to look. The inspect module provides a wealth of information about classes and its methods which I use to inspect for publicly available methods and fields to include in the Django template context.

literal_eval

django-unicorn supports calling methods from the frontend with Python objects as arguments.

<div u:model="dictionary">
    dictionary.name: {{ dictionary.name }}<br />
    <button u:click='set_dictionary({"name": 1, "nested": {"name": 2}})'>set dictionary</button>
</div>

The method arguments look like Python, but are actually strings because all interactions are via JSON. The argument in set_dictionary({"name": 1, "nested": {"name": 2}}) needs to be parsed. Originally, I built a simple parser to convert strings into Python objects, but then stumbled onto literal_eval which "can be used for safely evaluating strings containing Python values from untrusted sources without the need to parse the values oneself." It "may only consist of the following Python literal structures: strings, bytes, numbers, tuples, lists, dicts, sets, booleans, and None", but I end up manually handling datetime and UUID as well. Much safer than calling eval() and more sane than trying to handle all the cases yourself.

lru_cache

The standard library provides the lru_cache decorator that saves the results from up to the maxsize function calls. Once the maxsize+1 unique function argument is called, the cache evicts the first object that got pushed into it. cachetools provides similar functionality as a class so it can be used without the function decorator.

LRU caches are used in django-unicorn to prevent re-finding and re-constructing component classes, and to prevent re-serializing the same data from a Python dictionary to a string representation. Both processes can be comparatively slow and tend to happen multiple times with the same inputs.

typing

Typing is relatively new to the Python ecosystem (introduced with PEP 484), but I find them to be a useful addition, especially with mypy and an editor that understands the type annotations (personally, I have been pretty happy with VS Code and pylance).

Optional type annotations are only designed to help developers understand the code (they aren't used by the runtime for optimization -- at least not yet), but even so they have been useful for "future me" to better understand the context for my code. Coming from a previous static language, C#, I appreciate types to a certain degree, but I find this middle-ground to be particularly useful -- I have the freedom to prototype without a rigid type system in place, but as the design solidifies I tend to add in appropriate types where they might be useful.

Other helpful Python third-party packages

• orjson: chosen because 1) it appears to be one of the fastest JSON serializers for Python, and 2) it provides library support for serializing more datatypes than the out-of-the-box json library (plus, it provides a hook to "dump" other types as needed)
• shortuuid: used to create a unique identifier for components. The potential for possible collision is acceptable because of the limited potential number of components
• wrapt: decorators are easy to create, but deceptively hard to make correct and wrapt handles all the hard parts
• beautifulsoup4: sanely parse HTML without tearing your hair out

Javascript

I have primarily been a backend developer for most of my career and, besides a few side projects written in the early years of Node.js, I haven't worked substantially in Javascript besides adding little features here or there. Yet, as Michael Abrahamsen writes in his post about Flask-Meld, "...here I am, writing a whole lot of JavaScript so that I can write less JavaScript. I am an engineer, after all." It's a funny thought and I wouldn't say that I'm a particularly great Javascript developer, but I have learned a ton over the past 5 months about the DOM and the more "modern" Javascript ecosystem.

ES6

I'm not ashamed to say it: for a long time I didn't "get" the reasons to use anything other than ES5 Javascript on the browser for a long time. I didn't understand why I needed classes when I could do the limited DOM interactions I needed with prototype inheritance and functions. I also chafed at what I assumed was the requirement to use a transpiler like babel for what seemed like such minimal benefits.

In fact, when I first started django-unicorn it was all written in ES5 with lots and lots of unwieldy functions everywhere. Over time, it became really hard to follow the flow of code between all of the functions and I coudn't organize the code into understandable parts. Maybe it's just the "object oriented" mindset that has been drilled into me over the years, but I found Javascript modules and the ability to use class to be extremely useful to organize the code.

babel and rollup

Since the modern browser support is almost universal for the ES6 features I use (95%+) I can develop using ES6 Javascript and only transpile to ES5 when generating the minified version that I ship in the library. Originally, I just fought with babel, but quickly afterwards I looked for something easy to configure that could also minify the separate Javascript files into one file.

After looking at Livewire code again, I realized they use rollup which looked like it would fit the bill. There was quite a bit of fiddling and reading about IIFE to understand what I wanted to do, but now the build process is quick and painless.

ESBuild (potentially)

I also investigated esbuild because of the promise of even quicker build times. There is a ESBuild PR that seems to work as expected. It even creates a slightly smaller file size than rollup. However, there doesn't seem to be a way to integrate babel into the process and I am not quite ready to give up on ES5 for users on really old browsers. At some point, that trade-off will probably shift I expect, though.

ava and jsdom

I wanted a Javascript unit test framework that was fast and low ceremony (I think I see a recurring pattern) and ava seemed the best option. ava has been working great so far and fits my approach well.

One issue with testing Javascript is abstracting away the DOM so you don't end up needing functional tests like selenium (although I spent some time with web-test-runner and playwright and they were impressively fast to spin up). However, jsdom allows my tests to have enough of a DOM to test interactions like click events without requiring an actual web browser running. I did have to add hooks so that certain parts of my Javascript could use the jsdom DOM instead of the browser's DOM, but after that was added it seems to work well.

morphdom

Part of the magic of django-unicorn is how the DOM gets updated. That only works reasonably well because of the work of morphdom. A super impressive library and also a core part of Livewire, as well.

In general

Creating an open-source library isn't all about the code (as much as I really wish it was). I learned a few things that weren't related to either Python or Javascript, but about the entire process.

Start small

django-unicorn started as a germ of an idea, "Livewire for Django", but that was a daunting task. However, I knew what I considered the core functionality the library should have and could "see" a way to make it happen from the beginning:

• custom Django template tag that finds a component class and instantiates it
• expose all public fields on a component class to the Django template context
• Django view method that accepts a JSON object with a defined API
• Javascript that listens to events, converts them into a JSON object, and calls the Django view endpoint
• Django pieces to wrap everything together into an app that could be installed

The overall goal was overwhelming, but my first commit was relatively simple. I started with the basic functionality and iterated to add more and more functionality over time.

Breaking up a large project into smaller, achievable pieces is the best (or maybe only?) way I know of to build daunting, complicated software.

Everything needs marketing

As much as open-source is lauded as some idealized meritocracy, it really isn't. Building something cool and just waiting for people to find it is an exercise in frustration. I think Caleb Porzio does a great job of this with Livewire. Between "working in public" on Twitter, conference talks, interviews, and podcasts, it's obvious he understands how important marketing is for his numerous projects.

The whole Livewire website is also marketing. The first page is basically a landing page "selling" the library and why you should use it. The API documentation is clear and concise and the coup de grâce is the screencasts. It's clear that he understands that different people learn in different ways: some want detailed documentation, some want tutorials, some want a visual of how the library works. The screencasts also subtly counter some of the developer push-back about this approach. It's all brilliant marketing.

I knew for django-unicorn to be even moderately succesful it would need more than a GitHub readme with a GIF. Pretty early on I created a stand-alone documentation site with an initial landing page and comprehensive documentation with example code. It is also important to have actual components that developers can interact with and see just how well they work. I'm definitely not a designer and would love help to make the documentation site better, but having a standalone site seems key to encouraging more users to try django-unicorn.

Just showing up

For better or worse writing code is my day job and my hobby. django-unicorn incrementally gets better over time because of the time I spend on it. Some weeks that might be very limited, but the average is probably 10 hours a week. Each week I slowly add new features, improve unit tests, tweak site copy, respond to GitHub issues, and improve the documentation. It only works because I enjoy all parts of the process. Like most hobbies, showing up and slowly improving a project is how to build something great.

It also helps to have friends who nudge you to create a fix for a problem instead of just mindlessly complaining about it on Twitter!

Thanks for reading this far and I hope some of my lessons were useful on your journey! If you are interested in a full-stack framework for Django, please check out https://www.django-unicorn.com and consider sponsoring me on GitHub. 🦄

Cover image from photo by De'Andre Bush

Comments

[tombohub]

is this working? says web server is down

[Adam Hill]

Which link are you referring to?

[tombohub]

django-unicorn.com/

[Adam Hill]

Thanks, I just added a redirect to handle that.

[Ahmed Shahwan]

Great job! Thanks Adam for introducing livewire into django. I look forward to seeing your work flourish and be able to participate.