DEV Community: Daan Debie

Static Duck Typing in Python with Protocols

Daan Debie — Sat, 27 Nov 2021 16:55:23 +0000

At Source, we write most of our code in Python. It's a language that both our Software Engineers and Data Scientists are equally at home in. It's easy to be productive in Python, in part due to its dynamic nature. Not having to think too much about the types of your variables and functions, can make it easier to experiment, especially if you're not entirely clear yet on how you're going to solve a particular problem.

When moving our code to production however, we want to have more guarantees about the behaviour of our code. Writing (unit) tests is one way to get those guarantees, but we also make heavy use of type hints to give us more confidence in our code. Type hints can also provide a productivity boost, because not only humans can reason better about type hinted code, your editor can as well!

Sometimes though, using type hints everywhere can feel like you're losing out on a lot of the magic and speed that a dynamic type system brings you. One particular trait of dynamic typing that is pretty idiomatic in Python, is duck typing.

Duck Typing

Duck typing is a philosophy in programming where you care more about the behaviour and properties of an object than its stated type to determine if that object is useful in a certain situation. Duck typing is inspired by the duck test:

If it walks like a duck and it quacks like a duck, then it must be a duck

In practice this means that when you write a function that receives a certain input, you care only about the behaviour and/or attributes of that input, not the explicit type of that input.

One interesting question that arises is: if you don't want to be strict about the type of the parameters a function receives, are there still any static type guarantees to be had?

And the other way around is interesting as well: if you have a function with statically typed inputs, can you loosen up those parameters to make the function more universally useful, the way duck typing does?

As it turns out, Python provides a neat way to have our cake and eat it too!

Protocols to the Rescue

When reviewing some code recently, I came across a function that looked roughly like this:

def calculate_windowed_avg(
        measurements: Union[List[TemperatureMeasurement], List[HumidityMeasurement]],
        window_size: timedelta,
        field_name: str
    ) -> Dict[datetime, float]:
    window_upper_bound = measurements[0].timestamp + window_size
    current_window = []
    window_averages = OrderedDict()
    for m in measurements:
        # various calculations happen here
        # based on the timestamp of each measurement
        ...
    return window_averages

The goal of this function is to calculate the average of a certain field (identified by field_name) in a rolling window. At the time of writing this function, we were using it for TemperatureMeasurement and HumidityMeasurement, but it is very likely we'll want to use it for different types of measurements in the future.

If we look closely at how the function uses the input, it turns out that the only thing we want to be guaranteed off, is that the items we pass into the function have a timestamp field. So instead of specifying each different type that has adheres to this contract, we'd like to tell the type checker that we only care about having a timestamp field to work with.

Protocol from the typing module lets us do that. Just like with duck typing, Protocols let you specify the behaviour or attributes you expect, without caring about the type. Here is what that looks like:

from typing import Protocol, List, Dict
from datetime import datetime

class MeasurementLike(Protocol):
    timestamp: datetime

def calculate_windowed_avg(
        measurements: List[MeasurementLike],
        window_size: timedelta,
        field_name: str
    ) -> Dict[datetime, float]:
    window_upper_bound = measurements[0].timestamp + window_size
    ...

Now the type checker doesn't know exactly what the type is of whatever is provided as measurements but it does know what those items have a timestamp field because they adhere to the MeasurementLike Protocol.

In a sense, a Protocol acts like one side of an Interface as we know it from Java or Typescript. Instead of having to specify the behaviour and properties both on a type and on the functions that use it, we only have to specify it on a function, without caring about the types of the objects that are provided to the function.

Protocol and Generics

You can also use Protocols together with TypeVar for even more generic functions that are still type checked to some extend. One use-case that comes to mind, is when you don't care about the input type to a function, as long as it follows a protocol, but you also want to guarantee that the output of the function is of the same type as the input, no matter what the exact type is.

This works as follows:

from typing import Protocol, TypeVar
from datetime import datetime

class MeasurementLike(Protocol):
    timestamp: datetime

M = TypeVar('M', bound=MeasurementLike)

def measurement_as_timezone(measurement: M, tz: tzinfo) -> M:
    measurement.timestamp = measurement.timestamp.astimezone(tz)
    return measurement

Here we create a function that takes any object that has a timestamp field and guarantees that the output will be of the same type as the input.

Conclusion

Protocols in Python provide a nice way to use duck typing while still having some static type guarantees. You can define contracts for your functions without caring too much about the actual types of your inputs.

Update on 2021-11-23: There was a wrong type annotation in this article, as pointed out by ragebol on Hacker News, which is now fixed

How to Add An RSS Feed to a NextJS Blog

Daan Debie — Sun, 17 Oct 2021 17:45:41 +0000

Today, I released a new version of my website with only one new feature: syndication through RSS, Atom and JSON Feed. Even though by some accounts, RSS seems to be dead, I strongly believe RSS is an important feature in the fight to keep ownership over your own content while also increasing exposure. The way I approach publishing and sharing my content, is called POSSE:

Publish (on your) Own Site, Syndicate Elsewhere.

RSS is an important part of that strategy.

Luckily, I'm not the only one who values RSS. There are others

In this article I want to share how I implemented syndication feeds in my NextJS-powered website.

Options and Requirements

I did some Googling to see how other NextJS users were generating RSS feeds, and there turned out to be many people that had solved this particular problem and wrote about it I found a couple of different approaches to generating and rendering RSS feeds in NextJS:

Generating feeds:

Hand-build the required XML using templated strings 🤢
Use a library to do it for you. The most popular library for this in the JS ecosystem seems to be feed 🤩

Rendering feeds:

Use Server-side rendering

Advantages:

Create dynamic feeds (more on that later) that can be different for each visitor
Using NextJS pages to represent feeds feels natural in the NextJS way of doing things

Disadvantages:

You can't generate the website statically anymore (at least parts of it remain dynamic), which reduces performance As a side note: from a development perspective, server-side rendered pages and statically generatedpages in NextJS are so similar that the difference doesn't matter

Use Static generation

Advantages:

Because the feeds are generated at build time, the site remains snappy

Disadvantages:

You cannot set the Content-Type header for statically generated pages, so you can't serve those pages as application/rss+xml. I'm not how big of a problem this is and what black magic Vercel applies when serving my NextJS site

Picking the requirements

After looking at the options, I decided on the following requirements for my feeds:

I want to go for statically generated feeds, to keep my site fast and the implementation simple
Support for RSS, Atom and JSON Feed
I want to include the complete blog post contents in the feeds. This is an important one because I personally really hate it when my RSS reader only shows me a summary of a post and I have to go the the website to read all of it. Caveat though: my website is built using MDX, so I might include components in the future that are not easily convertible to static HTML without Javascript enabled. In that case, readers will have to click through to my site.

Implementation

As per my requirements, I wanted to generate the feeds at build time. But as mentioned before, NextJS doesn't support setting the Content-Type header for statically generated pages. The alternative that many people use, is to have a separate script generate your feeds and writing them to the public folder where all other static assets such as images are stored as well. That way, the feeds would be served as static assets instead of statically generated pages -- which, from the browsers perspective doesn't make a difference!

I found a good explanation by Ashlee Boyer of this technique.

My plan:

Write script to generate feeds, using the feed library from NPM
Run this script as a postbuild step so it would always be invoked after building the site using npm run build (this happens not only locally, but also when Vercel deploys my site)

Problem 1: running Typescript modules is hard

I immediately hit a snag with (1), because I couldn't manage to use ES6/Typescript modules in a script run outside of my normal website code.

I'm using Typescript, and apparently ts-node, the tool to run Typescript files, doesn't support modules. Writing the script in Javascript wasn't really an option for me because I wanted to reuse a lot of logic that I already wrote for reading and parsing MDX files in the website itself.

Solution

I decided to follow the route that Ashlee Boyer suggests in her blog post and sneak in the function to generate my feeds as a "stowaway" in the getStaticProps function of my index page. This works beautifully!

export const getStaticProps: GetStaticProps = () => {
    generateMainFeeds();
    const lastPosts = getAllPostsFrontMatter('blog', 3);
    return {
        props: {
            lastPosts,
        },
    };
};

Problem 2: including the full content in the feeds

The code of my website already supported translating MDX files into valid JSX to be rendered by React. But how to generate valid HTML from that content and include it in the feeds?

Solution

I couldn't find many examples of this, but did find out about ReactDOMServer.renderToStaticMarkup. This function will take a bunch of React components and render them into HTML. This is what is used by many React server-side rendering solutions (maybe also by NextJS?) and works perfectly here as well.

One caveat: if your content contains internal links -- which are often relative links -- then you have to be mindful that those relative links are meaningless in the context of an RSS feed. The way I solved this is by doing some regex-based replacements on the generated HTML.

The complete content generation part looks like this:

const url = `${baseUrl}/blog/${post.frontMatter.slug}`;
const htmlContent = ReactDOMServer.renderToStaticMarkup(
    <ChakraProvider resetCSS theme={theme}>
        <MDXRemote {...post.mdxSource} components={MDXComponents} />
    </ChakraProvider>
    .replace(/href="\/#/g, `href="${url}#`)
    .replace(/href="\//g, `href="${baseUrl}/`)
    .replace(/src="\//g, `src="${baseUrl}/`);
)

Problem 3: getting rid of style information

My site uses Chakra UI for theming, which uses Emotion -- a CSS-in-JS library -- under the hood. Emotion will happily render tons of <style> tags when statically generating HTML from your React components. For most use cases where you render React on the server (statically or not), this is desirable. In the case of RSS/Atom feeds, this is pretty useless.

Solution

The solution here is to strip all the <script> and <style> tags from the generated HTML. Rather than summoning The One whose Name cannot be expressed in the Basic Multilingual Plane by trying to use regex here, I found this library to help me with this task:

const cleanHtmlContent = stripHtml(htmlContent, {
    onlyStripTags: ['script', 'style'],
    stripTogetherWithTheirContents: ['script', 'style'],
}).result;

The end result

I now have serve RSS, Atom and a JSON Feed for your reading pleasure. Most of the relevant code can be found here

Future plans

At some point I want to diversify my writing output by not only writing about tech. And even within the topic of tech there are many sub-topics I could write about, not all of which are equally interesting to every reader (all 5 of them, including my mom 👩‍👦). I'm planning to introduce tags to allow filtering content once I have enough of it.

Once I have tags, I would like to start supporting dynamic feeds so readers can subscribe only to the stuff they actually want to read. I imagine building an endpoint like this:

/feeds/by-tags.xml?tags=tag1,tag2

I'm curious if others are doing this!

Use Tools that Suit You and the Problem

Daan Debie — Sun, 12 Sep 2021 18:48:29 +0000

A couple of weeks ago someone shared an article on Hacker News that discussed the drawbacks of using an IDE for programming. Needless to say, when you touch people's tools, you can be sure to spark a hefty discussion.

(To be fair, the author did clarify that they're not against IDE's in general and they actually use one on a daily basis.)

When reading the discussion, one of the first things I wondered is if there are places on the internet where carpenters or plumbers furiously criticize each other's choice of tools. But even that has a place on Hacker News

In this post I want to explore the fundamental problems with this type of discussion and how you can embrace freedom of choice.

It's all about context

When discussing someone else's choice of tools -- in this case the choice of editor or IDE -- it's good to be mindful of what we don't know about this person and their reasons for choosing certain tools. Aside from personal preference and/or familiarity, there can be lots of extra factors to consider:

We don't know what kind of problems they're working on
We don't know the context in which they work (e.g. security requirements laid down by the company they work for, available budget)
We don't know their level of experience

I think it's good form to voice an opinion only when we can see the whole picture. I also think that people should be encouraged to choose the right tool for the job. And to put it simply: the right tool for the job is the one that fits the problem and makes you the most productive.

It's also about definition

Another issue with this kind of discussion is that everyone has a different idea of what really constitutes an "IDE". We're inclined to frame the choice of editor as a binary choice -- "pure text editor" vs "IDE", almost like a false dichotomy -- while it really is a spectrum. On this spectrum you'll not only find different editors, but even different configurations of the same editor with varying degrees of "smartness" applied through plugins.

So when someone says "why use an IDE when you can do everything an IDE does in VIM if you install the following 20 plugins", is this really so different from an actual IDE (that was branded as such)?

Philosophies in tool choices

When looking at how people setup their development environments, I basically see 2 different approaches:

The "additive" approach: this basically follows the UNIX philosophy of composing many different tools that each do one thing well. It often involves a terminal-based editor like VIM, optionally with some plugins and supported by external tools like grep, sed and awk that can help with refactoring code. You basically stack tools until you have an "integrated" development environment
The "subtractive" approach: you take one "big" editor as the basis of your stack. This is often an editor that is branded as an IDE (e.g. Eclipse, Jetbrains editors). Then you take away/replace ("subtract") the parts that work better through dedicated tools (according to personal preference of course)

To me, both approaches have their merits and uses!

How to create freedom of choice?

I think the most important question around editors is not

How to choose the right editor?

but rather

How to enable each member of my team to choose the editor that makes them most productive, while enabling effective collaboration?

Here's my recipe to do that:

My basic starting point in teams: everyone can use the editor/IDE of their choice. To enable effective collaboration, the following rules must be followed:

Editor configuration is never checked into version control, unless it's generic and applicable across editors. Examples of generic editor configuration are: line endings, tabs vs spaces, max line length etc. There is some overlap with linters here
Enforce common code style through external tools (i.e. linters, code formatters)
Everyone should be able to build and run a project, regardless of the choice of editor, preferably through the terminal. This means that we'll always prefer external tools for this. Many languages provide tooling for this:
- Java → Maven/Gradle
- JS/TS → NPM
- Go → Go
- Rust → Cargo
- ...
Additionally, building and running code can be made easier and better reproducible by using Makefiles and/or Docker (combined with Docker Compose)

I have to admit that in some ways I also limit choice within my teams: building and running projects often involves shell scripts which are portable between Linux and MacOS, but not to Windows. This can be solved by using WSL on Windows.

This is my setup

My personal setup involves quite a lot of tools.

For any "serious" development work (i.e. long stretches of coding and big projects) I take the aforementioned "subtractive" approach. My basis is one of the Jetbrains tools, depending on the language(s) I'm working with. I find the Jetbrains tools make me really productive and their All products pack offers great value for money, especially with their huge loyalty discounts after 1 or more years.

I still use the terminal and external tools for many things in this case. I put quite some time in improving and maintaining my dotfiles as well.

I prefer to use the terminal for: building, linting, running tests, using git
I prefer to use the IDE for: writing and editing code, refactoring, searching for code, find/replace through regex

For quick and short edits and viewing files where speed is preferred and conveniences like syntax highlighting and intellisense are not important, I use Sublime Text or VIM. Example of this: editing my dotfiles.

Sometimes, but rarely, I use VSCodium for editing or viewing files which are not part of a project and which I want to reformat/reindent. This doesn't work well for me in Sublime Text. For example: viewing/editing minified JSON files.

Conclusion

To wrap this post up, I just want to reiterate: use whatever tools make you productive and help you solve the problems you're trying to solve and please let others do the same.

And here's some additional closing advice: should you ever grow unhappy with the tools you're using, just write a blog post extolling the virtues of your current tools. You can be sure people on the internet will tell you how much better their tools are, thereby providing you with potential alternatives to your current setup 😇

Thanks to Dan for proofreading this

The Pitfalls of Deploying a NextJS Frontend on AWS Amplify

Daan Debie — Sun, 29 Aug 2021 11:06:14 +0000

Have you ever gone above and beyond to give a product a fair chance, only giving up way past your own sanity and patience? I did. The product was AWS Amplify

Let's build something with NextJS

When I started developing the primary (web) UI for the product we're building at Source, I quickly decided to use the NextJS framework as the basis for the frontend. One thing I really like about NextJS is that it does filesystem-based routing. Even though I like to diss on PHP as much as the next person, this particular feature of NextJS reminds me a bit of PHP in a good way.

Another very compelling feature of NextJS, is the ability to have different rendering strategies for each different page. You can decide to use:

Static generation for pages and content that rarely change. These pages get pre-rendered at build time and can be aggressively cached on a CDN. These pages can of-course still be highly dynamic by using client-side data fetching methods
Server-side rendering for pages that need to return different content on each request and/or for each different user
Incremental static regeneration, which is a variation on Static generation that will cache pages for a period you specify and then regenerate the pages server-side when they're considered stale

Deployment

This particular feature requires a specific hosting setup that supports the different flavours of rendering. You basically want:

a good CDN to cache and serve your static pages
a server-side solution to serve the server-side rendered pages and facilitate incremental static regeneration. This should also take care of API routes, should you use them

The NextJS documentation mentions a couple of options for deployment:

Use Vercel. Vercel is the company behind NextJS. They offer a seamless experience where you can use any rendering method, and Vercel will deploy each part of your app in the right way. One neat thing they do is that they'll convert your server-side rendered pages and API routes into serverless unctions that they'll deploy to edge locations all over the globe.
Setup a NodeJS server somewhere for the server-side stuff and use a CDN manually to serve the static content.

At Source however, we've made a conscious decision to use AWS for all of our infrastructure needs. We're too early in the lifecycle of both our product and company to consider a hybrid cloud setup and AWS offers all the features that we need for prices that are reasonable for now. Through one of our investors we also got a boat-load of AWS credits to get started, which helps.

Amplify

I didn't want to manually set up a hybrid deployment with NodeJS and CDN - which I could probably do using Cloudfront and ECS - so I looked for alternatives. AWS Amplify claims to be the

Fastest, easiest way to build mobile and web apps that scale

And they claim to support NextJS as well as server-side rendering by virtue of Lambda@Edge

Sounds great! So how does that all work?

Amplify - the setup

The idea behind Amplify is that it lets you focus on building your app without having to worry about the infrastructure. This means that it will not only let you build and deploy your frontend easily, it will also let you hook-up backend components like authentication through Cognito, a GraphQL API using AppSync and a DataStore.

The way Amplify manages all of this, is through the idea of "backends". A backend contains one or more of the aforementioned components. You can setup different backend environments in Amplify and couple these to different (git) branches of your frontend.

Amplify - the developer experience

We are bringing our own database and API and wanted to use a pre-existing Cognito user pool. Amplify lets you import an existing Cognito user pool instead of creating one for you, and this worked fine... until I tried to create a second "backend environment" with a different Cognito user pool.

I want to allow different users on our development environment than our production environment. These environments correspond to the main branch in Git (production) and any other branches in Git (development).

How do I tie an Amplify backend environment to a frontend branch?

Interaction with Amplify is done through the web console or the
Amplify CLI. According to the documentation, we can create a new Amplify backend environment like this:

amplify env add

But nowhere is it made clear how this is linked to our frontend branch. The Amplify CLI tries to mimic the git CLI by having amplify push and amplify pull. So I assume we need to push our new environment to the cloud. Does this command depend on the git branch I'm on? I don't know.
I could just as easily overwrite my existing backend environment when using this command.

Eventually I was able to create a main environment and a dev environment, but no matter how hard I tried, I could not get those environments to use different Cognito user pools.

How do new team members setup an existing project?

The Amplify CLI will create some configuration files, some of which you need to commit to git, and some of them you need to ignore. These files should make the build reproducible. It is unclear however, what a new developer needs to do to be able to run an existing Amplify project.
The documentation mentions multiple Amplify CLI commands to setup a project:

amplify pull

amplify init

amplify configure project

amplify env checkout <my-env>

To be run in arbitrary order?

Three
similar
questions
on StackOverflow give 3 different answers on this topic.

When I was on-boarding new team-members, we basically tried some commands in different order until we were able to run the project locally for that engineer.

How do I figure out why my build failed?

In short: you don't. Or you enlist the help of AWS Support (which you will have to pay for). The Amplify web console doesn't share
detailed build logs, so you'll have to fix your problems by trial and error.

Amplify - the bugs

AWS Amplify seems to have quite some bugs. For example: when you use Gitlab, you cannot put the source code of your project in a sub-group. This will silently fail your build, with no clear error message in sight.

The aforementioned problem of using multiple Cognito user pools for different environments of one Amplify frontend, I also consider a bug.

Not having access to the actual build logs, I also consider a bug.

Amplify - no help included

The most painful issue I have with AWS Amplify - and AWS in general - is the complete lack of support. You are paying a lot of money for their services, but that does not come with any form of
support, unless you pay extra. And when you make use of that paid support, it goes something like this:

Contact AWS Business support through their website and try to explain your problem in a plain text field that you cannot resize or use formatting in
Wait far longer than the promised 4 hours
Get a response from someone telling you they'll forward it to "the team" (why not have someone respond who can actually help you with your problem?)
Get a canned response with a solution akin to "turn it off and on again"
Tell them you have an actual problem
Get forwarded to the next person who you have to clue in again on what your problem is
Go to 2 and repeat

This means that in practice, AWS Amplify is not usable unless you are willing to pay extra for support and have a lot of patience.

As a side note, I find it really frustrating that with big corporations like Amazon in general, you quickly feel helpless and ignored when you have problems. Stories about Google customer support (or lack thereof) are rampant on Hacker News, and I think Amazon is no different. You'd think that with so much revenue they should be able to set up actual proper customer support 🙄

The Pattern with AWS

I think that these problems with Amplify are endemic of a larger problem with AWS: their PaaS and SaaS solutions suck. Aside from pricing (which is quite high compared to competitors like
Digital Ocean), I think AWS' infrastructure-as-a-service offerings are fine. If you want to have a bunch of servers (EC2), a database here and there (RDS) and some serverless functions (Lambda), AWS
will serve you just fine.

But if you want something on top of that, like authentication (Cognito), or a PaaS solution like Amplify, be prepared for a sub-par offering compared to the competition. I think Amazon should
focus on strengthening their core offerings instead of offering everything and the kitchen sink.

The solution

In the end, I dropped Amplify and went with Vercel. $20/user/month is a small price to pay for actually being able to work on the product instead of fighting AWS services and support.