Lesson learned while working on my pet project for 8 years

Let me tell you a story about my pet project, a project I have been working on in nearly a decade. This project has taught me how to code, how to contribute to open-source projects, and taught me a lot about software in general.

This is a story about Episodehunter. The title of this post could as well have been "How to build a trakt.tv clone" or "How to migrate a PHP monolith into AWS lambda functions in typescript" or "A brief history of my pet project" but in the post, I will just focus on the lessons that I have learned under my last years working on my this project.

But first off, what is Episodehunter? – The idea is quite easy. Let's say you are following some different tv shows and you want to know which episodes you can watch tonight. How do you do? How do you know when the new season of your favorite tv show starts? How do you keep track of what tv show you have watched? Episodehunter will solve that issue. It will automatically track what you are watching if you are using Kodi or Plex, and you can get a history of what you have watched. If you ever have used trakt.tv, this is the same idea but with a different touch (I did not know about trakt.tv when I started developing Episodehunter).

I will not focus too much on the product itself in this post, but rather on what I have learned in the past few years. More particularly, my lessons learned.

Lesson one: Use PHP and all other shitty tools

Sorry, let me rephrase that; use PHP and all other tools that let you move fast. You can say a lot about PHP, for instance, why there is a mysql_real_escape_string and a mysql_escape_string function or why ?: is left-associative or a lot of other crazy things. But the thing is that it is straightforward to learn and easy to use. Especially with tools like Laravel. You can mix front end code with backend code in a single file and just hit refresh in the browser, and you will immediately get the result in front of you.
– Did the backend crash? It doesn't matter, just reload the page and the PHP file will recompile, and everything is back to normal, you do not need to restart anything.

The community is huge! and you will get an answer from stack overflow for almost everything. There is, of course, alternatives but sometimes it is good to take one step back and think one more time if you should start with the newest and coolest language/framework on the block, or if you just should pick the old, boring stack that you know you will move fast in, and once you have a working product to can start migrating to the new cool language/framework.

This is how I did it. I started with a simple PHP webpage and hosted it on a cheap web hosting company. I learned as much as I could about the language (PHP) and other web technologies like HTTP, REST, HTML, JavaScript. Ooo, about JavaScript; I hated it! I thought it was the worst language ever created and that all bugs that I shipped never were my mistake; no, it has to be Javascript false. It turned out that I was terrible at programming and did not understand the beauty of JavaScript. We will get back to this and why I was wrong.
When I had outgrown the hosting company, I moved my PHP website to a VPS server, and I was continually learning new things, and I made a ton of mistakes (like deleting a whole database or locked my self out from my own servers, so I had to create a new one. I have learned so much on this journey and I don't think it should have been possible if I had chosen the other language to start with, like C#, Java.

Lesson two: Use a monolith

Always start with a monolith; it will be times when micro/nano-services are more beneficial, but you will know when it is time to migrate. If you are not sure, use a monolith. Everything will be harder and probably slower with microservices (unless you have a big team around you). However, it will be exceptions, as always.

With Episodehunter, I started out with a monolith web app. One codebase, one API, one everything. And this was great at the beginning but as my user base started to grow, my security concerns did the same. Even though I had implemented authorization, it did not feel safe that the end-user had access to the same API endpoint as I had for internal/administrative work. Just to give you some example: it could be to update show information, get static reports, delete users, etc.
I started to separate the web app with the "real" backend. I created a new PHP app that only had a cli API, so the only way of interacting with the new service was through the terminal, on the machine. And I think this was a perfect example when it is a good idea to separate your app (separation of concerns on a higher level). With this separation, you could also create new users to your database/external services with different access.

This separation was so beautiful, so I just wanted to continue to split the service into smaller parts. I created a new service that only was responsible for the images (it is quite a lot of images that need to be resized and cached). Then I separated the REST API into a separate application (still in PHP) since it started to grow with a lot of logic to handle the clients (xbmc (now Kodi), web, and a native app).

This was late 2014, and Nodejs started to be a thing for real, so I was thinking; – maybe it's time to move to another language and create even more microservices. Perhaps I can create a message broker between the services and have a really good log service that tracks and analyze everything. I should also have a discovery service that automatically could detect new services and request new ones and when I'm at it, I should rewrite the front end, maybe to Angular 2, it seems to be the next big thing (it was in Alpha at the time). So I started and worked night and day to get something to work, and I learned a lot, but I never got finished. Why? Because it was to much work do be done for one person. The stack was awesome, and I think the architecture was fantastic if I was a team of 20 persons, so I went back to the "boring" technics and my old codebase. The real lesson here is to keep it really simple and dumb. Use a monolith.

Lesson three: Let somebody else handle that you don't like

I had my server at digitalocean, which is a fantastic service. It is cheap and straightforward to get you started, but it comes with another cost: you have to keep the server secure and up to date. Let me quickly go through some mistake I did.

• I wanted to upgrade the PHP version, but what I did not see was that it also required to update Apache, which I used in front of the web app. After the upgrade, all my Apache config files were not valid anymore, and it took mine like a day before a got it to work again. Lesson: always back up your server before doing a more significant upgrade. Digitalocean supports snapshot, which is easy and simple to use. Use them.

• I wanted to remove a folder that I had created outside my home directory that I didn't own. So I ran sudo rm -r / path/to/the/folder. Do you see my mistake? I was supposed to write /path/to/the/folder, but I did not; instead, I had started to remove all files on the server. Don't ask me how or why, but it happened. At first, I did not understand why the command took so long time. I cancel the process, and the server started to feel weird. I got a lot of weird errors that I did not understand, so I tried to log off and in again, but I was not able to log in again, ever. It wasn't until I scroll up in my terminal to see what I did that I understood what I had done. Lesson: do not do anything on the production server when you are tired. Use a tool that asks you if you are sure before deleting files (rm -i or trash or something else). Try to avoid sudo.

• So I wanted to install global node packages with npm without running it as root. So I just wanted to change the owner of the default npm folder, so let's do that: sudo chown -R user /usr/lib/npm_modules. That did not work, and at some point, I accidentally run sudo chown -R user /usr and after that, I could not use sudo anymore since I had changed the ownership of the sudo command and I could not change it back because I did not have access. Lesson: don't ever change any ownership on files outside your home directory if you are not entirely sure of what you are doing.

I made many more mistakes, and I also started to ignore some updates since I didn't want to spend a weekend on upgrading configuration files or wanted to read about all the security patches or how to config different security tools. I just wanted to write code and create an excellent product. So I decided to move all my services to a PaaS. Something I never regret.

– Can you use a PaaS? Do it! It's worth it (if you value your time).

I looked around and tried out Heroku, now, GCP, but landed at last at AWS.

Lesson three: Go serverless (FaaS)

I knew that I needed to rewrite the app, and I did not want to use PHP anymore. It has been great but it was not suitable for my use case anymore. At this point I needed:

• I flexible way of handling jobs. I need to continuously update the tv show information, which will build up a queue of tasks and I need a simple approach to managing this queue in an error-tolerant way.
• It has to bee cheap. I'm a single developer on this project and I do not earn any money at all. I do not sell any data, and I do not use any advertisement. This is a complete loss business from my part.
• I need a type safety language that I could move fast in.

After some experiments, I decided to try out AWS lambda function with typescript (and with serverless).

I hear you thinking, "Didn't you just say 1. Use simple tech stack 2. Javascript is the worst language ever created?". Yes, I did. Let me explain.

FaaS (function as a service) is quite an easy concept. You have a function that takes an input and gives back an output. You can call this function with an HTTP request, message broker, cli, or however you want. And the best part is that you only pay for the time it is executed, so if no one is calling your function, you don't pay anything (ish). I can get almost the same quick feedback loop as I did with PHP, but this time I can use a static language and microservices by default.

Regarding JavaScript/TypeScript: I think people who hate on JavaScript have not used it since ES2015 come out. When we got a real module system, block-scoped variables and much more, or they just don't understand the community around it. I did hate JavaScript but that was because I did not understand it. Once I learned it, I loved it, mostly because I can do anything with it: backend, frontend, desktop apps, mobile apps, home automation and much more. I do, however, hope that WebAssembly will be its successor (on all platforms).

Back to serverless (or FaaS): I would lie if I told you it was an easy setup and that I had all the answers before I start using it. It has been a bumpy ride, but in retrospect, I do not regret it. I do, however, not recommend all to jump on this FaaS-train because it will definitely not cover all use cases.

I have 13 different functions. Most of them are quite small but some of them are larger. I have for example one function that is the layer between the data store (database) and the consumer of the data. It exposes a GraphQL layer that all other services are talking to. You can see the overarching architect below. I could easily write a blog post about every service/function, and maybe I will if there is an interest.

Lesson learned: use FaaS but start small. FaaS can be an excellent platform that enables a lot, but it could also be super expensive and super hard to use.

Lesson four: Go open source

If you don't have anything to hide, show it to the public. This will make it easier for people to contribute, other people can learn from you, and it is easier to give something back to the community. And maybe most impotently: this will force you never to commit any secrets.

All backend services can be found here: https://github.com/episodehunter/episodehunter
And the web frontend is placed here: https://github.com/episodehunter/web

Next lesson:

I'm continually learning new things, and Episodehunter has been my playground for testing new ideas. The ideas I want to try out right now is mostly on the front end.

• It is quite common to put a lot of logic in a UI component. Even with libraries like moxb or redux, the business logic is close to the component and is running in the main thread. Should it not be easy to just move it to a web worker instead? The benefits would be a better separation of concerns, less work on the UI thread, and more natural code structure. I also believe that this will make it easier to make the application work offline.
• Push notifications. I know that iOS still doesn't support push notifications by a service worker, but everyone else should not be held back for Apple's decisions.
• Stop using webpack and use snowpack/pika instead. Is it possible without a performance hit?
• Background sync? Maybe?

This post was just an over glans of Episodehunter. I would love to go more in-depth on a specific topic if there is an interest. Like how to create a GraphQL API with lambda functions or why Firebase does not scale (in some use case) or how to use GraphQL types with eslint or how to create an image proxy in a lambda function or something else. Or if you want to join my journey, I will more than happily introduce you to Episodehunter and help you to set it up on your local computer.

The most important lesson that I want to share with you is: Do whatever you want to do. Be creative and learn, and the joy will come automatically.