DEV Community: Ferdinand Mütsch

Wakapi - Open-Source Time Tracking for Devs

Ferdinand Mütsch — Tue, 15 Aug 2023 05:49:00 +0000

Hi all!

I want to share with you an open-source project that I started back during my studies and am passionately improving ever since. Back then, I wanted statistics about my coding (what language, editor, projects, etc.), just like WakaTime.com, but as a student I wouldn't be willing to pay for it. Because I'm a developer, I simply decided to build my own instead. It has become my no. 1 hobby project and after years I still love to spend my evenings coding on it.

The project has reached more than 1,700 GitHub stars now, the hosted service at wakapi.dev got 1,800 registered users today and a super friendly, small community has evolved on GitHub - things, which I am super happy and appreciative about!

Wakapi is entirely open-source and can be self-hosted. But there is also a paid subscription plan for the official service, that gives users unlimited data history and a few other small perks and covers hosting, domain, etc.

Go check it out and, if you like Wakapi, consider opting for a paid subscription. I'd appreciate your support a lot!

P.S.: Consider sponsoring open-source 😊.

How development history keeps repeating itself

Ferdinand Mütsch — Tue, 08 Nov 2022 07:09:00 +0000

I want to share a couple of thoughts that repeatedly come to my mind the more experienced I become in (web-) development. If you're observing the ecosystem thoughtfully, you can kind of see history repeat itself every couple of years - and the wheel being reinvented over and over again. Here is my personal opinion about innovation in the web world, about technologies like PHP and SOAP versus modern JavaScript frameworks like Next.js and paradigms like Serverless, about developer productivity, and more.

Innovation on the web

Innovation in the world of web development, especially in the JavaScript ecosystem, has already become a meme among developers. People joke about new frameworks and libraries popping up every day. The moment you decide to pick up the latest and greatest frontend framework, it is most likely already outdated again, because the next big thing has already climbed up to the front page of HackerNews.

This is not necessarily a bad thing! Innovation is always great to have and many of the web frameworks and libraries out there are actually of very good quality (code, but especially also documentation) in my opinion. The hard part about this, however, is to not get distracted as a developer. It can be quite overwhelming and in the end you'll find yourself suffering from JavaScript fatigue.

My take here is: keep your eyes open for what's new and shiny, but decide for one technology, commit yourself to it, become and expert with it and only switch to something else if you really see a need to.

New technologies against boredom

But why is that even? Why are there such vast amounts of technologies, all of which serve very similar purposes in the end?

I cannot give a definitive answer to this question. But I reckon one reason is around the fact that the entry barrier to writing and publishing your own web framework is quite low. The web ecosystem is already so big that you'll find resources and support on virtually any topic – and platforms like Reddit, HackerNews or ProductHunt make it easy to get people aware of your project.

Another reason could be this: in a recent podcast episode of programmier.bar I heard the statement being made that developers need new technologies to stay interested and keep up their excitement. The interviewee claims that there is a trend of (tech) companies increasingly pushing for the development of shallow features, as opposed to deep features. That is, features, which are fairly straightforward to implement and where a developer doesn't have to put a lot of thoughts and brain power into. To prevent themselves from getting bored, developers jump into new technologies / frameworks / languages or start building their own ones.

Reinventing the wheel

All of these novel technologies (can be languages, frameworks, libraries, or also concepts / paradigms) aim to solve some problem. But most of those problems have already been solved before. Inspired by this Syntax.fm episode, and by a provocative tweet about Serverless being just the same as good old PHP scripts I saw recently (can't find the link anymore :-/), I want to share some of my thoughts on that same topic.

PHP vs. X

Let's compare a few allegedly "ageing" technologies with more recent approaches, starting with PHP scripts.

Please note: some of this is intentionally written in a slightly provocative way. Please note also: I'm neither a PHP developer, nor do I have a lot of experience with the below frameworks.

Server-side rendering (SSR)

I feel like this is one of the hottest topics right now. Recent frameworks with SSR support include Next.js, Nuxt.js, SvelteKit, Remix and others. But didn't PHP do server rendering already 20 years ago? Yes, but... PHP did only server-side rendering and nothing else. The strength of tools like Nuxt is to give developers the best of both worlds. They combine the benefits of SSR with those of single-page apps (SPA), which is extremely useful for very dynamic websites. A popular concept in this context is partial hydration. So, if this was a competition, I'd definitely give this point to the side of the "modern approaches".

Mixing template and code

Some developers appreciate JSX, because a component's HTML markup and its JavaScript business logic code are very close together, with one being an intrinsic part of the other. One could argue, though, that PHP did just that already since its very beginnings, so not too much new here.

File-based routing

Some frameworks advertise their mechanism of file-based routing, i.e. not having to declare HTTP routes / paths in code, but have them inferred implicitly from the app's directory structure. However, this is not at all new, either. PHP used to do just that - you place a .php into some folder and can call it from the browser by its file path.

Serverless

The idea of Serverless is (a) to have developers not having to worry about deploying, running and scaling their app in production (or even doing server administration), (b) to reduce boilerplate code for authentication, authorization, security and the web server itself and (c) to have very small, self-contained, single-purpose functions instead of giant coupled code bases. Serverless functions are triggered through an event (usually an HTTP call), then do their job, and are shut down again afterwards - they are entirely stateless. Sounds familiar? This is exactly what PHP scripts are. They are invoked per request and only live as long as the request itself. What is the novelty here? Indeed, Serverless is very, very close to what PHP scripts have been for years already. However, to be fair, Serverless (e.g. AWS Lambda) is - due to the way it is designed - probably much more scalable than PHP scripts running inside a single web server / FPM.

RPC technologies

Another field where I found a lot of parallels between new, "fancy" technologies and old, "legacy" approaches is remote-procedure calls (RPC). RPC is an alternative approach to API design and comparable with resource-based paradigms such as REST or "query"-based approaches like GraphQL.

gRPC, tRPC, ...

The most popular framework today in this space is gRPC by Google, but there are also Cap'n'Proto, tRPC, JSON-RPC (less of a framework, rather a convention) and others. And then there used to be SOAP for a long time. It's considered outdated and deprectated today. But in essence, it was doing the exact same thing - just in a slightly different way. Of course, gRPC has a few advantages over SOAP, especially more efficient message representation, thanks to binary encoding, and communication, thanks to HTTP/2 – on the other hand, though, SOAP has discoverability, thanks to to WSDL. The point I'm trying to make is that the fundamental concepts are almost exactly the same.

Old ≠ bad

The central point I want to make with this article is: technology is not necessarily bad, just because it's old. Not even PHP (especially with PHP 8.0). Many seemingly novel concepts have already been there before at some point, and not few of them do still have their reason to exist. When a new technology emerges, it is often times just an evolution, rather than a revolution - a slight improvement or the rejuvenation of a previous technology (cf. gRPC vs. SOAP, Serverless vs. PHP scripts, ...). Of course, this is important. Technology must be adapted to modern needs. But also, at the same time, developers should be careful about not ending up reinventing the wheel once again.

Key takeaways

From this discussion, my key takaways are:

Technology is not bad, just because it's old. Don't fooled by the novelty bias.
Don't ever rant about a technology - especially not just because everyone else on Twitter does. Even dreaded things like PHP or VBA have their right to exist. You can anyways only judge about a technology, if you have actually worked with it extensively.
Focus on building features, not tools - unless you really have a need to.

Originally published at muetsch.io.

Open-source, self-hosted location tracking with OwnTracks and Grafana

Ferdinand Mütsch — Fri, 25 Jun 2021 08:20:27 +0000

Introduction

If you are using any kinds of Google services on your smartphone, chances are high that Google continuously tracks your location and updates your Timeline.

(Source)

This is a pretty cool feature and for me it is super interesting to see in retrospect which places I have visited in the past. However, it is also a bit concerning to see how precisely Google tracks your daily life. That is why I turned the location tracking off and decided to go for a rather privacy-focused approach instead.

Your Own Setup

To build your own, self-hosted location tracking system, you are going to need a bit of technical expertise, a few different open-source software components and a small server to host them.

Requirements / Components

Android- or iOS smartphone
Web server (Caddy, nginx, Apache 2, ...)
PHP >= 7.x
MySQL or MariaDB
Grafana

OwnTracks

The first step to tracking your location is to record it anywhere you go. As most people almost always carry their smartphones with them and as most smartphones have GNSS sensors, the choice is quite obvious. The only thing missing is an appropriate app. This is where OwnTracks comes to play. It is an open-source mobile app that does exactly what we need - record your location and send it to a custom configured MQTT broker or HTTP endpoint. It is available for both Android (written in Kotlin) and iOS (written in Objective-C) and comes with super detailed, developer-focused documentation.

(Source)

It comes with different tracking modes, which essentially specify how often to send your location. Weirdly enough, these modes behave differently on Android an iOS.

	Tracking on iOS	Tracking on Android
Move mode	After `x` meters (default: 100) or `t` seconds (default: 300)	Every 30 seconds
Significant changes mode	After >= 500 meters and >= 5 minutes	After >= `x` meters and >= `t` seconds
Manual mode	On user request	On user request
Quiet mode	Never	Never

In fact, things are a bit more complex than this, as you will have to distinguish between how often the app requests your location from the device's sensors and how often it sends it. Also, there are different options regarding the desired precision (e.g. GPS is more accurate, but uses more power). Moreover, on iOS, there are region monitoring and iBeacon monitoring modes in addition.

I would suggest to read through the docs and decide which modes suits you best, depending on your smartphone's OS. I, however, decided to go for the significant changes mode on Android with the following custom config variables.

{
    "locatorDisplacement": 25,
    "locatorInterval": 60,
    "locatorPriority": 2
}

This basically tells the app to request my location with block-level accuracy (100 meters) every time I move by more than 25 meters, but at most every 60 seconds. These settings work quite well for me and I did not recognize any significant impact on battery consumption.

Server-side script

After setting up the client side, a server-side component to receive the OwnTracks app's requests is still missing. OwnTracks ships with its Recorder, which is a small and simple web application written in C. However, I did not like it a lot, as it does not look particularly beautiful and is very limited regarding its functionality. I rather wanted to visualize my data in Grafana. But to get it there, it first needs to be persisted to a database.

UPDATE: I found owntracks/frontend in the meantime, which seems to be a lot more advanced than the recorder's web UI. You may want to use this as an alternative to Grafana, which will make the setup a lot easier.

A heartbeat request's payload looks like this.

{
  "_type": "location",
  "acc": 13,
  "alt": 163,
  "batt": 91,
  "bs": 1,
  "conn": "w",
  "created_at": 1624607444,
  "lat": 48.9995682,
  "lon": 8.3940805,
  "t": "u",
  "tid": "l3",
  "tst": 1624607139,
  "vac": 3,
  "vel": 0
}

I quickly wrote a little PHP script to take OwnTracks location heartbeats and write them to a MySQL database. It can live at any web server with PHP support (I am using Caddy2 with php-fpm). Assuming it is deployed under https://my.server.tld/track.php then that is the URL you need to configure as an HTTP target endpoint in the OwnTracks app. Optionally, you can configure HTTP Basic auth inside your web server's config. OwnTrack's client app has support for that built in.

Grafana dashboard

The last missing part is to actually visualize your location data. I am a big fan of Grafana, as you can easily build beautiful visualizations with low effort. Grafana integrates with MySQL as a data source, so it can read the location data previously ingested by the above script. Using the grafana-map-panel plugin, you can visualize geo data in dashboard. In addition, I added another two graphs for plotting my velocity and my phone's battery level over time. This is what it looks like in the end.

The geo data panel is generated from this underlying SQL query:

SELECT
  tst AS "time",
  lat,
  lon,
  vel
FROM recordings
WHERE
  $__unixEpochFilter(tst) AND
  user = '$user' AND
  device = '$device'
ORDER BY tst

Conclusion

This is it! You have your own, self-hosted, Google-free location timeline now. Have fun!

Originally published at muetsch.io.

Aggregating and Visualizing DMARC Reports

Ferdinand Mütsch — Fri, 07 May 2021 20:48:25 +0000

DMARC, DKIM and SPF

If you are using a custom domain for your e-mail with providers like mailbox.org or even host your own mail server, it is likely that you came across these terms at some point. All three are technologies for improving e-mail security and authentication. Their whole purpose is to ensure that a mail was actually sent by the person who appears to have sent it, i.e. the person mentioned in the mail's From header. I don't want to go too much into detail here and only explain each of them very briefly.

SPF

SPF – the Sender Policy Framework – is quite easy to understand. As the owner of a domain, say example.org, you precisely specify in the DNS records of your domain what mail servers, identified by their IPs, are allowed to send mail for that domain. For instance, if you set a TXT record like "v=spf1 ip4:164.68.116.134 a -all", you are telling any receiving mail server in the world to only accept mail from *.example.org if the sending SMTP server has that very certain IP and drop all other incoming messages. Of course, you have to rely on the receiving server to fulfill its responsibility of actually performing that DNS lookup and verification.

Syntax of SPF records still goes a bit beyond the above example and allow for _include_s, which is basically a way of delegating the specification of the actual SPF record to a different domain. This is especially helpful when using external mail providers, as you, as a customer, obviously cannot know every single IP of their outgoing SMTP hosts.

DKIM

The concept of DKIM is similarly straightforward. Its idea is that a sending mail server, e.g. yours or your mail hoster's one, adds a cryptographic signature to all outgoing mail. Given the sender's public key, a receiving server can easily verify that signature then. To find out the public key belonging to the sending servers of example.org, that public key is stored as another TXT record in example.org's DNS zone. That's it.

DMARC

In contrast to the previous two methods, DMARC is not actually used for authentication, but rather for reporting. It is a way to notify a domain owner about what is going wrong out there in the internet with regard to e-mail. DMARC essentially tells (again, via DNS) receiving mail servers, including GMail, Outlook, GMX, etc., who to send reports about failed SPF and DKIM verifications to. Depending on the actual implementation, not only failure notifications are sent, but also regular, summarizing reports, even if all goes well. Moreover, DMARC also specifies an XML-based file format for those reports. You, as a domain owner, can then read and understand those reports and take action – whatever that might be.

An example for a full DMARC record can be found here. I only want to emphasize one part of the <record> section here.

<row>
    <source_ip>80.241.xx.xxx</source_ip>
    <count>1</count>
    <policy_evaluated>
        <disposition>none</disposition>
        <dkim>pass</dkim>
        <spf>pass</spf>
    </policy_evaluated>
</row>
<identifiers>
    <header_from>wakapi.dev</header_from>
</identifiers>

The report (from Google, in this case) notifies me, owner of wakapi.dev, that one mail from the above IP was received within a certain time span and that both SPF and DKIM checks were alright. Seems like my DNS records are correct and nobody tried to scam in my name. Perfect.

Configuring DMARC

Setting up SPF and DKIM for your domain is highly recommended, as the chances of your mails being considered spam by a receiver are significantly lower then. Afterwards, you still need to set up DMARC. It is probably best to just google how to do that, there are many great posts out there (like this (German-language) one).

In essence, you will end up with a DNS record similar to this one:

v=DMARC1; p=reject; adkim=r; aspf=r; rua=mailto:dmarc@muetsch.io; ruf=mailto:dmarc@muetsch.io

In the example, any mail server who actively implements DMARC reports, is instructed to send them to dmarc@muetsch.io. I would recommend to have a separate address (different from your main address) for DMARC records, to you can easily set up rules like "move everything addresses to dmarc@mydomain.tld into some IMAP folder and mark it as read".

Aggregation and visualization

As can be seen from the example above, DMARC records are a bit unpleasant to read and you probably do not want to go through every report (couple per day) manually. Luckily, a quick GitHub search led me to a project that helps with this, specifically it does:

Read DMARC reports from your inbox via IMAP
Parse them
Persist them into a database
Visualize them on a website

(Source: https://www.techsneeze.com)

To be precise, the tool is two separate scripts, the parser and the web dashboard.

Database setup

First, you will need a MySQL or MariaDB database, which the parser can write to and the dashboard can read from. If you do not already have a running MySQL instance anyway, you can easily set one up, e.g. using Docker.

docker run -d -p 3306:3306 -e MYSQL_RANDOM_ROOT_PASSWORD=yes -e MYSQL_DATABASE=dmarc -e MYSQL_USER=dmarc -e MYSQL_PASSWORD=sshhh --name mariadb-dmarc mariadb

This command already creates a database and user for you. If you don't want to spawn a whole new database instance or don't want to use Docker, you will have to create the database and user manually.

$ mysql -u root -p

$ USE dmarc;
$ CREATE USER 'dmarc'@'%' IDENTIFIED WITH mysql_native_password BY 'sshhh';
$ GRANT ALL PRIVILEGES ON 'dmarc'.* TO 'dmarc'@'%';
$ FLUSH PRIVILEGES;
$ quit;

Parser setup

The parser can be found at techsneeze/dmarcts-report-parser and is written as a single-file Perl script, alongside a configuration file. The repo's README explains the setup process quite well.

In the config file, you need to set your above database connection (usually something like imap.yourprovider.com, port 993, SSL) and credentials as well as your IMAP credentials to log in to your mail account. With $imapreadfolder you tell the parser which IMAP folder to search for DMARC mails. This should preferably not be your inbox' root folder to not distract the parser and not risk losing any mail. I, personally, set up a sub folder dmarc in my inbox (IMAP path is INBOX/dmarc then) and created a rule to automatically put all DMARC mails in there.

Since the parser is a one-shot script and not a long-running process, you will probably want to invoke it on a regular basis, which you can set up a cron job for, using crontab -e

@hourly     cd ~/dev/dmarcts-report-parser && ./dmarcts-report-parser.pl -i

Viewer setup

The viewer / web dashboard is located at techsneeze/dmarcts-report-viewer and implemented as a simple PHP application. That means, you will need a web server like Apache2, nginx or Caddy (I'd recommend the latter) alongside a PHP installation (e.g, using PHP-FPM). Explaining how to set these things up is beyond the scope of this article, but you can just google it. After putting your database credentials once again (this time for reading), you are good to go.

Hit http://localhost/dmarcts-report-viewer.php (or whatever your domain name is) in your browser and you will be presented a (not that nice-looking, but) very convenient and practical web UI to get an overview over all DMARC reports, grouped by domains, recipient providers and date. Much cooler than reading XML files one by one!

Thanks a lot for the great work by @techsneeze.

Decentralized cloud storage (Dropbox or Google Photos clone) on IPFS?

Ferdinand Mütsch — Fri, 16 Apr 2021 07:11:12 +0000

Today I woke up with the following idea.

I thought that utilizing a system like IPFS combined with client-side file encryption it would probably be quite straightforward to build a truly decentralized / distributed alternative to Dropbox & Co. No single authority would be in charge of your files as they are distributed among potentially hundreds of network participants' computer, like with Torrent back in the days. Yet, due to encryption, nobody participating in hosting parts of your files on the network would actually be able to access them.

I was wondering if someone already built something like this? Do you know of some implementation?

What I found so far is SharpPhotos, but it seems to be in a very early stage of development and is not open source.

Time Tracking for Developers – WakaTime vs. Hakatime vs. Wakapi

Ferdinand Mütsch — Sat, 10 Apr 2021 10:36:26 +0000

Automated time tracking

As a developer, you might be wondering how much time you are spending coding on a specific project, in a specific programming language, etc. Gathering detailed statistics about your work day might help you gain valuable insights on how to increase your productivity. Or maybe you are just a statistics freak, like me, who likes to track and analyze things. What you will need for that purpose is an automated time tracking tool. Many of these exist, but this article focuses on such that are specifically meant for developers, more specifically, on WakaTime and its derivatives in particular.

WakaTime and friends

WakaTime is one of the most popular and wide-spread automated time tracking softwares for developers. It was – and still is – built by Alan Hamlett and started out back in 2013. Its concept is very straightforward. Developers install a plugin into their code editor or IDE, which sends so called heartbeats to a web service on a regular basis. These heartbeats include all kinds of information on the project and the specific file you are currently working on. The backend-side web service aggregates and processes those heartbeats and generates detailed statistics and graphs from them, which you can then browse in a web dashboard. Many parts of WakaTime are open-source, however, the web service itself, which contains most of the core business logic, is not.

WakaTime comes with around 50 different IDE plugins (ranging from IntelliJ over VSCode to Vim and Emacs), which are all open source. Third-party projects like Hakatime and Wakapi started to build up on that basis and implement their own backends, which resemble the official WakaTime's API and therefore are (partially) compatible.

In the following, I want to present a brief comparison between these three software tools. Feel free to reach out if you are missing any aspect or if there is a mistake.

Disclaimer: I am the author of Wakapi.

Comparison

General

	WakaTime	Hakatime	Wakapi
Started in	2013	2020	2019
Open source	Partially (BSD-3)	✅ Yes (Unlicense)	✅ Yes (GPL-3.0)
GitHub	wakatime/wakatime	mujx/hakatime	muety/wakapi
Written in	Python	Haskell	Go
Databases	–	Postgres	SQLite, MySQL, Postgres, CockroachDB
API	✅ Yes	✅ Yes	✅ Yes
API Docs	✅ Yes	⛔️ No	✅ Yes
SaaS	✅ Yes	Demo only	✅ Yes
Self-hosted	⛔️ No	✅ Yes	✅ Yes
Docker deployment	⛔️ No	✅ Yes	✅ Yes
Pricing	$0 - $49 / month	–	–

Features

	WakaTime	Hakatime	Wakapi
Teams / Orgs	✅ Yes	⛔️ No	⛔️ No
Goals	✅ Yes	⛔️ No	⏳ TBD
Leaderboards	✅ Yes	✅ Yes	⛔️ No
Badges	✅ Yes	✅ Yes	✅ Yes
Data import	–	✅ Yes	✅ Yes
3rd-party integrations	✅ Yes	⛔️ No	⛔️ No
Prometheus export	⛔️ No	⛔️ No	✅ Yes
E-Mail reports	✅ Yes	⛔️ No	✅ Yes
Timeline charts	✅ Yes	✅ Yes	⏳ TBD
Activity charts	⛔️ No	✅ Yes	⏳ TBD

Statistics

	WakaTime	Hakatime	Wakapi
Projects	✅ Yes	✅ Yes	✅ Yes
Languages	✅ Yes	✅ Yes	✅ Yes
Editors / IDEs	✅ Yes	⛔️ No (?)	✅ Yes
Operating Systems	✅ Yes	⛔️ No (?)	✅ Yes
Machines	✅ Yes	⛔️ No (?)	✅ Yes
Files	✅ Yes	✅ Yes	⏳ TBD
Branches	✅ Yes	✅ Yes	⛔️ No
Commits	✅ Yes	✅ Yes	⛔️ No

Screenshots

WakaTime

Hakatime

Wakapi

For Wakapi, a major redesign of the user interface is planned.

Summary

Clearly, WakaTime is the most mature and feature-rich tool, but, of course, comes at a cost.

Hakatime has a nice-looking, clean user interface and beautiful visualizations. On the other hand, it felt a bit rough and sometimes buggy here and there during my tests. But of course, this is a subjective experience.

Wakapi is built on a solid, technical basis and provides lots of configuration options for developers. However, its current UI is quite minimalist and less fancy, compared to the other tools.

All three projects are under active development and with many new features about to come in the future. Wakapi also highly welcomes open source contributions to its code base.

If you are interested in tracking your coding statistics, I would like to encourage you to give all three tools a try. Feel free to share your thoughts!

My Experiences with the Oracle Java 11 Developer Certification

Ferdinand Mütsch — Fri, 19 Mar 2021 14:58:40 +0000

On the occasion of the Java programming language's 25th anniversary, Oracle just recently announced a pretty nice discount on the Java 11 SE Developer Certification. For a limited period of time you can get the exam, which normally costs something around $ 250 and the accompanying course, which, I believe, is even more expensive, for a total of $ 25.

At Frachtwerk, we decided to take the chance and give all backend developers the opportunity to get certified. Today, I had my exam and here are my thoughts on the journey to get there.

The Course

The course's target audience includes developers, who already have a decent amount of experience with programming in Java, but not necessarily are experts, yet. Motivation for taking the course could be to further improve your skills or to specifically prepare for the certification. A certification like this, in turn, can help developers set themselves apart from potential competitors on the job market.

I, personally, have been developing web applications in Java for a few years now and would consider my experience and language skills fairly advanced. However, I was surprised how much I could still learn, though. I even got to know feature of Java that I haven't had heard of before.

Contents of the course range from the most essential things like types, variables, loops and conditions over inheritance principles, streams, I/O and JDBC throughout to low-level multi-threading and concurrency concepts and details about the fairly new Java 9 module system.

Overall, my experience with the course was very good! All content is of overly high quality, explanations are clear and understandable and the topics covered are not at all only superficial, but indeed quite advanced. At the end of each section there are quizzes for reviewing your knowledge.

Here are a few examples of such questions:

Experienced Java developers might easily skip the first ten chapters, however, the higher ones were actually very interesting to me.

Among others, I got to learn about how type erasure of generics works, what heap pollution is, how the volatile type modifier functions and that interfaces can have private, non-abstract methods.

Especially since it is free, I would definitely recommend the course, even to experienced Java developers.

The Exam

After I had spent about 1.5 full work days for preparation, I eventually felt confident enough to register for the exam. It takes place online and you can choose from a variety of different dates and times.

I was perfectly satisfied with the learning path so far. However, there were a few points about the exam which I did not quite like that much.

While the certification program itself is offered by Oracle, the exam takes place through an external provider called Pearson VUE. Being spezialized on online exams, they take various measures to prevent cheating. In principle, this is a positive thing. I could not take a certificate seriously if I knew that you could google during the exam or do partner work. However, in my opinion, things were a bit too strict here.

First of all, you have to download and install a desktop application through which the exam itself is conducted. Unfortunately, it is only available for Windows and Mac and it turned out to be non-trivial to find a non-Linux computer with webcam and microphone at our office. The program runs full-screen and you can not exit to your desktop or other programs from it, which makes sense. Also, it automatically kills processes like teamviewer.exe, anydesk.exe and all kinds of software you might be able to use for cheating.

Before the exam starts, you are assigned an instructor, who, apparently, is an employee at Pearson VUE, sitting in a call center somewhere. He or she asks you to upload pictures of your surrounding environment, including your desk. Also, you need to provide your ID card. The guy asked me to remove nearly everything except my notebook from my desk. You are not allowed to have pen and paper on your desk, no smartphone, no smartwatch, no food or drinks, except water.

So far so good. However, at some point, things started to get a little ridiculous for my taste. I have an external monitor on my desk, which I was asked to unplug and show the unplugged power supply cable on webcam. During the exam, you are constantly being monitored through cam and microphone. If your face disappears on the webcam, you failed. If another person enters the room, you failed. A point at which I started to get annoyed was when the guy told me to not speak and not move my lips. I tend to speak questions and answers to myself while I analyze and think about them. Not allowed. I wanted to take a short break for a moment and look out of the window. Immediately, the guy reached out to me to remind me that if I looked away from my screen again, he would have to cancel my exam. Being monitored in such a strict way was distracting.

Eventually, I passed the exam after 90 minutes and 50 multiple-choice coding questions and got my certificate. The only disappointing thing about it is the fact that my last name, Mütsch, is instead written as M%C3%Btsch on it. I would not have guessed that non-ASCII characters like German umlauts would still be a problem in 2021. But anyway, I am happy that I passed the exam, which, indeed, was way harder than I would have expected.

All in all, I am still happy with this experience and would recommend it to any Java developer, too.

Consider Sponsoring Open Source

Ferdinand Mütsch — Sat, 12 Dec 2020 20:21:43 +0000

Open Source powers the world

What would the software world be without open source projects? Imagine there was no Linux, no Java or Python, no Android, no Firefox or Chromium, no Apache2 or nginx, no Kubernetes, no VSCode, no Angular, React or Vue, no Numpy, TensorFlow or PyTorch, ... The list of big open source projects, that power the vast majority of the entire internet, is almost infinitely long. Besides that, at least as important are the countless, small hobby projects driven by one or a few individuals. If you ever were looking for some script, tool, app or library, chances are high that you found exactly what you needed on GitHub.

While most of the projects mentioned above are backed by large companies like Google or Microsoft or non-profit organizations like the Open Source Initiative or the Apache Foundation, individual developers with projects with only a few hundred stars build excellent software as well. But they do it in their spare time and without getting paid anything. No matter if they build and maintain their software out of passion, in order to solve a problem for themselves or just for learning purposes – in my opinion, the efforts of individual contributors should be appreciated even more.

Accordingly, in this article, I want to encourage people to consider sponsoring for free software projects they like. It complements my previous article on Donating for a Good Cause.

Supporting the small ones

While the big corporations have the necessary funding and resources, most open source projects are developed by individuals in their spare time. However, it does require one’s efforts, time and probably includes some overhead costs too. Monetary supports surely help drive the project development. [1]

When maintaining a GitHub project, your aim is not to make money from it – although some lucky people can actually earn their living from their open source work. However, receiving small donations here and there, which express other people's appreciation for your efforts, will definitely boost your motiviation to keep on coding. Of course, monetary donations are only one of many options to support a project, but, indeed, probably the easiest one.

Ways to give

If you really like a product, buy it.

There are many different options to support a project.

Star it ⭐. This is obvious, doesn't cost you anything and the project's maintainer is likely going to be a bit proud for a moment.
Buy the Pro version 🦄. Apps or web services often have a free basic version and a paid version with some benefits. Buying the advanced version will not only give you more features or less ads, but is usually also quite cheap.
Give a one-time donation 💶. Most of the time, a small one-time donation of a few dollars is already enough to make the project's author happy and boost her motivation. And, of course, the more people do it, the better.
Set up recurring payments 🔄. If you really like a project and strongly want it to keep going – especially if it has explicit maintenance costs like hosting, etc. – consider donating a small amount on a regular basis.
Get sponsorware content 🔜. Sponsorware is a release strategy for open-source software that enables developers to be compensated for their open-source work with fewer downsides than traditional open-source funding models.

Sponsoring platforms

Different sponsoring / donations platforms have established over the years, many of them with an explicit focus on software projects and some of them also being non-profit.

GitHub Sponsors: First and foremost, there is GitHub's own sponsoring feature, that was introduced in early 2020. On one hand, it is a sponsoring platform for itself – mostly for recurring, monthly payments – where GitHub is in the role of managing the donations. Like with most other platforms, you do not directly sponsor a certain project, but a user. On the other hand, the new feature also allows project maintainers to link other, third-party donation platforms to their projects.
Liberapay: Founded in 2016, Liberapay is one of the major donation platforms in the open source world. It is partially open-source as well, has around 3000 active users and is free of fees, except for the payment provider's (Stripe) fees. Liberapay is for recurring donations on a weekly basis.
Opencollective: Equally as wide-spread as Liberapay is Opencollective, which has the same purpose.
Ko-Fi: Ko-Fi is another platform that you will occasionally see linked to GitHub or GitLab projects. It is 100 % free and supports both one-time donations as well as subscriptions.
Buymeacoffee: Very similar to Ko-Fi, Buy me a coffee is another donations platform with support for one-time donations and monthly subscriptions. You see it often used not only for software, but also on blogs or by content creators.
Bountysource: Lastly, there is Bountysource, which is specifically for software, but works a bit different than the above platforms in a way that you place a bounty on a specific issue on a repo. This way, you can push features of a project that are especially important to you. When the issue is resolved, the reward does not necessarily go to the project owner, but to the person who implemented the respective feature or bug fix.

There are more platforms, but these are the ones that I consider most common or relevant.

The purpose of this article is to give a few good reasons why sponsoring software projects is desirable and important and provide a brief overview of different ways to sponsor software projects. I would be happy to see more people being a bit more generous when it comes to free software 😊.

Watching stock prices with Node-RED and Webhook2Telegram

Ferdinand Mütsch — Mon, 30 Nov 2020 20:08:47 +0000

Motivation

I hold a few stocks and I want to stay up-to-date about their quotations. However, I found it a bit tedious to actively log in to my portfolio every day to see what has changed. So I decided that I needed a notification system that automatically informs me about the relative price changes for all of my stocks once a day. Since I get all kinds of notifications – including security alerts from my server, updates in my GitHub feed, and more – via Telegram, the choice to use that messenger for stock price notifications as well was quite obvious. Another obvious decision what have been to write a small Python script, that gets executed once a day via CRON. However, this time, I didn't want to write any code, but instead try the flow-based visual programming tool Node-RED.

Node-RED is a JavaScript-based platform to compose logical workflows through combining small, elementary building blocks together. Such building block, called nodes, include functionality to ingest data (e.g. via HTTP calls, MQTT subscriptions or reading a file), process it (e.g. string replacements, logical condition checks, aggregations, etc.) and output it in some way again (again, via HTTP, MQTT, files, etc.). Without writing any code, but only through configuring these elementary operations, whole programs can be built. While Node-RED is primarily used in IoT contexts, it basically serves any purpose. An even more comprehensive and "mature" alternative is, to some extent, Apache NiFi. However, while Node-RED is perfect for tinkering and small projects, NiFi focuses on scalability and Big-Data-like workloads.

Flow

The resulting flow, that fulfills the above mentioned purpose looks like this.

The flow's entry node is an inject node, that holds a JSON array of all my stocks' symbols (e.g. QCOM) and is automatically executed once every afternoon. The message is then split into multiple, individual messages, namely, one for every stock symbol. A http request node then calls the Alphavantage API once for every message to fetch the intra-day price changes. Subsequently, the response is parsed, post-processed and formatted as Markdown. Eventually all individual messages are combined into one again before my Webhook2Telegram bot is requested to send me the message as a last step.

Mastering Software Versioning in JavaScript Projects

Ferdinand Mütsch — Mon, 09 Nov 2020 09:52:42 +0000

Introduction

A frequently overlooked aspect of software development is the proper versioning of code. Consistent and descriptive version numbers not only help developers keep track of their own work, but can also inform users of your software about what to expect from a new release. While versioning is especially important for libraries and frameworks which other projects depend on, benefits apply to standalone applications equally.

In this article, we introduce techniques, conventions, and tooling that helped us establish a robust way of versioning our JavaScript- and/or TypeScript-based software projects.

Concepts

Semantic Versioning

One of the most important aspects to think of when it comes to versioning is the version number itself. Before caring about tooling and others, you need to come up with syntax and semantics for it.

A concept that is well established among open source software projects is Semantic Versioning, or SemVer. When following this specification, a version number consists of three digits separated by dots, like 1.5.4 or, more formally <MAJOR>.<MINOR>.<PATCH>, where each individual part has a meaning:

MAJOR: Incrementing it indicates that there have been fundamental changes to the software and the new version is most likely not backward-compatible with the previous one.
MINOR: Essentially indicates that new features were added, but backward-compatibiltiy is still guaranteed.
PATCH or BUG FIX: Gives a hint that minor changes or bug fixes had been introduced recently.

Strictly following these semantics helps to maintain a common understanding of what a certain version means and what to expect from a new release.

Conventional Commits

The second concept that we committed ourselves to follow is Conventional Commits. Similar to semantic versioning, the conventional commits specification provides common syntax and semantics for information provided by a developer. However, in this case, the convention is not about the version number itself, but about the commit messages composed by developers when checking in new code into version control. The goal is to standardize their format and make them machine-readable to a certain extent.

When following conventional commits, a commit message essentially has to be prefixed with one of a few keywords.

fix: – A commit message with this prefix indicates a bug fix
feat: – A commit message with this prefix indicates the introduction of a new feature or functionality
refactor: – A commit with, whose message is prefixed like this, contains code refactorings, i.e. internal, technical modifications of the implementation of certain logic
chore: – This prefix indicates minor, miscellaneous changes of any type, that do not necessarily affect the user immediately
BREAKING CHANGE!: A commit message with this prefix warns about comprehensive, fundamental changes and indicates that the newly released version is likely to be incompatible with the previous one

The conventional commits specification comprises a few more keywords and also allows developers to come up with custom ones. However, these are the most relevant ones.

Tooling

When having paid attention, one might have recognized a few similarities in the semantics of conventional commits and semantic versioning. Commits with fix-changes correspond to the PATCH version, feat goes well with the MINOR version and BREAKING CHANGEes will inevitably result in a new MAJOR version.

As a consequence of following the above conventions, we enabled our project for an automated versioning workflow.

standard-version CLI

standard-version is a JavaScript tool that utilizes conventional commits to automatically enforce semantic versioning. Moreover, it is capable of automatically generating a changelog in Markdown format, which developers can provide their users with.

When running standard-version, the tool scans your commit history since when it was last executed, searches for fixes, feats, or breaking changes, and adapts the project's version accordingly.

To add it to an existing project, all you need to do is:

Install it as a dependency

$ yarn add -D standard-version  # (or npm i --save-dev standard-version)

Optionally add it as an NPM script to your package.json

{
    "name": "your-cool-project",
    "version:": "0.0.1",
    ...
    "scripts:" {
        "release": "standard-version"
        ...
    }
    ...
}

Release Workflow

After the development team has committed to consequently follow the conventional commits specification and all tooling is set up, a typical workflow to release new versions of your software might look like so.

Once a new version is ready to be released, i.e. at the end of a sprint, a developer executes yarn release (or npm run release) to kick off standard-version. As a result ...

... the project's commit history is scanned to determine which part of the version number needs to be incremented
... the version property of the project's top-level package.json is set to the new version
... a CHANGELOG.md file is written, containing separate sections for features and bug fixes
... the changes are committed to Git
... the new commit is given a Git tag corresponding to the new version

Depending on your setup, a push to the remote repository might kick off your CI/CD workflow, which may automatically build a new Docker image with the newly introduced tag and push it to a public or private registry. Using tools like Watchtower, the new image might even be rolled out to production automatically.

The only manual steps required in this workflow were a single yarn release command and a Git push. Nothing more, nothing less.

Conclusion

The above workflow has proven to be a convenient and consistent way of managing and releasing new versions of our JavaScript- and TypeScript-based frontend-, backend- and library projects and is even more beneficial with proper CI/CD pipelines and tooling like GitLab, Docker, Watchtower, Portainer, and others. It might even be adapted to projects written in other programming languages.

GitHub Action to turn your profile README into a guestbook

Ferdinand Mütsch — Tue, 15 Sep 2020 20:23:31 +0000

I am happy to participate in the Actions Hackathon and to share my work with my fellow dev.to readers and the open source community. Hopefully, my little project inspires you to use it or to start hacking on your own actions.

My Workflow

As an open source contributor chances are high that you want to get feedback from the community and especially the lovely people who use your code to solve their problems. You need a way for them to interact with you and enable them to share their thoughts and thankfulness about your projects.

readme-guestbook turns your GitHub profile page into an interactive guestbook. People can post issues to your repo, which are then rendered to the Markdown of your README.

Try it out!

Submission Category: Wacky Wildcards

muety / readme-guestbook

A GitHub action to create a guestbook in your README from repository issues

Additional Resources / Info

My own profile as an example of the action in action 🤓

Modern, reactive web APIs with GraphQL, Go and Server-Sent Events – Part 1

Ferdinand Mütsch — Mon, 08 Jun 2020 00:00:00 +0000

Introduction

In the course of this two-part article, the interested reader is briefly introduced to the basic of GraphQL and how it compares to traditional approaches. In the second part, an example single-page web application (SPA) is built to demonstrate the use of GraphQL in combination with further modern web technologies. The final project provides a clean, opinionated code- and project structure for both backend and frontend and constitutes a good starting point for new apps based on the presented tech stack.

muety / go-graphql-sse-example

Basic example application using Go + GraphQL Subscriptions + Server-Sent Events + MongoDB + VueJS

go-graphql-sse-example

Basic example application to demonstrate an alternative way of building web APIs compared to REST. It combines these technologies:

Go as the primary backend-side language
GraphQL as a "protocol" for defining web interfaces
Server-Sent Events as a simple protocol for live updates, used as an implementation of GraphQL Subscriptions here
MongoDB as a flexible document database for storage
VueJS as a frontend framework to build single-page web applications

This project is intended to serve as a starting point to build modern, GraphQL-based single-page web application with a clean, opinionated code structure in backend and frontend.

Original blog post

Inspired by:

Requirements

Go >= 1.13
NodeJS >= 12.8.x
A MongoDB database
- You can get a free one at mlab.com

Usage

Backend (`/`)

Edit config.yml to configure your database
Set up the database (i.e. add users, collections and demo data)
Compile the GraphQL schema (located in schema…

View on GitHub

What is GraphQL?

GraphQL is a relatively new (proposed in 2015 by Facebook engineers) approach to designing APIs for (web) backend applications and can be considered an alternative to REST or remote-procedure-call (RPC) mechanisms like JSON-RPC. In other words, it's "an open-source data query and manipulation language for APIs" [1]. The specification is open-source and actively evolving on GitHub. While REST APIs are currently the de-facto standard on the web (although not necessarily all of them being fully mature) – GraphQL starts to gain traction.

Comparison with REST and RPC

In contrast to REST, which is primarily structured around resources or entities and RPC-based APIs, which focus on actions or methods, GraphQL is all about the underlying data itself. The consumer of an API – usually the frontend / client-side part of a SPA – only has to know the schema and structure of the data provided by the API to CRUD it. Compared to REST APIs, where the consumer heavily depends on the fixed data structure delivered by the backend API, this is especially beneficial as it introduced a lot more flexibility and decoupling and can save the developer some time making the client-side application tolerant. Also, you will probably not need backends for frontends anymore.

Essentially, with GraphQL, the consumer asks exactly for what it needs and how it needs it, i.e. your client application tells the backend exactly what to return and in which format. Consuming a GraphQL API is like querying a database, but with more guidance and control.

Example

Let's look at an example to get a better idea of how GraphQL works, especially in comparison to the REST principles.

Imagine you have an e-commerce application with products and orders. Every order consists, among others, of a set of products. As the operator of the web shop, you might want to get a list of all orders. With a more or less RESTful API (we neglect the hypermedia controls in the example though), your request-response pair could look like this:

Request
-------
GET /api/orders

Response Body
-------------
[
    {
        "id": 125,
        "customerId": 8977,
        "createdAt": "2020-06-06T13:40:49.038Z",
        "productIds": [ 49863176 ]
    }
]

So far so good, but potentially you will also want to view the actual products right away. What you got are only ids, for each of which you would have to issue another API call to retrieve it. Alternatively, the API could also return nested objects, like so:

[
    {
        "id": 125,
        "customerId": 8977,
        "createdAt": "2020-06-06T13:40:49.038Z",
        "products": [
            {
                "id": 49863176,
                "name": "Slim T-Shirt navy-blue",
                "price": 17.90,
                "options": [
                    {
                        "id": "size",
                        "name": "Size",
                        "description": "T-Shirt size",
                        "values": [
                            {
                                "id": "s",
                                "name": "Size S",
                            },
                            {
                                "id": "m",
                                "name": "Size M",
                            },
                            {
                                "id": "l",
                                "name": "Size L",
                            }
                        ]
                    }
                ]
            },
        ]
    }
]

However, that is — to my understanding – not truly RESTful anymore. Also, while the above example is still quite straightforward, things get ugly as nested objects include other nested objects, that include other nested objects, that... Quickly you get JSON responses of several tens or hundreds of kilobytes, although you're potentially only interested in two or three attributes. Moreover, on some pages of your shop you may be interested in all possible options (e.g. "size") of a product, but not on others. Should your API define different view models now and expose different endpoints? Or a single endpoints with query flags like ?expanded=true? Soon you might be catching yourself tailoring your API specifically to the needs of your client while neglecting REST conventions and a straightforward design.

With GraphQL, things are different. Your API is a bit dumber and less opinionated now and does not deliver data in a fixed structure, according to a specified GQL query, which looks a lot like JSON. The above example might look like this, now:

Request
-------
POST /api/graphql/query

{
    "query": "\{
        orders {
            id
            customerId
            products {
                name
                price
                options {
                    name
                }
            }
        }
    \}"
}

Response Body
-------------
{
    "data": {
        "orders": [
            {
                "id": 125,
                "customerId": 8977,
                "products": [
                    {
                        "name": "Slim T-Shirt navy-blue",
                        "price": 17.90,
                        "options": [
                            {
                                "name": "Size",
                            }
                        ]
                    }
                ]
            }
        ]
    }
}

This way, you get only the data you want. All your API has to know is how to fetch every piece of data. All your client has to know is how the data schema itself looks like.

Try it out

GitHub's official API offers GraphQL query endpoints. You can try it out using their GraphQL explorer.

GraphQL Basic

Since this article does not aim to be another introduction to GraphQL, you can read most of the basics about fields, data types, etc. in the official docs. However, it is worth mentioning that GraphQL supports three types of queries:

Query: "Standard" type of queries, used for fetching data (see above). Similar to what you would do with a GET in REST.
Mutation: Query type used to modify data. Similar to what you would do with a PUT, POST, PATCH or DELETE in REST.
Subscription: Query type to communicate your intent to subscribe to live data updates.

Subscriptions

While a basic GraphQL application will at least use the former two types, the latter is especially interesting in the context of this article. Using subscriptions, you can have your web frontend be notified when new data arrives at the server or existing data changes. For instance, the operator of the above web shop could have a live-updating dashboard, that shows new orders just as they are placed.

For subscriptions, the GraphQL standard does not define a lot more than their plain existence and purpose. Especially, it is not defined how and which technology to implement them. On the web, any publish/subscribe-like mechanism that provides bi-directional or uni-directional server-to-client communication is appropriate. For the sake of simplicity, Server-Sent Events are used in this article.

What's next?

This part gave a brief introduction to GraphQL. The next part is about actual code. We're going to build an example web app with live-updates using GraphQL, Go, MongoDB and VueJS.

Modern, reactive web APIs with GraphQL, Go and Server-Sent Events – Part 2

Ferdinand Mütsch for Frachtwerk GmbH ・ Jun 8 '20

#graphql #go #mongodb #vue

DEV Community: Ferdinand Mütsch

Wakapi - Open-Source Time Tracking for Devs

How development history keeps repeating itself

Innovation on the web

New technologies against boredom

Reinventing the wheel

PHP vs. X

Server-side rendering (SSR)

Mixing template and code

File-based routing

Serverless

RPC technologies

gRPC, tRPC, ...

Old ≠ bad

Key takeaways

Open-source, self-hosted location tracking with OwnTracks and Grafana

Introduction

Your Own Setup

Requirements / Components

OwnTracks

Server-side script

Grafana dashboard

Conclusion

Aggregating and Visualizing DMARC Reports

DMARC, DKIM and SPF

SPF

DKIM

DMARC

Configuring DMARC

Aggregation and visualization

Database setup

Parser setup

Viewer setup

Decentralized cloud storage (Dropbox or Google Photos clone) on IPFS?

Time Tracking for Developers – WakaTime vs. Hakatime vs. Wakapi

Automated time tracking

WakaTime and friends

Comparison

General

Features

Statistics

Screenshots

WakaTime

Hakatime

Wakapi

Summary

My Experiences with the Oracle Java 11 Developer Certification

The Course

The Exam

Consider Sponsoring Open Source

Open Source powers the world

Supporting the small ones

Ways to give

Sponsoring platforms

Watching stock prices with Node-RED and Webhook2Telegram

Motivation

Flow

Mastering Software Versioning in JavaScript Projects

Introduction

Concepts

Semantic Versioning

Conventional Commits

Tooling

standard-version CLI

Release Workflow

Conclusion

GitHub Action to turn your profile README into a guestbook

My Workflow

muety / readme-guestbook

A GitHub action to create a guestbook in your README from repository issues

Additional Resources / Info

Modern, reactive web APIs with GraphQL, Go and Server-Sent Events – Part 1

Introduction

muety / go-graphql-sse-example

Basic example application using Go + GraphQL Subscriptions + Server-Sent Events + MongoDB + VueJS

go-graphql-sse-example

Requirements

Usage

Backend (/)

What is GraphQL?

Backend (`/`)