DEV Community: szabi

Responsible programming

szabi — Sun, 22 May 2022 10:01:00 +0000

As programmers we come a across quite often a question: “Is the application safe to deploy?”. I had to ask myself as well sometimes: should I deploy X or Y feature even if there is an edge-case bug in there? Of course later those edge cases are the bugs, that the users find. Even though they aren’t looking for them, but this just shows sometimes how far we get from the way our users would expect an application to work.

Long time ago when someone would say: We can only deploy an application when it is safe to do open-heart surgery with it! I would smile a bit to myself thinking that I’m not coding medical equipment nor aviation software. But as years past and I got more experience working with spaghetti code and legacy systems that I inherited from here and there, my view started to change on this. It changed to the point, where today I would say the same to fellow programmers.

Save time today by cutting corners (e.g. cutting on writing tests for a fix or a small feature) and you will waste 10x more trying to fix it later or add a new feature to it. But if it would only be time that is wasted… but its not just time, it is never only time…

With bad code you will waste credit, clients will not trust your work and will have a fear from either deployment (what will break this time) or from not getting what they asked for (we got A and B, but C will only work if you sacrifice a chicken at a blood moon).

With bad code you will also waste the time of the users, if that user is trying to use your app to actually do their job, then you’re wasting their employer’s money as well.

With bad code you can cause your user to miss their monthly bonus, if you do an app that is used by employers to track employee performance.

With bad code you can also waste money of your client directly. Even if you do a small webshop and if any steps of the checkout or placing the product in the cart is acting badly, then your client will lose their clients.

With bad code you can get someone unemployed - doing an accounting application that doesn’t sum up everything correctly, or misses to add some taxes.

With bad code you can get someone imprisoned, as above bad tax reports can be seen as tax fraud in front of court.

With bad code you can kill someone even if you don’t do aviation or other safety-critical software.

There are lots of things that can go bad in software and while I enjoy making spaghetti code better, adding tests, breaking them down and adding meaning full naming. Every time I look at an app that is 100% spaghetti and I know that there are thousands of users using it daily, can’t help it but some questions comes to my head: “How did this make it out to production? Who thought this is okay to do? How is this still running?”

If you write code that someone will use, then please read the following three articles and take it to your heart. Think of them every time you code something! Even an accounting app can kill someone!

https://www.dailymail.co.uk/news/article-9502925/Post-Office-scandal-Dozens-staff-today-convictions-quashed.html
https://www.theatlantic.com/technology/archive/2017/09/saving-the-world-from-code/540393/
https://en.wikipedia.org/wiki/Volkswagen_emissions_scandal - this one is an exception. Here not the code quality, but the quality of the person who done the code knowingly is the issue.

Is your code good enough to do an open heart surgery with it? Would you bet your life on your code?

TrueNAS & Nextcloud

szabi — Sat, 07 May 2022 08:42:00 +0000

Awhile back I wrote about my experience in running TrueNAS and Nextcloud server at home. Thought I would give a small update how is that after 1.5 years.

I’m still very happy with the whole setup, even extended storage in the server. Had some interesting experiences when I tried to upgrade Plex and Nextcloud jails. There I just gave up, I still update and upgrade them, but not via TrueNAS’ update functionality, but by logging into the instance and issuing the update commands in the container. Also before updating I always make backups of them using the snapshot feature and when I managed to fubar my Nextcloud instance it was just ~4 clicks to restore it to normal without losing data or settings or anything. It is quite fiddly, still didn’t manage to figure out how am I going to upgrade the MySQL in that container. Tried to just use the system’s package manager, but that didn’t turn out well.. So might gonna re-create a new instance and just migrate settings and mounts over.

Managed to discover a plugin for Nextcloud recently that manages RSS feeds and has also an Android app, so that it can keep track of read and unread news no matter where the reading happened. Pretty much like feedly (used that service before finding this), so I can read some articles on the computer and those will be also marked as read on the mobile. Writing it down it feels like a laughable feature, but its really convenient! (one less google owned service that I use)

My only grievance with Nextcloud is a regression in their mobile app that they are really slow on fixing it. The issue is about that once you sync photos from Nextcloud to your phone, then the mobile’s gallery app won’t see them properly, so you cannot use those photos in other apps. E.g. would like to crop and only send the cropped image to someone on discord, well then you’re out of luck.. have to edit it on the computer.. and since I’m on the computer, might as well send it from there.. Not a deal breaker bug, but it is annoying. Like this Nextcloud for my wife is only about backup and easy access on the computer.

Over to backup feature of TrueNAS. Those files that are important to me and they are not in Nextcloud (e.g. full family photo collection) I setup TrueNAS to do daily cloud backups. I’m using Backblaze for it and like this I have an offsite backup for ~7 USD/month for all those files that are truly important. Also TrueNAS can do encrypted backups, so if the worst would happen and Backblaze would get hacked, then still nobody would have to see pictures of me sipping beer in my underwear. :D

So in this post I have mentioned:

Nextcloud 11 times and
TrueNAS 7 number of times

Safe to say that I’m very happy with this setup and that I’ve managed to repurposed an old PC that wouldn’t really get otherwise any use.

Rewrite, refactor, knee-deep in legacy - part two

szabi — Sat, 30 Apr 2022 10:07:00 +0000

I really don’t take on extra work apart of my full time job. At work I do get to try many different things. We get chance to work on both legacy and greenfield projects. So really don’t feel that I would need extra projects on the side.. :D

This project that I’m gonna write about is a webshop, that sort of fallen into my lap. Back in 2010, when I started out with programming I’ve been doing webhops and inventory sites, using PHP, MySQL. So it didn’t feel too alien to me to help out every now and then, when the person running it needed help. Even though I really don’t like webhops and always turned them away, this was the first and last exception.

I got to be a second dev on this project as the original author had less and less time to deal with it. Eventually the situation got to the point that I had to take over the maintenance fully (to much of my dislike).

State

The site was written in PHP using MySQL and NextJS for the front-end. The backend was producing normal JSONs, so nothing too exotic there. The issues started when one started to look under the hood. The original author also didn’t have much time to actually do the site and this was reflected in the code as well.

It was really chaotic and as I didn’t do programming in PHP for long it was hard for me to change it in a good way. Since the site was simple there was no framework used and as it was a rushed job the most common mistake was also made: no tests were written.

Front-end was also similar, with the exception that there NextJS gave a nice frame to it, but also zero tests to see if everything is working.

Kickoff

When I had to take over the maintenance including running the server, it was clear that I won’t be able to support the PHP backend specially in its current form. Wanted to fix that up and stick with it, but really couldn’t get a handle on it.

At that time I started to get to know NestJS as at my company we started to migrate services to this framework. So for me it was an easy choice to go with NestJS. The plan was to rewrite the backend using NestJS while keeping the database schema and HTTP API the same.

This time I’ve made sure to do it in TDD, so that every new route and service was fully tested. Some logic issues came out that were also present in the PHP version. It was an interesting journey, but at times it did feel monotone, doing the same over and over. But on the end all worked out and managed to deploy the backend with nearly zero change to the front-end.

With NestJS the default database ORM is TypeORM, it supports many different databases. In general I don’t trust libraries that can do anything while providing the same interface. But here it actually worked out nicely, as when I was looking for database providers I couldn’t find cheap MySQL provider while I managed to find a really cheap (actually free plan) for PostgreSQL. This was the first time where I actually migrated from one database to a different one with about 10 minutes job thanks to the ORM. I did concentrate on using the ORM’s methods for building queries and had zero raw SQL statements in the code.

Frontend

The frontend wasn’t in the best of shape either, but hey it is React thanks to NextJS, so that help a bit. It wasn’t in such a bad shape that I would want to rewrite from scratch, but I needed some proof after each and every change that all is working.

Tried to write tests using Jest, but the react part wasn’t written with the best practices, so it brought out all timing issues. This way proved to be really slow going. Here I decided to do an end-to-end testing using Cypress instead unit and functional tests with Jest.

Had zero experience with Cypress, just seen some PRs and nice comments at work from colleges. Before jumping into Cypress tests I crunched through a course from udemy, with double playback speed, think it took me day and a half. Sorry, don’t want to brag here but still makes me smile thinking about the instructor’s voice in double speed :D

After this I covered all the pages and all the features that I came across. With cypress it was truly a breeze, timing issues were still present, but manage do put some cy.wait commands and TODO comments for a later time.

Once cypress covered everything, refactoring the front-end and getting rid of all the timing issues was a dream. Okay maybe not a dream, but went with good speed and managed to do frequent deploys, knowing all is working.

Outro

Overall I’m happy that I chose rewrite, with the old code without tests I would still be having nightmares. Since then I managed to extend on the features of the site and also automated a lot of tasks on it. All these new features were done with tests, so at all time I know that the site is working and that the person relying on it won’t lose sales due to bugs.

Also this project gave me a good chance to deepen my knowledge with NestJS, Cypress, React and NextJS. Did make me a fan of NestJS and NextJS!

However I did put a lot of work into it. Looking back on it, eventually I rewrote the front-end as well. Via refactors think there is no line that I didn’t touch there. So in hind site for a webshop I would pick a provider rather than hand rolling it.

Rewrite, refactor, knee-deep in legacy - part one

szabi — Sun, 24 Apr 2022 16:37:00 +0000

From time-to-time we encounter some legacy projects - or the braver developers even work on them day after day. :) When working on such projects we do come across the exclamation that: “Starting from scratch would be quicker than trying to fix this mess!”. This normally comes from the frustration that legacy projects can cause.

There are already lots of articles out on the interwebs about refactor vs rewrite. Failure and success stories and would encourage those who don’t have experience in deciding this question to read many articles, pro and con. This article includes a few good one:https://medium.com/@herbcaudill/lessons-from-6-software-rewrite-stories-635e4c8f7c22

So where do I stand with this question? Personally I like to work on legacy systems, one can feel like Indiana Jones exploring hidden gems, funny comments and many odd solutions. Apart of this archaeological work when you just correct a small bit on legacy code that can sometimes yield huge performance improvements that makes work on them quite rewarding. So from this you can guess that I’m more towards the refactor end of the scope. In my experience rewrite from scratch doesn’t work on larger systems.

Got two rewrite stories to share. Long-long time ago I had the chance to partake in a full rewrite of a large system that turned out to be the text book example of the struggle and up hill battles. Second is a smaller project but was a good reminder for me why I don’t like rewrites.

The System

The system that I have had the chance to contribute in rewriting was like this: had a legacy code base in python using Django with a MySQL database, then there was a proof-of-concept app written next to it in NodeJS using MongoDB. This PoC app supposed to be temporary. Yes, this is the point where you can already start to laugh, like that ever happens… temporary…, of course it went to production and the temp PoC solution hunted the company for 3+ years.

We tried to fix up the temp solution to get it into shape and avoid throwing ever growing amount of money on CPU/memory/DB and network resources. But the beast always managed to grow another head. This was the point where the lead at that time decided to start over: clean slate.

Lesson #1: when you create a PoC or temp solution it still need to adhere to the highest quality standards. In our case it was a rushed steaming pile of spaghetti, straight from the stow.

Rewrite

Now we have an instable PoC system in production and a stable but painfully slow legacy system. How would you go about the rewrite? One could say divvy up the system and start rewriting small portions of it. Yeah, one could definitely take that path, but that wasn’t the one we picked.

We started to create a parallel system from scratch dreamed up all the microservices ahead of time. Decided to continue to use NodeJS with MongoDB as that was already the hype back then. We created a synchronization service that would actively bring data over from the old python system. The legacy and the PoC was all used in production and we wanted to test early the new microservices as well so we constantly synced data over to the system that nobody used yet.

The development teams were divided along the new and old systems. There was a larger development team supporting the old system that was actively bringing in money for the company and paying all of our salaries. Another smaller team had the marvellous job of learning about new technologies and write the microservices that will be the new system that will save us all.

I’ve had the chance to be in both of those parts (supporting the legacy and help with the rewrite) actually neither side is fun. Supporting the old application had the pressure from the clients that they needed newer and newer features, hunting bugs, dealing with downtimes. Reward part here was that when you have done something you got instant feedback on your job. In the rewrite team you get the pressure that everybody is waiting for this saviour system to come. They expect to get a system that will solve all their problems, it will solve their insomnia, it saves their marriage, they will be better parents and the new system will even make them coffee in the morning. Also there is more developers on the old system cranking out newer and newer features that the new system of course will support! :D

Catching the ever growing features of the system we would like to replace is just like Don Quichotte fighting with the windmills. You have very small chance of winning. We actually tried to push back on the adoption of the new system saying that its just not yet ready. But eventually we all agreed that it will never be fully ready and at one point we need to start using it.

Adaption of the new system

The whole team saw that we won’t be able to reach feature parity. For the adoption we pick a client that had the least need of custom features and set them up with the new system.

There were a lot of hick-ups, we were correcting a lot of bugs and mistakes that came from the fact that the new system was also rushed. It didn’t become spaghetti and thanks god we did write some tests - unlike the mentioned “temp PoC” project that had zero tests. But eventually the new system was working, it started to generate income for the company! To get to this point it took the teams about 2 years!!! Also there was one big challenge still remaining: get all the clients on it.

Migrating the clients was the next stage of the rewrite. Clients shouldn’t need to know about the changes we make in the background. They have a service that they expect to get regardless if the backend is python or nodejs or assembly. Interesting to note here that the service provided by the system relied on historical data. That data was all in the old system! All that data had to be converted and moved into the new system. This task was the one I actually ended up enjoying maybe the most. Luckily the conversion wasn’t too complex and it was interesting to see that we filled up the new databases through weeks of carefully controlled migrations. Had to be careful not to over load the old system. Then after all that: it worked! The system manage to show the same reports to the users just like the old system! 2-3 years and we managed to get to the same point as where we were before! :D

Different way

The temp PoC system was some sort of microservices architecture, sometimes we were joking about it saying its a micro-monolith as 4 services were serving it. The python legacy system was monolith. New system with the planning up front without good experience with microservices was more towards a distributed monolith with ~8 services serving the flow. We managed to change on this over time (without starting from scratch).

Thinking back of the decisions and what happened during the rewrite, frustrations from both development and customer service teams, if I would have to do the whole thing all over again, then would choose a different path.

Starting again with a monolith system would look at the processes and would try to move out small parts of the big service to a new microservice. Breaking down the monolith bit by bit, so that all parts are constantly in use and being validated. When we were doing the rewrite and testing we didn’t test the system with high loads. Of course everything was working on low loads and amazing developer computers, but when it went to production and came the load there were lots of performance issues.

Outro

In the above case rewrite was a bad decision in my opinion, but still we managed to get it working. There are cases where you have no other choice than starting from scratch. In another post I’ll share a story where I did a full rewrite and would do the same again in that situation.

Time travel and privacy

szabi — Mon, 22 Feb 2021 17:39:00 +0000

In December of 2020 I read some articles that made me want to go back in time and use tools that I considered to be old. One of these articles was about the productivity score by Microsoft’s O365, another was about how our phones betray our trust (no surprise there).

The content behind the following three links that made me think:

Mail and Calendar

After reading the article about O365 and that GSuite also has something similar to that, I decided to go back to some kind of e-mail and calendar application like Thunderbird or Evolution.

Yes I did use O365 via the browser before, it worked actually quite well didn’t miss notifications and it overall provided a good interface. But using the browser it also means that they can track:

where is my mouse on the screen,
if the tab is actually active on my screen
what is on my screen
typing and deleting and typing…

So really waaaay too much information that I would feel comfortable to share.

Must say it here: I’m not working for a company that collects such information or would do actions based on such information. If the company would change and go in this direction then I would have the choice to change my employer. Still I would rather avoid providing even the possibility for it.

Looked through possible e-mail clients for GNU/Linux, there were two things that are a must have for me:

no electron apps - don’t want to run a full browser with nodejs.. just don’t
to be able to integrate with a calendar app.

Checked BlueMail briefly, but fall short on a) being an electron app and b) not really maintained.

Looked at Thunderbird, it is still an okay with Lightning it can also satisfy the calendar and it is still the default e-mail client in Ubuntu. While I don’t use Ubuntu this will still keep maintainers motivated. However Thunderbird just didn’t feel right to me.

Lastly I tried Evolution. After installing and setting it up, realized that I used to use this client back when I started programming in 2010. Suddenly I’m back in 2010, just have more grey hair than I used to.

With evolution it was a huge surprise to me that it works with O365 out of the box, syncs calendar, contacts and even the todo items (if I would use them.. org-mode FTW!).

NOTE: MS did admit this wasn’t a good way and they are going to remove some low level reporting. See their blog post about it.

Chat applications

Okay this is bit of different topic than MS and phones tracking all of our moves, but still related to the back in time theme.

By moving my e-mail and calendar to an app I managed to close two pinned tabs from my browser and also managed to avoid using container tabs to be able to login to my son’s and daughter’s O365 account (they are in lower grades of primary school, so yes I do get to manage their accounts for when we’re in remote tuition mode). With this closing of tabs I noticed that there are other tabs that I also use they are just constantly pinned: whatsapp, facebook chat, discord and sometimes gitter. So this provided a new challenge: can I find an app to unite them all? Yes I can.

Again didn’t wanted to use something that is just another browser, so went with an old-old friend Pidgin. All the listed chat platforms have a plugin for it, open source with a clean code base (didn’t review all of the code, but all the files I visited was clean and relatively easy to understand), so very much tempting to contribute. Best of all: Pidgin will only eat up ~70 Mb of memory - opposed to gigabytes - and won’t send statistics about me to the facebook and other data hoarder and dealers.

Mobile front

All this has triggered the privacy demon in me so started to look for alternatives as well on mobile. First big hoarder to tackle: Google. The app that is actual for me to look at is Google Photos. I loved this app as made it very easy for me to share pictures with my wife or with anyone. Also could access all the photos on my computer quickly and all is backed up and safe.

All good things must come to an end at some point for me with Google Photos the end meant that - understandably - they don’t provide free storage anymore. In the past if you accepted that they re-compress the images with their algorithms, then they wouldn’t count towards used data - so free and unlimited storage.

Photos are actually a long story for me, tried a lot of different solutions when dealing with photos from my camera. Nothing work truly easily. A new motivation

apart of google - was that my wife started to make photos and videos with

the family DSLR. My wife isn’t tech savvy, so whatever solution I try next it has to be seamless to her, other way she won’t use it.

Now to the solution: TrueNAS. Yes, decided to run my own storage at home. As you may already guess I didn’t go with the most straightforward solution… nah life would be boring if we wouldn’t make is pointlessly over complicated!

So for NAS my criteria was that I want to be able to extend it later when and if I need more storage. For this you need more than just a two bay NAS, but rather a 4 or 5 bay NAS. Now those bigger NAS setups a for me just ridiculously expensive. Instead of spending big, decided to reuse my desktop PC from 2008 to build a NAS using TrueNAS system. It is really an awesome setup for me, but think I’ll write about that in a dedicated post.

On this TrueNAS setup I also installed Nextcloud, this is able to replace both Google Photos and Dropbox for me. Got a mobile app for it, easy enough to use and there is also a Gnome integration for it. Apart of images and files it can also serve as a calendar and contacts server (another stab at Google).

For larger files also configured NFS endpoints so that we can handle images from the DSLR. NFS works surprisingly well with Darktable, it is able to read the files quickly enough. If I would deal with images as a profession would probably invest into a better network setup, but it works well enough via WiFi already. Only secret is to keep Darktable’s SQLight database files on the local system, having that over NFS renders the app uselessly slow.

Having the above setup working reliable and well, I’m really considering going back to a privacy oriented Android setup on my phone as well, without all the bloatware and mandatory google applications installed.

News sources

RSS is still out there and still lives, I was surprised to find that most news sites will have RSS feeds as well. With RSS again one can use a simple application, even Evolution has RSS reader plugin and have all the news tracking free.

Outro

The journey with all those apps and different setups made me realize again that we do share a lot of information about ourselves on the internet. In the background we have no idea how they use all that data. The collected data shouldn’t be connectable the person behind it, but in truth it can be. If you don’t read all the three articles linked in the beginning, just read Martin Gundersen’s one.

What gets measured gets managed.

Through the data that we submit knowingly or not, voluntarily or not, will enable companies to manage our actions. So now I’m back in 2010, running my own server and using native application instead of browser based solutions.

Bad dependencies and bad interface changes

szabi — Tue, 19 May 2020 17:29:44 +0000

Recently one of my colleagues has shared his pain about a nodeJS package that we use to zip files.

His issue was that the library stopped generating the zip files and just hang. Eventually found, that the package in question has been upgraded,
but with a small issue. The newer version has changed the interface and the same method instead of returning the instance it returns a Promise.

Archiver.prototype.finalize = function() {
  if (this._state.aborted) {
    this.emit('error', new Error('finalize: archive was aborted'));
    return this;
  }

  if (this._state.finalize) {
    this.emit('error', new Error('finalize: archive already finalizing'));
    return this;
  }

  this._state.finalize = true;

  if (this._pending === 0 && this._queue.idle() && this._statQueue.idle()) {
    this._finalize();
  }

  return this;
};

Code Snippet 1:
Old version

source: https://github.com/archiverjs/node-archiver/blob/0.15.1/lib/core.js#L489-L507

Archiver.prototype.finalize = function() {
  if (this._state.aborted) {
    this.emit('error', new ArchiverError('ABORTED'));
    return this;
  }

  if (this._state.finalize) {
    this.emit('error', new ArchiverError('FINALIZING'));
    return this;
  }

  this._state.finalize = true;

  if (this._pending === 0 && this._queue.idle() && this._statQueue.idle()) {
    this._finalize();
  }

  var self = this;

  return new Promise(function(resolve, reject) {
    var errored;

    self._module.on('end', function() {
      if (!errored) {
        resolve();
      }
    })

    self._module.on('error', function(err) {
      errored = true;
      reject(err);
    })
  })
};

Code Snippet 2:
New version

source: https://github.com/archiverjs/node-archiver/blob/dd7f10d/lib/core.js#L759-L792

Bad interface changes

The most painful thing about the new code is not that it has the same name while returning something new, this can be forgiven, since the upgrade was not minor but major upgrade. Fair is fair in major upgrade methods can change and retain the same name, might not be the best, but can happen.

The thing that actually bad in my point of view, that it can return two completely different things. Till there is an if that is being triggered it returns the instance of the class, but once those are exhausted it returns a Promise.

Icing on the cake that in this case the instance that it returns is a stream, that we were just piping, but now at some point it will turn in to a Promise.

Generally it is a good rule of thumb that a method should always return with the same type, no matter on the logic inside. (Unless your method is a type selector or something like that :D )

For the above code I could imagine that the original finalize could have remained the same, always returning a stream. There could have been a new method created that would give back a promise and also retaining backward compatibility. E.g.: introducing an asPromised method.

Archiver.prototype.finalize = function() {
  if (this._state.aborted) {
    this.emit('error', new ArchiverError('ABORTED'));
    return this;
  }

  if (this._state.finalize) {
    this.emit('error', new ArchiverError('FINALIZING'));
    return this;
  }

  this._state.finalize = true;

  if (this._pending === 0 && this._queue.idle() && this._statQueue.idle()) {
    this._finalize();
  }

  return this;
};

Archiver.prototype.asPromised = function() {
  var self = this;

  return new Promise(function(resolve, reject) {
    var errored;

    self._module.on('end', function() {
      if (!errored) {
        resolve();
      }
    });

    self._module.on('error', function(err) {
      errored = true;
      reject(err);
    });
  });
}

Bad dependencies

Now over for the necessity of this npm package.

This package is used in NodeJS, so we don't send it to do magic on the
front-end, but we use it in a controlled environment: a NodeJS app running in a Docker container wrapped in a Kubernetes pod.

For the docker container we use GNU/Linux, so we can have other programs in there as well, that can be found for Linux.

The files that we zip are on an NFS drive, so we don't need to stream it, we know the location of the file as well.

Here comes the idea:

we have linux ✅
we have the files/folders accessible ✅
we could have the zip program on the system ✅

So in effect we could axe a third party npm package and use child_process.exec or child_process.spawn method's from NodeJS itself. The extra 3rd party in this case would be the zip program, but think it is safe to say that command is a battle tested, fire hardened program. Which is not likely to cause issues or change interface. For me an extra benefit of this approach, that we can offload processing from the main app to a separate CPU thread aaaand we can also use GNU's nice utility to set the importance of the task. Like this we can also
ensure that the main app will always have enough CPU time to respond and the zip command won't eat it up from the app as well.

Gains with this:

zip is a reliable program that has and is being tested by millions
offload processing from the main app, so it can still serve other requests without an issue
use built-in NodeJS module that is not likely to change interface and also tested by more than a lonely lib from npmjs.org

Conclusion

It is good to think through as what is the goal and not just pick an npm package off the shelf. Most issues have already been solved by someone previously.

Originally I've posted this on https://szab.it/posts/bad-dependencies-and-bad-interface-changes/

Custom keyboard layout in GNU/Linux

szabi — Fri, 08 May 2020 15:00:05 +0000

I generally like to use english keyboard layout for programming, makes it easier to get to the special characters. However, since I'm a hungarian, from time-to-time would like to use accented characters as well. E.g. á, é, ű, ú, ő...

In GNU/Linux to change keyboard layout isn't the easiest to do. As far as I remember (from about 10 years ago) in Windows there is a graphical app that helps you with this. In GNU/Linux this is all via text files.

Here are the steps I've done:

add a new section in the us symbol file - /usr/share/X11/xkb/symbols/us:

partial alphanumeric_keys
xkb_symbols "euro-hu" {

    include "us(basic)"
    name[Group1]= "English (US, with Hun)";

    include "eurosign(5)"

    include "level3(ralt_switch)"

    key <AD07> { [            u,            U,        uacute,           Uacute ] };
    key <AD03> { [            e,            E,        eacute,           Eacute ] };
    key <AD08> { [            i,            I,        iacute,           Iacute ] };
    key <AD09> { [            o,            O,        oacute,           Oacute ] };
    key <AD10> { [            p,            P,    odiaeresis,       Odiaeresis ] };
    key <AC01> { [            a,            A,        aacute,           Aacute ] };
    key <AE10> { [            0,   parenright,    odiaeresis,       Odiaeresis ] }; // 0 ) ö Ö
    key <AE11> { [        minus,   underscore,    udiaeresis,       Udiaeresis ] }; // - _ ü Ü
    key <AE12> { [        equal,         plus,        oacute,           Oacute ] }; // = + ó Ó
    key <AD11> { [  bracketleft,    braceleft,  odoubleacute,     Odoubleacute ] }; // [ { ő Ő
    key <AD12> { [ bracketright,   braceright,        uacute,           Uacute ] }; // ] } ú Ú
    key <AC10> { [    semicolon,        colon,        eacute,           Eacute ] }; // ; : é É
    key <AC11> { [   apostrophe,     quotedbl,        aacute,           Aacute ] }; // ' " á Á
    key <BKSL> { [    backslash,          bar,  udoubleacute,     Udoubleacute ] }; // \ | ű Ű
    key <LSGT> { [    backslash,          bar,  udoubleacute,     Udoubleacute ] }; // \ | í Í
};

the new keyboard variant should also be noted in the following four files as
well:

/usr/share/X11/xkb/rules/base.xml:

<variant>
  <configItem>
    <name>euro</name>
    <description>English (US, euro on 5)</description>
  </configItem>
</variant>
<variant>
  <configItem>
    <name>euro-hu</name>
    <description>English (US, with Hun)</description>
  </configItem>
</variant>
<variant>
  <configItem>
    <name>intl</name>
    <description>English (US, intl., with dead keys)</description>
  </configItem>
</variant>

/usr/share/X11/xkb/rules/base.lst:

euro            us: English (US, euro on 5)
euro-hu         us: English (US, with Hun)
intl            us: English (US, intl., with dead keys)

/usr/share/X11/xkb/rules/evdev.xml - same way as base.xml
/usr/share/X11/xkb/rules/evdev.lst - same as base.lst

Restart the system and the new keyboard layout should be available - for gnome - in the Regional & Language setting's Input Sources section.

If you wish to dig deeper in the topic there is a good documentation about this over on a polish programmer's site Michał Kosmulski's "Creating custom keyboard layouts for X11 using KB" article.

There is also an option to make keyboard layout modifications with Xmodmap, but I've had some bad experiences with it. Namely it was slow to load the config, so rather do it via XKB. Possible that I was doing something bad with Xmodmap... but we will never no that now. :D

Originally I've posted this on https://szab.it/posts/custom-keyboard-layout-in-gnu_linux/

MongoDB deprecating "count"

szabi — Sun, 01 Mar 2020 08:22:00 +0000

Prior warning: this post may turn into a rant.

Prologue

In the second half of 2019 at my company we have got an expected notice from our MongoDB provider, Atlas.

The notice was about the usual pain that they do every now and then: force upgrade for old versions. At that time we were running MongoDB v3.4 and so now we got the notice to ensure that we have a driver that supports v3.6 as all the clusters will be upgraded on the end of January 2020.

All is well we look at these upgrades as a necessary evil, that causes pain in short therm, but will bring benefits in the long run. The benefits with newer MongoDB versions was the performance. We have tested some of our heavier queries - with which we already have had issues in production - and behold they become 10 times faster. (We were comparing MongoDB v3.4 with v4.2 at that time)

We thought cool 10x the power! Lets do this!

So we have started our long journey of upgrades, tests, fixes and further upgrades and tests, tears and cries, laughter and anger.

Once we were satisfied with the upgrade we have deployed our first services,that was already in need of that performance boost. Cool we thought, surely we gonna have some co-workers coming to us saying: Boys don’t know what has happened, but the service is blazing fast!

Man, we were wrong! Surely the queries looked fast, but there was a tiny bit of issue: some of our calls to the database started to timeout. Worst of it those calls were actually fast previously. As an icing on the cake this didn’t come to our attention straightaway, but only a week later, when another new service wanted to sync data.

Once noted, we jumped into debugging. Looking at the database’s real-time operations (db.currentOp()) we were seeing aggregation calls on the biggest collection being called. As we didn’t recall to have used such heavy aggregations on that collection, we searched through our code base to find what could issue this command.

We have managed to find a couple of places where we have used aggregation, but none of them fit the match that we seen in the operations list.

Eventually one team member suggested that that aggregation is the way MongoDB does the count. I couldn’t believe it at first, but then we have read a bit more about the new countDocuments method that was suggested method by the documentation to use instead of the count and turned out that it is indeed slower as it is more accurate.

From MongoDB’s JIRA ticket NODE-1638:

count has been deprecated for a number of reasons. First, and foremost, the existing count command could not be run within transactions, so an alternative approach needed to be taken. Secondarily, the count command in most cases was not accurate. We have replaced count with countDocuments and estimatedDocumentCount. Please use estimatedDocumentCount for the performance you are looking for, and parity with the original count.

So the reasons against count:

not giving accurate results and
not transaction friendly

From my point of view those two are not really reasons to deprecate a core command, which think is quite needed.

`count` is not accurate

Okay it isn’t, but honestly what was accurate in MongoDB before? Like with iterating a cursor (with mongoose stream), you could easily miss documents or see others twice in the process. Unless you set read preference to snapshot, but even then if the process is long running and you have inserts in the mean time,then you won’t see the new documents, so it is still a meh solution.

For processing all the data in the database, even the ones that didn’t exist when we started the process, we were using a practice where we sorted the _id in ascending order, retrieving data in batches and using the last _id in the list with a greater than filter: { _id: { $gt: lastId } }. Like this we could processes all the documents without duplicates and if there were new documents created while the process run, no problem, still got them.

Now in case of the count, so far I haven’t seen a case where would have needed pinpoint accuracy. I can imagine that there are case where one needs it, but then just like with the streaming above there is a solution for it. The solution in this case comes in aggregation and I’m sure that before the countDocumentscommand developers were using it to get the accurate count that they needed.

It is nice that now there is a method in mongo, that can give you the accurate count, without fiddling around with aggregation. It is convenient for those who need it. Still in my point it is not a reason to deprecate count.

Not transaction safe

Well okay. It isn’t. Don’t know, never tried it. As I tend to work with micro-services, I never missed or wanted to use transactions. It is hard to implement across services. My preference for data consistency is to make operations idempotent and so it is safe to put them into job queues, that guarantee to run it at least once, hence getting eventual consistency.

Just to emphasize it: I do respect that in some cases transactions could be the best or only solutions and it is nice that countDocuments is transaction safe. Just still not a reason to deprecate count.

Solution

So count has been marked as deprecated in MongoDB v4.0, it is still well and alive in v4.2. Since the two replacement suggested to be used instead:

countDocuments - way too slow for us
estimatedDocumentCount - cannot provide a query

are both unsuitable for us, we have reverted all our calls to use the poor old count method and we have accepted that our terminal will be showing the deprecation warnings for a while.

For now we hope that they won’t remove it or they will improve the performance of the new countDocuments method to be on pair with count.

Finale

Okay, this has indeed become a rant, but you have been warned. :D Sorry.

DEV Community: szabi

Responsible programming

TrueNAS & Nextcloud

Rewrite, refactor, knee-deep in legacy - part two

State

Kickoff

Frontend

Outro

Rewrite, refactor, knee-deep in legacy - part one

The System

Rewrite

Adaption of the new system

Different way

Outro

Time travel and privacy

Mail and Calendar

Chat applications

Mobile front

News sources

Outro

Bad dependencies and bad interface changes

Bad interface changes

Bad dependencies

Conclusion

Custom keyboard layout in GNU/Linux

MongoDB deprecating "count"

Prologue

count is not accurate

Not transaction safe

Solution

Finale

`count` is not accurate