DEV Community: Valentin Boettcher

Subfigures in RevTex

Valentin Boettcher — Sun, 16 Feb 2025 00:32:00 +0000

I’m in the process of writing a paper (yay!). Merrily I add a formula here a plot there; but my supervisor zeroes in immediately on a two-panel figure with sub figures called “a” and “b” saying:

The sub figure labels go in the top-left corner. You’ve read papers before, haven’t you?

Having indeed read papers before, I must have neglected to pay proper attention to sub-figure labels – an inexcusable blunder – so this was news to me. Unwilling to fix the issues by monkeying around in inkscape, I turned to stack overflow. We have the pleasure to be using the RevTeX document class and annoyingly this class happens to be incompatible with both the subcaption package and the part of subfloat that controls where the captions go. The aforementioned packages are “the way” to do sub figures and most other solutions I found involved monkeying around with minipages.

So that’s what I did; but I wanted to do properly so I’d be able to give the sub figures referenceable labels. I came up with a surprisingly short piece of hacked-together LaTeX package which we will now walk through. If you’re impatient, you can just grab the sty file and jump ahead to the usage example.

We begin by declaring the package.

\NeedsTeXFormat{LaTeX2e}
\ProvidesPackage{poormanssubfig}[2025/02/14 A subfigure environment
  with working labels in the top-left corner that is compatible with revtex.]

Then, we create a new counter for the subfigures to be able to generate use labels to reference them.

\newcounter{subfigure}

The multifig environment resets the subfigure counter¹, and redefines the \thesubfigure macro to display references to the subfigure as [Figure Number] ([figure sublabel]). The environment temporarily increases the Figure counter so that \thefigure prints the right number. The three commands at the top of the below snipped allow the user to overwrite the way the label is printed and formatted.

\newcommand{\subfiglabelformat}[1]{\raggedright{#1}}
\newcommand{\subfiglabelstyle}[1]{\alph{#1}}
\newcommand{\subfiglabelsize}{\small}
\newenvironment{multifig}%
{\addtocounter{figure}{1}%
  \setcounter{subfigure}{0}%
  \renewcommand\thesubfigure{\thefigure~(\subfiglabelstyle{subfigure})}%
}
{\addtocounter{figure}{-1}}

This is the point where the real hackery begins. We define an internal variables \if@insubfig whose use will become clear later. The subfigure environment takes two arguments, the first one being the width of the subfigure and the second being an optional label string (it’s also where you put \label{[label]}). The environment then creates a minipage with the appropriate width and prints the label of the figure in the top-left corner.

\makeatletter
\ExplSyntaxOn
\newif\if@insubfig
\NewDocumentEnvironment{subfigure}{mO{}}{%
  \@insubfigtrue%
  \refstepcounter{subfigure}%
  \begin{minipage}{#1}%
    \subfiglabelformat{{\subfiglabelsize (\subfiglabelstyle{subfigure})~#2}}%
    }
    {\end{minipage}%
  \@insubfigfalse%
}
\ExplSyntaxOff

This final bit is truly horrendous. I wanted to be able to refer to the subfigures in the figure caption using their labels, but a standard \ref{[label]} would print the figure number, too. So I stole some code from the cleveref package to redefine the \label macro to write a new label format (with the label name [youlabel]@multifig) to the auxiliary (.aux) file upon compilation. LaTeX reads the auxiliary file on the second pass and evaluates the \newlabel{[label]} macros therein. These tell latex which text to print when we use the macro\ref{[label]}. In our case this is just the sufigure counter value, without the figure number. I define a new reference command \subref to expand \ref but append @multifig to its argument, so that the text we stored using\newlabel is used. These extra labels will only be generated for subfigures thanks to \if@insubfig.

\let\multifig@old@label\label%
\def\label#1{%
  \multifig@old@label{#1}%
  \if@insubfig%
    \@bsphack%
    \protected@write\@auxout{}%
    {\string\newlabel{#1@multifig}{{(\subfiglabelstyle{subfigure})}%
        {\thepage}%
        {\@currentlabelname}%
        {\@currentHref}{}%
      }}%
    \@esphack%
  \fi
}%
\def\subref#1{\ref{#1@multifig}}
\makeatother

With these details out of the way, let’s get to the part that you actually wanted to read about²: The package is used in a similar way to – who’d have thought it – the subfigure package, but puts your label it the top left corner – where it ought to be! Just put the .sty file somewhere TeXwill find it. For global installation, you can put it in your$TEXMFHOME/tex/latex folder. For usage with a specific project, you can dump it next to your .tex file. And now, behold the below usage example.

\begin{figure*}
  \begin{multifig}
    \begin{subfigure}{\columnwidth}{\label{subfig:descriptive-label}Some
        text label.}
      \begin{center}
        \includegraphics[width=.8\columnwidth]{example-image-a}
      \end{center}
    \end{subfigure}
    \begin{subfigure}{\columnwidth}{\label{subfig:another-descriptive-label}Another
        text label.}
      \begin{center}
        \includegraphics[width=.8\columnwidth]{example-image-b}
      \end{center}
    \end{subfigure}
  \end{multifig}
  \caption{\label{fig:beautiful-figures}As the discerning eye may have
    spied, the figures \subref{subfig:descriptive-label} and
    \subref{subfig:another-descriptive-label} have the potential to
    rock the foundations of society.}
\end{figure*}
Total nonsense, but I couldn't think of anything better to demonstrate
Figure \ref{subfig:descriptive-label} and \ref{subfig:another-descriptive-label}.

This produces the below result...

… and references in the text look like

Cheers!

Which could also have been accomplished by an argument to \newcounter. ↩︎
Who are you dear reader? Shoot me an email! ↩︎

Hours of my life

Valentin Boettcher — Fri, 10 Jan 2025 08:57:00 +0000

have been wasted why the line spacing and DIV calculation (typearea) of KOMA script was hanging with LuaTeX.

TL;DR: LuaTeX and setspace don’t go together.

I stripped out the custom fonts, I removed (seemingly) every package, I ported my document to PDFTex. At long last after HOURS spent over several days if found the culprit. This time, I actually removed everything and there it was: \usepackage{setspace} was the culprit. Why did I include it? I don’t anymore and I wont spend any more time on this.

Kindle Scribe Sync

Valentin Boettcher — Sun, 17 Nov 2024 17:13:00 +0000

TL;DR check out https://github.com/vale981/kindle_fetch.

I “recently” treated myself to a Kindle Scribe e-ink tablet which is somewhat like a ReMarkable¹ but on the cheap. There is no free lunch however, and price you pay is a somewhat sub-par software. One quite essential missing feature is convenient synchronization with other devices. As I use the tablet mainly for calculations, I rely on having the previous pages available at a glance (just as they would be if I were using paper). Furthermore, I would like to integrate the handwritten notes into my org-roam note taking system.

One solution would be to jailbreak the kindle, ssh into it and automatically download a notebook when it changes. The conversion to PDF could then be handled by the Calibre KFX input plugin. However, that ship has sailed for me as the firmware on my Kindle is too new.

This led me to come up with the following bodge. Amazon offers a service that converts a notebook to PDF and sends it to your email upon request and this we can exploit to our heart’s content. I wrote ascript which monitors your email inbox via IMAP² using aioimapliband detects incoming emails from Amazon containing download links for the exported notebooks. Once such an email is detected, the associated PDF is downloaded into a directory on the local file system (~/kindle_dump on my machine). The name is inferred from the contents of the email, although at the moment this doesn’t include any hint about which folder in the directory structure on the Kindle it came from. The email is then conveniently deleted so as to not clutter your inbox³.

Additionally, the latest downloaded PDF is copied into to a file of your choice (~/kindle_dump/.latest in my case). I then run zathura ~/kindle_dump/.latest which displays that PDF on my large 4K monitor and auto refreshes each time a new Kindle email comes in. I also configured Zathura to show four pages per row (set pages-per-row 4) so that I can see eight pages at a glance (see figure below).

Figure 1: An example of what my ugly handwriting looks like :),

Figure 2: A little demo (sped up).

Additionally, the following elisp snippet allows me to attach the latest note to an org-roam node with a couple of keystrokes.

(defun attach-kindle-note (text)
  "Attaches the latest kindle note to the current org buffer and links to it with TEXT."
  (interactive "sText: ")
  (let* ((name (org-id-new "kindle"))
         (tmp (format "/tmp/%s.pdf" name))
         (file "/home/hiro/kindle_dump/.latest.pdf")
         (text (if (string-empty-p text)
                   (format "handwritten note %s" (current-time-string)) text)))
    (copy-file file tmp t)
    (org-attach-attach tmp nil 'mv)
    (insert (format "[[attachment:%s.pdf][%s]]" name text))))

The integration into my note-taking system is the most important part the whole bodge for me.

Annoyingly, Amazon has already changed the email format twice since I initially hacked together this solution. However, each time I come up with a more robust way to get the PDF download link and one’s free time has to be spent in some way or another anyways.

And if you’re reading this sentence, you’ve made it to the end of this rather redundant post. Thank you for reading. I hope that at least some people find this bodge useful. If there’s sufficient interest, I might add OAuth2 support.

On the road-map for future (think of years, not months) posts are my mu4e/mbsync setup with OAuth support on NixOS, my Xournal++ setup with vim-like keybindings and a patch to make the output files more compatible with version management and finally, my org-roam braindump.

In other news, my fork of julia-vterm is now calledpy-vterm-interaction and available on MELPA.

Years ago, I sold my ReMarkable in to buy a cheap drum-set and used a graphics tablet instead. I have a pretty neat set-up for that, too but that’s a tale for another time. ↩︎
Unfortunately, OAuth is not supported, so GMail/Outlook won’t work out of the box. ↩︎
I additionally have a Sieve filter running on my server that automatically sorts the Kindle emails into a specific folder, although that is not strictly necessary anymore. You might want to setup a separate email account for this purpose altogether if you’re not comfortable with having a (potentially buggy) python script mucking around in your emails. ↩︎

Modern Python REPL in Emacs using VTerm

Valentin Boettcher — Sat, 11 May 2024 23:47:00 +0000

As alluded to in Poetry2Nix Development Flake with Matplotlib GTK Support, I’m currently in the process of getting my “new” python workflow up to speed. My second problem, after dependency and environment management, was that fancy REPLs like ipython or ptpython don’t jazz well with the standard comint based inferior python repl that comes with python-mode. One can basically only run ipython with the --simple-prompt flag which removes features like syntax-highlighting and auto-completion. Especially annoying is, that only the tkinter backend for matplotlib works in this mode.

The package elpy comes with some improvements, especially when it comes to sending part of a buffer to the repl, but it comes with all sorts of baggage that interfere with my emacs setup.

From my jolly Julia days I’m used to julia-vterm. This emacs package runs a Julia REPL using a full terminal emulator (emacs-libvterm). So in the pursuit of a nice hack, I M-x replace-string’d the word juliawith python and gave it a shot. Remarkably, the whole thing just worked without much tweaking and you can enjoy the result by checking out the GitHub repo.

The idea of extending the original julia-vterm package to support python as well is not without elegance. However, the code base is not too large and – owing to the differing sensibilities of the julia and python communities – the feature-set is likely to diverge in the future.

Figure 1: And this is the result!

Poetry2Nix Development Flake with Matplotlib GTK Support

Valentin Boettcher — Sat, 11 May 2024 21:57:00 +0000

I recently had the pleasure to dive back into python for work. In the past, I was happily using org-babel notebooks through emacs-jupyter. However, I have since switched to a more REPL/script driven workflow as I find that programming notebooks require a great deal of discipline to not end up as a horrible mess. For my new workflow, I need interactive plotting to work.

So let’s get straight to the meat. The following Flake dives you a development shell that tries to replicate the underlying poetry project in full nix using poetry2nix.

{
  description = "[your description]";

  inputs = {
    flake-utils.url = "github:numtide/flake-utils";
    nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable-small";
    poetry2nix = {
      url = "github:nix-community/poetry2nix";
      inputs.nixpkgs.follows = "nixpkgs";
    };
  };

  outputs = inputs @ { self, nixpkgs, flake-utils, ... }:
    flake-utils.lib.eachDefaultSystem (system:
      let
        pkgs = nixpkgs.legacyPackages.${system};
        poetry2nix = inputs.poetry2nix.lib.mkPoetry2Nix { inherit pkgs; };
      in
      {
        packages = {
          yourPackage = poetry2nix.mkPoetryApplication {
            projectDir = self;

            # set this to true to use premade wheels rather than the source
            preferWheels = false;

            # this enables interactive plotting support with GTK
            overrides = poetry2nix.overrides.withDefaults (final: prev: {
              matplotlib = with pkgs; prev.matplotlib.overridePythonAttrs 

                {
                  passthru.args.enableGtk3 = true;
                };
            });
          };
          default = self.packages.${system}.yourPackage;
        };

        # Shell for app dependencies.
        #
        # nix develop
        #
        # Use this shell for developing your app.
        devShells.default = pkgs.mkShell {
          inputsFrom = [self.packages.${system}.yourPackage];
          package = with pkgs; [
            # any development dependencies that you might have in nixpkgs
            ruff
            pyright
          ];
        };

        # Shell for poetry.
        #
        # nix develop .#poetry
        #
        # Use this shell for changes to pyproject.toml and poetry.lock.
        devShells.poetry = pkgs.mkShell {
          packages = [pkgs.poetry];
        };
      });
}

The workflow is as follows. Running nix develop .#poetry will give you a shell with poetry available. You can then poetry init and poetry addand poetry lock (not install) to your hearts content. A plain nix develop will then set up the environment according to the poetry.lockthat poetry generates. Note that this pull request has to be resolved before the above works with preferWheels = true.

You might want to checkout direnv and nix-direnv for added convenience.

KDE GSOC: Wrapping it up…

Valentin Boettcher — Mon, 16 Aug 2021 16:53:00 +0000

Well, we all know that the work on open source projects is never truly finished, but most of the core goals have been achieved and the time is up :). In this post I’ll briefly summarize my GSOC work and then talk about one last small but user-facing feature that I’ve implemented.

I’ve successfully implemented a new DSO backend and smoothed out most of the bugs. The python framework does work satisfactory and all existing catalogs have been ported. There remains the UGC catalog which will be imported in the future, either by me or by another member of the project. The latter option would be a good way to battle-test the documentation and I would prefer this option because I do not want to remain the only person familiar with the system.

To quantify my contributions during the GSOC period see the snippet below, although I do not think such numbers have much to say¹.

Valentin Boettcher <hiro@protagon.space>:
       insertions: 15193 (19%)
       deletions: 23402 (35%)
       files: 312 (21%)
       commits: 76 (23%)
       lines changed: 38595 (26%)

Furthermore there is the list of my merge requests which does go into more detail.

The user-facing side of my work is not very prominent. There is a small GUI for managing catalogs that allows importing, exporting, creating and editing catalogs.

There is also a basic CSV importer that should make it easier for users to get their own custom data into KStars.

Figure 1: The CSV importer. It sure needs some prettying up :P.

Nevertheless, the main goal of my work was to create a seamless replacement for the old DSO system of which the user should not be too aware. To that end, I’ve implemented a feature that should have been in my overhaul from the beginning: a mechanism to import custom objects from the old DSO database. Now, on startup the user is being asked whether the old database should be imported if it is present.

And finally: Colors!

The DSOs always had a distinct color depending on the catalog they’re from. Right from the outset one complaint from early testers were the garish colors that I chose for the catalogs. I “fixed” this problem by simply choosing more subdued colors. But colors are a matter of personal taste. Also, a single color can’t fit all of KStars' color schemes. Therefore colors can now be customized for each catalog and color scheme through a “pretty” dialog.

Figure 2: The “pretty” color picker.

Now you can do things like this:

Figure 3: Color Scheme: Moonless Night

Figure 4: Color Scheme: Starchart

And again I’ve learned that user feedback is very important. I would never have thought of this feature on my own but must admit that it enhances the usability of KStars greatly.

With that oddly specific foray into the world of colors I now conclude this blog post and thank you for your attention.

Cheers, Valentin

I deleted the old OpenNGC text catalog which contained more than ten thousand lines :P. ↩︎

KDE GSOC: Second Coding Period; Some Notes on the Catalog Repo.

Valentin Boettcher — Sun, 08 Aug 2021 10:15:00 +0000

TL;DR DSO catalogs in KStars are now generated reproducibly in the CI. A list of available catalogs and documentation can be found here.

As promised last time I’ll now go a little into the Catalogs Repository.

Usually DSO catalogs are pretty static and rarely change due to the nature of their contents. But although galaxies do not tend to jump around in the sky, catalogs still get updates to correct typos or update coordinates with more precise measurement. Our primary catalogOpenNGC for example gets updates quite regularly.

Figure 1: OpenNGC is being updated regularly.

And even though a catalog might not change, it would nevertheless be desirable to have a record on how it was derived from its original format in a reproducible way¹. Last but not least, having all catalogs in a central place in kind of the same format would make deduplication a lot easier.

The question is: how does one define a convenient yet flexible format that nevertheless enforces some kind of structure? My answer was: with some kind of package definition. What about the flexibility part? Well, basically every catalog is just a python module that must implement a class. By overwriting certain methods, the catalog can be built up. The framework provides certain support functionality and an interface to some catalog database features by way of a python binding to some KStars code. Apart from that one has complete freedom in implementing the details although some conventions should be followed².

A simple random catalog looks like the following listing.

def generate_random_string(str_size, allowed_chars=string.ascii_letters):
    return "".join(random.choice(allowed_chars) for x in range(str_size))

class RandomCatalogBase(Factory):
    SIZE = 100
    meta = Catalog(
        id=999,
        name="random",
        maintainer="Valentin Boettcher <hiro@protagon.space>",
        license="DWYW Do what ever you want with it!",
        description="A huge catalog of random DSOs",
        precedence=1,
        version=1,
    )

    def load_objects(self):
        for _ in range(self.SIZE):
            ob_type = random.choice(
                [ObjectType.STAR, ObjectType.GALAXY, ObjectType.GASEOUS_NEBULA]
            )
            ra = random.uniform(0, 360)
            dec = random.uniform(-90, 90)
            mag = random.uniform(4, 16)
            name = generate_random_string(5)
            long_name = generate_random_string(10)

            yield self._make_catalog_object(
                type=ob_type,
                ra=ra,
                dec=dec,
                magnitude=mag,
                name=name,
                long_name=long_name,
                position_angle=random.uniform(0, 180),

It implements only the load_objects build phase and is a kind of minimum viable catalog.

The basic idea behind the structure of a catalog implementation is that the build process can be subdivided into four phases which can be partially parallelized by the framework.

In the download phase each catalog defines how its content may be retrieved from the Internet or otherwise acquired. In the load/parse phase the acquired original data is being parsed and handed over to the framework which takes care of molding it into the correct format. During the deduplication phase each catalog can query the catalog database to detect and flag duplicates. And in the final dump phase the contents of each catalog are written into separate files which KStars can then import³.

If you are interested in the details I can recommend the documentationfor the catalog repository.

After implementing the framework porting over all the existing catalogs to the new system, I went on to configure the KDE Invent CI to rebuild the catalogs upon changes. The CI artifacts are sync-ed to the KNewStuff data server for KStars periodically and users are able to update their catalogs to the latest version.

To get the CI working I had to create a Docker image that encapsulates the more or less complicated build process for the KStars python bindings. This container is updated weekly by CI and is also suitable as a quick-and-easy development environment for new catalogs.

That’s it for today but do not fret. This is not all that I’ve done. There’s still more to come including something that has to do with the following picture.

Cheers, Valentin

And in a way that hopefully lasts for some time. Currently very few people know how to generate KStars' deep star catalogs… ↩︎
I haven’t yet worked those out yet TBH. ↩︎
The catalog package files actually do have the same format as the main DSO database :). ↩︎

GSOC: Second Coding Period; Some Notes on the Catalog Repo.

Valentin Boettcher — Sun, 08 Aug 2021 10:15:00 +0000

TL;DR DSO catalogs in KStars are now generated reproducibly in the CI. A list of available catalogs and documentation can be found here.

As promised last time I’ll now go a little into the Catalogs Repository.

Figure 1: OpenNGC is being updated regularly.

A simple random catalog looks like the following listing.

def generate_random_string(str_size, allowed_chars=string.ascii_letters):
    return "".join(random.choice(allowed_chars) for x in range(str_size))

class RandomCatalogBase(Factory):
    SIZE = 100
    meta = Catalog(
        id=999,
        name="random",
        maintainer="Valentin Boettcher <hiro@protagon.space>",
        license="DWYW Do what ever you want with it!",
        description="A huge catalog of random DSOs",
        precedence=1,
        version=1,
    )

    def load_objects(self):
        for _ in range(self.SIZE):
            ob_type = random.choice(
                [ObjectType.STAR, ObjectType.GALAXY, ObjectType.GASEOUS_NEBULA]
            )
            ra = random.uniform(0, 360)
            dec = random.uniform(-90, 90)
            mag = random.uniform(4, 16)
            name = generate_random_string(5)
            long_name = generate_random_string(10)

            yield self._make_catalog_object(
                type=ob_type,
                ra=ra,
                dec=dec,
                magnitude=mag,
                name=name,
                long_name=long_name,
                position_angle=random.uniform(0, 180),

It implements only the load_objects build phase and is a kind of minimum viable catalog.

The basic idea behind the structure of a catalog implementation is that the build process can be subdivided into four phases which can be partially parallelized by the framework.

If you are interested in the details I can recommend the documentationfor the catalog repository.

That’s it for today but do not fret. This is not all that I’ve done. There’s still more to come including something that has to do with the following picture.

Cheers, Valentin

And in a way that hopefully lasts for some time. Currently very few people know how to generate KStars' deep star catalogs… ↩︎
I haven’t yet worked those out yet TBH. ↩︎
The catalog package files actually do have the same format as the main DSO database :). ↩︎

KDE GSOC: Community Bonding and First Coding Period (May 17 - July 11)

Valentin Boettcher — Sun, 11 Jul 2021 13:00:00 +0000

Of course the task I described in the last post looks and is quite monumental. That is why I laid some of the groundwork for my GSOC beforehand (in the actual German semester breaks). This work continued in the community bonding and first coding period and will therefore be described here.

But first I want to thank my mentor Jasem Mutlaq for his support, his patience with me and his nerves of steel. My mood levels were somewhat similar to a huge-amplitude sine wave those last weeks.

Now to the meat…

I began by studying the existing deep sky object implementation in KStars to identify what structure the new catalogs should have and what the smallest irreducible core of functionality was I could replace to make integration easier. I discovered that the catalogs were a mix of SQL databases and text files, somehow loaded at startup and then appended to some linked list. There was some deduplication implemented but like most DSO code it was oddly catalog specific. Especially the Messier, IC and NGC catalogs were often mentioned in the code. Also the explicit distinction between stars and DSOs made writing general code complicated but I found a consistent set of data fields shared by all catalog objects which all admitted sane defaults. It wasn’t bad code or anything like that. Just the product of “organic groth” with many thing I wanted already present in some way but somewhat all over the place. I admit that I studied the code just enough to find out what exactly I had to replace and maybe I could have reused more of the existing code but I’ve picked this specific path in the multiverse, so let’s get on with it. Just a shout out to all who did previous work on the DSO code among whom are, just to name a few, Jason Harris, Rishab Arora, Thomas Kabelmann and Akarsh Simha.

With this knowledge I was able to go forward and devise a concrete plan for implementing the new DSO system. First of all, albeit I would love to use std::variant and some kind of entity component system for the different DSO types I settled with a one-for-all type for deep sky objects. The primary reason for this was, that KStars uses C++14which lacks variants (and the extremely usefulstd::optional). Furthermore the DSOs all share common structure, so this was just the simpler and thus preferable option. The second design decision was not to load all of the DSOs into memory, but instead to take inspiration from the deep star catalogs. For one they are dynamically loaded from a special trixel indexed format so this already was within the formulated goals of the endeavor. On the other hand the notion of having “canonical” copies of catalog objects in memory and syncing their mutation with the database system seemed overly complicated. The catalog database should be the single source of truth and not the (ephemeral) memory of KStars.

When a specific object is needed, it should just be retrieved from the database locally in the code instead of searching some in memory list in KStars or shooting around with pointers. This notion is somewhat at odds with how things were and are done in KStars which created some interesting problems later on as we shall see. These ideas somewhat dictated the rest of the plan which I (for the first time in my programming career) completely wrote down in advance. The heart of it all is the database manager which abstracts maintaining, reading from and writing to the database. As always one should justify the creation of a special data type. In this case it was the requirement that the database access should be painless and could be handled locally anywhere in the code just by creating another instance of the database manager. The manager should handle retrieving objects and catalog meta information as well as importing, editing and exporting catalogs.

The structure of the database itself was another point of consideration. Naturally each catalog should have its own table. But how should deduplication work? The method I settled on is really quite simple. Each object gets a (relatively stable) hash that is calculated from some of its properties which is henceforth called the ID. When two objects (from different catalogs or otherwise) are the same physical object, then they will both be assigned the same object id (OID) which is just the ID of the object in the “oldest” catalog (with the lowest catalog id), trying to make it stable under the introduction of new catalogs. Additionally each catalog is assigned a priority value which is just a real number (conventionally between zero and one). When loading objects from the database into KStars and there are multiple objects with the same OID only the one from the catalog with the highest priority will be loaded. This simple mechanism should cover the requirements of KStars quite well and is relatively easy to implement.

There I ran into an issue that demanded some research and table in the database. The simplest option would be just to create a benchmarking. Remember that each catalog is represented by its own so-called view, a dynamic “virtual” table that combines all the catalog into one homogeneous table. SQL magic could automatically perform the deduplication algorithm outline above and everything would be fine and dandy. However, benchmarking revealed that actually writing the view into its own table, henceforth called the master catalog/table, increased the performance quite considerably, enough so to justify the increased complexity in the implementation. And then I discovered SQL indexes. A gift from the heavens! They increased performance on loading objects in a trixel from the master catalog roughly threefold and I was sold on the master catalog approach. So to summarize it all; a deduplicated view of the combined catalog tables is being created and then written into the master table. This has to be done for every modification of the catalogs but is relatively fast (just not fast enough to be done 20 times per second). Later experimentation showed that this approach could accommodate catalogs up to a million objects in size.

I also created a catalog file format, which is just an sqlite database file with the application id set to a special value with almost the same structure as the catalog database proper. The application id enables KStars to check if the database is really a catalog file and not to rely just on the structure of the contained database for that. In the future the file command and other utilities like file managers could be made aware of this special application id to recognize the catalog files. We will leave it this level of detail for now. For more details please refer to my notes.

Of course the operations on catalogs have to somehow be accessible in the GUI of KStars so this was another point of action. Before that however the glue between the database manager and the usual sky composite system had to be implemented. In KStars different types of objects (Stars, Comets, Asteroids, etc. pp.) all are implemented as components with a unified interface. These components provide methods for loading and drawing objects, as well as utilities to find objects near a certain point on the sky and similar things. The loading and drawing part was relatively simple to implement. The drawing code could be straight up reused from the old implementation and the loading was essentially covered by the database manager but with a twist.

To support very large catalogs it would be desirable to only have objects in memory which are currently visible. Thus a LRU cache was implemented with the trixel id, which essentially labels a portion of the sky, as key. This cache is fully unit tested and relies completely on standard library containers so not a single pointer appears in the code.¹ As an added bonus, the cache is completely transparent by default and only takes effect if configured to so and therefore includes the typical use case of comparatively small catalogs up to ten-thousands of objects.

But here the culture clash between the new DSO implementation and the traditional KStars way of things became apparent. In many places KStars expects pointers to so called SkyObjects with no real clue as to where they are actually stored and how their memory is managed and with the implication that the object is expected to live forever. Well, the DSOs from the catalogs aren’t supposed to be kept around forever and thus a compromise is in order. So whenever a pointer to an object is required, it is inserted into a linked list² in a hash table with the trixel as index or is taken from there if it’s already present. I hope that we can eventually transition away from raw pointers and manage life time either explicitly or with smart pointers.

With this done and basic drawing working I went on to implement a basic GUI for catalog management³. I also wrote unit tests for the database functionality which proved itself as very useful later on. After that I couldn’t delay anymore. Back when I implemented the component for the new DSOs I went as far as getting it to compile and not much further⁴.

Now I had to go around and find out what broke. A lot broke and I did not find all of it until the big merge :P. A rather interesting source of work happened to be the way metadata like observation notes and image links were stored. They came from a text file and then were loaded into the sky objects at startup and somehow synchronized with the text files on mutation. This, of course, played not well with the new DSOs as they were ephemeral. So I replaced the whole shebang with a hash table which incidentally improved startup performance. The rest of the integration work was similarly interesting and continues today. I will not go into it further but feel free to look at the KStars commit history.

Just yesterday I added a feature back in that I had axed accidentally to the dismay of its original author. That showed me that I am not entitled to judge the merit of individual features and whether they could be sacrificed for the “greater good”. The answer is: They cannot! Another lesson I’ve learned is, that too much magic just ain’t no good. I had created a variadic template wrapper for the QSqlQuerytype for syntactical convenience and shot myself in the food with it. It ended up obscuring an error message and prevented me from reproducing a crash that users on certain platforms were experiencing. After a not-so-great couple of days I, with the help of two kind people, finally found the lowest common denominator of the problem: an old, but still supported version of QT which bundled an old version of sqlite which in turn did not support the NULLS FIRSTdirective that I was using. Turtles all the way down. Although I tested all my changes on KDE Neon (I am on NixOS primarily) the wise thing would have been to develop or at least test everything with an older QT version from the get go.

Also, although I had put in version checking into the database code, I didn’t provide a mechanism for upgrading the database format to new versions. This I now remedied by introducing a simple mechanism that applies database modifications successively for each version upgrade. So if I go from version two to version four it will be upgraded from version two to three and then to four which I understand is the way those things are usually done.

Now, I did do at least some “constructive” work, adding a (admittedly ugly) CSV importer so that users can import arbitrary CSV-ish catalogs. The greater chunk however I will cover next week: The python catalog package tooling with continuous integration and deduplication. The catalogs churned out by that framework are then installed via the KNewStuff framework. I discovered two interesting bugs in this framework because KStars seems to be almost the only program using the framework in this specific way.

If you made it this far, I applaud and thank you for your endurance. See you next time.

Cheers, Valentin

P.S. Currently I am working on documenting both the new DSO GUI and the python tooling. I hope eventually they will pass the “noob test” :P. But, as you may have recognized above, I am not the best explainer.

As a matter of fact, I set out with the goal not to do any manual memory management and not to use a single pointer in the new code. I have been successful thus far if you would be so lenient not to count glue code for legacy KStars systems. ↩︎
References to objects in linked lists are stable. ↩︎
See the KStars Handbook. ↩︎
I really appreciate c++ as a compiled language. ↩︎

KDE GSOC: Intro

Valentin Boettcher — Sun, 27 Jun 2021 13:00:00 +0000

Hi folks, talking to you over the interwebs is Valentin Boettcher who is overhauling the Deep Sky Object (DSO) system in the KStars Desktop Planetarium for the Google Summer of Code anno domini 2021.

This is the first post in a series and rather late in the coming, so let’s get right to it.

I’m currently studying for a master’s degree in physics at the TU-Dresden in, you’ve guessed it correctly, the beautiful city of Dresden (Germany). In Germany, we do have two study terms per year and the summer term usually coincides neatly with the GSOC so that I couldn’t participate in past years. This time around however, my schedule was finally sparse enough for me to have a go at it, and here we are :).

My first contact with KStars development was back in 2017 while I spent a year in New Zealand and had a lot of time at hand. My reasoning was, that I could learn mathematics and physics in UNI and should funnel my enthusiasm into familiarizing myself with software development and the open source software community. I promptly wiped my hackintosh laptop to put Linux with KDE on it¹. After reading ESR’s famous “How To Become A Hacker”, I followed the advice given therein, which was to find an open source project and start hacking on it. I already liked KDE and space, so KStars was in the center of the Venn-diagram :P. I went ahead and busied myself with one of the junior jobs listed on the KStars web-site². I quickly found that I liked figuring out how stuff in KStars worked and also got in contact with my mentor Jasem Mutlaq who was always available to answer questions and endure my barrage of instant messages on matrix :P. My second job was to draw comets a tail and learned that it is wise to do some code archaeology before going ahead and implementing functionality that is already present. From there on I contributed more or less regularly when I found the time in my semester breaks.

Now, finally, let’s talk a wee bit about the actual GSOC project. In KStars, everything that isn’t a Star or an object in our solar system, an asteroid, a satellite or a comet (I’m sure I forgot something) is a deep sky object (DSO). Prominent members of the DSO caste are galaxies (think M31, Andromeda), asterisms and nebulae. Of course there are a plethora of catalogs for specific types of DSOs (for example, Lynds Catalog of Dark Nebulae) as well as compilations like the New General Catalogue. The system for handling those catalogs in KStars has grown rather “organically” and is now a tangle between databases, CSV files and special case implementations. Many catalogs were mentioned explicitly in the code, making it hard to extend and generalize. Also, the sources of the catalogs and methods how they were transformed into the KStars format were inhomogeneous and hard to reproduce, making deduplication almost impossible. Finally, KStars just loaded all the DSOs into memory and computed their position on the virtual sky for every draw cycle, which made all too large catalogs infeasible. My task is now (and has been since the beginning of June) to implement a unified catalog format which can be loaded into a central database and supports deduplication. Furthermore, taking inspiration from the handling of star catalogs in KStars, the objects should be trixel³indexed and cached in and out of memory (but only for large catalogs). Finally, it would be very desirable to make the creation/compilation of the catalogs reproducible and easily extendable to facilitate future maintenance.

This sounds like a big heap of stuff to get done and in the next post I will be detailing how it’s going so far :).

Cheers, Valentin

Assigning each object to a trixel makes it efficient to retrieve all objects from a certain part of the sky.

which I knew from my school time when I used it on my netbook because there was a cool neon “Hacker” theme for it :P ↩︎
which had to do with figuring out why some faint asteroids where missing ↩︎
In KStars the sky is subdivided into triangular pixels “Trixels”. ↩︎

Installing without Fear

Valentin Boettcher — Sun, 16 Aug 2020 15:32:10 +0000

Note to self:

If you want to make sure some nice GNU/Linux installer does not touch certain drives just run
echo 1 > /sys/block/sdX/device/delete in a root shell and the drive will vanish from the system.

Shamelessly stolen from: https://askubuntu.com/questions/554398/how-do-i-permanently-disable-hard-drives

Fixing Linux Dualboot: Reinstalling the Windows EFI Bootloader Files

Valentin Boettcher — Fri, 10 Jul 2020 12:31:45 +0000

Note to my future self :).

Reloading my Linux install after a pretty radical 'nuke and pave' I had to get my Windows dualboot back to work.
There are a thousand guides on how to do that, but I'll add another one in case your setup is similar to mine.

I have installed windows on a separate drive and Linux on my main drive, along with the efi partition.

Don't follow this guide blindly. Think about every step you take, because you can seriously mess up your system :).

With that out of the way, the things you have to do are:

Boot a windows install medium.
Choose your language and enter the 'repair options'.
Go to advanced and select 'command line'.
To mount the efi partition type diskpart and in diskpart then type list volume. A list of volumes will be printed and one of them the efi partition (usually around 500mb ). Select this partition (select volume [number]) and assign a drive letter (X is the drive letter you assign).
Check where your windows partition is mounted. The diskpart list volume output will probably include it. I will assume that it is volume C. Exit diskart with exit.
To finally install the boot files type the command bcdboot c:\windows /s x: . This will generate boot files based on c:\windows and install them on the partition with the letter X .

Thats it, you can reboot now.
You may have to reconfigure grub (or whatever loader you use). On arch-linux, make sure you have os-prober installed :).