DEV Community: Garrett Mills

Salvaging a Corrupted Table from PostgreSQL

Garrett Mills — Sat, 11 Jan 2025 22:04:25 +0000

This post originally appeared on my blog.

⚠️ DO NOT DO THIS... well ever really, but especially on a server with failing disks. This is done on a server with perfectly fine disks, but corrupted Postgres blocks.

I spend a lot of time in my professional work and my home lab trying to learn and implement the “correct” or “responsible” way of implementing a solution — highly-available deployments, automated and tested backups, infrastructure-as-code, &c.

This is not that.

This is a very dirty, no-holds-barred, absolutely insane thing to do, and if you’re working in any kind of environment that matters, you should read this and hire a professional.

For unimportant reasons, I’ve been dealing with data corruption on the Postgres server in my home lab. The server was terminated uncleanly a couple times and the disk data was corrupted. Because there’s nothing more permanent than a temporary solution, I did not have backups for this server.

For most of the data, I was able to use pg_dump to dump the schemata and data and re-import it into my new Postgres server (which, yes, has backups configured now).

pg_dump -U postgres -h localhost my_database > my_database.sql

For databases with corrupted tables, though, pg_dump fails out with this unsettling error:

> pg_dump -U postgres -h localhost www_p1 > www_p1.sql
pg_dump: error: Dumping the contents of table "page_views" failed: PQgetResult() failed.
pg_dump: detail: Error message from server: ERROR:  invalid page in block 31869 of relation base/16384/16417
pg_dump: detail: Command was: COPY public.page_views (page_view_id, visited_at, hostname, ip, method, endpoint, user_id, xhr) TO stdout;

(…yes, that’s the database for my personal website. 👀) Somewhat to my surprise, I couldn’t find many details/strategies for how to “best effort” recover data from a corrupt Postgres table, so here we go.

Luckily, since the corruption was the result of unclean Postgres exits and not bad physical disks, it only affected table(s) with frequent writes at the time. In this case, that was the sessions table and the page_views table. The sessions table is entirely disposable — I just re-created it empty on the new server and moved on with my life.

It wouldn’t be the end of the world if I lost the page_views table, but there are some 6.5 million historical page-views recorded in that table that would kind of suck to lose. So… let’s do some sketchy shit.

My goal here isn’t to recover the entire table. If that was the goal, I would’ve stopped and hired a professional. Instead, my goal is to recover as many rows of the table as possible.

One reason pg_dump fails is because it tries to read the data using a cursor, which fails when the fundamental assumptions of Postgres are violated (e.g. bad data in disk blocks, invalid indices).

My strategy here is to create a 2nd table on the bad server with the same schema, then loop over each row in the page_views individually and insert them into the clean table, skipping rows in disk blocks with bad data. Shout out to this Stack Overflow answer that loosely inspired this strategy.

CREATE OR REPLACE PROCEDURE pg_recover_proc()
LANGUAGE plpgsql AS $$
DECLARE
    cnt BIGINT := 0;
BEGIN
    -- Get the maximum page_view_id from the page_views table
    cnt := (SELECT MAX(page_view_id) FROM page_views);

    -- Loop through the page_views table in reverse order by page_view_id
    LOOP
        BEGIN
            -- Insert the row with the current page_view_id into page_views_recovery
            INSERT INTO page_views_recovery
            SELECT * FROM page_views WHERE page_view_id = cnt and entrypoint is not null;

            -- Decrement the counter
            cnt := cnt - 1;

            -- Exit the loop when cnt < 1
            EXIT WHEN cnt < 1;
        EXCEPTION
            WHEN OTHERS THEN
                -- Handle exceptions (e.g., data corruption)
                IF POSITION('block' in SQLERRM) > 0 OR POSITION('status of transaction' in SQLERRM) > 0 OR POSITION('memory alloc' in SQLERRM) > 0 OR POSITION('data is corrupt' in SQLERRM) > 0 OR POSITION('MultiXactId' in SQLERRM) > 0 THEN
                    RAISE WARNING 'PGR_SKIP: %', cnt;
                    cnt := cnt - 1;
                    CONTINUE;
                ELSE
                    RAISE;
                END IF;
            END;

        IF MOD(cnt, 500) = 0 THEN
            RAISE WARNING 'PGR_COMMIT: %', cnt;
            COMMIT;
        END IF;
    END LOOP;
END;
$$;

There are some cool and absolutely terrible things here. In modern versions of Postgres, stored procedures can periodically commit their in-progress top-level transactions by calling COMMIT repeatedly. I’m (ab)using this here to flush the recovered rows to the new table as the procedure runs in case it fails partway through.

I'm doing some rough string analysis for error messages related to corrupt data and skipping the current row if that's the case. Another interesting edge-case: a couple times, I ran into a case where the INSERT into the recovery table failed because the SELECT query against the bad table was returning null values, even though that should technically never be possible. Told you we're violating some foundational assumptions about Postgres here. Adding an is not null to a different non-null column helped avoid this.

My original draft of this procedure was designed to keep looping and just skip the fatal errors caused by disk corruption (the various dirty POSITION checks in the error handler).

Quickly, however, I ran into a new error:

SQL Error [57P03]: FATAL: the database system is in recovery mode

Turns out, if you keep intentionally forcing Postgres to try to read data from bad disk blocks, eventually its internal data structures hit an inconsistent state and the server process restarts itself out for safety.

This is (obviously) a problem because we can’t catch that and force the procedure to keep running against its will. So instead I resorted to adding IF conditions to manually skip over primary key regions that caused the server process to crash. (I told you this was crazy.)

Every time the server would crash, I would dump out the rows I’d recovered so far, just in case:

pg_dump -U postgres -h localhost --table page_views2 www_p1 > page_views2-1.sql

Then I’d skip a new region of primary key, drop and re-create the recovery table, and try again. Why drop and re-create it? Because I discovered that when the server process crashed, it would occasionally write bad data to the recovery table, which is obviously no good:

pg_dump: error: Dumping the contents of table "page_views_recovery" failed: PQgetResult() failed.
pg_dump: detail: Error message from server: ERROR:  invalid memory alloc request size 18446744073709551613
pg_dump: detail: Command was: COPY public.page_views_recovery (page_view_id, visited_at, hostname, ip, method, endpoint, user_id, xhr) TO stdout;

Predictably, this got really annoying to do by hand, so I did what any good Linux nerd would do and wrote a script for it, which you can find here. The gist:

./pg-recover.sh postgres localhost www_p1 page_views page_view_id entrypoint

Of the 6,628,903 rows in the corrupt table, I was able to recover 6,444,118 of them. You know what they say — if it’s stupid and it works, it’s still stupid and you’re just lucky.

Mitigating the iconv Vulnerability for PHP (CVE-2024-2961)

Garrett Mills — Tue, 23 Apr 2024 02:33:44 +0000

This post originally appeared on my blog, here.

Recently, CVE-2024-2961 was released which identifies a buffer overflow vulnerability in GNU libc versions 2.39 and below when converting charsets to certain Chinese Extended encodings.

This vulnerability affects PHP when iconv is used to translate request encodings to/from the affected charsets and has the potential to be wide-ranging (e.g. the latest wordpress:apache image has iconv with the vulnerable charsets enabled).

Obviously, the best mitigation is to update to a patched version of glibc. However, if you are unable to (or it's not available on your OS yet), you can mitigate this issue by disabling the affected charsets in gconv.

I had a really hard time finding information on how to check for and mitigate this issue at the OS-level (possibly because the researcher who discovered the CVE is presently teasing details about the PHP exploit for his talk at a conference... 3 weeks after the CVE was announced. 🙄)

I've collected my notes here, in case they might be useful for someone else.

Check if your OS is vulnerable

ldd --version

The first line of the linker version info should include the version of glibc (either as GLIBC or GNU libc).

Example from Debian 12:

ldd (Debian GLIBC 2.36-9+deb12u4) 2.36
Copyright (C) 2022 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Written by Roland McGrath and Ulrich Drepper.

Example from Rocky Linux 9:

ldd (GNU libc) 2.34
Copyright (C) 2021 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Written by Roland McGrath and Ulrich Drepper.

You can also use your package manager to check (for example, rpm -q glibc).

If you are using glibc 2.39 or older, then the ISO-2022-CN-EXT encodings are vulnerable for your system's iconv and gconv.

Check for bad encodings

Check if the vulnerable encodings are enabled in iconv:

iconv -l | grep -E 'CN-?EXT'

If they are, you will see an output like:

ISO-2022-CN-EXT//
ISO2022CNEXT//

Disable bad encodings

We can modify the gconv-modules configuration to disable the affected charsets:

cd /usr/lib/x86_64-linux-gnu/gconv

This might be in slightly different locations for exotic systems. Try find / -name gconv-modules.

Disable the offending encodings in the gconv-modules config file. This will either be in gconv-modules directly, or in something like gconv-modules.d/gconv-modules-extra.conf:

cd gconv-modules.d
cat gconv-modules-extra.conf | grep -v -E 'CN-?EXT' > gconv-modules-extra-patched.conf
mv gconv-modules-extra-patched.conf gconv-modules-extra.conf
cd ..

Remove the cache file if present:

rm gconv-modules.cache

You can regenerate that cache file using iconvconfig(8).

Then re-check for the vulnerable encodings:

iconv -l | grep -E 'CN-?EXT'

There should be no output from this command.

Docker

For those using Docker images, here's a convenient Dockerfile blurb:

# Disable vulnerable iconv encodings (CVE-2024-2961)
RUN cd /usr/lib/x86_64-linux-gnu/gconv/gconv-modules.d \
    && cat gconv-modules-extra.conf | grep -v -E 'CN-?EXT' > gconv-modules-extra-patched.conf \
    && mv gconv-modules-extra-patched.conf gconv-modules-extra.conf \
    && rm -f ../gconv-modules.cache \
    && iconvconfig \
    && iconv -l | grep -E 'CN-?EXT' && exit 1 || true

That last line contains one of my favorite Dockerfile tricks (check-something && exit 1 || true) -- your Docker build will fail if the vulnerable charsets are enabled.

A previous version of this post kept gconv-modules-extra-patched.conf. Thanks to Anonymous for pointing out that a subsequent RPM update could re-introduce the file.

A previous version of this post indicated that glibc versions < 2.39 were vulnerable. Thanks to Geert for noting that 2.39 is also vulnerable.

Down the Rabbit Hole of Linux Terminal Emulators

Garrett Mills — Fri, 05 May 2023 04:14:58 +0000

For years now, I've used an excellent drop-down terminal emulator called Guake -- a GNOME port of Quake. It supported every feature I needed: Wayland (mostly), tabs, custom shortcuts, custom shell, transparency, and a toggle command (guake -t) that plays nice w/ custom keybinds in GNOME.

Recently, however, I've found myself wanting something akin to Vim's visual mode in Guake (specifically the ability to copy-paste chunks of output like file names into the command).

While Guake doesn't support this, the screen command does:

Press C-a [ to enter copy mode.
Move the cursor to the text you want to copy and press space.
Highlight the text you want to copy and press space again to copy the selection and return to normal mode.
Press C-a ] to paste at the cursor.

This is exactly what I wanted. Now, I just need to make screen fish --login my default shell in Guake. This presented a couple issues.

First, because I am using a GUI-based terminal emulator, I want to use its scroll-back functionality. By default, screen buffers output and provides its own implementation of scroll-back.

Luckily, this was easy enough to change with a line in my ~/.screenrc:

termcapinfo xterm*|xs|rxvt|terminal ti@:te@

The next problem was... not so easy to solve.

Guake's preferences dialog.

While Guake allows you to select a custom shell, it does not allow you to specify a custom command. I need to do this because screen does not register itself as a shell w/ the system.

Sooooo using Guake is out of the question if I want to use my new Franken-screen shell setup.

GNOME Terminator

GNOME Terminator's preferences dialog.

I tried several other shell emulators, but they all had game-breaking issues or missing functionality for me personally. Most notably, I actually got GNOME's Terminator tiling emulator completely set up to my liking. When I tried to replace my keyboard shortcut for Guake, thought, I discovered that there's no CLI flag to "toggle" the current window to make it behave like a drop-down shell.

Tilix

Tilix's Quake mode, Wayland edition.

I also tried Tilix, another tiling emulator. Like Terminator, it had preferences for everything I like to customize in Guake -- it even has a Quake mode that makes it behave as a drop-down shell! Unlike Terminator, it has a CLI flag to show/hide the overhead window.

Unfortunately, Tilix has limited support for Wayland in Quake mode. I was able to get it to work by updating my keyboard shortcut to use the env GDK_BACKEND=x11 environment.

Ah, Wayland, how we love thee.

This too has its own problem. One feature of Guake that I use a lot is the "Open on monitor with cursor" setting. This lets me quickly move Guake between monitors as I'm working.

There is a well-known limitation of Wayland where Wayland apps are restricted from accessing certain information about the displays and other running applications. This prevents Tilix from being able to determine which monitor the active cursor is on.

This problem is not unique to Tilix. I use the excellent Ulauncher tool. On X.org, Ulauncher opens on the monitor where the cursor is. On Wayland, it picks the monitor seemingly at random.

While searching around for this issue, I stumbled upon a comment on an issue in the Tilix GitHub which pointed me to a GNOME extension of all things called DDTerm.

DDTerm

DDTerm and its glorious, glorious preference dialog.

DDTerm is "Another Drop Down Terminal Extension for GNOME Shell" and has managed to be a perfect drop-in replacement for Guake for me. It supports, among other things:

A custom command to use for new sessions (screen fish --login)
A DBus command to toggle a single instance (gdbus call --session --dest org.gnome.Shell --object-path /org/gnome/Shell/Extensions/ddterm --method com.github.amezin.ddterm.Extension.Toggle)
Transparency
Customizable keyboard shortcuts/color theme/fonts

Plus, because it's running in GNOME directly, it can access the compositor to correctly determine the monitor where the mouse lives!

Conclusion

This rabbit hole was probably 2 hours of on-again-off-again tinkering w/ various terminal emulators, but I have landed on:

Shell: Fish Shell wrapped in GNU Screen
Emulator: DDTerm
Custom .screenrc to enable literal scrollback
Custom GNOME keybinding to toggle DDTerm

This post originally appeared on my blog, here.

Adventures in AI-assisted proof generation

Garrett Mills — Mon, 20 Mar 2023 12:00:00 +0000

Look, I know LLMs and ChatGPT are the big industry-consuming buzz-words right now. I'm as skeptical of the "ChatGPT is going to replace {industry}" mentality as anyone, but it's clear that something is coming and it's going to change the world, whether we like it or not.

Believe it or not, until a couple days ago, I had largely ignored the LLM craze. Sure, it's cool, and it's coming, but it's not relevant to what I do day-to-day yet. Recently, however, I was reading the excellent Web Curios blog by Matt Muir, and something he said stood out to me:

“But Matt!”, I hear you all cry, “what does this MEAN? Please tell me what I should think about all this breathless, breakneck upheaval and change and…is it…progress?” To which the obvious answer is “lol like I know, I’m just some webmong” (and also “is it normal to hear these voices?”), but to which I might also say “This is the point at which I strongly believe it’s important you start to learn how to use this stuff, because I reckon you’ve got maximum a year in which reasonable competence with this sort of kit makes you look genuinely smart”, but also “GPT4 is literally Wegovy for white collar office workers, insofar as everyone will be using it to give themselves a professional tweakup but people will be a bit cagey about admitting it”

So, begrudgingly, I fired up OpenAI, handed over my e-mail and phone number, and started playing around with ChatGPT. I tried all the normal things ("generate a Bash script to do X", "write a new cold open for X show in the style of Y"), but I stumbled upon something interesting, rather by accident.

The A Problem With Coq

(If you are familiar with Coq/provers, feel free to skip my basic and somewhat controversial explanation in this section.)

When I was in school, one of my areas of focus was formal methods, a computing discipline that concerns itself with the rigorous theory of programming languages and programs. One of the major applications of FM is so-called "verified computing" -- computing which has been formally proven correct. Applications of verified computing are legion, including security-sensitive environments -- like embedded devices, drones, and spacecraft -- but have gradually extended to broader applications like secure boot and mainstream languages like TypeScript.

The FM community has coalesced around a series of tools called interactive theorem provers which provide syntax for modeling theorems, composing them, and constructing proofs over them. The lab I worked with used one of the most popular, the Coq Proof Assistant. Coq contains a rigid, functional programming language and a less-rigid proof scripting language which operates over it.

Despite advocates claims that provers are "going mainstream any year now," these proof assistants struggle in general -- and Coq in particular -- with a lack of automation. Quite extensive knowledge is required to prove even basic lemmas, largely due to the strict nature of the runtime and its unwillingness to make assumptions which may turn out to be logically inconsistent. Undoubtedly, this will improve with time as automations and layers of abstraction are introduced to the system, but it remains the largest barrier to entry for programmers wanting to write provably-correct code.

The A Problem with ChatGPT

Of the many technical, societal, and moral quandaries spurred by ChatGPT is its complete willingness to fabricate lies disguised as the truth. Because all the LLM is doing is generating what it deems the statistically most likely response, it may be incorrect. However, its advanced language syntax model enables it to present its incorrect information in an eloquent, tidy, convincing format (the boon of misinformation farms, but I digress...).

Obviously proving whether or not the output of ChatGPT is correct generally is an insane and likely impossible proposition, there's an interesting rabbit hole to be found on the boundary of AI-generated code snippets and Coq proofs. A wedding between the embodiment of industry buzzwords and something which is decidedly not. You see, it turns out that one of the niche languages that ChatGPT understands reasonably well is Ltac, the proof scripting language built into Coq.

Factorial

A common baby's-second-proof in Coq is the factorial problem. Model the factorial function as a recursive function and an inductive relation, then prove that the two models are equivalent. So, I started by asking ChatGPT:

"Show me an implementation of the factorial algorithm in Coq."

This is what it came up with:

Fixpoint factorial (n : nat) : nat :=
  match n with
  | O => 1
  | S n' => n * factorial n'
  end.

This compiled in Coq and, upon inspection, this appears correct. Impressive. Next, I asked it something a bit more challenging:

"Now, show me an implementation of the factorial algorithm in Coq using an inductive relation instead of a recursive function."

Inductive factorial : nat -> nat -> Prop :=
| fact_base : factorial O 1
| fact_step : forall n m, factorial n m -> factorial (S n) ((S n) * m).

This also compiled fine, and I think this is much more impressive. The LLM produced a reasonable model of a factorial system, "inferring" what I meant by "inductive relation." ChatGPT also helpfully provided an example lemma for this relation:

Example factorial_5 : factorial 5 120.
Proof.
  apply (fact_step 4 24).
  apply (fact_step 3 6).
  apply (fact_step 2 2).
  apply (fact_step 1 1).
  apply fact_base.
Qed.

This is where things started to get interesting to me. I didn't even ask for an example, but in its quest to explain itself, ChatGPT generated Ltac code demonstrating an example usage of its inductive relation. I was curious, so I chucked it into Coqtail and tried it out:

Unable to unify "factorial 0 1" with "factorial 1 1".

Hm. It looks like the ChatGPT proof missed a step between apply (fact_step 1 1). and apply fact_base. Adding in an apply (fact_step O 1). between them pops it out.

After my initial doubts in ChatGPT's "knowledge" of Coq, I started to get a little more ambitious:

"Now, show me a proof that the inductive relation is equivalent to the recursive function definition."

Lemma factorial_equiv : forall n, factorial n (factorial_rec n).
Proof.
  induction n.
  - apply fact_base.
  - simpl.
    apply (fact_step n (factorial_rec n)).
    apply IHn.
Qed.

Lemma factorial_rec_equiv : forall n, factorial_rec n = factorial n.
Proof.
  intros n.
  unfold factorial_rec.
  induction n.
  - reflexivity.
  - simpl.
    rewrite IHn.
    apply (fact_step n (factorial n)).
    apply factorial_equiv.
Qed.

Now, that's something! There are so many interesting things about this response. First of all, it inferred that a "proof of equivalence" would require a lemma in both directions. Second of all, the Ltac it generated has structure and is organized using bullets, not just a large blob of proof commands. Even more, it was able to account for the conflict between the recursive factorial function it generated and the inductive factorial relation it generated, renaming the recursive function factorial_rec.

To my amazement, the factorial_equiv popped out without any modifications, and the bullets structure even made sense. The factorial_rec_equiv lemma however... has some problems. For one, neither factorial_rec nor factorial is single-parameter, so the lemma was missing an entire universal variable. I rewrote the theorem manually, to give the LLM a hint:

"Modify the factorial_req_equiv lemma to prove: forall n m, factorial_rec n = m -> factorial n m"

Lemma factorial_rec_equiv : forall n m, factorial_rec n = m -> factorial n m.
Proof.
  intros n m H.
  rewrite <- H.
  clear H.
  induction n.
  - apply fact_base.
  - simpl.
    apply (fact_step n (factorial_rec n)).
    apply IHn.
    reflexivity.
Qed.

Aside from the fact that the last reflexivity is not needed, this theorem popped out perfectly. Now, I not only had an AI-generated factorial relation, but I had an AI-generated proof that that relation was correct (or at least equivalent to the recursive definition I defined as "correct").

This is absolutely amazing to me, but of course there are caveats to note:

First, this is an exceedingly simple example, and as you add libraries, lemmas, and other code it would become difficult to provide ChatGPT enough context for it to generate coherent results.
Second, ChatGPT's generation of these outputs was slow. So slow, in fact, that several times when I returned to the tab from something else, it had timed-out and needed to be retried. But, it did finish.
Third, obviously my definition of "proven correctness" assumes logical consistency in the proof engine, but in this line of work that's a fair assumption to make.

Importantly, however, were I proving this for the first time, it would probably take more time to work my way through the various proof tactics and commands.

I think this has a few interesting implications: first, for theorem proving. The barrier to entry for theorem provers is largely an issue with how difficult it is to construct proof tactics. If ChatGPT generates a string of proof tactics which pops out a proof goal, then one of two things has happened. Either, (1) it has found a set of tactics which genuinely proves the statement, or (2) it has found a logical inconsistency -- a bug -- in the proof engine.

The latter is the stuff of nightmares, and is generally not much of an issue in mature tools like Coq. The former, however, is really cool. Imagine a utopic world where you can hand a proof and a universe of statements to your theorem prover and, rather than requiring carefully constructed hints, the auto tactic could use an LLM to generate the tactics for the proof. Even if it fails, it provides the user a much closer starting point than writing the proof from scratch.

There's a much broader and more interesting implication here, though. One problem with ChatGPT-generated code is that it may simply be completely wrong. So, what if we make ChatGPT prove that its code is correct? This, combined with Coq's excellent extractors, could free up not just FM people, but even normal developers to create verified implementations while spending less time in the weeds with the prover. auto on crack.

Conclusion?

I honestly don't know how I feel about ChatGPT in general, or even how it behaves in this particular situation. Maybe it simply does not scale to the complexity necessary to be useful. Maybe a rise in AI-generated Ltac will poison yet another vibrant FOSS community. It really feels like there's a social reckoning on the way.

Regardless, it seems like ChatGPT's usefulness as a programming tool may extend -- just a bit -- to the world of verified programming.

(I created a repo where you can find the generated Coq code as well as the output from my ChatGPT session.)

This post originally appeared on my blog, here.

Generalized Commutative Data-Types

Garrett Mills — Fri, 09 Dec 2022 10:58:26 +0000

Disclaimer: after I started writing about this, I found this paper from the Hydro project which presents a formulation of this idea using lattices & morphisms. What follows is my derivation of a similar technique, albeit significantly less formal. As far as I can tell, the Hydro paper does not separate "pseudo-commutative operations" instead opting to form reactive values which are re-computed as the PC operations are applied.

A "commutative data type" is one whose value is modified by a set of operations whose execution order is irrelevant. Such data types are useful in distributed & parallel systems which employ accumulator-style execution (i.e. many jobs perform a calculation then merge their result into a single, shared value).

What follows is a formulation of such a data type along with the structures required to define various operations over it.

Naïve Commutative Data Types: A First Draft

Begin with a base value v of some type T.

There are n many jobs which act in parallel to perform commutative operations on v.

An operation on v is commutative if, for all operations a and b of type (T -> T), a (b v) = b (a v).

Each of the jobs produces an operation of the type (T -> T) which are collected.

The result is a value (v : T) and a set of commutative operations (list (T -> T)).

The list of operations is applied to the value, chained, producing a final value v'.

For example, if we have the list ((v -> v+1) :: (v -> v+4) :: nil) and a base value of 0, resolving the value gives (v -> v+4) ((v -> v+1) 0) = 5.

Importantly, because the order of the operations is irrelevant, we can apply the operations as they are received by the reducer (the piece of software accumulating the result), rather than collecting them all at once.

This allows for efficient reduction of a shared result variable by many distributed parallel jobs.

Pseudo-Commutative Operations

Some operations, however, are not purely commutative. An example of this is multiplication.

If we introduce a job which produces a multiply operation into the above example, the list of operations is no longer commutative (herein referred to as asymmetric, or inconsistent).

However, the operation of multiplication is distributive in the sense that the TFAE:

c * (a + b)
(c * a) + (c * b)

Or, perhaps more interestingly for our case, TFAE:

d * (a + b + c)
(d * c) + (d * (a + b))

For example, say we receive the following operations in the following order. A C: prefix denotes a commutative operation, and a P: prefix denotes a pseudo-commutative operation:

C: (v -> v + 1)
C: (v -> v + 2)
P: (v -> v * 2)
C: (v -> v + 3)

If the base value of v is 0, we find that the "consistent" result should be (0 + 1 + 2 + 3) * 2 = 12.

(v -> v + 1) 0  => 1
(v -> v + 2) 1  => 3
(v -> v * 2) 3  => 6

(v -> v + 3) 6  => 9   (incorrect)
(v -> (v + 3) * 2) 6  => 18   (incorrect)
(v -> v + (3 * 2)) 6  => 12   (incorrect)

Because the commutative operation is opaque, there is no way of "pushing" the pseudo-commutative operation into the subsequent commutative operations, resulting in asymmetric results.

To address this, we re-define our naïve commutative operations like so:

A commutative operation is pair of the form (T, T -> T -> T) where the first element is the right-operand to a commutative binary operation. The second element is a function which takes the current accumulated value and the right operand and returns the new accumulated value.

This structure removes the opacity of the right operand in the operation, allowing us to push the pseudo-commutative operation into subsequent commutative operations.

We similarly re-define PC operations to have the form (T, T -> T -> T).

Now, the same example using the new structure:

C: (1, l -> r -> l + r)
C: (2, l -> r -> l + r)
P: (2, l -> r -> l * r)
C: (3, l -> r -> l + r)

This results in:

(l -> r -> l + r) 0 1  => 1
(l -> r -> l + r) 1 2  => 3
(l -> r -> l * r) 3 2  => 6
(l -> r -> l + r) 6 ((l -> r -> l * r) 3 2)  => 12

This is the fundamental insight of pseudo-commutative operations: if they are folded into the operands of all subsequent operations applied to the accumulator, the ordering of commutative and pseudo-commutative operations is irrelevant (insofar as the correct pseudo-commutative folds are performed).

Pseudo-commutative operations can even be chained to arrive at similarly-consistent results:

C: (2, l -> r -> l + r)
P: (2, l -> r -> l * r)
P: (3, l -> r -> l * r)
C: (2, l -> r -> l + r)

The expected result here is (0 + 2 + 2) * 2 * 3 = 24, and is computed as:

(l -> r -> l + r) 0 2  => 2
(l -> r -> l * r) 2 2  => 4
(l -> r -> l * r) 4 3  => 12
(l -> r -> l + r) 12 ((l -> r -> l * r) ((l -> r -> l * r) 2 2) 3)  => 24

Another added benefit of this representation is the lack of specialization of the operations. Both commutative and pseudo-commutative operations can be represented as generic functions over two parameters, and those functions reused for each operation.

Another GCDT: Sets

We will further formulate a GCDT over sets. A value of type (set T) is a collection of distinct, non-ordered values of type T.

A set has a characteristic commutative operation: append (or, more generally, union). Because sets have no order, the order in which unions are applied is irrelevant.

We use the ∪ operator to represent set union. So, A ∪ B is the union of sets A and B. For unions, the right operand is clear.

An operation, therefore, may be something like:

(B, l -> r -> l ∪ r)

Sets also have a clear pseudo-commutative operation: map (or set comprehension, if you prefer). This is the operation of applying a function (T1 -> T2) to every element in a set, resulting in a set of type set T2.

We represent set comprehension with the map function, which is of the form: map :: (T1 -> T2) -> set T1 -> set T2.

Here's an example, assuming we start with a base value v = {} (the empty set):

C: ({1}, l -> r -> l ∪ R)
C: ({1, 2}, l -> r -> l ∪ R)
P: ({}, l -> _ -> map (* 2) l)
C: ({3, 4}, l -> r -> l ∪ r)

Interestingly, map is a pseudo-commutative operation, but it is unary. To fit the structure, we implement it as a binary operation, but ignore the right operand, since it is always the one specified by the PC operation itself.

The expected result here is map (* 2) ({1} U {1, 2} U {3, 4}) = {2, 4, 6, 8}, and is computed as:

(l -> r -> l ∪ r) {} {1}  => {1}
(l -> r -> l ∪ r) {1} {1, 2}  => {1, 2}
(l -> _ -> map (* 2) l) {1, 2} {}  => {2, 4}
(l -> r -> l ∪ r) {2, 4} ((l -> _ -> map (* 2) l) {3, 4} {})  => {2, 4, 6, 8}

Pseudo-Commutative Operation Precedence

Now, let's introduce another pseudo-commutative operation over sets: filter (or set subtraction). Set subtraction removes all elements in the right operand from the left operand. For example, {1, 2, 3} - {2} = {1, 3}.

This can similarly be implemented using a function of type (T -> bool) which removes an element from set T unless the function returns true.

Based on the properties defined above, we can apply set subtraction in an example:

C: ({1, 4, 7}, l -> r -> l ∪ r)
P: ({}, l -> _ -> filter (< 5) l)
C: ({2, 5, 8}, l -> r -> l ∪ r)

The expected result here is filter (< 5) ({1, 4, 7} U {2, 5, 8}) = {5, 7, 8}, and is computed:

(l -> r -> l ∪ r) {} {1, 4, 7}  => {1, 4, 7}
(l -> _ -> filter (< 5) l) {1, 4, 7} {}  => {7}
(l -> r -> l ∪ r) {7} (filter (< 5) {2, 5, 8})  => {5, 7, 8}

Something problematic happens when we combine the two pseudo-commutative operators, however:

C: ({1, 4, 7}, l -> r -> l ∪ r)
P: ({}, l -> _ -> filter (< 5) l)
P: ({}, l -> _ -> map (* 2) l)
C: ({2, 5, 8}, l -> r -> l ∪ r)

Depending on whether we filter then map or map then filter, we arrive at {10, 14, 16} or {8, 10, 14, 16}, an asymmetric result. Unlike commutative operations, pseudo-commutative operations are not necessarily commutative with each other. Thus, the order in which pseudo-commutative operations are applies matters a great deal.

To resolve this inconsistency, we can require pseudo-commutative operations to be orderable such that, for a set of pseudo-commutative operations s1, there exists a list of these operations s2 such that, s2 has the form { s_i | s_i in s1 and forall j < i, s_i > s_j }.

This gives precedence to pseudo-commutative operations, allowing their order to be resolved when they are "pushed" into subsequent commutative operands, but how do we handle the case when a greater PC operation is received after a lesser PC operation is processed?

One approach to this is to specify the inverse of an operation, allowing it to be efficiently re-ordered.

For example, say we have an initial value v0 and a PC operation ({}, pc1, pc1') (where pc1' inverts pc1). If a subsequent PC operation with a greater precedence is applied, ({}, pc2, pc2'), we compute the accumulator like so:

v = v0
v = pc1 v {}
v = pc1 (pc2 (pc1' v {}) {}) {}

This approach has a few benefits:

First, by inverting and re-applying operations on-the-fly, we avoid the need to re-compute the accumulator all the way from the initial value. Instead, we only need to re-compute the operations which were PC and of a lower priority.
Second, because of this rewinding approach, you will never have to rewind a PC operation of equal or greater precedence, as the operations of lesser precedence will always be "closest" to the end of the chain.
Finally, commutative operations need not be re-applied during a rewind. Instead, the resultant value is treated as a pre-existing member of the set to be re-computed, since the commuted operation is preserved through the inverse of the PC operations.

However, there are a few drawbacks:

Depending on the order in which the PC operations are received, the reducer may be forced to perform unoptimally-many re-computation.
Fundamentally, some PC operations will lack easily computable inverses. For example map sqrt.

This last case is perhaps the most serious drawback to this approach, but it also has a fairly simple solution.

Because the entire domain of a PC operation is known when the operation is applied, we can trivially define an inversion of the operation by building a map from the range -> domain and storing that after the PC is applied (we call this "auto-inversion").

This will require updating the mapping as the PC is applied to subsequent commutative operations, but such updates are considered relatively minor overhead.

This allows us to auto-invert any PC operation. The trade-off here is between time and space complexity.

In cases where the domain operand is small, but the inverse operation complex or impossible to define, defining the inverse as a mapping is more efficient.

However, in cases where the domain operand is large, the resultant auto-inverse may require a large amount of memory. In these cases, if the inverse operation is efficiently computable, defining an inverse function is more efficient.

Applications

The motivation for this thought exercise came from Swarm: a modular & massively-parallel distributed programming language I've been building w/ Ethan Grantz for the past year.

Swarm provides set-enumeration constructs which are natively parallelized and shared variables whose synchronization is handled by the runtime.

However, the language still relies on the developer to avoid asymmetric operations. For example:

enumeration<number> e = [1, 2, 3, 4, 5];
shared number acc = 0;

enumerate e as n {
    if ( n % 2 == 0 ) {
        acc += n;
    } else {
        acc *= n;
    }
}

This example is somewhat contrived, but it is easy to see that the order in which the enumerate body executes for each element of e determines the value of acc.

This example could be made consistent by treating acc as the initial value of a GCDT of type number, and each execution of the body would submit one of two operations:

-- If n % 2 == 0:
C: (n, l -> r -> l + r)

-- Else:
P: (n, l -> r -> l * r)

Then, using the method described above, this result is always consistent, regardless of the order in which the jobs are executed.

This post originally appeared on my blog, here.

Rancher K3s: Kubernetes on Proxmox Containers

Garrett Mills — Tue, 19 Apr 2022 05:54:40 +0000

For a long time now, I've self-hosted most of my online services like calendar, contacts, e-mail, cloud file storage, my website, &c. The current iteration of my setup relies on a series of Ansible playbooks that install all of the various applications and configure them for use.

This has been really stable, and has worked pretty well for me. I deploy the applications to a set of LXC containers (read: lightweight Linux VMs) on Proxmox, a free and open-source hypervisor with an excellent management interface.

Recently, however, I've been re-learning Docker and the benefits of deploying applications using containers. Some of the big ones are:

Guaranteed, reproducible environments. The application ships with its dependencies, ready to run.
Portability. Assuming your environment supports the container runtime, it supports the application.
Infrastructure-as-code. Much like Ansible playbooks, Docker lends itself well to managing the container environment using code, which can be tracked and versioned.

So, I have decided to embark on the journey of transitioning my bare-Linux Ansible playbooks to a set of Kubernetes deployments.

However, there are still some things I like about Proxmox that I'm not willing to give up. For one, the ability to virtualize physical machines (like my router or access point management portal) that can't be easily containerized. Having the ability to migrate "physical" OS installs between servers when I need to do maintenance on the hosts is super useful.

So, I will be installing Kubernetes on Proxmox, and I want to do it on LXC containers.

What We're Building & Rationale

I'm going to deploy a Kubernetes cluster using Rancher's K3s distribution on top of LXC containers.

K3s is a lightweight, production-grade Kubernetes distribution that simplifies the setup process by coming pre-configured with DNS, networking, and other tools out of the box. K3s also makes it fairly painless to join new workers to the cluster. This, combined with the relatively small scale of my deployment, makes it a pretty easy choice.

LXC containers, on the other hand, might seem a bit of an odd choice. Nearly every other article I found deploying K8s on Proxmox did so using full-fat virtual machines, rather than containers. This is certainly the lower-friction route, since it's procedurally the same as installing it on physical hosts. I went with LXC containers for two main reasons:

LXC containers are fast. Like, almost as fast as bare metal. Because LXC containers are virtualized at the kernel level, they are much lighter than traditional VMs. As such, they boot nearly instantly, run at nearly the same speed as the host kernel, and are much easier to reconfigure with more RAM/disk space/CPU cores on the fly.
LXC containers are smaller. Because the containers run on the kernel of the host, they need to contain a much smaller set of packages. This makes them require much less disk space out of the box (and, therefore, makes them easier to migrate).

So, to start out, I'm going to create 2 containers: one control node, and one worker node.

Prerequisites

I'm going to assume that you (1) have a Proxmox server up and running, (2) have a container template available on Proxmox, and (3) you have some kind of NFS file server.

This last one is important since we'll be giving our containers a relatively small amount of disk space. So, any volumes needed by Kubernetes pods can be created as NFS mounts.

You'll also want to set up kubectl and helm tools on your local machine.

Creating the LXC Containers

Because our LXC containers need to be able to run Docker containers themselves, we need to do a bit of additional configuration out of the box to give them proper permissions.

The process for setting up the 2 containers is pretty much identical, so I'm only going to go through it once.

In the Proxmox UI, click "Create CT." Make sure you check the box to show advanced settings.

Make sure to uncheck "Unprivileged container."

Fill in the details of the container. Make sure to uncheck the "Unprivileged container" checkbox. On the next screen, select your template of choice. I'm using a Rocky Linux 8 image.

I elected to give each container a root disk size of 16 GiB, which is more than enough for the OS and K3s to run, as long as we don't put any volumes on the disk itself.

The CPU and Memory values are really up to whatever you have available on the host, and the workloads you intend to run on your K8s cluster. For mine, I gave 4 vCPU cores and 4 GiB of RAM per container.

For the network configuration, be sure to set a static IP address for each node. Additionally, if you use a specific internal DNS server (which I highly recommend!), you should configure that on the next page.

Finally, on the last page, make sure to uncheck the "Start after created" checkbox and then click finish. Proxmox will create the container.

Additional Configuration

Now, we need to tweak a few things under-the-hood to give our containers proper permissions. You'll need to SSH into your Proxmox host as the root user to run these commands.

In the /etc/pve/lxc directory, you'll find files called XXX.conf, where XXX are the ID numbers of the containers we just created. Using your text editor of choice, edit the files for the containers we created to add the following lines:

lxc.apparmor.profile: unconfined
lxc.cgroup.devices.allow: a
lxc.cap.drop:
lxc.mount.auto: "proc:rw sys:rw"

Note: It's important that the container is stopped when you try to edit the file, otherwise Proxmox's network filesystem will prevent you from saving it.

In order, these options (1) disable AppArmor, (2) allow the container's cgroup to access all devices, (3) prevent dropping any capabilities for the container, and (4) mount /proc and /sys as read-write in the container.

Next, we need to publish the kernel boot configuration into the container. Normally, this isn't needed by the container since it runs using the host's kernel, but the Kubelet uses the configuration to determine various settings for the runtime, so we need to copy it into the container. To do this, first start the container using the Proxmox web UI, then run the following command on the Proxmox host:

pct push <container id> /boot/config-$(uname -r) /boot/config-$(uname -r)

Finally, in each of the containers, we need to make sure that /dev/kmsg exists. Kubelet uses this for some logging functions, and it doesn't exist in the containers by default. For our purposes, we'll just alias it to /dev/console. In each container, create the file /usr/local/bin/conf-kmsg.sh with the following contents:

#!/bin/sh -e
if [ ! -e /dev/kmsg ]; then
    ln -s /dev/console /dev/kmsg
fi

mount --make-rshared /

This script symlinks /dev/console as /dev/kmsg if the latter does not exist. Finally, we will configure it to run when the container starts with a SystemD one-shot service. Create the file /etc/systemd/system/conf-kmsg.service with the following contents:

[Unit]
Description=Make sure /dev/kmsg exists

[Service]
Type=simple
RemainAfterExit=yes
ExecStart=/usr/local/bin/conf-kmsg.sh
TimeoutStartSec=0

[Install]
WantedBy=default.target

Finally, enable the service by running the following:

chmod +x /usr/local/bin/conf-kmsg.sh
systemctl daemon-reload
systemctl enable --now conf-kmsg

Setting Up the Container OS & K3s

Now that we've got the containers up and running, we will set up Rancher K3s on them. Luckily, Rancher intentionally makes this pretty easy.

Setup the control node

Starting on the control node, we'll run the following command to setup K3s:

curl -fsL https://get.k3s.io | sh -s - --disable traefik --node-name control.k8s

A few notes here:

K3s ships with a Traefik ingress controller by default. This works fine, but I prefer to use the industry-standard NGINX ingress controller instead, so we'll set that up manually.
I've specified the node name manually using the --node-name flag. This may not be necessary, but I've had problems in the past with K3s doing a reverse-lookup of the hostname from the IP address, resulting in different node names between cluster restarts. Specifying the name explicitly avoids that issue.

If all goes well, you should see an output similar to:

Once this is done, you can copy the /etc/rancher/k3s/k3s.yaml as ~/.kube/config on your local machine and you should be able to see your new (admittedly single node) cluster using kubectl get nodes!

Note: you may need to adjust the cluster address in the config file from 127.0.0.1 to the actual IP/domain name of your control node.

Setup the worker node

Now, we need to join our worker node to the K3s cluster. This is also pretty straightforward, but you'll need the cluster token in order to join the node.

You can find this by running the following command on the control node:

cat /var/lib/rancher/k3s/server/node-token

Now, on the worker node run the following command to set up K3s and join the existing cluster:

curl -fsL https://get.k3s.io | K3S_URL=https://<control node ip>:6443 K3S_TOKEN=<cluster token> sh -s - --node-name worker-1.k8s

Again, note that we specified the node name explicitly. Once this process finishes, you should now see the worker node appear in kubectl get nodes:

You can repeat this process for any additional worker nodes you want to join to the cluster in the future.

At this point, we have a functional Kubernetes cluster, however because we disabled Traefik, it has no ingress controller. So, let's set that up now.

Setting up NGINX Ingress Controller

I used the ingress-nginx/ingress-nginx Helm chart to set up the NGINX ingress controller. To do this, we'll add the repo, load the repo's metadata, then install the chart:

helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm repo update
helm install nginx-ingress ingress-nginx/ingress-nginx --set controller.publishService.enabled=true

Here, the controller.publishService.enabled setting tells the controller to publish the ingress service IP addresses to the ingress resources.

After the chart completes, you should see the various resources appear in kubectl get all output. (Note that it may take a couple minutes for the controller to come online and assign IP addresses to the load balancer.)

We can test that the controller is up and running by navigating to any of the node's addresses in a web browser:

In this case, we expect to see the 404, since we haven't configured any services to ingress through NGINX. The important thing is that we got a page served by NGINX.

Conclusion

Now, we have a fully-functional Rancher K3s Kubernetes cluster, and the NGINX Ingress Controller configured and ready to use.

I've found this cluster to be really easy to maintain and scale. If you need to add more nodes, just spin up another LXC container (possibly on another physical host, possibly not) and just repeat the section to join the worker to the cluster.

I'm planning to do a few more write-ups chronicling my journey to learn and transition to Kubernetes, so stay tuned for more like this. The next step in this process is to configure cert-manager to automatically generate Let's Encrypt SSL certificates and deploy a simple application to our cluster.

This post originally appeared on my blog, here.

Runtime Data Validation from TypeScript Interfaces

Garrett Mills — Fri, 14 Jan 2022 21:03:38 +0000

For the last year or so, I've been (slowly) building a TypeScript-based Node.js framework called Extollo. One of the design goals with Extollo is to only expose the user (i.e. the developer) to ES/TypeScript native concepts, in order to reduce the amount of special knowledge required to get up and running with the framework.

Runtime schemata: a plague of DSLs

One of my biggest pet-peeves with the current Node.js framework scene is that nearly every ecosystem has to re-invent the wheel when it comes to schema definitions. Because JavaScript doesn't have a native runtime type-specification system (at least, not a good one), if you want to encode details about how a data structure should look at runtime, you need to design a system for passing that information along at runtime.

For example, a prolific MongoDB ODM for Node.js, Mongoose, gives users the ability to specify the schema of the records in the collection when the user defines a model. Here's a sample schema definition from the Mongoose docs:

import mongoose from 'mongoose';
const { Schema } = mongoose;

const blogSchema = new Schema({
  title:  String, // String is shorthand for {type: String}
  author: String,
  body:   String,
  comments: [{ body: String, date: Date }],
  date: { type: Date, default: Date.now },
  hidden: Boolean,
  meta: {
    votes: Number,
    favs:  Number
  }
});

I'm currently building the request validation system for Extollo. Because it has to process web requests with dynamic input, the validator interfaces need to be specified at runtime, so they data can be checked against the schema. To do this, I'm using the fantastic Zod schema validator library written by Colin McDonnell.

However, Zod falls victim to the same fundamental problem with runtime schemata in JavaScript as Mongoose. Because its schemata need to be available at runtime, you have to use Zod's custom schema builder to define your interfaces. Here's an example of a schema for some data that might come from a login page:

import { z } from 'zod'

export const LoginAttemptSchema = z.object({
    username: z.string().nonempty(),
    password: z.string().nonempty(),
    rememberMe: z.boolean().optional(),
})

That's not too bad, but it does require the developer to learn Zod's specific schema definition language. I find this especially annoying since TypeScript already has an interface definition language! This is a situation where I'd like to avoid making the developer learn an equivalent system if they already know the one built into the language.

Let's rewrite this schema in TypeScript for a start:

export interface LoginAttempt {
    /** @minLength 1 */
    username: string

    /** @minLength 1 */
    password: string

    rememberMe?: boolean
}

Okay, that's an improvement! We can use TypeScript's native type syntax to define the interface, and augment it with JSDoc comments for any properties that can't be natively expressed. So, to use this with Zod, we need to convert it from the TypeScript syntax to the Zod syntax. Luckily, Fabien Bernard has spearheaded the excellent ts-to-zod project, which looks through interfaces defined in a file and outputs the equivalent Zod schemata for them.

Hmm.. so now the user can write their schema definitions in (mostly) native TypeScript syntax, and, with a bit of helper tooling, we can convert them to the Zod format so we can use them at runtime. Perfect! Well, almost...

We have a subtle problem that arises when we want to actually _use _a schema at runtime. Let's look at an example:

import { Validator } from '@extollo/lib'
import { LoginAttempt } from '../types/LoginAttempt.ts'

class LoginController {
    public function getValidator() {
        return new Validator<LoginAttempt>()
    }
}

This class has a method which returns a new Validator instance with the LoginAttempt schema as its type-parameter. Intuitively, this should produce a validator which, at runtime, validates data against the LoginAttempt schema. Let's look at the compiled JavaScript:

"use strict";
Object.defineProperty(exports, "__esModule", { value: true });
const Validator_1 = require("@extollo/lib").Validator;
class LoginController {
    getValidator() {
        return new Validator_1.Validator();
    }
}

Uh, oh. Ignoring the boilerplate noise, we see that our nice, type-parameterized Validator instance has been stripped of its type information. Why? TypeScript is a transpiler. So, it takes TypeScript code and outputs the _equivalent _JavaScript code. Because JavaScript has no concept of types at runtime, the transpiler (in this case, tsc) strips them out.

So now we have a problem. We've improved our user-interface by only requiring the developer to specify the TypeScript types, but now we can't use them at runtime, because the TypeScript types get stripped away. 'What about the Zod schema we just generated?' you ask, wisely. Well, unfortunately, there's no mapping between the interface and the Zod schema it induced, and there's no easy way to create such a mapping, because it has to be done at compile-time.

A very deep rabbit-hole

Ordinarily, this is where the story ends. You need some kind of mapping between the interface and the Zod schema (which, remember, the developer has no idea exists thanks to our ts-to-zod magic) to make the Validator work. In a generic TypeScript project, you'd have to have some kind of naming convention, or expose the schema to the user somehow to create the mapping.

However, Extollo has a unique advantage that I suspected could be used to solve this problem transparently: excc. Extollo projects are primarily TypeScript projects, but they also contain other files like views, assets, &c. that need to be included in the built-out bundle. To standardize all of this, Extollo uses its own project-compiler called excc for builds. excc is primarily a wrapper around tsc that does some additional pre- and post-processing to handle the cases above.

Because Extollo projects are all using excc, this means that we can do arbitrary processing at compile time. I suspected that there would be a way to create a mapping between the interfaces and the schemata we generate for runtime.

Zod-ifying the Interfaces

The first step was converting the TypeScript interfaces to Zod schemata using ts-to-zod. In excc, this is implemented as a pre-processing step that appends the Zod schema to the .ts file that contains the interface. So, the processed LoginAttempt.ts might look something like:

import { z } from "zod";

export interface LoginAttempt {
    /** @minLength 1 */
    username: string

    /** @minLength 1 */
    password: string

    rememberMe?: boolean
}

export const exZodifiedSchema = z.object({
    username: z.string().nonempty(),
    password: z.string().nonempty(),
    rememberMe: z.boolean().optional(),
});

This has some drawbacks. Namely, it assumes that only one interface is defined per-file. However, Extollo enforces this convention for other concepts like models, middleware, controllers, and config files, so it's fine to make that assumption here.

This gets us closer, but it still doesn't do the mapping for the runtime schema. The first step to this is going to be devising some way of referencing a schema so that we can easily modify the TypeScript code that uses its related interface.

I don't love the initial system I have for this, but what excc does now is generate a unique ID number for each interface it Zod-ifies. Then, when it is writing the Zod schema into the interface's file, it adds code to register it with a global service that maps the ID number to the Zod schema at runtime. So, the above file would actually look something like:

import { z } from "zod";
import { registerZodifiedSchema } from "@extollo/lib";

export interface LoginAttempt {
    /** @minLength 1 */
    username: string

    /** @minLength 1 */
    password: string

    rememberMe?: boolean
}

/** @ex-zod-id 11@ */
export const exZodifiedSchema = z.object({
    username: z.string().nonempty(),
    password: z.string().nonempty(),
    rememberMe: z.boolean().optional(),
});
registerZodifiedSchema(11, exZodifiedSchema);

This may not seem like much, but this is a huge step toward our goal. We now have, at compile time, a mapping of interfaces to IDs and, at runtime, a mapping of IDs to schemata. So, we can use the compile-time map to modify all the places that reference the interface to set a runtime parameter with the ID of the schema for that interface. Then, at runtime, we can look up the schema using the ID. Bingo! No, how do we actually do that...

Wrangling the AST

Now that we have our mapping, we need to make sure that a look-up is done whenever the type is referenced in code. That is, anywhere where we create a Validator<LoginAttempt>, we should set the ID of the Zod schema for LoginAttempt on that Validator instance.

To accomplish this, I wrote a couple of transformer plugins for TypeScript. Now, tsc doesn't support plugins by default. (You may have seen plugins in the tsconfig.json for a project, but they are plugins for the editor's language server, not the compiler.) Luckily for us, again, there exists a fantastic open-source package to solve this problem. Ron S. maintains a package called ts-patch which, aptly, patches the tsc installation for a project to allow the project to specify compiler-plugins.

These plugins operate on the abstract syntax-tree of the TypeScript program. If you're not familiar with ASTs, they're basically the compiler's internal representation of the program you're compiling. They are data structures which can be manipulated and optimized. When you install a plugin, it is called repeatedly with the AST for each source file in the TypeScript project you're compiling. Importantly, the plugin can replace any of the nodes in the AST, or return a completely different one, in the file, and tsc will output the modified version instead of the original.

First, Identify

The first plugin operates on the entire AST for each file in the project. Its job is to walk through each file's AST and look for interface declarations that we generated Zod schema for. When it finds one, it parses out the ID number we wrote into the file earlier and stores a mapping between that ID number and the symbol TypeScript uses to identify the interface internally.

Because we were the ones that wrote the Zod schema into the file, we can know that it -- and the call to register it with the global service -- are the last statements in the file. So, we can quickly look them up and parse out the ID from the registration call.

The TypeScript AST for the augmented file, at this point, looks something like this:

(As an aside, I used the ts-ast-viewer web app to generate this hierarchy. ts-ast-viewer is a project started by David Sherret that allows you to visualize and explore the AST for any TypeScript program. It was invaluable in helping me figure out the structures for this project.)

By recursively walking the AST, we can look for the InterfaceDeclaration nodes. If we find one in a file, we can check the root of the file to see if an Identifier called exZodifiedSchema is defined. If so, we grab the last statement in the file (an ExpressionStatement containing the call to registerZodifiedSchema) and pull out its first argument, the ID number.

Once this transformer finishes, we've identified all of the interfaces for which we generated Zod schemata and created a mapping from the interface to the ID number we need at runtime.

Then, Modify

The second plugin runs after the first has finished going through all the files in the project. This plugin's job is to replace any NewExpression nodes where the type parameters contain Zod-ified interfaces with an IIFE that sets the __exZodifiedSchemata property to an array of the ID numbers used to look up the schemata for those interfaces.

That is, the plugin transforms this:

new Validator<LoginAttempt>()

into this:

(() => {
    const vI = new Validator<LoginAttempt>();
    vI.__exZodifiedSchemata = [11];
    return vI;
})()

And because the NewExpression is an expression just like the CallExpression is, anywhere where we have a NewExpression can instead have this CallExpression that wraps it with additional logic. The transformer is able to look up the ID numbers associated with the interfaces because the Identifier that references the interface in new Validator<LoginAttempt>() has the same symbol set on it as the InterfaceDeclaration we identified with the first plugin.

These symbols are created by something in the TypeScript compiler called the linker, which resolves all the identifier declarations and matches them up with the uses of those identifiers, even if the actual variables have been renamed along the way. So, we can use these symbols to match up uses of the interface with the declarations of the interfaces we care about.

This is the magic sauce that finally makes it all work. After this plugin runs, the program TypeScript finishes compiling has all of the runtime type mappings linked up to the Validator instances based on which interface was specified when the Validator was instantiated.

Conclusion

This was a long, and kind of hacky journey, but the end result is excellent. From the developer's perspective, they can type this:

const validator = new Validator<LoginAttempt>();

and, at runtime, the validator instance will have a Zod schema and will be able to parse data against the schema. No custom schema definition languages, no validator/schema mapping, nothing. To the developer, it's all just pure TypeScript, which was the goal all along.

The code for this feature is still very much work-in-progress, and I have to remove a lot of unused code and clean up what I keep, and probably rewrite part of it to be a less... jank. But, at the end of the day, I'm really happy with this "magic" feature that will help keep Extollo projects TypeScript-native, and easy to maintain.

You can find a minimal working example matching this post here.

This post originally appeared on my blog, here.

Rethinking Dependency Injection in TypeScript

Garrett Mills — Tue, 30 Mar 2021 19:06:41 +0000

This post originally appeared on my blog, here.

Anyone who has read my blog before knows that I have a particular interest in dependency injection and inversion-of-control paradigms.

Over the last few years, I've implemented DI in JavaScript for various projects, and I'm currently in the process of rewriting my framework and its DI implementation, so I wanted to share some observations about different JavaScript/TypeScript DI strategies.

In particular, we'll explore named-injection, constructor-injection, and property-injection.

Named Injection

My first foray into DI in JavaScript relied on purely-runtime code and allowed injecting services from a container by name:

const Injectable = require('./Injectable')

class SomeInjectableClass extends Injectable {
    static services = ['logging']

    myMethod() {
        this.logging.info('myMethod called!')
    }
}

This was a fairly efficient and scalable paradigm, and defining the services as a property on the class itself made it easy to account for the services required by parent classes:

// ...
class AnotherInjectableClass extends SomeInjectableClass {
    static get services() {
        return [...super.services, 'another_service']
    }

    myMethod() {
        this.another_service.something()
        super.myMethod()
    }
}

In fact, this mechanism was reliable enough that it became the basis of the injector used in my Flitter framework.

Drawbacks

This method is not without its downsides, however. For one, all classes must extend a common Injectable base class. If your class extends from, say, a base class from a library, then it can't be injected directly.

Likewise, relying on service names makes it hard to know exactly what's being injected into your class. Especially as I am transitioning more projects and my framework over to TypeScript, relying on named-injection just wasn't going to cut it. This would require referencing properties with the any type annotation:

class SomeInjectableClass extends Injectable {
    static get services(): string[] {
        return [...super.services, 'another_service']
    }

    myMethod(): void {
        (this as any).another_service.something()  // NOT type safe
    }
}

Relying on named services also makes the injectable classes inflexible, as the services have to be injected into properties with the same name. Say, for example, I have a service called models, and a class that uses it. If that class wants to keep an array called models, it will conflict with the injected service:

class SomethingThatUsesModels extends Injectable {
    static get services() {
        return [...super.services, 'models']
    }

    // CONFLICT with the injected 'models' service
    protected models: Model[] = []
}

Because a named-injector would have to bypass type-safety, this could lead to a situation where the TypeScript compiler types models as Model[], but the injector overrides it to be the injected models service, which would cause runtime errors.

Constructor Injection

Since we're working in TypeScript, we want to do away with named-injection entirely. The TypeScript compiler has a flag which, when enabled, emits the type metadata for classes and properties, making it available via the Reflection API.

This is useful because it effectively enables "naming" a dependency based on its type, rather than an arbitrary string. So, when defining typed injectable classes, each property contains two pieces of information, rather than just one.

Likewise, we can enable the experimental "decorators" functionality, which can allow us inject any arbitrary class rather than requiring it to extend a base Injectable class. For example:

@Injectable()
class SomethingThatUsesModels {
    protected models: Model[] = []

    constructor(
        protected readonly modelsService: ModelsService,
    ) { }
}

Anyone who has used the Angular framework is familiar with this format. The Angular DI historically worked this way, using type reflection to handle injection. Nowadays, it uses its custom compiler to handle injection at compile time, but that's beyond the scope of this writeup.

How does this work?

Okay, so we have a decorator and some type annotations. But, how do we actually do the injection from that?

The key is that Injectable decorator. In essence, this decorator is a function that accepts the class it decorates. Then, this function uses the reflect-metadata package to get a list of type annotations for the constructor's parameters, then stores that information as additional metadata.

Here's a (simplified) example from the Extollo DI (Flitter's TypeScript successor):

/**
 * Get a collection of dependency requirements for the given target object.
 * @param {Object} target
 * @return Collection<DependencyRequirement>
 */
function initDependencyMetadata(target: Object): Collection<DependencyRequirement> {
    const paramTypes = Reflect.getMetadata('design:paramtypes', target)
    return collect<DependencyKey>(paramTypes).map<DependencyRequirement>((type, idx) => {
        return {
            paramIndex: idx,
            key: type,
            overridden: false,
        }
    })
}

/**
 * Class decorator that marks a class as injectable. When this is applied, dependency
 * metadata for the constructors params is resolved and stored in metadata.
 * @constructor
 */
export const Injectable = (): ClassDecorator => {
    return (target) => {
        const meta = initDependencyMetadata(target)
        Reflect.defineMetadata(DEPENDENCY_KEYS_METADATA_KEY, meta, target)
    }
}

In essence, all this decorator does is read the type annotations from the class' meta-data and store them in a nicer format in its own meta-data key (DEPENDENCY_KEYS_METADATA_KEY).

Instantiating the Class

Okay, so we have the type annotations stored in meta-data, but how do we actually inject them into the class? This is where the container comes in.

In our old paradigm, the container was a class that mapped service names (another_service) to factories that created the service with that name. (e.g. another_service to instanceof AnotherService). In the type-based system, the container is a class that maps types to factories that create the service with that type.

This result is very strong as it enables type-safe injection. In the example above, the "token", ModelsService is mapped to an instance of the ModelsService by the container.

So, when we ask the container to inject and create an instance of our SomethingThatUsesModels class, the container goes through all the items in the DEPENDENCY_KEYS_METADATA_KEY meta-data key and resolves them. Then, it passes those instances into the new class to instantiate it. For a (simplified) example:

class Container {
    resolveAndCreate<T>(token: Instantiable<T>): T {
        const dependencies = Reflect.getMetadata(DEPENDENCY_KEYS_METADATA_KEY)
        const params = dependencies.orderByAsc('paramIndex')
            .map(dependency => this.resolveAndCreate(dependency.key))

        return new token(...params)
    }
}

So, we can instantiate our SomethingThatUsesModels class like so:

const inst = <SomethingThatUsesModels> container.resolveAndCreate(SomethingThatUsesModels)

Drawbacks

The constructor-injection paradigm works well and addresses many of the features we cared about between named-injection. In particular:

Provides type-hinted injection
Separates class property names from injection tokens

However, one way this falls behind named-injection is in the sense that the child classes must know and provide the dependencies of their parents.

For example, assume I have a class:

@Injectable()
class ParentClass {
    constructor(
        protected logging: LoggingService
    ) { }
}

Now, I want to define a child of this class that has its own dependencies:

@Injectable()
class ChildClass extends ParentClass {
    constructor(
        protected another: AnotherService,
    ) { super() }  // ERROR!
}

This will immediately fail to compile, since the ChildClass doesn't pass the required dependencies into the parent. In reality, the child class must also specify the dependencies of the parent as parameters in its constructor:

@Injectable()
class ChildClass extends ParentClass {
    constructor(
        protected another: AnotherService,
        logging: LoggingService,
    ) { super(logging) }
}

The issue with this becomes immediately obvious. All of the dependencies and imports of the parent must also be specified in all of the children. As the classes become larger and the inheritance chain becomes longer, you can quickly run into ridiculously long constructor signatures:

@Injectable()
class LargerControllerClass extends ParentControllerClass {
    constructor(
        protected logging: LoggingService,
        protected config: ConfigService,
        protected models: ModelsService,
        socket: SocketService,
        renderer: ViewRenderer,
        other: OtherService,
        another: AnotherService,
        more: MoreService,
    ) { super(socket, renderer, other, another, more) }
}

Here, not only does the child need to be aware of the dependencies of the parent, it needs to take into account the order of the constructor parameters, which might be irrelevant in practice, but could break between versions.

Property Injection

To improve upon this, we want to divorce the injected dependencies from the constructor while still maintaining the type-hinted and property-name benefits we gained from constructor-injection

This has the additional benefit of keeping the constructor signatures smaller, and keeping the non-injected constructor parameters distinct from the injected ones.

Luckily, in TypeScript, properties of a class also emit type annotations, and can be decorated. So, we can change our

ParentClass and ChildClass definitions to look as follows:

@Injectable()
class ParentClass {
    @Inject()
    protected readonly logging!: LoggingService
}

@Injectable()
class ChildClass extends ParentClass {
    @Inject()
    protected readonly another!: AnotherService
}

How does this work?

The "magic" bit here is the @Inject() decorator, which looks at the type annotation of the property it decorates and stores that property and its token value as meta-data on the class. Here's a simplified example of Extollo's implementation:

/**
 * Mark the given class property to be injected by the container.
 * @constructor
 */
export const Inject = (): PropertyDecorator => {
    return (target, property) => {
        const propertyMetadata = new Collection<PropertyDependency>()
        Reflect.defineMetadata(DEPENDENCY_KEYS_PROPERTY_METADATA_KEY, propertyMetadata, target)

        const type = Reflect.getMetadata('design:type', target, property)
        if ( type ) {
            const existing = propertyMetadata.firstWhere('property', '=', property)
            if ( existing ) {
                existing.key = key
            } else {
                propertyMetadata.push({ property, key })
            }
        }

        Reflect.defineMetadata(DEPENDENCY_KEYS_PROPERTY_METADATA_KEY, propertyMetadata, target)
    }
}

Now, when the container creates an instance of a class, instead of passing in the dependencies as parameters to the constructor, it instantiates the class, then sets the properties on the class that have @Inject() decorators. For example:

class Container {
    resolveAndCreate<T>(token: Instantiable<T>): T {
        const inst = new token()
        const dependencies = Reflect.getMetadata(DEPENDENCY_KEYS_PROPERTY_METADATA_KEY, token)
        const instances = dependencies.map(x => {
            inst[x.property] = this.resolveAndCreate(x.key)
        })

        return inst
    }
}

There's a problem here, though. Say we were to resolveAndCreate<ChildClass>(ChildClass). Because of the way JavaScript works, the instance returned by this call would ONLY have the properties defined in the child class, not the parent (i.e. another, but not logging).

To understand why, we need a bit of background.

The Prototype Chain

In JavaScript, inheritance is prototypical. Say we have the following:

const parent = new ParentClass(...)
const child = new ChildClass(...)

The object created as parent has a "chain" of prototypes that comprise it. So, if I try to access a method or property on parent, JavaScript will first check if the property exists on parent itself. If not, it will check if the property exists on ParentClass.prototype, then ParentClass.prototype.prototype, and so on.

If you follow the prototype chain long enough, every item in JavaScript eventually extends from Object.prototype or Function.prototype. (For classes, it's the latter.) From any of these prototypes, we can access the constructor they are associated with using Class.prototype.constructor.

So, to get the ParentClass constructor from its prototype, we could do ParentClass.prototype.constructor.

The Issue

When our @Inject() decorator saves the meta-data about the property type annotations, it does so by defining a new meta-data property on the prototype of the class where the property was defined.

Since the logging property was first defined and decorated in the ParentClass, the meta-data property with the information we need is actually defined on ParentClass.prototype.

However, when @Inject() is called for the another property in the ChildClass, it defines a new meta-data key with ChildClass's defined properties on the ChildClass.prototype.

Thus, in order to get all the properties we need to inject, we must check the meta-data defined for all prototypes in the inheritance chain of the constructor being instantiated. So, the container implementation might look something like:

class Container {
    resolveAndCreate<T>(token: Instantiable<T>): T {
        const inst = new token()
        const meta = new Collection<PropertyDependency>()
        let currentToken = token

        do {
            const loadedMeta = Reflect.getMetadata(DEPENDENCY_KEYS_PROPERTY_METADATA_KEY, currentToken)
            if ( loadedMeta ) meta.concat(loadedMeta)
            currentToken = Object.getPrototypeOf(currentToken)
        } while (
            Object.getPrototypeOf(currentToken) !== Function.prototype
            && Object.getPrototypeOf(currentToken) !== Object.prototype
        )

        meta.map(x => {
            inst[x.property] = this.resolveAndCreate(x.key)
        })

        return inst
    }
}

Now, inst will have all properties defined as injected for all parent classes in the inheritance chain.

Best of Both Worlds

This approach combines the benefits of named-injection with the type-safety of constructor-injection:

Child classes don't need to account for the dependencies of their parents
Injected dependencies can be type-hinted
Property names of dependencies are independent of their types
Parent dependencies are automatically accounted for

After using it for a while, I really like this paradigm. It provides a type-safe way to do dependency injection reliably, while still keeping the class definitions clean and parent-agnostic.

Drawbacks

While its still my preferred solution, property-injection in TypeScript still isn't without its drawbacks. Namely, it requires use of the non-null assertion operator since the properties are being filled in by the container.

Because of this, if you were to instantiate a class manually outside the container and not fill in all the properties, the compiler wouldn't catch it. Accessing properties on that instance would lead to runtime errors. However, assuming you always instantiate Injectable classes with the container, this problem is largely mute.

Another downside that I didn't explore much in this article is the container code. Generalizing the container (for either constructor- or property-injection) requires use of the any operator at some point since factories are matched by key. At least in my implementation. I'd be interested to see alternatives.

Conclusion

There will, undoubtedly, be another iteration of this article wherein I discover a new paradigm I want to try. But for the foreseeable future, I'll be implementing and running with property-injection in my projects. As I mentioned in the article, constructor-injection and property-injection support form the basis of the dependency injector for the Extollo framework, my new project.

I tried not to dive too deep into the actual code required to implement the various strategies in this article, so if you're interested in seeing how I've implemented them for my projects, here are some links:

The Flitter DI (named-injection)
The Extollo DI (constructor- and property-injection, WIP)

As always, I'd love to hear any other strategies or ways people have implemented this, so feel free to leave a comment or get in touch.

Converting an Ionic/Angular Site into a Progressive Web App

Garrett Mills — Tue, 01 Dec 2020 18:00:05 +0000

This post originally appeared on my blog, here.

For the past year, I've been working on a web application called Noded. Noded is built in Angular on the Ionic framework and provides tools for building a personal tree of information. (If you're curious, you can try it out here.)

A screenshot from Noded.

Because Noded is meant to replace whatever note-taking application a person uses, it's important that it be available offline (on your phone, for instance). So, one of the goals for Noded was to make it work as a progressive web app so it could be loaded even when the client doesn't have Internet access.

For the uninitiated, a progressive web app (or PWA) is a type of web app that can make use of native-integration features like push notifications, storage, &c. On mobile platforms, this also enables the "Add to Home Screen" functionality which enables users to "install" a PWA to their device so it appears as a native application and opens in full-screen mode, rather than in a browser.

Noded, running as a PWA on my phone.

Service Workers

In order for a web app to become a PWA, it needs two things. First, it needs a web manifest, which tells the browser the location of all resources used by the web app, and other information like the icon and background color. Second, it needs to have a service worker registered. Service workers are event-based JavaScript programs that run in the background on a user's browser.

These background programs can run even when the app itself isn't open and enable things like offline mode and push notifications. Ever wonder how applications like Google Docs can still load even when the browser is offline? This is enabled by the service worker API.

Your application's service worker sits like a layer between your application and its back-end server. When your app makes a request to the server, it is intercepted by the service worker which decides whether it will be forwarded to the back-end, or retrieved from the local cache.

PWAs work offline by having the service worker cache all of their app resources offline automatically. Then, when the back-end server is unreachable, the resources are served from the service worker transparently to the application. Even when your app is online, service workers can dramatically speed up load times for people with slow or latent connections (especially those in developing areas).

Angular Service Worker

Because of their structured nature, Angular apps can make use of the Angular Service Worker which can automatically integrate with Angular apps to cache the built modules offline. This can be much easier to configure than writing a service-worker from scratch.

We'll start by adding the @angular/pwa package to our app, which will automatically bootstrap the manifest and service worker config:

ng add @angular/pwa --project app

(Where app is the name of your Angular project in angular.json.) This will create the ngsw-config.json config file, as well as the manifest in src/manifest.webmanifest.

`ngsw-config.json`

The Angular service worker can be configured through the ngsw-config.json file. By modifying this file, we can tell the service-worker for our app to automatically pre-fetch all assets for the application. That way, when the app goes offline, it can still load the front-end resources.

Note that the service-worker will cache other XHR headers with the proper cache headers, but if your application relies on API requests to start, you should account for that in the app's code using things like IndexedDB or localStorage.

{
  "$schema": "./node_modules/@angular/service-worker/config/schema.json",
  "index": "/index.html",
  "assetGroups": [
    {
      "name": "app",
      "installMode": "prefetch",
      "resources": {
        "files": [
          "/favicon.ico",
          "/index.html",
          "/manifest.webmanifest",
          "/*.css",
          "/*.js"
        ]
      }
    },
    {
      "name": "assets",
      "installMode": "prefetch",
      "updateMode": "prefetch",
      "resources": {
        "files": [
          "/assets/**",
          "/*.(eot|svg|cur|jpg|png|webp|gif|otf|ttf|woff|woff2|ani)"
        ]
      }
    }
  ]
}

Here's a sample config file. The index key specifies the entry-point to your application. For most Angular apps, this will be index.html since that's the file first loaded.

Then, the front-end assets are split into two groups. The app group matches any built files that are necessary to boot the Angular app. The assets group matches any additional assets like images, fonts, and external files.

In this example, I've set both groups to prefetch, which means that the service-worker will try to cache them in the background the first time the app is loaded. This ensures that they are always available offline, as long as they had time to load once. However, it can be more taxing for the first load.

To avoid this, you can set an asset group to installMode: lazy. This will cache the resources offline only once the front-end tries to load them.

Web Manifest

The @angular/pwa package will also generate a web manifest for your application in src/manifest.webmanifest. Here, you can customize things like your application's name, background colors, and icons:

{
  "name": "Noded",
  "short_name": "Noded",
  "theme_color": "#3A86FF",
  "background_color": "#fafafa",
  "display": "standalone",
  "scope": "./",
  "start_url": "./index.html",
  "icons": [
    {
      "src": "assets/icons/icon-72x72.png",
      "sizes": "72x72",
      "type": "image/png",
      "purpose": "maskable any"
    },
    {
      "src": "assets/icons/icon-96x96.png",
      "sizes": "96x96",
      "type": "image/png",
      "purpose": "maskable any"
    },
...

Angular will auto-generate PWA icons in the assets/icons/ directory, so you'll want to customize those to match your app. These icons will become the home-screen icon for you app when a user installs it.

Noded's PWA icon when added to my home screen.

A few other notes about the web manifest:

The scope property defines the scope of pages in the web app that can be navigated to in the "app mode." If your app tries to load a route that's outside of the scope, the client will revert to a web-browser rather than immersive mode.
- This property is relative to the entry point of the application. So, if the entry point is /index.html, then the scope ./* matches all routes /**.
The start_url is the route that is loaded when the user launches the PWA. Usually, this should match the entry point in the ngsw-config.json file as index.html.

Building your application

Now that we've set up the Angular service-worker, you should be able to build your app and have it appear as a PWA in the browser. You can do this as you normally would. Since Noded is an Ionic app, I'll use:

./node_modules/.bin/ionic build --prod

Using the ngsw-config.json, this will generate a few new files. If you look at www/ngsw.json, you can see the compiled config for the service-worker telling it the locations of all generated files for your app:

{
  "configVersion": 1,
  "timestamp": 1606842506052,
  "index": "/index.html",
  "assetGroups": [
    {
      "name": "app",
      "installMode": "prefetch",
      "updateMode": "prefetch",
      "cacheQueryOptions": {
        "ignoreVary": true
      },
      "urls": [
        "/10-es2015.8900b72b6fdc6cff9bda.js",
        "/10-es5.8900b72b6fdc6cff9bda.js",
        "/11-es2015.82443d43d1a7c061f365.js",
        "/11-es5.82443d43d1a7c061f365.js",
        "/12-es2015.617954d1af39ce4dad1f.js",
        "/12-es5.617954d1af39ce4dad1f.js",
        "/13-es2015.eb9fce554868e6bda6be.js",
...

This is how the service-worker knows what to fetch and cache when running your application. It also writes the ngsw-worker.js file, which is the actual service worker code that gets run by the browser in the background. The web manifest is also included in the build.

Once you deploy your app and load it in the browser, it should now appear to have both a web manifest and a service worker:

You can view this on the "Application" tab of your browser's dev-tools.

Note that the service worker will only register and run if it is configured properly and your application is served over HTTPS.

Running in a sub-route (`/app`, &c.)

You may have noticed in the screen-shot above that the service-worker for Noded is registered for noded.garrettmills.dev/i. This is because the Angular app for Noded runs in the /i sub-route of the domain. This requires special consideration for the service-worker.

Recall that the manifest has a scope and start_url, and the ngsw.json has an index key. These are relative to the root of the domain, not the application. So, in order to serve our Angular app from a sub-route, we need to modify the PWA configs. Luckily, the Angular service-worker has a CLI tool that makes this easy for us. After we build our application, we can use the ngsw-config command to re-generate the config to use a sub-route:

./node_modules/.bin/ngsw-config ./www/ ./ngsw-config.json /i

The last argument is the sub-route where your application lives. In my case, that's /i. This command will modify the service-worker config to use the sub-route for all resources:

{
  "configVersion": 1,
  "timestamp": 1606843244002,
  "index": "/i/index.html",
  "assetGroups": [
    {
      "name": "app",
      "installMode": "prefetch",
      "updateMode": "prefetch",
      "cacheQueryOptions": {
        "ignoreVary": true
      },
      "urls": [
        "/i/10-es2015.8900b72b6fdc6cff9bda.js",
        "/i/10-es5.8900b72b6fdc6cff9bda.js",
        "/i/11-es2015.82443d43d1a7c061f365.js",
        "/i/11-es5.82443d43d1a7c061f365.js",
        "/i/12-es2015.617954d1af39ce4dad1f.js",
        "/i/12-es5.617954d1af39ce4dad1f.js",
...

This ensures that your service worker caches the correct files. (Note that this doesn't actually need to modify the web manifest.)

Debugging

Once you've deployed your built app, it should start caching assets through the service-worker. However, if this doesn't happen, here are a few things to consider.

Don't modify the compiled Angular code

Once your app has been compiled to the www/ directory, never modify these files. If you need to make changes, use substitutions in the angular.json, or just change the original source files.

  "hashTable": {
    "/i/10-es2015.8900b72b6fdc6cff9bda.js": "d3cf604bab1f99df8bcf86d7a142a3a047c66dd2",
    "/i/10-es5.8900b72b6fdc6cff9bda.js": "8fcf65ea8740ae0364cd7371dd478e05eadb8b35",
    "/i/11-es2015.82443d43d1a7c061f365.js": "bc50afb2730b9662fc37a51ae665fd30a9b0637c",
    "/i/11-es5.82443d43d1a7c061f365.js": "300d5e62ec8ed5a744ac0dc1c2d627d6208499d7",
    "/i/12-es2015.617954d1af39ce4dad1f.js": "465dd6ae6336dee028f3c2127358eea1d914879d",
    "/i/12-es5.617954d1af39ce4dad1f.js": "5549d758aea47ab6d81a45d932993a6da9f5289c",
    "/i/13-es2015.eb9fce554868e6bda6be.js": "2ca9cc161ae45c0a978b8bebce3f6dd7597bba07",
    "/i/13-es5.eb9fce554868e6bda6be.js": "1dadc7f0083a1d499ea80f9c56d9ad62de96c4f3",
...

The reason for this is because the Angular service-worker generates hashes of the generated files and checks them on download. This is how it knows whether it has cached the latest version of the file or not. If you manually modify the compiled file, the hash won't match, and the service-worker will invalidate its entire cache.

Bypass the service-worker

As mentioned above, the service-worker will attempt to cache other outbound requests, provided that the server responds with appropriate cache headers. However, there may be instances where you want to prevent this behavior (for example, when checking if the app is online and can access the server). To do this, you can add the ?ngsw-bypass query parameter to the URLs of your requests.

Example: /api/v1/stat?ngsw-bypass.

View service-worker logs

If you are having issues with the service worker's cache, it can be difficult to narrow them down without logs. You can view debugging output from the Angular service-worker by navigating to the /ngsw/state route in your app. In my case, that's https://noded.garrettmills.dev/i/ngsw/state.

NGSW Debug Info:

Driver state: NORMAL ((nominal))
Latest manifest hash: none
Last update check: never



=== Idle Task Queue ===
Last update tick: never
Last update run: never
Task queue:


Debug log:

If you are having issues, the Debug log section can provide more info on cache invalidation and other issues.

View cached files

You can view the status of cached files in the "Storage" section of your browser's dev tools. This can help you see if the service worker was unable to find files (invalid route configurations), or was invalidating cached files.

Files cached locally by Noded's service worker.

Conclusion

This was a cursory look at getting your Angular/Ionic app set up as a PWA and caching assets offline using Angular service-workers. If your app relies on back-end resources (like an API), you'll still need to account for that when adding offline support using tools like IndexedDB and localStorage.

For example, Noded has an API service that sits between the app and the server and caches API resources locally in the IndexedDB. Perhaps we'll look into this more in a future post.

DEV Community: Garrett Mills

Salvaging a Corrupted Table from PostgreSQL

Mitigating the iconv Vulnerability for PHP (CVE-2024-2961)

Check if your OS is vulnerable

Check for bad encodings

Disable bad encodings

Docker

Down the Rabbit Hole of Linux Terminal Emulators

GNOME Terminator

Tilix

DDTerm

Conclusion

Adventures in AI-assisted proof generation

The A Problem With Coq

The A Problem with ChatGPT

Factorial

Conclusion?

Generalized Commutative Data-Types

Naïve Commutative Data Types: A First Draft

Pseudo-Commutative Operations

Another GCDT: Sets

Pseudo-Commutative Operation Precedence

Applications

Rancher K3s: Kubernetes on Proxmox Containers

What We're Building & Rationale

Prerequisites

Creating the LXC Containers

Additional Configuration

Setting Up the Container OS & K3s

Setup the control node

Setup the worker node

Setting up NGINX Ingress Controller

Conclusion

Runtime Data Validation from TypeScript Interfaces

Runtime schemata: a plague of DSLs

A very deep rabbit-hole

Zod-ifying the Interfaces

Wrangling the AST

First, Identify

Then, Modify

Conclusion

Rethinking Dependency Injection in TypeScript

Named Injection

Drawbacks

Constructor Injection

How does this work?

Instantiating the Class

Drawbacks

Property Injection

How does this work?

The Prototype Chain

The Issue

Best of Both Worlds

Drawbacks

Conclusion

Converting an Ionic/Angular Site into a Progressive Web App

Service Workers

Angular Service Worker

ngsw-config.json

Web Manifest

Building your application

Running in a sub-route (/app, &c.)

Debugging

Don't modify the compiled Angular code

Bypass the service-worker

View service-worker logs

View cached files

Conclusion

`ngsw-config.json`

Running in a sub-route (`/app`, &c.)