DEV Community: Sebastian Kurfürst

Using Nginx to Migrate Legacy Systems

Sebastian Kurfürst — Sun, 27 Oct 2019 19:11:30 +0000

tl;dr: Using Nginx to migrate legacy systems is very powerful. By clever combinations of rewrite rules and reverse proxies, one can implement seamless transitions!

We demonstrate the above by replacing a CMS with Neos CMS, but similar ideas can be applied to many kinds of systems.

The Problem - Adopting a new CMS incrementally

For a customer of us, it became evident that he could not implement his needs with his previous CMS – and after we have showed Neos CMS, it became quickly appearent that they would profit a lot by its functionalities, especially flexible content modelling, redirect management, and SEO functionalities.

However, the customer had quite a lot of content in the old CMS; and also special functionalities like a shopping basket and different kinds of forms – so we figured out that replacing everything in a big bang would be too expensive.

For us, the core question was: How can we unlock the potential of Neos CMS for the customer, in a very cost-effective way?

On the plus side, the customer had recently adopted a new design which he was very happy with; so we wouldn't need to touch any of this.

Solution Idea: Deliver Neos Content first, then fall back to the old CMS

At [Neos Conference 2018](http://neoscon.io], I've attended a talk by Sebastian Heuer and Arne Blankerts about escaping legacy hell - where they introduced the pattern of using Nginx reverse proxies, configured in a clever way. Basically, the idea is to ask the new system first, and when the new system returns a 404 page, intercept that at the Nginx-layer; and instead ask the old system for the same URL.

This way, we were able to re-build the template in Neos, and migrate incrementally: We took a page where the current structure was not as expected, and we re-built this page in the new system.

Nginx Configuration to migrate legacy systems

Let's zoom in a bit on the required Nginx configuration - which, at the core, boils down to the following settings:

location / {
    # default for Neos CMS; to redirect all pages which do not exist on disk to index.php
    try_files $uri $uri/ /index.php?$args;
}

# default config - passing PHP calls to PHP-FPM via FastCGI
location ~* \.php$ {
    fastcgi_pass   unix:/tmp/php7-fpm.sock;
    # ... quite some more fastcgi parameters here

    # when PHP delivers an HTTP error, do not send this to the end-user directly,
    # but instead evaluate the error_page directive.    fastcgi_intercept_errors on;     

    # redirect to the named @legacy location in case of a 404 from the new system.    
    error_page 404 = @legacy;     
    # also allow if that happens multiple levels.   
    recursive_error_pages on;
}

# definition for the legacy backend
location @legacy {
    # ... (see below)
}

Now, I had some trouble defining the legacy backend correctly, because with my first tries defining proxy_pass, the URL /index.php was always called on the legacy system. Then, a bit of debugging and docs reading revealed what was going on:

The try_files directive works like some internal redirect; effectively rewriting the Nginx internal variables such as $uri to /index.php.
then, the PHP location block is executed - and if PHP returned a 404 error, the current request (which was still a request to /index.php) was returned to the legacy system.
the legacy system (of course) could not deal with requests to /index.php and returned an error.

So, the question was: How can we use the original URL (before starting to rewrite) and pass that to the legacy upstream system?

The solution is quite easy, but it took a bit to figure this out. The core idea was to use variables inside proxy_pass, to build up the target URL dynamically:

location @legacy {
    # we use the ORIGINAL request URI (including all arguments) here.
    # To quote the docs at https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_pass:
    #     When variables are used in proxy_pass the URL
    #     is passed to the server as is, replacing the original request URI.
    proxy_pass https://old.my-url.de$request_uri;

    # we need to define a resolver, as we use variables and a full domain name in proxy_pass
    resolver 8.8.8.8;

    # for the dynamic parts to work, we need to rewrite Location headers to be relative again, because we use a
    # dynamic $request_uri in proxy_pass - and thus the default rewriting of proxy_redirect does not happen anymore.
    proxy_redirect https://old.my-url.de/ /;
    proxy_redirect http://old.my-url.de/ /;
}

Add a Fast-Path for static assets

In our setup above, the dynamic pages are handled by php-fpm; and all static files (like images, css, ...) are delivered directly from the nginx system which also does the proxying. Thus, we want to avoid asking our new backend for every static asset; just to get the 404 returned, and then to ask the legacy system (which has the asset).

To fix this, a rather simple additional rule is needed for handling static file types directly:

location ~* \.(?:css|png|js|woff|woff2|svg|jpg|jpeg|ttf|pdf)$ {
    # if these static files do not exist on disk, directly ask the legacy backend.
    error_page 404 = @legacy;
}

Adding Caching

Finally, to further improve response times, we added a simple cache which caches all legacy-upstream resources for 10 minutes, by using the rules below:

proxy_cache_path /tmp/nginx-cache levels=1:2 keys_zone=legacy:50m inactive=24h  max_size=1g;

location @legacy {
    #  ... the config from above, and additionally the following lines:

    proxy_cache legacy;
    proxy_cache_background_update on;
    proxy_cache_use_stale  error timeout invalid_header updating
                           http_500 http_502 http_503 http_504;
    proxy_cache_valid 200 302 10m;
}

Summary

In this post, we have shown how Nginx can be used to direct requests dynamically between a new and a legacy system - which can be a powerful and cost-effective way to migrate from a legacy system. Let's finish this post with some properties of this approach, to ease the decision whether this approach is worth it.

This approach (for websites) can only be used if the old and the new system have the same visual look-and-feel; i.e. HTML markup can be re-used.
It best works for systems which are well-segmented; so we do not have lots of interconnections between the "old" and the "new" system. It is a lot harder if the old and the new system need to access the same underlying data store.
The approach does not contain an in-built way to access shared data; i.e. a page is always rendered purely in the new, or purely in the old system. This makes it a bit harder to deal with global data like a search index, menus, or sitemap.xml (which has to be taken care of differently).
The approach works best for read-heavy systems.
This approach allows to use many powerful Neos features, like redirect management, also for content in the old system (as redirects are returned first, from the new system).

Looking forward to your feedback!
All the best,
Sebastian

How To Stop Poking and Start Structured Debugging

Sebastian Kurfürst — Mon, 07 Jan 2019 20:08:25 +0000

This post explains why structured debugging is hard; and introduces a method to make this process more straightforward. However, before that, we first need to introduce a basic mental model about software development, which will serve us as an analogy for later.

A Mental Model: How We Develop Software

Usually, when we start developing inside an existing, bigger application, I feel this process resembles a lot exploring an unchartered island. At the beginning, we only roughly know what we want to archive, but we do not know yet how to reach that goal. We have to first explore our nearby environment.

Then, we go a few steps, and adjust our course as we go. At first, the steps we take are rather small; but then steps will become bigger as we know more and more about our island (depicted by the big gray area trailing behind you) and are thus able to make more correct assumptions about how the system behaves.

Thus, over time, we are improving our development speed because our assumptions are getting better and better; and thus we can go multiple steps at once, before validating them all together by trying out the application again. Turns out this will become a problem lateron when we start debugging.

Developing in big existing applications is like charting a previously unchartered island.

Debugging Unexpected Behavior

When an unexpected behavior happens, I often see people poking around; often changing various parameters at once before re-running the application. Basically, we this is what happens when we apply our thinking of efficient software development (see above) to debugging - but this leads us to a totally wrong path.

We basically think "we know how the system will behave" - and thus, we are going too far at once; hitting walls (depicted in orange) which do not lead us to our solution. When one of these routes does not work, we start again from scratch and retry a different direction - however again going too far into the other direction as well. The problem: By doing this, we do not learn anything new about our system anymore (we just become confused).

We need to apply another kind of thinking to solve this debugging challenge. Let's assume for a second we did not know anything about our software when we hit a behavior violating our assumptions. How could we progress then?

The Solution: Design Experiment; Run it; Analyze; Repeat

We could assume our system is a black, opaque box, which we do not understand. Now, let's apply what a scientist would do to understand it better:

Design an experiment which divides the problem space at around 50%, to figure out whether our unexpected behavior resides in the left or the right half of the problem space. It is important that this experiment returns definitive answers (and no assumptions and no probabilities). This is depicted with the blue line and the A/B sections in the image above.
Run the experiment.
Analyze the experiment results. You should now know definitely at which part of the problem space your unexpected behavior occurs.
Repeat with a new experiment, again bisecting the relevant part of the problem space again.

Of course, even when doing this,it can happen that an experiment fails without yielding good results. Then, you need to understand that this has happened and design a new experiment, basically discarding the old experiment.

Practical Hints

Designing good experiments is the hardest part of the process above.
Designing good experiments needs some experience; and you will see this gets easier over time.
When you start with designing experiments, you should explicitely write down the experiment's assumption which should be validated or falsified. Furthermore, write down the experiment setup.
For systems consisting of multiple coarse components, it is usually a good start to create an experiment to isolate which component is responsible for the unexpected behavior – so first bisecting at component boundaries is a good start. For client-server webapps, this means first figuring out whether the client-side portion of the application or the server-side portion of the application is responsible for the unexpected behavior.

Closing Thoughts

Basically, when developing, you want to understand as much as possible from the system; so I call this White-Box Thinking. On the other hand, when debugging, you often need Black-Box Thinking. Switching between the two is often difficult; I hope this essay can help to make this a little easier.

Looking forward to your feedback,

Sebastian

PS: This is quite similar to the behavior of git bisect: In case you need to figure out when a certain regression has entered your codebase, you can use this to identify the commit since when the problem existed.
PPS: This essay has been first published at our company blog.

Writing React with TypeScript

Sebastian Kurfürst — Wed, 02 Jan 2019 07:05:28 +0000

Just a quick note today as I stumbled over a a nice GitHub repo:

We've made good experiences writing React Applications with TypeScript -- and I just now stumbled across a good cheat sheet which summarizes many of the lessons learned:

typescript-cheatsheets / react-typescript-cheatsheet

Cheatsheets for experienced React developers getting started with TypeScript

React+TypeScript Cheatsheets

Cheatsheets for experienced React developers getting started with TypeScript

Web docs | 中文翻译 | Español | Contribute! | Ask!

👋 This repo is maintained by @swyx, @ferdaber, @eps1lon, @IslamAttrash, and @jsjoeio, we're so happy you want to try out TypeScript with React! If you see anything wrong or missing, please file an issue! 👍

All React + TypeScript Cheatsheets

The Basic Cheatsheet (/README.md) is focused on helping React devs just start using TS in React apps
- Focus on opinionated best practices, copy+pastable examples.
- Explains some basic TS types usage and setup along the way.
- Answers the most Frequently Asked Questions.
- Does not cover generic type logic in detail. Instead we prefer to teach simple troubleshooting techniques for newbies.
- The goal is to get effective with TS without learning too much TS.
The Advanced Cheatsheet (/ADVANCED.md) helps show and…

View on GitHub

We've summarized our own best practices at sandstorm in a GitHub boilerplate (created by my great colleague Robert Baruck), with the best practices summarized in a GitBook:

sandstorm / typescript-react-app-kickstart-guide

Kickstart guide for React Redux Apps written in TypeScript

React TypeScript kickstart

This the source for a guide for kickstarting a new React project with preconfigured Redux, TypeScript, Storybook and testing - all powered by docusaurus. If you find errors, inconsistencies or want to contribute: Feel free to submit issues & pull requests.

Prerequisites

These tools are needed to start developing.

# node version management
brew install nvm

# node (latest lts version)
nvm install --lts

# yarn
brew install yarn --without-node

Editor

We recommend to use VSCode as editor. Install the following Extensions:

TSLint -> linting support directly in editor (marketplace)
Type safe React & Redux Snippets in TypeScript -> Snippets for boilerplate code as shown in Cookbook (marketplace)

Cookbook

In the cookbook you will find skeletons for important modules. These skeletons are also available for VSCode as snippets in the VSCode marketplace.

Example (Component with defaultProps)

Look, Ma' - Intellisense!

View on GitHub

We've also published [a VSCode Snippet Extension(https://marketplace.visualstudio.com/items?itemName=Sandstorm.vscode-awesome-ts-react-redux-snippets) which helps to write TypeScript/React with little boilerplate.

What are your experiences writing React + TypeScript?

All the best,
Sebastian

Event Sourcing and CQRS

Sebastian Kurfürst — Tue, 01 Jan 2019 14:10:07 +0000

I've had the pleasure to join an EventSourcing and CQRS workshop with Mathias Verraes in Frankfurt, to figure out the next steps on how to apply Event Sourcing and CQRS to the Neos Content Repository.

This blog post is about my learnings in general, and not about our specific implementation plans in Neos.

Setting the stage: How state is handled and modified

Normally in web applications, people use relational databases to store their application state. Many frameworks apply patterns such as Active Record or Domain Model to make state accessible and to modify it. No matter what pattern is used, usually an object is responsible for both reading from the state as well as updating it. This is different when using CQRS.

CQRS stands for Command Query Responsibility Seggregation; basically meaning that you'll use one object to read from the state (the Query), and a different object to modify the state (the Command).

This allows you to separate the read from the write side; allowing to use different representations for state modification and for reading – and it allows multiple commands and multiple read models to be used (more on that later)!

Normally, you store state in your database. You have tables which contain e.g. the current user name, or the contents displayed currently on your website, or your currently available products. When this state is modified, this happens rather implicitely (e.g. through some method calls resulting in database queries). We'll now think of this state modification as an event.

The core idea behind Event Sourcing is different from the conventional approach: We won't store the current state, but rather all the events that led to it. These events will be stored in an event store, which is conceptually just a long, ordered, append-only list of all events which have ever happened in the system.

This way, we can build up the current state by taking a "blank" state, then applying all events which have ever happened, and then we have the current state. Of course this would be horribly inefficient (more on this later, we're currently on the conceptual level). More than that, we're able to travel forth and back in time: We can also compute all previous states as they existed in the system.

What if we don't store the current state, but rather all events that led to it?

Having access to all previous states which happened in the system is an extremely powerful paradigm: You can, for example, see, how often your data was modified. By whom. What the old value was. So you have an audit log.

Additionally (and that's for me the core benefit), you are able to take all old events in your system and use them to build up additional state for answering different questions than originally anticipated. We'll cover this in a bit.

It turns out that event sourcing and CQRS work together beautifully:

On the write side, you "basically" will just create events which you store in the event store.
On the read side, you consume the events and build up the state you need for your application. This is called a projection.

Commands, Events, and Projections

Let's ensure we are on the same page regarding some core vocabulary:

A command is a state-mutation-intent. It has not yet happened. It might still fail. It corresponds to the "write" side of CQRS. Usually a command is written like "Create User" or "Add Node to the system".

An event, on the other hand, is the indication that a state mutation has happened. Usually it is written in past tense, like "User has been created" or "Node has been added to the system". An event is persisted in the event store, which is an append-only log of all events.

A projection reads all events and creates read-only state out of it. It corresponds to the "read" side of CQRS. When a new event comes in, the state is updated accordingly. Often, the projection state is stored in standard database tables. Projections, by nature, can always be re-created by clearing their state and applying all events (from the beginning of time) again. Basically, a projection is something like a cache.

Event Sourcing and CQRS – The Unified Model

Following is a diagram which shows how CQRS and Event Sourcing work together. The diagram is split into three parts:

On top is the external/API side of our system.
On the right is the command side which mutates the state by generating events.
On the left is the read side which contains different projections which can be queried.

There is inherent asynchronity through the event store. This means that when a command is executed (and stored as event in the event store), it might take some time until the projections on the read side have been updated.

1 - Simple Case - No Constraints

In the most simple case, here's what happens:

1a - A command is sent which can always be applied ("Update the user's last name")
2a - The command is directly converted to an event ("User's last name has been updated") and stored in the event store.
(asynchronity)
3 - Sometimes later, the projections are updated, taking the event into account.
4 - Now, when the user queries the projections, he'll see the updated state.

2 - Hard Constraints - Enforced through Aggregates

Sometimes, you need to ensure that an invariant on the state is always true. For example, it might be useful to define: "It is never possible to allocate a username twice." We basically need a safeguard to ensure that the "UserCreate" command always has distinct user names. This safeguard is called an aggregate. (Do not confuse it with the term "Aggregate" in Domain-Driven-Design.) How does this safeguard work? It cannot read a projection, because this projection might be out-of-date.

Thus, an aggregate works usually in the following way:

1b - A command is sent to an aggregate/command handler.
The Aggregate's state must be reconstituted so it can do its decision:
- 2b - The aggregate first receives all events which it has previously emitted from the event store.
- This way, it is able to reconstitute its state from previous events.
Then, the hard constraint is checked; i.e. in our example it is checked whether the username is already taken.
2c - If the constraint is fulfilled, the event is emitted to the store.
Otherwise, the command cannot be executed, and an error is returned.

3 - Soft Constraints - Query the read side to ensure invariants are "almost always" met

It turns out that hard constraints are not always needed; quite to the contrary: If you think hard enough about a problem, you need them quite rarely. A soft constraint, on the other hand, is not a real constraint in the sense that an invariant is always met; but it can be extremely likely that the invariant is still met. We'll cover this more in-depth later. A workflow covering soft constraints works in the following way:

1c - A command is dispatched.
Before the command is transformed to an event, a soft constraint check can take place in the following way:
5 - We query some projection, knowing that it might not be fully up-to-date yet.
2a - Based on the projection result, we either accept the command and dispatch its corresponding event to the store, or we discard the command.

Consistency, Constraints and Invariants

Before we dive into which constraints should be used when, let's talk a little about consistency.

We often like to think about systems being immediately consistent (also called strongly consistent). This means that an event enters the system at a certain point in time, and everybody who accesses the system will be able to retrieve the new state directly after this event has been applied.

On the other hand, systems might be not consistent at all – this is generally a state which you try to avoid very hard; as usually you cannot easily recover back to consistent state from there.

Between systems being immediately and not consistent, there's a big space in between which is called eventually consistent. The core idea is that certain periods of time where the system is inconsistent might be acceptable, but it needs to be ensured that after "some time" (eventually) the system is consistent again. We'll explore this idea a bit further!

Eventual Consistency

So, let's explore what Eventual Consistency actually can mean. Basically, we want to ensure the system should appear as consistent as possible. "Appearing" consistent to me mostly means that a user gets a "consistent" view of what he did, i.e. when he created a comment on an article and reloads the page, he should see his comment; so we need to have a high probability that the relevant projection has already been updated.

On the other hand, it does not really matter that much when other users see the comment; it is fine if there is a delay of a few seconds (which is a long time for "eventual consistency"). It also does not matter when e.g. a search index is updated.

So, there are two tricks we can apply to make the system perceived as consistent, although "by the book" it is eventually consistent:

We usually do not need to update all projections synchronously, but rather many can be updated asynchronously.
When a change is visible for one user, it does not immediately need to be visible to other users of the system.

A small sidenote: Often it is suggested that a user interface should "simulate" the behavior, e.g. showing the new comment already before we got the confirmation from the server that the comment was successfully added. I personally think this is a nice trick to improve the perceived responsiveness of applications; but I still would ensure that the needed projections run mostly synchronous. Otherwise, you will get strange effects when e.g. pages are reloaded etc.

Why Soft Constraints are often sufficient

I think this is best explained with an example. In the Neos content tree, there is a "MovedNode(source, target)" event indicating that the "source" node has been moved to "target". Let's say user 1 tries to move "A" inside "B", while user 2 tries to move "A" inside "C". Thus, there's a conflict.

When I thought of preventing conflicts, I was directly thinking how a hard constraint with an aggregate could prevent this conflict. However, it might actually be not so easy enforcing this constraint with an aggregate.

Let's for a moment think what happens when a soft constraint would be applied. If two people move the same node into different directions at the same time, the soft constraint check (5 in the image above) would succeed, because the projection was not yet updated. Thus, both move events would hit the event store. Now, during projection time, we have a few options:

We can move the node from A inside B, discarding the move of user 2
We can move the node from A inside C, discarding the move of user 1
We can move the node from A inside B, and then move the node from inside B to inside C (or vice versa, depending on the event ordering)

In all cases, one user will be a little confused because his change looks as if it was ignored by the system; while for the other user the end-result is the expected one. If we had hard constraints in place, then the "second" user would get an error that the move could not happen – this is unexpected for him as well. Thus, no matter whether we use hard or soft constraints, we'll always have one confused and one non-confused user in this example. Additional to that, the user whose command failed will probably just re-try his action; and chances are extremely high that the user's action will now work. (Unless, of course, another user moves the node again at exactly the same time :-) ). Thus, why should you use a hard constraint here if it has roughly the same experience for your end-user than when using a soft constraint?

To sum it up, soft constraints are important because they catch the common case. They ensure your system is perceived as being consistent. They make it extremely likely that the system behaves predictable for your users.

Often, you'll find very few cases where compensating an error would be very costly; these are usually good candidates for hard constraints. Then, try to create aggregates, where each of them should only care for one of these constraints.

Soft constraints catch the common case. They ensure your system is perceived as being consistent.

Implementation Details

Here are some further implementation details which are important on a detail level:

As already stated, try to reduce the number of hard constraints as much as possible.
Treat your aggregates as implementation details of the write side. They are just very strong constraint checkers, after all.
Name your aggregates after the invariants they protect. Call them UserUniquenessAggregate instead of UserAggregate.
If one aggregate becomes too imperformant, you can often partition these further; e.g. have one UserUniquesAggregate per first-character of the user name.
Soft constraints should be checked inside your command pipeline [I still dislike this name]; e.g. before the command is converted to an event and persisted in the event store.

There's one topic which I did not cover at all yet, but which is important for many real-world applications: The concept of a Process Manager. Basically, a process manager listens to emitted events (pretty much just like a projection); and can react to events by emitting new commands. They allow to encapsulate side-effects, such as accessing configuration or getting the current system time. Furthermore, they allow to orchestrate bigger flows, pretty much like React Redux Sagas.

How we'll use all this in the Neos Content Repository

This article covers my thinking and mindset which I currently use to approach the Neos Content Repository domain. It is not showing yet the specifics we want to use to apply the CQRS / Event Sourcing methodology. This is the topic for another blog post!

PS: All views expressed here are my own; and of course, all errors and misunderstandings as well – @mathias - I hope there are not too many of them in here :-)
PPS: This is a cross-post of my post at our company blog