DEV Community: Peter Ruttkay-Nedecký

Don’t let your RxJS observables die

Peter Ruttkay-Nedecký — Sun, 10 Oct 2021 20:08:55 +0000

RxJS is a powerful library for reactive programming. We use it in Angular all the time, from simple reactions to HTTP client responses to full-blown state management solutions. But sometimes, it is easy to miss some details and create a code that can break at the worst possible moment.
Therefore, we will dive into a few scenarios that you should remember when dealing with errors in RxJS observable executions.

Keep it alive

Let’s start with an example right away.

of(1, 2, 3, 4, 5)
  .pipe(
    tap(v => {
      if (v === 3) {
        throw new Error('some error');
      }
    })
  )
  .subscribe({
    next: console.log.bind(null, 'next:'),
    error: (err: Error) => console.log('error:', err.message),
  });

We have an observable that emits numbers from 1 to 5, and if the number is 3, we throw an error.

We can see the following output.

next: 1
next: 2
error: some error

Numbers 4 and 5 were not processed at all.

This might be surprising at the beginning, but if we check the RxJS observable documentation, we can find a clear definition of the observable contract as:

In an Observable Execution, zero to infinite Next notifications may be delivered. If either an Error or Complete notification is delivered, then nothing else can be delivered afterwards.

In our case, the execution has delivered an error for the third item, and thus it can not deliver any other notifications.

Let’s move into something more complex - we have routing in our application, and we want to build an Angular component that displays the current URL.

this.router.events
  .pipe(
    filter(event => event instanceof NavigationEnd),
    tap((e: NavigationEnd) => this.thisCanThrowAnError(e.url))
  )
  .subscribe({
    next: e => this.currentPath = e.url,
    error: this.notifyError.bind(this),
  });

Here we subscribe to router events and use the tap operator to call a function that can potentially throw an error. If an error occurs, we show a notification. But as we already know, we will lose the subscription if an error is thrown. If we navigate to another route afterward, the displayed path won’t be updated anymore, and we would have to call the subscribe again.

So, how to handle this problem? One option would be to catch the exception right inside of the tap handler. You can imagine this would be a hassle because we would have to do it for each operator that can throw an error. Instead, we would like to handle it in one central place.

Luckily, RxJS has us covered, and we can use the catchError operator. The catchError operator catches an error and enables us to handle it in some way. The vital thing to remember is that the catchError operator expects a new observable that will replace the old failed one. In our case, we would like to continue with the observable we already had, so all we have to do is return the observable sent to our handler after we display the error notification.

currentPath$ = this.router.events
    .pipe(
    filter(event => event instanceof NavigationEnd),
    tap((e: NavigationEnd) => this.thisCanThrowAnError(e.url)),
    map(e => e.url),
    catchError((err, failedObservable) => {
      this.notifyError(err);
      return failedObservable;
    })
  );

As a side note, we are not handling the error inside of the observer anymore. Therefore we can leverage the Async pipe provided by Angular and remove the explicit subscribe.

One thing to remember, when an error occurs, the currentPath$ will not emit any value to the Async pipe for the URL that caused the error. Therefore we won’t see any update to the current path. The reason is that we are resubscribing to router events, and the last event has already been emitted and processed. We can prepend a fallback value to the observable returned from the catchError operator to solve this issue.

currentPath$ = this.router.events
  .pipe(
    filter(event => event instanceof NavigationEnd),
    tap((e: NavigationEnd) => this.thisCanThrowAnError(e.url)),
    map(e => e.url),
    catchError((err, failedObservable) => {
      this.notifyError(err);
      return failedObservable.pipe(startWith('failed to retrieve'));
    })
  );

We will, in this case, display the message “failed to retrieve” instead of the old URL.

Break the loop

So far, so good, but we still have some caveats to explore.

Let’s return to our first example with the simple of observable and apply the same fix.

of(1, 2, 3, 4, 5)
  .pipe(
    tap(v => {
      if (v === 3) {
        throw new Error('some error');
      }
    }),
    catchError((err, failedObservable) => failedObservable)
  )
  .subscribe();

Do you see the problem? The catchError operator will resubscribe to the returned observable, and therefore the observable will emit all of its items again. We will fail on item 3 again, and we will also process the previous items multiple times.

In a more real-world case, this problem would occur if we use any kind of replay. Let’s rewrite the simple example using ReplySubject.

const subject = new ReplaySubject(1);
subject
  .pipe(
    tap(v => {
      if (v === 3) {
        throw new Error('some error');
      }
    }),
    catchError((err, failedObservable) => failedObservable)
  )
  .subscribe();

subject.next(1);
subject.next(2);
subject.next(3);
subject.next(4);
subject.next(5);

In this case, ‘3’ will be emitted again after the resubscribe in the catchError operator. The error in the tap operator will be therefore thrown again, resulting in an infinite loop.

How can we skip the failing item? The first idea could be to simply skip the failing item when returning the new observable from the catchError.

Something like this.

failedObservable.pipe(skip(1))

This will not work because failedObservable already contains the tap operator where the error occurs. We have added the skip operator after it, so it is too late. We can overcome this problem by moving the handling above the tap operator.

let lastValue;
const subject = new ReplaySubject(1);
subject
  .pipe(
    filter(v => v !== lastValue),
    tap(v => {
      lastValue = v;
      if (v === 3) {
        throw new Error('some error');
      }
    }),
    catchError((err, failedObservable) => failedObservable),
  )
  .subscribe();

subject.next(1);
subject.next(2);
subject.next(3);
subject.next(4);
subject.next(5);

This way, we successfully skipped the failing item. But let’s face it, it looks somehow ugly. We need another variable, and the RxJS pipeline is not neatly self-contained anymore.

Let’s try to go a step back.

Until now, we have been trying to keep the failing observable alive by retrying it. But what if we accept that it will become unsubscribed instead? We will prepare for it by creating another observable that we can sacrifice in the case of an error.

const subject = new ReplaySubject(1);
subject
  .pipe(
    mergeMap(v =>
      of(v).pipe(
        tap(v => {
          if (v === 3) {
            throw new Error('some error');
          }
        }),
        catchError(() => EMPTY)
      )
    )
  )
  .subscribe();

subject.next(1);
subject.next(2);
subject.next(3);
subject.next(4);
subject.next(5);

In this case, we create another observable from the current value and flatten it using the mergeMap operator. We catch the error for item 3 inside the mergeMap operator as a part of the inner observable. We will cause the inner observable to complete by returning the EMPTY observable, but the important thing is that it will not end the outer observable.

Service calls

We could also apply this pattern to our example with the router.

currentPath$ = this.router.events.pipe(
    filter(event => event instanceof NavigationEnd),
    switchMap(event =>
      of(event).pipe(
        tap((e: NavigationEnd) => this.thisCanThrowAnError(e.url)),
        map(e => e.url),
        catchError(err => {
          this.notifyError(err);
          return of('failed to retrieve');
        })
      )
    )
  );

Note that because we are not using any reply, it doesn’t make much sense to use the inner observable in this case. Still, it brings us closer to a real-world example - a back-end API call to retrieve some information as a reaction to an observable emission.

Let’s say that instead of getting some information from the activated route, we would like to call an API based on the current path. Such a call can fail anytime, e.g., because the API server is currently unavailable. Therefore we need to prepare for that case.

currentPath$ = this.router.events
  .pipe(
    filter(event => event instanceof NavigationEnd),
    switchMap((e: NavigationEnd) =>
      this.someApiCallThatMightFail(e.url).pipe(
        catchError(err => {
          this.notifyError(err);
          return of('failed to retrieve');
        })
      )
    )
  );

We catch the potential error inside the switchMap operator, and thus we ensure that the error will not take down the whole observable execution.

Conclusion

We have looked into some cases that we should remember when dealing with a code that could throw an error in our observable execution.
Remember to catch errors correctly if you wish to keep the execution alive, and also beware of loops if you rethrow those errors.

You can find the example code with the router events here:

How to use an Android device for ASP.NET development

Peter Ruttkay-Nedecký — Tue, 05 Jan 2021 19:11:28 +0000

In this article, I will show you how to turn your Android device into a machine for web development with .NET 5. For the sake of fun, we will do all the necessary configuration steps directly on a tablet without the aid of another computer. The basic idea is to use code-server, which enables us to run VsCode on a remote machine and access it in a browser.

Set up a virtual machine

We are going to use DigitalOcean as a provider for our virtual machine (VM). DigitalOcean offers a Droplet (Linux-based VM) with an already configured code-server. The problem is that it contains an old version of code-server, and I had a hard time trying to update it to the newest version. Therefore, we will go with the clean Ubuntu Droplet, and we will install code-server ourselves.

The cheapest option is enough for our testing, but you can go with the $10/mo options if you plan to do some serious stuff. The nice thing about DigitalOcean is that you are charged only for the time when the VM is allocated to you. So, you can even go with the $80/mo option, play with the VM for an hour, then destroy it, and you are going to be charged only $0.119.

We are going to use SSH for accessing our server during setup. For this, we can use Termux, which is an Android terminal emulator and Linux environment app. You can use Termux for many development related tasks - even to install NodeJS and use it to develop web applications to some degree. Sadly, at least at the time of writing this article, it is impossible to use it for .NET development. But it is worth to keep an eye on Termux/.NET compatibility for the future - https://github.com/termux/termux-packages/issues/516.

For SSH access, we first need to generate an SSH key, so let’s install OpenSSH.

pkg upgrade 
pkg install openssh

And then generate our SSH key pair.

ssh-keygen -t rsa -b 2048

Keep the default location when prompted, and then print the public SSH key.

cat ~/.ssh/id_rsa.pub

We can now go back to the Droplet setup and enter the public key in the Authentication section.

And that’s it. You can also change the name of the server and then finish the creation of the server. It will take only a few seconds for the setup to finish.

Let’s go back to Termux and run the following command to connect to the newly created server.

ssh root@[ip address of your server]

You can obtain the IP address from the list of Droplets.

code-server user

Before we install code-server, we need to prepare a user under which code-server will run. The reason for this is that code-server enables us to run commands directly on the server through the terminal in VsCode, and it is safer if we don’t run them under the root user by default.

Let’s create a new user using the following command.

adduser codeuser

You can, of course, pick any name you like instead of “codeuser”. To run administrative tasks under the “codeuser” user using the “sudo” command, we need to add the user to the “sudo” group.

usermod -aG sudo codeuser

code-server

We are now ready to install code-server. We will use the official code-server documentation as a guide, but mind that the guide also contains steps for creating a VM on Google Cloud, but we already have one. Therefore, we can skip right to the installation of code-server package.

curl -fsSL https://code-server.dev/install.sh | sh

Let’s enable the code-server service using the following command.

systemctl enable --now code-server@codeuser

The “codeuser” string after the “@” character specifies that the code-server service should run under the user we created before. From the technical point of view, we have simply defined the instance’s identifier, which is used as a value for the “User=” option in the unit template.

We can see that by printing out the “/lib/systemd/system/code-server@.service” template.

cat /lib/systemd/system/code-server@.service

And inspecting the following line.

User=%i

Expose code-server

We must expose code-server to the internet using a reverse proxy to access it through a browser. For the sake of security, we are going to expose code-server only through HTTPS. We will therefore need an SSL certificate. We can use Let’s Encrypt, which provides SSL certificates for free. The only downside is that we also need a domain because Let’s Encrypt doesn’t offer certificates for plain IP addresses.

Our first step after obtaining a domain is to point it to the server IP. We can do that by creating an “A” record in the DNS configuration of your domain provider. Set the record’s value to the IP address of your server and Host to a subdomain that you like (e.g. “www”). Mind that the DNS propagation may take up some time (24-48 hours, but usually it is much faster).

We are going to use the caddy server for the reverse proxy. It makes HTTPS setup with Let’s Encrypt really easy. First, we need to install caddy by following steps from the official documentation.

sudo apt install -y debian-keyring debian-archive-keyring apt-transport-https

curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/cfg/gpg/gpg.155B6D79CA56EA34.key' | sudo apt-key add -

curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/cfg/setup/config.deb.txt?distro=debian&version=any-version' | sudo tee -a /etc/apt/sources.list.d/caddy-stable.list

sudo apt update

sudo apt install caddy

Now let’s configure the reverse proxy. Replace the content of “/etc/caddy/Caddyfile” with:

[my-domain]
reverse_proxy 127.0.0.1:8080

Where “[my-domain]” is the domain that we configured “A” record for – e.g. “www.my-domain.com”.

Now, let’s reload the caddy server.

sudo systemctl reload caddy

We are ready to access the code server. Open “https://[my-domain]” in a browser (replace [my-domain] with your domain), and we will see the login screen of our code-server. We can find the password by running the following command.

cat ~codeuser/.config/code-server/config.yaml

If you wish to change it, modify the “config.yaml” file and reload the code-server service using the following command.

sudo systemctl restart code-server@codeuser

We can run the rest of the configuration directly from the VsCode terminal, and we may therefore close the SSH connection in Termux.

For additional security, we are going to enable the UFW firewall. We need to allow SSH connections and HTTPS connections on the default port (443).

ufw allow OpenSSH
ufw allow https

Now we can enable the firewall.

ufw enable

Install .NET SDK

We can install the .NET SDK by following the steps from https://docs.microsoft.com/en-us/dotnet/core/install/linux-ubuntu.

Run the following command (for Ubuntu 20.04 - if you are using another Linux distribution, go to https://docs.microsoft.com/en-us/dotnet/core/install/linux and select it).

wget https://packages.microsoft.com/config/ubuntu/20.04/packages-microsoft-prod.deb -O packages-microsoft-prod.deb
sudo dpkg -i packages-microsoft-prod.deb

And then.

sudo apt-get update; \
    sudo apt-get install -y apt-transport-https && \
    sudo apt-get update && \
    sudo apt-get install -y dotnet-sdk-5.0

Our first ASP.NET application

We are ready to create our first ASP.NET application using the code-server terminal in the usual way.

mkdir webapp1
cd webapp1
dotnet new web

And we can start it.

dotnet run

The only problem that is left before us is how to access the application because it runs on localhost on the remote server. We can, of course, inspect the response using curl.

curl http://localhost:5000

But that would be rather painful. We would typically create a reverse proxy in the caddy server to expose the local addresses, but code-server has us already covered. Just head to “https://[my-domain]/proxy/5000”, and we can see our ASP.NET application. That is because code-server acts as a simple reverse proxy where each request to the “/proxy/[port]” path is forwarded to the appropriate port.

Conclusion

We have managed to create a development environment out of our Android device. And on top of that, we managed to do the whole setup from the Android device itself – so you can do it even on a bus 🙂. If you own an iPad instead, then luckily, you can follow the same steps, but instead of Termux, you will use Blink Shell, or Ish.

So, that’s it, happy coding!

Sources

Integrating React into ASP.NET Core using Razzle with all the goodies like SSR, routing, code splitting, and HMR – Part 2/2

Peter Ruttkay-Nedecký — Sat, 17 Oct 2020 20:56:02 +0000

In the first part of this article, we have created a React application with server-side rendering (SSR) in ASP.NET Core. We did not use the well-known library ReactJS.NET but instead went a different way that gives us greater flexibility. We helped ourselves with a tool called Razzle and used Javascript.NodeJS to call NodeJS from ASP.NET Core.

In this part, we will add data loading to both the client and the server-side. We will look into code splitting and wrap up with a deployable package of our application.

Data loading

We need to be able to access data on three occasions.

When the request first comes to the application, we need to return HTML containing a fully rendered page. For that, we need to provide data into our React application while rendering it on the server side using renderToString.
During hydration, we must render the same HTML on the client-side. That means we need the data again.
Finally, when we do a client-side routing, we need to load the data from the server using AJAX.

Let’s make a simple in-memory data provider that we will use in our HeroController.

public class HeroDb : IHeroDb
{
    private readonly Hero[] _items = new[]
    {
        new Hero{
            Id= 1,
            Name= "Luke Skywalker",
            Height= 172,
            Mass= 77,
            BirthYear= "19BBY",
        },
        ...
    };

    public Hero[] GetAll()=>_items;
    public Hero Get(int id)=>_items.SingleOrDefault(h => h.Id == id);
}

Client-side data loading

First, we will focus on client-side routing, which is going to be the most straightforward part. We just need to be able to retrieve the data using AJAX. To simplify things, we will always load all the data for the current page using a single AJAX call.

Let’s add two additional actions to our controller.

public class HeroController : Controller
{
…
    [Route("data/")]
    public IActionResult IndexData() => Ok(_db.GetAll());

    [Route("/data/{id:int}")]
    public IActionResult DetailData(int id) => Ok(_db.Get(id));
}

Both actions correspond to actions that we created last time for SSR. Each has a "data/" prefix in its URL. This way, we have a convention for accessing page data based on the URL of the current page without any additional configuration. In a real-life application, we would merge data and non-data actions to a single action to prevent duplication. We could achieve it, for instance, by using URL rewrite, but that is outside of the scope of this article.

To keep the data loading on the client side in one place, we are going to introduce a simple higher-order component.

const page = (WrappedComponent) =>
  ({ staticContext }) => {
    const location = useLocation();

    if (!staticContext) {
      useEffect(() => {
        fetch(`data${location.pathname}`)
          .then(r => r.json())
          .then(setPageData);
      }, [location]);
    }

    const [pageData, setPageData] = useState(null);

    return (
      pageData && <WrappedComponent pageData={pageData}></WrappedComponent>
    );
  };

We are retrieving the data using fetch API. The useLocation hook provided by React router is helping us to construct the URL. Note, that we are ignoring query string because we are not using it in our examples.

As you can see, we are retrieving data only if the staticContext is not set, which means that we are rendering the application on the client-side. We will use another way for the server-side later.
We are fetching data in an effect hook with location dependency to update data each time the location changes due to client-side routing.
In production code, we would also add cancelation of old requests and error handling, but we will omit it here to keep the example simple.

Thanks to page component, we can now easily add data into the HeroList and HeroDetail components.

const HeroList = ({ pageData }) => (
  <div>
    <h2>List of heroes</h2>
    <div>
      <ul>
        {pageData.map(hero => (
          <li key={hero.id}>
            <Link to={`/${hero.id}`}>{hero.name}</Link>
          </li>
        ))}
      </ul>
    </div>
  </div>
);
export default page(HeroList);

const HeroDetail = ({ pageData }) => (
  <div>
    <h2>{pageData.name}</h2>
    <div>
      Height: {pageData.height}
    </div>
    <div>
      Mass: {pageData.mass}
    </div>
    <div>
      Year of birth: {pageData.birthYear}
    </div>
    <div>
      <Link to="/">Back to list</Link>
    </div>
  </div>
);
export default page(HeroDetail);

Server-side data loading and hydration

To add data loading on the server-side, we have to make small adjustments to SsrResult and RenderService classes.

public class SsrResult : IActionResult
{
    ...
    private readonly object _data;
    public SsrResult(string url, object data)
    {
        ...
        _data = data;
    }
    public async Task ExecuteResultAsync(ActionContext context)
    {
        ...
        var renderResult = await renderService.RenderAsync(_url, _data);
        ...
    }
}

public class RenderService : IRenderService
{
    ...
    public Task<string> RenderAsync(string url, object data) => 
        _nodeJSService.InvokeFromFileAsync<string>(_serverJsPath, 
            args: new object[] { url, data });
}

public class HeroController : Controller
{
    ...
    [Route("/")]
    public IActionResult Index() => new SsrResult("/", _db.GetAll());
    [Route("/{id:int}")]
    public IActionResult Detail(int id) => new SsrResult("/:id", _db.Get(id));
    ...
}

We are receiving data in the SsrResult constructor and passing them straight into server.js through RenderService and INodeJSService.

We can now use the data in server.js to render the application.

const server = (cb, url, data) => {
  const context = { data };
  const markup = renderToString(
    <StaticRouter context={context} location={url}>
      <App />
    </StaticRouter>
  );

    ...

  cb(null, `<!doctype html>
      <html lang="">
      <head>
          ...
          <script>window.__ROUTE_DATA__=${JSON.stringify(data)}</script>
          ...
      </head>
      ...
    </html>`);
}

We are passing received data into the context of the StaticRouter and thus making it available to our page component. By using an inline script, we are assuring that we can access the data also during hydration.

We are ready to take advantage of the data in our page higher-order component during SSR and hydration.

const page = (WrappedComponent) =>
  ({ staticContext }) => {
    const location = useLocation();

    let initData = null;
    if (staticContext) {
      initData = staticContext.data;
    } else if (window.__ROUTE_DATA__) {
      initData = window.__ROUTE_DATA__;
      delete window.__ROUTE_DATA_;
    }

    if (!staticContext) {
      useEffect(() => {
        if (!initData) {
          fetch(`data${location.pathname}`)
        ...

We are retrieving the data from the staticContext (during SSR) or the __ROUTE_DATA__ window field (during hydratation). You might have noticed that we are clearing the __ROUTE_DATA__ field after filling the initData variable. This way, we ensure that the initial data is used only during hydration and not for another page during client routing.

Let’s check the browser. When we open the https://localhost:5000/4 URL, we can see that the initial request contains fully rendered HTML with all the data.

When we navigate to the list using the “Back to list” link, we can see that only an AJAX call was executed.

Code splitting

We have fully functional SSR now. It's time to add a cool feature that is currently not supported by ReactJS.NET - code splitting. Code splitting enables us to split our scripts into multiple chunks and lazy load them only when necessary. That means faster load times for our users.

We are going to use the Loadable Components library that, unlike React.lazy, also supports SSR. Thankfully, Razzle has a nice example for Loadable Components, so our work will be rather easy.

First, we need to install a few dependencies.

npm i @loadable/component @loadable/server -d
npm i @loadable/babel-plugin @loadable/webpack-plugin -D

Now we can update razzle.config.js to include the installed Loadable Webpack plugin using the following code.

if (target === "web") {
    const filename = path.resolve(__dirname, "build");
    config.plugins.push(
        new LoadableWebpackPlugin({
            outputAsset: false,
            writeToDisk: { filename },
        })
    );
}

Loadable Components also requires a Babel plugin (@loadable/babel-plugin) for SSR to function properly. Razzle supports modifications to Babel config through a ".babelrc" file in the folder where the razzle.config.js is. Razzle will then automatically pick it up during its initialization.

{
    "presets": [
        "razzle/babel"
    ],
    "plugins": [
        "@loadable/babel-plugin"
    ]
}

We are using the razzle/babel preset, which will give us all the defaults provided by Razzle, so we don’t have to configure them manually.

Next, we need add chunk extractor from Loadable Components into server.js file.

const server = (cb, url, data) => {
  const context = { data };
  const extractor = new ChunkExtractor({
    statsFile: path.resolve(__dirname, 'loadable-stats.json'),
    entrypoints: ['client'],
  });
  const markup = renderToString(
    <StaticRouter context={context} location={url}>
      <ChunkExtractorManager extractor={extractor}>
        <App />
      </ChunkExtractorManager>
    </StaticRouter>
  );

  const scriptTags = extractor.getScriptTags();
  const linkTags = extractor.getLinkTags();
  const styleTags = extractor.getStyleTags();

  cb(null, `<!doctype html>
      <html lang="">
      <head>
          <meta http-equiv="X-UA-Compatible" content="IE=edge" />
          <meta charset="utf-8" />
          <title>Welcome to Razzle</title>
          <meta name="viewport" content="width=device-width, initial-scale=1">
          <script>window.__ROUTE_DATA__=${JSON.stringify(data)}</script>
          ${linkTags}
          ${styleTags}
      </head>
      <body>
          <div id="root">${markup}</div>
          ${scriptTags}
      </body>
    </html>`);
}

Notice that we have also replaced assets in the HTML template with the ones that come from the chunk extractor.

We want to lazy load both our pages, so we need to wrap their imports in the App.js file with the loadable function provided by Loadable Components.

const HeroList = loadable(() => import('./HeroList'))
const HeroDetail = loadable(() => import('./HeroDetail'))

To wait for all asynchronously loaded scripts required to render the application, we must also wrap the hydrate call in client.js with the loadableReady function.

loadableReady().then(() => {
  hydrate(
    ...
  );
});

With that, we finished the integration of the code splitting into our application. Notice that we did not have to do anything special just because we are using ASP.NET Core as our backend, which is awesome.

Publishing the application

In the previous part of the article, we have bootstrapped our application using the standard React template provided by ASP.NET Core. Thanks to this, the publish profile was created for us, and we don’t need to change a single line in it. If we open the csproj file, we can see that the PublishRunWebpack target runs

npm install

and then

npm run build

The build npm script was created in package.json automatically by create-razzle-app when we bootstrapped the client side of our application.

The only thing that we need to do is a small modification of the Webpack configuration. Razzle is using webpack-node-externals to exclude all node_module packages from the server bundle. It makes sense for a NodeJS backend, but in our case, it would just make things harder during deploy. We would need to copy package.json, package-lock.json, and install packages on the destination server. It is much easier for us to let Webpack bundle all the dependencies into the resulting package – we are not using any dependency that couldn’t be bundled like this.

Let’s do a final modification to razzle.config.js.

if (dev) {
    ...
} else {
    if (target === 'node') {
        config.externals = [];
    }
}

You can read more about Webpack externals in the official documentation of Webpack.

And we are done. Execute the publish using the following command.

dotnet publish

The result is a fully functional package of our application.

Conclusion

This concludes our SSR React + ASP.NET Core backend integration. What I personally really like about this way is that we are free to use any React library that requires special handling for the SSR to function. We can use cool stuff like code splitting and most likely anything that will Webpack provide in the future because we have nicely decoupled the ASP.NET Core backend and the Webpack/React part.

You can access the complete code of the example application here https://github.com/pruttned/AspNetCoreReactRazzleExample .

Integrating React into ASP.NET Core using Razzle with all the goodies like SSR, routing, code splitting, and HMR – Part 1/2

Peter Ruttkay-Nedecký — Thu, 25 Jun 2020 20:20:40 +0000

To create a React application with server-side rendering (SSR) in .NET, you can usually use a well-known library, ReactJS.NET. Sadly, you might miss some useful features like code splitting or HMR (Hot Module Replacement). You are also constrained in which React libraries with SSR support you can use. E.g., in time of writing this article, there is no support for Typography.js.

What I’m going to present you in this two-part article, is an alternative way, where you have full control over the React side, while still keeping things simple thanks to Razzle and Javascript.NodeJS. We will also fully embrace React and replace Razor views in the ASP.NET Core application with it.

In the first part, we will take a quick look into Razzle. We will create a base Razzle application and integrate it with ASP.NET Core backend.
In the next part, we will add data loading to both the client and the server-side. We will explore possibilities of extending our application with cool stuff like code splitting and wrap up with publishing our application using Visual Studio.

Why server-side rendering ?

By using server-side rendering, we can deliver HTML content with a single request. It has two main benefits. User doesn’t need to wait while all javascript assets are loaded and initialized to see the actual content, and we make it much easier for SEO bots to crawl our page.

Why Razzle

A common way to create a React application is to use create-react-app (CRA). It allows us to focus on writing applications instead of wasting time with configuration. But sadly, it doesn’t come with server-side rendering, and without ejecting, we don’t have enough access to underlying Webpack configuration. Thankfully, there is a great alternative called Razzle. Similarly to CRA, it works with zero configuration. But in contrast to CRA, Razzle gives us the power to tap into the build process and thus modify the Webpack configuration. On top of that, it comes with server-side rendering right out of the box, while still keeping server-side code in our power.

Create Razzle App

We are going to use the standard React template provided by ASP.NET Core. Although it uses CRA, it still offers useful things, like base SpaServices configuration or publish configuration. Let’s run the following command to create the base of our application.

dotnet new react

Now we can focus on transforming CRA to Razzle. First, we will remove the whole ClientApp directory that contains the actual CRA client application and replace it with Razzle. Run the following command to initialize Razzle application inside of the ClientApp folder.

npx create-razzle-app ClientApp

We need to change the root path of SPA static files in the Startup.cs file. We will use a different path for development and production according to paths generated by Razzle.

services.AddSpaStaticFiles(configuration =>
{
    configuration.RootPath = _env.IsDevelopment() ? 
     "ClientApp/build" : "ClientApp/build/public";
});

Next, we must remove the UseSpa code from the Configure method, because we are not going to use ASP.NET Core spa proxy (more on that later).

app.UseSpa(spa =>
{
  …
});

Razzle architecture

Razzle application has two entry points for two different Webpack bundles:

index.js – the entry point for the server-side bundle
client.js - the entry point for the client-side bundle

In the server.js file, we can see the actual server-side rendering inside of an Express server.

const server = express();
server
  .get('/*', (req, res) => {
    const context = {};
    const markup = renderToString(
      <StaticRouter context={context} location={req.url}>
        <App />
      </StaticRouter>
    );
      res.status(200).send(
        `<!doctype html>
        ...
        ${markup}
        ...
        </html>`
      );
    ...
  });

The App component is the root of our React application (same as with CRA). This component is converted to an HTML string using renderToString provided by React and then sent to a client inside of an HTML template.

Let’s start the Razzle app to see how it works (for now without the ASP.NET Core part). Execute the following command in the ClientApp folder.

npm start

Razzle application is running on port 3000 (even if the output is saying that it is 3001 – more on that later), so let’s open http://localhost:3000/ in our browser. We are greeted by the default Razzle app. By opening developer tools and checking the network tab, we can see the SSR in action.

The document returned from the server contains fully rendered HTML. If we make any change to the Home.js component, it gets updated in the browser automatically. So HMR is also working.

Now, what’s up with the http://localhost:3001? Razzle uses a smart trick to achieve client-side HMR and blazing fast recompiles during development. It runs a separate webpack-dev-server for client-side assets. To connect these assets with the server-side, Razzle explicitly sets “webpack.output.publicPath” to localhost:3001. Therefore if we examine the rendered HTML from http://localhost:3000, we can see that scripts (alongside other assets) are actually hosted on http://localhost:3001.

This trick is going to be quite helpful. Webpack will serve client-side assets for us independently of our ASP.NET Core side, and therefore, we will have HMR without any hassle. We won’t even need to use a proxy between ASP.NET Core and Webpack dev server in contrast to a CRA solution.

Extending Razzle app

Let’s extend our application a little bit to be able to experiment with routing. For that, we will replace the Home.js component with two other components - HeroList.js and HeroDetail.js. For now, these will be just dummy components and we will extend them later.

const HeroList = () => (
  <div>
    <h2>List of heroes</h2>
    <div>
      TODO: list 
          <ul>
        <li>
          <Link to="/1">Hero 1</Link>
        </li>
      </ul>
    </div>
  </div>
);
const HeroDetail = () => (
  <div>
    <h2>Hero ...</h2>
    <div>
      <Link to="/">Back to list</Link>
    </div>
    <div>
      TODO: Hero detail
        </div>
  </div>
);

We also need to create routes for our new components in App.js.

<Switch>
  <Route exact path="/" component={HeroList} />
  <Route exact path="/:id" component={HeroDetail} />
</Switch>

Let’s check the http://localhost:3000/. We can see the HeroList component rendered, and clicking on the “Hero 1” link brings us to HeroDetail, as you would expect.

ASP.NET Core integration

We have successfully created a base of our application, but the client-side (React/Razzle) and the server-side (ASP.NET Core) are still disconnected. It’s time to integrate both sides by introducing React server-side rendering into the ASP.NET Core backend.

Rendering to string

We need to execute React rendering from the ASP.NET Core backend, but Razzle provides server-side rendering through an Express server by default. An easy way to achieve this integration is to use Javascript.NodeJS library that enables us to call NodeJs scripts from .NET Core. We only need to replace the Express server with a simple function that just returns the rendered HTML.

const server = (cb, url) => {
  const context = {};
  const markup = renderToString(
    <StaticRouter context={context} location={url}>
      <App />
    </StaticRouter>
  );
  
  cb(null, `<!doctype html>
      <html lang="">
      <head>
          <meta http-equiv="X-UA-Compatible" content="IE=edge" />
          <meta charset="utf-8" />
          <title>Welcome to Razzle</title>
          <meta name="viewport" content="width=device-width, initial-scale=1">
          ${assets.client.css
      ? `<link rel="stylesheet" href="${assets.client.css}">`
      : ''} 
          ${process.env.NODE_ENV === 'production'
      ? `<script src="${assets.client.js}" defer></script>`
      : `<script src="${assets.client.js}" defer crossorigin></script>`}
      </head>
      <body>
          <div id="root">${markup}</div>
      </body>
    </html>`);
}

The first parameter (cb) is a callback function to signal async completion to Javascript.NodeJS.

The entry point for the server-side is index.js, so we need to modify that one too.

let app = require('./server').default;
module.exports = app;

We have removed all of the original code, and now we are just exporting the server.

We can now try out the server function. First, we need to build the Razzle application with the following command.

npm run build

This command generates a production bundle of our application in the “./build/server” directory. It includes the server.js file that contains the whole application built by Webpack.

To call the server function, just run the following command.

node -p "require('./build/server')((_,x)=>console.log(x), '/')"

Great, we can see the rendered HTML of our React application. We are making some progress, but we surely don’t want to call build each time we change something in our React part during development. Luckily, Razzle does this automatically for us when we start the application in development mode. There is just one little catch. Razzle creates a bundle that also contains server-side HMR, and it will get in our way. If we try to run our rendering function from the server.js generated in the development mode, it will keep running and waiting for changes. Therefore, we need to remove it. Thankfully, we can keep the client-side HMR, which is more significant for us. To remove the server-side HMR, we need to tap into the underlying Webpack configuration by creating a “razzle.config.js” file with a “modify” function.

module.exports = {
    modify: (config, { target, dev }) => {
        if (target === 'node' && dev) {
            config.plugins = config.plugins.filter(p => 
                p.constructor.name !== 'HotModuleReplacementPlugin');
            config.entry = config.entry.filter(e => e !== 'webpack/hot/poll?300');
        }
        return config;
    },
};

Now, after we have disabled the server-side HMR, we are ready to start the development server again with the following command.

npm start

We can also try to rerun the server function.

node -p "require('./build/server')((_,x)=>console.log(x), '/')"

It prints HTML, same as in case of production bundle, but now, we can make changes to our React components, and by rerunning the command, we can see that the bundle was updated.

Calling NodeJs from ASP.NET Core

Finally, we are ready to integrate our React client-side with the ASP.NET Core backend. For now, we will do it without any data coming from the ASP.NET Core backend.

We are going to use the already mentioned Javascript.NodeJS library to call server.js from ASP.NET Core backend. So, let’s add it to our project through NuGet.

dotnet add package Jering.Javascript.NodeJS --version 5.4.0

We need to register it in the ConfigureServices method inside of the Startup.cs file.

services.AddNodeJS();
services.Configure<OutOfProcessNodeJSServiceOptions>(options =>
{
    options.EnableFileWatching = _env.IsDevelopment();
});

Well, that was rather simple, and we are ready to go. Let’s create a service for calling our function exported from server.js.

public class RenderService : IRenderService
{
    private const string ServerJsRelPath = "./ClientApp/build/server.js";
    private readonly INodeJSService _nodeJSService;
    private readonly string _serverJsPath;

    public RenderService(INodeJSService nodeJSService, 
      IWebHostEnvironment webHostEnvironment)
    {
        _nodeJSService = nodeJSService;
        _serverJsPath = webHostEnvironment.ContentRootFileProvider
                .GetFileInfo(ServerJsRelPath).PhysicalPath;
    }
    public Task<string> RenderAsync(string url) => 
        _nodeJSService.InvokeFromFileAsync<string>(_serverJsPath, 
                args: new object[] { url });
}

We are using ContentRootFileProvider.GetFileInfo to get the path to the server.js file. In RenderAsync, we are calling Javascript.NodeJS (InvokeFromFileAsync) to invoke the exported function in the server.js file. InvokeFromFileAsync returns whatever we send to the callback (cb) in the js function. In our case, it is HTML of the specified route provided as the “url” parameter.

We can now create a custom action result for executing React rendering from MVC actions.

public class SsrResult : IActionResult
{
    private readonly string _url;
    public SsrResult(string url)
    {
        _url = url;
    }
    public async Task ExecuteResultAsync(ActionContext context)
    {
        var renderService = context.HttpContext.RequestServices
            .GetRequiredService<IRenderService>();
        var renderResult = await renderService.RenderAsync(_url);
        var contentResult = new ContentResult
        {
            Content = renderResult,
            ContentType = "text/html"
        };
        await contentResult.ExecuteResultAsync(context);
    }
}

Here we are calling our render service and converting its result into a ContentResult.

We have everything ready to create the actual hero controller.

public class HeroController : Controller
{
    [Route("/")]
    public IActionResult Index() => new SsrResult("/");
    
    [Route("/{id:int}")]
    public IActionResult Detail(int id) => new SsrResult("/:id");
}

For now, it is rather simple. We have defined two routes that just return the SSR action result.

We have everything ready to try out our application. Open https://localhost:5001/, which is the default development URL of ASP.NET Core applications. In the network tab of developer tools, we can see that we are receiving a complete HTML from the backend, so the SSR is working.

SSL

We have only one small problem left. In development, Razzle serves scripts through HTTP, while ASP.NET Core uses SSL by default. This inconsistency is going to be a problem for us because all modern browsers today disallow mixed content for security reasons. To force Razzle to use SSL, we need to tap into the Webpack configuration again using the “modify” function in razzle.config.js.

if (dev) {
    config.output.publicPath = config.output.publicPath
       .replace('http://', 'https://');
    if (target === 'web') {
        config.devServer.https = true;
        config.devServer.pfx = process.env.RAZZLE_PFX;
        config.devServer.pfxPassphrase = process.env.RAZZLE_PFX_PASSPHRASE;
    }
}

First, we are setting publicPath, because HTTP is hardcoded in the Razzle configuration (see https://github.com/jaredpalmer/razzle/blob/2535fbcfc8d8f99d9d0ada4eaed3e994335f2af4/packages/razzle/config/createConfig.js#L90). Then we are telling Webpack dev server to serve content through HTTPS and which certificate to use. For more info about SSL in the Webpack dev server, please see https://webpack.js.org/configuration/dev-server/#devserverhttps.

For storing the certificate configuration, we can use “.env” file. Razzle is then going to provide all keys with the “RAZZLE_” prefix to “process.env” automatically.

Alternatively, we could also just switch the ASP.NET Core backend to HTTP by removing https://localhost:5001 URL from the launchSettings.json file.

Conclusion

We are going to wrap it here for now. We have successfully integrated Razzle with ASP.NET Core backend into a working application with server-side rendering while having HMR during development.

Next time, we are going to add data loading, look into code splitting, and also tackle the problem of deploying our application.

Preventing unnecessary allocation in .NET collections

Peter Ruttkay-Nedecký — Sun, 09 Feb 2020 14:17:16 +0000

.NET is doing a really good job in the case of memory management, and usually, you don’t have to think about it at all. But if you are creating an application where performance really matters, for instance, a game in Unity, then it is important to have allocations in check. Sometimes you will be surprised how a rather innocently looking code could do a lot of allocations and mess with the performance. Let’s look into a few such cases.

Heap vs. Stack

Before we jump straight to examples, let’s first take a quick detour to grasp some basics of memory management in .NET.

In .NET, we have two kinds of types - Reference types (class) and value types (e.g., struct).

Reference types are allocated on the heap, while value types (when used for parameters, local variables, or return values) are allocated on the stack.

You can perceive stack as a LIFO (last in first out) structure of blocks (stack frames) containing the method being executed, its parameters, and local variables. Each time a method execution finishes, its frame is removed from the top of the stack. This removal is done outside of .NET garbage collection.

Heap is a memory place where you can allocate much bigger memory blocks and keep them alive as long as you need them. Garbage collection is responsible for freeing memory allocated on the heap. Disadvantage is that the garbage collection is slower then removing objects from the stack.

When we are writing a performance-critical code, we should prevent unnecessary heap allocations.

IEnumerable and heap allocations

Suppose that we are creating a custom collection, and we would like to be able to enumerate over it. For our example, we are going to create a simple wrapper over List<float> whose purpose is to prevent negative numbers in the collection.

public class NonNegativeCollection : IEnumerable<float>
{
    private List<float> mItems = new List<float>(10);
    public void Add(float value) => mItems.Add(Math.Abs(value));
    IEnumerator IEnumerable.GetEnumerator() => mItems.GetEnumerator();
    public IEnumerator<float> GetEnumerator() => mItems.GetEnumerator();
}

We have an internal list of items, Add method that is doing all the hard work and methods for implementing IEnumerable<float>.

Now let’s put our new and shiny collection to good use.

var collection = new NonNegativeCollection
{
    -1.1f, 1.1f,
};
// game loop
for (int i = 0; i < 1000; i++)
{
    var sum = 0f;
    foreach (var item in collection)
    {
        sum += item;
    }
}

We are creating an instance of our collection filled with numbers, and then we are computing the sum of them (yes without Linq ). We are doing this multiple times to simulate a game loop or a complex computation. How many allocations would you expect to be happening in this simple code? There is, of course, the obvious one, the allocation of the NonNegativeCollection class. But it happens outside of the game loop. Let’s open a memory profiler to check if there are any other allocations.

As you can see, apart from expected NonNegativeCollection allocation, we have 1000 allocations from GetEnumerator method. One for each iteration in the game loop.

We must be doing something wrong.

We are returning an enumerator from our inner mItems List in the GetEnumerator method. If we check the type of this enumerator, we can see that it is a structure. So why do we see allocations on the heap? The reason for this is boxing. It happens each time we convert a value type to System.Object, or to an interface. In our GetEnumerator implementation, we are returning the enumerator as an IEnumerator<float>. Because of this, the enumerator instance is moved onto the heap.

Let’s fix it

The fix is rather simple. According to documentation on foreach loop, we don’t have to implement IEnumerable if we wish to support foreach loop. All we need is to have a public parameterless GetEnumerator method that returns an enumerator. Let’s change GetEnumerator to implicit interface implementation and provide another GetEnumerator method that returns List<float>.Enumerator. Our resulting code looks like this.

public class NonNegativeCollection : IEnumerable<float>
{
    //...
    public List<float>.Enumerator GetEnumerator() => mItems.GetEnumerator();
    IEnumerator<float> IEnumerable<float>.GetEnumerator() => 
      mItems.GetEnumerator();
}

In reality, we could even remove the whole IEnumerable<float> implementation and still be able to use foreach loop on our collection class, but I prefer to keep it there to make it explicit that the class supports enumeration.

If we run the memory profiler again, we would see that the extra heap allocations are gone (there is no boxing anymore), and we have gained some precious performance.

Casting collection to IEnumerable

Let’s say that we want to make our sum computation reusable and to support also other collection types. Therefore we will move the sum computation to a separate method, and we will define its parameter as IEnumerable<float>.

private static float Sum(IEnumerable<float> collection)
{
    var sum = 0f;
    foreach (var item in collection)
    {
        sum += item;
    }
    return sum;
}

After running the memory profiler again, we see that we are again doing heap allocations. It is happening because we are casting the input collection to IEnumerable<float>. Therefore we are again using IEnumerable<float>.GetEnumerator() method and thus doing boxing.

How to solve this case? Sadly, there is no silver bullet for this problem, at least none, that I’m aware of.

We could partially solve it in multiple ways:

We could create a separate method for each collection type, which would cause a lot of duplicate code.
We could replace IEnumerable<float> parameter with, e.g., IReadOnlyList<float> (with Count and indexer) and use for-loop instead of foreach. But this is limiting the usage of our sum method.
We could create a new IEnumerable interface that would return an Enumerator structure via generic type.

public struct NonGcEnumerator<TEnumerator, TValue> : IEnumerator<TValue>
    where TEnumerator : IEnumerator<TValue>
{
    private TEnumerator mEnumerator;
    public TValue Current => mEnumerator.Current;
    object IEnumerator.Current => mEnumerator.Current;
    public NonGcEnumerator(TEnumerator enumerator)
    {
        mEnumerator = enumerator;
    }
    public bool MoveNext() => mEnumerator.MoveNext();
    public void Dispose() { }
    public void Reset()
    {
        mEnumerator.Reset();
    }
}
public interface INonGcEnumerable<TEnumerator, TValue>
    where TEnumerator : IEnumerator<TValue>
{
    NonGcEnumerator<TEnumerator, TValue> GetEnumerator();
}
public class NonNegativeCollection : 
INonGcEnumerable<List<float>.Enumerator, float>
{
    private List<float> mItems = new List<float>(10);
    public void Add(float value) => mItems.Add(Math.Abs(value));
    public NonGcEnumerator<List<float>.Enumerator, float> GetEnumerator() =>
        new NonGcEnumerator<List<float>.Enumerator,float>
    (mItems.GetEnumerator());
}

private static float Sum(INonGcEnumerable<List<float>.Enumerator, 
float> collection)
{
    var sum = 0f;
    foreach (var item in collection)
    {
        sum += item;
    }
    return sum;
}

This way, we can define GetEnumerator with a non-interface return type and still be able to do it a general way.

Linq

Up to this point, we were ignoring the fact that in .NET, we have a built-in method for computing the sum of items in a collection. For that, we use the Sum extension method provided by System.Linq namespace.

The problem with System.Linq, in our case is, that all the methods are provided as an extension method on IEnumerable<T> interface. E.g., the Sum method looks like this.

namespace System.Linq
{
    public static class Enumerable
    {
        ...
        public static float Sum(this IEnumerable<float> source);
        ...
    }
    ...
}

Therefore if we used Sum provided by Linq, we would be back on doing allocation on the heap due to boxing. In general, if you are creating a game or something similar, you should sadly forget about the entire Linq in performance-critical parts.

Conclusion

We have looked in a few cases, where .NET is doing unwanted heap allocations that are rather easy to miss. In most cases, you don’t have to care, because the garbage collector is rarely a performance bottleneck. But if you are creating a game, then you should pay attention to boxing and always check a memory profiler if you are not sure.

Turning Git into an application database

Peter Ruttkay-Nedecký — Sat, 28 Dec 2019 15:47:45 +0000

Today when you are creating an application, you have many options for a database to choose from.

What you might not realize is that you could consider Git as one of the options with some great advantages. Why and how would you do that? Well, read ahead.

Recently I was creating a simple invoicing application with my brother as a personal project for our needs. From the very beginning, we intended to run our application locally in a single-user mode, but we required to be able to access data from multiple places.

In the past, we discussed the possibilities of using file system along with Git to create a simple, yet powerful database, so we decided to use this opportunity to make a proof of concept of this idea.

Why Git?

Let’s look at the benefits of using Git backed filesystem as an application database.

History - Let's face it, managing history in common database is usually not fun. You'll end up with tables full of records where you must check dates just to get the currently active record. Or you must move records to historical tables. Either way, it is tedious. But with Git, you'll get history basically for free. You are even able to compare versions, and you also know who, when, and why did the change.
Hosting for free – Today, you can create a private repo in Bitbucket and Github without any costs. Compared to ordinary database free hostings, there are no row constraints, no read/write constraints, and I consider both Github and Bitbucket reliable providers.
Human readable data store - By storing data in plain text files, you can read and even edit data with your favorite text editor. Thanks to this, you can quickly hack some data, or rapidly prototype new features. E.g., you need to clone an old invoice, and you still had no time to implement this feature into your application? No problem, simply copy the file, open your favorite text editor, and you are done. It's much easier than in most database admin tools.
Distributed and disconnected database - Great thing about Git is its distributed nature. Each local repository contains the full history. If one of the repositories dies, it doesn't usually mean a loss of data. It is also possible to work in offline mode without any additional headaches.
Merging of conflicts - When two users change the same entity, there must be a conflict resolution. That often means that either the later change wins or the first one. Git, on the other hand, gives users the ability to merge conflicts manually and in many cases, can merge changes automatically.

Now, when we know what the benefits of a Git application database are, let us dive into implementation.

How to store documents

First, we need to decide how we are going to model our database. We will use folders for collections and files for documents.

This way, it is easy to collect all the entities stored in one collection, and it is also more readable for a human. We are going to use a non-bare repository, so we will be able to access files directly, which I consider to be a big advantage.

Let's point out here an interesting article https://www.kenneth-truyers.net/2016/10/13/git-nosql-database/ that is also dealing with the concept of using Git as a database, but it is using bare repositories.

To speed up lookups, we are going to store document ID directly in the file name. This way, it is not necessary to open a file to get the ID. And it is again more readable for a human. It would also be possible to store more fields in the file name and thus make a sort of index.

As a file format for document files, we will use YAML. YAML is great for serializing data while keeping them human-readable, so checking diffs in the Git history will be easier for us.

Let's have some performance

Ok, so now when we have our file structure, to have any sense of speed, we need caching. We are going to do caching on two levels. First, we will cache collection and then also individual documents.

Caching of individual documents is more interesting, so let us start with that. We should better not keep documents in memory forever, but rather they should have TTL (time to live) specified. For actual implementation, we will use node-cache library.

async function getDocument(collection: string, id: string): Promise<any> {
  const documentCacheKey = getDocumentCacheKey(collection, id);
  let document = documentCache.get(documentCacheKey);
  if (!document) {
    const file = getDocumentPath(collection, id);
    const fileContent = await readFile(file, 'utf-8');
    document = yaml.safeLoad(fileContent) || {};
    (document as any)['id'] = id;
    documentCache.set(documentCacheKey, document);
  }
  return document;
}
function getDocumentCacheKey(collection: string, id: string) {
  return `${collection}-${id}`;
}
function getDocumentPath(collection: string, id: string) {
  return path.join(baseDir, collection, `${id}.yaml`);
}

First, we retrieve the cache key for a given document. For this, we concatenate the collection (folder) name and ID of the required document. We then use this key to access the document in the cache. If it is not there, we need to load it from the file system and then store it in the cache under our cache key.

For parsing YAML files, we will use js-yaml library.

For caching collections, a simple dictionary of arrays is enough. We don’t need TTL in this case, because we will query collections more often than actual documents and actual content in memory is rather small.

async function getCollection(collection: string): Promise<string[]> {
  if (!collectionCache[collection]) {
    collectionCache[collection] =
      (await glob(path.join(baseDir, collection, '*.${yaml}')))
        .map(p => ({
          id: path.basename(p, path.extname(p)),
          path: p
        }));
  }
  return collectionCache[collection]!
    .map(f => f.id);
}

Text editor to the rescue

Because we have all documents nicely accessible from the file system, it would be a shame not to support changes outside of the application. This way, it is possible to handle usability patterns, that we are not directly supporting in the application. E.g., as we said previously, we will be able to copy invoices before adding this feature into the application.

For this, we are going to introduce file watcher using chokidar library.

watcher = chokidar.watch(`**/*.yaml`, { cwd: baseDir });
watcher
  .on('add', file => onChange(file, 'add'))
  .on('unlink', file => onChange(file, 'unlink'))
  .on('change', file => onChange(file, 'change'));

function onChange(file: string, changeType: 'add' | 'unlink' | 'change') {
  const collection = path.basename(path.dirname(file));
  const id = path.basename(file, path.extname(file));
  invalidateDocumentInCache(collection, id, changeType !== 'change');
}
function invalidateDocumentInCache(collection: string, id: string,
  invalidateCollectionCache: boolean) {
  const documentCacheKey = getDocumentCacheKey(collection, id);
  documentCache.del(documentCacheKey);
  if (collectionCache && invalidateCollectionCache) {
    collectionCache[collection] = null;
  }
}

In case of detecting a change to a YAML file, we remove it from the node-cache. If the file is removed, added, or renamed (add + unlink), then we also clear the whole collection in the collection cache.

Let's search

Finally, we have everything prepared for doing an actual querying. We are going to do it on two levels. Query by ID and by content. We will use the micromatch library for querying by ID.

async function getAllIds(query?: IdQuery): Promise<string[]> {
  let ids = await getCollection(collectionName);
  if (query && query.id) {
    ids = ids.filter(d => micromatch.isMatch(d, query.id!));
  }
  return orderBy(ids, 'id');
}

Micromatch enables us to run simple queries based on ID. E.g., in the case of invoices, each ID consists of year and number in the specific year, so it looks like this "201933". To get all invoices created in a specific year, we can use "${year}*" query.

For querying by content of files, we are going to use simple filtering.

let docs = await Promise.all(
    collection.map(doc => doc.getDocument(collectionName, doc)));
if (query && query.where) {
  docs = docs.filter(query.where)
}

Save it & load it

To keep local and remote Git repositories in sync, we need to pull and push regularly. For a single page application, the best time to pull is during page refresh. It is possible to push after each change, so the remote repository is updated as soon as possible. For actual Git handling, we are going to use simple-git library, which makes git operations easy.

async function pull(): Promise<any> {
  await ensureRepo();
  await repo!.pull();
}
async function commitAndPush(message: string): Promise<any> {
  await ensureRepo();
  await repo!.add('.');
  var commitRes = await repo!.commit(message);
  var pushRes = await repo!.push();
}
async function ensureRepo() {
  if (!repo) {
    repo = simplegit(db.dir);
  }
}

And then in actual update code:

async function update(invoice: InvoiceUpdateModel): Promise<InvoiceDocument> {
  let invoiceDocument = await db.invoices.single(invoice.id);
  invoiceDocument = { ...invoiceDocument, ...invoice };
  invoiceDocument = await db.invoices.update(invoiceDocument);
  await repoService.commitAndPush(`invoice(${invoice.id}):updated`);
  return invoiceDocument;
}

Conclusion

I must say that I'm happy with the result. For this specific use case, git works quite nice as a database for application data. Of course, the performance is much lower compared to common database solutions. The whole concept would also be far from ideal in case of an application hosted on a remote server because it would make dealing with conflicts much more difficult. But on the other hand, it is really simple, cheap and it gives us the ability to do quick hacking of documents.

You can access the whole code of the invoicing application here https://github.com/pruttned/owl-invoice