DEV Community: Ingvar

Localization in AvaloniaUI

Ingvar — Mon, 02 Aug 2021 20:55:30 +0000

Hey there! This time I gonna explain how to add localization support in your AvaloniaUI app. As always I'm using my app called Camelot as good working example.

What is localization and why do we need it

Localization is a process of adding support for different locales in your app. Locales always include language, date/time formats etc. It allows users from multiple countries use your app in their native language.

How to localize AvaloniaUI app

Create resources

First step is to create set of resources files like here. It could be done automatically via IDE. Resources.resx is the main file with default culture translations inside, IDE also autogenerates Resources.Designer.cs class based on it. This class could be used in code if needed. Other *.resx files contain translations for different cultures etc. Please note that for same language it's possible to keep many locales, like en-US and en-GB.

Integrate resources into views

Resources could be used as static objects in views. Translation for current locale will be shown. Here is an example how to use them:



<ToolTip.Tip>
    <TextBlock Classes="mainWindowTextBlock" Text="{x:Static p:Resources.GoBack}" />
</ToolTip.Tip>

Add language selection logic

For usability it's good to provide ability to select language in UI. All you need on backend in this case is to set current culture:



Thread.CurrentThread.CurrentUICulture = CultureInfo.GetCultureInfo(languageCode);

See LanguageManager for more info

Add translations

IDE support adding resources over UI:

Tips: use clear translation string names. If text is too long don't try to put full text in resource name. If you have resource that ends with colon I recommend to name it ...WithColon because you might need similar resource w/o colon later and adding : in view to your text won't work for some locales as expected.

Conclusion

In this post I described process of localization AvaloniaUI app. It's pretty straight-forward process. Feel free to contact me in comments if you have any questions!

How to Automate UI testing in AvaloniaUI App

Ingvar — Sun, 28 Mar 2021 14:57:37 +0000

Hello, in this post I gonna discuss details of automated UI testing. As always I gonna use my app Camelot as working example.

Why do you need UI testing

Why do you need to automate your UI testing? Because it saves time if compare with manual testing. It's more accurate and helps to find regression bugs. The only disadvantage here is time spent on writing tests and initial setup.

What UI tests could do

UI tests could emulate everything user does. Move mouse, click, press keys etc. I recommend to check a single test case/scenario in a test, like you do during manual testing. Example from my app: open create directory dialog, enter directory name and create it, observe that it was created. This test could be automated easily. I recommend to start automating from smoke testing and finish with full regression testing.

UI testing in AvaloniaUI

AvaloniaUI has functionality that allows to run custom code few seconds after app setup. You can add your testing code there and run your scenarios. AvaloniaUI 0.10.0 also introduced Headless platform. Headless app doesn't have UI so it could be started even on non-gui OS. It could be useful for running tests on servers w/o GUI. I run UI tests on Github via github actions so for me this option is useful. I prefer to run real app and execute tests on it but you can also test your controls etc w/o running whole app.

Avalonia app inside uses static fields etc so it's not possible to create new instance of app per test. I had to create single instance and reuse it across tests, so parallel execution of tests should be disabled for UI tests project. Also every test should have proper cleanup otherwise it could break other tests.

Setup infrastructure for UI testing

I use Xunit for testing, it doesn't work with UI tests out of box. I had to add my own runner for Xunit:

private class Runner : XunitTestAssemblyRunner
{
    // constructor

    public override void Dispose()
    {
        AvaloniaApp.Stop(); // cleanups existing avalonia app instance

        base.Dispose();
    }

// this method is called only if test parallelization is enabled. I had to enable it and set max parallelization limit to 1 in order to avoid parallel tests execution
    protected override void SetupSyncContext(int maxParallelThreads)
    {
        var tcs = new TaskCompletionSource<SynchronizationContext>();
        var thread = new Thread(() =>
        {
            try
            {
                // DI registrations
                AvaloniaApp.RegisterDependencies();

                AvaloniaApp
                    .BuildAvaloniaApp()
                    .AfterSetup(_ =>
                    {
                        // sets sync context for tests. avalonia UI runs in it's own single thread, updates from other threads are not allowed
                        tcs.SetResult(SynchronizationContext.Current);
                    })
                    .StartWithClassicDesktopLifetime(new string[0]); // run app as usual
            }
            catch (Exception e)
            {
                tcs.SetException(e);
            }
        })
        {
            IsBackground = true
        };

        thread.Start();

        SynchronizationContext.SetSynchronizationContext(tcs.Task.Result);
    }
}

Full code
One more headless tests example

Also I added AvaloniaApp class that is actually wrapper for real app:

public static class AvaloniaApp
{
    // DI registrations
    public static void RegisterDependencies() =>
        Bootstrapper.Register(Locator.CurrentMutable, Locator.Current);

    // stop app and cleanup
    public static void Stop()
    {
        var app = GetApp();
        if (app is IDisposable disposable)
        {
            Dispatcher.UIThread.Post(disposable.Dispose);
        }

        Dispatcher.UIThread.Post(() => app.Shutdown());
    }

    public static MainWindow GetMainWindow() => (MainWindow) GetApp().MainWindow;

    public static IClassicDesktopStyleApplicationLifetime GetApp() =>
        (IClassicDesktopStyleApplicationLifetime) Application.Current.ApplicationLifetime;

    public static AppBuilder BuildAvaloniaApp() =>
        AppBuilder
            .Configure<App>()
            .UsePlatformDetect()
            .UseReactiveUI()
            .UseHeadless(); // note that I run app as headless one
}

UI testing

Now it's time to write test! Here is an example of test that opens about dialog via F1:

public class OpenAboutDialogFlow : IDisposable
{
    private AboutDialog _dialog;

    [Fact(DisplayName = "Open about dialog")]
    public async Task TestAboutDialog()
    {
        var app = AvaloniaApp.GetApp();
        var window = AvaloniaApp.GetMainWindow();
        // wait for initial setup
        await Task.Delay(100);

        Keyboard.PressKey(window, Key.Tab); // hack for focusing file panel, in headless tests it's not focused by default
        Keyboard.PressKey(window, Key.Down);
        Keyboard.PressKey(window, Key.F1); // press F1

        await Task.Delay(100); // UI is not updated immediately so I had to add delays everywhere

        _dialog = app
            .Windows
            .OfType<AboutDialog>()
            .SingleOrDefault();
        Assert.NotNull(_dialog);

        await Task.Delay(100);

        var githubButton = _dialog.GetVisualDescendants().OfType<Button>().SingleOrDefault();
        Assert.NotNull(githubButton);
        Assert.True(githubButton.IsDefault);
        Assert.True(githubButton.Command.CanExecute(null));
    }

    public void Dispose() => _dialog?.Close(); // cleanup: close dialog
}

Looks simple, right? I had to add some extra code for delays but in fact test is simple. Here is an example of delay service:

public static class WaitService
{
    public static async Task WaitForConditionAsync(Func<bool> condition, int delayMs = 50, int maxAttempts = 20)
    {
        for (var i = 0; i < maxAttempts; i++)
        {
            await Task.Delay(delayMs);

            if (condition())
            {
                break; // stop waiting for condition
            }
        }
    }
}

More tests examples
Official example

Conclusion

AvaloniaUI provides good infrastructure for running UI tests. I recommend to use them in your project if it's complicated enough because they save a lot of time. Are you using any UI tests in your project? Tell me in comments

How to Use AvaloniaUI Developer Tools

Ingvar — Mon, 01 Mar 2021 20:48:13 +0000

Hello, in this post I will briefly explain how to install and use developer tools in your AvaloniaUI project. As always, I will use my app Camelot as working example.

What is developer tools window?

Developer tools window is similar feature to Chrome developer console. It allows you to observe tree of elements, their styles and positions, including margins/paddings etc. Also it has ability to log UI events and even to show FPS!

How to add it

Add following lines of code inside your main window code behind class inside constructor:



#if DEBUG
    this.AttachDevTools();
#endif

Example of code

Usage

Open developer tools window using F12.

Watch visual tree and edit controls parameters:

Watch events:

Check FPS:

Highlight elements in tree:

Official documentation

Why Do You Need Management Port in Your Dotnet Web App

Ingvar — Thu, 04 Feb 2021 14:55:22 +0000

Hello, in this short post I gonna explain why enterprise level apps should use management port.

What is management port

By default app uses only one port and exposes all information to clients via it. But what if we want some data to be private for clients? Let's say we want health check to be available for k8s but not for consumers of our API. How to do that? The answer is separation of concerns. You should use separate ports in your app for different purposes. It's secure and simple way to handle this.

Management port is a private port for service endpoints. It is often used for:

1) Health checks
2) Metrics
3) Custom endpoints for testing

Management port is usually not exposed outside so
you can be sure that clients won't access your private metrics or health checks data.

Configuring

Asp.net core provides set of environment variables for configuring ports. ASPNETCORE_MANAGEMENTPORT is responsible for management port, so your typical config looks like this:

"ASPNETCORE_URLS": "http://*:5000/;http://*:5001/",
"ASPNETCORE_MANAGEMENTPORT": "5001"

This config uses ports 5000 and 5001 in Kestrel, 5001 is management port.

Conclusion

Management port is fast and convenient way to expose private service endpoints in your app. Do you have those in your app? Feel free to answer in comments

Dialogs in AvaloniaUI

Ingvar — Wed, 06 Jan 2021 21:59:35 +0000

Hello, in this post I gonna explain how to create dialogs infrastructure in your Avalonia app! As always I gonna use my app Camelot as an example of implementation.

Why do you need dialogs

Why do we need dialogs? Sometimes apps contain functionality that could be shown in separate window. Another option is to replace part of main window with this content, but sometimes it's not possible or doesn't look good from UI/UX side. In this case dialogs could help

How typical dialog looks like

Typical dialog looks like a small window shown in front of main window. In most implementations background of main window is blurred or semi-transparent that indicates that main window is not active at the moment. Here is an example from Camelot:

Also please note that dialog window is centered relatively to parent window. Also dialog is not always just a window, it could be a selector that returns some value to parent window (for example, create directory dialog returns new directory name).

Dialog view and view model requirements

Dialog view is not a simple user control because dialog itself is a window. Also it requires communication between view and view model because dialog can be closed from vm code. For this purpose I added Close request to base dialog view model:



public class DialogViewModelBase<TResult> : ViewModelBase
    where TResult : DialogResultBase
{
    public event EventHandler<DialogResultEventArgs<TResult>> CloseRequested;

    protected void Close() => Close(default);

    protected void Close(TResult result)
    {
        var args = new DialogResultEventArgs<TResult>(result);

        CloseRequested.Raise(this, args);
    }
}

Here TResult represents type of dialog result. I created empty DialogResultBase that should be inherited by specific result but you can omit this step. Also I created class for dialog without result:



public class DialogViewModelBase : DialogViewModelBase<DialogResultBase>
{

}

For views I created similar structure (code here). Dialog view centers itself and subscribes view model events.

What if we want to pass some parameter to dialog before opening? For example, if I want to create new directory in current one, I will pass current directory path to create directory dialog because it validates if directory name is correct and doesn't exist. For this purpose I added thing called navigation parameter. I added base class for such dialog too:



public abstract class ParameterizedDialogViewModelBase<TResult, TParameter> : DialogViewModelBase<TResult>
    where TResult : DialogResultBase
    where TParameter : NavigationParameterBase
{
    public abstract void Activate(TParameter parameter);
}

Activate is called when dialog is shown, TParameter is our parameter types that extends empty NavigationParameterBase class. Full VM code is here. Note that this view model doesn't need specific view to work, all navigation parameter logic is done on backend side.

Dialog service interface

Let's encapsulate dialogs opening logic in separate dialog service. Interface for it looks like this:



public interface IDialogService
{
    Task<TResult> ShowDialogAsync<TResult>(string viewModelName)
        where TResult : DialogResultBase;

    Task ShowDialogAsync(string viewModelName);

    Task ShowDialogAsync<TParameter>(string viewModelName, TParameter parameter)
        where TParameter : NavigationParameterBase;

    Task<TResult> ShowDialogAsync<TResult, TParameter>(string viewModelName, TParameter parameter)
        where TResult : DialogResultBase
        where TParameter : NavigationParameterBase;
}

Note that it supports result and navigation parameter as generic overloads and accepts view model name. View model name is mandatory. I use it to determine which dialog should be opened (different dialogs could have same results and navigation parameters so there is no other way to understand which one should be used).

I put IDialogService into view models project because view models open dialogs. But implementation itself is done in UI project because it depends on some Avalonia-specific things, view models shouldn't care about this.

Dialog service implementation

How to implement this interface? Let's dive deeper inside implementation. Full code is available here.

Typically I have 6 steps inside dialog service:
1) Create view
2) Create view model
3) Bind view and view model
4) Activate dialog with navigation parameter
5) Show dialog and wait for result (also here I show/hide overlay on main window)
6) Return dialog result if needed

Looks complicated? Let's go through it step by step:

Create view

I have naming convention that view model and view have almost similar names, like CreateDirectoryDialogViewModel and CreateDirectoryDialog. It allows me to find view by view model type name easily:



private static DialogWindowBase<TResult> CreateView<TResult>(string viewModelName)
    where TResult : DialogResultBase
{
    var viewType = GetViewType(viewModelName);
    if (viewType is null)
    {
        throw new InvalidOperationException($"View for {viewModelName} was not found!");
    }

    return (DialogWindowBase<TResult>) GetView(viewType);
}

private static Type GetViewType(string viewModelName)
{
    var viewsAssembly = Assembly.GetExecutingAssembly();
    var viewTypes = viewsAssembly.GetTypes();
    var viewName = viewModelName.Replace("ViewModel", string.Empty);

    return viewTypes.SingleOrDefault(t => t.Name == viewName);
}

private static object GetView(Type type) => Activator.CreateInstance(type);

Note that I use reflection for creating instance of view.

Create view model

Flow here is similar. Note that here I use Locator to create instance of view model. It's required because dialogs often have dependencies. You should register your view model with Splat to make this code work. See more about dependency injection in AvaloniaUI in my previous blog post.



private static DialogViewModelBase<TResult> CreateViewModel<TResult>(string viewModelName)
    where TResult : DialogResultBase
{
    var viewModelType = GetViewModelType(viewModelName);
    if (viewModelType is null)
    {
        throw new InvalidOperationException($"View model {viewModelName} was not found!");
    }

    return (DialogViewModelBase<TResult>) GetViewModel(viewModelType);
}

private static Type GetViewModelType(string viewModelName)
{
    var viewModelsAssembly = Assembly.GetAssembly(typeof(ViewModelBase));
    if (viewModelsAssembly is null)
    {
        throw new InvalidOperationException("Broken installation!");
    }

    var viewModelTypes = viewModelsAssembly.GetTypes();

    return viewModelTypes.SingleOrDefault(t => t.Name == viewModelName);
}

private static object GetViewModel(Type type) => Locator.Current.GetRequiredService(type);

Bind view and view model

Bind is super simple. I set DataContext property of dialog with my view model:



private static void Bind(IDataContextProvider window, object viewModel) => window.DataContext = viewModel;

Activate dialog

If view model supports activation, I call Activate method.




switch (viewModel)
{
    case ParameterizedDialogViewModelBase<TResult, TParameter> parameterizedDialogViewModelBase:
        parameterizedDialogViewModelBase.Activate(parameter);
        break;
    case ParameterizedDialogViewModelBaseAsync<TResult, TParameter> parameterizedDialogViewModelBaseAsync:
        await parameterizedDialogViewModelBaseAsync.ActivateAsync(parameter);
        break;
    default:
        throw new InvalidOperationException(
            $"{viewModel.GetType().FullName} doesn't support passing parameters!");
}

Show dialog

This part is a bit tricky. I have to get main window, show overlay on it, show dialog and hide overlay:



private async Task<TResult> ShowDialogAsync<TResult>(DialogWindowBase<TResult> window)
    where TResult : DialogResultBase
{
    var mainWindow = (MainWindow) _mainWindowProvider.GetMainWindow();
    window.Owner = mainWindow;

    mainWindow.ShowOverlay();
    var result = await window.ShowDialog<TResult>(mainWindow);
    mainWindow.HideOverlay();
    if (window is IDisposable disposable)
    {
        disposable.Dispose();
    }

    return result;
}

Overlay is a semi-transparent Grid. Show/hide methods just change ZIndex for it:



public void ShowOverlay()
{
    OverlayGrid.ZIndex = 1000;
}

public void HideOverlay()
{
    OverlayGrid.ZIndex = -1;
}

This Grid has following style:



<Style Selector="Grid#OverlayGrid">

    <Setter Property="ZIndex" Value="-1" />

    <Setter Property="Background" Value="{DynamicResource DialogOverlayBrush}" />

    <Setter Property="Opacity" Value="0.2" />

</Style>

Return result

Easy part. Avalonia supports dialogs results:



var result = await window.ShowDialog<TResult>(mainWindow);

// some code here

return result;

Conclusion

In this post I explained how to create and use dialogs in your app. Do you have dialogs in your app? Tell me in comments.

Monitoring Your Dotnet Service Using Prometheus

Ingvar — Sat, 26 Dec 2020 11:03:48 +0000

Hello everyone, in this post I gonna explain how to monitor your .Net Core service using Prometheus and Grafana.

Why do you need monitoring of your app

Let's start from identifying reason to add metrics to your app. Why do we need metrics? Metrics allow us to check performance of app, find hotspots in your code, analyze app data and collect statistics. How many users do you have? How fast is your app for them? How often does your app call database and how fast is it's response? How many of users do some action on your website? Metrics allow you to answer all this questions. Typically in your app you could use 2 kinds of metrics:
1) Application monitoring tool that could monitor app performance and microservices communication (examples: New Relic, AppDynamics)
2) Monitoring that allows to add metrics into your app, mostly business or performance metrics. This is what Prometheus designed for.

Metrics could be collected on app side and sent to some storage (often it's time-series database) using 2 models:
1) Push model. Every interval (for example 30s) app sends metrics to some endpoint in predefined format
2) Pull model. App exposes some endpoint with metrics and external tool collects them from time to time. Prometheus mostly uses this model. I prefer it over push because it's less complicated and app shouldn't care about sending metrics somewhere.

Prometheus

Prometheus is a monitoring system and time-series database. It uses pull model for collecting metrics.

Official site

Grafana

Grafana is a tool that allows you to visualize metrics. Is supports multiple target databases with metrics and provide a way to query them and show output as a chart.

Official site

Prometheus-net

Prometheus-net is a .NET library that supports Prometheus metrics collection and sharing them on specific endpoint.

Github

Metric server

For exposing metrics you should enable metrics server in following way:



public void Configure(IApplicationBuilder app)
{
    // some code

    app.UseMetricServer(5000, "/prometheus");  // starts exporter on port 5000 and endpoint /prometheus
}

Monitoring middleware

Out of box library provides middleware that collects metrics of http requests. It tracks count of currently executed requests and timing. It could be enabled in following way:



public void Configure(IApplicationBuilder app)
{
    // some code

    app.UseRouting();
    app.UseHttpMetrics();

    // some code
}

It's recommended to call UseHttpMetrics after UseRouting because Prometheus tracks controller and action names. It's good if you want to have detailed statistics per endpoint or controller but it skips all previous middlewares! Middleware shouldn't be slow but it's still a problem that metrics are not accurate.

Custom metrics

1) Counter
Counter is simple counter that only increments it's value:



var counter = new Counter("cache_misses_total", "Cache misses total count");

counter.Inc(); // +1
counter.Inc(100); // +100

Note that every metric require 2 mandatory fields - metric name (I recommend to use default Prometheus format like in example above) and short description.

In Grafana mostly you will visualize increase of your metric value during some period of time:



increase(cache_misses_total[5m])

Note that it's not necessary to hardcode 5m value, there are dashboard variables that could be used for that purpose: link

2) Gauge

Gauge is a counter with decrement operation support:



var gauge = new Gauge("cache_misses_balance", "Cache misses balance");

gauge.Inc(); // +1
gauge.Inc(100);  / +100

gauge.Dec(); // -1
gauge.Dec(20); // -20

gauge.Set(50); // set 50 as gauge value

3) Histogram

Histogram is used for tracking number of events splitted between buckets. It often used for tracking time of operations. Prometheus also supports calculation of quantiles based on histogram value but it's not 100% accurate because Prometheus doesn't track exact values of events inside buckets. If buckets are small it's fine but if not - linear approximation is used that could result in lost of metrics accuracy.

4) Summary

Summary is close to histogram but it's calculated on client side, so it gives more accurate values but uses more CPU. I don't recommend to use it because in most cases trend is more important that 100% accurate metrics values.

Tracking time

All mentioned metrics has support for tracking time of operations in convenient way:



using (histogram.NewTimer())
{
    ExecuteOperation();
}

No need for awkward StopWatch usages in your code!

Labels

What if we have single operation that could have 1 or more results? What if we have single operation that could have 1 or more possible options of execution? For example, single HTTP request could be processed by N controllers with M actions, how to process that case? This is what labels were designed for. Let's say we try to access the cache and we have 2 possible result: success (value was found) and fail (value is missing). We can track those cases separately using labels:



var counter = new Counter("cache_usages_count", "Cache usages count", new[] {"result"}); // result is label name

counter.WithLabels("Success").Inc(); // success is "result" label value
counter.WithLabels("Fail").Inc();

Monitoring application

In application I recommend to use wrapper around prometheus-net library. It could be useful for changing prometheus-net to AppMetrics or similar library in future. This is what I've used:



public interface IMetricsFactory
{
    ICounter CreateCounter(CounterConfiguration configuration);

    IGauge CreateGauge(GaugeConfiguration configuration);

    // etc...
}

Where metrics interfaces are similar to prometheus-net interfaces:



public interface ICounter
{
    void Increment(double value = 1, params string[] labels);
}

Also all interfaces inherit interface with timer support:



public interface IMetricWithTimer
{
    ITimer CreateTimer(params string[] labels);
}

public interface ICounter : IMetricWithTimer
{
    // etc...
}

I don't like static way of using metrics libraries because in this case all of your nugets/projects depend on exact implementation so I use more canonical OOP way and moved all abstractions into one nuget and implementations into second one. So now all child projects depends on interfaces only while root project has package with implementations installed.

Inside package with implementations I used wrappers around default Prometheus metrics:



public class Counter : ICounter
{
    private readonly PrometheusCounter _counter;

    public Counter(PrometheusCounter counter)
    {
        _counter = counter;
    }

    public IDisposable CreateTimer() => _counter.NewTimer();

    public void Increment(double value = 1, params string[] labels) => _counter.WithLabels(labels).Inc(value);

    // etc...
}

Instance of prometheus counter is injected by metrics factory:



using PrometheusCounter = Prometheus.Counter;

public class MetricsFactory : IMetricsFactory
{
    public ICounter CreateCounter(CounterConfiguration configuration)
    {
        var prometheusCounter = new PrometheusCounter(configuration.MetricName, configuration.MetricDescription);

        return new Counter(prometheusCounter);
    }

    // etc...
}

where CounterConfiguration is simple:



public class CounterConfiguration
{
    public string MetricName { get; }

    public string MetricDescription { get; }

    public CounterConfiguration(string metricName, string metricDescription)
    {
        MetricName = metricName;
        MetricDescription = metricDescription;
    }
}

Enable monitoring in Prometheus

Now you should start to monitor your app from Prometheus itself. You should modify your default prometheus.yml file to achieve that:



scrape_configs:
  - job_name: prometheus
    static_configs:
      - targets: ['localhost:5000']

Doc

Grafana dashboards

Great, we added metrics to our app. How to add cool charts like in hacker films to our Grafana? You should setup data source in your Grafana first: link. After that you can create dashboard add charts there. You can use
this one as a base.

Prometheus queries

Enhanced dotnet metrics

If you need more dotnet metrics please check this library. Please note that adding dotnet metrics can affect performance!

Conclusion

In this article I showed an example of basic usage prometheus-net library for monitoring your dotnet app. What are you using for monitoring your app? Tell me in comments

AvaloniaUI: Dependency Injection

Ingvar — Wed, 30 Sep 2020 15:57:14 +0000

Hello, in this post I gonna explain how to use dependency injection in AvaloniaUI application and why is it so important. As always I will use my app Camelot as working example of used approach.

What is dependency injection (DI) and why is it needed

Consider following example:

private readonly TerminalService _terminalService;
private readonly DirectoryService _directoryService;

public TopOperationsViewModel()
{
    _terminalService = new TerminalService();
    _directoryService = new DirectoryService();
}

What's wrong with this code? It's a constructor of typical view model that depends on few services with business logic inside. In this snippet I created them in a constructor explicitly. That's bad approach because of following reasons:
1) What if we want to use another implementation of TerminalService for example that has same API but another implementation (lets call it AnotherTerminalService)? We will have to loop over all our files and change usages of TerminalService to AnotherTerminalService! A lot of work
2) What if we have to use services that depends on other services and so on? In this case in each view model we will have something like this mess:

public TopOperationsViewModel()
{
    _terminalService = new TerminalService(new PathService(), new DriveService());
    _directoryService = new DirectoryService();
}

And if we have some runtime dependencies it's even worse:

public TopOperationsViewModel()
{
    _terminalService = new TerminalService(new PathService(), RuntimeInformation.IsOSPlatform(OSPlatform.Linux) ? new LinuxSpecificService() : new WindowsSpecificService());
    _directoryService = new DirectoryService();
}

Absolutely unreadable code 😃

3) What if used service has some methods/properties that shouldn't be available in our view model? We can do something like this:

private readonly ITerminalService _terminalService; // note interface here

public TopOperationsViewModel()
{
    _terminalService = new TerminalService(); // saving by interface
    _directoryService = new DirectoryService();
}

In this case nobody could prevent programmer from modifying this code to use implementation directly based on knowledge what type is used.

Dependency injection provides flexible approach that solves all mentioned problems. In this approach you INJECT your dependencies into constructor of your type. Usually additional framework is used to help with this. Your code starts to look like this:

private readonly ITerminalService _terminalService;
private readonly IDirectoryService _directoryService;

public TopOperationsViewModel(
    ITerminalService terminalService,
    IDirectoryService directoryService) // note that interface is used
{
    _terminalService = terminalService;
    _directoryService = directoryService;
}

In example above dependencies are passed by interface as constructor parameters. DI framework takes responsibility for passing these parameters in correct order. Interfaces hide details of implementation and allow you to change injected classes w/o modifying dependent code. Your view model starts to depend on abstractions instead of implementations.

Using DI in Avalonia app

As I mentioned before under hood Avalonia uses ReactiveUI framework. It has support for DI using library called Splat. I will show how to use it in your project.

ReactiveUI Splat doc

Adding Splat

Splat is a dependency of Avalonia.ReactiveUI, but you can install it into your root project explicitly using Nuget package manager.

After installing you can access Splat.Locator instance and use it for registering/resolving your services.

Service locator

Internally Splat uses service locator pattern for resolving dependencies. Service locator is actually anti-pattern. It provides static accessible everywhere services resolver (locator). Splat.Locator.Current is an example of such locator. It could be used like this:

public override void OnFrameworkInitializationCompleted()
{
    if (ApplicationLifetime is IClassicDesktopStyleApplicationLifetime desktop)
    {
        desktop.MainWindow = new MainWindow
        {
            DataContext = Locator.Current.GetRequiredService<MainWindowViewModel>()
        };
    }

    base.OnFrameworkInitializationCompleted();
}

Why is it anti-pattern? It used as a way to get rid of dependencies in your code but instead of that it adds one dependency - service locator itself! I recommend to avoid using this pattern if possible. Sometimes it could be useful, but you don't need in most cases.

Registering dependencies

I moved dependencies registration into separate static class Bootstrapper which is called from program entry point:

Bootstrapper.Register(Locator.CurrentMutable, Locator.Current); // note that I'm passing Splat service locators as parameters

Locator.CurrentMutable allows you to register new services while Locator.Current allows to resolve registered services.

Bootstrapper itself looks like this:

public static class Bootstrapper
{
    public static void Register(IMutableDependencyResolver services, IReadonlyDependencyResolver resolver)
    {
        services.Register<IPlatformService>(() => new PlatformService());  // Call services.Register<T> and pass it lambda that creates instance of your service
    }
}

What if you want to create service with constructor parameters? You can achieve this using following code:

services.RegisterLazySingleton<IUnitOfWorkFactory>(() => new LiteDbUnitOfWorkFactory(
    resolver.GetService<DatabaseConfiguration>()
));

Look how IReadonlyDependencyResolver instance is used for creating service instance:

resolver.GetService<DatabaseConfiguration>() // call resolver.GetService<T>() to get instance of type T

IMutableDependencyResolver has 3 methods for registering your service:
1) RegisterConstant allows you to inject some constant value of specific type. I use it for injecting configuration:

var filePanelConfiguration = new FilePanelConfiguration();
configuration.GetSection("FilePanel").Bind(filePanelConfiguration);
services.RegisterConstant(filePanelConfiguration);

2) Register. Service registered with this name will be created each time requested:

services.Register(() => new GeneralSettingsViewModel( 
    resolver.GetService<LanguageSettingsViewModel>()
));

3) RegisterLazySingleton. Once created, service instance will be reused in future so it acts like a singleton.

services.RegisterLazySingleton<IBitmapFactory>(() => new BitmapFactory());

Registering dependencies properly

What if Splat doesn't find implementation for your service? Modern frameworks mostly throw exception in this case but Splat injects null instead. Why is it not good? You want to see error on startup but you will have runtime error (NRE somewhere in your code) that much harder to detect. How to avoid this? I recommend to use extension methods that will throw an error if resolved type instance is null:

public static TService GetRequiredService<TService>(this IReadonlyDependencyResolver resolver)
{
    var service = resolver.GetService<TService>();
    if (service is null) // Splat is not able to resolve type for us
    {
        throw new InvalidOperationException($"Failed to resolve object of type {typeof(TService)}"); // throw error with detailed description
    }

    return service; // return instance if not null
}

Usage of this method is similar:

services.Register<ITopOperationsViewModel>(() => new TopOperationsViewModel(
    resolver.GetRequiredService<ITerminalService>(),
    resolver.GetRequiredService<IDirectoryService>(),
    resolver.GetRequiredService<IFilesOperationsMediator>()
));

Example of code

With this approach you won't have unexpected NullReferenceException in your code.

Examples of using DI

Usage in view model:

public TopOperationsViewModel(
    ITerminalService terminalService,
    IDirectoryService directoryService,
    IFilesOperationsMediator filesOperationsMediator)
{
    _terminalService = terminalService;
    _directoryService = directoryService;
    _filesOperationsMediator = filesOperationsMediator;

    SearchCommand = ReactiveCommand.Create(Search);
    OpenTerminalCommand = ReactiveCommand.Create(OpenTerminal);
}

Registration of this type:

services.Register<ITopOperationsViewModel>(() => new TopOperationsViewModel(
    resolver.GetRequiredService<ITerminalService>(),
    resolver.GetRequiredService<IDirectoryService>(),
    resolver.GetRequiredService<IFilesOperationsMediator>()
));

All registrations code is available here

DI testing

Developers often forget to register implementations of used interfaces. How to automate check that everything is registered properly? For this purpose solution is to write tests. You can do following:
1) Call registrations code and try to resolve specific service/view model
2) Call registrations code and try to create instances of all registered types.

In my project I used combined approach. I checked that all registrations are correct (using wrapper for IMutableDependencyResolver) but also I have dynamically created from code instances of dialogs so I checked them separately. Full tests code is available here.

Conclusion

Hope that now you know how to use DI in your Avalonia app and why is it needed. Feel free to contact me via comments if you have any questions. Thanks for reading!

Database Migration in Production: Tips and Tricks

Ingvar — Fri, 18 Sep 2020 20:25:19 +0000

Hello guys, today I gonna describe such complicated process as switching database on production server.

When and why is it needed

1) Sometimes currently used database stops to comply with all your requirements. It often happens when it reaches some critical size and starts to work much slower. If no performance improvements are available consider idea about changing your database.
2) Changing infrastructure. If you're using DynamoDb but decided to use Azure instead of AWS it's not an option to continue using your current db. You should switch to another database which could be used on Azure.

Used approach

Following approach was used many times and recommended itself as the most stable and clear one. It allows to migrate your databases w/o downtime what is often required. If long downtime is acceptable you can follow less strict and complicated way.
So approach is the following: add new database support into your code so it would be able to write to both destination databases and add some kind of dynamical flag. Based on this flag your app should decide what database(s) is being used for read and write operations at the moment. Start from using only old database (old mode), later switch to dual write read old mode so code starts to write to all databases. Reading from old database is used because new database is almost empty. After that you can start actual database migration, dual write will prevent possible race conditions w/o downtime. After that you can switch to dual write read new mode. In this mode you still write to both databases but use new one for read operation. You can live on this mode for some time if you want to be sure that all data is migrated correctly. Next mode is called new because it uses only new database for read/write. It's a final state of migration. Cleanup your old database code and job is done. Sounds easy? Let's go through it step by step

Initial implementation

Start implementing code before entering old mode. Your code should be able to read and write data from/to both databases. This example assumes you have something like repository pattern used. If you don't have a repository/unit of work patterns you still do following but it will be harder.

Here is an enum that shows all possible options:

public enum RepositoryMode
{
    Old,
    DualWriteReadOld,
    DualWriteReadNew,
    New
}

You can now create your repository for using new database. Lets say you have repository similar to this simple example:

public interface IRepository<T>
{
    T Get(int id);

    void Insert(int id, T data);
}

After that you can create AbRepository that reads/writes data to/from repository based on flag:

public class AbRepository<T> : IRepository<T>
{
    private readonly Func<RepositoryMode> _getRepoModeFunc;
    private readonly IRepository<T> _oldRepository;
    private readonly IRepository<T> _newRepository;

    public AbRepository(
        Func<RepositoryMode> getRepoModeFunc, // if restart is an option you can inject static configuration, options pattern is also could work here
        IRepository<T> oldRepository,
        IRepository<T> newRepository)
    {
        _getRepoModeFunc = getRepoModeFunc;
        _oldRepository = oldRepository;
        _newRepository = newRepository;
    }

    public T Get(int id)
    {
        var mode = _getRepoModeFunc();

        // we read data from current active database
        return mode == RepositoryMode.Old || mode == RepositoryMode.DualWriteReadOld
            ? _oldRepository.Get(id)
            : _newRepository.Get(id);
    }

    public void Insert(int id, T data)
    {
        var mode = _getRepoModeFunc();

        if (mode != RepositoryMode.New)
        {
            // write data to old db if needed
            _oldRepository.Insert(id, data);
        }

        if (mode != RepositoryMode.Old)
        {
            // write data to new db if needed
            _newRepository.Insert(id, data);
        }
    }
}

This class could be more complex and process writing errors etc.

Old mode

After finishing code deploy it to production in old mode. App should work as previously, only old database is used.

Dual write read old mode

In this mode app starts to write to new database too. It's useful if you can't do a downtime during migration because it prevents possible race conditions. Also you can check your write performance to new database on this step. I recommend to start migration right after switching to this step (of course don't forget to check that write to new database works as expected).

Migration

It's the hardest and the most interesting part of whole process. For migration you should write migrator program. It's mostly console application that reads data from old database and writes it to new. But good implementation is hard and not so simple. Here are things that you should do in your migrator:
1) Read/write balancing. You shouldn't read faster than you can write to new database. Also I don't recommend to read too fast because it could affect production performance. Check your read speed and read items per second(minute) is above some limit pause reading for small amount of time. Same with write operations. In write operations I recommend to write in few (limited) count of threads, it speeds up migration process.
2) Use replica for read from old storage if possible
3) Add retries everywhere. If operation fails it should be executed again after some timeout and recorded to log. It prevents you from loosing any data during migration procedure.
4) Add pausing option. If possible you should be able to stop migrator and continue your migration from saved point later. I achieved this using reading by index. You can save your latest write position (note, write to new database position, not read from old database position!) into JSON file and read it on next start.
5) Split read and write code. Read and write operations should be separated in your code and shouldn't know anything about each other. I did it following way: moved read and write to different classes. Read class writes to BlockingCollection instance and write code reads from it and submits data to new storage.

Migrator high-level design
6) Don't afraid to modify your entities from source db before writing to destination db. You can fix bugs/issues with entities during migration process, skip empty items or use completely different model in your new code. That's fine.
7) Try to reuse part of your code for migrating different entities. Create generic base classes for read/write.
8) Use write with version if possible. If your source database has records versioning option you can read item before write (don't read one by one, do bulk read instead) and try to put them with incremented version. It prevents possible race conditions. If race condition occurred you should process it in your code (retry operation or cancel if dual write mode did job).
9) Monitor databases load during migration for safety.
10) Print some statistics to console/log file. You will be able to see progress of operation in this case.

Dual write read new mode

After migration both databases have up to date data so you can finally start to use new database for read operations. This mode is required because it gives you time to check if migration was successful. It's still an option to rollback to dual write read old mode if something went wrong and do migration again if needed. I recommend to live with this mode in production for a while before switching to next mode.

New mode

In new mode only read/write to new database is used. Old code still exists but no more called. Note that you can't rollback to previous mode now because your old database has outdated data after switching to this mode so please double check that you're ready to switch to this mode.

Cleanup

After all other steps done don't forget to cleanup obsolete code that still uses your old database. It's a common mistake when people forget to do that. Migration process is finished only after this step.

Thanks for reading, hope it was useful for you. In next post for experts I will write about monitoring your services using Prometheus. Stay tuned!

Styles in AvaloniaUI

Ingvar — Sat, 29 Aug 2020 20:38:18 +0000

Hello guys! In this post I gonna explain how to stylish your AvaloniaUI app according to Avalonia best practices. As always I gonna use my app Camelot as an example.

Difference from WPF

In WPF styles are added in separate xaml files using xaml syntax like here:

<Style TargetType="TextBlock">
    <Setter Property="Foreground" Value="Gray" />
    <Setter Property="FontSize" Value="24" />
</Style>

Styles in WPF target specific element types. If you want to set different styles on different views for same control you should include different styles files into your view. Avalonia provides more flexible approach here. It allows to add classes (similar to CSS) to your element and specify different styles for classes. Also it has selectors support (also similar to CSS approach) that allows you to avoid using triggers and write your styles in similar way.

Styling an element

In Avalonia it's also recommended to separate view and styles files. Let's start from view example:

<TextBlock Classes="dataGridColumnHeaderTextBlock" Text="{x:Static p:Resources.Extension}" />

In this example I added a TextBox. Note that I don't have any inline styles here. The only thing (related to styling) that was set is Classes="dataGridColumnHeaderTextBlock". This property adds class to element. I use this class for adding styles:

<Style Selector="TextBlock.dataGridColumnHeaderTextBlock">
    <Setter Property="Foreground" Value="{DynamicResource InactiveTabForegroundBrush}" />
</Style>

In selector field I add css-like selector for my element. It always should contain element type (otherwise it's not possible to determine set of properties available for customizing) and optionally has classes, inner elements and events. For example, for hover event you will have something like this:

<Style Selector="Button.tabButton:pointerover">
    <Setter Property="Background" Value="{DynamicResource TabButtonHoverBrush}" />
</Style>

:pointerover sets styles for button if only pointer is above this button, looks simple, right?

I don't want to describe all available selectors here because they are listed in official documentation. It's available here: Avalonia official styling docs

Creating app theme

After adding basic styling you would probably think about adding themes support. For example, some apps have both dark and light themes and allow user to choose between them. I will explain how to achieve this.

I extracted all theme-related styles into single file. I didn't add it to App.xaml like it's done by default but I load it dynamically on app start. For this purpose I modified my App.xaml.cs and added themes loading:

public override void Initialize()
{
    AvaloniaXamlLoader.Load(this);
    LoadTheme();
}

private void LoadTheme()
{
    Styles.Add(new DarkTheme());
}

For now I have only one dark theme so I simply add dark theme to list of app styles. In future I gonna light theme as well so in this method I will read value from config and load theme based on this value. Pretty simple. Also I had to add cs file for using styles from code:

using Avalonia.Markup.Xaml;
using AvaloniaStyles = Avalonia.Styling.Styles;

namespace Camelot.Styles.Themes
{
    public class DarkTheme : AvaloniaStyles
    {
        public DarkTheme() => AvaloniaXamlLoader.Load(this);
    }
}

Themes best practices: files structure

For better using themes I used following solution structure:
1) I completely separated view and styles, there are no styling in views except basic stuff like margin etc.
2) I moved themes independent styles into their own files
3) I created a directory for each style and added theme file there. It could include other style files too if needed
4) Into theme file I put colors and basic options values, so theme file modifies color scheme and nothing more

Themes best practices: colors

Coloring an Avalonia app is definitely the hardest part of styling your project. There are no good docs regarding this so I had to learn on my own mistakes here 😃

Initially I tried WPF-like way. I specified brushes in common styling files and tried to override them in themes file. I called my brushes like TransparentButtonBorderBrush so it was obvious where is it used. But sometimes it didn't work. Avalonia continued to show me it's default control colors! It ignored all my settings even if I added them into view itself. It was really confusing. Where does it get those weird colors? Obviously it loaded them from somewhere and as longer I didn't have anything except my custom and Avalonia default styles in my app, I decided to look into default styles and check how they work. I expected similar approach there - different brushes per each control etc. I was wrong.

Initially I had Avalonia dark theme used which is available here. I opened that file and found out that it has only colors/brushes specified and nothing else! I opened control styles (available here) and realized that all Avalonia controls has been styled in following way:
1) In control file set of brushes for control is specified in both view code and styles code
2) Controls brushes are reused across different controls and all listed in theme file

I understood that I can style my app by modifying default Avalonia brushes. For example here is part of my dark theme file:

<SolidColorBrush x:Key="ThemeBackgroundBrush" Color="{DynamicResource MainBackgroundColor}" />
<SolidColorBrush x:Key="ThemeForegroundBrush" Color="{DynamicResource WhiteColor}" Opacity="0.8" />
<SolidColorBrush x:Key="HighlightBrush" Color="{DynamicResource OrangeColor}" />
<SolidColorBrush x:Key="HighlightForegroundBrush" Color="{DynamicResource WhiteColor}" Opacity="0.9" />
<SolidColorBrush x:Key="ThemeAccentBrush4" Color="{DynamicResource OrangeColor}" />
<SolidColorBrush x:Key="ThemeAccentBrush3" Color="{DynamicResource BrightOrangeColor}" />
<SolidColorBrush x:Key="ThemeControlLowBrush" Color="{DynamicResource BlackColor}" Opacity="0.2" />

If I need to override some styles I used same selectors like default style has, for example:

<Style Selector="ToggleButton:checked /template/ ContentPresenter">
    <Setter Property="Background" Value="Transparent" />
</Style>

My algorithm of styling is following:
1) Check default brush for control that I want to style
2) If it's possible override default brush with your custom value
3) If it's not possible write new selector that will add color to control inner part that needs to be styled. This style could use another default brush or your custom one.

This algorithm allowed me to style whole app exactly as I wished. Screenshots on different platforms are available here

I wanna say that AvaloniaUI styling system it neat and flexible. It completely covers all needs in css-like manner. It's a bit hard to understand initially but still easy even for beginners. What do you think about styles in Avalonia? Let me know in comments 😃

.Net Core 3: Garbage Collector Freeze Issue

Ingvar — Wed, 22 Jul 2020 19:51:07 +0000

Hello guys, in this post I gonna write about garbage collection freeze issue in .Net Core 3 that we found on our production. This post is dedicated to dotnet experts mostly. I doubt that this issue could be reproduced on small projects so it will be interesting for devs from enterprise size projects.

Introduction

After migration to .Net Core described in previous post we finally got our production working on Linux. We had a lot of Linux machines with our services and they worked pretty well. But after some time we faced issue. Sometimes one of our services stops working on few machines randomly. It started to happen pretty often (~20 times per day) so we decided to investigate this issue.

Initial look on issue

Initially we thought that our services crashes on these machines. When it happened again we connected to this machine via ssh and it appeared that dotnet process was working there. Anyway none of our requests (even health check) didn't work because dotnet process was stuck and didn't respond to incoming requests.

We created a dump from this stuck process. Dotnet provides tools for creating and analyzing dumps such as dotnet-dump etc. Script that we used for dump creation:



createdump <pid> -u -f <dump_file>

Linux dumps could be analyzed on Linux only. I prefer to use dotnet-dump itself for analyzing but you can consider using lldb if you need more details. Anyway it's a bit hard to find exact lldb version that works for specific net core and sos plugin. For analyzing I use Docker container mounted to directory with dump.

Dump investigation

We collected few dumps from stuck machines and started to investigate them. It was a bit problematic to grab a dump because we had automatic restart of not responding services. Also it takes some time which sometimes is not acceptable for high-load on production. Note that we didn't use gcdump because it triggers full GC and doesn't work on stuck processes like we had. Even for working processes it fails sometimes (especially if process uses a lot of RAM).

For dump analyzing I used dotnet-dump tool itself. dotnet-dump is some kind of old-school SOS analyzer. It uses same commands but it supports .Net Core and different platforms. This how I checked it:
1) Initially I thought we have a deadlock somewhere in the code. So I used clrthreads command that showed me list of all CLR threads. After that I used clrstack -all to print all threads stack traces. I checked few dumps and most of them had completely different stack traces! It wasn't result that I expected. No sign of deadlock or similar issue.

Example of clrthreads output
2) Also I checked objects in memory. I used dumpheap for this purpose. For example, dumpheap -stat -min 10000 showed me statistics per type for objects large than 10k bytes. It's the fastest way to detect large objects. I found some large (2-4 MB objects) but nothing really large.
3) Dump had some signs of coming GC: objects that were ready to be removed, gcframe in stack traces. But what if it's just a coincidence?

Basic performance metrics

In order to check that GC is the issue and to look back in past before freeze I added more metrics. For this purpose I used Prometheus. It was used because it allows to grab metrics with different intervals w/o any need to send batches of metrics somewhere because it uses pull model instead of push. Also dotnet version has a lot of good built-in metrics that I used for investigation. Later in one of my future posts I will explain details of using Prometheus in dotnet apps.

I collected a lot of metrics from production with 1 second pulling interval. Metrics showed me following:

1) Full garbage collection interval is too low. Every few seconds it collected garbage in all generations! It's obvious that something went wrong.

Every time memory decreases after GC to minimal value, that means we do full GC each time
2) A lot of 2nd generation objects were created during app lifetime. Most of them were obviously objects from LOH. It looked weird because I didn't expect we're creating so many large objects.
3) Each time before freeze memory stat showed growth or even was on peak like on image above. GC starts collection approximately on same memory usage levels so I thought it's good proof that GC was the reason of our issue.

Improving performance

Freezing GC seems to be a bug in GC itself but what exactly triggers it? Large objects that force full GC each time I assumed. I opened a dump again and checked largest objects there. This is what I found:
1) Large strings - 4 MB in UTF-16.
2) Large byte arrays - 2 MB each.
These objects looked suspiciously so I decided to check them. It appeared that we used a two-level cache for large object that were used pretty often. First level was in memory cache (IMemoryCache) and the second one was compressed cache in database. These objects were shared. They contained some configuration for users and different users could have same configuration so we also used additional intermediate entity. It looked like this:



public class UserSettings
{
    public string UserId { get; set; }

    public string SettingsHash { get; set; }
}

SettingsHash is a hash of large settings object that was used as primary key for settings search in cache and database. So initially we retrieved this hash value from cache by user id and later loaded settings by user settings hash. This way of caching helped us to reuse our large objects but it's a bit complicated. As you know complicated code has more bugs - this is exactly what happened to us.

I started to play with our caching code. I asked myself following questions:
1) Does it work at all or cache misses rate is too high?
2) If it works do we need additional configuration for it, for example ttl tuning?
3) Is there a way to simplify or improve it?
4) We used compressing for settings in database using LZ4 - I decided to check that we're compressing everything properly too
5) Does our database cache work too slow? It required loading large byte array, decompressing into string and deserializing it, sounds like good reason to be slow

I added metrics for cache so I could get exact cache stats. I used Prometheus for this purpose as previously, it has good syntax for such checks. Also I checked our (de)serialization and (de)compressing operations speed using it. Here is an example of neat Prometheus syntax:



using (_histogram.NewTimer()) 
{
   // some code here
}

This example tracks execution time of code inside using block and allows to build histogram for it later. Looks easy, right? No need to use StopWatch or similar stuff.

I checked our compression code. Code itself was almost good but it missed few using's - it created memory streams w/o calling Dispose. Ok, it's not so serious issue. But after looking closer I found much more serious one. Before compressing object was converted to json, but json serialization settings were incorrect and object was saved with all formatting! So after serializing and compression it was even larger that after proper serializing w/o formatting. I fixed that and compression started to work as expected.

I wasn't able to improve ttl for our cache because it was pretty good. Large objects had long ttl and small objects were recreated much often. Large objects change less often so it's reasonable. Looks good, let's go further.

Last thing that I checked was caching logic itself. We used following algorithm:
1) Check for UserSettings object in local cache
2) If it's in cache return it and check for large settings object in local cache or db
3) If it's not in cache load it from db or calculate and load settings from db

See a bug? If user settings object is not in cache we don't even try to look for large settings object in local cache! And because ttl of these types of objects are completely different and large objects are reused across user settings it's a serious mistake. We created a lot of large objects on each request and load them into local cache replacing same objects. Good way to waste your resources 😃 I fixed that bug - added check for local settings objects before asking external cache.

APM

Right at that moment we tried new APM system on our production. After some time it detected our GC problem as well:

I strictly recommend to use (good) APM on production because it could save you a lot of time.

Fix results

During deployment to production I immediately started to monitor performance. I was shocked because right after deploy service latency went down by 50%:

Metrics also verified that cache works good (cache misses count is close to 0):

Local cache

Local + db cache

Also gc stats showed that we started to create much less objects in 2nd generation:

Right after deploy large objects count became closer to 0

Second generation on few machines

Execution time for search in cache became much faster too because local cache started to work as a charm. Wow!

We got so high cache success rate, more than 90% of our requests were almost instant!

After some time we checked our restart job. It appeared that in 72h after deploy we didn't have any stuck processes! Amazing result after 20+ freezes in 24h. Later we had few freezes due to other performance issues but rate was at least 10x times lower than ever before.

Conclusion

After all improvements GC freeze became rare issue for us (yes, it is still reproducible because we also have another type of too large objects that are extremely difficult to get rid of, we will try to do it on close future). Also performance of our hottest endpoint was improved significantly. We contacted Microsoft regarding this issue but still are waiting for their response. GC freeze is not fixed in dotnet itself so it could happen in your project too. Are you ready?

AvaloniaUI: Introduction for Beginners

Ingvar — Sat, 18 Jul 2020 11:25:14 +0000

Hello! This post starts my new post series dedicated to beginners willing to dive into Avalonia. In this post I gonna describe Avalonia usage basics. In later posts I will explain some undocumented sides of developing Avalonia app.

AvaloniaUI is cross-platform UI framework for .Net Core. It supports Windows, Linux and macos out of box. Avalonia follows WPF way in building desktop apps but it has few improvements if compare with WPF.

I wanna explain Avalonia principles using my own pet project called Camelot as an example. It has everything that I gonna show in this blog post series and I hope it could be good example for beginners. I think that one good example is better than 100 docs. Even for Avalonia where docs cover most of needs.

Short architecture overview

Desktops frameworks have completely different solution structure from structure that web developers used to see. Avalonia apps could be separated into following components:

1) Views (UI markup). Markup is represented by XAML files, one XAML file per view/control. XAML is XML-based markup language which was introduced in WPF. It's some kind of HTML in web world.
2) Styles. Styles are added in separate XAML files. This way is similar to web css files. In styles file root tag is Styles
3) Code-behind files. Similar to old ASP.NET code-behind approach. Every view should have a related class in code-behind file. In most cases it only loads XAML of a class. In some cases it also contains some logic. Previously it was popular to write business logic in code-behind files but it's wrong approach. Code-behind files are attached to their views so they are not reusable and depends on view. View update can easily break code-behind file. I will describe better approach below.
4) View models. Here is proper replacement for code-behind files which is very popular now. View model is a class which is connected to view using binding mechanism, in Avalonia each view model inherits ReactiveObject. In XAML you can specify property that you want bind to and your control will be connected to this property. For example, you can connect text box and string property. Every change in text box will change property and every change of property will change text on UI. Sounds better than processing events or manual reading text box value, right?
5) App business logic. Represented by C# classes. Mostly it's separated into business and domain logic. This part is completely independent from UI and view models layers.
6) Tests. Optionally you can have tests for your UI and view models or business logic.

View

As mentioned before, view is XAML file with XML-like syntax:

<Window xmlns="https://github.com/avaloniaui"
        xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
        xmlns:d="http://schemas.microsoft.com/expression/blend/2008"
        xmlns:mc="http://schemas.openxmlformats.org/markup-compatibility/2006"
        xmlns:p="clr-namespace:Camelot.Properties"
        xmlns:dialogs="clr-namespace:Camelot.ViewModels.Implementations.Dialogs;assembly=Camelot.ViewModels"
        mc:Ignorable="d" d:DesignWidth="800" d:DesignHeight="450"
        x:Class="Camelot.Views.Dialogs.CreateDirectoryDialog"
        Width="400" Height="150"
        Title="{x:Static p:Resources.CreateDirectoryTitle}">

    <Design.DataContext>
        <dialogs:CreateDirectoryDialogViewModel />
    </Design.DataContext>

    <Grid RowDefinitions="Auto,Auto,Auto" Margin="10" HorizontalAlignment="Center">
        <TextBlock Classes="dialogTextBlock" Text="{x:Static p:Resources.EnterDirectoryNameWithColon}" />
        <TextBox Grid.Row="1" x:Name="DirectoryNameTextBox" Text="{Binding DirectoryName}" />
        <StackPanel Grid.Row="2" Orientation="Horizontal" HorizontalAlignment="Center">
            <Button Classes="dialogButton transparentDialogButton" Content="{x:Static p:Resources.Cancel}" Command="{Binding CancelCommand}" />
            <Button Classes="dialogButton" Content="{x:Static p:Resources.Create}" Command="{Binding CreateCommand}" />
        </StackPanel>
    </Grid>
</Window>

It could be window or user control. It has connected code-behind class (x:Class="Camelot.Views.Dialogs.CreateDirectoryDialog")
<Design.DataContext> contains reference to view model for design time purpose. It enables intellijsense for bindings and helps to show preview of your view.

View itself represented by xml tags with attributes like in html. Let's look closer at following tag:

<TextBox Grid.Row="1" x:Name="DirectoryNameTextBox" Text="{Binding DirectoryName}" />

TextBox is control name. Text is attribute which is connected to DirectoryName property of my view model via binding. x:Name attribute specifies unique control name.
Note that like in WPF or HTML some controls act like containers (panels) for other controls, for example Grid from code snippet above. For better understanding of Avalonia controls I recommend to read official Avalonia docs. They are pretty good.

Useful links:
1) XAML quick start
2) User controls
3) Panels

Code-behind files

Typical code-behind files looks like this one:

using Avalonia.Controls;
using Avalonia.Markup.Xaml;

namespace Camelot.Views.Main
{
    public class OperationsView : UserControl
    {
        public OperationsView()
        {
            InitializeComponent();
        }

        private void InitializeComponent()
        {
            AvaloniaXamlLoader.Load(this);
        }
    }
}

All it does is XAML loading for itself. In some complicated cases you can write some logic there. Here is an example. As I said before it's not good approach to put your business logic here but in some cases there is no other option due to missing features in Avalonia.

Avalonia code-behind docs

Styles

Style files contains a lot of <Style> tag with styles for elements. Example:

<Style Selector="Button">
    <Setter Property="BorderThickness" Value="0" />
    <Setter Property="Background" Value="Transparent" />
    <Setter Property="Cursor" Value="Hand" />
</Style>

Every style is pretty simple. Selector describes what control we want to restyle, in every setter there is name and value for styled property. For example <Setter Property="Cursor" Value="Hand" /> sets hand cursor for a buttons which should be default for buttons in framework I think :) Selectors could be complex like in css, for example ComboBoxItem:selected TextBlock will style text block inside selected combo box item. Avalonia has pretty good style docs, I recommend to check them out.

View model

View model is a key element of MVVM pattern. It allows you to connect your UI and and backend properly and update UI implicitly from your code without need to know UI details. Also with this pattern your view and view model are separated because they are connected via binding instead of element names like in code-behind files. So you reuse your view models with different views w/o adjusting your code and so on.

Data-binding docs

In Avalonia ReactiveUI is used in view models. It's open source framework that allows to write view models code in asynchronous "reactive" way. It allows to create chains of functions that reacts on user input. Here is an example:

this.WhenAnyValue(x => x.CurrentDirectory, x => x.SelectedTab)
    .Throttle(TimeSpan.FromMilliseconds(filePanelConfiguration.SaveTimeoutMs))
    .Subscribe(_ => SaveState());

In this example if any of observable properties changed after filePanelConfiguration.SaveTimeoutMs ms after change view model saves its state to database. Although I'm using reactive way too, I prefer using old-school WPF approach for view models mostly (actually I'm using mixed approach). What is the difference? WPF approach doesn't use function chains for configuring view models. It's simple and more readable in most cases because there is no need to analyze list of operations in chains which can be long and difficult to understand. I will explain details in next posts.

Business logic

Nothing interesting here, just business logic. I connected my view models to business logic (services) using dependency injection approach. In this approach view models get all services by their interfaces in constructor so they don't know exact implementation details. This approach is flexible and allows to modify both services and view models independently. I will provide implementation details in one of next posts. Here is an example of using DI:

namespace Camelot.ViewModels.Implementations.Dialogs
{
    public class CreateDirectoryDialogViewModel : ParameterizedDialogViewModelBase<CreateDirectoryDialogResult, CreateDirectoryNavigationParameter>
    {
        // readonly fields for injected services
        // note that only interfaces are used
        private readonly IDirectoryService _directoryService;
        private readonly IFileService _fileService;
        private readonly IPathService _pathService;

        // some code

        // constructor with injected dependecies
        public CreateDirectoryDialogViewModel(
            IDirectoryService directoryService,
            IFileService fileService,
            IPathService pathService)
        {
            _directoryService = directoryService;
            _fileService = fileService;
            _pathService = pathService;
        }

        // some code here
    }
}

Tests

Also I added tests for my app (link) using XUnit and Moq. As for now I have only business logic and view models tests, but in future I gonna add UI tests as well. I will write post with detailed explanations later in this series (Avalonia docs miss that information). I wanna note that you can test your view models too (example). Why did I do that? For most beginners it's not obvious why tests are so important. In fact they help to prevent regression bugs and allow to reduce manual testing effort. In my case I refactored and modified too much code so unit tests saved a lot of time for me and helped to find many bugs.

What is next

In next post in this series I gonna touch Avalonia styling approach which differs from WPF and could be confusing for WPF developers. Also it has some undocumented things every Avalonia dev should know. Stay tuned!

Our migration from .Net Framework to .Net Core: pitfalls and tips

Ingvar — Mon, 06 Jul 2020 22:02:00 +0000

Hello everyone, it's my first post here. In this post I wanna talk about our migration of large enterprise project from .Net Framework 4.7.2 to .Net Core 3.1, problems that we faced and how we overcame them. It will be a long post so take a cup of coffee or tea and let's go!

Our goal

Our primary goal was to make our large monorepo solution work on Linux, so latest LTS Net Core version was selected in order to achieve this. Previously we had our production running on Windows which is pretty expensive. Also it was hosted by another company so we had no real control over it. Our own data centers have all required infrastructure for running our services but they support Linux only so our target was obvious and clear - port existing code and infrastructure to Net Core and run it on Linux environment.

Let's go through all our long way step by step!

Porting all libraries and nugets to netstandard2.0

During migration process we had to support both old and new versions of code and it's too difficult when you have to keep two separate codebases. Our first step was to migrate all our shared code to netstandard2.0. Netstandard2.0 is supported by both Net Core 3 and .Net472 so it was a good choice.

In most cases migration to netstandard doesn't require too much effort. We had to remove a bit of .Net Framework specific code, update all our libraries to versions with netstandard support. Anyway even here we got few pitfalls. They were related to .Net core transitive dependencies. As longer we still had web api project on 472 then it doesn't load dependencies required by converted to netstandard projects that ended with runtime errors. Also in different projects we had different versions of same nugets installed. After switching to netstandard we missed that point and got classic runtime errors about missing methods and so on.

Every time you want to update library - update it in whole solution, that's important!

Porting all tests to .Net core 3.1

If compare with libraries there is only one huge difference. Test can't target net standard. They target specific platform instead. So we targeted .Net Core 3.1 instead. We also had to update all libs to latest version and got few hundreds compile errors due to changed fluent assertion syntax 😃 Anyway it was easy to fix. Auto replace did it for us 😃 Fixed tests showed that we have no issues in code except transitive dependencies mentioned above 😃 Also we had to skip our current DI tests because we had nothing to test at that moment

Migration of web project itself: new project structure

We created new Net Core project alongside with old one that was under active development at that moment. We created project from scratch and decided to copy logic from old project step by step. We didn't have ability to keep only old or only new project so it was hard to copy-paste changes from old project to new one, it was merge hell 😃

Migration of web project itself: middleware and filters

We decided to start from pipeline in our migration process. Previously on .Net472 we used few global and few local filters, some of them didn't work as real filters because they actually added some info to request-response and didn't filter anything 😃 Instead of global filters Net Core recommends to use middleware. Middleware concept came to dotnet world from NodeJS I suppose. We rewrote our global filters as set of middlewares. Here we did our first serious mistake. In middleware world order is important. First registered middleware will be executed first before request execution and last after and so on. We had invalid order initially and didn't check it later because of time trouble. We had logging middleware that processed all unhandled errors, but it was executed after middleware that suppressed all errors so we lost logs about exceptions on production. Oops.

Always check your middlewares order!

Also we had few filters. We rewrote them using new filter syntax. Also we used Net Core type filter that allowed us to inject dependencies into filter constructor (on .Net472 we had terrible property injection, don't use it unless you want to obfuscate your code). Here we did another serious mistake. It appeared that we had our filters as singletons previously while Net Core version were recreated each time. We resolved it using IsReusable property for filter. Our code:

public class CustomFilterAttribute : TypeFilterAttribute // attribute for acion or class
{
    public CustomFilterAttribute(bool someArgument = true)
        : base(typeof(CustomFilter)) // filter itself
    {
        Arguments = new object[] {someArgument}; // some attribute argument that we want to pass to filter too
        IsReusable = true;
    }
}

In order to improve cod quality some code was moved to global filter. It was our third mistake here - we missed a point that global filter is being executed before others. Here is good article about filters

Don't recreate filters each time unless you really need this behavior

Migration of web project itself: controllers

Controllers syntax changed a bit in Net core but we didn't have any real problems with it. We refactored all our controllers - removed injections in methods, property injections and other horrible things. It took some time because we had a lot of controllers with many actions inside. Also we splitted too large controllers into separate ones for readability purposes. When you have 90% of all actions in one controller think if your code violates SRP 😃

I should also mention problem with serialization. In Net Core default serializer is different so we got different JSON. In some services we had custom formatters and adapted it to new framework. In other services we had to change serializer settings as well.

Migration of frontend stuff

Also we had few views used mostly for testing purposes. We moved them to wwwroot and added static files middleware. It was enough to make them work but also we found out that in html our scripts and styles were included with names in wrong case. It worked fine on case-insensitive Windows but on Linux it created a lot of 404 errors.

Check if case is correct everywhere if you wanna use Linux!

The most problematic part on frontend side was SignalR. Latest version was completely different from version that we used before. After update we had issue with transport because previously default one was LongPolling and it was switched to WebSockets. Our nginx liked that and ate all websockets requests so we had to use LongPolling again 😃

Docker

We used Docker for creating containers for our apps. Dockerizing Net Core app is easy task and probably it was the easiest part of our journey 😃

Final fixes

During local testing on final phase we realized that our DI framework doesn't instantiate controllers for some time after start. After digging it appeared that we had no warmup for it and tried to create too much services on first request. So we added warmup for DI. It created all controllers and services on start for us so our app was able to execute requests right after start. This thing replaced DI tests for us as well. Example of such simple warmup is provided below (don't forget to add .AddControllersAsServices() to make it work):

private void WarmUpControllers(IServiceProvider serviceProvider)
{
    var controllersList = _serviceCollection.Where(x =>
            x.ServiceType.IsSubclassOf(typeof(ControllerBase)) &&
            !x.ServiceType.IsAbstract)
        .Select(x => x.ServiceType);

    foreach (var controller in controllersList)
    {
        serviceProvider.GetRequiredService(controller);
    }
}

Also we added some technical stuff like health checks. We also had funny moment here. I wrote health check that pinged Mongo database, it worked on Windows. But on Linux it throwed OperationCanceledException immediately if cancellation token is passed! I had to use older version of Mongo driver to make it work because I didn't want to remove that cancellation token 😃 Also please note that we have separate port (called management port) for health checks and similar stuff. These port is not exposed outside.

Test your code on env similar to production

Testing phase: how everything went wrong

Finally we were ready to ask our QA to start testing. Our api is consumed by mobile clients mostly while we tested it using Swagger mostly so we didn't know details of client implementation. Soon after start of testing QA found strange problem. 2nd request to endpoint failed each time for them. We started to digging this issue and faced situation from these horror films where tech debt eats developers. In Net472 solution DI framework called LightInject was used. We copied it too because we didn't want to spent a month on fixing registrations. As its name says it's light, but in fact it allows to do a lot of weird things because of its "lightweight" checks. So in old solution we found a lot of cases where devs tried to optimize DI calls and injected scoped services into singletons. In this case scoped service becomes a singleton too. In order to avoid this in most cases factory Func<T> was injected and resolved into services right in method! It worked in old Net Framework but it didn't in Net Core.

After a week playing with this we found a lot of related cool bugs. For example we had bug when each first request to action fails but others work. Even if different actions have same code inside first requests to each one failed O_o. Finally we found a hack that helped us fix that. Instead of using default LightInject factories for services we used such factory:

var serviceProvider = httpContextAccessor.HttpContext.RequestServices;

return serviceProvider.GetRequiredService<TService>();

It's a terrible hack and I hope I won't use it again (we had to copy it to another service anyway) 😃 Hopefully this hack helped us to fix our issue and our QA started to test our dotnet service once again.

Use frameworks that limit their incorrect usage. I recommend to use default frameworks if possible. For example Net Core is good for 99% of projects.

Testing and deployment to production

I was kindly surprised but we didn't have any business logic related bugs in our code after migration (probably we had but QA didn't find them 😃). All we had was poor performance and missing logs 😃 I think performance issue was related mostly to our Mongo driver that showed worse speed on Net Core (in fact we used Mongo in a wrong way but it's a long story), but difference wasn't critical. Probably on third or forth attempt we successfully deployed our work to production with all fixes. So now our production uses .Net Core 3.1 and it's running on Linux.

Conclusion

Migration to Net Core was a real challenge for us. It's really painful to migrate large enterprise solution from old framework, especially when you are in time trouble and have technical debt like ours 😃 Anyway risks worth it. Net Framework is obsolete so I recommend to switch to cross-platform Net Core and enjoy dotnet apps running on Linux!

Thanks for reading!