DEV Community: Sotirios Mantziaris

Parallelize work using parwork

Sotirios Mantziaris — Fri, 28 Mar 2025 19:29:43 +0000

In order to process a lot of work we have to parallelize work across all cores, and especially if it's CPU bound.
Go has goroutines, which can be used to parallelize the work, but there is the cost of context switching for a lot of goroutines.
Minimizing this context switching can be achieved by using a fork-join model when processing work.

Parwork solves this problem by using goroutines, channels and waitgroups. It creates workers (goroutines) that pull
work of a queue (channel), process the work and report the work back to a queue (channel).
This is done in a abstracted way so the user has to provide implementation for:

a Work interface
a WorkGenerator function
a WorkCollector function

Work interface

type Work interface {
    Do()
    GetError() error
    Result() interface{}
}

The work interface defines a method Do() which contains all the processing logic of the work item. The GetError() error method can be used to flag the work item as failed and return a error. The Result() interface{} defines a method which returns the result of the work. Due to the lack of generics the data return has to be cast from interface{} to the actual result type in order to be usable in the WorkCollector.

The following example work implementation shows a work item that calculates a MD5 hash of a string:

// md5Work defines a structure that holds the value to be hashed and the result of the hashing
type md5Work struct {
    data   []byte
    hashed []byte
}

// Do calculates the hash of the given value
func (gw *md5Work) Do() {
    gw.data = md5.New().Sum(gw.hashed)
}

// GetError returns nil since the work does not fail
func (gw *md5Work) GetError() error {
    return nil
}

// Result returns the hashed result
func (gw *md5Work) Result() interface{} {
    return gw.data
}

Check out the examples folder of the Github repo for a complete example.

WorkGenerator function

type WorkGenerator func() Work

The WorkGenerator function allows the user to provide a implementation that returns on each call a work item to be processed. If the generator returns nil the generation of work has finished.

Check out the examples folder of the Github repo for a implementation of the generator.

WorkCollector function

type WorkCollector func(Work)

The WorkCollector function takes as a argument a completed Work item. It can check for a failure by calling the GetError or the Result method of the Work item and handle it appropriately.

Check out the examples folder of the Github repo for a implementation of the collector.

Check out

Head over to the Github repo to see the code, with a working example, try it and if you find something, like a bug or a improvement, don't hesitate do open a issue or better yet create a PR.

Thanks and enjoy!

Various .NET Benchmarks

Sotirios Mantziaris — Fri, 28 Mar 2025 19:29:42 +0000

A lot of times i was wondering what is the best performing code.
In order to determine that i had to benchmark my code, but benchmarks are hard to write.
Luckily there is a open source project that does this work perfectly good and very easy.

The name of the library is BenchmarkDotNet and the documentation can be found here.
The only thing you have to do is:

Create a console application project (.NET/.NET Core)
Install the nuget BenchmarkDotNet and it's dependencies
Always run in release mode!

The following code is needed in the Program Main

public static void Main(string[] args)
{
    BenchmarkSwitcher.FromAssembly(typeof(Program)
                    .GetTypeInfo().Assembly).Run(args);
}

which will scan the assembly for benchmarks and asks you which one to run.

A typical benchmark is just a class with methods that are annotated with the [Benchmark] attribute like the following one:

[MemoryDiagnoser]
public class DateTimeToStringBenchmark
{
    private static readonly DateTime _dateTime = 
        new DateTime(2016, 12, 31, 23, 59, 59, 999);

    [Benchmark(Baseline = true, Description = "DateTime ToString")]
    public string DateTimeToString()
    {
        return _dateTime.ToString();
    }

    [Benchmark(Description = "DateTime ToString with format")]
    public string DateTimeToStringFormat()
    {
        return _dateTime.ToString("g");
    }
}

Pretty simple isn't it?

After the run the result is the following:

Method	Mean	Error	StdDev	Scaled	ScaledSD	Gen 0	Gen 1	Allocated
'DateTime ToString'	848.8 ns	2.8059 ns	2.4874 ns	1.00	0.00	0.0410	-	132 B
'DateTime ToString with format'	790.0 ns	18.0956 ns	16.0413 ns	0.93	0.02	0.0391	0.0104	124 B

I get the timings of the method and by using the [MemoryDiagnoser] attribute i get event the GC Stats.

Conducted Benchmarks

Given the easiness to create such benchmark i have started a github repository named DotNetBenchmarks.

There you can find some of the performance questions that i have about some components like logging frameworks, XML Serialization, String concatenation etc.
Every time i have to check the performance of some component i will add a new benchmark to this repository. If anyone likes to contribute, even better!
Make a PR!

Please note that:

These benchmark are not conducted in order to make certain libraries look good or bad
Please create a issue or better make a PR if you think that the benchmark methodology is wrong or the setup is wrong
Do not take the results as granted and conduct your own benchmarks to see if in the context of your application the results differ

Thanks and enjoy!

New go event sourcing library named incata

Sotirios Mantziaris — Fri, 28 Mar 2025 19:29:41 +0000

Event sourcing is capturing all changes of an application state as a sequence of events.
Since we only store events we only have to add events to a store, in contrast to updates for keeping the application state.
A much simpler model that scales very well. When needing the application state we just aggregate the events into

More on this can be read all over the internet but two excellent links are available below:

The library has support for the following RDBMS's:

Microsoft SQL Server
PostgreSQL

incata is very easy to setup and to use like illustrated on the github page.
Any ideas or improvements are highly welcome. Enjoy!

Initial release of adaptlog

Sotirios Mantziaris — Fri, 28 Mar 2025 19:29:40 +0000

Almost every application logs data one way or another. There are a plethora of logging packages available for golang.

There is the one that comes with the standard packages which takes a simple approach.
There are many logging packages that follow the well established leveled approach, and there are really a lot of them.

The decision of choosing a specific library comes with the cost of a direct dependency.
But why should we depend directly on a specific package?
How painful is it to exchange a logging package for another when we already created a lot of code with a direct dependency?

This is the reason why apaptlog came to life.

apaptlog is just a logging abstraction, which itself does not implement any log related stuff.
A logger has to be provided by the developer using any of the previous mentioned logging packages or any custom implementation.
The developer has to implement only a logging interface (standard or leveled), configure it at application start and use the abstraction throughout the code.
This is hardly something new. There are many libraries in other languages that do exactly this.

apaptlog is very easy to setup and to use. Follow the sample for the standard logger implementation.
Any ideas or improvements are highly welcome. Enjoy!

dotNETZone.gr meetup .NET core presentation

Sotirios Mantziaris — Fri, 28 Mar 2025 19:29:39 +0000

The presentation was about the current state of .NET Core in the Linux World.
Here are the slides of the presentation
and the github repository.

Thanks for attending.

The Repository and Unit of Work pattern

Sotirios Mantziaris — Fri, 28 Mar 2025 19:29:38 +0000

Yes, i know not this again. Is this not the one millionth time that someone blogs about that?
Yes, yes and yes but…
It is always good to repeat things and we all know that

“Repetition is the mother of learning, the father of action, which makes it the architect of accomplishment.” ― Zig Ziglar

There are still implementations out there that might benefit from this…

So let’s start with some definitions.

Repository

Quoting Martin Fowler’s Definition:

A system with a complex domain model often benefits from a layer, such as the one provided by Data Mapper (165),
that isolates domain objects from details of the database access code. In such systems it can be worthwhile to build another layer of abstraction
over the mapping layer where query construction code is concentrated. This becomes more important when there are a large number of domain classes or heavy querying.
In these cases particularly, adding this layer helps minimize duplicate query logic. A Repository mediates between the domain and data mapping layers,
acting like an in-memory domain object collection. Client objects construct query specifications declaratively and submit them to Repository for satisfaction.
Objects can be added to and removed from the Repository, as they can from a simple collection of objects, and the mapping code encapsulated by the Repository
will carry out the appropriate operations behind the scenes. Conceptually, a Repository encapsulates the set of objects persisted in a data store and the operations
performed over them, providing a more object-oriented view of the persistence layer. Repository also supports the objective of achieving a clean separation and
one-way dependency between the domain and data mapping layers.

Reading different sources (MSDN The Repository Pattern,
Martin Fowler: Repository etc) about the repository pattern the following properties emerge:

It maps between Domain Objects and Data objects
It does not expose the data layer to the outside world
It consolidates all data access patterns in one place thus help with code deduplication
It has a single responsibility
It is simple to implement
It has a one way dependency between the domain and the data layer

A simple example is the following application repository(C#):

public interface IApplicationRepository
{
    Task DeleteAsync(int id);
    Task<ApplicationModel> GetAsync(int id);
}

By providing an interface we can leave the implementation up top the developer to choose the data access library they wish.
The argument and return values of this interface should be domain specific objects and not the data objects to avoid spilling the data
into other layers and have a clean separation.

By using the above i had the chance to change the underlying implementation with anything i wished to experimented with.
First everything was EF, then Simple.Data then Dapper etc. You could even mix and match any of the above since every implementation
in the end will use a SqlConnection. It is really easy to change the underlying implementation.

You may think that changing the implementation happens not that often (migrate from EF to Dapper or from nHibernate to EF or Dapper etc)
but it can happen and is a really cheap abstraction over your data layer implementation. It further promotes clean separation which is always something worth doing.
This allows the application to not depend directly on the data access library and allows for future change with little cost.
For example if you have a application that uses nHibernate, which was maybe a good choice in the past, you are missing out some things
that other ORM provide like async calls or even the new .Net Core which may or may not happen for nHibernate. Dapper and EF already have the above.

The implementation of the interface does need something in order to work with the data layer. This can be a SqlConnection, DbContext (EF), Session (NHibernate) etc.
This will be injected to each repository and will generally be implemented in the Unit of Work.

Unit of Work

Quoting Martin Fowler's Definition:

A Unit of Work keeps track of everything you do during a business transaction that can affect the database.
When you're done, it figures out everything that needs to be done to alter the database as a result of your work.

So the UoW (Unit of Work) is responsible for keeping the db object (SqlConnection, DbContext) and handling the final commit in order to persist everything to DB.

A simple interface (C#) that has to be implemented is the following:

public interface IUnitOfWork : IDisposable
{
    IApplicationRepository Applications { get; }
    Task CommitAsync();
}

This is just a wrapper around our db object (SqlConnection, DbContext etc) and the implementation of the commit.
When we have a UoW we have at our hands all the necessary repositories, so interacting with them is really easy.

EF baked implementation and usage

Now we have the the following implementation for the application repository

public class ApplicationRepository : IDataAccess<ApplicationDbModel>, 
                                        IApplicationRepository
{
    public ApplicationRepository(DbContext dbContext, IMapper mapper) : 
        base(dbContext, mapper)
    {
    }

    public Task DeleteAsync(int id) => base.DeleteAsync(id);

    public async Task<ApplicationModel> GetAsync(int id)
    {
            var application = await GetAll()
                        .Where(p => p.Id == id)
                        .SingleOrDefaultAsync();
            return mapper<ApplicationModel>(application);
    }
}

Where the base repository is a EF implementation of the following interface:

public interface IDataAccess<T> where T : class
{
    IQueryable<T> GetAll();
    Task<T> GetByIdAsync(params object[] keyValues);
    void Add(T entity);
    void Update(T entity);
    void Delete(T entity);
    Task DeleteAsync(params object[] keyValues);
}

It is fairly easy to implementing another data access library. A dapper implementation of the application repository has
as constructor parameter a SqlConnection and the actual implementation of the interface methods. That’s it.

The unit of work implementation is the following:

public sealed class UnitOfWork : IUnitOfWork
{
    private readonly IMapper _mapper;
    private DbContext _dbContext;

    public UnitOfWork(DbContext dbContext, IMapper mapper)
    {
        _dbContext= dbContext;
        _mapper = mapper;
    }

    public Task<int> CommitAsync()
    {
        return _dbContext.SaveChangesAsync();
    }

    public IApplicationRepository Applications => 
        new ApplicationRepository(_dbContext, _mapper);

    //Implement IDisposable
}

This is a simple implementation of the UoW. Do not mind that some features are missing like transaction handling
(DbContext.Database.BeginTransaction() and then commit or rollback) a repository factory etc which are fairly easy to implement.

And how is this used?

Let’s assume we have a Unit Of Work Factory implemented so the code would be:

using (var uow = _unitOfWorkFactory.Create())
{
    var application = await uow.Applications.GetAsync(1, 1);
    await uow.Applications.DeleteAsync(application.Id);
    await uow.CommitAsync();
}

Easy and clean, isn’t it? Everything is in one place, at the end it get’s committed and properly disposed.
Since EF exposes the connection through the DbContext we can actually use Dapper also and have a mixed data access layer repository
in order to handle some hotspots where EF does not play well.

Conclusion

The repository and the unit of work patterns are fairly easy to implement. They provide a proper data access abstraction and expose only the needed domain object
and do not spill the data object into the upper layers. The only thing needed in order to use this is to inject the unit of work factory and we have our db in our hand.
Hope this is helpful. Any comment, discussion or fix is highly welcome.

Tripping the circuit

Sotirios Mantziaris — Fri, 28 Mar 2025 19:29:37 +0000

This is probably one of the most useful "cloud" patterns out there and it is fairly easy to implement.

There are great articles and implementations, like Polly,
already on the internet about this pattern so why another one?

Κρείττον οψιμαθή είναι ή αμαθή.

Socrates 469-399 BC., Philosopher

Which translates to:

Better too have learned lately than never, as he tried to explain why he learned to play
guitar in his old age.

I have learned better by reading, implementing and writing about something so stick with me.

Problem

Almost every application communicate with other services or resources, and they fail...

The reasons:

slow network connections
timeouts
the resources being over-committed or temporarily unavailable
buggy release
etc

When this happen our system becomes unstable, unreliable, brittle and failures cascade.

Let's go with an example of a failing remote service, let's say we have the following scenario

the remote service times out after 60sec
our service gets 30 req/s
the usual response time is 1s
each request takes up 1MB of RAM

What happens in our application?

The current request takes up resources in order to make the call and blocks (or awaits) for 60sec
All requests for the same service will suffer the same fate
Almost 1800 requests will be waiting for response at the end of the first 60s
Almost 1800MB of RAM is used up at the end of the first 60s
All clients that call our service fail in the same way
The failure cascades
The response times go through the roof and will be 60s for each request due to the timeout
The SLA, we might have, will be breached
The perfect storm is about to form

The above is a simplified example but is not that far fetched.

Solution

Using a circuit breaker can improve the stability and resilience of our application.
The circuit is actually a state machine with 3 states

Closed, meaning the execution will proceed
Open, meaning the execution will not proceed and throw a exception or return an error (implementation specific)
Half-Open, meaning that some executions are allowed after some time in order to check the remote service and open or close the circuit based on the response

For deeper knowledge on the pattern please read the following excellent articles

Benefits

By using a circuit breaker we have the following benefits:

Fail in a controlled manner
Fail fast
Save server resources
Maintain response times (SLA)
Handle failures differently when the circuit opens eg
- Redirect to another resources
- Save for later retry
- etc

Implementation

There are two implementation of the circuit breaker.
They share the same philosophy but are written in C# and Go.
Both implementations have a setting provider interface which can be implemented
in order to get the settings from anywhere. There is a in-memory settings implementation which
holds the settings in memory.
Both implementation are key based which means that for every key
the implementation provides a separate circuit which is actually the state.
The following setting exist for each key:

Failure Threshold at which the circuit opens
Retry Timeout defines after how much time after the circuit trips will the state be half-open
Retry Success Threshold defines after how many successful retries will the circuit reset and Closed
Max Retry Execution Threshold defines how many retries are allowed in the half-open state

The C# implementation can be found @ clouddotnet.
The implementation is generic, asynchronous and thread safe and lock-free.

The Go implementation can be found @ gocloud.
The implementation is idiomatic and "goroutine" safe.

Diffs

Since Go does not have generics the action to be executed returns a interface and a error so type casting is necessary

Epilogue

My blog is hosted in github so for any change, improvement or fix
you can either open a issue or submit a pull request.

The same goes for both implementations.

TT || !TT aka To TAP or not to TAP

Sotirios Mantziaris — Fri, 28 Mar 2025 19:29:36 +0000

Ramblings on Tasks, async-await and ASP.NET. The slides from the meetup
do contain common issues encountered when working with TPL, async-await and ASP.NET along with some good practices. These are off course not all. Please let me know if you have any suggestions. Thanks for attenting.

Setup a blog with Hugo and Github Pages

Sotirios Mantziaris — Fri, 28 Mar 2025 18:55:30 +0000

It was long my desire to write a blog with stuff that interests me.

Lately i was studying Golang and i came across Hugo which is a really nice and fast site generation utility.

This was a great opportunity to start my own blog by using Hugo and Github Pages in order to host it. Why?

it's free
it's Github
it's easy and fast

This is a walk through on how you can have a blog easy, fast and free! Let's start! The only thing you need is:

a Github Account
a Domain (optional)

Steps

The following steps are needed for the initial setup and creation of the blog:

Github repository for source code of the blog
Github Pages repository for the generated site
Setup Hugo
Create blog
Publish blog to Github Pages
Generate content and publish
(Optional) Setup sub domain to point to blog

1. Github repository for source code of the blog

Create a repository (public or private).

2. Github Pages repository for the generated site

Follow the instruction on Github Pages to create a repository with your Github username. Clone it to your local drive.

3. Setup Hugo

Download Hugo to your local drive. Unpack it to a folder and set the path in your OS to the executable. Almost all OS are supported!!!

4. Create blog

Create a folder for your blog source code and cd into it.
Execute hugo new site .
Execute git init
Add as remote repository the repository created in Step 1. (git remote add origin https://github.com/{username}/{repository}.git)
Add .gitignore file to exclude the path public/, which is the default directory of the generated static files
Execute git add .
Execute git commit -m "initial commit"
Execute git push -u origin master

Please refer to Hugo's documentation for generating content, using themes etc.

5. Publish blog to Github Pages

When we are ready to deploy our blog we do the following:

Execute hugo -d {path}, where path is the cloned repository path from step 2
cd into the above path
Execute git add .
Execute git commit -m "initial commit"
Execute git push origin master

After this we can enjoy our newly created blog under http://{username}.github.io where username should be replaced with your Github's username.

6. Generate content and publish

After our initial commit we can now generate more content and publish it (Step 5).

7. (Optional) Setup sub domain to point to blog

Let's assume you have a sub domain blog.domain.com. The only thing you need to do is to setup a CNAME entry in your DNS configuration and point it to {username}.github.io, add a file with name CNAME and content blog.domain.com to the root folder of the repository created in step 2, commit and you're done.

Conclusion

With these six (the seventh is optional) easy steps we have created a fully functional and fast blog. The cherry on top: for free!!!

(If we used public repositories in Github). And if we assume that almost everybody nowadays has a domain it is free even with the 7th step!

Happy blogging!