DEV Community: Bnaya Eshet

Proactive Architecture Guarding

Bnaya Eshet — Sat, 04 May 2024 19:46:01 +0000

Architecture Guarding with Static Code Analyzer and Code Fix

Ever worked on a codebase where the original architecture seems like a distant myth?

In large organizations, it’s easy for software architecture to drift as documentation becomes outdated and implicit knowledge fades. This wastes valuable knowledge and makes onboarding new developers a nightmare.

The Problem of Architecture Drift

In medium to large organizations, software architecture tends to drift away from its intended design over time. Documentation like wikis gets neglected or forgotten. The cognitive load on developers’ minds is too high to remember all architectural decisions. New developers are often unaware of the intended architecture.

Using Static Code Analysis integrated into the IDE & CI pipelines is the remedy.

Static Code Analysis

Static code analysis is the process of analyzing source code to detect potential issues, vulnerabilities, or deviations from coding rules and best practices without actually executing the program. It helps enforce quality standards and architectural guidelines early in the development lifecycle.

There are different types of static analyzers ranging from simple linting tools to sophisticated data flow analyzers that can detect complex coding issues, security vulnerabilities, and architectural violations.

.NET Roslyn Analyzers

For .NET projects, Roslyn Analyzers leverage a deep understanding of the .NET codebase to identify architectural violations.
These analyzers can dissect the code’s syntax tree and scrutinize code declarations to uncover even subtle architectural rule drifts. This fine-grained analysis makes Roslyn Analyzers, coupled with Roslyn Code Fix, incredibly powerful tools. By enforcing architectural best practices and automatically correcting common mistakes, they significantly improve the system quality. Furthermore, integrating seamlessly with developer IDEs like Visual Studio and VS Code shortens the issue-fixing cycle. Developers receive real-time feedback as they write code, allowing them to address problems before they reach the code repository. This proactive approach minimizes the number of issues that make it into the codebase, ultimately leading to cleaner, more maintainable software.

Integrating Static Code Analysis into CI/CD

Once you’ve established your architectural rules within static code analyzers like Roslyn Analyzers, the next step is to integrate them seamlessly into your CI/CD pipeline. This ensures that architectural violations are caught early in the development process, preventing them from reaching production.

Several popular tools facilitate integrating static code analysis into your CI/CD workflow. Here are a few examples, including both dedicated static code analyzers and platforms that manage them:

Static Code Analyzers:

Roslyn Analyzers (for .NET): As mentioned previously, offer a powerful way to enforce architectural best practices specifically for .NET projects. They integrate directly with developer IDEs and can be configured to fail builds in CI/CD pipelines upon detecting violations.

CI/CD Integration Platforms:

SonarQube: This popular platform goes beyond basic static code analysis. It aggregates the results from various static code analyzers, including Roslyn Analyzers, providing a consolidated view of code quality across your entire codebase. SonarQube integrates seamlessly with most major CI/CD tools and allows you to set quality gates within your pipelines. Builds will fail if these quality gates, which can include architectural compliance metrics, are not met.
GitHub Actions, Jenkins, Azure DevOps: While these are primarily CI/CD platforms, they offer extensive plugin support for integrating various static code analyzers. These plugins allow you to configure analysis jobs within your pipelines and often integrate with platforms like SonarQube for centralized reporting and quality gate management.

By choosing the tools that best suit your CI/CD platform, development environment, and desired level of analysis depth, you can establish a robust system for enforcing architectural compliance throughout the development lifecycle. This proactive approach ensures that your software adheres to its intended design, leading to a more maintainable and reliable codebase in the long run.

Complementary Tools for a Holistic Code Quality Approach

Static code analysis backed into the IDE & C/CD pipelines are foundational elements for enforcing architectural best practices within your .NET projects. However, a well-rounded code quality strategy encompasses a broader range of considerations. Here are some complementary tools you can integrate with your CI/CD pipeline to address various aspects of code quality:

Security-Focused Tools:

SAST (Static Application Security Testing) tools: These tools analyze code to identify potential security vulnerabilities.
SCA (Software Composition Analysis) tools: These tools scan your codebase for known third-party library vulnerabilities. Check this blog post for deeper insights.

By incorporating these tools alongside static code analysis, you can significantly enhance your security posture.

Best Practices and Code Style Enforcement:

NDepend: offers advanced code analysis capabilities, including code metrics, dependency visualization, and rule-based code inspections. It integrates seamlessly with CI/CD pipelines for .NET development.

By incorporating these complementary tools alongside static code analysis and SonarQube, you can establish a comprehensive approach to code quality that safeguards your .NET projects from security vulnerabilities, promotes code maintainability, and fosters a well-structured codebase.

Let’s Get Our Hands Dirty

Don’t be intimidated by the idea of creating your own static code analyzer , it’s more approachable than it may seem at first glance. The process is relatively simple.

Create a new Analyzer with Code Fix (.NET Standard) project, and you'll have a working sample right off the box.

The real challenge lies in translating your analysis logic into Roslyn syntax, but even that can be made easier with the help of tools like GitHub Copilot. While Copilot won’t provide a complete, ready-to-use solution, it can guide you through the process and suggest code snippets to build upon. You’ll likely still need to refine and debug Copilot’s suggestions, but it can streamline the development process significantly.

So don’t hesitate — roll up your sleeves and dive into creating your very own static code analyzer! With the right tools and a bit of persistence, you’ll be analyzing code like a pro in no time.

Conclusion: Building a Strong Code Foundation

Enforcing architectural best practices and fostering a culture of code quality are essential for building secure, maintainable, and well-structured software. By leveraging static code analysis and CI/CD guards like SonarQube, plus a strategic selection of complementary tools, you can establish a robust system for safeguarding your .NET projects throughout the development lifecycle.

This proactive approach yields significant benefits:

Reduced Architectural Drift: Explicitly defined architectural rules baked into the development process prevent deviations from the intended design.
Enhanced Code Security: SAST and SCA tools proactively identify and mitigate security vulnerabilities.
Improved Code Maintainability: Consistent code style, best practice enforcement, and code duplication reduction lead to cleaner, more maintainable code.
Faster Development Cycles: Catching issues early in the development process minimizes rework and accelerates delivery timelines.

By prioritizing code quality, you empower your development team to deliver high-performing, reliable applications that meet the evolving needs of your organization, and ship faster.

Getting Started with Event Sourcing and EventSourcing.Backbone

Bnaya Eshet — Mon, 29 May 2023 05:28:10 +0000

Getting Started with Event Sourcing and EventSourcing.Backbone

This post is the first of a series about event sourcing and an exciting framework called EventSourcing.Backbone.

In this post, we’ll explore the fundamentals of event sourcing and a Hello World sample of the framework.

Understanding Event Sourcing

Event sourcing is an architectural pattern that captures and persists state changes as a sequence of events. It provides a historical log of events that can be used to reconstruct the current state of an application at any given point in time. This approach offers various benefits, such as auditability, scalability, and building complex workflows.

Event sourcing, when combined with the Command Query Responsibility Segregation (CQRS) pattern, offers even more advantages. CQRS separates the read and writes concerns of an application, enabling the generation of dedicated databases optimized for specific read or write needs. This separation of concerns allows for more agile and flexible database schema designs, as they are less critical to set up in advance.

By leveraging EventSourcing.Backbone, developers can implement event sourcing and CQRS together, resulting in a powerful architecture that promotes scalability, flexibility, and maintainability.

You can check this article in order to learn more about event sourcing.

Introducing EventSourcing.Backbone

One notable aspect of EventSourcing.Backbone is its unique approach to event sourcing, instead of inventing a new event source database, EventSourcing.Backbone leverages a combination of existing message streams like Kafka, Redis Stream, or similar technologies, along with key-value databases or services like Redis.

This architecture enables several benefits. Message streams, while excellent for handling event sequences and ensuring reliable message delivery, may not be optimal for heavy payloads. By combining key-value databases with message streams, EventSourcing.Backbone allows for message payload to be stored in the key-value database while the stream holds the sequence and metadata. This approach improves performance and facilitates compliance with GDPR standards by allowing the splitting of the personal data aspects of the messages into different keys easily.

It’s worth noting that EventSourcing.Backbone currently provides an SDK for the .NET ecosystem. While the framework may expand to other programming languages and frameworks in the future, at present, it is specifically focused on the .NET platform.

Leveraging Existing Infrastructure

One of the major advantages of EventSourcing.Backbone is its compatibility with widely adopted message streaming platforms like Kafka or Redis Stream. These platforms provide robust message delivery guarantees and high throughput, making them ideal for handling event streams at scale.

Furthermore, EventSourcing.Backbone seamlessly integrates with popular key-value databases such as Redis, Couchbase, or Amazon DynamoDB. This integration allows developers to leverage the strengths of these databases for efficient storage and retrieval of auxiliary data related to events.

Behind the scenes

EventSourcing.Backbone is taking a somewhat different approach. It defines messages contract via interfaces (rather than data classes). This approach is friendly for versioning and easier for handling diverse message types. It can also help with GDPR by segregating personal data into different method parameters. In a nutshell, a message type build-out of each method parameter in a way that is transparent to the developer.

Each producer method call gets serialized as key-value pairs based on the method parameters (by default) and stored in the key-value storage (Redis Hash in our case). Then the metadata of the message is pushed into the stream (Redis Stream in our case).

The consumer is listening to the stream and reconstructing the message into a method call right into the subscriber interface.

The framework is capable of handling different message patterns and different strategies, more on this will be posted in future posts.

Let’s have a taste of it.

Prerequisites:
In order to run the sample you need

Docker Desktop (for hosting REDIS)
.NET 7 SDK (or higher)

Get Redis up & running:

docker run -p 6379:6379 -it --rm --name redis-event-source-sample redislabs/rejson:latest

Create a solution with 3 projects (Producer, Consumer, Common Abstraction). or clone the following repo as a starting point:

git clone https://github.com/bnayae/HelloEventSourcing

Defining mutual interest codes like the common contracts:

Add the following NuGet packages into the project of the common abstraction:

The first step is to define the events schema.
“EventSourcing.Backbone” is taking the approach of defining an interface for the event’s data structuring.

Put the following code into the event abstraction project.

using EventSourcing.Backbone;
namespace EventSourcing.Demo;

[EventsContract(EventsContractType.Producer)]
[EventsContract(EventsContractType.Consumer)]
public interface IShipmentTracking
{
    ValueTask OrderPlacedAsync(User user, Product product, DateTimeOffset time);
    ValueTask PackingAsync(string email, int productId, DateTimeOffset time);
    ValueTask OnDeliveryAsync(string email, int productId, DateTimeOffset time);
    ValueTask OnReceivedAsync(string email, int productId, DateTimeOffset time);
}

The EventsContract attribute is where the magic begins.
By decorating the interface with EventsContract _a compile-time source generator will generate the code necessary for producing and consuming events (you don’t have to write boilerplate code).
And in a result will generate the code of the _IShipmentTrackingProducer interface for the producer and IShipmentTrackingConsumer for the consumer.

Note: all methods should return ValueTask (this is the convention), yet void _will be fine but will result in _ValueTask _on the generated version of the interface (producer/consumer). furthermore, all generated methods will have the _Async suffix.

Creating a producer:

Add the following Nuget for using a Redis-based producer (using Redis Stream + Redis Hash).

EventSourcing.Backbone.Channels.RedisProducerProvider And don’t forget to add a reference to the common abstraction project we created earlier.

using EventSourcing.Backbone;
using EventSourcing.Demo;

IShipmentTrackingProducer producer = RedisProducerBuilder.Create()
                                    .Uri("hello.event-sourcing")
                                    .BuildShipmentTrackingProducer();

User user = new(1, "someone@gmail.com", "someone");
var product = new Product(1234, "Something you must have", 10000);
await producer.OrderPlacedAsync(user, product, DateTimeOffset.UtcNow);

That's all you need to have a working producer.

Take a look at BuildShipmentTrackingProducer which is an extension method generated from the IShipmentTracking interface and returns an IShipmentTrackingProducer .

Creating a consumer:

Add the following Nuget for using a Redis-based consumer (using Redis Stream + Redis Hash).

EventSourcing.Backbone.Channels.RedisConsumerProvider And don’t forget to add a reference to the common abstraction project we created earlier.

Add a class that implements the consumer (generated) interface:

using EventSourcing.Demo;

class Subscription : IShipmentTrackingConsumer
{
    public static readonly Subscription Instance = new Subscription();

    public ValueTask OrderPlacedAsync(
                        User user, Product product, DateTimeOffset time)
    {
        Console.WriteLine($"Order Placed: {user.name}, {product.name}, {product.id}");
            return ValueTask.CompletedTask;
    }

    public ValueTask PackingAsync(string email, int productId, DateTimeOffset time)
    {
        Console.WriteLine($"Packing: {email}, {productId}, {time}");
        return ValueTask.CompletedTask;
    }

    public ValueTask OnDeliveryAsync(string email, int productId, DateTimeOffset time)
    {
        Console.WriteLine($"On Delivery: {email}, {productId}, {time}");
        return ValueTask.CompletedTask;    
    }

    public ValueTask OnReceivedAsync(string email, int productId, DateTimeOffset time)
    {
        Console.WriteLine($"On Received: {email}, {productId}, {time}");
        return ValueTask.CompletedTask;   
    }
}

The final step is to attach this class to the event stream.

using EventSourcing.Demo;
using EventSourcing.Backbone;

IConsumerLifetime subscription = RedisConsumerBuilder.Create()
                                          .Uri(URIs.Default)
                                          .SubscribeShipmentTrackingConsumer(Subscription.Instance);

await subscription.Completion;

If you recall the producer was sending a single event of OrderPlacedAsync,
To complete the picture we’ll add a combination of consumer & producer which will change the state along the shipping flow.

Create a new project and add both the Redis-based producer & consumer NuGets:

Add an implementation of IShipmentTrackingConsumer.

using EventSourcing.Backbone;
using EventSourcing.Demo;

class Subscription : IShipmentTrackingConsumer
{
    public static readonly Subscription Instance = new Subscription();

    private readonly IShipmentTrackingProducer _producer = RedisProducerBuilder.Create()
                                .Uri(URIs.Default)
                                .BuildShipmentTrackingProducer();


    public async ValueTask OrderPlacedAsync(User user, Product product, DateTimeOffset time)
    {
        await _producer.PackingAsync(user.email, product.id, DateTimeOffset.Now);
    }

    public async ValueTask PackingAsync(string email, int productId, DateTimeOffset time)
    {
        await _producer.OnDeliveryAsync(email, productId, DateTimeOffset.Now);
    }

    public async ValueTask OnDeliveryAsync(string email, int productId, DateTimeOffset time)
    {
        await _producer.OnReceivedAsync(email, productId, DateTimeOffset.Now);
    }

    public ValueTask OnReceivedAsync(string email, int productId, DateTimeOffset time)
    {
            return ValueTask.CompletedTask;
    }
}

This way we can handle the state’s transition.

The last step is to plug it into the stream:

using EventSourcing.Demo;
using EventSourcing.Backbone;

Console.WriteLine("Consuming and Producing Events");

IConsumerLifetime subscription = RedisConsumerBuilder.Create()
                                                .Uri(URIs.Default)
                                                .Group("transit")
                                                .SubscribeShipmentTrackingConsumer(Subscription.Instance);

await subscription.Completion;

Notice the Group(“transit”) this one is important for separating the consumer group which deals with the state transition from the one that reports the current state.

Consumer groups distribute the workload of processing messages among multiple consumers, allowing parallel processing while ensuring each message is handled only once. This ensures efficient and reliable message processing, scalability, and fault tolerance when dealing with large volumes of data or high-throughput message streams.

Putting both consumers in the same consumer group (or under the default one) will result in messages either handled by the transit or by the reporter and lead to a mess.

The full code is available at:
https://github.com/bnayae/HelloEventSourcing/tree/HelloWorld
notice that it is under the HelloWorld branch

Conclusion

In this introductory post, we’ve explored the concept of event sourcing and introduced the unique aspects of EventSourcing.Backbone. This framework stands out by leveraging a combination of message streams and key-value databases to achieve better performance, support GDPR standards, and provide flexibility in event-sourcing implementations.

EventSourcing.Backbone integrates seamlessly with popular message-streaming platforms and key-value databases, enabling developers to leverage their strengths for scalable and efficient event sourcing.

When combined with CQRS, event sourcing can enhance database schema design, making it more agile and flexible, and enabling the generation of dedicated databases optimized for specific needs.

In future posts, we’ll drill down to more advanced patterns, concerns, pros & cons, and best practices.