DEV Community: Thai Anh Duc

Observation - Watch Out Boundaries

Thai Anh Duc — Sun, 23 Jun 2019 05:07:01 +0000

Original posted on my blog June 23, 2019

When I first started my software development career I think writing software was hard. And it is true. However, the definition of writing software at that time was different. What I really meant was my code met the functional requirement (and it was not always true) and ran. That was when I did not see in production. So everything was working fine in my machine.

Over the time, I have had chances to bring my code into production and seen them running. No surprise that they did not work well in production. Every developer knows the famous "It works well in my machine". And it might work well in test. But it always has problems in production.

Why? There is no single answer for that problem. Software are developed by developers with different level of skills, experience, intelligent, ... Even the team has the most talented developers in the world, their products still have bugs. So I am not trying to find a solution for that problem. Instead, I embrace and observe the fact. There is no silver solution but there are tips and tricks to prevent as much as possible.

From my own experience, from Pluralsight courses, from youtube, ... from any source I touched, I want to document what I have observed. I do not intend to go in the detail of each item. Rather I want to have a list and some explanations, references. The detail varies project by project.

SQL

If you build applications with SQL Server as data storage, watch out 2 common unbounded patterns

Select N+1
Select * without top n

Asking Google for "Select N+1" you will know what it is immediately. There are detail explanation with code example. Usually developers that have worked in ORM know it very well.

The second watch out is a bit tricky. That is when your applications issue a query to SQL in this pattern

SELECT * FROM dbo.Employee WHERE [Predicate]

In the test environment, there is no problem. However, in the production, the data is huge and that query might return millions of records.

These days many applications do not talk to the database directly. Instead there are ORM/LinqToSQL in between. And this piece of code is not uncommon

var employeesByName = ctx.Employees.Where(x => x.Name.StartWith("Smith")).ToList();

Some ORMs might put a limit on the generated queries. What developers need to do is to review all the generated queries.

The rule of thumb is that always control the number of returned records. Put the max on everything.

Connections

There are many kinds of connections that applications make - connect to the database, connect to external services. When making such those calls, there are some watch out

Timeout: Make sure a timeout value is set on everything. Usually the modern frameworks have default values. Just make sure there is and you are aware of them.
Close connections properly. Just imagine what happens if you have your door opened? Bad things happen.

JSON - Serialization and Deserialization

JSON is cool. Developers work with JSON everyday in one form or another. Many take it for granted. We rarely pay attention to the size of data. I once experienced such a problem here - hidden cost of an architecture.
So if you have to explicitly use JSON directly in your code, ask these questions

Do I have to use it? Is there any other options?
What size? Is the size under controlled?

Some might argue that RAM is cheap. So why should we care too much about the size? Yes. RAM is cheap but it has limit. Once it reaches the limit, your application will freeze or crash. And if your applications are running on the cloud, everything your applications consume, there is cost involved.

Enumerable, List, ForEach

Does developer write code without the usage of loop? Have we ever wondered how many items are there in a list? When talking about a list, we should be aware that all items are stored in memory. So the size really matters here even it is trivial.
Another trap is at the Enumerable. Enumerable represents a sequence that mostly expected to iterate only once. By nature, we do not know the size of a sequence (There is no Count property on the IEnumerable interface). Therefore, when calling .ToList(), be aware of all the nasty things can happen.

I brought them here for reference. It might not a thing that takes down production. But it is nice to be aware of as well.

There might be more about boundaries to watch out. Those are what I have come up so far. What's yours?

C# 7 Tuple Better Test Assertion

Thai Anh Duc — Sat, 20 Apr 2019 01:38:34 +0000

Original posted on my blog April 18, 2019

Recently I read C# In Depth 4.0 where I met the new Tuple design in C# 7. It is really a cool feature. Besides the syntax sugar, it offers capacities that developers can leverage.

At the time of reading it, I was tasked with writing unit tests in my job. It triggered my memory about Semantic Comparison with Likeness. The main idea of semantic comparison is to compare 2 objects with certain properties. It allows developers to define what equality means. The tuple supports equality by default. So maybe I should be able to use the tuple to accomplish the same thing as Likeness.

In this post, I will write a simple unit test without Likeness or tuple, then refactors it with Likeness, finally uses Tuple. Let's explore some code.

public class Product
{
    public Guid Id { get; set;}
    public string Name { get; set;}
    public double Price { get; set;}
    public string Description { get; set;}
}

[TestFixture]
public class ProductTests
{
    [Test]
    public void Test_Are_Products_Same()
    {
        var expectedProduct = new Product
        {
            Name = "C#",
            Price = 10,
            Description = "For the purpose of demoing test"
        };

        var reality = new Product
        {
            Name = "C#",
            Price = 10,
            Description = "For the purpose of demoing test"
        };

        // Assert that 2 products are the same. Id is ignored
    }
}

The task is simple. How are we going to assert the 2 products?

Old Fashion

Very simple. We simply assert property by property.

[TestFixture]
public class ProductTests
{
    [Test]
    public void Test_Are_Products_Same()
    {
        var expectedProduct = new Product
        {
            Name = "C#",
            Price = 10,
            Description = "For the purpose of demoing test"
        };

        var reality = new Product
        {
            Name = "C#",
            Price = 10,
            Description = "For the purpose of demoing test"
        };

        Assert.AreEqual(expectedProduct.Name, reality.Name);
        Assert.AreEqual(expectedProduct.Price, reality.Price);
        Assert.AreEqual(expectedProduct.Description, reality.Description);
    }
}

Some might think that we should override the Equals method in the Product class. I do not think it is a good idea. The definition of equality between production and unit test are tremendously different. Be careful before overriding equality.

The product class has 3 properties (except the Id property). So the code still looks readable. Think about the situation where there are 10 properties.

Likeness - Semantic Comparison

There is a blog post explaining it in the detail. In this demo, we can rewrite our simple test.

[TestFixture]
public class ProductTests
{
    [Test]
    public void Test_Are_Products_Same()
    {
        var expectedProduct = new Product
        {
            Name = "C#",
            Price = 10,
            Description = "For the purpose of demoing test"
        }.AsSource()
        .OfLikeness<Product>();

        var reality = new Product
        {
            Name = "C#",
            Price = 10,
            Description = "For the purpose of demoing test"
        };

        Assert.AreEqual(expectedProduct, reality);
    }
}

Likeness is a powerful tool in your testing toolbox. Check it out if you are interested in.

Tuple - Customized

The idea is that we can produce a tuple containing asserted properties and compare them. This allows us to flatten the structure if wished.

[TestFixture]
public class ProductTests
{
    [Test]
    public void Test_Are_Products_Same()
    {
        var expectedProduct = (Name = "C#", Price = 10, Description = "For the purpose of demoing test");

        var reality = new Product
        {
            Name = "C#",
            Price = 10,
            Description = "For the purpose of demoing test"
        };

        Assert.AreEqual(expectedProduct, (reality.Name, reality.Price, reality.Description));
    }
}

It might not look different from the Likeness approach. And I do not say which approach is better. It is just another way of doing things.

Summary

So which approach is better? None of them. Each has its own advantages and disadvantages. They are options in your toolbox. How they are used depends on you, developers. Definitely, I will take advantages of the new Tuple in both production and unit test code.

The Process of Making Elegant Unit Tests

Thai Anh Duc — Fri, 12 Apr 2019 04:45:49 +0000

Original posted on my blog April 12, 2019

Unit tests are part of the job that developers do while building software. Some developers might not write unit tests. But, IMO, majority does. If you are one of those, how do you treat the unit test code comparing to the production code?

Do you think about maintainability?
Will you refactor the test to make it better? Note: I use the term refactoring from the Refactoring book by Martin Fowler.
Have you used the advantages that the test framework offer?

To be honest, I had not thought about that much. In the beginning of my career, I wrote tests that followed the current structure of the projects. I did not question that much. Over the time, I started to feel the pain so I made changes Unit Test from Pain to Joy which has served me well in that project.

Write Tests

Recently, I have had a chance to work on another project. I was tasked with writing unit tests (and integration tests) to get used to the system and to be able to run tests with multiple credentials, AKA login user.

Here is the test, not a real one of course. But the idea is the same. I want to run the test with different credentials. The username and password must be passed to the parameters. This is very useful when you look at a test report. By parameterizing the report will show values passed to the test.

[TestFixture]
internal class MultiCredentialsTest
{
    [Test]
    [TestCase("read_user", "P@ssword")]
    [TestCase("write_user", "P@ssword")]
    public void RunWithDifferentCredentials(string username, string password)
    {
        // The test body goes here.
    }
}

And there will be many of them. It worked as expected. But there are potential problems. Can you guess the problem?

What happens when one of test users is changed either username or password?
What if we want to add more test users into the test suites?

Think about the situation where there are hundreds even thousands of them. It will be a pain. I need a solution to centralize the test data. My process has started.

Make Them Better - Manageable

It was time to look at what NUnit offers. NUnit has supported TestCaseSource. You should check it out first if you have not known it. In the nutshell, it allows developers to centralize test data in a manageable manner. That was exactly what I was looking for.

I created a TestCredentialsSource to produce the same test data. I would prefer the name TestCredentialsFactory, but seems the "source" fits better in the unit test context.

internal class TestCredentialsSource
{
    public static  IEnumerable<object> ReadWriteUsers = new IEnumerable<object>{
        new object[]{"read_user", "P@ssword"},
        new object[]{"write_user", "P@ssword"}
    }
}

The test was rewritten with version V1. There are 2 versions for comparison.

[TestFixture]
internal class MultiCredentialsTest
{
    [Test]
    [TestCase("read_user", "P@ssword")]
    [TestCase("write_user", "P@ssword")]
    public void RunWithDifferentCredentials(string username, string password)
    {
        // The test body goes here.
    }

    [Test]
    [TestCaseSource(typeof(TestCredentialsSource), "ReadWriteUsers")]
    public void RunWithDifferentCredentials_V1(string username, string password)
    {
        // The test body goes here.
    }
}

In the test, I did not have to deal with test values. The test data was encapsulated in the TestCredentialsSource with the ReadWriteUsers static field.

Make Them Even Better - Reuse and Duplication

It was good with known specific set of users. There were certain tests that want to run with a specific user. It should be fairly easy with another property in the TestCredentialsSource

internal class TestCredentialsSource
{
    public static  IEnumerable<object> ReadWriteUsers = new IEnumerable<object>{
        new object[]{"read_user", "P@ssword"},
        new object[]{"write_user", "P@ssword"}
    }

    public static   IEnumerable<object> SpecificUser = new IEnumerable<object>{
        new object[] {"special_user", "P@ss12345"}
    }
}

What if I wanted to test with only "read_user"? What if I wanted to combine the "read_user" with "special_user" for another test? One option was to define them in the TestCredentialsSource. Which was still fine because it was still manageable in a single file. But it was awkward.

Was there any better alternative?

Yes, there was. Let's encapsulate the data in a class. Welcome to TestCredentials class.

internal class TestCredentials
{
    public string Username { get; }
    public string Password { get; }
    public TestCredentials(string username, string password)
    {
        Username = username;
        Password = password;
    }
    ///<summary>
    /// Convert the object into an array of properties object which can be used by the TestDataSource
    ///</summary>
    public object[] ToTestSource() => new object[] { Username, Password };

    public static TestCredentials ReadUser = new TestCredentials("read_user", "P@ssword");
    public static TestCredentials WriteUser = new TestCredentials("write_user", "P@ssword");
    public static TestCredentials SpecialUser = new TestCredentials("special_user", "P@ss12345");
}

The class supplied 3 factory methods to construct the needed credentials. This was the only single place where the data was provided without any duplication.
The TestCredentialsSource became much cleaner

internal class TestCredentialsSource
{
    public static  IEnumerable<object> ReadWriteUsers = new IEnumerable<object>{
        TestCredentials.ReadUser.ToTestSource(),
        TestCredentials.WriteUser.ToTestSource()
    }

    public static   IEnumerable<object> SpecificUser = new IEnumerable<object>{
        TestCredentials.SpecialUser.ToTestSource()
    }
}

Cool! The data has gone from the source definition. But there was still one thing that I did not like much - the setup of "SpecificUser" in the TestCredentialsSource. Having a source for a single value did not sound right to me.

There was a solution - convert the TestCredentials to a source that NUnit can understand. Implement the IEnumerable<TestCaseData>. TestCaseData is defined by the NUnit framework

internal class TestCredentials : IEnumerable<TestCaseData>
{
    public string Username { get; }
    public string Password { get; }
    public TestCredentials(string username, string password)
    {
        Username = username;
        Password = password;
    }
    ///<summary>
    /// Convert the object into an array of properties object which can be used by the TestDataSource
    ///</summary>
    public object[] ToTestSource() => new object[] { Username, Password };

    public static TestCredentials ReadUser = new TestCredentials("read_user", "P@ssword");
    public static TestCredentials WriteUser = new TestCredentials("write_user", "P@ssword");
    public static TestCredentials SpecialUser = new TestCredentials("special_user", "P@ss12345");

    public IEnumerator<TestCaseData> GetEnumerator()
    {
        return new List<TestCaseData>
            {
                new TestCaseData(Username, Password)
            }.GetEnumerator();
    }

    IEnumerator IEnumerable.GetEnumerator()
    {
        return GetEnumerator();
    }
}

With the in place, I could write the 2 below tests. There was no limitation of what combinations I could make of test credentials.

[Test]
[TestCaseSource(typeof(TestCredentials), "SpecialUser")]
public void RunWithDifferentCredentials_SpecialUser(string username, string password)
{
    // The test body goes here.
}

[Test]
[TestCaseSource(typeof(TestCredentials), "SpecialUser")]
[TestCaseSource(typeof(TestCredentials), "WriteUser")]
public void RunWithDifferentCredentials_CombinedUsers(string username, string password)
{
    // The test body goes here.
}

The End

The good result does not come by accident. I could have stopped at any step in the process. By pushing a little bit further, by asking the right questions, the end result was more than I had expected.
If you are writing tests code, should you look at them again and ask questions? Give it a try and see how far it takes you.

Avoid Potential Problems with Explicit API Design

Thai Anh Duc — Sat, 06 Apr 2019 10:46:34 +0000

A year ago, I wrote Leaky Abstraction - Linq Usage. I use the design whenever I meet the same challenge. The system works as expected. I am happy about the design.

Until recently, there was a need to order the collection. Say that we need to order a TeacherCollection by years of experience. Assuming that there is a requirement to print the result in the years of experience order.

It is a pretty simple requirement. In the TeacherCollection constructor, this code will do the job.

public TeacherCollection(IEnumerable<Teacher> teachers)
        {
            if (teachers != null)
                _teachers = teachers.Where(x => x.IsStillAtWork).OrderBy(x => x.StartedOn).ToList();
        }

Everything works as expected.

Boom! The system is very slow when having more data. The profiler shows that a high number of time spent on the ordering. In that system, the ordering logic is much more complicated. It is not a pure Linq sorting as in the example. Still, the sorting cannot be a problem. That is for sure.

The problem is that the collection is accessed too many times. It is also an expected result because the collection is designed to filter data, to work in a pipeline in a safe way.

The ordering logic should not be placed here. The collection itself has all the information to do the sorting, filtering.

What should we change in term of the design to solve the problem and also support the sorting?

Identify Responsibilities

In my opinion, this is the most difficult part of writing code. I have not found any exact formula to get it right. Identifying responsibilities is a heuristic. Experience matters here.

Filtering and ordering should be treated as two separated responsibilities. It is very easy to mix them in one implementation and thus error-prone. When defining responsibility, one should consider at least 2 factors

The purpose of each: One is for filtering, the other is for sorting. They are 2 different operations.
When it is used and the usage frequency. Filtering is used a lot to extract sub collection from the original collection. Ordering is, on the other hand, only used when a final result is displayed to the end user or other form of presentation such as Console screen, word document.

It is kind of tricky to see them as separated responsibilities. In many case I even not bother to think about it. Well, It proves that I was wrong. Sometimes, it sounds cool and simple if just order the list.

Extract Explicit Interfaces

Before moving on, let's take a look at the TeacherCollection. The additional feature we need is the ability to get exact index of a teacher.

ITeacherCollection - The default interface is extracted from the current TeacherCollection. The School now holds an instance of ITeacherCollection, instead of TeacherCollection implementation. This refactoring step will not break anything.

IIndexedTeacherCollection - A simple interface which supply only GetIndex API. A key point is that a consumer cannot instantiate it. The only way to have this API is a transition from ITeacherCollection.BuildIndex.

The sorting cost is paid only whenever a need arises.

The actual implementation of the improved design is almost identical with the original version. All the major implementation logic is there in the TeacherCollection class. The refactoring is safe because the compiler will tell us what goes wrong.

The client (Program class) only deals with interfaces and interfaces transition.

Why didn't I have the ITeacherCollection at the first time? Well, we have not needed it at that time. We should not make thing complicated if there is no demand. Design is evolved.

I just solved a bug beautifully.

Watch Out Abstract Dependency

Thai Anh Duc — Sat, 06 Apr 2019 04:07:23 +0000

The context is a C# .NET Core WebAPI system. The system employs the Onion Architecture. There is an API layer which is ASP.NET MVC controller. Then there is an Application Service layer inside the API layer. In the core of the onion, it is the domain. One can quickly google for Onion Architecture. If you do not know it, I suggest you take a look at it first.

One of a key points in the Onion Architecture is the inner layer must not know the outer layer. Which translates into the C# project that the Application Service Layer must not reference directly to the API project. The API consumes the Application Service NOT the other way around.

There is a requirement that application configuration (note that it is not system configuration like connection string, certificates ...) should be maintained in a JSON file. Let's call it appConfig.json. The file is deployed with the API. The implementation takes the advantage of .NET Core JSON configuration so the runtime can load the file.

Let simplify the requirement to focus on the architecture. The appConfig defines the default currency. The value is used to display in the UI and for other processing in the Application Service.

Come to the tricky part. Where do we should place the implementation: API or Application Service?

My answer is always in the API. What if the implementation is in the Application Service? Let's see the implementation and analyze.

By using the IConfiguration interface, the Application Service has a dependency on Microsoft.Extensions.Configuration package which seems fine. Because it is abstraction.

Has the dependency between API and Application Service changed? Implementation-wise NO. Because there is no direct reference from Application Service to the API.

But there is in term of design. The Application Service implementation is using the infrastructure supplied by the API - the configuration is managed by the API layer.

Will it cause any problem? Well, it depends on what we care most about. If the architecture is the main concern, then yes, it is a problem. The architecture is broken unintentionally. Think about a scenarios where the Application Service is by another client, such as WPF application. Does it make sense to bring the Microsoft.Extensions.Configuration package to WPF application?

The Application Service defines the interface, which says "hey! I need to know the default currency, but I cannot figure it out by myself." The outer layers, can be a WebAPI, can be WPF application, will supply the implementation, which says "Not a problem! I know where to get the default currency for you."