DEV Community: NewDay Technology

The Backend Testing Breakup 💔

Jay Glass — Mon, 16 Dec 2024 16:23:39 +0000

Or: How to maximize the value tests provide

Preamble

Ah, testing. If you’re like me and have a love-hate relationship with tests, this article is for you.

Having the right testing at the right point in the development process can be a lifeline and prevent costly bugs getting into production, but test suites also have the tendency to grow into unwieldy monsters.

Have you ever worked on a project where understanding and updating the tests was more complex than implementing the actual (money-making) business logic? If not, I guarantee at some point you will.

But it doesn’t have to be this way.

Tests can be simple and manageable, and my hope in writing this article is to help steer testing in a less painful direction for everyone involved - myself especially.

Goals of this article

To suggest a battle-tested way of making testing as painless as possible.

To provide a way to implement this form of testing without impeding feature development.

To cover the importance of buy-in and how to get it.

But first

Let’s not forget the goal of testing: to give us confidence that our system will behave as expected in production.

Any test we write should support this goal.

So…

How do we make testing as painless as possible?

With these simple principles:

Test for each thing as early as possible

Only test for each thing once

But to dig into these principles we need to cover (and establish common language for) the levels of testing.

Levels of testing

I am yet to work at a company that doesn’t have at least one form of testing named differently from everywhere else I’ve worked.

We generally all agree on what a unit test is but if some of the other names differ from what you’re used to, please bear with. I’ve even half-jokingly suggested coming up with new, less-opinion-loaded names for the various levels of testing but I usually get shot down because “there are existing names already” - and everyone then spends the rest of the session debating what those existing names' responsibilities actually are…🤦‍♀️ so for now let’s just go with what appears to be most common.

The test pyramid trophy tetrahedron hierarchy

The test pyramid/trophy/(insert shape here) is great but it does tend to direct focus to the point that there should be more tests at one specific level rather than some other specific level. Which is often translated as “level x should have more tests than level y”. In the Pyramid paradigm, it means writing more Unit Tests than Component Tests, more Component Tests than Contract Tests, and so forth all the way up.

#1

Which detracts from the point: We shouldn’t be focusing on the amount of tests. We should be focusing on the practicality. And this is where “Test each thing as early as possible” comes in:

If some logic can be verified in a Unit Test, we should do so in a Unit Test.

Because these test single things, they have single points of failure. A failing Unit Test tells us immediately what unit of logic failed which makes it extremely fast to find and fix.

As we go up the testing levels, the amount of moving parts increases so if we skip Unit Tests completely and something breaks, did it break because of an internal logic issue? Or an integration issue? A network issue?

Sure you can start debugging or trawl through logs but if you could, why not have the Unit Test to tell you exactly which bit of logic the failure came from?

And the same applies as we go up the levels: we shouldn’t be verifying integrations or contracts in Smoke Tests, putting them in the earlier levels means we can catch issues in a more isolated manner making the root cause easier to identify.

A failing Contract Test takes you directly to the contract that is causing a problem. If we skip Contract Testing and a Smoke Test fails, we need to do a lot more digging to get to the information which tells us it is a contract causing the issue.

So, when adding some new functionality, we should (repeat with me) “test each thing as early as possible”.

#2

And if we have a Contract Test verifying that a contract is as expected, we don’t need to do so again in an Integration Test. Some third-party dependencies may not even need any Integration Tests at all, especially if all we are concerned with testing is that given a specific request format, we get a specific response format.

We may find that maintaining the additional Integration Test gives us no value.

The same goes for Component Tests: we shouldn’t be testing any logic. That should already be done in the Unit Tests. So if a Component Test fails, we know we’ve already proven the internal logic so it must be something else - like a configuration or internal dependency registration issue.

If we start asserting logic in Component Tests, contracts and integrations in Smoke Tests we are duplicating our assertions, crossing the responsibilities of our levels of testing, and creating more test code to understand, maintain and update leading to an eventual monster of a test suite that developers will treat with an appropriate level of dread.

So mantra #2: Only test for each thing once

Defining the test types

…or each level of tests' responsibilities

To help illustrate this we have the following scenario:

A web API for a Zoo which allows zookeepers to:

add an animal, which will result in the animal being added to the database and an email notification being sent.
change the feeding time of an animal, which will result in the time being updated in the database and an email notification being sent.

For the tests, following the mantra of “Only test for each thing once”, we get the following breakdown:

Unit Tests

These are done on the smallest possible unit of logic. We are only concerned with validating that the unit of logic behaves as expected given different inputs. To ensure we are only testing our logic and not any dependencies, dependencies are mocked to return an expected response to help us assert the unit of logic’s behavior for a specific scenario.

These tests are usually written against the public functions defined in business logic.

Example Test Cases

AddAnimal returns success when the mocked AnimalRespository reports that the record has been created.

AddAnimal returns an AnimalAlreadyExists error when the mocked AnimalRepository reports that the animal already exists.

AddAnimal returns success when the mocked EmailClient reports it successfully sent the email.

AddAnimal returns an InvalidEmail error when the mocked EmailClient only forwards the email request to the mocked EmailClient if the call to the mocked AnimalRepository returned successfully.

etc

and similar for the ChangeFeedingTime function of the FeedingTimeService.

💰 Tip: Keeping functions/units-of-logic small results in smaller, more manageable and easier to understand Unit Tests.

✅ Now we know that each individual piece of business logic works as expected.

Component Tests

The next step is to test that all these functions can be chained together and execution flows through the running application as expected.

Since we are only concerned with testing the execution flow, we can mock external dependencies (the Database and Email Web API).

We have also already tested all the logic within the various functions so we don’t need to check any of those again either.

All we assert is that after entering the system, execution flows through to the correct external dependencies and back resulting in the expected response.

Example Test Cases

When an add animal request is sent to the API, the animal is added to the database, an email is sent and we get a success response.

When an add animal request is sent to the API but the animal already exists, we get an “animal already exists” error response.

etc

✅ Now we know that when pieced together, execution flows through the individual pieces of business logic as expected.

💰 Tip: If you need to provide a mock of your entire application: these mocked external dependencies can be used in place of the actual implementations (e.g. by clever in-memory injection) and viola: you have a version of your application which has all the logic, requires almost no additional maintenance, and isn’t beholden to external dependencies causing trouble.

Contract and Integration Tests

We now know our application works when the external dependencies behave as expected - but how do we confirm that the external dependencies behave as expected? This is what Contract and Integration Tests do.

Contract Tests

These ensure that the external dependencies contracts conform to the schema that we expect.

For example: We would have tests which send the Email Web API the various requests we use, to ensure they are accepted with the fields and their values in the format in which we are sending them.

This means that if the Email Web API were to suddenly make a field mandatory that we are not supplying, the Contract Tests would start to fail and we would know there is something we need to change.

Example Test Cases

Email Contract Tests

When an email request is sent with all the required properties, we get a success response.

When an email request is sent with a body greater than 5000 characters, we get an error response stating the body is too large.

Database Contract Tests

When an add animal request is sent with all the required properties, we get a success response.

When an add animal request is sent without a name, we get an error response stating the animal name is required.

etc

Integration Tests

Similarly, Integration Tests ensure the external dependencies behave as we expect.

E.g. The Database may have a rule which prevents Junior Zookeepers from setting feeding times. In this case we can have an Integration Test which asserts that if a Junior Zookeeper tries to change a feeding time, the database returns an error.

Example Test Cases
Email Integration Tests

When an email request is sent, a test email account receives the email.

When an email request is sent which is identical to a previous request, we get an error response stating the email is a duplicate.

Database Integration Tests

When a Senior Zookeeper updates the feeding time, we get a success response.

When a Junior Zookeeper updates the feeding time, we get an error response stating permission denied.

etc

✅ Now we know that the application behaves as expected and the external dependencies behave as expected.

Smoke Tests

So what’s next? Well, when we deploy our application it may be running on different hardware, with different configuration, network rules and restrictions, and who knows what else.

Smoke Tests help us identify any major issues which haven’t and can’t be caught by the previous levels of testing.

The usual convention is to pick a few of the mission-critical features that touch on different dependencies. Because we know the application behaves as expected in all other regards, we can keep the amount of tests in this level to very few. This has the bonus of saving us test maintenance headaches as smoke testing is often the most time-consuming and complex, both in functionality and time to set up.

Example Test Cases
When an add animal request is sent to the API, we receive a success response.

Note: We could create another test case:

When a feeding time update request is sent to the API, we receive a success response.

But the first test case already touches both the database and the email dependency so for this example, the one is sufficient.

✅ Now we know that the application behaves as expected, the dependencies behave as expected, and when deployed, the application can start up, dependencies can be reached and some of the mission-critical features run as expected.

To summarize

We know our logic works thanks to Unit Tests.

We know execution flows through the application correctly thanks to Component Tests.

We know external dependencies conform to the contracts we expect thanks to the Contract Tests.

We know external dependencies behave as expected thanks to the Integration Tests.

And now we know that the application runs in the various environments thanks to the Smoke Tests.

Why does this work?

Because each individual piece has been tested, and we’ve tested the individually tested pieces work together, an we’ve tested that the external pieces behave as our individually tested pieces expect them to, and we’ve tested that our entire system still runs as expected in deployment environments.

Levels of testing conclusion

We’ve got a clear split in the responsibilities of the various levels of testing.

We have a common set of names to use for the various test levels.

We know exactly what to test in each level.

We know to test anything in the earliest possible level.

We know not to test for anything that has already been proven in an earlier level.

And the result is

Happy Developers 🥳 Need I say more?

…but in case I do:

A much more manageable test suite with simple tests that are easy to understand, update and debug.

A clear idea of how to implement testing when picking up a new task.

A common testing paradigm to use across backend teams.

And the real money saver: Quicker feedback on issues with lower levels of testing highlighting issues sooner.

What about other forms of testing?

To reduce the scope of this article non-functional forms of testing have been intentionally omitted.

Performance, security, scalability etc are all important however they differ greatly and have different levels of relevancy depending on the particular project. In my experience, the functional forms of testing tend to be bigger time sinks as they are run and updated far more frequently - so the focus is on the area with the biggest potential for improvement - the functional testing that forms part of the typical testing done for every code change from the developer’s machine to production - as part of the typical software development lifecycle.

Let’s not overload ourselves by trying to do absolutely everything in one go.

Putting into practice

Before this article suffers the same fate of so many before it and gets lost in an abundance of browser tabs, if you see value in the concepts - why not do something about it?

Implementation Guide

The problem with a lot of tech improvements is that they’re seen as blockers. Drop everything and work on this improvement which will prevent us from working on any new features or business as usual for some amount of time.

This may work for smaller tasks than improving an entire, mature test suite - but for large tasks, this may and should have the business quaking in their boots at the prospect of falling behind on delivery schedules.

But there is a way to develop your cake and eat it:

The strangler pattern

Following the concept of the programming pattern, instead of updating the entire test suite in one go, we can methodically update tests as we touch on them as part of the day-to-day software development process.

Decide on and agree a direction for the test suite across the team.
When a particular area of code is worked on, as part of that work update any of the tests that code uses to conform to the new direction.

This allows the migration of tests, bit by bit, without having a massive impact on the day-to-day commitments to features and business-as-usual development tasks.

AND means any new tests created, are created following the new direction - so don’t add to the mess that needs to be migrated later.

After a few months of this, Engineering Improvements tasks can be raised to migrate the still outstanding tests. These can be small enough to slowly pick away at without affecting the team’s cadence much.

And before you know it, you’ll be reaping the benefits of a smoother, quicker and easier test suite.

Getting buy-in

But before doing anything, you’ll need to first convince your team and then convince the people who decide how your team spends its time that this is a worthy endeavor:

For the team:
- if the current test suite is painful enough that you’re reading this article - that’s the job done. Otherwise one might suggest that this is a way to ensure tests don’t get to that point of pain.

💰 Tip: Make this one of your or your team’s official objectives to get additional visibility and support within your company.

For the business:
- increase feature delivery by reducing the overhead/cost of developing new features due to the time spent updating unwieldy tests
- maintain and improve client confidence with the software reliability increase due to improved testing
- this will also make for happier developers which means less developer churn - less costly onboarding
- if multiple teams adopt this, the company will benefit from the knowledge-share across teams and the cohesion that comes from having a shared testing paradigm.
- and can be done without additional cost if done with a strangler approach - which is to only move testing to the new way of doing so when the test code for that specific feature needs to be modified anyway

Fin

And there you have it. The lessons I’ve learned (and what many others have shared with me) over the years to help take testing from a brittle behemoth to a simple, structured, manageable, set of utilities to give us confidence in our system.

We’ve covered:

The levels of testing
Testing each thing as early as possible
Only testing for each thing once
How to migrate old tests without costing the company
How to get understanding and buy-in from the company

Thanks for reading and happy testing! 🤖

Estimating the RU throughput requirements for Azure Cosmos DB and predicting cost

Alessio Franceschelli — Tue, 30 May 2023 15:42:26 +0000

Azure Cosmos DB is a globally distributed, multi-model database service designed to scale seamlessly to handle variable workloads.

One key aspect of Azure Cosmos DB's performance and scalability is its usage of Request Units (RUs). In Cosmos DB, RU throughput is important because it determines how many operations a database can perform per second. This means that by varying it, you can scale the performance of a database up or down, ensuring that your application can handle the expected workload and maintain the desired level of performance. Of course, this will also influence the cost you are billed for your database.

Understanding Request Units (RUs)

Request Units (RUs) are a measure of the resources needed to perform a particular operation in Cosmos DB. Each database operation in Cosmos DB, such as reading, writing, or querying data, requires a specific number of RUs.

RUs are a fundamental part of the Cosmos DB pricing model. They determine the amount of throughput capacity that is required to perform a given set of operations on a database. The more RUs a database requires, the higher the cost of the database.

RUs are used to abstract the complexity of underlying hardware and software from the user. Instead of worrying about hardware and software configurations, users can simply specify the required RUs for their workloads, and Cosmos DB will handle the rest.

To estimate the required RU throughput for Azure Cosmos DB, it's important to consider the data operations that will be performed, the size of the data being accessed, and the performance characteristics of the database. The number of RUs required for a given data operation depends on the complexity of the query, the amount of data being accessed, and the consistency level of the database. In addition, the size of the data being stored, and the access patterns can affect the required RU throughput. For example, larger datasets may require more RUs to perform operations efficiently, and high-throughput workloads may require more RUs to ensure that the database can keep up with demand. By carefully considering these factors, users can estimate the required RU throughput for their specific workloads and ensure that their Cosmos DB database is provisioned with the appropriate amount of resources to handle their needs.

We will now explore some common Azure Cosmos DB operations and their associated RU costs to give you a better understanding of how RUs affect your database performance and cost.

Reading a single document

Reading a single document by its ID and partition key is advertised as the solution that consumes the lowest number of RUs, making it the most efficient read operation in terms of RU usage. However, the size of the document drives the RU consumption and, for large documents, it will become more onerous that a query operation.

For example, reading a 1KB document will consume 1 RU, while reading a 100KB document will consume 10 RUs. Indexing has understandably no impact on RUs consumption of these operations.

Creating a Document

Creating a new document in Azure Cosmos DB typically consumes a larger number of RUs than reading it, making it not particularly cost effective for write heavy scenarios. The RU cost for this operation depends on the size of the document and the amount of indexed fields.

For example, without any indexed field, writing a 1 KB document, will consume 5.5 RUs, while a 100KB document will consume 55 RUs. As you can see, as the document grows the cost scale the same as with reads.
Indexing has also an important impact, as inserting a 1KB document, with all fields indexed, which is the default behaviour, would consume about 16 RUs.

Updating a Document

Updating a document consumes about double the RUs compared to creating a new document, however the RU consumption is potentially less affected by indexes as not necessarily all indexed fields are updated.

For example, without any indexed field, updating a field in a 1 KB document, will consume about 10 RUs, while in a 100KB document will consume 100 RUs.

Sample RU consumptions

This table show measured RU charges for different operations on a sample 1Kb and 100Kb documents. Of course, there would be variation based on the actual shape of your documents.

	Insert	Point Read	Upsert	Delete
1KB Document No Index	5.5	1	10.3	5.5
1KB Document All fields indexed (default)	16.2	1	10.7	16.2
100KB Document No Index	48.8	10	96.7	48.8
100KB Document All fields indexed (default)	59.4	10	97.1	59.4

Queries RUs consumption

When looking into the RUs consumption of queries, there are many different variables at play, which makes it more difficult to predict the performance and cost of performing the queries required to support your application behaviour.

Indexes

In Azure Cosmos DB, indexes play a vital role in optimizing the performance of read and query operations. While indexing can increase the RU consumption of write operations, such as inserts and updates, the benefits of using indexes often outweigh the associated costs.

One of the main advantages of using indexes in Azure Cosmos DB is the substantial improvement in query performance. Indexes enable the database engine to quickly locate and retrieve documents based on specific attribute values. Without indexes, Cosmos DB would need to perform a full scan of the data, which is much slower and consumes more RUs. By using indexes, you can efficiently run complex queries with multiple filters, sorts, and joins, resulting in a significantly reduced query execution time and lower RU consumption.

It’s important to note that the number of total documents doesn't affect RUs consumption when running a query on an index in Cosmos DB because the query is executed on the index. However, the number of documents returned by the query does affect the RUs consumption, as it determines the amount of data that needs to be read and returned.

Cosmos DB offers automatic index management, which means that the database engine automatically maintains indexes for all properties in your JSON documents by default. This eliminates the need for manual index creation and maintenance, simplifying your database administration tasks. Automatic index management also ensures that new properties added to documents are indexed automatically, making your data model more flexible and adaptable to changes in your application. However, as we had seen previously, indexes come at a cost, so if you don’t need to perform queries on all fields, you should optimize your indexing policy to balance RU consumption and query performance.

Although indexing might increase the RU consumption of write operations, the benefits of faster query execution, automatic index management, and flexible indexing policies often outweigh the associated costs. By carefully perfecting your indexing strategy, you can strike the right balance between performance and cost efficiency in your Azure Cosmos DB deployment.

Choosing the partition key

The choice of partition key in Azure Cosmos DB plays a crucial role in the overall performance and scalability of your database. A well-chosen partition key not only helps distribute your data evenly across multiple partitions but also affects the RU consumption of various operations.

Cross-partition queries

One of the main factors influencing RU consumption is the type of queries your application needs to perform. Queries that can be resolved within a single partition typically consume fewer RUs compared to queries that require scanning multiple partitions (cross-partition queries). By selecting a partition key that aligns well with your most common query patterns, you can minimize the number of cross-partition queries and reduce RU consumption.

Hot partitions

An inappropriate partition key choice may lead to a scenario where a single partition receives a disproportionately high amount of requests compared to others. Hot partitions can cause increased RU consumption and may result either in throttling, affecting your application's performance, or requiring you to increase the provisioned RUs of the whole database.
For this reason, to keep the RU consumption in check, it is critical to choose a partition key that evenly distributes the data and request load across all partitions.

Strategies for selecting an optimal Partition Key

While this is a big topic on its own, there are many resources available as it is not a problem specific to Azure Cosmos DB but a common requirement for most of the non-relational databases.
The best way to approach the problem is to first understand your application's most common query patterns and then choose a partition key that allows for efficient single-partition queries. Also consider the data distribution, select a partition key that ensures even distribution of data and request load across all partitions to prevent the problem of hot partitions.

Unfortunately, once a partition key is put in place, it is no longer possible to change it. However, this should not prevent you to continuously monitor your database's performance and RU consumption to identify potential issues with your partition key choice. Be prepared to migrate the data to a new container if the need to adjust your partition key strategy becomes necessary.

Sometimes, finding a partition key that works successfully in most scenarios is not possible and duplicating the data could unintuitively become the best option.

Leveraging data duplication for perfecting the partition key usage

There might be cases where a single partition key does not meet all query requirements efficiently. In such scenarios, duplicating data to another container with a different partition key can be a powerful strategy to optimize query performance and reduce RU consumption.

Duplicating data to another container with a different partition key can significantly improve the performance of queries that would otherwise require cross-partition operations. Cross-partition queries consume more RUs and have higher latency compared to queries that can be resolved within a single partition. The RU consumption of writing data to a new container could become negligible considering the saving on queries, depending on the volume of your operations.

Change Feed for data Synchronization

Azure Cosmos DB's Change Feed is a powerful feature that enables you to capture changes in your source container and replicate them to a target container with a different partition key. Change Feed ensures near-real-time data synchronization between the source and target containers, allowing you to maintain consistent data across multiple containers with different partition key configurations. This process is generally low effort and resilient, however it consumes RUs to read the data from the source container, on top of the obvious charges to write to the target container.

Sample RU consumptions

There is a sample RUs consumption of a query using indexes on a container with documents of 1KB.

Method	Items Returned	RUs consumed with 1KB documents	RUs consumed with 100KB documents
Point read	1	1	10
Query	1	2.8	4.5
Query	10	3.2	19.9

Based on the results of the sample test, the point read method appears to be the most efficient in terms of RUs consumption as it only consumes 1 RU for retrieving a single document. However, if the requirement is to retrieve multiple documents, then the query method becomes more efficient. It is important to note that the number of documents returned by the query has a significant impact on the RUs consumed. Therefore, when designing queries, it is important to consider the number of documents that will be returned and use indexing and query optimization techniques to minimize the RUs consumed.

In the last column we can see a test for larger documents that highlights an unexpected result due to a quirk of the Cosmos DB RU charges calculation. In fact, the point read are not actually the cheapest way to retrieve a single document as generally presented in the documentation, but for large documents retrieving items via query is actually cheaper.

Translating RUs into cost

Now that we have a clear picture of the RUs consumption of different operations, we are left with the fundamental question on how RUs translate to cost in Azure Cosmos DB, so that we can better optimize our databases for cost efficiency.

Azure Cosmos DB offers different pricing models, but the most relevant for this analysis are provisioned throughput and serverless.

Serverless is designed for workloads with variable or unpredictable throughput requirements, and you pay for the actual RUs consumed by your database operations instead of pre-allocating throughput capacity. It has, however, limitations on features availability and on how much your database can grow.

With provisioned throughput, you allocate a specific number of RUs per second to your database or container. This pre-allocated capacity determines the maximum throughput your database can handle at any given time. You are billed for the provisioned throughput, whether or not you fully utilize it.

Luckily, you can enable autoscaling on top of provisioned capacity, so that the provisioned throughput would instantly scale based on load, up to the specified maximum provisioned throughput and down to 10%. However, even if the scaling is near instantaneous, for each wall-clock hour you are charged for the maximum value reached of provisioned throughput in that hour, so supporting spiky workload can be expensive.

Impact of partitioning and storage amount

One of the limitations of Azure Cosmos DB is that the provisioned throughput of all partitions scale together at the same time. While this doesn’t represent an issue for well distributed workloads, if you have many partitions and experience spikes of load in a small subset of those partitions, your required provisioned throughput and consequently cost will be significantly larger than would be expected given the amount of operations.

For example, if you have a large database of 40TB of data, you will probably have about a thousand physical partitions, due to the constraints on partitions size. If your data is not particularly well distributed and you have a spiky workload, it could easily be that you have a bunch of operations happening at the same time in one of the partitions at least once per hour.
Or, alternatively, you could have an infrequent unoptimized query requiring to scan multiple documents and hence consuming lots of RUs. In these cases, it would not be surprising to have a spike of consumption of a couple thousand RUs on specific partitions. However, given all the partitions have to scale together, it would require a total provisioned capacity of a couple millions of RUs! If this happens every hour for just a second, you would be constantly paying for this high amount of RUs even if you are not actually performing that many operations on your database to justify this.

Impact of data consistency and high availability on cost

In distributed databases that rely on replication, there is a fundamental trade-off between the read consistency, availability, latency, and throughput. The choice of data consistency and high availability settings can also a substantial impact on the cost of running the database.

First of all, adding more regions will multiply the cost as you are essentially paying for each region where your data is replicated to, independently from the amount of operations served by it. In fact, Cosmos DB provisioned throughput is reflected on all region, so even regions not serving any traffic will incur the same cost as the active region.

Azure Cosmos Db also offer the ability to setup a database in multi-region write, which offers better availability, while sacrificing the data consistency, but this causes the cost to double. So, if for example we have configured a database to have 3 regions with multi-region writes enabled, we would be paying 6 times compared to using a single region.

Regarding of the different consistency models, when using the Strong Consistency or the Bounded Staleness models to achieve better consistency, to the detriment of write latency, read operations will consume double the amount of RUs as they require a local minority quorum.

One final consideration is regarding the use of availability zones to improve the availability inside a single region. This introduces a 25% increase in cost for that region.

It is important to carefully evaluate the trade-offs between consistency, availability, and cost to ensure an optimal and cost-effective Cosmos DB solution.

Estimating RUs requirements and predicting cost

Now that we understand the cost model of Azure Cosmos DB, we need to determine the amount of provisioned RUs that would be needed to support our applications, so that we can predict the running cost.

While there are many factors, as we discussed before, that influence how many RUs will be required to perform the operations happening every given second, if we managed to achieve a well-structured database, in particular with well-balanced partitions, we can obtain a pretty accurate figure focusing only on the main aspects: the amount of data, the size of the documents, the distribution of the different operations and the number of regions we are going to replicate the database to.

To quickly obtained an estimate based on this simplification, you can leverage the Azure Cosmos DB Capacity Planner which is provided for free by Microsoft to assist you in determining your RU needs based on your workload's characteristics.

You can access the Azure Cosmos DB Capacity Planner tool at the following URL: Azure Cosmos DB Capacity Calculator

The Capacity Planner tool requires you to input your workload details to provide a good estimate of RUs consumption and cost. It also provides the ability to define indicatively the difference in load between peak and quiet times, as well as to combine different workloads together. You will also need to specify the number of regions you are going to replicate your database to and if you intend to use multi-region writes which, has we had seen, has major cost implications.

The main omission in this tool is the lack of estimation of the cost of indexes, so your actual spend may vary based on how many indexes you have. As we have seen the major impact these have to the RUs consumption of create and update operations, so the variance can be significant for write heavy scenarios, which however are not ideal use cases for Cosmos DB.

The Capacity Planner tool provides a starting point for estimating your RU requirements. However, once you go live with your database, it's essential to monitor and adjust your provisioned throughput settings based on your actual usage patterns and performance metrics in a real-world scenario, and review your operations as needed.

Monitoring RUs usage

Effectively monitoring and tuning Request Unit (RU) usage in Azure Cosmos DB is crucial for maintaining optimal performance and cost efficiency.

Azure Cosmos DB provides various tools and metrics to monitor your RU usage. The Azure Portal provides built-in metrics to inspect the RU usage of your Cosmos DB account. Key metrics include provisioned throughput, total requests, average RU consumption, and 429 (Too Many Requests) responses that show throttling due to exceeding provisioned throughput. You can also configure Azure Monitor to collect Cosmos DB metrics and create custom dashboards, alerts, and reports to track your RU usage.

Furthermore, when executing operations in Cosmos DB via the SDK in your application, you can retrieve metrics from the response, which include information on RU consumption, query execution time, and retrieved document count.

If you need more detailed information, you can enable diagnostic settings in your Cosmos DB account to collect logs and metrics, and send them to a storage account, event hub, or Log Analytics workspace for further analysis and reporting. Please note that there is a cost involved.

Conclusion

In conclusion, this blog post has explored the significance of Request Units (RUs) in the context of Azure Cosmos DB, emphasizing their role in managing resources and controlling costs. We delved into the intricacies of query RU consumption, the translation of RUs into cost, estimation of RU requirements, and monitoring of RU usage.

The key takeaways from this post are:

RUs are the primary measure of resource consumption in Cosmos DB and understanding them is crucial for optimizing performance and cost.
The kind of operation and document size influence the RUs throughput requirements.
Query RU consumption varies depending on factors like query complexity, indexing policies, and data size.
Point reads are advertised as the most efficient way but in practice they consume more RUs than queries when dealing with large documents or where multiple documents need to be retrieved.
RUs directly impact cost; hence, it's important to have a thorough understanding of the relationship between RUs and pricing.
Estimating RU requirements and predicting costs is essential for budgeting and capacity planning.
Regular monitoring of RU usage can help in identifying bottlenecks, optimizing performance, and avoiding unexpected costs.

We encourage you to make use of the provided resources to calculate and optimize your own RU throughput requirements for Cosmos DB. By taking advantage of this knowledge and the Azure Cosmos DB Capacity Planner tool, you can ensure that your database operations are both efficient and cost-effective. However, don't hesitate to explore further and tailor your Cosmos DB setup to suit your unique requirements. Happy optimizing!

Easily making internal libraries debuggable in .NET

Alessio Franceschelli — Thu, 15 Dec 2022 14:51:02 +0000

Easily making internal libraries debuggable in .NET

At NewDay, we deeply care about software testing and, while we are big fans of TDD, there is no denying that being able to debug software effectively is a crucial aspect of software engineering.

Favouring a debugging session instead of writing a new unit test is a big topic on its own, so here we will only focus on a specific aspect: being able to step into an internal library from a consuming application or service in .NET.

.NET has a long history of providing a fully featured debugger via Visual Studio. When working with Microsoft libraries or many open-source ones, we get the ability to step into the code, how is that possible? And how we can achieve the same behaviour for our company internal libraries?

Leveraging debug symbols

In general, to debug a compiled language, on top of having access to the source code, you need the debug symbols, which map the binary to the original source code. In .NET, this role is fulfilled by the Portable Database (or PDB).

For Microsoft libraries or many open-source ones, the PDBs are automatically retrieved by Visual Studio from the Microsoft public symbols server or from the NuGet.org symbol packages (.snupkg) feed, respectively.

When working in an enterprise scenario, we don’t usually have either of those mechanism available for our internal libraries: symbols servers have never been popular and are tricky to setup. On the other hand, while snupkg are the standard for modern .NET libraries, they are not supported by the most popular software used to host internal package feeds, like JFrog Artifactory or GitHub Packages registry. For this reason, the best approach is to include the PDBs as part of our actual library NuGet package.

Including PDBs in a NuGet package

The common solution to provide debug symbols for private packages used to be to include the PDBs in the NuGet packages alongside the dll.

For example, in early versions of .NET Core you could add a, admittedly quite hard to remember, line to your csproj to achieve this.

Unfortunately, with changes in .NET Core 3.0 to the project system, it is no longer possible to consume the PDB files included in NuGet packages.

Luckly, we can rely on another mechanism that, while deeply different in nature, allow us to achieve the same result: embedding the PDB inside the DLL. It has a few drawbacks, like having larger packages and not being able to easily strip out PDBs from deployables, however these problems are rarely relevant for internal libraries, especially when used in internal microservices and APIs.

In current .NET versions, this is easily achievable by adding the <DebugType>embedded</DebugType> property to your library csproj, which is much easier to remember than the previous technique and one day it could become the default behaviour.

There is a final aspect that we need to mention when talking about including a PDB: making sure it is deterministic. While .NET builds have become deterministic by default in .NET, ensuring the same DLL is produced no matter where it is built, to have PDBs with normalized paths to our source file, we also need to add the true property, usually with a conditional to only apply the normalization when building in CI, but we will look into an easy way to achieve all these later on.

Getting the source code

Now that our library contains the required metadata to map the DLL to the source code, we need to be able to easily retrieve it. Nowadays we all use version control systems, so the easier way is to retrieve it directly from it. The .NET platform provides Source Link, a nifty tool integrated into Visual Studio, which provides the ability to automatically include metadata into NuGet packages that point to the source code hosted in our version control hosting platform of choice from which Visual Studio can then automatically download it when stepping into the code. Thanks to the support to multiple authentication mechanism as well as on-premises solutions like GitHub Enterprise, this works well in the enterprise scenario, allowing us to leverage the same tool as the opensource community without extra configuration, as easy as adding a package reference and a couple of project properties. But can it be even easier to integrate it?

Putting it all together

As we had seen, by putting a few pieces together and using the right configuration, we can provide a brilliant debugging experience for our internal libraries which has nothing to envy to opensource or Microsoft libraries. Of course, we would not want to have to replicate a bunch of settings across every project and, while creating shared props or targets is an option, there is no need for it.

In fact, there is a .NET Foundation’s package called DotNet.ReproducibleBuilds that takes care of all these steps necessary to get Source Link fully working and CI builds deterministic, including logic to detect your correct CI server. All you need to do is to add this package either to your csproj, or more convenient if you have multiple projects published as library in the same solution, to your Directory.Build.props file.

Investigating Source Link issues

If you are having issues in navigating to the source code, you should check the logs in Navigate to External Sources in the Visual Studio’s Output View.

Also make sure you have Enable Source Link support enabled and consider enabling the integration with Git Credential Manager if you experience authentication issues.

Furthermore, don’t forget that to be able to step into libraries, apart from thrown exceptions, you need to disable Just My Code and set the Symbols loading rules appropriately.

Extra goodies

In recent versions of Visual Studio 2022 there is now the ability to use Go To Definition on references of our internal libraries and magically go to the source code relying on the same mechanism previously described used during debugging. While the feature is still not perfect, it works in most scenarios, and it keeps getting refined with each feature update!

So easy!

Thanks to the ability of the IDE to automatically retrieve debug symbols and the source code, allowing both to "step in" during debug as well as using Go To Definition when navigating the codebase, working with private libraries has never been so easy!

Adopting SwiftUI

Olivier Rigault — Fri, 11 Jun 2021 16:53:32 +0000

Introduction

There has been a debate among the iOS developer community about adopting SwiftUI since Apple introduced this framework in 2019. Adopting a new technology is always risky, and comes with many challenges, but we still decided, at NewDay, to adopt this new technology very early in 2020.

This post explains the reasons why we made this decision, the challenges that we faced, and how we embraced SwiftUI in our iOS development process.

What is SwiftUI?

SwiftUI is an innovative, exceptionally simple way to build user interfaces across all Apple platforms with the power of Swift. Build user interfaces for any Apple device using just one set of tools and APIs. With a declarative Swift syntax that’s easy to read and natural to write, SwiftUI works seamlessly with new Xcode design tools to keep your code and design perfectly in sync.

Apple Inc.

SwiftUI is Apple’s take on Declarative UI Programming, and is a faster, cleaner and more interactive way to code UI on the iOS platform. It also comes with some caveats.

Pros:

Declarative syntax
Fast, clean and interactive
Works well with Combine
UIKit compatibility

Cons:

Supported only on iOS 13+
Limited list of UI components
Lack of support from the developer community
Tightly coupled views

Btw, what is Combine? 🤔

It would not be fair to talk about SwiftUI without mentioning Combine, which is Apple’s take on Reactive Programming. Like SwiftUI, Combine is only available from iOS 13+.

Combine is, by essence, very similar to SwiftUI, and therefore still perfectible. It is a strong contender to replace, in the next few years, other native reactive programming frameworks such as RxSwift or ReactiveSwift.

Combine can be used without SwiftUI, but SwiftUI is very dependent on Combine.

Is SwiftUI Ready for Production?

That's the big question. We at NewDay believed very early in 2020 that we could push SwiftUI code to production, and in doing so, would benefit our development and delivery processes.

Why did we adopt SwiftUI?

Our decision was based on several factors:

iOS 13 adoption rate
Pushing new features
Replacing web views by native code
Launching new applications
Adopting the latest technologies

High iOS 13 adoption rate

The adoption rate of new versions of mobile operating systems has always been much faster on iOS than on Android, and iOS 13 was no exception. Apple released iOS 13.0 on September 2019, and by January 2020, 80% of iOS users (worldwide) already adopted it.

We assumed that given time, this rate would go much higher. We originally thought about pushing our first SwiftUI code to production in September 2020, so very close to the iOS 14 launch date, but we postponed this to later in the year - December 2020.

In February 2021, more than 92.5% of our iOS users use our apps on iOS 14, and with about 4% using iOS 13. These numbers seem to confirm that our original assumption, which was shared with our stakeholders, was correct, and the lack of support of older versions of iOS was not a blocker for fully embracing SwiftUI at NewDay.

A New Feature - Aqua Coach

We decided to use SwiftUI to implement new features, whenever possible. The very first feature benefitting from this was Aqua Coach, a credit score functionality that we added in the Aqua Card app in December 2020.

A New Native Screen - Account Summary

Our mobile apps are currently partly hybrid. Some screens just use web views to display contents, which give users a poor user experience.

We plan to keep these web views as long as our apps support iOS 12 (or under), and present brand new native screens, coded in SwiftUI, for our users using iOS 13+ devices.

The new Account Summary will be the first of such screens, and will be available soon.

A New Application - bip

We have used SwiftUI and Combine to write our brand new bip iOS app. We originally planned to launch the app in September 2020, but decided to postpone to Q2 2021 instead. This decision gave our users, even more time, to adopt iOS 13 (and now iOS 14).

bip is an innovative product, a mobile app only service, targeting a younger generation of users, who, potentially, own recent phones running on the latest version of the operating system.

Using SwiftUI to code all the screens helped us tremendously as there were numerous screens in bip mobile app in terms of application forms.

bip is now available on the App Store and Google Play Store.

Adopting The Latest Technologies

NewDay is a tech-driven company. We want to adopt the best practices, the best technologies in order to deliver better, faster and create reliable apps. We also want to attract new talent who want to embrace these goals.

SwiftUI is surely one of the best example of such tools on the iOS platform.

The Future of SwiftUI at NewDay

We strongly believe that SwiftUI and Combine will play an important part in iOS development in the years to come. Even though we admit that these frameworks are still perfectible, there is no doubt that Apple will refine and improve them overtime.

Apple being Apple, it is also likely they will start to gradually deprecate UIKit pretty soon, and entice iOS developers to adopt SwiftUI instead.

Measuring performance using BenchmarkDotNet - Part 3 Breaking Builds

Tony Knight — Sat, 22 May 2021 00:25:13 +0000

Introduction

Previously we discussed the absolute bare minimum to run BenchmarkDotNet in your CI pipeline. Your code builds, benchmarks are taken, and you have to drill down into the numbers.

But what if bad code was committed? A small change sneaks in, probably under tight deadlines, that suddenly makes your nice fast code run like treacle?

How would you know about - and more importantly stop - such horrors? That's what we'll try and address in this post.

As before, we're only talking pure code here, that is, your class methods and algorithms. APIs, services, applications are much more complex, and we haven't considered I/O. So let's keep it simple and just focus on the performance of pure code.

What this post will cover

Discuss ways to stop builds upon degraded code performance
Installing tools in a sandbox environment
Collecting benchmark data for analysis
Analysing results and breaking builds

What you'll need for this post

.NET 7 SDK installed on your local machine
A BenchmarkDotNet solution
An IDE, such as VS, VS Code or Rider.

The simplest possible way to break a build...

...is, surprisingly, not a sledgehammer. It's even simpler than that.

For the vast majority of build platforms, to stop a build you normally need your script to return a non-zero return code. That age old trick is so simple and effective: it stops bad things dead in their tracks. So let's use that:

We want our benchmark analysis to return 0 on success and 1 on failure.

Easy! but this leaves us with a trickier problem.

How to detect performance has degraded?

You've got the stats from BenchmarkDotNet. You now need to monitor each build's performance results, or more accurately, detect deviance from accepted performance.

What is acceptable performance?

This is a very broad subject, and it's often difficult to put precise time limits on micro code performance. For much optimisation work, you'll be iteratively changing code so that performance should always improve with each commit. Therefore, you can stop optimising when the results are good enough.

So, as we often do not have absolute time requirements and we iteratively improve our performance as a matter of course, we'll take a broad view:

Accepted performance is the best recorded benchmark time

That leads us onto deviance.

Deviance from acceptable performance

Why do we want deviance and not absolutes? Because we cannot guarantee that repeated benchmark runs, even with a static codebase and the same infrastructure, will yield exactly the same time measurements over each iteration.

And as each build feeds many time-critical activities - user acceptance, security validation and the like - we don't want a tiny deviation to choke off this supply of new features.

How do we know what is an appropriate deviance is and how do we measure it? That's another very broad subject and depends entirely on your circumstances. For now, let's take a simple (& admittedly crude!) method just to illustrate the key point of stopping slow code getting into our main codebase.

Acceptable deviance, expressed as a percentage, falls in between [baseline measurement] < [new measurement] < [baseline measurement + deviance%]

Here we're simply allowing some slippage from the best recorded time.

Please remember: the subject is extremely broad and this article is just an introduction to the subject. But for now, the main take-away point is: whatever the current performance, keep improving it and never degrade!

It seems we need a tool for this

You could build your own, but here's something from our own stables: a dotnet tool to detect deviance in BenchmarkDotNet results:

NewDayTechnology / benchmarkdotnet.analyser

A tool for analysing BenchmarkDotNet results

BenchmarkDotNet.Analyser (BDNA) is a tool for iteratively collecting and analysing BenchmarkDotNet data. It's distributed as a dotnet tool, so you can use it locally and on almost any CI platform. You just need .NET 7 installed.

BDNA is in beta and we want to continually improve it. We welcome bug reports and feature suggestions.

Installing

The latest version is distributed via Nuget.

For the remainder of this section, I'll lead you through installing BDNA in a sandbox environment so if you do get into any problems you can simply destroy the directory and start again without any side effects.

Create a new sandbox

The sandbox will be a directory on your local drive.

We won't be pushing this directory to source control in this article. But the same steps are used in a cloned local repository.

mkdir c:\projects\scratch\tools
cd
cd projects\scratch\tools

Create a tools manifest

The tools manifest is simply a version list of the repo's tools, to ensure version consistency and stability: just like your own project's package dependencies. As we want these tools installed locally we'll create a new manifest in our directory:

dotnet new tool-manifest

Globally installed .NET tools are very convenient: you install it once on your machine and keep updating as necessary. But they place nasty dependencies on your build platform, and there's no guarantee your team members will use exactly the same version. Locally installed tools provide consistency, and are installed to the local repository.

Install BDNA

All that's left now is to download and install BDNA:

dotnet tool install bdna

This will install the latest non-preview version. If you want to install a specific version, just give the version, say:

dotnet tool install bdna --version 0.2.263

BDNA packages are listed here.

Check that BNDA is correctly installed:

dotnet tool list

and you will get a list of repo-local tools:

Package Id      Version              Commands      Manifest
-------------------------------------------------------------------------------------------------------
bdna            0.2.263              bdna          projects\scratch\tools\.config\dotnet-tools.json

Check that it's up and running:

dotnet bdna

and you should be greeted with a banner, like:

The installation is done!

You have successfully installed BDNA into your directory, and exactly the same steps will apply in a cloned git repository.

Checking benchmarks

What remains now is to get some benchmarks. If you've followed this series, you'll have some demonstration projects that generate benchmarks, such as

NewDayTechnology / benchmarking-performance-part-2

A simple demonstration of BenchmarkDotNet

Get some benchmarks

Clone the repo and start building:

dotnet clean
dotnet restore
dotnet build -c Release

cd src\benchmarkdotnetdemo\bin\Release\net7.0
dotnet Benchmarkdotnetdemo.dll -f *

The results will be found under **\BenchmarkDotNet.Artifacts\results.

Collect the data from your recent BenchmarkDotNet run

BDNA works by aggregating sequential benchmark runs. To aggregate (from the repo's root directory):

dotnet bdna aggregate -new ".\src\benchmarkdotnetdemo\bin\Release\net7.0\BenchmarkDotNet.Artifacts\results" -aggs ".\bdna" -out ".\bdna" -runs 30

To see all options try dotnet bdna aggregate -?

Repeatedly run benchmarks (dotnet Benchmarkdotnetdemo.dll -f *) and aggregate (dotnet bdna aggregate ...) to build a dataset.

When benchmarking you'll need points of reference for each datapoint. You can use --build %build_number% when aggregating each benchmark run to annotate with the build number. Tags are also supported.

Analyse the data

Now, we want to check the dataset for deviances. To see some errors we'll assume very strict deviance (0%) and allow no errors:

dotnet bdna analyse --aggregates ".\bdna" --tolerance 0 --maxerrors 0 --verbose

dotnet bdna analyse will send results to the console. If all is well you'll see a nice confirmatory message:

But if there are degraded benchmarks they'll be listed:

If too many errors are found the tool's return code will be 1: your CI script will need to watch for this return code and fail the build accordingly.

Reporting on the data

Console logs are often fine for CI pipelines. Wouldn't it be good to get some graphs of performance over time?

dotnet bdna report --aggregates ".\bdna"  --verbose -r csv -r json -f * -out ".\bdna_reports"

For help see dotnet bdna report --help

BDNA will build a CSV (and/or JSON) file containing selected benchmarks. Each benchmark is exported with namespace, class, method, parameters and annotations (build number, tags, etc).

Import the report file in your favourite BI tool and:

These measurements were taken from a machine with a lot of background procesing going on, and so you see peaks and troughs in the measurements. The general trend is flat. This is good, as the code didn't change between builds.

What have we learned?

We've discussed a very simple method of determining degraded performance where we compare results against a best-known result.

We've described how to set up local dotnet tools and nuget configurations.

We've introduced a tool that can collect, report & detect performance degradations, and how it can be used in a sandbox environment.

My sight and accessibility tools

Matthew Lane — Fri, 07 May 2021 10:47:06 +0000

Introduction

If I don’t have my glasses on or my contact lenses in, most visual interfaces are hard to use, the nature of my eyesight is such that anything over 20-30cm from my face becomes blurred, out of focus and indistinct. Even with WCAG AA recommendations the contrast level is often not enough unless I have also pushed up the zoom size. I can easily dig around in the sofa to find and put my glasses on, but there are many forms of degeneration in sight affecting our customers/users that can’t be corrected. A deeper dive into how low vision is described, its incidences and causes can be found in footnote 1.

I invite you to walk through an experience of this and join me for a three stage experiment at the end.

Without my glasses the following image is roughly what I see, only the larger and bolder fonts are readable, images have little internal detail and none of the legal compliance text is readable.

What I see

This is my world without my glasses. Thankfully there are built in tools in most operating systems to help me with this problem.

What I do when my glasses are down the back of the sofa?

One of the most used features for users with low vision is the zoom, allowing content to be enlarged, if a website is well designed the content will re-flow so that content is kept within the browser window. Being able to re-size the interface without a loss in functionality ensures that we meet WCAG 2.1 - 1.4.4: Resize text; a AA standard. All browsers should support zooming using the ctrl and plus keys, ctrl and 0 will reset your zoom.

A system wide tool that enables text to be read more clearly is high contrast which improves clarity and readability, whilst this tool is useful it is system wide, meaning the entire operating system is in high contrast mode.

Our base level of contrast supports WCAG 2.1 - 1.4.3: Contrast (Minimum) a AA standard, but without a control on our sites to increase contrast we cannot meet WCAG 2.1 - 1.4.6: Contrast (Enhanced) a AAA standard.

Enabling high contrast for Windows can be found in start → settings → ease of access → high contrast.

Using zoom and high contrast together gives me a much better chance at understanding what’s happening on the login page, being readable (for me without glasses) with no content hidden by horizontal scroll.

The design systems we use at New Day support our adherence WCAG 2.1 AA guidelines when implemented to meet these standards.

A note about magnifiers: whilst zooming in the browser is supported as a standard, there are tools which magnify a specific portion on the screen in all major operating systems see footnote 2.

Use case: Focus is important

All browsers by default provide a focus state that will meet or exceed WCAG level A guidelines. If we change these default states we must make sure that:

“It must be possible for a person to know which element among multiple elements has the keyboard focus. If there is only one keyboard actionable control on the screen, the success criterion would be met because the visual design presents only one keyboard actionable item.”

The preceding statement is from the understanding success criterion and is a Level A standard see footnote 3.

An Experiment in Three Parts

Part One - The Low Vision Simulator

An experience of low vision can be simulated with a pair of old sunglasses and some sticky back plastic, simply wrap one or two layers of the tape around the glasses and pop them on - you now have an approximation of low vision. (If like me you are lucky enough to use glasses for near sight, just pop them off, or make you own glasses and experience the world of someone with more limited vision).

Part Two - See what it’s like

With your new vision simulators firmly on your face, open a webpage or download an application that you are unfamiliar with and attempt a login or registration journey without using any assistive tools - zoom, high contrast, screen readers, keyboard navigation etc. It’s important to be conscious of your experience, take notes about what actions you find difficult to achieve.

Part Three - Explore assistive technologies

Re-visit the same webpage still with your fancy eyewear on, this time make use of as many accessibility tools as you feel comfortable with. Most mobile and desktop platforms have built in assistive technologies, find a mix that enables you to use application or site you were visiting, take notes about your experience and share that with your colleagues.

Footnotes

Consumer Driven Contract Testing

Elham Khani — Fri, 16 Apr 2021 12:36:15 +0000

Introduction

At NewDay, we used to create backend services and front-end web apps within the same team (or between teams who worked together closely). By growing fast and becoming more mobile oriented, the system architecture is now developing into BFF (Backends for Frontends). That means front-end developers should trust APIs created by other teams and API teams should feel safe to change their APIs.

Contracts are essential in API based architectures (Microservices or BFF), but what if the contract needs to be changed? Who should be notified of the changes and how should they be notified?

These are the issues Consumer Driven Contract Testing solves.

It’s essential that an API team (Provider) and front-end team (Consumer) agree on a contract and develop their code accordingly. In BFF model, the idea is that there are many Consumers of an API - for example mobile and web app teams. When the Provider changes part of the contract, the traditional way of emailing or putting a message in a Slack/Teams Channel is not practical - each Consumer may not know exactly which part of the contract they are using at their end.

Consumer driven contract testing is an API test approach which doesn’t test if the API is working fine, but checks if the contract is not-broken.

Example

Let's say we have an API for setting marketing consent and both mobile and web apps are using them; they are consumers. The API accepts an object with 5 Boolean fields:

{
  "email": true,
  "post": true,
  "sms": true,
  "phone": true,
  "push": true
}

The web app only needs to send four of these fields because "push" (push notifications) are not used in the web app. If the API team decides to change one of the fields, "email", "post", "sms" or "phone", they need to notify both the web app team and mobile team; but if they want to change the "push" field, they don't need to notify the web app team because they are not using that field.

In this scenario, mobile and web app teams write their own contract tests and will run these tests in their own CI pipeline.

This approach allows the consuming team to understand contract changes from the provider, and if there is an impact to them; i.e. CI of mobile / web app teams will fail if the contract tests fail, thereby, notifying themselves as consumers of contract changes from the provider.

Tooling

There are different tools to write Contract Tests. The most common ones are:

Postman/Newman
Pact
Spring Cloud Contract

At Newday, we use Postman to write contract tests, Newman to run the tests in CI (TeamCity) and the Newman reporter to see reports in TeamCity.

Writing contract tests in Postman

Writing API tests in Postman is explained fully here: https://www.postman.com/use-cases/api-testing-automation/

Notice at this point we call our tests, “API tests”, this is because Consumer driven contract tests are only an approach; they are basically API tests.

Here is an example of an API test to validate the schema of our above mentioned consent API scenario. This is the test written by the web app team. Recall they are not interested in the, “push” field, they haven’t put it as part of the schema validation. By clicking, “Send” the test will run and the results are displayed in the, “Test Results” tab.

Save API collection

To be able to run the tests in CI/CD they need to be in a repository. First Export the API collection as a json file and push them to your desired repository. If you are using variables in your collection, you should export the variables to a separate json file. The variable file will be used when running the collection in next step.

Run the tests locally

The next step is to run the API tests in your local CLI and then we will do the same steps in TeamCity.

Newman is the CLI companion of Postman and it can execute Postman collections.

Install Newman globally like so:

npm install -g newman

and run the saved collection

newman run mycollection.json

add -e flag to use an environment variable file

newman run mycollection.json -e dev_environment.json

Now you can run your tests locally and see the result. hopefully all passed!

Run tests in CI (TeamCity)

newman-reporter-teamcity is a newman report for TeamCity. It’s optional but recommended.

Install newman and the above package globally in your build:

npm install -g newman
npm install -g newman-reporter-teamcity

Add another step to run the newman and generate the report.

I use two parameters to locate the postman collection file and postman environment file.

newman run %postman.collection.path% -e %postman.environment.path% --suppress-exit-code --reporters teamcity,cli

Now you can be sure that whenever there is a change in API the contract tests will run and the correct team can be notified if the contract is broken.

Big Questions

Who owns the contract tests?
The Consumer driven contract tests can be done in many ways and you should implement the approach best suited to your products and teams.

I believe the Consumer is the owner of the contract tests and Provider should not change these tests.

Where to put the tests (in this example the postman collection)?
I prefer to put them in the same repository as the API (Provider’s repository).

What stage of CI/CD process should contract tests be executed?
The idea is to notify both the Provider and Consumer when the contract is broken. I prefer to notify the Provider as soon as possible (for example a git commit hook to prevent the Provider to push breaking changes). If the Provider is sure that the changes are correct then they need to change the contract tests but that means Consumer might never get notified. So having them in the Provider’s build and setting up an automated notification for Consumer is the earliest time to get the best results.

Measuring performance using BenchmarkDotNet - Part 2

Tony Knight — Thu, 01 Apr 2021 16:07:43 +0000

Introduction

Previously we discussed what BenchmarkDotNet gives us and how to write simple benchmarks. As a quick reminder:

We use benchmarks to find code performance
BenchmarkDotNet is a nuget package
We use console apps to host and run benchmarks

So what's next to do? We need to run it and get benchmarks as easily and frequently as we can.

Running Benchmarks locally

We have a sample .Net core console application coded up and ready to go in Github:

NewDayTechnology / benchmarking-performance-part-2

A simple demonstration of BenchmarkDotNet

Build and run

Once you've cloned the repo, just run a dotnet publish from the local repository's root folder:

dotnet publish -c Release -o publish

If you're unfamiliar with dotnet's CLI, dotnet publish will build and integrate the application, pushing the complete distributable application to the ./publish directory. You can read more here.

At this point, you've got a benchmarking console application in ./publish that's ready to use. Because I like my command line clean, I'm going to change the working folder:

cd publish

...and we're almost ready to start.

Before you run, prepare your machine

Whenever you're measuring CPU performance you've got to be mindful of what else is running on your machine. Even with a 64 core beast your OS may interrupt the benchmark execution and skew results. That skew is not easy to measure or counter: it's best to assume that the interrupts and switches always happen.

Whenever you run final benchmarks make sure the absolute minimum software and applications are running. Before you start, close down all other applications before running your benchmarks. Browsers, chat, video, everything!

For now, don't close down everything: we're just exploring BenchmarkDotNet here and you need a browser open to read. But, when capturing real results always remember to run on idle machines.

And now to get some benchmarks

To run them all we need to:

dotnet ./benchmarkdotnetdemo.dll -f *

-f * is a BenchmarkDotNet argument to selectively run benchmarks by their fully qualified namespace type. We've elected to select all of them with the wildcard *; if we want to run only selected benchmarks, I'd have to use -f benchmarkdotnetdemo.<pattern> as all these benchmarks fall in the benchmarkdotnetdemo namespace. For instance, -f benchmarkdotnetdemo.Simple* will run all the "Simple" benchmarks.

Each console application with BenchmarkDotNet has help automatically integrated. Just use --help as the arguments, and you will get a very comprehensive set of switches.

So now all we have to do is wait, and eventually your console will give you the good news:

// ***** BenchmarkRunner: End *****
// ** Remained 0 benchmark(s) to run **
Run time: 00:03:44 (224.56 sec), executed benchmarks: 3

Global total time: 00:08:03 (483.58 sec), executed benchmarks: 15
// * Artifacts cleanup *

All good! The result files will have been pushed to the BenchmarkDotNet.Artifacts folder:

    Directory: C:\...\benchmarking-performance-part-2\publish\BenchmarkDotNet.Artifacts


Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
d-----          4/1/2021  11:50 AM                results
-a----          4/1/2021  11:50 AM         128042 BenchmarkRun-20210401-114253.log

The .log file is simply the benchmark console echoed to file.

Within the /results directory you'll find the actual reports:

    Directory: C:\...\benchmarking-performance-part-2\publish\BenchmarkDotNet.Artifacts\results


Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
-a----          4/1/2021  11:47 AM         109014 benchmarkdotnetdemo.FibonacciBenchmark-measurements.csv
-a----          4/1/2021  11:47 AM         103104 benchmarkdotnetdemo.FibonacciBenchmark-report-full.json
-a----          4/1/2021  11:47 AM           3930 benchmarkdotnetdemo.FibonacciBenchmark-report-github.md
-a----          4/1/2021  11:47 AM           6632 benchmarkdotnetdemo.FibonacciBenchmark-report.csv
-a----          4/1/2021  11:47 AM           4484 benchmarkdotnetdemo.FibonacciBenchmark-report.html
-a----          4/1/2021  11:50 AM          83537 benchmarkdotnetdemo.SimpleBenchmark-measurements.csv
-a----          4/1/2021  11:50 AM          53879 benchmarkdotnetdemo.SimpleBenchmark-report-full.json
-a----          4/1/2021  11:50 AM           1215 benchmarkdotnetdemo.SimpleBenchmark-report-github.md
-a----          4/1/2021  11:50 AM           2119 benchmarkdotnetdemo.SimpleBenchmark-report.csv
-a----          4/1/2021  11:50 AM           1881 benchmarkdotnetdemo.SimpleBenchmark-report.html

As you can see, it's a mix of CSV, HTML, markdown and pure JSON ready for publication and reading.

These formats are determined by either the benchmark code or the runtime arguments. I've included them all in the demo repo to give a feel of what's on offer.

Interpreting the results

We've previously discussed the various reports' contents. But suffice to say BenchmarkDotNet runs & reports benchmarks but does not evaluate them.

Evaluating these benchmarks and acting on them is a fairly complex problem: what analysis method to use? How do we run and capture results? Can we use benchmarks as a PR gateway? This will be the subject of a future post.

But before we run, we'd would like benchmarks running on all git pushes, right?

Running benchmarks in CI

Let's implement the simplest possible approach:

build benchmarks
run them
capture the report files
present for manual inspection

In short, benchmarks are built, run, and the results published as workflow artifacts. Anyone with access can download these artifacts.

Because our repo is in Github, and we want to show this in-the-flesh we'll be using Github Actions.

One day, Github Actions will support deep artifact linking and one-click reports, just like Jenkins and TeamCity have provided for years. But until that day dawns the tedium of download-extract-search is our lot :(

Here's a super-simple Github Actions workflow:

If you're unfamiliar with Action workflows, one of the best hands-on introductions is from Pete King.

This workflow file is in the sample Github repository, under ./.github/workflows/dotnet.yml.

Looking at the workflow, let's skip past the job's build steps as they're self explanatory.

`Publish`

- name: Publish
      run: dotnet publish -c Release --verbosity normal -o ./publish/

Here we prepare a fully publishable .Net core application.
We always need to build with Release configuration: BenchmarkDotNet will not adequately run without normal compiler optimisations. The application with its dependencies, including the code-under-test, is pushed to a ./publish/ directory within the job.

One glorious day, both Windows and linux will finally and completely converge on a single standard for directory path separators. Until that time, please be careful if you're writing these workflows on Windows!

`Archive`

- name: Archive 
      uses: actions/upload-artifact@v2
      with:
        name: benchmarkdotnetdemo
        path: ./publish/*

We're just arching the binaries here, in case we want to distribute and run locally.

`Run Benchmarks`

- name: Run Benchmarks    
      run: dotnet "./publish/benchmarkdotnetdemo.dll" -f "benchmarkdotnetdemo.*"

This is where we run the benchmarks.

As of now there are no Github Actions to support benchmark running, so all we do here is run the console application itself within the Github Actions job.

We're running all benchmarks in the benchmarkdotnetdemo namespace, and we expect the results to be pushed to the same working folder.

Note the double quotes! In windows you won't need to quote these arguments, but you will need to for Github Actions. If you don't, you'll see strange command line parsing errors.

Previously I remarked that you should only run benchmarks on an idle machine. Here we'll be running these on virtualised hardware, where OS interrupts are an absolutely unavoidable fact of life. Clearly we're trading precision for convenience here, and the code-under-test is simple enough not to worry too much about single-tick precision metrics.

`Upload benchmark results`

- name: Upload benchmark results
      uses: actions/upload-artifact@v2
      with:
        name: Benchmark_Results
        path: ./BenchmarkDotNet.Artifacts/results/*

This is where we present the results for inspection.

We just zip up the benchmark result files into a single artifact called Benchmark_Results.

And lastly...

That's it! Every time you push changes to this solution, benchmarks will be run. Performance degradations won't fail the build as we're not analysing the results, and we're certainly not applying quality gates in this solution. But you've got the minimum useful visibility, albeit very simply:

What have we learned?

Running benchmarks is very simple on face value, but there are considerations when doing so: you don't want to run while you're rendering videos!

Incorporating benchmark reporting into a CI pipeline is straight forward, although the lack of build reporting in Github Actions is a disappointment.

We've yet to act on those benchmarks' results. For instance, we don't yet fail the build if our code-under-test is underperforming.

Up next

How to fail the build if your code's underperforming.

Measuring performance using BenchmarkDotNet - Part 1

Tony Knight — Mon, 15 Mar 2021 18:15:31 +0000

Introduction

We all must build fast software, right? Right? It’s true that microservices tend to introduce latencies - stateless functions mean a whole lot more network calls, and you can wave goodbye to data locality. But a microservice is still dependent on its own code being fast, and at least fast enough.

In the past we’ve relied on profilers, stopwatches, dedicated performance teams, and sometimes plain old complaints from the field. All of these methods require some form of measurement; unfortunately they tend to be “big picture” performance that lacks detail - and often without concrete scenarios. This gets very expensive very quickly.

Very often, you just want to measure the code’s performance without the baggage of dependencies. You might have a critical piece of code that absolutely must meet certain performance criteria. Measuring such microcode cam obviously be done with profilers - dotTrace, ANTS to name just two. The problem is they bring their own baggage as well, and worse can’t be easily relied upon in a CI pipeline. So how can you measure microcode performance in CI? Unit tests are a terrible idea, what else is there? Step forward BenchmarkDotNet.

TL;DR

Measure your code’s performance with benchmarks at near zero cost and. All you need are:

.NET 7 SDK
VS/VSCode
BenchmarkDotNet from Nuget

We’ll talk about how to write simple benchmarks, how to run them and how to interpret the results.

What is BenchmarkDotNet?

BenchmarkDotNet does what it says on the tin: benchmark .net code. It’s available as a Nuget packaged library for inclusion into your .net console applications. It is very widely used by all major players in the .net world, including the dotnet core runtime project itself.

What does a HelloWorld benchmark look like?

Let’s say you have a very basic Fibonacci implementation - and you want to measure its resource usage growth as more numbers are generated.

By “resource usage” I mean time and memory consumed per method call.

In other words, you'd want to know how it scales. Here's an implementation of "get the first N Fibonacci numbers":



public static IEnumerable<int> GetFibonacci(this int count)
{
    var w = 1;
    var x = 1;

    yield return x;
    foreach (var _ in Enumerable.Range(1, count - 1))
    {
        var y = w + x;
        yield return y;
        w = x;
        x = y;
    }
}

No prizes are sought for best efficiency here. Please do not take this as a reference implementation of Fibonacci!

To answer the scaling question, we would implement a benchmark, run it and analyse the results. Skipping forward a rendered benchmark report would look something like the below:

What do all the headers actually mean?

The column	What it means
Method	The name of the code-under-test; a single benchmark may have several methods under test for, e.g. scenarios. This value is lifted directly from your benchmark code.
Count	An arbitrary parameter: in this case the number of Fibonacci numbers generated by the method under test.
Mean/Error/StdDev/StdError	Execution time statistics. Note that these can be given down to nanoseconds, depending on how fast your code is. Low is best.
Min/Q1/Median/Q3/Max	Quartile execution time statistics: note the time units. Low is best.
Ops/sec	The number of operations executed per second for the method/parameter combination. High is good.
Rank	The fastest performing method/parameter combination. Low is best.
Gen 0/1/2	The total number of collections per generation
Allocated	Total bytes allocated against all generations

Note the header information in the report! It’ll give details on the OS, CPU, .Net version, JIT method and GC configuration. Always benchmark like-for-like!

OK… what do those numbers really mean?

Let’s look at each iteration of Count, and we’re using it here to get the first Count numbers of the Fibonacci sequence.

Where Count is 1 the mean execution time is 103.4 nanoseconds. That’s 0.1 microseconds, or 0.0001 milliseconds. I like that: nice and fast.

Where Count is 13 (yes, the parameters themselves follow Fibonacci!) the mean time is 407.2 ns: four times what Count=1 is, yet the Count is 13 times bigger. I’ll take that, for now.

Where Count is 34 the mean time is 1,077.9 ns, or 1.077 microseconds, or just over 0.0001 milliseconds. That’s taking 2.6 times more time than Count = 13. Let’s compare against Count = 1: Count is 34 times bigger , yet takes 10 times the time. I’ll take that too.

If we plot Count against the time ratio we see this:

In other words, time used does not grow as Count grows. If it did, the lines would be parallel.

So the benchmarks are showing that the implementation has reasonably acceptable scaling. It's not constant time, but it’s better than O(n) time: a pleasant surprise.

If you're not satisfied with the performance results, simply make your changes, re-run the benchmarks & re-analyse. That's it.

You haven’t mentioned the memory yet, have you?

Trust me, I’m getting to that.

Pay particular attention to memory usage. Garbage collections and memory allocations are as important as sheer speed!

Count=1 used 128 bytes.
Count=13 used 312 bytes
Count=34 used 744 bytes.

If we plot Count against the allocation growth ratios, we see this:

This means the used memory isn’t constant either: the memory used for Count=34 is greater than the memory used for Count=1. Again it's better than O(n). To my mind this is OK, but not great: we need more investigation. It's probably incurred with yield return, but do we want to sacrifice the readability? Probably not, but in any case we’re getting new perspectives on our code. This is a good thing.

What other rendered reports can you get?

You can output a markdown version of your report and many other formats; Markown output is GitHub inspired.

You can use the following attributes to output the many different types of rendered reports:



[JsonExporterAttribute.Full()]
[CsvMeasurementsExporter()]
[CsvExporter(CsvSeparator.Comma)]
[HtmlExporter()]
[MarkdownExporterAttribute.GitHub()]

An example of the GitHub Markdown report:

Charting is supported through the R project. As R is a world in itself, I’m going to skip the subject.

If you want charts, consider importing the rendered JSON into Excel. The CsvExporter attribute will generate a CSV with the data you need.

Full code example

What does the benchmark code look like using BenchmarkDotNet? It might surprise you to see how simple it is.

BenchmarkDotNet relies on declarative code over which it will reflect. Leaving aside the class attributes (more on those later), note the [Params] attribute over Count from the report above, likewise [Benchmark] and Fibonacci().



using System.Linq;
using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Exporters.Csv;

namespace benchmarkdotnetdemo
{
    [InProcess()]
    [MemoryDiagnoser]
    [RankColumn, MinColumn, MaxColumn, Q1Column, Q3Column, AllStatisticsColumn]
    [JsonExporterAttribute.Full, CsvMeasurementsExporter, CsvExporter(CsvSeparator.Comma), HtmlExporter, MarkdownExporterAttribute.GitHub]
    [GcServer(true)]
    public class FibonacciBenchmark
    {
        [Params(1, 2, 3, 5, 8, 13, 21, 34)]
        public int Count { get; set; }

        [Benchmark]
        public void Fibonacci()
        {
            var xs = Count.GetFibonacci().ToList();
        }
    }
}

You’ll notice that the benchmarks have a return type of void and do not have any assertions. Remember: we’re not proving functional correctness here, we’re measuring resource usage.

Show me the code!

I’ve created a simple BenchmarkDotNet implementation here:

NewDayTechnology / benchmarking-performance-part-1

A simple demonstration of BenchmarkDotNet

There’s only the one C# project in there - benchmarkdotnetdemo.csproj - that contains the minimal files.

BenchmarkDotNet will only work if the console project is built with a Release configuration, that is with code optimisations applied. Running in Debug will result in a run-time error.

Setup

This is the Program.cs file, and like all C# console apps you need an entry point:



using System;
using BenchmarkDotNet.Running;

namespace benchmarkdotnetdemo
{
    class Program
    {
        static int Main(string[] args)
        {
            try
            {
                BenchmarkSwitcher.FromAssembly(typeof(Program).Assembly)
                    .Run(args);

                return 0;
            }
            catch (Exception ex)
            {
                Console.ForegroundColor = ConsoleColor.Red;
                Console.WriteLine(ex.Message);
                Console.ResetColor();
                return 1;
            }
        }
    }
}

Aside from the standard method entry point let’s go over it bit by bit.



using BenchmarkDotNet.Running;

For bootstrapping BenchmarkDotNet, this is the only import you need.



BenchmarkSwitcher.FromAssembly(typeof(Program).Assembly).Run(args);

This one-line-to-rule-them-all will perform all command line parsing, all help, all benchmark execution and all report generation.

One point here is .FromAssembly(typeof(Program).Assembly) - this informs BenchmarkDotNet of its benchmark search scope. Benchmarks are internally discovered by reflection - you’ll see soon enough.

NOTE: If you were to run the project without any command line arguments, BenchmarkDotNet will assume an interactive CLI.

.Run(args) returns a sequence of report objects comprised of the same data used for rendered reports: I’ve excluded them for simplicity. If you want to run benchmarks and fail CI builds if performance dips they are your first place to look.

Create a new benchmark

There is a file called SimpleBenchmark.cs. Let’s have a look.



using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Exporters.Csv;

namespace benchmarkdotnetdemo
{
    [InProcess()]
    [MemoryDiagnoser]
    [RankColumn, MinColumn, MaxColumn, Q1Column, Q3Column, AllStatisticsColumn]
    [JsonExporterAttribute.Full, CsvMeasurementsExporter, CsvExporter(CsvSeparator.Comma), HtmlExporter, MarkdownExporterAttribute.GitHub]
    [GcServer(true)]
    public class SimpleBenchmark
    {
        [Benchmark]
        public void NoopTest() { }

        [Benchmark]
        public int AddTest() => int.MaxValue + int.MinValue;

        [Benchmark]
        public int MultiplyTest() => 11 * 3;
    }
}

FibonacciBenchmark.cs

Just for completeness: note the similar declarations as SimpleBenchmarks.cs. In this case, we’re adding a [Params] parameter to support benchmark permutations.



using System.Linq;
using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Exporters.Csv;

namespace benchmarkdotnetdemo
{
    [InProcess()]
    [MemoryDiagnoser]
    [RankColumn, MinColumn, MaxColumn, Q1Column, Q3Column, AllStatisticsColumn]
    [JsonExporterAttribute.Full, CsvMeasurementsExporter, CsvExporter(CsvSeparator.Comma), HtmlExporter, MarkdownExporterAttribute.GitHub]
    [GcServer(true)]
    public class FibonacciBenchmark
    {
        [Params(1, 2, 3, 5, 8, 13, 21, 34)]
        public int Count { get; set; }

        [Benchmark]
        public void Fibonacci()
        {
            var xs = Count.GetFibonacci().ToList();
        }
    }
}

How are benchmarks executed?

Without going into too much detail, BenchmarkDotNet will attempt to run your benchmarks many many times over to settle on mean and median values.

When you run the benchmarks you may first be confused by just how many iterations are involved, so let’s give a simplistic explanation. Modern OSs are preemptive multitaskers, CPUs have pipeline caches as well as instruction reordering features. .NET itself has the JIT compiler. This means that no single execution of code can be relied upon to give a canonical result.

This is part of the reason why unit tests are terrible for benchmarking! They only run once and incur their own (unaccounted) overheads.

BenchmarkDotNet will run warm up iterations before it can take representative values. These show up as various stages: OverheadJitting & WorkloadJitting, WorkloadPilot, OverheadWarmup, OverheadActual.

JIT comes at a cost: the first time any .NET code executes it must first be JIT compiled. The more complex the code the higher the JIT cost, usually showing as CPU and time costs. As we’re interested only in runtime performance, these steps eliminate JIT costs from measurements.

In the same vein other warmup steps are run to eliminate other “once only” costs, for instance to warm up pipelining caches.

After these steps have completed, BenchmarkDotNet will iterate these operations to yield the final statistics; these are shown as WorkloadActual steps.

If you want more detail, please refer to BenchmarkDotNet’s own documentation. In these code samples we’re using the default Throughput strategy for microbenchmarking.

How long does it take?

It depends ;) Simple calculations, such as in the demo project, will run in under a minute. Adding permutations (such as with [Params]) will linearly increase the benchmarking time, as each parameter will be benchmarked in its own right.

With that in mind, it’s quite clear that resource hungry algorithms, benchmarked with a large variety of parameters, will take a considerable amount of time.

Don’t expect to parallelise BenchmarkDotNet: it runs benchmarks sequentially. Thread context switching is itself a cost and extremely difficult to compensate for.

What have we learned?

We’ve seen how to get BenchmarkDotNet
We’ve seen how to integrate it in a simple console application
We’ve seen the minimum work needed to build benchmarks
We’ve had a taste of the reports and inferences we can gain from BenchmarkDotNet

Next Steps

How to incorporate into CI?

More Information

What is benchmarking - Wiki
BenchmarkDotNet
BenchmarkDotNet on Github
A real world use case of BenchmarkDotNet
GitHub repository:
NewDayTechnology / benchmarking-performance-part-1

A simple demonstration of BenchmarkDotNet
Measuring performance with BenchmarkDotNet part 1

A simple demonstration of the superlative BenchmarkDotNet and its integration into Github Actions.

Measuring code performance is self evidently a vital discipline to software engineering and yet is so often skipped, usually for false economies. BenchmarkDotNet makes this essential task simplicity itself, with a syntax and style that's immediately intuitive to anyone versed in unit testing.

Just exercise your code in a declarative way, include it in your CI pipeline, and enjoy the results.

This project just demonstrates the basics: the .net project, the CI pipeline and the resultant reports.

The Benchmarks
- NoopTest The absolute minimum function that can be benchmarked - it does nothing.
- AddTest A simple addition metric, again of minimal complexity.
- MultiplyTest A simple multiplication metric, again of minimal complexity.
- Fibonacci Benchmarking a Fibonacci implementation, measuring the computation time for the first N Fibonacci numbers.
Builds

Builds are managed with love…
View on GitHub