DEV Community: chaitanya.dev

Using Dapper over EntityFramework for database operations in .NET Core

chaitanya.dev — Mon, 28 Jun 2021 17:15:58 +0000

We’re entering a new world in which data may be more important than software.
--Tim O’Reilly

In most of the use cases for today’s software, data access plays an important role.
.NET core supports multiple framework options to store and access data in your software application may it be a microservice or a web application.

Entity Framework Core is one of the most recommended options suggested for applications working with relational database. EF Core is an object-relational mapper(ORM) that enables .NET developers to persist objects to and from a data source. It eliminates the need for most of the data access code developers would typically need to write.

Dapper is a simple object mapper for .NET and owns the title of King of Micro ORM in terms of speed and is virtually as fast as using a raw ADO.NET data reader. Dapper operates directly using the IDbConnection interface which is extended by database providers like SQL Server, Oracle, MySQL etc. for their database.

Dapper is known for its high performance and is especially used by developers when they want to write the SQL query themselves with optimal performance or use stored procedures.
But how fast is it compared to EF Core?

To figure this out, as always, I have created a demo project here that compares the same read and write operations between EF Core and Dapper using BenchmarkDotNet. I have used SQL Server as the relational database to insert and retrieve data.

Structure of the database

For this demo, I have created two tables called Athletes and Sports. Before the execution of the benchmarks, I am also clearing down the database and reseeding the data with 16 records in Athletes and 5 records in Sports.

Read Operation

I have created repository methods for EFCore and Dapper each to fetch data from the database. I have used the same criteria for both, which is to fetch the names of all indoor sport athletes older than 25 years of age.

Write Operation

I have created repository methods for EFCore and Dapper each to insert Athlete data into the database. I have used the same data to be inserted in both the methods.

Performance benchmarks for read and write

I have used BenchmarkDotNet to benchmark the performance for the database read and write operations using both EF Core and Dapper.

BenchmarkDotNet helps you to transform methods into benchmarks, track their performance, and share reproducible measurement experiments. The results for these benchmarks are presented in a user-friendly form that highlights all the important facts about your experiment.

I have set the iteration count as 100 iterations. This can be configured by assigning the desired value to the variable numberOfIterations.

Benchmark results

On running the benchmark project, the below summary is what is displayed as output which shows that for Dapper both read and write are faster, almost x2 times, than EF Core for the same kind of operation.

The summary also shows that the memory allocated for Dapper operations is far less than the memory allocated for EF Core operations.

The detailed results for both the read and write operations give detailed statistical information for the benchmark. It also gives a histogram for the the time interval for the database operation. BenchmarkDotNet also allows the measured data to be exported in different formats including plots.

While benchmark results clearly state that Dapper has a higher performance than EF Core for both read and writes, in most instances the performance for database writes is almost comparable. Due to this, some developers prefer to use Dapper for read operations and EF Core for write operations involving complex objects to be written into the database as transforming them into Dapper queries itself may be a cumbersome task.

Dapper is safe from SQL injection with the use of parameters using anonymous objects and also allows you to use custom queries with performance enhancement which gives you better control over the database operations for eg. using with(nolock) in select queries.

Thus we have seen why Dapper is known as King of Micro ORM and how in our example, BenchmarkDotNet helped us showcase that it has a better performance to EF Core. It is important to note that the performance of both EF Core and Dapper vary based on the use case and the type of database operations being performed, but Dapper in general performs better with database read operations.

You can find my demo project that I have created to run this benchmark on my github repo here https://github.com/chaitanya-suvarna/efcore-dapper-benchmark

I hope you found this interesting and useful.

Thank you for reading!

Event Sourcing pattern for microservices in .Net Core

chaitanya.dev — Sat, 05 Jun 2021 15:44:17 +0000

Event Sourcing pattern is a Domain Driven Design pattern that defines an approach to handling operations on data that’s driven by a sequence of events, each of which is recorded in an append-only store.

Why should we follow an architecture pattern?

Every project’s software development life cycle has 2 opposing forces, the force of doing things and the force of doing things right.

While both lead to a working software, the force of doing things, or the force of getting things done as I like to call it, leads to a greater amount of rework.

By not following a tried and tested architecture pattern and just trying to get a tangible output asap always leads to rework. The rework may be due to the issues faced at a later point in the life cycle due to the incorrect domain definition or frequent changes in design due to the lack of domain knowledge at the design phase. Rework could also be due to the inevitable technical debt which one has to pay at some point.

What is Event Sourcing pattern? How is it different?

The typical approach for most applications that work with data is to maintain the current state of the data by updating it as users work with it. For example, in the traditional CRUD model a typical data process is to read data from the store, make some modifications to it, and update the current state of the data with the new values—often by using transactions that lock the data.

The fundamental idea of Event Sourcing is that of ensuring every change to the state of an application is captured in an event object, and that these event objects are themselves stored in the sequence they were applied in the EventStore. Not only can we query these events, we can also use the event log to reconstruct past states.

Some key facts to remember while implementing Event Sourcing pattern

An event is something that has happened in the past.
Events are expressions of the ubiquitous language.
Events are not imperative and are named using past tense verbs.
Have a persistent store of events. (Event Store)
Append-only, no delete. A delete or update action(updateEvent) will also be appended in the event store to maintain the history but no existing records will be modified.
Replay the (related)events to get the last known state of the entity. (Event aggregator)

How does Event Sourcing work?

Let’s take an example of a conference management system that needs to track the number of completed bookings for a conference so that it can check whether there are seats still available when a potential attendee tries to make a booking.

In our conference management system we have 3 microservices for the sake of this example. This example also implements the CQRS pattern along with the Event Sourcing pattern.

Reservation : On receiving the CreateReservationCommand or UpdateReservationCommand it creates the corresponding ReservationCreatedEvent/ReservationUpdatedEvent and publishes the event to the microservices listening to the event(s) on the message bus. It also checks for the availability of seats for a given conference and availability of the attendee at the specific time slot by using GetConferenceDetailsQuery and GetAttendeeStatusQuery respectively. It also persists the event to the Event Store which is used when a GetConferenceReservationStatusQuery is received. The Event Aggregator is used to create the current reservation state of a conference by aggregating the sequence of events stored in the Event Store.
Attendee : On receiving the ReservationCreatedEvent/ReservationUpdatedEvent in the Event Store, an attendee object is created/updated in the database. On receiving the GetAttendeeStatusQuery to check the attendee’s availability for a specific time slot, the Event Aggregator is used to aggregate the sequence of ReservationEvents and find out the availability of the attendee for the specific event.
Conference : The Conference microservice uses the Event Aggregator to aggregate the ReservationCreatedEvent/ReservationUpdatedEvent in the Event Store to create the current state of number of seats available for the conference. This can be used to send the response for GetConferenceReservationStatusQuery.

Now let’s find out what the terms Event Store and Event Aggregator, which were repeatedly used in our example, mean.

Event Store

In software, persistence is made of four key operations through which developers manipulate the state of the system. They are CREATE, UPDATE, and DELETE, to alter the state, and QUERIES to read the state without altering. A key principle of software and especially CQRS inspired software, is that asking a question should not change the answer.

Similarly an Event Store is where the Events are persisted in the form of streams of immutable events. The event store is the permanent source of information, and so the event data should never be updated. The only way to update an entity to undo a change is to add a compensating event to the event store.

The CREATE operation in an event-based persistence scenario is not much different from classic persistence(CREATE). The event is is created with the following attributes in the Event Store.

EventId : used to uniquely identify the event.
Event TimeStamp : used to identify when the event was created and is also used to identify the sequence of events.
EventName : used to recognise the type of Event
EventData : the Event object stored in a serialised form
Aggregator Id : used to identify the entity to which the event belongs. This is used while materialising the view for the entity

EventStore table in database:

EventId	Event TimeStamp	EventName	EventData	Aggregator Id
1	2019-10-31T01:48:52	ReservationCreatedEvent	{ ReservationCreated..	1000129
2	2019-10-31T02:12:35	ReservationCreatedEvent	{ ReservationCreated..	1000129
3	2019-10-31T02:30:15	ReservationUpdatedEvent	{ ReservationUpdated..	1000129
4	2019-10-31T07:23:18	ReservationCreatedEvent	{ ReservationCreated..	1000129

The UPDATE operation on an entity is different from a classic update. Here you don’t override an existing record, but just add another record with similar information as create operation, unique event id, timestamp of event, event name, serialised UpdateEvent data and aggregator Id.

The DELETE operations are analogous to UPDATE operations. Here also you don’t delete any existing record, but just add another record to specify the delete event. Subsequently, we can say that the deletion is logical and consists in writing that the entity with a given ID is no longer valid and should not be considered for the purposes of the business. This can be specified by the unique event name to specify the entity is deleted.

Event Aggregator

The main aspect of event sourcing is the persistence of messages, which enables you to keep track of all changes in the state of the application. Recording individual events doesn’t give you immediate notion, however, by reading back the log of events, you can rebuild the present state of the entity. This aspect is what is commonly called the replay of events and is done by the Event Aggregator.

Event aggregation is a two-step operation. First, you grab all events stored for a given aggregate in sequence using AggregateId and TimeStamp. Second, you look through all events in some way and extract information from events and copy that information to a fresh instance of the aggregate of choice.

A key function one expects out of an event-based data store is the ability to return the full or partial stream of events. This function is necessary to rebuild the state of an aggregate out of recorded events.

In our example, let’s consider the conference with conferenceId 1000129 has the capacity of 50 seats.

Reservation m/s receives a CreateReservationCommand to reserve 1 seat for conference 1000129 followed by 2 seats. Therefore 2 CreateEvents are stored in the event store.
Reservation m/s the receives a UpdateReservationCommand to cancel the reservation for 1 seat for conference 1000129. 1 UpdateEvent is stored in the Event Store.
Reservation m/s receives a CreateReservationCommand to reserve 1 seat for conference 1000129. 1 CreateEvent is stored in the event store.

Reservation m/s now receives the GetConferenceReservationStatusQuery. The Command Handler calls the Event Aggregator called ReservationAggregator which gets the conferenceId 1000129 from the query and fetches all events in the EventStore where AggregatorId=1000129. It then finds 4 events from the event store (3 CreateEvents and 1 UpdateEvents). On processing these events the seat availability is calculated and the SeateAvailabilityAggregate is created.
The response is sent back by the Reservation m/s specifying 47 seats out of the 50 seats are available.

Advantages of Event Sourcing pattern

Performance improvement in insert/update of data — Events are immutable and can be stored using an append-only operation. Therefore, we would expect to have much less(or none) deadlocks(or concurrency exceptions) happening whenever and event is stored in Event Store.
Simplification of implementation – Events don’t directly update a data store. They’re simply recorded for handling at the appropriate time. This can simplify implementation and management. However, the domain model must still be designed to protect itself from requests that might result in an inconsistent state.
Audit trail — since state is constructed from sequence of events it is possible to extract detailed log since the begging and up to current date. This is becomes very useful for a production incident post-mortem. The list of events can also be used to analyse application performance and detect user behaviour trends, or to obtain other useful business information.
Projections & queries — The append-only storage of events can be used to monitor actions taken against a data store, regenerate the current state as materialised views or projections by replaying the events at any time. In addition, the requirement to use compensating events to cancel changes provides a history of changes that were reversed, which wouldn’t be the case if the model simply stored the current state.

Thus we have seen how Event Sourcing pattern can be used in microservices using .net core.

Most of the content that I have referred to get a better understanding of Event Sourcing Pattern and to write this post are as below:

Modern Software Architecture course by Dino Esposito on Pluralsight.
Microsoft Docs
Martin Fowler’s blog

This post has been created by me as I was learning and implementing Event Sourcing pattern in the code I develop. There may be some parts missed or not completely covered. Please feel free to reach out to me, I’d love to talk about software development patterns and learn along the way.

I hope you enjoyed reading this.

Thank you!

Creating Deployment & Rollback SQL Scripts from EntityFrameworkCore migrations

chaitanya.dev — Mon, 05 Apr 2021 13:56:56 +0000

If you have worked on an application implemented in dotnet core, chances are high that changes in data models and database schemas are managed using EF Core. The migrations feature in EF Core provides a way to incrementally update the database schema to keep it in sync with the application’s data model while preserving existing data in the database.

There are various strategies for applying EF Core migrations, with some being more appropriate for production environments, and others for the development lifecycle.

Microsoft’s EF Core documentation suggests that the recommended way to deploy migrations to a production database is by generating SQL scripts. The advantages of this strategy are stated as following:

SQL scripts can be reviewed for accuracy; this is important since applying schema changes to production databases is a potentially dangerous operation that could involve data loss.
In some cases, the scripts can be tuned to fit the specific needs of a production database.
SQL scripts can be used in conjunction with a deployment technology, and can even be generated as part of your CI process.
SQL scripts can be provided to a DBA, and can be managed and archived separately.

Let’s have a look at how to generate SQL scripts for your migrations

Generating SQL scripts for applying migrations

The below command can be executed using the .NET core CLI to generate SQL script for your migrations. This command generates a SQL script from a blank database to the latest migration.

dotnet ef migrations script

Script generation accepts the following two arguments to indicate which range of migrations should be generated:

The from migration should be the last migration applied to the database before running the script. If no migrations have been applied, specify 0 (this is the default).
The to migration is the last migration that will be applied to the database after running the script. This defaults to the last migration in your project.

You can also mention the migration from which you want to create the SQL Script to the latest migration by adding the from migration name as mentioned below

dotnet ef migrations script FromMigrationName

If you prefer to generate a SQL script from the specified from migration to the specified to migration, you could mention the from & to migration as mentioned below

dotnet ef migrations script FromMigrationName ToMigrationName

Generating Rollback SQL Scripts for your migrations

Whenever you deploy changes to any higher environment such as a UAT environment, any other testing environment or a Production/DR environment it is always necessary to have a rollback script ready in case there are some issues faced during deployment and you need to rollback the changes so that the user experience/testing is not impacted.

You can generate rollback scripts using the ef core script generation command. The only difference would be the from and to migrations would be inverted.

If your command that generates SQL script for applying migrations looks like below

dotnet ef migrations script ThirdMigrationName FifthMigrationName

then the command to generate Rollback SQL Script would be :

dotnet ef migrations script FifthMigrationName ThirdMigrationName

If you want to rollback all migrations then you can specify 0 in the to argument and from argument would contain the migration from which you want the rollback script to start.

Idempotent SQL scripts

EF Core also supports generating idempotent scripts, which internally check which migrations have already been applied (via the migrations history table), and only apply missing ones. This is useful if you don’t exactly know what the last migration applied to the database was, or if you are deploying to multiple databases that may each be at a different migration.

The following generates idempotent migrations :

dotnet ef migrations script --idempotent

It is a good practice to always generate idempotent scripts so that you don’t end up adding the same migration multiple times on your database leaving the database in an inconsistent state.

Before deploying to production the DBA should always review the script for accuracy. This is to ensure that the SQL script is updated in case the production database might have some minor changes which are not present in the other environments and a DBA might be aware of these changes.

Thus, we saw how we can generate SQL scripts and rollback scripts for EF Core migrations.

I hope you found this informative and helpful.

Thank you for reading!

Implementing factory pattern for dependency injection in .NET core

chaitanya.dev — Tue, 23 Mar 2021 15:53:40 +0000

ASP.NET Core supports the dependency injection (DI) software design pattern, which is a technique for achieving Inversion of Control (IoC) between classes and their dependencies.

A dependency is an object that another object depends on. If there is a hardcoded dependency in the class i.e. the object is instantiated in the class, this could create some problems. In case there is a need to replace the dependency with another object, it would require the modification of the class. It also makes it difficult to unit test the class.

By using dependency injection we move the creation and binding of the dependent objects outside of the class that depends on them. Thus, solving the problems that we face with hardcoded dependency and making the classes loosely coupled.

What is Factory Pattern?

The Factory Pattern was first introduced in the book “Design Patterns: Elements of Reusable Object-Oriented Software” by the “Gang of Four” (Erich Gamma, Richard Helm, Ralph Johnson, and John Vlissides).

The Factory Pattern is a Design Pattern which defines an interface for creating an object but lets the classes that have a dependency on the interface decide which class to instantiate. This abstracts the process of object generation so that the type of object to be instantiated can be determined at run-time by the class that want’s to instantiate the object.

When is Factory Pattern for DI required?

Factory Pattern is useful when there are multiple classes that implement an interface and there is a class that has a dependency on this interface. But this class will determine at run-time, based on user input, which type of object does it want to instantiate for that interface.

To understand this scenario further, let’s take an example of an interface called IShape.

And we also have two classes called Cube and Sphere that implement IShape.

Now, imagine we have a service class called ShapeCalculationService which has a dependency on IShape. It takes the input from user to choose either Cube or Sphere and based on the input it has to instantiate the Cube or Sphere object at runtime, how would that be possible?

The service class needs to use this corresponding Shape object to Get the input and display the SurfaceArea and Volume for the shape.

This scenario where multiple classes implement an interface and the object that needs to be instantiated is decided at runtime, is where Factory Pattern comes to use.

How to implement Factory Pattern for DI?

Continuing with above example, we will try to implement factory pattern in this scenario.
To abstract the instantiation of the correct Shape object at runtime, we will create a ShapeFactory class who’s responsibility is to resolve which concrete class is required to be instantiated for a given selection by user.

Here you can see that the ShapeFactory class has a dependency on IServiceProvider so that ShapeFactory only resolves which concrete class needs to be instantiated but should ask the built-in service container of .Net core to get the instance and resolve its dependencies.

This is because we want to rely on IoC container of .Net core to resolve our dependencies and don’t want to make changes in out factory every single time a new dependency is introduced in either of Cube or Sphere classes.
This further decouples the code and makes it easier to manage.

The place where you setup the ServiceCollection for your project should look like below so that the DI of .Net core could figure out the required dependencies by either of the service and could resolve them at run time.

In my example, since this is a simple console application, this code resides in Program.cs

Finally, I’d also like you to see the actual ShapeCalculationService that takes an input from the user and uses ShapeFactory to get the Shape class at runtime which is the used to display the surface area and the volume.

Conclusion

Thus we have seen how we can inject a factory to get total control of creation of our dependencies at runtime, while still using .Net core’s IoC container to resolve our dependencies.

If you are like me, who needs a small yet complete demo solution to clearly understand how this works, I have created a demo project in my github repository which would help you understand this better.
You can have a look at it here : DotNetCoreFactoryPattern

I hope you found this interesting. Thanks for reading!

The $81 million Bangladesh bank heist that was assisted with improper software security practices in place

chaitanya.dev — Mon, 30 Nov 2020 06:59:39 +0000

In February 2016, hackers broke into Bangladesh’s central bank and were able to steal $81 million from the bank. The ‘breaking in’ was not done physically, instead they broke into the bank’s computer systems due to improper security controls in place. The hackers used a malware that allowed them to hack into the bank’s SWIFT software to transfer money, as well as hide their tracks.

What is a SWIFT message?

SWIFT stands for the Society for Worldwide Interbank Financial Telecommunication and is a consortium that operates a trusted and closed computer network for communication between member banks around the world. The consortium, which dates back to the 1970s, is based in Belgium and is overseen by the National Bank of Belgium and a committee composed of representatives from the US Federal Reserve, the Bank of England, the European Central Bank, the Bank of Japan and other major banks. The SWIFT platform has some 11,000 users and processes about 25 million communications a day, most of them being money transfer transactions. Financial institutions and brokerage houses that use SWIFT have codes that identify each institution as well as credentials that authenticate and verify transactions. The SWIFT message contains instructions through these standardised system of codes.

Now that we know what SWIFT messages are, let’s have a look at the sequence of events that took place leading to this heist and also discuss the learnings from this event.

Pre-heist

In January, weeks before the heist, the hackers obtained the computer credentials of a SWIFT operator at Bangladesh Bank by installing a malware on the bank’s systems. The hackers did a series of test runs, logging into the system briefly several times between Jan 24 and Feb 2. One day they left monitoring software running on the bank’s SWIFT system; on another they deleted files from a database. All this to make sure they have the necessary control over the systems.

February 4, Thursday

During the late evening of Feb 4, when most of the bank’s employees were off work, the hackers initiated 35 fraudulent payment orders via SWIFT worth $951 million from the Bangladesh Bank’s account with the Federal Reserve Bank of New York to transfer funds. Out of these 35 requests, 30 requests worth $851 million were flagged for review by the Fed while 5 requests were granted; $20 million to Sri Lanka and $81 million to Philippines. These successful transactions were then forwarded to the correspondent banks to be later transferred to the destination bank accounts. The $20 million transfer to Sri Lanka gained suspicion from Deutsche bank, one of the routing banks, because the hackers misspelled the word “Foundation” as “Fundation” thus putting the transaction at halt until clarifications are received from Bangladesh Bank. The Fed had also sent multiple notifications to the Bangladesh Bank but received no response.
After the requests are sent, the malware checks the SWIFT messaging system on the terminal and deletes any incoming messages or confirmation messages relating to the fraudulent transfers before they are sent to the office printer. Thus cleaning all tracks.

February 5, Friday

Friday, being a weekend day in Bangladesh, only a handful of staff come into the bank finding an empty printer tray, that would normally contain the SWIFT transaction related messages that are printed automatically. They also find out that the printer is broken, which was not unusual. The boss asks for the printer to be repaired and heads off with his daily routine. At the same time, the $81 million lands in 4 bank accounts in Manila branch of a Philippine bank called RCBC. These accounts were later found to be created with fraudulent IDs. The funds were were then transferred to a foreign exchange broker to be converted to Philippine pesos, returned to the RCBC, consolidated in an account.

February 6, Saturday

The employees of Bangladesh Bank returned to work on Saturday around 9am and tried again to use the printer only to discover the SWIFT software was not starting up while showing an error that said a file NROFF.EXE “is missing or changed”. When they finally got access to the SWIFT messaging system after a series of approvals from senior officials to use other means to access the system and manually print the SWIFT messages, did they realise what had happened. They promptly contacted the New York Fed through phone, emails and fax details available on their official website but there were no response from the office as it was the weekend.

February 8, Monday

SWIFT remotely fixes the messaging system and now Bangladesh Bank has realised that the money has gone to RCBC in Philippines. They send a SWIFT message to RCBC asking them to STOP the transfers, but it’s a public holiday in the Philippines on 8th Feb due to the Chinese New year and the message was received by them only a day later.

February 9, Tuesday

The SWIFT message sent to RCBC asking them to STOP the transfers was sent as normal SWIFT message and not as a CANCEL message. Due to this they are added to a pile of other routine messages at the RCBC headquarters and sent to the RCBC branch containing the accounts. By the time the branch gets to the message, the money in the accounts is already transferred to other accounts with most of it ending up into 4 Philippine casinos.
Casinos in Philippines are not covered by anti-money laundering laws, which means there are gaps in record-keeping around where money goes once a casino obtains it.
To this day the entirety of $81 million has not been recovered and there are multiple investigations going on at Bangladesh Bank, Philippine’s Anti money Laundering Council and the US Fed.

Why was this heist successful?

Carefully designed heist by the hackers. The 4 bank accounts in RCBC bank used to transfer the money were created in May 2015 months ago with fake IDs and were laying cold with just $500 in them for months until the attack. The hackers also planned around the fact that Friday would be a weekend at Bangladesh and Monday would be a holiday in Philippines, not allowing all parties to be available at the same time.
Delay at Bangladesh Bank. Had they proactively worked towards finding out why the SWIFT acknowledgement messages were not printed on Friday, they could have found out about the heist and contacted the New York Fed who’d have responded on Friday instead of the off-duty hours on Saturday.
No 24*7 support by Federal Reserve Bank of New York for such emergencies and dependency on the the automated system that mostly reviews just the format of the SWIFT messages and flags suspicious messages to be manually reviewed.
Weak Anti Money Laundering laws in Philippines which led the money to be vanished in the Casino system where they can keep customer and account details private without proper record keeping.
Lack of proper software security controls in place at the Bangladesh bank was one of the major reasons this hack was successful and I’d like to discuss more on this.

How did improper software security practices enable the heist?

The Bangladesh bank had not protected its computer systems with a firewall, and it had used second-hand $10 electronic switches to network computers linked to the SWIFT global payment system. Hackers may have exploited such weaknesses after Bangladesh Bank connected a new electronic payment system, known as real time gross settlement (RTGS) in November the previous year. One of the reports also said that the malware might have installed when one of the employees accessed their mailbox on the same network as the SWIFT system and opened a contaminated email which could have been easily prevented with a proper firewall in place.
“Banks should conduct SWIFT transactions only on computers that are isolated from other devices on their networks”, says Sean Sullivan, an adviser at the security firm F-Secure. “It should be a dedicated computer for its single task”, Sullivan says. Despite Swift’s warnings, the bank had not segregated its Swift server from the rest of the computer network. In addition to all this, the bank had not updated the SWIFT systems to the latest version of the softwares that had the latest security patches included.
Without these processes in place, the hackers were already in the Bank’s network and with enough access to override any local security settings and hiding in plain sight for months gaining an understanding of banks business operations and collecting user credentials and other information to get into the Swift server. This helped the hackers to turn one of Swift’s defining features — its global reach — into a vulnerability.

SWIFT’s statement

In a statement provided to Information Security Media Group, SWIFT notes that it is aware of the risks and was taking steps to help banks shore up security.

“We understand that the malware is designed to hide the traces of fraudulent payments from customers’ local database applications and can only be installed on users’ local systems by attackers that have successfully identified and exploited weaknesses in their local security,” the statement says. “We have developed a facility to assist customers in enhancing their security and to spot inconsistencies in their local database records.”

“However, the key defense against such attack scenarios remains for users to implement appropriate security measures in their local environments to safeguard their systems – in particular those used to access SWIFT – against such potential security threats. Such protections should be implemented by users to prevent the injection of malware into, or any misappropriation of, their interfaces and other core systems.”

Do we pay enough attention to our security controls and the technology debt?

All the folks working in a software development related role have come across the dilemma of having to prioritise between features focused on enhancing security controls in applications/products that are currently LIVE or delivering features that would result in additional functionality. There is also the technical debt backlog that we accrue because of multiple reasons such as lack of time or understanding. These technical debts might sometimes lead to security loopholes in our system that would make it easier for someone with malicious intent to break into our systems and cause damages.
More often than not, teams put off paying-off this technical debt or making sure proper security controls or practices are in place to have a fool-proof system that is unbreakable or is easy to recover before it is too late. Bangladesh bank made the mistake of not having proper firewall and other network security features in place that would have isolated and protected the SWIFT server, hindering the hackers’ actions to get into the network using the malware and would have saved millions for them which was lost in a matter of hours.

Simpsons Against DevOps

@simpsonsops

"it has been scheduled"

02:01 AM - 27 Nov 2020

Aftermath

The New York Fed has now set up a 24-hour hotline for emergency calls from some 250 account holders, mostly central banks, around the world. SWIFT has advised banks using the SWIFT Alliance Access system to strengthen their cyber security posture and ensure they are following SWIFT security guidelines. The case threatened to reinstate the Philippines to the Financial Action Task Force on Money Laundering blacklist of countries that made insufficient efforts against money laundering. The Bangladesh Bank continued its efforts to retrieve the stolen money and had only recovered about $15 million, mostly from a gaming junket operator based in Metro Manila. In February 2019, the Federal Reserve pledged it would help Bangladesh Bank recover the money and SWIFT has also decided to help the central bank rebuild its infrastructure.

This case was an interesting research for me and I have written this blog post based on multiple articles that I have read online. I may have gotten certain details wrong as I am not an expert, please reach out to me in this case.

I have written this post to highlight how important software security practices are and how a minor oversight of these security controls could lead to individuals causing disruption remotely even with trusted systems in place like SWIFT that is used worldwide for huge amounts of money transfers.

I hope you enjoyed reading this post.

Thank you!

How a developer broke the internet by un-publishing his package containing 11 lines of code

chaitanya.dev — Sun, 22 Nov 2020 13:52:27 +0000

All Javascript developers might have used npm at some point in their lifetime. npm is the default package manager for node.js. For those who don’t know what npm is, npm – short for Node Package Manager is a package manager for the JavaScript programming language. npm, Inc. is a subsidiary of GitHub. npm can manage packages that are dependencies of a particular project, allowing users to reuse modules or pieces of code that are already distributed and present in npm’s remote registry. A package, that your project depends on, can itself have a dependency on another package, which has a dependency on another package and so on.. But the good thing is, with npm you don’t have to worry about the other dependencies, npm handles it for you and does all of it for free.

Now that we know what npm is and how it works, let’s see what happened on 22nd March 2016 that caused highly used packages like React, Node, Babel etc to break with multiple JavaScript programmers around the world receiving a strange error message when they tried to run their code.

Background

Azer Koçulu is an open-source developer who had been publishing and maintaining his packages on npm for other developers to use and include in their packages. Out of his ~270 packages on npm, one of them was called kik, which helped programmers set up templates for their projects.
Kik also happens to be the name of a freeware instant messaging mobile app available on both android and iOS, from the company Kik interactive based in Ontario, Canada.

The E-mail

One fine day, Koçulu received an email from one of Kik’s patent agents, asking him to rename his package called kik as they were planning to publish a package on npm and Koçulu’s NPM package could have caused a confusion. The entire e-mail thread can be found here but to give you the gist, Azer declined Kik’s request. Bob Stratton, Kik’s patent agent, put forward Kik’s request for the name kik to NPM team, again citing the company’s trademark and potential confusion.
NPM decided to side with Kik and took kik away from Azer, handing the name over to the company.

The Liberation of Modules

On finding out that NPM had sided with the corporate, Koçulu wrote to NPM saying that he wanted all of the packages he had published on npm taken down, or they should let him know how he can take them all down quickly. Two days after Koçulu sent this email, on Tuesday 22nd March, Programmers all over the world were left staring at broken builds and failed installations. Out of the multiple lines of errors, one of the lines read as :

npm ERR! 404 'left-pad' is not in the npm registry.

What this error means is that, the code that you’re trying to build/run requires a package called left-pad that does not exist in the npm registry (the one that we talked about at the start of this post) . Where did this package named ‘left-pad’ go?
It seems Koçulu did exactly what he had written in his email, he unpublished all his packages from npm and left-pad was one of the packages published by Koçulu. He wrote a blog post explaining why he had unpublished all his modules. “This situation made me realize that NPM is someone’s private land where corporate is more powerful than the people, and I do open source because Power To The People,” Koçulu said in his blog.

Un-Un-publishing left-pad

To fix all the failing projects and packages around the world, on 23rd March Laurie Voss, CTO and cofounder of NPM, decided to do something unconventional and restore the unpublished left-pad 0.0.3 that apps required on NPM. His tweet read “Un-un-publishing is an unprecedented action that we’re taking given the severity and widespread nature of breakage, and isn’t done lightly.”
With that, all the failing packages started building and running successfully thus fixing the internet. Laurie also said “Even within npm we’re not unanimous that this was the right call, but I cannot see hundreds of builds failing every second and not fix it.” while I agree with him on this, it also made me wonder and think about a couple of things that we are gonna talk about next.

What was left-pad ?

Let’s have a look at the contents of left-pad and try to figure out why were so many projects all over the world making use of this package.
left-pad, as the name suggests, pads the lefthand-side of strings with characters or spaces and the entire package contains only 11 lines of code. 11 lines . This is left-pad in it’s entirety.

module.exports = leftpad; 
function leftpad (str, len, ch) {
   str = String(str);
   var i = -1;
   if (!ch && ch !== 0) ch = ' ';
   len = len - str.length;
   while (++i < len) {
     str = ch + str;
   }
   return str;
}

Why did this cause so many packages on NPM to break?

React, Babel, and a bunch of other high-profile packages on NPM broke on March 22nd 2016 because all these packages and projects took on a dependency for a simple left padding string function on the package left-pad. Most programmers, who were facing these build errors, might not have even heard the name left-pad but their code was breaking because their apps were dependent on some packages, which in turn were dependent on some packages and down the line, one of the dependencies might have been left-pad.
Ideally programmers don’t have to worry about all these dependencies as NPM takes care of this for them and has always been reliable in doing so. In this case though, the package was unpublished, and there was no way npm could find this dependency, thus causing these unforeseen errors.

Why did so many packages depend on left-pad to, well, left pad?

With only 11 lines of code, left-pad is just a function exported as a module that so many packages took a dependency on rather than the developers of those packages writing a basic function to left pad by themselves. It would hardly take a few minutes for a well versed programmer to write such a function and yet they decided to depend on another developer for this. Tying together multiple third-party dependencies or packages and developing a project with minimal code should not be considered ideal. Any issues with the third-party dependency would cause your code to break and you’ll be dependent on another developer to fix their work so that you can get your project to start functioning properly again. This has to be considered a serious issue when web services like Facebook, for example, become indirectly dependent on code written by other programmers that don’t even know the impact their code might be having down the line.

How many dependencies are too many dependencies?

Have we become so lazy that we require a package to check if an object isArray? The package contains one line of code and is downloaded 39,001,468 times weekly as I write this post. Do we really need to publish packages containing just one function? And should we be creating dependencies on packages for few lines of code that we can easily write?
I think this method of software development needs to change and dependencies should be created on ‘libraries‘ that provide an array of interrelated complex functionalities. Why would you import a package to add, subtract, multiply instead of importing a package that provides all Math functionalities?
Ease of writing code by adding dependency after dependency for minimal functionality, leads to difficulties in maintaining code when you don’t have control on third-party packages and increases the points of failure.
There should be least amount of dependencies and only on well-known libraries that offer many complex functionalities that would be hard/time consuming to write by oneself so that the risk of errors are worth it.

To all programmers, all I say is that in the near future if there is a small and simple functionality that you need to use in your project, choose to write a few lines yourself rather than add a dependency on an unknown package. Take the frustrated programmers as an example who chose to directly or indirectly depend on 11 lines of code instead of writing it themselves.

I have not written this blog to discuss the ethics/legality of the actions taken by Kik, NPM or Azer as I am no expert in that area and my opinions on that would not be valid.

I have written this blog to discuss how the methodology of depending on so many small APIs( should 1-liner functions classify as API? ) can be disastrous and to re-think this way of programming.
I’d also suggest you to go through this reddit post to see what other programmer’s opinions are on this event.

All of this blog’s content comes from google search, reading articles etc and I may have been wrong in some places, please reach out to me if I have missed something.

Note: The featured image is by xkcd and all credits go to the artist.

I hope you enjoyed reading this post!
Thank you.

Fetch Service Status & Storage Info remotely in C# using WMI

chaitanya.dev — Sun, 20 Sep 2020 08:24:39 +0000

I was recently working on a pet project where I wanted to find the status of some services on multiple server instances and display this on a web page so that I have a single page to look at if I want to check the status if any of my application services have stopped on any of the sever instances. I also wanted to be able to Start or Stop these services from the same page. With this in place I no longer needed to log into any of the remote servers to check up on the services. And just as an add-on, I also wanted to be able to look at the storage information on these servers so that I can take action on any of the drives that are filling up quickly.

While looking around trying to figure out the easiest way to create such a Web App I came across Windows Management Instrumentation which is the infrastructure for management data and operations on Windows-based operating systems. Developers can use WMI to remotely monitor the hardware and software on remote computers. Remote connections for managed code are accomplished through the System.Management namespace.

Let’s have a look at how we can go about fetching Service and Storage info remotely using WMI with the System.Management namespace.

Setting up ConntectionOptions

The ConnectionOptions class from the System.Management namespace specifies all settings required to make a WMI connection. You can create a ConnectionOptions object and use it without setting up any of it’s properties to connect to the remote computer with default connection options. In this example I have setup some of the properties based on my connection requirements to the remote servers.

ConnectionOptions connection = new ConnectionOptions();
connection.Username = "User";
connection.Password = "AStrongPassword";
connection.Authority = "ntlmdomain:DOMAINNAME";
connection.EnablePrivileges = true;
connection.Authentication = AuthenticationLevel.Default;
connection.Impersonation = ImpersonationLevel.Impersonate;

Connecting to remote server using ManagementScope

The ManagementScope class represents a scope for management operations. You can initialize a new ManagementScope with a specific path and then connect the scope object to a namespace on a remote computer using the ConnectionOptions object.

ManagementScope scope = new ManagementScope(
                        $"\\\\{serverName}\\root\\CIMV2", connection);
scope.Connect();

Fetching management information using ManagementObjectSearcher and ObjectQuery

The ObjectQuery class is used to specify a query in the ManagementObjectSearcher.
The ManagementObjectSearcher class is used to retrieve a collection of management objects based on a specified query. This class is one of the more commonly used entry points to retrieving management information. In this example I have created the ObjectQuery to fetch LogicalDisk info and I am using ManagementObjectSearcher to get each disk’s info.

ObjectQuery query = new ObjectQuery("SELECT * FROM Win32_LogicalDisk");

ManagementObjectSearcher searcher = new ManagementObjectSearcher(scope, query);

foreach (ManagementObject managementObject in searcher.Get())
{
   Console.WriteLine("Drive Name :" +
                      managementObject["Name"].ToString());
   Console.WriteLine("Volume Size :" +
                      managementObject["Size"].ToString());
   Console.WriteLine("Free Space :" +
                      managementObject["FreeSpace"].ToString());
}

Similarly we can also fetch Service information like below

ObjectQuery query = new ObjectQuery("SELECT * FROM Win32_Service");

ManagementObjectSearcher searcher = new ManagementObjectSearcher(scope, query);

foreach (ManagementObject managementObject in searcher.Get())
{
   Console.WriteLine("Service Name :" +
                      managementObject["DisplayName"].ToString());
   Console.WriteLine("Service State :" +
                      managementObject["State"].ToString());
}

Using InvokeMethod() on a ManagementObject

We can use the method InvokeMethod() on a ManagementObject to perform operations on it asynchronously. In the below example, I have created the ObjectQuery to fetch the ManagementObject to represent a specific service who’s DisplayName is stored in service_name. I am then calling InvokeMethod() to invoke the StartService method with no options which will start the service on the remote server asynchronously.

ObjectQuery query = new ObjectQuery("SELECT * FROM Win32_Service where
                                     DisplayName= '" + service_name + "'");

using (ManagementObjectSearcher searcher = new
                                   ManagementObjectSearcher(scope, query))
{
  foreach (ManagementObject myservice in searcher.Get())
  {
     myservice.InvokeMethod("StartService", null);
  }
}

Thus we have seen how we can use System.Management namespace and it’s classes to fetch management info from Windows machines and perform actions remotely. You can see the entire ASP .Net Core Web App that I have created on my Github profile here.

I hope you found this interesting. Thanks for reading!

Writing unit tests for HttpClient using NUnit and Moq in C#

chaitanya.dev — Mon, 07 Sep 2020 16:23:09 +0000

You have a class that sends a GET request using HttpClient and consumes the response and performs further actions. Now that you have written this class you want to go ahead and write Unit Tests using NUnit for this class so that you can make sure the right URI is being called with the correct Request Headers and Request Method. How do you go about this?

Let’s take a look at the demo project that I have created to understand this better. You can find this project on my Github Page here.

I have created a class called EmployeeApiClientService that has a method called GetEmployeeAsync() that takes an employeeid as a input parameter, sends a GET request to the employee API with the employeeid in the uri and returns the Employee object back.

public class EmployeeApiClientService
{
    private readonly HttpClient employeeHttpClient;

    public EmployeeApiClientService(HttpClient httpClient)
    {
        employeeHttpClient = httpClient;
    }

    //environment specific variables should always be set in a seperate config file or database. 
    //For the sake of this example I'm initialising them here.
    public static string testDatabase = "SloughDB";
    public static string environment = "TEST";

    public async Task<Employee> GetEmployeeAsync(int employeeId)
    {
        Employee employee = null;

        //Add headers
        employeeHttpClient.DefaultRequestHeaders.Add("Accept", "application/json");
        employeeHttpClient.DefaultRequestHeaders.Add("tracking-id", Guid.NewGuid().ToString());

        //Conditional Headers
        if (environment == "TEST")
        {
            employeeHttpClient.DefaultRequestHeaders.Add("test-db", testDatabase);
        }

        HttpResponseMessage response = await employeeHttpClient.GetAsync($"http://dummy.restapiexample.com/api/v1/employee/{employeeId}");
        if (response.IsSuccessStatusCode)
        {
            employee = JsonConvert.DeserializeObject<Employee>(await response.Content.ReadAsStringAsync());
        }
        return employee;
    }
}

Once you have finished writing your class, you start writing your unit tests but you are stuck. There’s no interface for the HttpClient class which you can use to Moq and you do not want to hit the actual end-pints during the unit tests. What do you do here?

There are multiple ways to tackle this problem :

Method 1 : Write a wrapper class for HttpClient class
This method would require you to write a wrapper class eg. HttpClientWrapper and implement all of HttpClient’s methods in your wrapper class and then use this wrapper class as a dependency instead of HttpClient in your actual class.
Then you can Mock this Wrapper class in your unit tests and verify the request details. For me, this method seemed like too much work and not a neat implementation.

So I looked around a bit and found out that the HttpClient has a constructor overload that takes a HttpMessageHandler.

public HttpClient(HttpMessageHandler handler);

And that’s how I came to the second method.

Method 2 : Mock HttpMessageHandler and pass it to your HttpClient class
HttpMessageHandler has one protected method SendAsync() which is the underlying method called by all of HttpClient’s GET/POST/PATCH/PUT Async methods. All we have to do is mock this class and setup SendAsync to accept and return our desired values as per our test cases. We can use Moq for this purpose.

Moq is a Mocking Framework used in .NET to isolate units to be tested from the underlying dependencies. We can create a Mock for HttpMessageHandler and pass it to the overloaded constructor for HttpClient.

Let’s see how we can implement this.

Mock HttpMessageHandler

We will create a mock object of HttpMessageHandler using Moq and pass it to the HttpClient class constructor and pass this HttpClient object to our EmployeeApiClientService constructor in the Test SetUp.

EmployeeApiClientService employeeApiClientService;
Mock<HttpMessageHandler> httpMessageHandlerMock;

[SetUp]
public void setUp()
{
    httpMessageHandlerMock = new Mock<HttpMessageHandler>(MockBehavior.Strict);
    HttpClient httpClient = new HttpClient(httpMessageHandlerMock.Object);
    employeeApiClientService = new EmployeeApiClientService(httpClient);
}

Setup SendAsync method

Moq does not allow us to directly setup SendAsync() method because this method is protected in the HttpMessageHandler class and cannot be accessed outside the class.
We can use the Moq.Protected api, which gives us some additional methods on the mocked object, where we can access the protected members using their names using the .Protected() method.
We will now Setup SendAsync() method of the mocked HttpMessageHandler object so that it returns StatusCode 200 with Employee object in json format for which we will use the ReturnsAsync() method and we’ll make this object verifiable by using Verify() method so that we can verify the number of calls to this method, request details etc. in the assertion section.

httpMessageHandlerMock.Protected().Setup<Task<HttpResponseMessage>>(
    "SendAsync",
    ItExpr.IsAny<HttpRequestMessage>(),
    ItExpr.IsAny<CancellationToken>()
    ).ReturnsAsync(new HttpResponseMessage()
    {
       StatusCode = HttpStatusCode.OK,
       Content = new StringContent(JsonConvert.SerializeObject(new Employee()), Encoding.UTF8, "application/json")
    }).Verifiable();

Verify the call to SendAsync

In our assertion section of the Unit Test we’ll verify that the SendAsync() method is called only once, it’s called with a GET request, the request contains the expected target uri and the request contains all the required headers.

httpMessageHandlerMock.Protected().Verify(
    "SendAsync",
    Times.Exactly(1), 
    ItExpr.Is<HttpRequestMessage>(req =>
    req.Method == HttpMethod.Get  
    && req.RequestUri.ToString() == targetUri // veryfy the RequestUri is as expected
    && req.Headers.GetValues("Accept").FirstOrDefault() == "application/json" 
    && req.Headers.GetValues("tracking-id").FirstOrDefault() != null 
    && environment.Equals("TEST") ? 
                      req.Headers.GetValues("test-db").FirstOrDefault() == testDatabase : 
                      req.Headers.GetValues("test-db").FirstOrDefault() == null 
    ),
    ItExpr.IsAny<CancellationToken>()
    );

Complete Test Class

The complete test class with one test case to test the Request is created correctly looks something like this.

[TestFixture]
class EmployeeApiClientServiceTests
{
    EmployeeApiClientService employeeApiClientService;
    Mock<HttpMessageHandler> httpMessageHandlerMock;

    //environment specific variables should always be set in a separate config file or database. 
    //For the sake of this example I'm initialising them here.
    string testDatabase = "SloughDB";
    string environment = "TEST";

    [SetUp]
    public void setUp()
    {
        httpMessageHandlerMock = new Mock<HttpMessageHandler>(MockBehavior.Strict);
        HttpClient httpClient = new HttpClient(httpMessageHandlerMock.Object);
        employeeApiClientService = new EmployeeApiClientService(httpClient);
    }


    [Test]
    public async Task GivenICallGetEmployeeAsyncWithValidEmployeeId_ThenTheEmployeeApiIsCalledWithCorrectRequestHeadersAsync()
    {
        //Arrange
        int employeeId = 1;
        string targetUri = $"http://dummy.restapiexample.com/api/v1/employee/{employeeId}";
        //Setup sendAsync method for HttpMessage Handler Mock
        httpMessageHandlerMock.Protected().Setup<Task<HttpResponseMessage>>(
            "SendAsync",
            ItExpr.IsAny<HttpRequestMessage>(),
            ItExpr.IsAny<CancellationToken>()
            )
            .ReturnsAsync(new HttpResponseMessage()
            {
                StatusCode = HttpStatusCode.OK,
                Content = new StringContent(JsonConvert.SerializeObject(new Employee()), Encoding.UTF8, "application/json")
            })
            .Verifiable();

        //Act
        var employee = await employeeApiClientService.GetEmployeeAsync(employeeId);

        //Assert
        Assert.IsInstanceOf<Employee>(employee);

        httpMessageHandlerMock.Protected().Verify(
            "SendAsync",
            Times.Exactly(1), // verify number of times SendAsync is called
            ItExpr.Is<HttpRequestMessage>(req =>
            req.Method == HttpMethod.Get  // verify the HttpMethod for request is GET
            && req.RequestUri.ToString() == targetUri // veryfy the RequestUri is as expected
            && req.Headers.GetValues("Accept").FirstOrDefault() == "application/json" //Verify Accept header
            && req.Headers.GetValues("tracking-id").FirstOrDefault() != null //Verify tracking-id header is added
            && environment.Equals("TEST") ? req.Headers.GetValues("test-db").FirstOrDefault() == testDatabase : //Verify test-db header is added only for TEST environment
                                            req.Headers.GetValues("test-db").FirstOrDefault() == null
            ),
            ItExpr.IsAny<CancellationToken>()
            );
    }
}

Thus we have seen how we can easily write unit test cases to test HttpClient calls to verify various aspects of the requests and how your class processes the responses. This method can also be used for POST/PUT/PATCH requests as all these methods in HttpClient use HttpMessageHandler’s SendAsync() method under the hood.

I hope you found this interesting. Thanks for reading!

SQL Server Service Broker for Asynchronous Applications

chaitanya.dev — Sat, 05 Sep 2020 09:41:18 +0000

Imagine an online retail web application which has an order fulfilment system that comes into play whenever a customer wants to place an order. This system might have different business processes such as payment processing, a CRM, inventory management, shipping etc. and each of them have their own databases. In a scenario when one of the systems is down, customers may not be able to place an order if there is no way to queue their order request and guarantee that their order request will be delivered to subsequent systems for processing. This is where an asynchronous queueing and messaging system like SQL Service Broker comes into picture.

SQL Server Service Broker provides native support for messaging and queuing in the SQL Server Database Engine and Azure SQL Managed Instance. Application developers can use Service Broker to distribute data workloads across several databases without programming complex communication and messaging internals. Service Broker ensures that all tasks are managed in the context of transactions to assure reliability and technical consistency. It’s part of SQL Server so if you’re already utilising SQL Server for your project, then there’s no additional licensing required.

Now that we know what SQL Service Broker is, let’s look at how we can use Service Broker in our order fulfilment system. We can leverage Service Broker to exchange data between the business processes. Even if there is a payment processing outage, customer’s order will be accepted and a request for payment processing will be queued and held as a message. The system can continue accepting other orders and queueing payment request messages. Once the payment processing system is back online, it’ll dequeue and process the payment request messages. Once done, the subsequent CRM systems will be updated with the sale information and the shipping systems may do the needful and ship the products to the customer that made the order. Service Broker ensures that the payment processing request has reached the destination system and as the processing task will be done in the context of a transaction it ensures consistency.

Let’s try and set up a simple Service Broker queue and see how we can send or receive messages through Service Broker.

Things you’ll need:

SQL Server
- Developer Edition available here.
Microsoft Docs
- Documentation for reference here.

Enabling Service Broker in database

Let’s start off by creating a a database and enabling Service Broker for that database.
Note: To ensure that no users are connected to the database when you enable Service Broker we’ll use the ROLLBACK IMMEDIATE option and this works as we are currently executing this in our DEV database.

CREATE DATABASE PaymentProcessingDB
GO
ALTER DATABASE PaymentProcessingDB
      SET ENABLE_BROKER
      WITH ROLLBACK IMMEDIATE;
GO

Create Message Types

Message Type is the basic object of the messaging infrastructure that is used to define what kind of messages can and cannot be sent.
You can set the type of a message by defining the ‘VALIDATION’ of a message type as follows EMPTY, NONE, WELL_FORMED_XML, VALID_XML WITH SCHEMA COLLECTION.
We’ll create a PaymentRequestMessage which will be sent by the service initiating the DIALOG and a PaymentResponseMessage which will be sent back to the Initiator Service by the Target Service.

CREATE MESSAGE TYPE
       [PaymentRequestMessage]
       VALIDATION = WELL_FORMED_XML;
CREATE MESSAGE TYPE
       [PaymentResponseMessage]
       VALIDATION = WELL_FORMED_XML;

Create Contract

The CONTRACT is used to define the kind of message that will be sent by the INITIATOR – the party that sends the first message and the TARGET – the party that receives the message and sends a response.

CREATE CONTRACT [PaymentProcessContract]
      ([PaymentRequestMessage]
       SENT BY INITIATOR,
       [PaymentResponseMessage]
       SENT BY TARGET
      );

Create Queues

A queue is a storage space where the messages reside. You can also query queues to see what messages are currently present in it.
We will be creating two queues, the TagetQueue will hold messages sent be the INITIATOR to the TARGET and the InitiatorQueue will hold messages sent by the TARGET back to the INITIATOR.

CREATE QUEUE PaymentProcess_TargetQueue;
GO
CREATE QUEUE PaymentProcess_InitiatorQueue;
GO

Create Services

A service is created on top of a queue which defines what contract is to be adhered to send messages to that queue. A queue can have multiple services adding messages to it by adhering to different contracts.
For our example we are going to create two services for the Target and Initiator queues that adhere to the PaymentProcessContract.

CREATE SERVICE
       [PaymentProcess_TargetService]
       ON QUEUE PaymentProcess_TargetQueue
       ([PaymentProcessContract]);
CREATE SERVICE
       [PaymentProcess_InitiatorService]
       ON QUEUE PaymentProcess_InitiatorQueue
       ([PaymentProcessContract]);

Send a Request Message

To send a message from the InitiatorService to the TargetService we will start a DIALOG and mention the CONTRACT that will be used for this DIALOG. We will also use a unique identifier(GUID) for this DIALOG which will be used for all the conversations that will be sent to the TargetQueue.

Once the DIALOG has started, we can send messages on the CONVERSATION using the unique identifier used to start the DIALOG and mention the MESSAGE TYPE and the message body which, for our example, contains a PaymentProcessingRequest number.

DECLARE @conversation_handle UNIQUEIDENTIFIER;
DECLARE @message_body XML;

BEGIN TRANSACTION;


--Begin conversation
BEGIN DIALOG @conversation_handle
     FROM SERVICE [PaymentProcess_InitiatorService]
     TO SERVICE N'PaymentProcess_TargetService'
     ON CONTRACT [PaymentProcessContract]
     WITH ENCRYPTION = OFF;

SELECT @message_body = N'<PaymentRequestMessage>PAYREQ001</PaymentRequestMessage>';

--send message on conversation using the same GUID
SEND ON CONVERSATION @conversation_handle
     MESSAGE TYPE [PaymentRequestMessage] 
(@message_body);

COMMIT TRANSACTION;
GO

You might have noticed that queues are never mentioned while sending messages on a CONVERSATION, as a DIALOG can start only between two services that logically defines the contract to be adhered to send messages to the queue.
Once the message has been sent to the TargetService, we can view the message in the TargetQueue using a simple select statement as below.

SELECT *, CAST(message_body AS XML) AS message_body_xml
FROM PaymentProcess_TargetQueue
;

Receiving and Processing a Request Message

The initiator has sent the PaymentRequestMessage to the Payment processing system and it is now free to accept other orders. The Payment processing system will receive messages from the TargetQueue asynchronously and respond back to the INITIATOR once the processing is done.

To process the request message, we will have to first RECEIVE the message from the TargetQueue. Once the message is received we can enter the logic to process the request message, but for simplicity we are just going to display the message contents here. When we receive the message body, we also receive the conversation handle and message type.

We can use this conversation handle to send a response message on the initiator queue on the same conversation. We can also check the message type we received and process the message accordingly. As said earlier, in this example we are just going to respond back with a PaymentResponseMessage containing the PaymentProcessingRequest number indicating we have finished processing the Payment Request.

-- Receive the request and send a reply
DECLARE @conversation_handle UNIQUEIDENTIFIER;
DECLARE @message_body XML;
DECLARE @message_type_name sysname;

BEGIN TRANSACTION;

WAITFOR
( RECEIVE TOP(1)
    @conversation_handle = conversation_handle,
    @message_body = message_body,
    @message_type_name = message_type_name
  FROM PaymentProcess_TargetQueue
), TIMEOUT 1000;

SELECT @message_body AS ReceivedPaymentRequestMsg;

--check message type
IF (@message_type_name = N'PaymentRequestMessage')
BEGIN
     DECLARE @reply_message_body XML;


--You can enter the logic to process the message here
     SELECT @reply_message_body = N'<PaymentResponseMessage>PAYREQ001</PaymentResponseMessage>';

     SEND ON CONVERSATION @conversation_handle
          MESSAGE TYPE [PaymentResponseMessage]
     (@reply_message_body);
END

SELECT @reply_message_body AS PaymentResponseMessage;

COMMIT TRANSACTION;
GO

You can then check for the PaymentResponseMessage sent to the InitiatorQueue by executing the below select query on the InitiatorQueue.

-- Check for the reply message in the initiator queue
SELECT *, CAST(message_body AS XML) AS message_body_xml
FROM PaymentProcess_InitiatorQueue;
GO

Understanding the relationship between DIALOG, CONTRACT, SERVICE and QUEUE

By sending the response message on the same DIALOG by using the same conversation handle, the contract PaymentProcessContract specified in the DIALOG will be used to send the message.
The PaymentProcessContract specifies PaymentResponseMessage is sent by Target to the Initiator and the DIALOG also specifies the conversation initiator is the PaymentProcess_InitiatorService.
The PaymentProcess_InitiatorService logically represents the PaymentProcess_InitiatorQueue and thus the response message arrives at the InitiatorQueue.

I hope this helps you understand the relationship between DIALOG, CONTRACT, SERVICE and QUEUE.

Ending the Conversation

Now that we have sent the PaymentResponseMessage back to the InitiatorQueue we are going to use the same method to RECEIVE the message from the queue.
If the received message is of the type PaymentResponseMessage we will use END CONVERSATION with the conversation handle received from the response message.

The idle conversations occupy space and are of no further use, so it is always a good practice to END CONVERSATION once you have received the desired response and clear out the conversation.

DECLARE @message_body XML;
DECLARE @conversation_handle UNIQUEIDENTIFIER;
DECLARE @message_type_name sysname;

BEGIN TRANSACTION;

WAITFOR
( RECEIVE TOP(1)
    @conversation_handle = conversation_handle,
    @message_body = CAST(message_body AS XML),
    @message_type_name = message_type_name
  FROM PaymentProcess_InitiatorQueue
), TIMEOUT 1000;

IF (@message_type_name = N'PaymentResponseMessage')
BEGIN
    END CONVERSATION @conversation_handle;
END

SELECT @message_body AS ReceivedPaymentResponseMessage;

COMMIT TRANSACTION;
GO

We have ended the conversation on the Initiator side, this sends an End Dialog message to the TargetQueue. If you check the TargetQueue using the below query, you should see the EndDialog message.

--Check for EndDialog message in TargetQueue
SELECT *, CAST(message_body AS XML) AS message_body_xml
FROM PaymentProcess_TargetQueue;
GO

We also need to end the conversation from the Target side. We will do something similar to what we did with the InitiatorQueue but here we’ll look for a message type of EndDialog rather than PaymentResponseMessage.
Ending the Conversation here does nothing but just cleans up the conversation. There is no message sent back to the initiator.

-- Receive the End Dialog and clean up
DECLARE @conversation_handle UNIQUEIDENTIFIER;
DECLARE @message_body XML;
DECLARE @message_type_name sysname;

BEGIN TRANSACTION;

WAITFOR
( RECEIVE TOP(1)
    @conversation_handle = conversation_handle,
    @message_body = CAST(message_body AS XML),
    @message_type_name = message_type_name
  FROM PaymentProcess_TargetQueue
), TIMEOUT 1000;

--check if message type is EndDialog
IF (@message_type_name = N'http://schemas.microsoft.com/SQL/ServiceBroker/EndDialog')
BEGIN
     END CONVERSATION @conversation_handle;
END

COMMIT TRANSACTION;
GO

With this we have successfully end the CONVERSATION or DIALOG associated with the conversation_handle. We can have a look at both the queues, they should be empty unless a new PaymentRequestMessage is being processed.

-- Check for a message in the target queue
SELECT *, CAST(message_body AS XML) AS message_body_xml
FROM PaymentProcess_TargetQueue;
GO

-- Check for a  message in the initiator queue
SELECT *, CAST(message_body AS XML) AS message_body_xml
FROM PaymentProcess_InitiatorQueue;
GO

This is now the end of message processing between two systems that has happened asynchronously. The TargetQueue can have PaymentRequests added at any point of time and the Payment Processing System can dequeue and process messages at any point of time and respond back with the results.

Service Broker can also be used for batch processing of data wherein you can queue messages during business hours and process the whole chunk of messages during off-business hours or process it periodically during the day. One of the advantages of using SQL Server Service Broker is that as the messages and queues reside in the database, whenever you backup and restore the database all of your messages and queues are retained.

Thus, we have had a glimpse of both queueing and messaging technology that SQL Server Service Broker offers and also the basic components required for a simple asynchronous application. Service Broker has a lot more to offer and you can get more information here.

I hope you found this interesting. Thank you for reading!

Using SQLite database with EntityFramework Core 3

chaitanya.dev — Sat, 29 Aug 2020 18:59:47 +0000

Recently I have been working with .Net Core web applications which used Entity Framework with SQL Server Database. I wanted to figure out how to use EF Core 3 with a portable database like SQLite. This is how I got the idea for my latest pet project, a simple .Net Core Web API that uses EF Core 3 to store and retrieve data from a SQLite Db.
With this blog post I aim to demonstrate how easy it is to create this project with Visual Studio 2019.

You can find my source code here at my GitHub.

Things required for this project :

Visual Studio 2019
- Community edition available here.
Postman
- Download available here.
Microsoft Docs
- There’s no better reference than the official documentation!

Creating New Project

On Visual Studio, choose to create a new project and in the templates section select ASP.NET Core Web Application and click Next.

In the next screen, give a name to your Project and click Create.
In the final screen, choose API template and make sure ASP.NET Core 3.1 is selected in the drop-down menu.

Once you click Create you should see your project in the Solution Explorer.
Get rid of the sample WeatherForecast Controller and class as we will be creating our own.
Add three new folders called Contexts, Controllers and Entities to your project.
Once done, your project structure should look like this.

Adding EF Core and related packages

Now we will be adding EF Core NuGet package for SQLite using the Package Manager Console using the below command. This will also add the dependency EntityFramework Core package.
Tools > NuGet Package Manager > Package Manager Console

Install-Package Microsoft.EntityFrameworkCore.Sqlite

Once installation is complete you should see Microsoft.EntityFrameworkCore.Sqlite & Microsoft.EntityFrameworkCore in your Project at Dependencies -> Packages .
Now we are all set to start creating our entities and the DbContext.

Creating the Model

EF Core can read and write entity instances from/to the database, and if you’re using a relational database, EF Core can create tables for your entities via migrations.
By convention, each entity type will be set up to map to a database table with the same name as the DbSet property that exposes the entity.

For our example, let’s create an Entity called Athlete that’ll be used to store an Athlete’s information.
Add a new class file called Athlete.cs in the Entities folder in our project. The class will have 3 properties Name, Age and Sport along with an identifier called Id.

using System.ComponentModel.DataAnnotations;

namespace SQLiteDemoWebApi.Entities
{
    public class Athlete
    {
        [Key]
        public int id { get; set; }

        [Required]
        [MaxLength(100)]
        public string Name { get; set; }

        public int Age { get; set; }

        [Required]
        [MaxLength(100)]
        public string Sport { get; set; }
    }
}

Now let’s create a DbContext which will be used by EF Core to gather details about the application’s entity types that are exposed in DbSet properties in the context and how they map to a database schema.

Add a new class file called AthleteSQLiteDbContext.cs in the Contexts folder in our project which will expose the Athlete entity as Athletes DbSet.

using Microsoft.EntityFrameworkCore;
using SQLiteDemoWebApi.Entities;

namespace SQLiteDemoWebApi.Contexts
{
    public class AthleteSQLiteDbContext : DbContext
    {
        public DbSet<Athlete> Athletes { get; set; }

        public AthleteSQLiteDbContext(DbContextOptions<AthleteSQLiteDbContext> options) : base(options)
        { }
        protected override void OnModelCreating(ModelBuilder modelBuider)
        {
            base.OnModelCreating(modelBuider);
        }
    }
}

Adding Migrations and Updating the Database

Add-Migration command scaffolds a migration to create the initial set of tables for the model i.e. the details we have mentioned in the DbContext and the corresponding Entities.

Update-Database command creates the database and applies the new migration to it.

To execute these commands we need to install the PMC tools for EF Core. This can be done by executing the below command on the Package Manager Console.

Install-Package Microsoft.EntityFrameworkCore.Tools

Once the package is installed you should see Microsoft.EntityFrameworkCore.Tools added to your Dependencies.

Before we start creating migrations, we need to add a connection string for our SQLite database. We’ll be adding this connection string to the appsettings.json file. For the sake of this project, we’ll be creating our SQLite database at the root directory.

{
  "Logging": {...},
  "ConnectionStrings": {
    "SQLite": "Data Source=Athlete.db"
  },
  "AllowedHosts": "*"
}

We also need to make sure that AthleteSQLiteDbContext is added to the container at startup. This can be done by adding it to the services collection in the ConfigureServices() method in Startup.cs.

The UseSqlite method requires the connection string which we fetch from appsettings.json using the Configuration class.

The ConfigureServices method should look like this:

public void ConfigureServices(IServiceCollection services)
{
    services.AddControllers();
    services.AddDbContext<AthleteSQLiteDbContext>(o =>
                o.UseSqlite(Configuration.GetConnectionString("SQLite")));
}

We can now add our initial migration to the database with the Add-Migration command.

Add-Migration InitialMigration

This will create a Migrations folder in your project.
Three files are added to your project under the Migrations directory:

XXXXXXXXXXXXXX_InitialMigration.cs–The main migrations file. Contains the operations necessary to apply the migration (in Up) and to revert it (in Down).
XXXXXXXXXXXXXX_InitialMigration.Designer.cs–The migrations metadata file. Contains information used by EF.
AthleteSQLiteDbContextModelSnapshot.cs–A snapshot of your current model. Used to determine what changed when adding the next migration.

The timestamp in the filename helps keep them ordered chronologically so you can see the progression of changes.

Update-Database command can be used to apply migrations to a database. While productive for local development and testing of migrations, this approach isn’t ideal for managing production databases. For our example, we’ll be using this command.

Update-Database

You should see a newly created Athlete.db file in your Project directory after the update-database command is executed successfully.

Creating a Controller for the API

Now that we have the database and it’s connectivity setup, all that’s remaining is to create a controller that’ll use EF Core to fetch and update data from the SQLite database.
As part of this blog post, I do not want to get into the details of creating an API Controller.
So we will be using the Visual Studio scaffolding magic to get a controller created for us that uses the AthleteSQLiteDbContext to GET, PUT, POST, DELETE Athlete data.

To do this, Right-Click on the Controllers directory in your project -> Add -> Controller.
Select API Controller with Actions using EntityFramework and click OK.
Select the Model: Athlete and DbContext: AthleteSQLiteDbContext from the drop-down and click Add.

You should now see a AthletesController.cs file in your Controllers directory. There’s no changes needed to be made in this controller class except, make sure he Controller’s Route is set to "api/Athletes" .

Test the API

To test the API we’ll use Postman so that we can send requests to the API endpoints.
We’ll use the App URL that is mentioned in the Debug section which can be found by Right-click on Project -> Properties -> Debug.

Once your project is running, in Postman we’ll first create a new POST request to the below endpoint so that we can create an Athlete record in the database.
https://localhost:43325/api/Athletes (Note that the port number may differ for you)

Once you get a 201 Created Response, we can be sure that the Athlete data is inserted into the database.

This can also be confirmed by creating a new GET request for the same endpoint URL.
You should receive a response similar to this :

[
    {
        "id": 2,
        "name": "John Doe",
        "age": 35,
        "sport": "Swimming"
    }
]

We have now successfully configured EF Core in our Web API to connect to an SQLite database. I hope you find this helpful and got to learn something new.
As stated earlier, you can find the finished project here at my GitHub page.

Wireless lock with Arduino Uno

chaitanya.dev — Sat, 29 Aug 2020 15:20:26 +0000

Few years ago when I got to know about Arduino Uno, I was really curious about the capabilities of this open-source microcontroller board. For those who might not be familiar with Arduino, it is a popular tool for IoT product development and Arduino Uno is one of their most popular products. It contains everything needed to support the microcontroller. Simply connect it to a computer with a USB cable or power it with a AC-to-DC adapter or battery to get started. You can tinker with your Uno without worrying too much about doing something wrong. It has 14 digital input/output pins (of which 6 can be used as PWM outputs), 6 analog inputs, a 16 MHz ceramic resonator (CSTCE16M0V53-R0), a USB connection, a power jack, an ICSP header and a reset button.

As always when I want to get to know a new piece of technology better, I decide to use it for a pet project. With the Arduino Uno, I decided to make a wireless door lock. For this I used the below components :

Arduino Uno
LED lights
Solenoid Lock
Relay Circuit
Breadboard
Ethernet Shield
Some Android skills for developing the App to control the lock and C skills to write the ‘Sketch’ for Arduino Uno Computer

I got all the components required from a local hardware store, luckily all of it was available from the same place. Make sure you get the authentic Arduino Uno board as there are many aftermarket copies out there which don’t work as well.

Once I had all the components setup on my table, I had no clue where to begin with. Luckily, Arduino has a very helpful getting started guide here. I installed all the necessary softwares and drivers, wrote my first sketch in C to have an LED blink every 5 seconds, connected my Arduino Uno to the PC and uploaded my first sketch. The feeling when you get something to work for the first time, even if it’s something as small as a blinking LED, is just wonderful. I was hooked after that! I wrote multiple programs to do stuff with LEDs. I had a spare motor that I had got for another project and I uploaded a sketch to rotate it clockwise and anticlockwise within intervals.

After some googling (yes, I use that as a verb), I found out that I could use the Arduino Uno as a Web Server. My mind was blown, I quickly wrote a sketch to host a simple web page with 2 buttons: ‘Turn Left’ and ‘Turn Right’. I had my mobile on the same network as Uno and I could access a webpage that had two buttons which when I pressed, I could control the direction in which the motor connected to the Arduino Uno was turning. And slowly I realised how capable this small microcontroller board in my hand was.

Now coming to my actual project, I had a solenoid lock which is very simple to work with. It has a magnet inside it, if you pass electricity through the lock, it will pull the latch in thus leaving the lock in an ‘unlocked’ state. When there’s no electricity passing through, the latch will be out in the ‘locked’ state. After connecting this lock to the Arduino Uno on the breadboard, all I had to do was pass on/off signal on the port it was connected to. For those trying to work with modules like a Solenoid lock, make sure that you get a power adapter as the usb power from a computer won’t be enough to get it to work.

To control the lock remotely, I created a Web server on the Arduino Uno which could be used by an Android app for the lock/unlock functionality similar to the ‘Turn Left’ and ‘Turn Right’ buttons with the motor. There are tons of tutorials out there which will help you with this part and the one that pointed me to the right direction was this .

I had really enjoyed working on this project with my friend Abhishek, as he’s as intrigued with new tech as I am. I know some of you may ask why didn’t I use a Raspberry Pi? The answer is Arduino Uno was good enough for the task. Even though Raspberry Pi is a much stronger device which can practically run an Operating System(Raspbian) on it, I didn’t think it was a good decision to shell out almost thrice the price to get the same thing done.

I hope you’ll enjoyed reading this and if you have any questions please feel free to drop a comment or reach out to me. For those looking for some awesome project ideas involving Arduino, you can check out https://create.arduino.cc/projecthub .