Cinema Seat Reservation System — Part 3: Problems, Decisions and Some Areas Of Improvement

Introduction

In the two previous parts, I finished sharing my main checkpoints while I was building the system till I deployed it. Today, I need to share two interesting problems that faced me and how I took decisions to solve them. I hope someone can benefit from them.
Also, I will mention the areas of improvement in the system, how it can be better and considering their negative consequences on the system. System design is all about Trade-offs.

You can find more decisions regarding the system in /docs/decisions/ folder on GitHub repository.

mazenaly256 / Cinema-Seat-Reservation-System

Cloud-native microservices-based cinema seat reservation system developed. Designed for modularity and scalability.

Cinema Seat Reservation System

A cloud-native, microservices-based ecosystem developed in ASP.NET Core with REST APIs. This system features deliberate design decisions optimized for modularity, scalability and resilience — implementing proven distributed systems patterns such as Retries and Circuit Breakers and Request Timeouts alongside automated CI pipelines and manual deployment workflow.

---

Architectural Overview

Diagram

Services

Service	Responsibility
API Gateway	Centralized authentication, rate limiting, request timeouts, and routing to the main services
Identity Service	Centralized Identity Provider — user management and JWT issuance
Movie Service	Movies and showtimes management
Reservation Service	Seat layout, seat holds, and seat reservations

---

Getting Started

The system is live and ready to test via Postman Workspace

The workspace includes:

• Complete API documentation, with organized request collections and folders for each service
• All API endpoints with example requests and responses, featuring interactive testing
• Pre-configured variables containing hosting Azure VM IP with ports to access the services

How to…

View on GitHub

Problems

Latency Problem

One of the bottlenecks that appeared through load-testing, is that the latency of a specific endpoint (Get all showtimes with filters) takes the latency was around 850ms.

Availability Problem

Availability is simply the ability of the system to be alive and respond to requests, and do not completely die.

Another problem appeared when I tried to stress test the system, I simulated huge traffic on the system and it was across the critical path in the system (that is the longest trip for a request across the services).
Due to the large traffic the service took long time to respond, and also under stress testing the service crashed, due to exceeding the memory that limit that I determined for the container.

Furthermore, from a user experience perspective, a service that returns a quick, graceful error is far better than one that hangs indefinitely. Under stress testing, the P95 latency reached a painful 16 seconds, a frustrating experience where users are highly likely to lose patience and abandon their requests entirely.

Decisions To Solve The Problems

Solution of Latency Problem

I accepted the idea that the latency has to be slightly high, due to the network overhead that caused by having the database and service running on different servers in different countries so there is a long trip for data over the network between the service and database, but the 850ms latency is also very high even if we consider the network delay, there must be a problem with my system design.

At first glance, since the problem was related to database reading operations (data retrieval), then database indexing and caching are obvious answers to improve performance and reduce latency, but guess what? Neither of them was the solution in my case.

Why indexing did not reduce the latency:

Generally, indexing is a go-to solution in cases like this and I already created a database index on the table, but guess what? There was not any reduction in latency.
What I learned is that due to the high latency, and the relatively small number of rows (which is ~10,000 rows), the impact of indexing on the latency is invisible especially since the network delay itself was already massive.

Why caching is not a solution (common problems in both distributed and in memory):

Actually, I found caching would make things much worse, there are many cons for caching my situation:

• Range-based queries (from and to date filters) produce highly variable cache keys as if data of 2 days is requested then data of five days that span those two days is requested again, then actually we absolutely need to hit database again as we do not know if there are showtimes in those excess 3 days or not, cache usage rate would be low.

• Also full-list caching becomes impractical as the dataset grows to 100k+ rows.

• Cache invalidation on create/update/delete for the domain resources adds complexity that it actually does not pay off.

Those are problems in both in-memory and distributed caching, but even each one has specific cons.

the main problem with the in-memory caching is in a multi-containered deployment, each container instance maintains its own isolated cache, this produces inconsistent responses, as each container will have its own data separately — unacceptable for a booking system. Furthermore, syncing all those data will add significant complexity.

also in my case distributed caching (via Redis or any similar) will introduce its own network overhead to transfer data between the service and the Redis instance, that introduce more network delay that will increase latency.

The real solution, eliminating unbounded query results

It is pagination, instead of retrieving huge number of rows for a single request, return only a few number of rows like 10 or 20. Practically speaking, users never need more than them, they will see them and if they need more, they can load more and in this case they will accept waiting a few additional milliseconds to see new 10 or 20 showtimes, instead of waiting 850ms to retrieve 10,000 rows that user will never consider them all. This solution improved the performance reducing the P95 latency to ~515ms, and increased the throughput (the handled traffic) by ~25%.

Theoretically, there is an issue here that may require changing the pagination technique, the issue is that the database still reads every row it skips. Page 5 is fast. Page 5000 is slow. This becomes visible only when the number of becomes really huge.
However, practically speaking, this number of rows will not requested as it require very long date window that contains this enormous number of showtimes, that is not requested in the real life.

 

Solution of the availability problem

To solve this I applied rate limiting on the API gateway, this ensures that any traffic exceeding or limit fails at the edge, guaranteeing that only a manageable number of requests are processed within a given time window.

Additionally, applying request timeouts helped as it prevented misbehaving requests that are unusually slow for some specific reason from degrading all the system as they take long time to be processed and take some resources. Now the system automatically cancels any request after 8 seconds and disconnect all the related connections and frees up all its allocated resources to be available to another incoming request to consume.

And prevented the latency from exceeding 8 seconds.

---

Possible Areas of Improvement

In this section I would like to say that honestly, I do not know if these features will actually improve the system, or their effect will be minimal. For sure they will come with their overhead and trade-offs and will require a decision to determine whether they will introduce an actual improvement.

1. Caching

I discussed the caching when addressing the latency problem and I mentioned why it did not work, but caching may be useful in another part in the system specially when service is read heavy, and also with data that do not change frequently or the data that the way of querying them is consistent (that was not the case in my problem above).

2. Event-Driven Architecture and Message Brokers like Kafka and RabbitMQ

At first glance, this solution clearly introduces a major architectural trade-off: Database Replication. On the other hand, it has benefits including that no inter-service network calls are required so the system will be faster, and also no cascading failures, suppose that movie service crashed, the reservation service has all data on its database replica and can actually use and do its work (reserving seats in showtimes) without even knowing that movie service is down.
This will reduce the synchronous HTTP calls that happen frequently between the services in cases like checking movie when adding a new showtime, or checking showtime when trying to reserve a seat. All those are synchronous calls that enforce the upstream service (service that makes a call) to wait for the downstream service (the called service) to respond. Also upstream service fails to do its job if the downstream service goes down.

3. Change the database location

There are two options to achieve this.
Option A: is to just get a nearer the server for our database. There is no architectural side effects and never presents more challenges, just host the DB on a nearer server to the deployment server of the services, this reduces the travelling distance for data thereby reduce network overhead.

Option B: is hosting the database on the same machine as the services, this will be actually better in terms of performance due to eliminating the network overhead. But it will introduce its own overhead like the configuration headache and managing the DBMS on the server.
Additionally, it comes with a trade-off, which is coupling the downtime of DB with the downtime of the system. When the services and the database are hosted on different servers, the services can still be available and return friendly errors when the DB server can not be reached, instead of being fully down if they all are on one machine. This is a violation for the principle of single point of failure, that is "eliminating cascading failures by separating concerns across different machines", but as I said above ad want you to remember: System design is all about Trade-offs.