DEV Community

Akif Keklik
Akif Keklik

Posted on

The Code Didn't Change. So Why Did the Application Go Down?

What a hospital appointment system taught me about production failures, database availability, security, and testing.

A few months ago, I deployed a hospital appointment system as part of my portfolio.


Everything seemed fine.

Patients could book appointments, doctors could manage their schedules, and administrators could manage the system.

Then, one day, I noticed something unexpected.

The application had been down for around ten days.

I hadn't made any changes that should have broken it. So what happened?

That question led me to investigate something every software developer should understand:

A production system is more than its source code.

1. First, What Did I Build?

The project was a hospital appointment management system with three main roles:

  • Patients could book appointments.
  • Doctors could manage appointments and medical records.
  • Administrators could manage doctors, departments, and applications.

The backend was built as a modular monolith.

In simple terms, it was a single deployable application divided into separate modules based on business responsibilities.

For example, appointment, patient, and doctor operations had their own responsibilities.

I chose this approach instead of introducing microservices because the project did not require the additional operational complexity.

2. Security Was Not Just About Login

A hospital system handles sensitive personal information.

That makes security an important part of its design.

Here are some of the decisions I made.

Password Storage

Passwords were not stored as plain text.

I used BCrypt hashing to make password verification more resistant to brute-force attacks.

I also made password reset responses consistent whether an account existed or not. This helps prevent account enumeration.

Protecting the Login Endpoint

Login endpoints can be targeted by automated password-guessing attacks.

I implemented rate limiting to restrict repeated requests.

When the limit is exceeded, the application returns:

429 Too Many Requests

JWT Storage

A common approach is to store JWTs in localStorage.

However, JavaScript running on the page can access this storage. If an XSS vulnerability exists, an attacker may be able to steal the token.

For this project, I used an HttpOnly cookie so that client-side JavaScript could not directly read the token.

Preventing Duplicate Appointments

Imagine two patients trying to book the same doctor's appointment slot at exactly the same time.

A simple availability check is not enough.

Both requests might check the database before either one saves the appointment. Both could see the slot as available.

This is a race condition.

I used a database-level UNIQUE constraint to enforce the appointment slot rule.

@Column(name = "active_slot_id", unique = true)
private String activeSlotId;
Enter fullscreen mode Exit fullscreen mode

The database becomes the final authority: only one request can successfully claim the same slot.

Preventing Unauthorized Data Access

A patient should never be able to access another patient's information simply by changing an ID in the URL.

For example:

GET /api/patients/5

What happens if the patient changes 5 to 6?

This is a common form of IDOR (Insecure Direct Object Reference).

Role-based authorization alone is not enough. We also need ownership checks.

I used authorization logic similar to:

@PreAuthorize(
    "hasRole('ADMIN') or @securityService.isPatientOwner(#id)"
)
Enter fullscreen mode Exit fullscreen mode

The application checks whether the authenticated user is actually allowed to access the requested record.

3. The Application Went Down

Now, back to the incident.

I opened the deployment dashboard and noticed that the recent deployments were failing.

My first assumption was that one of my code changes had caused the problem.

But instead of immediately changing code, I asked a different question:

When did the failure actually begin?

I checked the first failed deployment.

The commit only contained a small change in the appointment service. Nothing there explained why the application could not start.

The tests were also passing.

So I started looking outside the application code.

The backend had one important external dependency: its database.

I opened the Aiven dashboard.

The database was powered off.

It turned out that the free plan automatically stopped services after a period of inactivity.

The backend could not establish a database connection during startup, so every deployment failed.

The fix was simple:

  1. Start the database.
  2. Deploy the application again.
  3. Verify that the application could connect successfully.

The application was working again.

But the incident revealed a much bigger problem.

4. What Else Had I Missed?

The application was back online, but I wanted to understand whether there were other issues hidden in the project.

So I reviewed it again.

I found two actual business logic problems.

Problem 1: Patients Could Change Appointment Status Incorrectly

Patients could update the status of their own appointments.

That included marking an appointment as completed.

The ownership check was working, but ownership alone did not define which operations a user was allowed to perform.

I changed the rules so patients could only cancel their appointments.

Problem 2: Doctors Were Never Receiving Their Temporary Passwords

This was more serious.

When an administrator approved a doctor application, the system generated a temporary password and returned a message saying that the password had been sent.

But no email was actually being sent.

The doctor was approved but could not access the system.

The problem was not immediately obvious because the existing tests only checked that the password was not exposed in the API response.

They did not verify that the doctor actually received it.

I fixed the workflow so the temporary password was sent to the doctor's email address.

This taught me something important:

A test can pass while the actual business process is still broken.

5. Another Lesson: Database Schema Validation

I also wanted Hibernate to validate the production database schema instead of automatically modifying it.

However, enabling ddl-auto=validate caused the application to fail during startup.

Why?

The production database had changed over time and no longer matched the schema represented by the migration files.

I temporarily reverted the change.

The correct next step was to reconcile the actual production schema with the migration history before enabling validation.

This was another reminder that database schema management requires careful attention.

6. What Did I Learn?

This incident changed how I think about production systems.

Here are three lessons I took away:

1. Investigate the first failure, not just the latest change.

The most recent commit is not always the reason an application goes down.

2. Your application is more than your code.

Databases, hosting providers, deployment environments, and external services are all part of the system.

3. Passing tests does not mean everything works.

A test only proves the behavior it actually checks. If an important business scenario is missing, the test suite cannot protect you from it.

My next step is to add regular health monitoring and alerts so that I know when the application becomes unavailable.

Because a production system that nobody monitors can fail without anyone noticing.

Have you ever encountered a production issue that had nothing to do with your latest code change?

I'd be interested to hear what happened and how you diagnosed it.


Project source code: Hospital Management System

Top comments (0)