Concurrency and API Protection - Part 3

#database #restapi #apigateway #concurrency

This article is part of a series that explores practical patterns for protecting your systems.

Concurrency and Row Versioning - Part 1
Concurrency and Transactions - Part 2
Concurrency and API Protection - Part 3 ⭐

❓ Concurrent requests and external services

Occasionally, applications encounter broken state errors that are difficult to debug, especially when the issue affects only a small subset of resources. Investigation often reveals that the root cause is a UI that fails to disable the submit button, allowing users to double-click and send multiple requests to the same REST API.

In the previous series, we explored how to protect against such cases using our internal services. Today, modern applications almost always rely on third-party services, with in-house teams focusing primarily on core functionality. That approach enhances both security and development efficiency, but it also introduces a new challenge.

For example, Perci Health uses a famous service for Fast Healthcare Interoperability Resources (FHIR). Transactions are very limited, and conditional row versions are not supported. During testing, we found that a nurse can double-click on a submit button to update an appointment. This could lead to a broken state: while only one appointment encounter should exist, two or three resources appear with incorrect statuses.

A similar case was found during the contract work at Checkatrade. Multi-clicking on the Salesforce button led to two or three modification requests with a slight sub-second difference.

To eliminate or at least reduce such issues, we can move resource protection from the DB to the upper level—the REST API. This idea was initially implemented at Checkatrade with database transactions. This article shows the improved transactionless version.

🚫 No out-of-the-box solutions

At the time of writing, none of the major cloud providers offer a built-in solution to protect REST APIs from concurrent modification requests. Services like Amazon API Gateway, Google Cloud API Gateway, and Azure API Management Gateway provide only basic rate-limiting capabilities.

💡 REST API protection idea

Here is the idea of API protection against concurrent requests.

Consider only modification methods: POST, PUT, PATCH, and DELETE. Do not protect GET requests.
Protect APIs requiring authentication, so we do not touch /auth/sign-in and other public methods.
If the request path has a parameter, then it is the exact resource modification, and we must protect it, for example, DELETE "/appointments/:appointmentId". Then, if there is no parameter and a user is authenticated, we also need to protect the API, and we do it by adding the user ID prefix, so PUT /me becomes PUT /:userId/me, POST /appointments becomes POST /:userId/appointmetns. Otherwise, it is a public API without a parameter, such as POST /auth/sign-in, and there is no need for protection.
Group API parametrized paths per single parent resource, for example, all the following modification requests
- PUT /appointments/100
- POST /appointments/100/end-call
- DELETE /appointments/100
have one parent resource: "/appointments/100".
Mark in a database that the request for a special resource is in process.
If another resource tries to own the modification, return the HTTP 409 Conflict status code (or HTTP 423 Locked). A UI client can fetch fresh data and notify a user what to do with a conflict.
The owner completes the request and unlocks the modification resource.

🎯 REST API protection with Firestore DB

Here, we consider a NodeJS ExpressJS example with Firestore DB. A similar one can be built with almost any REST API framework, like NodeJS Fastify, ASP.NET, Python Jango/FastAPI, Java Vert.x/Spring Boot, Go Gin/Fiber, etc. Any database that supports conditional updates with row versioning, transactions, or a lock mechanism can be used: key-value RedisDB or KeyDB, document-oriented Fistore, Supabase, or MongoDB, a modern SQL database, such as Postgres, MS SQL, Oracle, MySQL, etc.

Now, let us write our Express and Firestore implementation step by step in TypeScript.

Introduce the acquire lock function returning a type result and an optional release lock function. The type result can be skipped - lock is not applicable, locked - lock was successfully applied, conflict - not used due to the resource conflict.
```
  const acquireLock = async (req: Request, res: Response): 
    Promise<{ type: 'skipped' } | 
            { type: 'locked'; release: () => Promise<void> } |
            { type: 'conflict' } > => {

    // implementation
  };
```

Create an ExpressJS middleware that can be reused with multiple APIs

import { Request, Response, NextFunction } from 'express';

export const lockMiddleware = async (
    req: Request,
    res: Response,
    next: NextFunction,
  ) => {        
  const lockResult = await acquireLock(req, res);

  if (lockResult.type === 'conflict') {
      // send HTTP 409 Conflict or 423 Locked code
      res.sendStatus(409); 
      return;
  }

  try {
    next(); // API request handler        
  } finally {
    if (lockResult.type === 'locked') {
      await lockResult.release();
    }
  }
};

Protect only for modification methods.

const PROTECTED_METHODS = new Set(['POST', 'PUT', 'PATCH', 'DELETE']);

if (!PROTECTED_METHODS.has(req.method)) {
  // Not a modification, no need to lock
  return { type: 'skipped' };
}

Assume there is an auth middleware. It is out of the scope of this article to fully implement it, for example, by inspecting the Authorization: Bearer <token> header. One practice is to store the authentication results in the res.locals object. We assume that if res.locals.userId is not empty, the user is authenticated.

Ignore public endpoints, such as /auth/sign-in.

if (!res.locals.userId) {
  // Public endpoint, no need to lock
  return { type: "skipped" };
}

Create two helper functions. First, canonicalize the path so that the action endpoints /resource/:id/action and lock on /resource/:id and /me on /:userId/me.

const canonicalizePath = (
  path: string,
  params: Record<string, string>,
  userId: string,
): string => {
  // clean up the request path from query parameters,
  // so /appointments/1/cancel?abc=1&def=some_value becomes /appointments/1/cancel
  path = path.split('?')[0];
  path = path.length === 0 ? '/' : path;

  // remove double slashes and trim trailing slash (except root)
  path = path.replace(/\/+/g, '/');
  if (path.length > 1 && path.endsWith('/')) {
    path = path.slice(0, -1);
  }

  // If any segment matches a param, keep only the collection path + id
  // e.g. /resource/:id/subresource/:subId/action becomes /resource/:id
  // e.g. /prefix/resource/:id/action becomes /prefix/resource/:id
  const segments = path.split('/').filter(Boolean);
  if (segments.length >= 2) {
    // Iterate through segments; stop when a param value appears
    for (let i = 0; i < segments.length; i++) {
      const segment = segments[i];
      const isParamSegment = Object.values(params).includes(segment);

      if (isParamSegment && i > 0) {
        // Keep prefix up to this param
        const normalized = `/${segments.slice(0, i + 1).join('/')}`;
        return normalized;
      }
    }
  }

  // No params matched in path.
  // Prefix with userId to avoid clashes between different users
  // e.g. /learn/articles/saved becomes /:userId/learn/articles/saved
  // e.g. POST /appointments becomes /:userId/appointments
  return `/${userId}${path}`;
};

The path can be too long for a Firebase collection or have unsupported symbols, so we can hash it.

import crypto from 'crypto';

const buildLockId = (canonicalPath: string) => {
  // path may contain special symbols or be too long, hash it
  return crypto.createHash('sha256').update(canonicalPath).digest('hex');
};

Our acquireLock function becomes

const acquireLock = async (
  req: Request,
  res: Response
): Promise<
  | { type: "skipped" }
  | { type: "locked"; release: () => Promise<void> }
  | { type: "conflict" }
> => {
  if (!PROTECTED_METHODS.has(req.method)) {
    // Not a modification, no need to lock
    return { type: "skipped" };
  }

  if (!res.locals.userId) {
    // Public endpoint, no need to lock
    return { type: "skipped" };
  }

  const canonicalPath = canonicalizePath(
    req.path,
    res.params,
    res.locals.userId
  );
  const lockId = buildLockId(canonicalPath);

  // FOLLOWING IMPLEMENTATION
}

For brevity, the following code snippets will be part of the // FOLLOWING IMPLEMENTATION body section.

A cloud provider can unexpectedly destroy a running application instance for many reasons. Our lock cannot be released and gets stuck forever. The first thing to do is check whether the request is stale and, if so, delete the stuck lock.

// How long a single request may hold the lock before it’s considered stale
const LOCK_TTL_SECONDS = 5;

const db = getFirestore();
const lockRef = db.collection('request-locks').doc(lockId);

const staleLockDoc = await lockRef.get();
if (
    staleLockDoc.updateTime &&
    staleLockDoc.updateTime.toMillis() + LOCK_TTL_SECONDS * 1000 <
      Timestamp.now().toMillis()
) {
  try {
    await lockRef.delete({
      lastUpdateTime: staleLockDoc.updateTime,
    });
  } catch {
    // ignore delete error
  }
}

The lastUpdateTime precondition prevents the accidental removal of a concurrently updated version, which is covered in Concurrency and Row Versioning—Part 1.

The next step is to check for ongoing processing and mark the conflict.

const existingLockDoc = await lockRef.get();
if (existingLockDoc.exists) {
  // Someone else holds a fresh lock
  return { type: 'conflict' };
}

Now it's time to insert a row lock indicator. If another processor did it before us, the row insert will fail, indicating the conflict.

let createdLockDoc: FirebaseFirestore.DocumentSnapshot<
  FirebaseFirestore.DocumentData,
  FirebaseFirestore.DocumentData
>;
try {
  await lockRef.create({
    method: req.method,
    path: req.path,
    canonicalPath,
  });

  createdLockDoc = await lockRef.get();
} catch (error) {
  const firebaseError = error as { code?: string | number; message?: string };
  // error code `6` - the document already exists
  if (firebaseError.code === 6) {
    // Someone else created a lock
    return { type: 'conflict' };
  } else {
    throw error; // rethrow unexpected error
  }
}

The last function part is to return the release lock function.

const release = async () => {
  const logger = new Logger(`${appName}:lockMiddleware`);
  try {
    // Only delete if we still own it (avoid nuking a refreshed lock)
    await createdLockDoc.ref.delete({
      lastUpdateTime: createdLockDoc.updateTime,
    });
  } catch {
    // skip release error
  }
};

return { type: 'locked', release };

To use the lock mechanism in multiple requests, apply the lockMiddleware.

  import { default as express } from 'express';
  import { lockMiddleware } from '../path-to/lockMiddleware';
  import { yourRouterOne } from '../path-to/yourRouterOne';

  const api = express();
  api.use(lockMiddleware); // applied to all routers below
  api.use(yourRouterOne);
  // rest of the routers

Full middleware version

The complete middleware example can be found at lockMiddleware.ts.

Testing

To test concurrent requests, run curl commands almost simultaneously, for example, a POST request.

seq 20 | xargs -n1 -P5 curl -X POST -s -o /dev/null -w "%{http_code} %{errormsg}\n" \
--data-raw '{ "id": "abcde" }' \
-H 'Content-Type: application/json' \
-H 'Authorization: Bearer <TOKEN>' \
https://your-api-path

🚀 Result

With this protection in place, concurrent API requests no longer cause invalid resource states. The best part is that it achieves this without Firestore transactions, keeping it "blazingly" fast.