Building APIs: Backend API Concepts And When To Use Them

Every few weeks someone posts the same checklist: as a backend engineer, learn System Design, APIs, Databases, Distributed Systems, Caching, Security, DevOps, Performance, Cloud, Monitoring… and the list is correct but can be confusing without specifying when any of it matters.

Lets take one endpoint, POST /transfers, which moves money from one account to another. I picked the scariest endpoint on purpose: when an order endpoint has a bug you ship a duplicate t-shirt; when a transfer endpoint has a bug you move someone's rent twice. Each thing that breaks teaches one concept, in the order you'd actually hit it.

Code is Spring Boot. Lessons aren't (These are applicable in any backend framework of your choice)

Stage 0: It works on my machine

Here's version one. A controller, a repository, done.

@PostMapping("/transfers")
public Transfer create(@RequestBody Transfer transfer) {
    return transferRepository.save(transfer);
}

It compiles, it demos, you feel powerful. It also contains the first mistake almost everyone ships: it accepts and returns the JPA entity directly.

That's a problem on both ends. On input, the client can set fields you never meant to expose (status, id, settledAt). On output, you're serializing a Hibernate-managed object, which lazily loads associations during JSON serialization, giving you surprise queries or a LazyInitializationException outside the transaction.

Fix it before anything else: the wire format is not your database. Use DTOs (records are perfect).

public record CreateTransferRequest(String fromAccount, String toAccount, BigDecimal amount) {}
public record TransferResponse(Long id, String status, BigDecimal amount) {}

@PostMapping("/transfers")
public ResponseEntity<TransferResponse> create(@RequestBody CreateTransferRequest req) {
    Transfer saved = transferService.create(req);
    return ResponseEntity.status(HttpStatus.CREATED).body(toResponse(saved));
}

Now the API has a contract that's independent of your schema. Everything below builds on that.

Stage 1: The user double-clicks (idempotency)

First real customer hits Send, the network stalls, they tap again. You just moved the money twice. This is the nightmare scenario for a payments API, so it's the first thing you harden.

POST is not idempotent, two identical requests legitimately mean two transfers. The standard fix is an idempotency key: the client sends a unique key, and a retry with the same key returns the original result instead of moving money again.

The naive version is an in-memory Map, and it's wrong, because two concurrent requests with the same key both check the map before either writes, a classic time-of-check/time-of-use race. The real guard is the database (a unique constraint) or Redis SETNX:

@PostMapping("/transfers")
public ResponseEntity<TransferResponse> create(
        @RequestHeader("Idempotency-Key") UUID key,
        @RequestBody CreateTransferRequest req) {
    try {
        Transfer saved = transferService.create(key, req);    // INSERT, key column UNIQUE
        return ResponseEntity.status(HttpStatus.CREATED).body(toResponse(saved));
    } catch (DataIntegrityViolationException dup) {           // someone got there first
        Transfer existing = transferService.findByKey(key);
        return ResponseEntity.ok(toResponse(existing));
    }
}

Let the database be the referee. Concurrency is exactly where hand-rolled idempotency falls apart, and with money, "falls apart" means a customer's account is wrong.

Stage 2: The garbage payload (validation + honest errors)

Now requests arrive with amount: -500 (a transfer that pulls money the wrong way) and a missing toAccount. If you don't stop them, that nonsense flows straight into your ledger.

Validate at the edge with Bean Validation, and, this is the part people skip, say what went wrong with the right status code.

public record CreateTransferRequest(
    @NotBlank String fromAccount,
    @NotBlank String toAccount,
    @Positive BigDecimal amount) {}

@PostMapping("/transfers")
public ResponseEntity<TransferResponse> create(@RequestBody @Valid CreateTransferRequest req) { ... }

The status code is the error contract, it tells the client what to do next, before they read a single word of your message:

• 400 — malformed/missing field. The request is broken.
• 422 — well-formed but semantically invalid (amount: -500).
• 409 — conflict (insufficient funds, stale balance, next stage).
• 401 vs 403 — not authenticated vs authenticated-but-not-allowed. Different problems, different fixes.

Spring 6 gives you RFC 9457 ProblemDetail for free, so every error has the same machine-readable shape:

@RestControllerAdvice
class ApiErrors {
    @ExceptionHandler(MethodArgumentNotValidException.class)
    ProblemDetail onValidation(MethodArgumentNotValidException ex) {
        ProblemDetail pd = ProblemDetail.forStatusAndDetail(
            HttpStatus.UNPROCESSABLE_ENTITY, "Validation failed");
        pd.setProperty("errors", ex.getFieldErrors().stream()
            .collect(toMap(FieldError::getField, FieldError::getDefaultMessage)));
        return pd;
    }
}

A 500 for a bad input is a lie. Don't lie to your clients.

Stage 3: Two debits race on the same account (optimistic locking)

A customer fires two transfers from the same account at the same instant. Both read balance = 100, both subtract 80, both save, last write wins, and now 160 has left an account that only had 100. That's a lost update, and in a payments system it's an overdraft you created yourself.

You don't need to lock rows. Add a version column and let Hibernate detect the stale write:

@Entity
class Account {
    @Id String id;
    BigDecimal balance;
    @Version Long version;     // Hibernate checks this on UPDATE
}

If a transaction tries to update a row whose version has already moved, Hibernate throws OptimisticLockException, which you map to 409 Conflict, telling the client "your view of the balance was stale, refetch and retry." This pairs with knowing your transaction boundaries: keep @Transactional tight, mark pure reads readOnly = true, and never wrap a slow external bank call inside a DB transaction, you'll hold a connection (and a row lock) hostage while the network dawdles.

Stage 4: "Give me all the transfers" (pagination + N+1 + caching)

GET /accounts/{id}/transfers works great with 50 rows and falls over at 5 million. Three separate things bite here, usually at once.

Pagination — never return an unbounded list. Offset paging (Pageable) is built in and fine for admin screens, but the DB still scans and discards every skipped row, so deep pages crawl and new transactions arriving mid-scroll cause dupes. For a transaction history, append-heavy and user-facing — use keyset/cursor paging on an indexed column:

// SELECT * FROM transfers WHERE account_id = :acct AND id < :after ORDER BY id DESC LIMIT :size
@Query("""
    SELECT t FROM Transfer t
    WHERE t.account.id = :acct AND (:after IS NULL OR t.id < :after)
    ORDER BY t.id DESC""")
List<Transfer> page(@Param("acct") String acct, @Param("after") Long after, Pageable pageable);

N+1 — listing transfers and touching transfer.getCounterparty() in a loop fires one query per transfer. The fix is a fetch join or entity graph:

@EntityGraph(attributePaths = "counterparty")
List<Transfer> findByAccountId(String acct);     // one query, not 1 + N

For deeper graphs set hibernate.default_batch_fetch_size so lazy loads batch into a few IN (...) queries instead of hundreds. Watch your SQL logs, N+1 is invisible until the data grows.

Caching — for hot, rarely-changing reads (an already-settled transfer never changes), add HTTP ETag + Cache-Control so unchanged data returns a bodiless 304, and @Cacheable (Redis/Caffeine) so the read never touches the DB:

return ResponseEntity.ok()
    .eTag(Long.toString(transfer.getVersion()))
    .cacheControl(CacheControl.maxAge(60, SECONDS))
    .body(toResponse(transfer));

Stage 5: A stranger finds the endpoint (auth + rate limiting)

You're public now. Two new questions on every request: who are you, and how often are you allowed to ask.

Authentication vs authorization are different jobs. A JWT answers the first, but a JWT is signed, not encrypted, so it proves the token wasn't tampered with while the payload stays readable. Never put secrets in it, always set a short expiry, and authorize separately with scopes/roles. Spring Security as a resource server makes validation declarative:

http.authorizeHttpRequests(a -> a
        .requestMatchers(POST, "/transfers").hasAuthority("SCOPE_transfers:write")
        .anyRequest().authenticated())
    .oauth2ResourceServer(o -> o.jwt(withDefaults()));

A valid token is not a free pass. The single most important check in this whole API: the fromAccount actually belongs to the authenticated user. Skip that ownership check and anyone with a token can drain anyone else's account, this is the #1 class of real-world API breach (broken object-level authorization).

Rate limiting stops one client, malicious or just stuck in a retry loop, from drowning everyone else. Token bucket via Bucket4j, one bucket per API key, returning 429 with a Retry-After so well-behaved clients back off.

Stage 6: The bank rail stops answering (timeouts + circuit breakers)

Your endpoint calls a downstream payment rail to actually settle the money. One day it doesn't respond. With no timeout, your threads pile up waiting, and your healthy service goes down because of someone else's outage. This is the failure the listicles never mention and the one that actually pages you.

Two non-negotiables for every outbound call: a timeout, and a circuit breaker that stops hammering a service that's clearly down (Resilience4j):

@CircuitBreaker(name = "bankRail", fallbackMethod = "queueForLater")
@TimeLimiter(name = "bankRail")
public CompletableFuture<Receipt> settle(Transfer t) { ... }

private CompletableFuture<Receipt> queueForLater(Transfer t, Throwable ex) {
    // degrade gracefully: park it as PENDING and retry, don't take the whole API down
}

Decide now what happens when a dependency dies, not during the incident.

Stage 7: The work is too slow to do inline (queues + async)

Settling a transfer now means: call the bank rail, run a fraud check, write the ledger entry, notify both the sender and the recipient, update analytics. Do all of that inside the request and your endpoint is only as fast as its slowest dependency, and it fails entirely if any one of them hiccups.

The fix is to do the essential part synchronously (record the transfer as PENDING) and offload the rest to a message queue — RabbitMQ, Kafka, SQS, so the API returns immediately and workers handle the slow work on their own time.

@PostMapping("/transfers")
public ResponseEntity<TransferResponse> create(@RequestBody @Valid CreateTransferRequest req) {
    Transfer saved = transferService.create(req);          // record intent, fast
    queue.publish(new TransferInitiated(saved.getId()));   // settle + notify async
    return ResponseEntity.accepted().body(toResponse(saved));   // 202, not 201
}

Three things you must get right the moment a broker enters the picture:

• Idempotent consumers. Queues deliver at least once, so the same message will eventually arrive twice. The worker that calls the bank rail has to dedupe (Stage 1's logic, now on the consumer side) — otherwise you settle the same transfer twice and move the money twice, asynchronously, where it's far harder to notice. Dead-letter queues + retries. A message that keeps failing must not loop forever or silently vanish, that's a customer's money stuck in limbo. Retry with backoff, then park it in a DLQ for a human. Spring's @RabbitListener / @KafkaListener have retry and DLQ wiring built in. Backpressure. A queue is a shock absorber: the API keeps accepting transfers at 202 speed during a spike while workers drain the backlog at a sustainable rate. Decoupling accepting work from doing work is the entire point.

This also reshapes the contract. Settlement shouldn't hold a request open, so the pattern becomes 202 Accepted + a status endpoint: return an id now with status PENDING, let the client poll GET /transfers/{id}, or fire a webhook when it settles. If you send webhooks, sign them (HMAC) so the receiver can trust the payload, and retry them, because the receiver's endpoint is exactly as flaky as the bank rail was in Stage 6.

Stage 8: You need to change the response (versioning + consistency)

The transfer shape has to change, and there are clients in the wild you can't redeploy. The rule that saves you: additive changes are free; breaking changes get a new version (/v1/transfers → /v2/transfers, or a media-type header). Adding a field is fine. Renaming or removing one is a v2.

By now you've split work across services and async listeners, which means not every read is instantly up to date, and that's where you have to think hard, because this is money. The discipline is to be deliberate about what gets which guarantee:

• Strong consistency for the money itself. The debit, the balance check, the "do we have the funds" decision — these are read-your-writes, inside a transaction, no exceptions. You cannot let a balance be eventually consistent in a way that permits an overdraft.
• Eventual consistency for everything derived from the money: the notification, the analytics rollup, the search index, the "transactions this month" widget. These can lag a few seconds and nobody gets hurt.

@TransactionalEventListener     // runs only after the transfer tx commits
public void onSettled(TransferSettled e) { ledger.projectAsync(e); }

Just make those consumers idempotent (Stage 7 again) — at-least-once delivery means the same event will eventually arrive twice.

Stage 9: You can't see what it's doing (observability + docs)

The endpoint is live, fast, and secure, and one morning it's misbehaving and you have no idea why, because you can't see it. An API that moves money and that you can't observe is not an API, it's a liability.

The minimum kit:

• Correlation IDs in the logs (a filter that puts a request ID into the MDC) so you can trace one transfer across the API, the queue, and the bank rail instead of grepping blind.
• Metrics via Micrometer + Actuator, scraped by Prometheus, drawn in Grafana, latency percentiles (p99, not averages, averages hide the pain), error rates, throughput, and business signals like settlement failure rate.
• Health checks — liveness/readiness probes via Actuator (/actuator/health), so your orchestrator knows when to route traffic to a fresh instance and when to restart a sick one.
• Docs via springdoc-openapi, so the contract is discoverable and your future self isn't reverse-engineering past you:

@Operation(summary = "Initiate a transfer")
@ApiResponse(responseCode = "202", description = "Transfer accepted and pending settlement")
@PostMapping("/transfers")
public ResponseEntity<TransferResponse> create(@RequestBody @Valid CreateTransferRequest req) { ... }

The endpoint, grown up

Same POST /transfers we started with. Now it returns a DTO, dedupes retries through the database, validates input and answers with honest status codes, refuses to overdraft under concurrent debits, pages and caches its reads, checks not just who is calling but whose account they're touching, degrades gracefully when the bank rail dies, offloads settlement to a queue and answers 202, versions its contract, keeps the money strongly consistent while letting the dashboards lag, and tells you exactly what it's doing.

None of that showed up in the demo. All of it shows up in production, and with money, all of it shows up as a customer complaint if you skip it.

You don't need to learn the whole backend-checklist wall of nouns in one sitting. You need to ship one honest endpoint, then keep asking the next question: what breaks when a thousand strangers hit this at once? The concepts arrive in order, every time. Meet them on a quiet afternoon, not at 2am.

Comments

[Mustafa ERBAY]

I like the way this grows one endpoint through actual failure modes instead of introducing backend concepts as a checklist.

One failure I’d add around Stage 7 is the gap between transferService.create() and queue.publish(). The database commit can succeed and the broker publish can still fail, leaving a PENDING transfer with no event to process it.

That’s usually the point where I’d introduce a transactional outbox rather than trying to make the database and broker behave like one transaction. It fits the progression here nicely because it’s exactly the kind of problem that doesn’t exist in the demo and suddenly matters a lot in production.