DEV Community

Muneeb Ahmad
Muneeb Ahmad

Posted on

Building a Job Portal Backend: From Database Design to Deployment

#backenddevelopment #node #systemdesign #postgres

Summary

This project is a multi-tenant job portal built with Node.js, TypeScript, PostgreSQL, Redis, BullMQ, and Docker. It supports three roles: admin, recruiter, and applicans, with strict role-based access control and complete data isolation between companies.

The system covers the full lifecycle: company onboarding, job posting, applications with profile snapshots, interview scheduling and feedback, admin moderation, async email and resume processing, rate limiting, structured logging, and graceful shutdown. The deployment is containerized with multi-stage Docker builds, Nginx reverse proxy, and a one-off migration service.

This blog walks through every major decision, like why relational over document, why raw SQL over ORM, why cursor pagination over offset, why files don't belong in the database, how multi-tenant isolation works, and why two tokens are better than one.

Table of Contents

Why a relational database for a job portal

I had two options for databases, one a relational database like postgresql and second a non-relational/document based database like mongodb.

For the specific case relational db was the obvious choice,

Firstly because the data is going to be spread across different entities like user, application, jobs etc, and using a document based approach would require me to strictly hold all the schemas/data values at the application level.

Secondly, operations like job applications specifically need complete information and for all the intermediate steps to be completed. As the operations following submission require them, operations like profile snapshot, application review etc.

The document based database does not support important checks or invariants at the database level.

So choosing postgresql provides me with ACID transactions and ability to enforce relations between data.

The Question then is: Why not just implement these at application level?

Answer:

  • More code.
  • Could be bypassed raising security concerns
  • Database should be the main defence wall as this is where we store and retrieve data from.

Choosing your stack & tools

Typescript vs javascript

I chose typescript over javascripts as typescript provides type safety during the compile time.

SQL vs Query Builder vs ORM

I used raw SQL to get under the hood and not compromise on speed, get complete control of the queries and avoid the n+1 problem.

Zod

The project uses Zod to catch the validation errors at runtime using the already set typescript configurations. No need for separate manual configurations that might introduce inconsistencies as well.

Redis

The project uses redis to cache specific pages and to act as a record for the locally scheduled jobs.

Bull MQ

Bull MQ is used for a worker program that runs in the background for tasks like sending emails and processing resumes etc.

S3 storage

The project uses S3-compatible storage to store applicant resume pdfs.

Pino

Pino is used in the project for clear, structured logging for easier debugging.

Docker Local env

I used docker for the local environment setup, for PostgreSQL and for Redis. That was to keep the versions similar across different devices for consistent development and to avoid it works on my device issue in development. I used docker compose to orchestrate these services. Besides these I used minio as an s3-storage and mailpit to track the emails.

Configs & Secrets

I used a config file that acts as a centralised place to access all the environment variables related to the backend. No program file accesses the variables directly from the .env files. The config file reads the variables, validates the schema, applies default values and then passes them on.

Another reason to use it to fail fast right at startup if config is invalid.

Migrations

I used migrations to set up the database in order without facing the table dependency problems we might face setting the db up manually. For example, in case of a manual setup, we are trying to create an application table but it references a job table or an applicant table which we have not created yet. These sorts of problems are what we avoid with migrations.

Foreign Keys and Constraints

I specifically used RESTRICT constraint on Primary Keys to not hard delete all the dependent Foreign Keys. As in most cases we want to keep the underlying records like if we are to delete a job we do not want to delete the underlying applicant/application records. I did use CASCADE on the user -> applicant dependencies as when I delete a user I want its record in its specific user type table to be deleted as well

JSONB attributes

Attributes like different perks a company provides, and questions they ask are arbitrary. So we use JSONB to store their values. We might need to search jobs based on these but due to their arbitrary nature I went with JSONB. And JSONB still provides search capabilities though not as fast as the indexed sql columns.

Request Validation Layer

I used a request validation layer that validates the request schema when it hits the server to catch it immediately without rolling it forward. On a match we let it pass and on a violation we throw an error. That prevents any inconveniences we might face in future by preventing the malformed data from reaching the database. The schema is validated using Zod.

Consistent Errors

I have set a centralised error file that defines the errors with default messages and error code. Different files can import the defined types from here and raise errors with their specific messages. With this we have standard error types and standard mechanisms to throw errors. Since I am using express 4 I use try/catch blocks to catch the errors and pass them as next(err) for it to reach the handler. (Express 4 does not catch async errors automatically.)

Hashing Passwords

We hash the passwords using bcrypt so that brute-forcing is expensive. Bcrypt is intentionally slow to increase the brute-force time and includes a salt per password which makes the precomputed hash attacks less useful. No two identical passwords have a similar hash. And attackers cannot use precomputed tables either.

JWT Access Tokens and Refresh Tokens

I used a pair of tokens, access token and refresh token for the access flow.

The refresh token is stored in the database, encrypted and has time to live (TTL) of 7 days. While the access token is stored as a signed jwt cookie, with a TTL of 15 mins.

Now When the access token is expired, the frontend makes a call to the backend to refresh it based on the refresh token hash. It matches the token hash with the hash in the database and on a match returns a new access token.

Now if it does not match or if it has expired we need to login again and are redirected to the login page.

Now why use these two the way I am using them?

Why sign the access token?

Answer: to prevent session undetected modification of its signed claims. It is signed using HMAC (hash based message authentication code)

Why hash the refresh tokens?

To not allow a potential attacker to gain long term login access in case the refresh tokens are compromised. It is a 32-byte random string hashed using SHA-256 which is stored in the database. This prevents an attacker who compromises the database from directly using the stored values as refresh tokens.

What happens in case of an access token being hacked?

The hacker retains the control for a maximum of 15 minutes then he needs to have a refresh token to generate a new access token. In this time he can use it directly until it expires.

What happens if the refresh token is compromised?

If an attacker attains the raw token, they can use it until it expires. He can now access everything till the token lives.

After that he needs to login again.

What happens if we know that a hacker has access to our refresh token?

We destroy all the associated refresh tokens from the db. But the hacker can still enjoy the access till the access token lives and that is a max of 15 minutes. That is the price we pay to reduce the db calls for authentication.

Why not just use a single token?

Using a single token sure seems the easy straightforward option. But this way we would need to call the database on every db call we make that needs authentication. Or have the user login again after small intervals.

Auth Middleware and RBAc

Auth middleware is added before all the secured routes, and it checks for if the access_token matches the signed hmac, for authentication. Besides this auth middleware, I have also added Role-Based protection by allowing only some roles for specific routes. This is done by getting the user_role from the request and verifying it against the allowed list of roles. With this no applicant can access the routes of an admin.

Now the problem is that the request could be modified or lets say it could be tampered with to change the roles.

That issue is resolved in the auth middleware. After the user is authenticated and the token is generated there we add the user id and user role in the request and then that updated request moves to the next part of requireRole or authorization.

Multi-tenant Isolation

Multi-tenant isolation is achieved in a similar way. The backend does not expect the different ids, user id or company ids etc to be passed by the frontend as request headers or params. What the backend does is query the database for different ids against their authenticated information. For example if an applicant wants to search its applications, the backend first verifies the authentication status then authorization and then the database is queried using the applicants-id referencing all the related applications. So all the ids and stuff are managed in the backend providing complete isolation to tenants.

This concept of avoiding using params and querying the backend with specific scope of applicant-ids etc is called scoping queries.

This is also called IDOR prevention which means not exposing the database ids to unauthorized users.

I have also used clear ownership checks, directly using databases to verify the ownership. For example there is a method assertJobOwnership, that compares the recruiter’s company id with the jobs company id and if it does not match the recruiter is faced with a not found error. This pattern hides details that the recruiter is not authorized to see.

Within-company roles

When a user is recruited or invited to the company he is assigned one of the 4 company roles, owner (initial), recruiter, hiring manager and hr manager. The main distinction between them is who can manage other members and that is the owner. The rest of the permissions are shared like managing jobs, managing applications etc.

Inviting Members

The owner can invite new members by their email and a role. When the invitation function is call, the backend first checks for if the company exists. Then it checks for if the inviter is authorized to invite. Then we check if the invitee is already a member of the company. Then we check if there exist any pending invitations already. After all these checks we generate an invitation.

The invitation consists of 32 random bytes hashed using sha-256 to prevent brute-force attempts. This is then stored in the db and the raw token is sent to the user email through brevo. The email task is scheduled in redis and BullMQ workers complete it asynchronously. The user clicks the link sent with the token as params. The token is then compared with the one in db. The token is then validated for expiry etc. Then if the user already has an account with the platform he gets the role. And if it's a new user he is redirected to add password etc and is logged in the same way with the company role.

Posting Jobs

When a job posting request hits the backend, first the user is checked for authentication, then for authorization if he’s authorized as a recruiter role. Then the company specific role is verified (for this case every company role is allowed to manage jobs) and then the request schema is validated. And then the job is posted, the details being saved in the database.

N + 1 Problem

The N+1 problem is when you make one query to get a list of records, then make N additional queries, one for each record to fetch related data. In my public job board, I initially fetched all open jobs, then looped through each job to fetch its company name. For 20 jobs, that was 21 queries.

I fixed it by joining the jobs and companies tables in a single query. The job board now loads in one query instead of 1 + N, regardless of how many jobs are returned. In cases where join was not possible I made 2 queries one to fetch the record ids and the second query to fetch the records using ANY($1) reducing the N + 1 count to just 2. This helps us from overwhelming the db client.

Cursor Pagination

This project uses cursor pagination chosen after comparison with the offset pagination.

Cursor pagination uses a unique identifier which is a combination of different attributes that define the sort order. The attributes I am using are date and id. The composite of these two is then encoded with base64 and is passed to the user. Based on this encoded base64 string the next query is made, where first this is decoded and then based on this the query is made. The fetch query is told to fetch one extra record than what is asked for by the user, and then the base64 string is made from the attributes of that extra record. This extra record is a lookahead and is not returned to the user. If the length of the query response is less than expected it means we have no other records left.

Offset pagination simply uses the page size, an offset and a page number based on these two the database query is made and the results are displayed.

Why choose cursor pagination over offset pagination?

In applications like the job-portal systems, new objects/jobs are being added and deleted all the time. So when a user is searching through jobs and a new job is added or deleted we want to make sure that the user does not miss any jobs or have to see the same job twice. These problems exist with offset pagination. As when the user lets say move to page two and during that a new job is created the user will now see the last job from the previous page as its first job. In case of deletion if a job from page 1 is deleted during the transition to page 2 we may miss the first job from page 2 as based on the index and order it now belongs to page 1.

As cursor pagination uses attributes like date and id, these experiences for the mentioned reasons can not happen. We simply ask the database to provide us responses from a specific id of a specific object and not an index that could change for different reasons.

Another reason is that cursor pagination is faster and has a consistent speed at scale compared to offset pagination that slows down. Offset pagination slows down as the database has to go through all the records just to fetch the records for a specific page. Like on page 20 with offset 20 it has to go through 400 records just to return the last 20 records. It's evident how bad that would be at scale. We could use high offset value to reduce frequent searches but it would still cause similar problems at scale.

Cursor pagination on the other hand could even be sped up if index the attributes used for querying the records which for us include date and id (though id is already indexed as its a PK).

What is wrong with cursor pagination?

With cursor pagination if the encoding string is lost we can not resume our search or scroll from the same point we will have to start from the very first record. Simply put the user cannot jump between pages. But that is not a problem for a job portal where users scroll through the jobs and do not jump pages.

Now for cursor pagination there could be a question for why not just use id or just use date for sorting. The answer is if we were to use just the date, there would be instances where multiple jobs could have the same date of insertion/creation. Or if we just used id the sort/order would be stable but would not be newest first which is the natural expectation from the user. So for it to be newest first we need to sort using date and to avoid the tie breakers we need to use the ids.

Indexing

I have indexed different attributes from different tables to make frequent queries faster. I will discuss here what those attributes are and why indexing them was a good option.

Idx_companies_status

companies are filtered based on their status and the records returned are with status verified. Any other status is not considered. Idx_companies_status supports queries that filter companies by status, such as listing verified companies or finding companies in a particular state. For other queries its usefulness depends on the user of relevant indexes.

idx_invitations_expires_at, idx_refresh_tokens_expires_at

The job cleanup cron jobs use invitations expires_at and refresh_tokens expires_at attributes to delete the expired jobs, so they are also accessed quite frequently. And indexing them really helps the worker to clean these up faster (as there could be a lot of expired invitations and refresh_tokens at scale).

Idx_invitations_token_hash

invitations _ token_hash is also accessed frequently, it is accessed whenever a new member is added. So querying from the invitation token to match the token_hash is going to be slower to speed it up. I have indexed the token_hash as well.

Idx_jobs_search

The jobs search idx is based on the search_vector that is gonna be needed at every search i.e. frequently needed. As the search vector is going to be a huge collection indexing them does increase the size but it also makes the search faster at scale.

idx_jobs_status_created

the jobs status_created idx is a composite idx based on two attributes, the jobs status and the time it is created_at. The jobs board filters the jobs based on status like only the public jobs could be listed at the public board (that’s why it is on the left i.e primary filter) then the jobs are filtered based on how recent they are (new jobs first). A public job board is the most accessed service of any job portal system. That's why the use of composite index to speed up the reads. It however introduces a problem and that is the writes get slower at scale. But slower job writes are not really a problem for the job board, as writes are less frequent than reads.

Foreign key indexes

  • idx_refresh_tokens_user_id
  • idx_email_verifications_user_id
  • idx_recruiters_company_id
  • idx_recruiters_user_id
  • Idx_jobs_company_id

These are the foreign key indexes for the most frequent joins. I picked the most foreign keys needed in the most frequent joins for indexing.

Search & Filtering

Applications, Jobs, Companies, and Members for different user dashboards are filtered based on different attributes. For example companies in admin panel are filtered based on there status, jobs are filtered in the recruiter panel based on if there status draft, open or private and similar filters hold for other options too. These attributes used for filtering are indexed for faster queries as mentioned before.

For searching (specifically used for jobs) I have used a search_vector/ts_vector which is created using a trigger on job insert. It breaks the job title and descriptions tokenized into a vector. It uses GIN index for faster response. The problem with this is that because of the index it uses extra storage, for a small change we need to rebuild all ts_vectors, the write performance is slower.

It is preferred over the other approach of inline computation where the PSQL would have to parse every row in the table during execution.

Redis Caching

The public job portal, as would have the most requests, I have implemented cache storage for that page. Besides this, async jobs like, sending emails for different purposes i.e. member invitation, otp message etc. and for cron jobs like refresh_token cleanup etc. And rate limiting also uses redis caching to track different client requests and block those that are potential attacks.

Cache Invalidation

Caches need to be invalidated, because the data is changed frequently so it could get stale. Like the new jobs could have been added, old ones could have been deleted etc. So what I did is add a TTL of 60 seconds to the cache so after every 60 seconds the cache expired and a new request to the db was made for updated data.

Then in scenarios where user updates the records may be add or delete records the redis cache is invalidated and then updated with data from the database. In scenarios like these when a change is made doing a two phased commit can introduce inconsistencies. Like what if the redis commit fails and what if the db commit fails? If redis commit fails the specific user would have stale cache for the max time duration of TTL of the cache. If the db commit fails all the users no one sees the change, the cache still holds old data and the db never gets the update. That's worse than before as the data is inconsistent between what the user expects and what’s stored.

Why don't files belong in DB?

Large files do not belong in DB because of three costs.

Storage cost

relational db costs around 0.18 to 0.25 usd per gb per month while S3 storages like amazon costs around 0.023 about 10x cheaper.

Throughput cost

PSQL serves data over a connection pool. Every pool is a finite source capped at around 20-100 connections. A database query that returns a resume, must hold the connection during the duration of the transfer that could be a few seconds. If the file size is 500kb and the throughput is 5 Mbps it would take 0.8 seconds for the complete transfer and during that time the connection can not serve any other query. S3 based storages on the other hand are built specifically for large file transfers and they do not hold the connection either.

Operational Cost

Backups

storing resumes in PSQL results in increased sizes for the backups, backing up would take hours.

Migrations

20gbs of Blob-heavy tables would take hours migrating.

Replicas

WAL replication sends all the changes to replicas. Binary writes are replicated byte for byte adding huge pressure on bandwidth.

Presigned Uploads

Presigned uploads are used in s3-uploads without revealing the server side secrets to the client. The server sends a request to the S3 api along with its S3-credentials. The server returns a presigned-url that is sent to the client. The client then uses that presigned url to upload the resume.

Profile Snapshots

Profile snapshots are an important part of a job application. Profile snapshots are snapshots of applicant details taken when the applicant applies for a job. The snapshot holds the historical record of the user profile which allows the recruiters to judge the applicant profile based on how it was at the time of application.

Application Transaction

The need for application transactions arises from the problems we could face from the pre-signed url. For example lets say the query to store the presigned url (to later retrieve the resume) is completed but the update fails. What will happen now is that we have a reference url but we don’t have any file stored against it. To make these queries atomic and to avoid these inconsistencies we wrap these up in a transaction. The psql transactions run on a single connection as multiple connections does not guarantee atomicity. It rolls back on any error to avoid the inconsistencies.

Idempotent Application

The job applications are idempotent. Idempotent means that if an applicant applies to the same job multiple times the state of the db is gonna stay the same to what it would be applying once. The applicant is allowed to apply multiple times but only the first application would be considered and the user would just get the success message every time, regardless the nth count it is. This is to maintain a state that is expected throughout the system, like no multiple applications per applicant per job. The applicant always gets the same success message, regardless of whether the application was new or a duplicate

Stage Pipeline

Each application goes through a stage pipeline. The pipeline has stages: screening, interview, final_interview, offer, hired, rejected. The transition rules are:

  • The targetStage must be a valid stage name in stage order
  • The current stage must not be hired or rejected.
  • The targetStage index must be greater than the current stage index (forward only)
  • Exception: rejected can be reached from any non-terminal stage (it is a special terminal that can be entered from any point)

Why are backward transitions blocked?

They are blocked because moving an application backward from interview to screening would cause confusion and is not consistent with the natural flow.

The status field provides another dimension to the application with ‘withdraw’. The applicant can withdraw his application and when that happens the stage cannot be changed by the recruiter as there is no point in doing so.

Interview Scheduling

An interview scheduling feature is added where the recruiter can schedule an interview with a specific applicant, against an application. The interview table references the application table. When an interview is scheduled the stage is implicitly transitioned to interview. This transition would be expected by the recruiter most of the time and the absence of this feature could lead to inconsistencies where an interview is scheduled for an application that is staged at screening. An email is delivered to the applicant informing him of the meeting through the smtp server. The email transfer is asynchronous and is carried out by the BullMQ worker. On failure it automatically retries and keeps retrying.

Interview Feedback

The recruiter can add the interview feedback and then can either move the candidate to the next stage which would be the logical next stage or it could reject the application.

The interview feedback is immutable which means once a feedback is given it cannot be overwritten as that could cause a data integrity problem.

What if the recruiter makes some sort of mistake with the feedback?

The project does not solve this problem but a solution to it could be adding a small time window in which the recruiter is allowed to edit the feedback. How we can implement this is by using an attribute feedback_created_time and allow the edits when the difference between feedback_created_time and current time is less than the window we have decided on.

Admin moderation

The admin gets an overall view of the system entities. It has access to all the companies, jobs and users.

When a user registers a company the status for that company is set to pending. That status is then verified by the admin and then it can post jobs and do hirings. If a company violates some standards, the admin can suspend that too. A suspended company can be reverified after investigation.

What happens to the company jobs when it is suspended?

If a company is found to be engaged in fraudulent activities and the admin wants to suspend it, it's only obvious that the admin doesn’t want any more applications to the company jobs so the backend suspends all the company jobs in the backend.

The admin can view all the jobs and can target specific jobs changing their status, closing them.

The admin can also view all the users and can suspend specific users and then also verify them again. Suspending a user deletes or suspends all the refresh tokens associated with him. But the access token remains valid until it expires (max 15 minutes). After that expires the user loses all his access.

An admin can mistakenly suspend itself and lock itself out. To prevent that an id check is added where the admin user_id from the jwt token (added at the time of login) is matched against the user_id we are trying to suspend, and the operation only proceeds if the two do not match.

Why async

Slow operations should not block the HTTP response. Email delivery, pdf processing and api responses can take 100 of milliseconds or more. Making the user wait for them increases latency and the user perceives the application as slower. And if the slow operation fails the user gets an error unrelated to what he submitted.

A job queue is durable; an in-process event emitter is not. Node.js EventEmitter fires and forgets inside the same process. If the process crashes between emitting the event and handling it, the event is lost with no record of it ever happening. A job queue writes the job to Redis before the HTTP response returns. The job survives a worker crash or a server restart.

BullMQ stores each job as a Redis key. When you call queue.add('job-name', payload), BullMQ serialises the payload to JSON and writes it into a Redis sorted set. Workers poll that sorted set and atomically move jobs to a "processing" state. If the worker crashes mid-job, the job moves back to waiting after a configurable lock expiry.

At-least-once delivery means jobs can run more than once. BullMQ guarantees that a job will eventually be processed at least once. It does not guarantee exactly once nor does it guarantee the success of a job. If a worker completes the job but crashes before acknowledging it, the job may run again. Workers must be idempotent running the job twice should produce the same outcome as running it once.

Three things should happen before returning HTTP 200 (or 201). Validate the input. Write to the database. Enqueue the background job. Everything else, sending the email, processing the file, calling the third-party API etc. happens after the response is returned. The user gets a fast response and the side effect happens within seconds.

Queue vs event bus

The question is "what happens if no worker is running?" A job queue holds the job in Redis until a worker picks it up. The job can wait hours or days. An event bus fires the event immediately. If no worker is listening, the event is gone. For this system emails, file processing, a queue is the right choice because worker downtime must not cause data loss.

Concurrency is explicit in BullMQ. When you create a Worker, you set a concurrency number. { concurrency: 5 } means the worker processes up to 5 jobs simultaneously. Default is 1 (sequential). This is different from the HTTP server, where every request is handled concurrently by Node.js's event loop.

Queue and Worker setup

BullMQ is the queue this project uses to queue the jobs in redis and the worker reads those jobs and processes them. Both BullMQ and the worker have separate redis connections that is because:

  • BullMQ uses blocking Redis commands and Lua scripts that require dedicated, non-shared connections to avoid blocking other operations. For that purpose the connection is not shared between the queue and the worker

The job request is made by calling the job-key stored as a key value pair in BullMQ, from there the job is stored in the redis. A worker retrieves the job, processes it and after completion marks it as done. If a job is failed the worker retries unless the job is completed.

If the job fails for some reason the worker retries it again and again. To handle this we add a specific case where we implement the logic of preventing the retries to go on forever. We add a limit to the number of retries.

If the job succeeds but the worker fails to register its completion the worker retries the job later and then updates the status. The jobs are idempotent so the state says the same no matter if the job was executed once or more than once.

Retries and Idempotency

The default job options from BullMQ uses attributes like attempts to specify the number of retry attempts to make before stopping and backoff to specify the delay between each retry and how that wait increases after each failure like sequential increase, exponential increase etc. The project uses exponential increase.

Retries are important because if a job fails without completion it becomes a permanent data loss unless a retry attempt is made.Backoff is important because without it the worker immediately retries the failed job flooding the already struggling service.

The job is safe to retry because the worst case is a duplicate email, it is annoying, but not data corruption. True idempotency would require deduplication.

For other jobs we have explicitly handled the conflict at database level using ON CONFLICT DO NOTHING. So when a conflict happens the database just returns a successful response without doing anything.

Resume Processing

The user apploads a resume to the frontend. The request to the backend is made with the file. The backend requests the s3 server which responds with a presigned url. This url is then used to upload the resume. After completion a job is run to parse resume and count words and update the database with the word count. The job is enqueued by BullMQ and picked up by the worker.

We await the s3 stream chunks and collect them in a chunks array which is then concatted to a buffer.The buffer is passed to the pdf parse after which it is available for garbage collection. While being parsed a resume occupies more space then its initial size. With a 100mb resume the worker could experience high memory pressure, slow processing and risk of OOM errors if multiple large files are processed concurrently, So a size limit is applied rejecting the oversized resumes.

Scheduled Jobs

Repeatable jobs are scheduled as cron jobs with defined schedules. For example this project schedules jobs to clean up expired refresh tokens and OTPs and also schedules a job to send an email digest to the recruiters.

Rate limiting

This project uses express rate limiting and redis stores the request counts for consistent rate limiting across instances. Each request is counted in redis. if the limit is reached the request is rejected. All of this happens in the middleware. The middleware supports both a global limit and route-specific limits for example, rate limiting for auth.

Rate limiting prevents the project from abuse. The attackers cannot bruteforce a login because rate limiting increases the cost.

The reason we are specifically using redis and not the servers storage is because if a server fails the counter is set to 0. And for multiple servers the counter would be split across servers, making the limit unreliable. With redis the count is the same for all the servers.

Structured Logging

This project uses pino for structured logging. Each log has its own request id. Structured logging writes a JSON object per line. Log aggregators index these fields. They are then queried with the aggregator's UI.

Using console.log produces a plain string. To find all login events for a specific user, you would need grep. grep on a log file works fine on one server. It breaks when you have:

  • Multiple servers (logs are on different machines)
  • High volume (millions of log lines per day)
  • Complex queries ("all 500 errors in the last hour on the /api/auth/login route")

Health & graceful shutdown

The project has two health points: /ready to check if the server is ready and /health to check if all the services are healthy, it checks for the database and the redis services for if they are working correctly or not. If everything is good the endpoint returns 200 else 503 the internal server error.

Graceful shutdown is implemented with which on SIGTERM the server gracefully shutdowns the services and blocks new requests and also waits for requests being processed to be completed. The worker is also gracefully shutdown, it waits for the ongoing jobs to complete and then shuts down.

Deploy

The project uses docker images for the api, the worker and the one time migration task. These images use multi-stage build to reduce the image size and also reduce the attack surface by not including the source code.

Docker compose starts the full stack: PSQL, Redis, MinIO(s3-storage), the API, the worker and nginx as a reverse proxy. The migrate service runs before the API starts. It requires the PSQL to be healthy. The API and worker depends on the success of migration, that is to avoid starting with a missing schema. Environment secrets are injected on the run and are not included in the image. THe worker runs as a separate process so API failures do not affect jobs processing. Health checks on the services helps docker restart them.

A note on live deployment

The project is containerized and ready to deploy, but I haven't hosted it live. Render's free tier does not support background workers, and the worker is important to this system. Without it, emails and resume processing do not run. Paying $7/month for a portfolio project wasn't suitable for me as a student, so the deployment configs are included in the repo and the system can be brought up locally or on any paid host with docker compose -f docker-compose.prod.yml --profile migrate up -d --build.

Conclusion

This project took me from basic CRUD logic to building a full-fledged multi-tenant system. It helped me understand how production systems are built and held together. It helped me understand and implement the concepts of idempotency, preventing IDOR, dual-token authentication, RBAC, and more.

Every decision was a trade-off between different options. I learned that no single choice fits every system.

This project serves as proof of my backend design skills. It shows I can build a multi-tenant system with async processing and clean deployment pipelines. The repo is public, the deployment configs are included, and it runs locally or on any host with Docker Compose.

If you have questions or feedback, reach out at: https://www.linkedin.com/in/muneeb-ahmad-se/

Repo: github.com/Muneeb-Ahmad-404/Talent-Hunt

Top comments (0)