DEV Community: Francisco Luna

I built Envoy - Control Plane for Painless PostgreSQL Schema Migrations

Francisco Luna — Mon, 23 Mar 2026 00:35:13 +0000

If you have some years of experience as a FullStack Developer, Dbadmin or DevOps engineer, you already know managing database migrations across Staging, QA, and Production is a mess. You already know how this game work:

We switch .env files manually is dangerous and error-prone
Then forget GRANT permissions in pgAdmin causes production downtime
Audit logs are non-existent, scattered in sql files, or hard to verify
Multiple tools required for simple database operations
We build custom scripts we forget how to use

Here's how every Friday was like in a regulated environment from an actual registered startup:

I get the development database credentials
Then replace the environment variable in .env file
Run migration command with the orm
Verify migration was successful
Try changes in development
Realize I forgot to give permissions to new tables
Go back to pgAdmin/psql to fix permissions
Try again
Repeat next week across all environments

Introducing Envoy

This month I've built Envoy. It is an open source control plane for your PostgreSQL database schema migrations built as a Golang 1.25.0 API, which is consumed by a React/Vite Frontend. It connects to your database using PGX and encrypts all your connection strings using AES-256.

Envoy does not care if you wrote the SQL yourself or if it comes from your ORM. It allows you to visualize the GRANT permissions, log migrations and work by environments in case your infrastructure has multiple projects or you manage multiple clients.

How to use Envoy

Clone the repository and initialize it using Docker Compose
Create an account and login
Create a new project and set up your first environment
Connect to your Postgres database as an admin or any user capable of modifying the Postgres schema (I advice you to experiment with the PostgreSQL database from the dockerized environment first)

Why Envoy

I built Envoy because no ORM considered actual compliance requirements and multiple users when working in multiple databases. They asssume you're always running the database as doadmin or postgres, which is a lie in actual startups with compliance.

Every deployment was full of anxiety and rushed SQL commands to see permissions, not because I was careless, but because I was managing 3 environments at the same time on my own; each one with different users and needs. Also, the previews I got were simple command line tables that were gone and forced me to open pgAdmin to further inspect the schema.

I realized this was not sustainable in the long term, and I started building Envoy to manage my clients' database schema migrations seamlessly.

Key Features

Environment Management: Add/switch between dev, staging, prod databases
Migration Editor: Write SQL with syntax highlighting
Schema Preview: See changes before applying
Permission Auditing: GRANT verification per user and environment
Audit Trail: Complete history of all schema migration operations

Next Steps

All feedback is welcome! As I'm in the MVP stage, I'll be working on UI enhancements, finishing the environments CRUD, integrating both unit and integration tests in the codebase and simplifying the onboarding process.

You can support Envoy by giving it a star in the following repo:

https://github.com/franciscoluna-28/envoy

Leave a comment and share if you've found this tool useful :)

The Founding Engineer’s Checklist: Every Essential System for a Battle-Tested Launch

Francisco Luna — Wed, 18 Mar 2026 21:55:41 +0000

I've developed products for startups. Most of them have failed and not because of the code. The backend was functional, queries were fast and the infrastructure could handle actual users, even if the traffic was small.

Why did them fail? Lack of PMF, scope creep and tedious operation management that made the 24/7 IT support guy for months.

The real world of startup software development is not as easy as deploying your app and asking users to come; you also need to be able to add new features and see how users interact with your app.

I had to learn these things with downtime and scope creep, but they will ensure you have a battle tested launch.

Note: This blog goes beyond rate limiting and using environment variables, often times, real world services are not mentioned in coding bootcamps or university.

🔴 Critical - Infrastructure and Deployment

Provisioning: Ensure you have your infrastructure documented, whether in Terraform, Pulumi or simple documentation if your product doesn't have too many moving parts yet. Your infrastructure and setup needs to be replicable.
CI/CD Pipelines: Make sure you have deployment scripts or actions to deploy your system and application. Always double check the secrets, permissions and environments available in your setup.
Secrets Management: Annotate all your application secret templates in .env.example files, secure them in your cloud provider and make sure to rotate the credentials accordingly.
Testing: For God's sake, don't skip testing even for MVP. Prioritize testing inputs, happy paths and use cases. I recommend using Playwright to test the entire application workflow and integration tests to test your use cases.

🔴 Critical - Data Layer Reliability

Database Migrations: My least favorite one. Make sure you have a clear way to do this. You can use your ORM built in migration tools, custom scripts, CI/CD pipelines or even manual deployments.
Backups & Recovery: Create backups of your database at least daily if you're using a managed database. Back up your server in case you're using a VPS.
Caching Strategy: Make sure your Redis setup and materialized views work as expcected and have clear policies.

Remember this small checklist before running schema migrations:

A new field in a table should never be NOT NULL. It should always have a default value or be nullable.
Don't take normalization too seriously. Sometimes it's better to use JSONB over complicating a new product with new tables.
Your new schema should be compatible with previous features. Whatever trade-offs you made, will have consequences later.
Don't trust your ORM too much. Always double-check the SQL code yourself.
Apply the principle of least privilege to everything. This means if your user needs to read the database, give them read-only access.

(I'm also building a tool to make this process less painful in PostgreSQL! - It'll be revealed soon.)

🔴 Critical - Identity and Security

JWT tokens: It's up to you if you use JWT with a long TTL or refresh tokens. What matters is that you save the JWT secret in an environment variable and you have a disaster recovery strategy if it gets exposed.
Auth: Don't reinvent the wheel. Keep it simple and don't implement more services than needed whether you use custom auth or a managed service. If you use a manage service, keep the data in your own database.
RBAC & ABAC: Have your roles and policies documented. Create custom policies and checks to prevent repeating if statemements across your app.
SSL & Proper Configuration: Make sure your domains and infrastructure run over HTTPS, you have your robots.txt files, metadata, icons, SEO, app redirections and external services.
Transactional Messaging: Set up a service such as Resend or AWS SES and configure the SPF and DKIM records in your DNS. Use proper email domains and avoid spamming your users. To create templates you can use MJML, as it allows you to create professional and responsive emails.

Tips:

If you're using images or fonts in your transactional emails, make sure they belong in your domain (e.g., images.mydomain.com/logo).
Keep your emails as simple as possible and make sure your links work as expected.

🟡 Important - Observability

Frontend Analytics: Don't let your marketing team go blind. You can use Umami for this, but other tools may work as well.
Frontend error handling: You need to see if your application is not working well for other users. I recommend using Sentry to monitor your frontend errors.
Logging: You need to use structured logging in services such as CloudWatch or BetterStack. Even a well structured Pino in a Docker container is valid if you're in the MVP stage. One important note, never display passwords or credentials in your loggers. Stick to non-sensitive fields like the user ID.
Health Checks and Alerts: Keep it simple for MVP. Use a service like BetterStack or your cloud provider to ping your /health endpoint every 2 minutes or thereabouts. This way you'll receive an email when one of your services is down.
Server Monitoring and Observability: This step is not often needed if your system is small. You can implement Prometheus and Grafana to monitor your infrastructure state (RAM usage, clusters metrics), although most cloud services have their built in monitors and metrics.

Tips:

Make sure to ping the database and other critical services such as Redis when calling your /health endpoint.
Prometheus and Grafana are powerful but high-maintenance. Stick to built-in cloud metrics for your MVP to keep your focus on the product, not the dashboard.

🟢 Good to Have - Operations

Internal modules: Make the admin modules and initial app configuration seamless to use. Remember marketing, product and your internal team also need to use the application. Give the other departments what they need so you're not the IT support guy all day :)
Setup should be easy: You should not be using manual SQL and scripts every single time you deploy the system. Make the first app setup to initialize the platform intuitive, even if it's for yourself only. Don't develop hate towards your own system.
Architecture and scope matters: I once had to build a product solo with no guidance. I built poor architecture and I took every single request from the client. And of course, it collapsed under its own weight...

Tips:

Learn software architecture (how to create code that can be tested and maintained in the future)
Internal tooling is as important as customer facing features in most serious apps. Yes, your CEO and the marketing team are also your clients.
Get the user count and basic metrics for the team please. This will save you hours of IT support.
Learn about negotiation (how to set boundaries, build a MVP with proper scope and educate non-technical people about product development).

🟢 Good to have - Documentation

OpenAPI documentation: OpenAPI 3.0 (OAS 3.0) is a specification you can use to document your API. You don't need it, but it makes collaboration and scalability easier. If you're building a serious product that will be maintained over time, consider adding it.
Onboarding docs: Document the current state of the system and what needs to be done, feature flags, how to run things locally, important trade-offs made, the architecture and tech stack of the system and alongside the most critical technical debt to pay.
Scalability plans: They don't need to be AWS architecture diagrams yet; a Notion page is enough. But you need to identify the current bottlenecks in your system and have a plan to scale.

Tips:

Make sure your API has consistent response structures, examples and clear body data.
Even if you have plans to work solo, the documentation is useful for your clients and your future self.
Most bottlenecks to scale will show up in the server and application layer. For this, you'll need to use load balancers, consider serverless architecture or use scalability groups.
Node.js is good, but sometimes Golang can be better for specific services. Understand your technologies well.

You can learn more about my work on my website, where I help companies build their MVPs and Backend Applications: https://www.itsfranciscoluna.com/

Thanks for reading! Leave a comment if you want to share other tips you know for a successful MVP launch :)

The Trench Doctrine: A Third-World Founding Engineer's Post-Mortem

Francisco Luna — Sun, 18 Jan 2026 16:33:16 +0000

Introduction

Most people think I'm stubborn about startups, products and the standards I developed are insane.

Maybe they are. But when you've faced user complaints directly, you have to manage a database at 12pm on a Saturday because of broken SQL scripts, phantom docker containers, extreme technical debt, talking with the CEO directly and falling for scope creep, you already know I had to learn how this game worked.

Let's mention the 7 most important points and lessons I learned while being a founding engineer from a third world country, Venezuela.

1. Scope Creep

I've seen this a lot. Adding features because they're cool or "just in case". When you're a founding engineer, there's a skill you must develop. Negotiating scope. Because the CEO and the non-technical part of your team want to add features all the time.

I'm skeptical about scope nowadays. Because I was responsible of building products no one understood and I was saying yes to every single one of the CEO's ideas.

My advice? Please, keep scope simple for MVP and introduce the core features. Iterate as you go and always validate your feedback with other users and the competition.

Always prefer a tough discussion that'll save the project over sweet acceptance.

2. Poor Software Architecture

Let's be honest. Proper software architecture is not overengineering. It's about using the 20% of techniques and patterns that will give your codebase of results. This is called the Pareto principle.

For a MVP you don't need:

Utility classes
Complex mappers
Aggregators

But:

DTOs
Repositories
Entities
Handlers
Use cases

The core goal of software architecture is to make the software flexible and to iterate faster through decoupling. It's a simple concept: Your code should not be glued together like a wall, but rather it's a set of flexible Lego blocks you can work with.

Because if your stakeholders ask you for a change of the requirements (spoiler: they will) or if you need to fix a bug you, you're not spending 5 hours going through spaghetti code but 30 minutes tweaking a use case, making things easier to test and deploy.

3. Complexity

One day some guys on LinkedIn were talking to me about their architecture. They were telling me they were using microservices. I asked them about their DAU rate. It was less than 100.

In that moment, I wanted to turn off the computer just to process this.

The problem is not microservices themselves. They're a tool to be used once your company and products scale and you need to separate responsibilities. But are you aware of the maintenance overhead you're introducing with this choice?

Choose a proper modular monolith when you're getting started, collect metrics, talk to users and iterate your business idea as you go.

The simpler it's to get started and iterate, the more chances of winning you have. Because you have more room to test and experiment, and in the real world, users don't care about your API gateway, only if the feature works and if it's reliable enough.

4. Not everyone will get it

Sometimes, it's better to be alone with senior engineers, CEOs, CTOs and those who are with you in the trenches.

No. I'm not telling you to stop networking but to get your priorities right.

When you're building a complex multi tenancy architecture, writing tests to prevent lawsuits and stress testing the system you don't need to be with people who are asking you how the Python syntax works or how they can center a div.

The best choice you can make is to bet on yourself, your future, skills and the people who are with you in the trenches.

Mentor others and document, but value your time for deep work.

5. Testing DOES matter

I still remember the tragic day of launch where a major stakeholder found they could not use specific characters when trying to login into the system. We had little to no tests and we were assuming our input DTOs were working fine.

Spoiler, they were not. You don't need 100% test coverage, but always consider testing the following modules through both unit and integration tests:

Auth
Payments
Input DTOs
Permissions
Access control systems

6. Academia might betray you

I'm not telling you to stop studying or learning, but the opposite, knowledge is power. However, you need to develop pragmatism and avoid falling for dogmatism (dogma).

Dogma is having the belief things should always be done in a specific way no matter the situation. Pragmatism, on the other hand, is knowing when to break a specific rule while considering the trade-offs.

Look, once I designed 35 tables using the 5 possible NFs of Database Design for MVP.

For what?

87 users in 2 months
Complex queries with 3 joins
Servers down
User complaints

When I read Designing Data Intensive Applications and I learned it was fine to just have a JSONB row in my tables, it changed my life.

Because sometimes the only way to win and keep your sanity is to break the rules. Sadly, this is something you learn only through experience.

Which is why I'll still insist about avoiding dogma.

7. Life outside startups

I wish someone would've told me these things before. I learned these the hard way and I hope you can learn from them.

Protect your relationships at all cost
Learn how to read and understand legal contracts
Learn about human psychology and power dynamics
Learn how to say no
Learn how to negotiate
The startup is a medium, not the end
Equity is NOT free. Make sure it aligns with your future plans
Save yourself. Not a company. Not a startup. Your own self

Conclusion

As Ging Freecs says:

"You should enjoy the little detours to the fullest. Because that's where you'll find things more important than what you want."

Thank you for reading. Don't take this advice literally. This is only my philosophy about startups at 21.

An Introduction to Docker: Stop asking your stakeholders to install Postgres! 🚀

Francisco Luna — Sun, 11 Jan 2026 20:32:58 +0000

I once asked a major stakeholder to install Postgres on his laptop just to see my progress.

It was September 2024. I was earlier in my career, and the shame of that moment still fuels my obsession with Docker and DX today.

The project was in its early stages. When the stakeholder asked to try it himself, I gave him a to-do list:

Install Node.js and PNPM.
Set up a local Postgres instance.
Switch to a Linux environment.
Manually configure credentials and run migrations.

I didn't share a project, I shared a liability. Today, I don't ship code; I ship environments.

That experience was my wake-up call. I realized that as a developer, my job isn't just to write code, but delivering value. If a stakeholder or a new teammate can't run the project in minutes, I've failed.

I'll help you save time and be more effective with one of my favorite development tools ever created: Docker.

Why using Docker? Is not that a DevOps thing?

Yes. Docker is used for DevOps. But almost no one tells you it's used for professional development environments. Would you prefer to install Docker Desktop and run a few commands to start building or:

Manually set up Postgres in your machine
Set up WSL
Ensure each service is running
Set up each service individually
Ensure the versions match
Etc

This process took my company's onboarding time from 5 days to a couple of hours.

Getting Started with Docker

1. Install Docker Desktop

The industry standard for Mac and Windows. If you’re on Linux, you already know the drill.

Docker Desktop for Windows

2. Verify Installation

Open your terminal and check the installed version


docker --version

3. Your First Container

Run this to see Docker pull an image and spin it up in seconds:


docker run hello-world

About Docker

Docker lets you package your app with everything it needs to run; from code to runtime, and libraries into a single unit called Container; which can run in any OS and machine.

It's like a bassline. You don't notice it's important until it's gone.

Docker vs Virtual Machines (VMs)

VMs are heavy, they require a full copy of the operating system and consume significant RAM and CPU.

On the other hand, Containers share your machine's OS kernel and they only install the libraries and dependencies they need.

The Three Pieces of Docker

Dockerfile: This is your blueprint. It allows you to specify Docker how you want to build the environment (OS, the Node.js version, and your dependencies)
Image: The snapshot. It's what you get when you build the blueprint given by the dockerfile. It's a read-only template that contains your application and its environment.
Container: This is the image in action. You can start, stop and delete environments without affecting your actual computer.

Solving the "Postgres Headache" with Docker Compose

Remember the list I gave that stakeholder? "Install Postgres, run migrations, set credentials..."

With Docker Compose, we turn that entire manual process into a single command: docker compose up.

Think of each service (application, database, message queue) as an instrument. You're combining them to create a band, but in YAML.

It's important to mention this tool is used for development environments and small MVPs in production. You'll need load balancers and eventually K8s for scaling a complex system.

Time to Build

Suppose we want to dockerize a Next.js app and run a database with it, we're going to create 3 files in the root folder of the project.

docker-compose.yaml
.dockerignore
dockerfile

Dockerfile

This is how Docker will build the image to run our Next.js application.


FROM node:20-alpine

RUN npm install -g pnpm

WORKDIR /app

COPY package.json pnpm-lock.yaml* ./

RUN pnpm install

COPY . .

EXPOSE 3000

CMD ["pnpm", "dev"]

Dockerignore

Just as we have our .gitignore we also have .dockerignore. This file is responsible for preventing packaging sensitive or unnecessary content from the Next.js app into the container.

node_modules
.pnpm-debug.log

.next
out
build

.env
.env.local
.env*.local

npm-debug.log*
yarn-debug.log*
yarn-error.log*
.pnpm-debug.log*

.git
.gitignore

.DS_Store
Thumbs.db

Docker Compose File

This file is responsible for orchestrating the containers. You declare each component of your application as a service. Remember to create a Dockerfile for the core Next.js app. The data of postgres will persist on the volume we have created.


services:
  db:
    image: postgres:15-alpine
    container_name: nextjs-db
    restart: always
    environment:
      POSTGRES_USER: admin
      POSTGRES_PASSWORD: supersecretpassword
      POSTGRES_DB: db
    ports:
      - "5432:5432"
    volumes:
      - postgres_data:/var/lib/postgresql/data
  app:
    build: .
    container_name: nextjs-app
    ports:
      - "3000:3000"
    environment:
      DATABASE_URL: postgresql://admin:supersecretpassword@db:5432/db
    volumes:
      - .:/app
      - /app/node_modules
    depends_on:
      - db
    command: pnpm dev
volumes:
  postgres_data:

Notice I use node:20-alpine. It uses Alpine Linux under the hood; you can choose other Node.js images depending on your needs. Remember that in production environments, we optimize for image size and security surface. We also use depends_on because your app shouldn't start if the database isn't alive.

We're saving the database data in a named volume called postgres_data. This way, when we turn off the container, the data will still persist.

Tip: For resilience, combine depends_on with a healthcheck to ensure the DB is actually ready to receive traffic, not just "running".

One Command to Rule them All

With these three files in your root directory, you’ve turned a 1-hour headache setup into a 10-second command:

docker compose up

By running this, Docker will build your Next.js app, pull the database image, set up the network between them, and start your dev server with hot-reload enabled.

You can even add startup and seeding scripts if needed.

Stop Selling Hours, Start Delivering Systems

I used to think Docker was only a complex DevOps tool until that afternoon in 2024. Where I learned that seniority isn't just about writing fancy code, but about using code as a tool for the business.

I stopped being just a programmer and became a Software Engineer. I reclaimed my time, improved my team's DX, and made sure that no stakeholder ever has to install a database on their laptop again.

If you want to scale your project or your career, start containerizing today. Your future self and your team will thank you.

What was your worst setup story? Share below in the comments!

Building a complex application or a high-performance system?

I help startups engineer scalable applications and internal tools that stay fast as they grow. See how I build scalable systems here: 👉 www.itsfranciscoluna.com

Learn Docker: Stop asking your stakeholders to install Node.js

Francisco Luna — Sat, 10 Jan 2026 13:36:13 +0000

I once asked a major stakeholder to install Postgres on his laptop just to see my progress.

It was September 2024. I was earlier in my career, and the shame of that moment still fuels my obsession with Docker and DX today.

The project was in its early stages. When the stakeholder asked to try it himself, I gave him a to-do list:

Install Node.js and PNPM.
Set up a local Postgres instance.
Switch to a Linux environment.
Manually configure credentials and run migrations.

I didn't share a project, I shared a liability. Today, I don't ship code; I ship environments.

I'll help you save time and be more effective with one of my favorite development tools ever created: Docker.

Why using Docker? Is not that a DevOps thing?

Manually set up Postgres in your machine
Set up WSL
Ensure each service is running
Set up each service individually
Ensure the versions match
Etc

This process took my company's onboarding time from 5 days to a couple of hours.

About Docker

Docker lets you package your app with everything it needs to run; from code to runtime, and libraries into a single unit called Container; which can run in any OS and machine.

It's like a bassline. You don't notice it's important until it's gone.

Docker vs Virtual Machines (VMs)

VMs are heavy, they require a full copy of the operating system and consume significant RAM and CPU.

On the other hand, Containers share your machine's OS kernel and they only install the libraries and dependencies they need.

The Three Pieces of Docker

Dockerfile: This is your blueprint. It allows you to specify Docker how you want to build the environment (OS, the Node.js version, and your dependencies)
Image: The snapshot. It's what you get when you build the blueprint given by the dockerfile. It's a read-only template that contains your application and its environment.
Container: This is the image in action. You can start, stop and delete environments without affecting your actual computer.

Solving the "Postgres Headache" with Docker Compose

Remember the list I gave that stakeholder? "Install Postgres, run migrations, set credentials..."

With Docker Compose, we turn that entire manual process into a single command: docker compose up.

Think of each service (application, database, message queue) as an instrument. You're combining them to create a band, but in YAML.

It's important to mention this tool is used for development environments and small MVPs in production. You'll need load balancers and eventually K8s for scaling a complex system.

Time to Build

Suppose we want to dockerize a Next.js app and run a database with it, we're going to create 3 files in the root folder of the project.

docker-compose.yaml
.dockerignore
dockerfile

Dockerfile

This is how Docker will build the image to run our Next.js application.


FROM node:20-alpine

RUN npm install -g pnpm

WORKDIR /app

COPY package.json pnpm-lock.yaml* ./

RUN pnpm install

COPY . .

EXPOSE 3000

CMD ["pnpm", "dev"]

Dockerignore

Just as we have our .gitignore we also have .dockerignore. This file is responsible for preventing packaging sensitive or unnecessary content from the Next.js app into the container.

node_modules
.pnpm-debug.log

.next
out
build

.env
.env.local
.env*.local

npm-debug.log*
yarn-debug.log*
yarn-error.log*
.pnpm-debug.log*

.git
.gitignore

.DS_Store
Thumbs.db

Docker Compose File

services:
  db:
    image: postgres:15-alpine
    container_name: nextjs-db
    restart: always
    environment:
      POSTGRES_USER: admin
      POSTGRES_PASSWORD: supersecretpassword
      POSTGRES_DB: db
    ports:
      - "5432:5432"
    volumes:
      - postgres_data:/var/lib/postgresql/data

  app:
    build: .
    container_name: nextjs-app
    ports:
      - "3000:3000"
    environment:
      DATABASE_URL: postgresql://admin:supersecretpassword@db:5432/db
    volumes:
      - .:/app
      - /app/node_modules
    depends_on:
      - db
    command: pnpm dev

volumes:
  postgres_data:

We're saving the database data in a named volume called postgres_data. This way, when we turn off the container, the data will still persist.

Tip: For resilience, combine depends_on with a healthcheck to ensure the DB is actually ready to receive traffic, not just "running".

One Command to Rule them All

With these three files in your root directory, you’ve turned a 1-hour headache setup into a 10-second command:

docker compose up

By running this, Docker will build your Next.js app, pull the database image, set up the network between them, and start your dev server with hot-reload enabled.

You can even add startup and seeding scripts if needed.

Stop Selling Hours, Start Delivering Systems

I stopped being just a programmer and became a Software Engineer. I reclaimed my time, improved my team's DX, and made sure that no stakeholder ever has to install a database on their laptop again.

If you want to scale your project or your career, start containerizing today. Your future self and your team will thank you.

What was your worst setup story? Share below in the comments!

Building a complex application or a high-performance system?

I help startups engineer scalable applications and internal tools that stay fast as they grow. See how I build scalable systems here: 👉 www.itsfranciscoluna.com

Junior vs. Lead: The 8 High-Stakes Mistakes I’m Leaving in 2025

Francisco Luna — Tue, 23 Dec 2025 20:38:32 +0000

In 2025, I learned that the distance between a junior Engineer and a lead is measured in the mistakes you stop making. It was a year of challenging product launches and pivots that changed my technical philosophy. I didn't just ship code; I paid the cost of admission in the form of technical debt and manual labor.

That's when I learned that efficiency is a business strategy, not just a technical preference. Here are the major mistakes I made during my professional career in 2025.

1. Poor Communication with Stakeholders

I once told a Product Manager:

"I’ve configured the NGINX reverse proxy to support 5MB image uploads."

Technically true, but useless to them.

As you grow, your job shifts from writing code to solving business problems. Stakeholders don't care about the how; they care about the result.

The Shift:

Junior: "I updated the NGINX config."

Lead: "Users can now upload high-res images. The marketing bottleneck is gone."

It's the same as when you play a guitar solo. Of course you're not going to tell people the fancy scale you're using. They only care about the feeling.

Stakeholders care about results and making the product better, not much about the fancy tech you use.

2. Infrastructure as Manual Labor

If your "deployment process" involves opening pgAdmin to manually add users, or if your documentation is a list of 20 manual steps to configure a server, you’ve fallen for the Sysadmin Trap.

It's the same when you have to set up the entire AWS architecture using raw dashboards.

You are acting as a manual gatekeeper instead of an architect.

Manual steps are invitations for human error. If you have to "remember" to set a config variable or manually assign a role to a new team member, you haven't built a system but a bottleneck.

The Shift:

Junior: Manages the database and servers through a GUI (pgAdmin/AWS Console) and manual docs.

Lead: Automates the environment. Use Migration Scripts to manage roles and Ansible/Terraform to define infrastructure.

The business shouldn't depend on your memory to set things up. It should depend on your code.

3. Configuration Overhead

Rebuilding a Docker image just to change a URL is a failure of architecture.

I’ve seen it happen: a stakeholder decides to change a domain or an API endpoint, and the engineering team has to trigger a full CI/CD pipeline, wait for the build, and redeploy everything for a single string change...

You should never have to rebuild your code to change how it behaves in its environment.

The Problem: Hardcoding Behavior

Whether it's a "feature flag," a timeout limit, or a third-party URL, if it's baked into your source code or your Dockerfile, your system is rigid. Rigid systems break under the pressure of fast-moving startups.

The Shift:

Junior: Hardcodes "minor" settings or bakes them into the Docker image during build time.

Lead: Uses Environment Variables and Secrets Managers (like AWS Secrets Manager or Parameter Store).

Ensure that the Docker image is immutable. It should be the exact same image in Staging as it is in Production. The only thing that changes is the environment it plugs into.

This doesn't just save time, it saves the business from deployment fatigue and unnecessary downtime.

4. Poor Testing Strategies

If your "Testing Strategy" is just you clicking around the UI before a deploy, you don't have a strategy, you have a prayer.

I once saw a production launch stumble because a user couldn't use a "dot" in their username. It sounds small until it’s the reason your conversion rate drops by 15% on launch day.

You need to set up testing from day one and let your CI/CD pipeline be the "bouncer" that keeps bad code out of production.

What Actually Matters (The 80/20 Rule):

You don’t need 100% code coverage. That’s a vanity metric. You need to protect the Critical Path:

Permissions & Validations: Can a user see data they shouldn't? Can they sign up with valid characters?
Critical Business Logic: Does the checkout work? Does the subscription upgrade trigger correctly?
Edge Cases & Race Conditions: What happens if two users click "buy" on the last item at the exact same millisecond?

The Shift:

Junior: Tests manually and hopes for the best. Thinks tests "slow them down."

Lead: Uses Unit Tests for logic, Integration Tests for the database/API flow, and E2E for the "happy path."

I know that tests aren't about finding bugs; they are about confidence. They allow the team to move fast without the fear of breaking the product every time they push code.

Set standards when writing tests and ensure your team is proactively working on them as well.

5. Poor Permission Management

I still remember these checks as a form of technical debt:

if (user.role === 'admin') {
  // Execute logic to ban user
}

This is fine for an MVP. But as the system grows, combining business logic with hardcoded roles becomes a nightmare.

The Problem: Role Inflation

Imagine we're building a social media platform. Initially, Admins ban users and regular Users view posts. Simple.

Then the business scales. You introduce a Moderator role. They need to manage reports and suspend accounts, but they shouldn't have full Admin access to the billing system. Suddenly, your code looks like this:

if (user.role === 'admin' || user.role === 'moderator') {
  // Execute logic to suspend account
}

Now imagine this check is scattered across 50 different files. If the CEO decides to add a "Junior Moderator" with limited powers, you have to tweak every single if statement. Congrats, now you're fighting with your own system.

The Shift: Ability-Based Access

A Lead Engineer stops thinking about who the user is and starts thinking about what they are allowed to do.

if (ability.can('suspend', 'User')) {
  // Execute logic
}

The Transformation:

Junior: Checks the "ID card" (Role) at every door.

Lead: Implements a "Keycard System" (Abilities/Permissions).

By using a library like CASL or a specialized Policy class, you decouple the business logic from the organizational hierarchy. When the business pivots or adds new roles, you change one configuration file, not the entire codebase.

And by the way, your tests will be simpler and auditing your permissions is a walk in the park now!

6. No Seeding Strategies

I used to spend 20–30 minutes manually clicking through the UI, creating fake accounts, and spamming forms just to get "real-world" data to work with.

I was doing the work of a script, but manually.

Seeding is the process of populating your database with a predictable set of data for development and testing. Without it, you aren't engineering; you're just messing around.

Why Seeding is a Business Strategy:

Handling Edge Cases: You can instantly recreate that one specific, weird bug that only happens to users with 50+ transactions and an expired subscription.
Stress Testing: You don't need to "spam" the system. One command generates 100,000 records to see if your PostgreSQL queries or Redis cache actually hold up.
Instant Onboarding: A new engineer (or a collaborator) should be able to clone the repo, run one command, and have a fully functional environment.
Testing the "Golden Path": Imagine you are debugging business logic for premium subscribers. Every time you want to test a change, you have to manually trigger a Stripe checkout, click through 5 different screens, and wait for webhooks. That's 10 minutes of manual labor per test. A seeding script handles this in 2 seconds. If you don't automate your state, you aren't testing; you're just doing expensive data entry.

The Shift:

Junior: Manually creates "test_user_123" every morning.

Lead: Automates the state of the world.

If you don't have a seeding strategy, you’re wasting the company’s most expensive resource: your time.

7. The "Hard Delete" Sin

In a startup, data is the only asset you can't recreate. I used to think that DELETE FROM was just part of the CRUD flow. I was wrong.

A hard delete is an irreversible loss of business knowledge. If a user deletes their account today, but the CEO wants to know why churn increased six months from now, that data is gone. The business no longer has access to the history.

The Shift:

Junior: Deletes records to "keep the database clean."

Lead: Implements Soft Deletes (deleted_at timestamps) and Audit Logs.

Yes, it's kind of tedious to only update a flag. However, it's critical for audit purposes and compliance.

8. Thinking a Company Was Going to Save Me

This was the hardest lesson of 2025. I spent too much time being reactive, waiting for a "safe" job or a promotion to change my life.

I realized that my security doesn't come from a contract; it comes from my reputation, my Network, and my skills.

The Shift:

Junior: Waits for instructions and a paycheck.

Lead: Acts as a business ally. I treat every project like I’m a partner, and I focus the most on my own knowledge and skills.

TL;DR — Engineering Maturity Checklist

Communicate outcomes, not configs: Stakeholders care about results, not technical steps.
Automate infrastructure (IaC): Use Terraform/Ansible; don’t be a manual gatekeeper.
Keep configs flexible: No hardcoding; rely on environment variables and secrets managers.
Test for confidence, not coverage: Protect the critical path with unit, integration, and E2E tests.
Design permissions with abilities: Move beyond role checks; implement ability‑based access control.
Seed your environments: Automate realistic data for onboarding, edge cases, and stress testing.
Preserve data with soft deletes: Use deleted_at flags and audit logs; never throw away business knowledge.
Own your growth: Security comes from skills, reputation, and network — not contracts.

About 2026 and the Future

If you’ve made it this far, thank you for reading! :)

I wish you a Merry Christmas and a productive New Year with your family; of course, if you celebrate it.

To start 2026 right, I’m currently diving into The Pragmatic Programmer. My goal is to become a more effective leader for my team and to ship features that are not just fast, but reliable.

I’m also returning to pen, paper, and whiteboards. I’ve realized that relying too much on AI and digital tools made me drift. There is no substitute for the clarity of drawing a system by hand before writing a single line of code.

Talk to other engineers outside your startup, rest, and plan your days. This is a marathon, not a race.

Here’s to a year of less entropy and more architecture.

What was your biggest lesson in moving from Junior to Lead? I'd love to hear from you.

Building a complex application or a high-performance system?

I help startups engineer scalable applications and internal tools that stay fast as they grow. See how I build scalable systems here: 👉 www.itsfranciscoluna.com

🚀 The Black Box Principle: Decoupling API Clients with OpenAPI and TypeScript

Francisco Luna — Sat, 01 Nov 2025 18:54:05 +0000

When I was primarily a frontend engineer, I had a terrible relationship with APIs. If I wasn't manually rewriting types and interfaces from the backend, I was debugging runtime errors because someone changed an endpoint without telling me.

Back then I read the following advice:

'The best way to get types from the server was to get the types from it.'

What I didn't realize was this was some of the worst advice I've ever received; until I became the backend engineer responsible for maintaining both sides.

The code you're about to see isn't a hypothetical example. It's the actual technical debt that almost burned me out as a solo engineer building a startup.

I didn't understand how the backend was supposed to work in a real world environment.

So this is what I came up with:

// Are you telling me I also need the backend to JUST get some schemas? 
import { 
  SignInWithEmailOrUsernameResponseSchema,
  SignInWithEmailOrUsernameSchema 
} from "@/backend/validations/auth";

// What is this? Your own flaky OpenAPI schema?
import { AuthAPI } from "../services/auth";

export function useSignIn() {
  return useMutation<
    z.infer<typeof SignInWithEmailOrUsernameResponseSchema>,
    AxiosError<ErrorResponse>,
    z.infer<typeof SignInWithEmailOrUsernameSchema>
  >({
    mutationFn: (user) => AuthAPI.signIn(user),
    onSuccess: (data) => {
      const { accessToken } = data;
      // Only God knows how this was processed...
    }
  });
}

The Tragedy

You might be wondering. Francisco, what is wrong with this, really? Everything, at least when you're working with both a separate backend and frontend.

❌ 1. Coupling

You're assuming your monolith repository will exist forever. But what happens if:

The backend moves to another repository?
You transition to microservices?
You need to support multiple client applications?
The backend team starts using languages other than TypeScript?

💀 2. Compilation and build nightmares

// Wait, why is my frontend bundle including this?
import { UserEntity } from "@/backend/entities/User";
import { PasswordHelper } from "@/backend/utils/crypto";
import { DatabaseConfig } from "@/backend/config/database";

Bundle Bloat: Your frontend is now shipping backend entities, utilities, and configuration files
Build Complexity: Your CI/CD now needs the entire backend codebase just to build the frontend
Dependency Hell: Backend-specific packages leaking into frontend builds
Docker Disasters: Trying to untangle this mess in containerization

⚠ 3. Hardcoding and context switching

This was the workflow with the former setup:

Add the new API endpoint

// This needs to be extended or changed manually 😭
export const AUTH_API_ENDPOINTS = {
  LOGIN: `/${API_ENDPOINT_NAMES.AUTH}/login`,
  REGISTER: `/${API_ENDPOINT_NAMES.AUTH}/register`
};

Create the new API service


// Yes, we also need backend dependencies and duplicate logic! 
const signIn = api<
  z.infer<typeof SignInRequest>,           // ← Manual type inference
  z.infer<typeof SignInResponse>           // ← More manual work
>({
  method: "POST",
  path: AUTH_API_ENDPOINTS.LOGIN,          // ← Hardcoded string
  requestSchema: SignInRequest,            // ← Duplicated validation
  responseSchema: SignInResponse,          // ← More duplication
  type: "public",
  withCredentials: true,
});

Create the queries and mutations

You already saw the code in the introduction, but see how duplicate things are and how prone this is to break, again.

export function useSignIn() {
  return useMutation<
    z.infer<typeof SignInWithEmailOrUsernameResponseSchema>,
    AxiosError<ErrorResponse>,
    z.infer<typeof SignInWithEmailOrUsernameSchema>
  >({
    mutationFn: (user) => AuthAPI.signIn(user),
    onSuccess: (data) => { 
      const { accessToken } = data;
      // Why are we guessing types? 
    }
  });
}

Now pray that something does not break and repeat the same exact process for 60+ API endpoints... right?

There's a BETTER way

I performed a refactor at work where we migrated all this Zod chaos, DTOs, schema validations, and backend spaghetti into a proper API contract.

Before I walk you through the recovery, let me introduce the tool that changed everything: OpenAPI.

OpenAPI

OpenAPI is a specification format used to describe and document REST APIs in a way that can be understood by human and used by machine. It’s not just documentation, but also a contract.

Here’s what it gives you out of the box:

🗺 1. Available endpoints:
Every route, method, and path is clearly defined.

🛠 2. Query and path params for each API endpoint:
You know exactly what each endpoint expects.

✅ 3. Response format for each status code:
You can validate what success, failure, and edge cases look like.

We'll see now how we can create a professional API specification on the server side.*

Generating OpenAPI Schemas

This process involves generating a .json or .yaml file containing your API contract.

Warning: This step varies dramatically based on:

The scale of your project (monolith vs microservices)
Your framework (Nest.js vs Spring Boot vs Go)
Your team's technical maturity (senior engineers vs junior team)
Your deployment complexity (do you need API versioning?)

The Nest.js Approach: Decorator-Driven

In Nest.js you use decorators to automatically generate your OpenAPI spec:

// task.controller.ts
@Controller('tasks')
export class TasksController {
  @ApiTags('Tasks')
  @Get()
  @ApiOkResponse({ 
    type: TaskEntity, 
    isArray: true,
    description: 'Returns all tasks for the authenticated user' 
  })
  @ApiUnauthorizedResponse({ description: 'Invalid or missing token' })
  findAll() {
    return this.tasksService.findAll();
  }
}

// task.entity.ts
export class TaskEntity {
  @ApiProperty({ 
    type: String,
    description: 'Unique task identifier',
    example: '550e8400-e29b-41d4-a716-446655440000'
  })
  id: string;

  @ApiProperty({
    type: String,
    description: 'Task title',
    example: 'Finish API documentation',
    maxLength: 255
  })
  title: string;

  @ApiProperty({
    type: Date,
    description: 'When the task was created',
    example: '2024-01-15T10:30:00.000Z'
  })
  createdAt: Date;
}

The Reality Check

If look at Reddit or technical blogs/forums, some developers prefer writing OpenAPI specs manually. The code-first approach has trade-offs:

✅ Pros:

Single source of truth (your code generates docs)
Harder for docs to drift from implementation
Faster iteration in early-stage startups

❌ Cons:

Can get messy with complex APIs
Limited control over exact OpenAPI features
Decorator pollution in your business logic

The Framework for Success

Mirror your domain entities and find a DRY way to manage them
Document controllers methodically. Descriptions and examples are important too
Display with SwaggerUI or Scalar for team collaboration
Generate client SDKs automatically as part of your CI/CD

Choosing Your API Visualizer

Honestly? Whatever. The important thing is that you have a visual interface for your API. Choose the one that's the most suitable for your team needs. Let's explore the common options with more detail:

Swagger UI - The battle-tested veteran

✅ Universally recognized, extensive customization
❌ Can feel dated, heavier bundle

Scalar - The modern contender

✅ Beautiful UI, faster, better DX
✅ Embracing by major frameworks (.NET's new default)
❌ Less ecosystem maturity

⭐ Bonus: Frontend Implementation

Now you might be wondering, how does this actually work on the frontend?

Well, it’s not difficult.

Let me introduce you to OpenAPI TypeScript

Imagine if your axios or fetch client had complete type safety and could interact with your API without manual guesswork. That’s exactly the problem OpenAPI TypeScript solves. All you need is the OpenAPI specification, which can be from your backend or a remote URL.

🚀 Getting Started with OpenAPI TypeScript

Install the tool

npm install -D openapi-typescript

Generate your types

From a local schema:

npx openapi-typescript ./path/to/my/schema.yaml -o ./path/to/my/schema.d.ts
# 🚀 ./path/to/my/schema.yaml -> ./path/to/my/schema.d.ts [7ms]

From a remote schema:

npx openapi-typescript https://myapi.dev/api/v1/openapi.yaml -o ./path/to/my/schema.d.ts
# 🚀 https://myapi.dev/api/v1/openapi.yaml -> ./path/to/my/schema.d.ts [250ms]

Now you can import types and contracts directly from your schema:

import type { paths, components } from "./my-openapi-3-schema";

// Schema object
type MyType = components["schemas"]["MyType"];

// Path parameters
type EndpointParams = paths["/my/endpoint"]["parameters"];

// Response objects
type SuccessResponse =
  paths["/my/endpoint"]["get"]["responses"][200]["content"]["application/json"]["schema"];

type ErrorResponse =
  paths["/my/endpoint"]["get"]["responses"][500]["content"]["application/json"]["schema"];

Wait, what about type-safe API clients?

Even with generated types, you’d still need to manually wire up your API calls!

That’s where OpenAPI Fetch comes in.

It gives you a fully type-safe client. No more guessing and no more duplicated logic.

Example: Type-Safe API Calls

import createClient from "openapi-fetch";
import type { paths } from "./my-openapi-3-schema"; // generated by openapi-typescript

const client = createClient<paths>({ baseUrl: "https://myapi.dev/v1/" });

const {
  data, // only present if 2XX response
  error, // only present if 4XX or 5XX response
} = await client.GET("/blogposts/{post_id}", {
  params: {
    path: { post_id: "123" },
  },
});

await client.PUT("/blogposts", {
  body: {
    title: "My New Post",
  },
});

This can also be implemented seamlessly with React Query, SWR or you favorite data fetching/mutation library.

Conclusion: The Black-Box

Please, always remember this:

Your frontend application must treat the backend as a black box.

It does not care about your database schema, crypto helpers, or specific implementation details. It only sees the API and the public interfaces.

By forcing your entire system to rely on a generated, language-agnostic OpenAPI Contract, you achieve true separation. You replace the questionable monorepo setup with an actual handshake.

This is the difference between a codebase that scales and a codebase that burns you out.

Stop debugging what you didn't write and start building what you control.

🚀 Join Escaping From the Abyss

Thank you so much for reading! :)

I’m launching a newsletter where I share hard-earned lessons from the trenches of startup engineering, especially in backend development and product planning.

I’ve seen engineers suffer from broken workflows and scope creep. I’ve even lived it. And I believe every dev deserves clarity, autonomy, and the tools to build well; both technically and emotionally.

This isn’t just about clean code. It’s about clean contracts, clean boundaries, and clean teams.

Let’s make startups a better place, through better engineering, better business practices, and better mentorship.

Join me here! Let’s escape from the abyss.

Lessons Learned as a 21 YO CTO

Francisco Luna — Sat, 18 Oct 2025 15:20:55 +0000

The truth is, the greatest architectural failure was not in code, but in my career path.

I carried the title of CTO (Chief Technology Officer) at 21. It sounds great, because you're able to select the technologies a company uses, you're an executive and you manage client relationships.

But internally, my CPU was at 100%. I was the solo founding engineer sustaining the product. I had to be a generalist in every single domain:

Frontend
Database Optimization
DevOps
Backend
Client PM

The context-switching, multi-tasking and lack of proper management experience or a technical team turned the position into a bottleneck. The problem was never the position. The problem were the conditions and responsibilities that came with it and I was not prepared for.

I'm still coding and delivering results, but I'm no longer the former CTO I used to be.

Let me share with you the 5 most important lessons I could've learned during this journey of almost 1 year.

1. Map your capabilities to your aspirations

Before you can build anything great, you must survey the land. I failed to do this.

I was too focused on the business results and outputs (features, deadlines, metrics, etc) that I was neglecting my own skills. I didn't know IaC (Infrastructure as Code), proper Networking or scalable Systems Design. The "CTO" role was nothing but a mask to hide those gaps from others, and more dangerously, for myself.

Your career is a system that needs maintainance. Audit it regularly. Be honest about the gap between your current skills and your future goals.

Think of it like climbing a mountain. The title is the summit.

Most people believe you have two choices:

Bare Hands: The brutal and manual grind. You fight for every inch of ground through sheer force and overtime.
The Helicopter: The smart, strategic path using the right tools and skills.

This is a lie.

The truth is, you use your bare hands to build the helicopter. I was so exhausted from the manual labor of firefighting and solo-coding that I never stopped to actually build the vehicle that could lift us to new heights. I was stuck in permanent manual mode, confusing hard work for smart progress.

Don't just climb with your hands. Build the damn helicopter.

2. Specialize to scale

Before we talk about specialization, let me ask you something more revealing than what programming languages you know: What are your hobbies?

Mine is music composition. I'm obsessed with it. In music, you don't just throw notes together. You use music theory, a structured system of rules and patterns. You make changes through careful voice leading, ensuring each note moves smoothly to the next. You work within constraints like scales and tempo to create something beautiful and coherent.

I realized I don't just love music; I love the architecture of music. (I'm a music theory nerd, too. 🤣)

And that's why I'm specializing in backend and cloud engineering. The backend is the music theory of software. It's the hidden structure that makes everything else possible. Cloud engineering is the orchestration, ensuring every service is in tune and in time.

This doesn't mean I can't do frontend. I can. Just like a composer can learn to paint, it's a different skill. But I'd rather be the one writing the symphony than designing the concert poster.

I want to build systems that scale and grow without me.

Specialization isn't about closing doors; it's about mastering the architecture you're truly passionate about building. It’s the difference between playing every instrument and becoming a virtuoso of the one that speaks to your soul.

For me, that’s the symphony of systems.

3. Growth over titles

Let's be honest. You're a 21-year-old CTO at a startup. It sounds impressive.

Now imagine you have to sit down and negotiate with other CTOs. They're discussing microservices, Kubernetes, CEO negotiations, hiring strategies, scaling to millions, and complex systems design on AWS.

And it all sounds like buzzwords to you.

This exact scenario is why I quit.

Not because I was scared, but because I was strategic. I saw two paths:

The Ego Path: Stay in the room, fake it, and slowly be exposed as inexperienced while my actual technical skills rust.
The Growth Path: Leave the room, admit what I didn't know, and go build the skills to earn my way back in authentically.

When you're starting out, you must prioritize your growth, not your title. It's great to learn from other CTOs, but not at the expense of your technical foundation.

You're better off monetizing a SaaS, being a freelancer, or excelling as an engineer at a company where you can learn new tech.

Specialize. Grow. Learn.

Don't try to be a pro F1 driver when you're still learning to drive a kart. Get proper training first.

4. Find a team and mentors

I'm done being the solo engineer.

I'm done guessing. Done sending pull requests into a void, with no one to question my decisions or suggest a better way. Done being unaware of the teamwork and best practices that transform good code into great systems. This isolation probably even stunted my Git skills. When you only answer to yourself, you never learn the true power of collaboration.

Don't be the only technical person in the room unless it's your own venture. Even then, it's a temporary state, not a sustainable strategy. Solitude is a career liability that only limits your knowledge.

That's why I'm joining a team at work. To immerse myself in code reviews, architectural debates, and shared ownership. I'm also studying Web Scalability for Startup Engineers. This is the book I wish I'd read before writing a single line of professional code.

It's the book that would have taught me that scaling isn't a feature you add later; it's a property you design for from the beginning.

It's never too late to stop being a one-man band and learn how to play in an orchestra.

5. Burnout and mental health

Protect your mental health, relationships, and happiness at all costs. No equity or title can buy your freedom.

I'm focused on growing and learning now, not managing. Rest. Accept that it's okay to not know everything.

The moment your heart stops choosing something, even though your mind has rational excuses, it's no longer for you.

Whether it causes trauma, breaks you, or holds you back; let it go. Your most important system is the life you build.

Conclusion

A lot of people say I need therapy.

They misunderstand. My choice to leave wasn't a cry for help, but a declaration of independence.

Look:

Therapy doesn't build resilient systems at 3 AM.
Therapy doesn't master cloud infrastructure or elegant code.
Therapy doesn't deliver the profound satisfaction of creation.

I found my therapy in building. In the logic of systems, the flow of music, and the confidence that comes from genuine mastery.

They see a former CTO. I see a builder who finally learned that the ultimate scale is a life of freedom, surrounded by family and friends, doing work that matters; on my own terms.

The title was a cage. The freedom is the system I'm building now.

(In case you're wondering, I didn't quit. I'm a Backend and DevOps Engineer at the company now.)

So, what do you actually want?

Not what your resume wants. Not what your ego wants. Not what other people think you should want.

I know what I want. I want to build apps. I want to be with my people. I want to deepen my backend and cloud engineering skills. I want to contribute my knowledge, mentor others, educate myself and own my time.

I want that more than a place on a cap table. More than chasing investors. More than a fancy title that costs me my freedom.

The title was a lesson. The building is the reward.

Now it's your turn. What do you actually want? Share your "Path" in the comments.

Building a complex application or a high-performance system?

I help startups engineer scalable applications and internal tools that stay fast as they grow. See how I build scalable systems here: 👉 www.itsfranciscoluna.com

10 Database Design Lessons Learned Working for a Startup

Francisco Luna — Sun, 31 Aug 2025 14:48:45 +0000

Hi! The abyss has been quite intense these weeks, but we're back with a new entry. I remember when I was working in the database design of my first position a year and a half ago. The amount of things I was blundering: bad practices, poor naming conventions... I guess that's how we all start and learn at the beginning.

I've been working on database design at a startup, and this time, it feels like the system is no longer against me. It's all about best practices, scalability, and clear conventions. But this didn't happen suddenly; it took countless mistakes, hours, and pain.

I've gone through the process of learning the best practices on my own and ensuring my schemas can actually scale, so you don't have to go through the same painful process. Here are the 10 database design lessons I learned the hard way.

1. Avoid overusing enums

An enum (enumeration) is a data type that lets you define a list of possible string values a column can hold. The database then enforces that the data inserted into that column must be one of those values. They are great for small, unchanging sets of values but they're a pain to update because it requires a schema migration.

Lookup tables are separate tables that contain a list of values, and your main table references them using a foreign key. This makes the values easy to manage, add, or remove without changing the main table's schema.

Simple Enum in PostgreSQL:

CREATE TYPE user_status_enum AS ENUM ('active', 'inactive', 'suspended');
CREATE TABLE users (
  id SERIAL PRIMARY KEY,
  username VARCHAR(50) NOT NULL,
  status user_status_enum NOT NULL
);

Scalable Lookup Table Schema:

CREATE TABLE post_categories (
  id SERIAL PRIMARY KEY,
  name VARCHAR(50) UNIQUE NOT NULL
);
CREATE TABLE posts (
  id SERIAL PRIMARY KEY,
  title VARCHAR(255) NOT NULL,
  category_id INTEGER REFERENCES post_categories(id)
);

Backend Dictionaries

These are data structures in your application code that hold a list of valid values for a specific field.

// This data structure is dynamically populated from the database
interface CategoryDictionary {
  [categoryName: string]: number;
}
const CATEGORIES: CategoryDictionary = {
  'Tech': 1,
  'Music': 2,
  'Travel': 3
};
// Example usage
const categoryId = CATEGORIES['Tech']; // Returns 1
const isValidCategory = !!CATEGORIES[categoryName]; // Checks if a category exists

While they can be a useful tool for validation, simply using a static array of strings in the database isn't a robust solution on its own. It directly violates the principles of database normalization by creating a redundancy between your application and your database schema.

If the list of valid values changes, you have to update your code and potentially your database, leading to a disconnect between the two sources of truth. A more normalized approach is to use these dictionaries alongside a lookup table. The dictionary in your backend can be populated from the lookup table in your database on application startup. This way, the database remains the single source of truth for the list of values, and your application always has the most up-to-date data for validation.

2. Design for the Business, Not for Convenience

This principle is about creating a database schema that accurately models the real-world business concepts, rather than what's easiest for your code. A DATE data type, for instance, represents a full calendar date. While a developer might find it convenient to store a birthday as three separate integers for year, month, and day, the business understands a birthday as a single date. Using the correct data type ensures your database reflects the truth of the business.

Bad Design:

CREATE TABLE user_birthdays (
  user_id SERIAL PRIMARY KEY,
  birth_month INTEGER,
  birth_day INTEGER,
  birth_year INTEGER
);

Good Design:

CREATE TABLE users (
  id SERIAL PRIMARY KEY,
  name VARCHAR(100),
  birthday DATE
);

3. Consistency Matters

Consistency means using the same naming conventions, data types, and structural patterns throughout your entire database. For example, if you choose snake_case (e.g., user_id), you should use it everywhere. Inconsistent naming like user_id in one table and userId in another leads to confusion and bugs.

Inconsistent Naming (Bad):

CREATE TABLE users (
  user_id SERIAL PRIMARY KEY
);
CREATE TABLE user_posts (
  post_id SERIAL PRIMARY KEY,
  userId INTEGER
);

Consistent Naming (Good):

CREATE TABLE users (
  id SERIAL PRIMARY KEY
);
CREATE TABLE user_posts (
  id SERIAL PRIMARY KEY,
  user_id INTEGER REFERENCES users(id)
);

4. Everything Should Serve a Purpose

Every table, column, and index in your database should exist for a specific, current business requirement. Over-engineering is the act of building features or adding complexity that isn't currently needed, often with the thought of "just in case." This adds unnecessary complexity and can harm performance. The goal is to keep the database as simple as possible while meeting all requirements.

Over-engineered (Bad):

CREATE TABLE products (
  id SERIAL PRIMARY KEY,
  shipping_notes TEXT,
  supplier_id INTEGER,
  is_on_sale BOOLEAN,
  sale_price DECIMAL(10, 2)
);

If your business only has one supplier and products are never on sale, these columns are just clutter. Don't try to cover all possible future cases when a feature hasn't even been requested yet.

5. Don't Let the Database Handle Authentication

Your database is a data store, not a security server. Don't rely on features like PostgreSQL's Row-Level Security (RLS) for core authentication, please. It sounds easy to do, but youre introducing provider-lock in. What if you change the database youre working with in the future? Thats why all authentication and password hashing should be handled in your application layer.

Example RLS Policy (to show what to avoid):

-- Now your app's security is coupled to the DB...
CREATE POLICY user_access ON user_posts
  FOR SELECT
  USING (user_id = current_setting('app.user_id')::integer);

6. The Database Shouldn't Handle Business Logic Validations

Validation is the process of ensuring data is correct and meaningful. The database should only perform basic validations like ensuring a field is not null or that a value is unique. Complex business logic, like checking if a number is between 1 and 12 for a month, should be handled by your application code. This separation of concerns keeps your database clean and your application flexible.

Application-level validation (Better):

// In your backend code
if (month < 1 || month > 12) {
  // Error handling
}

7. Separate Staging and Production Databases

Production is the public-facing environment. Staging is a pre-production environment used to test new features. You should never work directly on your production database. Mistakes are inevitable. Having separate databases creates a safety net, so you don't accidentally delete real customer data. I recommend you using a product like Hashicorp Vault to properly save the credentials and URLs to work with these environments.

8. Never Take Scalability for Granted

Scalability is the ability of a system to handle a growing amount of work. Its a mindset you must adopt from the beginning. A key part of this is understanding the impact of your design choices. An index is a data structure that improves the speed of data retrieval operations on a database table. Forgetting to add them to foreign keys is a common mistake that can lead to major performance issues as your data grows.

Adding an Index

CREATE TABLE user_posts (
  id SERIAL PRIMARY KEY,
  user_id INTEGER REFERENCES users(id)
);
CREATE INDEX idx_user_posts_user_id ON user_posts (user_id);

Think About the Long Term

Always build your tables with the expectation that there will be changes and new features.

Spoiler: There will always be changes.

This means you should design for the long term. A critical part of this is adding a new column. To avoid database conflicts during a migration, a new column must be nullable or have a default value. This prevents live applications from failing when they try to insert new data. While a nullable column is a common and necessary workaround, using a default value is often a better practice from a data integrity standpoint, as it avoids nulls and maintains a cleaner dataset.

Document Your Schemas

In a startup, your data structures will evolve quickly. When using flexible data types like JSON or JSONB to store semi-structured data, it's easy to lose track of the schema. That's why you should document your JSON/JSONB schemas and structures through a formal specification like OpenAPI. This ensures that your entire team and any future consumers of your API understand the data format, preventing errors and ensuring consistency.

9. Use the Right Data Structures

Choosing the correct data type (e.g., BOOLEAN, DATE, JSONB) for a column is important for performance and clarity. JSONB is a PostgreSQL data type that stores JSON data in a binary format which allows you to index and query it efficiently. Its perfect for semi-structured data where the schema might be flexible.

Example with JSONB:

CREATE TABLE social_media_posts (
  id UUID PRIMARY KEY,
  metadata JSONB
);
INSERT INTO social_media_posts (id, metadata)
VALUES (
  'a0eebc99-9c0b-4ef8-bb6d-6bb9bd380a11',
  '{"tags": ["newmusic"], "likes": 150}'
);

10. Always Choose UUIDs Over `auto_increment` IDs

An auto_increment ID is a simple integer that automatically increases with each new row. A UUID (Universally Unique Identifier) is a 128-bit number that is globally unique. UUIDs are essential for distributed systems because they can be generated anywhere, on a client, in a microservice; without a central authority to ensure uniqueness. This prevents ID conflicts and makes your system much more scalable and secure.

Why UUIDs are better:

No Conflicts in Distributed Systems: With auto_increment, two different services inserting data at the same time might generate the same ID. UUIDs solve this problem.
Security & Data Leaks: Predictable, sequential IDs like 1, 2, 3... are a major security risk. A malicious user can simply try incrementing IDs in an API endpoint (e.g., /api/users/1, then /api/users/2) to scrape or access other users' data. With UUIDs, the IDs are random and impossible to guess, protecting against these types of attacks.

Example in PostgreSQL:

CREATE TABLE users (
  id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  email VARCHAR(100) UNIQUE NOT NULL
);
-- The UUID is generated automatically
INSERT INTO users (email) VALUES ('janedoe@example.com');

Conclusion

Database design is a brutal teacher in software engineering. The most painful lessons aren't in a tutorial when you're developing; they're in production, where a small mistake can become a colossal monster. Don't be discouraged by your blunders or failures as they're the very thing that forges your mastery.

Think of it like Dark Souls. You dont get good by avoiding the bosses; you get good by facing them, learning their patterns, and finally beating them. The ten lessons in this post are your strategies. Now you're ready to face your own production monsters. I know I'll be facing more challenges, and I'm willing to keep sharing with you everything I learn on this journey.

See you in the next entry. Dont let the abyss consume you.

Uploading Images to AWS S3 with Next.js and React Dropzone — A Complete Guide

Francisco Luna — Fri, 27 Sep 2024 01:41:16 +0000

I’ve been learning how to set up AWS S3 with Next.js at work for a few weeks. At first, it felt a bit overwhelming with different approaches and configurations to consider. After a while, this task got easier and I integrated S3 to my tech stack.

Adding S3 can boost your project's functionality and performance, especially when you need to work with images and files in your application. That's why In this guide, I'll take you step-by-step through setting up S3 with Next.js. We'll create a small application where users will be able to add images from their devices through a drop zone component, to be uploaded to S3.

What is AWS S3?

Amazon S3 or Simple Storage Service is one of the core services of Amazon Web Services. It gives you a highly scalable, reliable and secure object storage service on the cloud.

S3 offers developers and businesses a highly reliable and cost-effective solution for storing and retrieving unlimited amounts of data, accessible anytime and from anywhere on the web.

And here's an interesting tidbit: the platform you're reading this on, Dev.to, uses Amazon S3 to store all the blog images! It's a perfect real-world example of how S3 is seamlessly integrated into the digital experiences we use every day.

Requirements

Before diving into this guide, it’s essential to have a foundational understanding of the following technologies:

1. Next.js: Familiarize yourself with the basics of Next.js, a React framework that enables server-side rendering and static site generation.

2. React: Ensure you have a good grasp of React, as Next.js is built on top of it.

3. AWS IAM User: Set up an AWS Identity and Access Management (IAM) user with the necessary permissions to access and manage your S3 bucket.

4. S3 Bucket: Create and configure an Amazon S3 bucket to store your files. Make sure to note down the access key and secret key for your IAM user, and configure the bucket’s permissions and CORS settings as needed. Save the bucket name as well.

What you'll be building

In this tutorial you'll be building a simple application that allows users to upload images to S3 thanks to a dropzone component. The images will be retrieved programmatically from a S3 bucket and the users can also delete each image.

This will teach you:

How to use Next.js Server Actions: Learn how to use serverless functions to leverage Next.js latest features and best practices to do form submissions seamlessly.
How to use the AWS S3 SDK for Node.js: You'll learn how to upload images, retrieve them and how to delete images programmatically.
Integrate React Dropzone into your application: You're going to add React Dropzone with validations, type safety, hooks and constants to use best practices and ensure a nice user experience.

Installing and Configuring Next.js

To install Next.js, you can use the following command: npx create-next-app@latest.

Learn more on the official docs.

Setting up dependencies

Once you have installed Next.js, you'll need to install React Drop Zone and the AWS S3 SDK for Node.js.

To install these dependencies, you can use the following command: npm install react-dropzone @aws-sdk/client-s3

Learn more about the AWS S3 SDK here

Learn more about React Drop zone here

Creating the Server Actions

Before building the user interface focus on creating the business logic through server actions. They're the same as the lambda functions from AWS and you can use them for form submissions.

For this guide, you'll need to create 2 server actions; One to upload an image to S3 and another one to delete an image.

Create a new folder called actions inside your src folder and start creating your first server action there:

Server Action to Upload an Image to a Bucket

This server action is used to upload images to an S3 bucket. It takes two arguments: formData and payload.

formData: Contains the image data sent from the frontend.
payload: An object with two fields, bucket and key.

Payload Fields

bucket: The name of your S3 bucket.
key: A unique identifier for each image in the bucket.

Every image in S3 needs a unique key to ensure that each file is stored separately.

// /src/actions/s3.ts
type UploadImageToS3Payload = {
  bucket: string;
  key: string;
};

export async function uploadImageToS3(
  formData: FormData,
  payload: UploadImageToS3Payload
): Promise<any[]> {
  const s3 = new S3Client({
    region: process.env.AWS_REGION,
    credentials: {
      accessKeyId: process.env.AWS_ACCESS_KEY as string,
      secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY as string,
    },
  });

  const { bucket, key } = payload;

  try {
    const files = formData.getAll("file") as File[];

    const response = await Promise.all(
      files.map(async (file) => {

        const arrayBuffer = await file.arrayBuffer();
        const buffer = Buffer.from(arrayBuffer);

        const fileUploadParams = {
          Bucket: bucket,
          Key: key,
          Body: buffer,
          ContentType: file.type,
        };

        const imageParam = new PutObjectCommand(fileUploadParams);
        await s3.send(imageParam);
      })
    );

    revalidatePath("/");
    return response;

  } catch (error) {
    console.error("Error uploading image to S3:", error);
    throw new Error("Failed to upload image to S3.");
  }
}

What's Going On Here?

Let's break down what's happening in the code step by step.

1. Initializing the S3 client

First, we create an instance of the S3 client using the following code. Remember to use environment variables to keep credentials safe as well:

const s3 = new S3Client({
  region: process.env.AWS_REGION,
  credentials: {
    accessKeyId: process.env.AWS_ACCESS_KEY as string,
    secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY as string,
  },
});

Here you're passing the AWS region and the necessary credentials (access key and secret access key) to the S3Client class. These credentials are typically stored in environment variables for security reasons. The S3Client is the object that allows you to interact with Amazon S3. Giving you the possibility to upload files.

2. Extracting Payload and Files Information

Next, you'll need extract the bucket and key from the payload, which are essential for specifying where and how each file will be stored in S3:

const { bucket, key } = payload;

Then, we gather all the files from the formData object:

const files = formData.getAll("file") as File[];

The formData.getAll("file") method retrieves all files associated with the "file" field from the form data submitted by the frontend. This array of files will be processed and uploaded to S3.

3. Processing and Uploading Each File

For each file, we perform the following steps:

const response = await Promise.all(
  files.map(async (file) => {
    const arrayBuffer = await file.arrayBuffer();
    const buffer = Buffer.from(arrayBuffer);

    const fileUploadParams = {
      Bucket: bucket,
      Key: key,
      Body: buffer,
      ContentType: file.type,
    };

    const imageParam = new PutObjectCommand(fileUploadParams);
    await s3.send(imageParam);
  })
);

Convert File to Buffer:

We start by converting the file into an ArrayBuffer, which represents the file's binary data. Then, we create a Buffer from this ArrayBuffer. This Buffer is what we'll actually send to S3 as the file's content.

Prepare Upload Parameters:

We define fileUploadParams, an object that specifies the details of the upload, including the S3 bucket name (Bucket), the unique key (Key), the file content (Body), and the file type (ContentType).

Upload the File:

The PutObjectCommand is used to create a command for uploading the file to S3.

Finally, we execute this command using s3.send(imageParam), which uploads the file to the specified bucket with the given key.

By using Promise.all, we ensure that all files are processed and uploaded in parallel, improving the efficiency of the operation.

4. Revalidating the Path

Once the files are successfully uploaded, we call revalidatePath("/") to refresh the content or invalidate the cache for the root path:

revalidatePath("/");

This is useful if your application needs to update the displayed content or reflect the changes made by the file uploads.

5. Error Handling

If anything goes wrong during the upload process, we catch the error, log it to the console, and throw a new error to ensure the issue is properly handled:

catch (error) {
  console.error("Error uploading image to S3:", error);
  throw new Error("Failed to upload image to S3.");
}
This helps in diagnosing issues and ensures that the application can gracefully handle failures.

Server Action to Delete an Image from S3

This server action deleteImageFromS3, is designed to delete an image stored in an Amazon S3 bucket. It accepts an object containing the Bucket name and the Key of the image to be deleted.

type S3ParamsPayload = {
  Key?: string;
  Bucket?: string;
};

export async function deleteImageFromS3({
  Bucket,
  Key,
}: S3ParamsPayload): Promise<DeleteObjectCommandOutput> {
  const s3 = new S3Client({
    region: process.env.AWS_REGION,
    credentials: {
      accessKeyId: process.env.AWS_ACCESS_KEY!,
      secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
    },
  });

  const command = new DeleteObjectCommand({ Bucket, Key });

  console.log(command);

  try {
    const res = await s3.send(command);
    revalidatePath("/");

    console.log(res);
    return res; 

  } catch (error) {
    console.error("Error deleting image from S3:", error);
    throw new Error("Failed to delete image from S3.");
  }
}

What's going on here?

Let's break down what's happening in this server action:

1. Type Definition

First, we define the S3ParamsPayload type:

type S3ParamsPayload = {
  Key?: string;
  Bucket?: string;
};

Where key is the unique identifier of the image (name or path) of the image within the S3 bucket and Bucket refers to the name of the S3 bucket where the image is stored.

2. Initializing the S3 client

We create again the S3 client which is necessary to interact with AWS S3.

// Provide the respective region and credentials from your AWS account 
const s3 = new S3Client({
  region: process.env.AWS_REGION,
  credentials: {
    accessKeyId: process.env.AWS_ACCESS_KEY!,
    secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
  },
});

3. Creating the Delete Command

The DeleteObjectCommand is an AWS SDK command that specifies the exact file to delete from the specified bucket.

const command = new DeleteObjectCommand({ Bucket, Key });

4. Executing the Delete Command

This line sends the DeleteObjectCommand to AWS S3, which processes the request and deletes the specified image from the bucket.

const res = await s3.send(command);

5. Revalidating the Path*
Invalidate the cache for the root path of the application after deleting an image so the changes are reflected in real time.

revalidatePath("/");

6. Error Handling
Add basic error handling to manage any issues that might arise during the deletion process.

catch (error) {
  console.error("Error deleting image from S3:", error);
  throw new Error("Failed to delete image from S3.");
}

Retrieving the Images

Let's create a new folder inside src called services. This folder will be used to perform async operations with external services such as AWS S3. Create a new file called s3.ts and add the following function to get the signed URLs from a bucket. This is the way we can get images using AWS S3 in a secure way.

Signed URLs are temporary, secure URLs that allow access to objects in an S3 bucket for a limited period of time. By signing the URLs, you control who can access your S3 objects and for how long, without making the bucket or objects publicly accessible.

// src/services/s3.ts
export async function getAllObjectsSignedUrls({
  Bucket,
}: S3ParamsPayload): Promise<{ key: string; url: string }[]> {
  const s3 = new S3Client({
    region: process.env.AWS_REGION,
    credentials: {
      accessKeyId: process.env.AWS_ACCESS_KEY!,
      secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
    },
  });

  try {
    // List all objects in the bucket
    const listCommand = new ListObjectsV2Command({ Bucket });
    const listObjectsOutput: ListObjectsV2CommandOutput = await s3.send(
      listCommand
    );

    const signedUrls = await Promise.all(
      (listObjectsOutput.Contents || []).map(async (object) => {
        if (object.Key) {
          const getObjectCommand = new GetObjectCommand({ Bucket, Key: object.Key });
          const url = await getSignedUrl(s3, getObjectCommand, { expiresIn: 3600 });
          return { key: object.Key, url };
        }
        return null;
      })
    );

    // Filter out any null results (in case an object didn't have a Key for some reason)
    return signedUrls.filter((item): item is { key: string; url: string } => item !== null);
  } catch (error) {
    console.error("Error retrieving objects from S3:", error);
    throw new Error("Failed to retrieve objects from S3.");
  }
}

What's going on here?

1. Initialize the S3 client

As with the server actions, you need to initialize the S3 client once again:

// Provide the respective region and credentials from your AWS account
const s3 = new S3Client({
  region: process.env.AWS_REGION,
  credentials: {
    accessKeyId: process.env.AWS_ACCESS_KEY!,
    secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
  },
});

2. List Objects in the S3 Bucket

We use the ListObjectsV2Command to retrieve all objects in the specified bucket:

const listCommand = new ListObjectsV2Command({ Bucket });
const listObjectsOutput: ListObjectsV2CommandOutput = await s3.send(listCommand);

This command will list all the objects stored in the S3 bucket, returning metadata about each object, including the key and last modified date.

3. Generate Signed URLs

Once we have the list of objects, the next step is to generate signed URLs for each object:

const signedUrls = await Promise.all(
  (listObjectsOutput.Contents || []).map(async (object) => {
    if (object.Key) {
      const getObjectCommand = new GetObjectCommand({ Bucket, Key: object.Key });
      const url = await getSignedUrl(s3, getObjectCommand, { expiresIn: 3600 });
      return { key: object.Key, url };
    }
    return null;
  })
);

We've generated temporary urls which grant secure access to the objects stored in S3. In this case, we use the getSignedUrl function to generate URLs that expire after 3600 seconds (1 hour).

We also use Promise.all to handle all the asynchronous URL generation tasks concurrently which improves performance by reducing the total time needed to process all the objects.

4. Filter Out Null Values

In some cases, an object might not have a key, or the operation might fail for some objects. To ensure that the final result only contains valid entries, we filter out any null values:

return signedUrls.filter((item): item is { key: string; url: string } => item !== null);

5. Error Handling

As with any operation that involves external services, there’s a chance something could go wrong. To handle potential issues gracefully, the function includes error handling:

catch (error) {
  console.error("Error retrieving objects from S3:", error);
  throw new Error("Failed to retrieve objects from S3.");
}

Creating the Drop zone Component

The Dropzone component allows users to drag and drop images for upload. We set it up to handle a maximum of 3 images, each up to 2MB in size. The react-dropzone library is used to create the drag-and-drop area and manage the upload process.

Here's the code for the Dropzone component:

"use client";

import { FileError, FileRejection, useDropzone } from "react-dropzone";
import React, { useState } from "react";
import { uploadImageToS3 } from "@/app/actions/s3";

const MAX_FILE_SIZE_MB = 2;
const MAX_FILE_SIZE_BYTES = MAX_FILE_SIZE_MB * 1024 * 1024;
const MAX_IMAGE_COUNT = 3;

const Dropzone = () => {
  const [uploading, setUploading] = useState<boolean>(false);

  const typeValidator = (file: File): FileError | null => {
    if (file.size > MAX_FILE_SIZE_BYTES) {
      return {
        code: "size-too-large",
        message: `Image file is larger than ${MAX_FILE_SIZE_MB}MB.`,
      };
    }
    return null;
  };

  const onDrop = async (
    acceptedFiles: File[],
    rejectedFiles: FileRejection[]
  ) => {
    if (rejectedFiles.length > 0) {
      alert(
        `You're trying to upload a file larger than ${MAX_FILE_SIZE_MB}MB. Please try again.`
      );
      return;
    }

    setUploading(true);

    try {
      const formData = new FormData();
      acceptedFiles.forEach((file) => formData.append("file", file));

      await uploadImageToS3(formData, {
        bucket: process.env.NEXT_PUBLIC_BUCKET!, 
        key: `${Date.now()}`, 
      });

      alert("Files uploaded successfully!");

    } catch (error) {
      console.error("Error uploading image to S3:", error);
      alert("Failed to upload image to S3. Please try again.");
    } finally {
      setUploading(false);
    }
  };

  const { getRootProps, getInputProps, isDragActive } = useDropzone({
    onDrop,
    validator: typeValidator,
    accept: {
      "image/jpeg": [],
      "image/png": [],
      "image/webp": [],
      "image/jpg": [],
    },
    maxSize: MAX_FILE_SIZE_BYTES,
    maxFiles: MAX_IMAGE_COUNT,
  });

  return (
    <div>
      <div
        {...getRootProps()}
        className="border-2 border-dashed border-slate-200 rounded-lg hover:bg-slate-100/50 cursor-pointer duration-200 p-8 text-center"
      >
        <input {...getInputProps()} />
        {isDragActive ? (
          <p>Drop the files here ...</p>
        ) : (
          <p className="text-slate-400">
            {`Drag and drop some files here, or click to select files (up to ${MAX_IMAGE_COUNT} images, max ${MAX_FILE_SIZE_MB}MB each)`}
          </p>
        )}
      </div>

      {uploading && <p>Uploading...</p>}
    </div>
  );
};

export default Dropzone;

Explanation:

Validation: We validate file types and sizes. If a file is too large, an error message is displayed.

Handling Uploads: Files are uploaded using uploadImageToS3, a function that sends files to the S3 bucket via a server action we set up earlier.

Feedback: We provide feedback to the user, like showing an "Uploading..." message while files are being uploaded.

The Image Container Component

The ImageContainer component displays each uploaded image along with a "Delete" button. This button allows users to delete images from the S3 bucket.

Here's the code:

export function ImageContainer({ keyProp, url }: Props) {
  return (
    <div key={keyProp} className="relative h-48 w-full">
      <Image
        src={url}
        alt={keyProp}
        layout="fill"
        objectFit="cover"
        className="rounded-lg relative"
      />
      <button
        onClick={() =>
          deleteImageFromS3({
            Bucket: process.env.NEXT_PUBLIC_BUCKET!,
            Key: keyProp

          })
        }
        className="absolute bg-red-500 rounded-full left-2 top-2 p-1 px-4 text-sm text-white font-medium hover:bg-red-600 duration-200"
      >
        Delete
      </button>
    </div>
  );
}

Explanation:

Image Display: Each image is displayed using the next/image component, which optimizes image loading in Next.js.

Delete Functionality: The "Delete" button triggers deleteImageFromS3, a function that removes the image from the S3 bucket.

The Final Page: Page.tsx

Finally, the Page.tsx file ties everything together. It displays the Dropzone component and the images that have been uploaded.

Here's how it looks:

import Dropzone from "../components/Dropzone";
import { getAllObjectsSignedUrls } from "@/services/s3";
import { ImageContainer } from "../components/ImageContainer";

export default async function Home() {  
  const imageUrls = await getAllObjectsSignedUrls({
    Bucket: process.env.NEXT_PUBLIC_BUCKET!,
  });

  return (
    <main className="max-w-[800px] m-auto p-8">
      <Dropzone />
      <div className="grid grid-cols-3 gap-4 mt-8">
        {imageUrls.map((image) => (
          <ImageContainer key={image.key} keyProp={image.key} url={image.url} />
        ))}
      </div>
    </main>
  );
}

Explanation:

Dropzone Component: This is where users upload images.

Displaying Images: After uploading, images are fetched from S3 and displayed using the ImageContainer component.

Fetching Images: We use getAllObjectsSignedUrls to get the URLs of all images stored in the S3 bucket. This function was defined in the services file earlier.

How to Improve the Code

As this blog post was created for educational purposes and I was relatively new to AWS at the time it was written, in a real-world application, you should implement the following improvements:

Singleton Client for the S3 class: Instead of instantiating a new S3 client for each operation, you should create a singleton instance of the S3 class. This avoids unnecessary overhead and improves performance.

More robust Typescript types: Enhancing TypeScript types will help prevent potential errors and provide a better developer experience. You can define types for common operations like file upload and metadata handling, ensuring better code quality and maintainability.

Security Enhancements

IAM Roles & Policies: Ensure that the S3 bucket's permissions are correctly set up using IAM roles, allowing only authorized users and services to interact with it.

Server-Side Encryption: Consider enabling server-side encryption to ensure your files are encrypted at rest.

Presigned URLs: If your app allows file uploads or downloads from the client, use presigned URLs to provide secure temporary access.

Image Compression: Consider using a library like Sharp to compress the users' images and save storage and resources.

Conclusion

Uploading images to AWS S3 with Next.js and React Dropzone offers an efficient and scalable solution for handling media uploads in web applications. This guide covered the basic steps to integrate S3 into your project, including file uploads, error handling, and using Next.js Server Actions.

By implementing the recommended improvements, such as creating a singleton S3 client, leveraging stronger TypeScript types, and enhancing security, you can ensure your application is both performant and secure in production environments.

Thanks for reading, and I wish you the best in your AWS and development journey!😊

Getting started with Tables using Next.js, Tanstack Table and Typescript

Francisco Luna — Fri, 17 May 2024 01:58:11 +0000

Have you ever wondered how some admin panels handle their users in a efficient and comfortable way? Well, tables have come to the rescue!

They're an efficient way to display a huge amount of information with a proper user experience and design. However, before you'd need to work with libraries and dependencies such as Material UI table; and this is a good library, the problem is that it's about 300kb! You're delivering all that Javascript to your client which is not efficient.

Times have changed and now we have alternatives such as Tanstack Table and Shadcn/UI. Amazing UI libraries that allow us to build intuitive and professional tables easily with Typescript support.

That's why in this guide you'll be learning how to build tables using Next.js, a modern fullstack framework based on React, Shadcn/UI, an amazing and modern UI library for React and JSONPlaceholder as our mockup data.

Let's get started!

What We'll be Building

We're going to build a table to display users using Shadcn/UI, Next.js, Tanstack Table and JSONPlaceholder. You'll be able to work with real world data and tables after reading this guide with the mentioned technologies to deliver higher quality products and software.

Project Requirements

To be able to understand and get the most out of this tutorial you need to have Node.js 18.17 or later installed on your computer, a solid understanding of React and Next.js and how to use third party libraries.

Installing Next.js and Shadcn/UI

We're going to install Next.js using the following command from the official website documentation:

npx create-next-app@latest

Use a name of your preference to name your project and since we're going to use Typescript and the App Router, we'll have the following setup:

Would you like to use TypeScript? Yes Would you like to use ESLint? Yes Would you like to use Tailwind CSS? Yes Would you like to use src/ directory? Yes Would you like to use App Router? (recommended) Yes Would you like to customize the default import alias (@/*) Yes What import alias would you like configured? @/*

To learn more about the installation process of Next.js, you can check of the official Next.js documentation

Now once the dependencies have been installed go to the project root folder and install Shadcn/UI:

npx shadcn-ui@latest init

You can select the settings you feel the most comfortable with to set up your components.json file. There will be your Shadcn/UI setup.

Learn more about the installation process of Shadcn/UI using Next.js in the official Shadcn/UI installation documentation with Next.js

Installing Shadcn/UI data table and Tanstack Table

Now that you've installed Shadcn/UI it's time to install the table component from the library. This component can be extended using Tanstack Table so we'll need to install that dependency as well.

Install the table component from Shadcn/UI using the following command:

npx shadcn-ui@latest add table

And install Tanstack Table, which uses React Table under the hood and will allow you to add filtering and pagination in the future:

npm install @tanstack/react-table

See the official TanStack Table Documentation

Now we can start by fetching data and creating the Typescript types.

Getting the data

In a real world application you're going to use actual API endpoints with actual client data. However, in this tutorial we'll be using JSONPlaceholder.

And what is this service? According to the official website:

JSONPlaceholder is a free online REST API that you can use whenever you need some fake data.

In this case, we'll be fetching fictional user data using the following endpoint:

https://jsonplaceholder.typicode.com/users

As you can see, the data we're getting is huge and you might be tempted to create the types yourself. I wouldn't recommend you to do so, but instead, using a website like Transform Tools. This is a website that'll allow us to convert files to other formats such as JSON to Typescript Types or React Props.

Copy the response data you're getting from the API and paste it here: https://transform.tools/json-to-typescript

Create a new file in your src file called types.ts and paste the interface from the tool you've used before. You're going to save the types of your project there since this is a small project. Keep in mind you'll need to create and organize different folders for types as your project scales.

Now after some tweaks in the TS interfaces, your data could look like this:

// src/types.ts
export interface User {
  id: number
  name: string
  username: string
  email: string
  address: Address
  phone: string
  website: string
  company: Company
}

export interface Address {
  street: string
  suite: string
  city: string
  zipcode: string
  geo: Geo
}

export interface Geo {
  lat: string
  lng: string
}

export interface Company {
  name: string
  catchPhrase: string
  bs: string
}

Fetching Data and Creating the Services

Let's get started by creating a new folder in our src folder called services. In this folder we're going to create our async functions responsible for fetching data.

Create a new file called index.ts, since we'll only have a single file here for now. Remember to add the return types explicitly as well.

// src/services/index.ts
import { User } from "@/types";

const getUsers = async (): Promise<User[]> => {
  const data = await fetch("https://jsonplaceholder.typicode.com/users");

  return data.json();
};

export default getUsers;

Columns and How to Use them

Once you've created the services, you can define the columns of your table. Columns are where you define the core of what your table will look like. They define the data that will be displayed, how it will be formatted, sorted and filtered.

Create a new file called columns.tsx. We're going to display the following fields from the user:

Username
Email
Phone Number
Name

Hence we can create the columns as the following:

"use client"

import { ColumnDef } from "@tanstack/react-table"
import { User } from "@/types"

export const columns: ColumnDef<User>[] = [
  {
    accessorKey: "username",
    header: "Username",
  },
  {
    accessorKey: "email",
    header: "Email",
  },
  {
    accessorKey: "phone",
    header: "Phone Number",
  },
  {
    accessorKey: "name",
    header: "Name"
  }
]

What's happening in this code? First, we're creating the column definitions importing the ColumnDef type definition from Tanstack Table. This is a generic type, so we add the type we want to infer, in this case it's User from our types.

We create an array with the object definitions we want to display in our table. Each object definition has the properties accessorKey and header. accessorKey refers to the core object field we'll be using, and header refers to the property's display header in the table.

Learn more about the columns definitions in the official Shadncn/UI documentation

Data Tables and their Purpose

Now it's time to create a reusable <DataTable /> component to render our table.

Go to the folder called ui inside our components folder in src. Create a new file called data-table.tsx.

You're going to use the following code from the official Shadcn/UI documentation:

// src/components/ui/data-table.tsx
"use client"

import {
  ColumnDef,
  flexRender,
  getCoreRowModel,
  useReactTable,
} from "@tanstack/react-table"

import {
  Table,
  TableBody,
  TableCell,
  TableHead,
  TableHeader,
  TableRow,
} from "@/components/ui/table"

interface DataTableProps<TData, TValue> {
  columns: ColumnDef<TData, TValue>[]
  data: TData[]
}

export function DataTable<TData, TValue>({
  columns,
  data,
}: DataTableProps<TData, TValue>) {
  const table = useReactTable({
    data,
    columns,
    getCoreRowModel: getCoreRowModel(),
  })

  return (
    <div className="rounded-md border">
      <Table>
        <TableHeader>
          {table.getHeaderGroups().map((headerGroup) => (
            <TableRow key={headerGroup.id}>
              {headerGroup.headers.map((header) => {
                return (
                  <TableHead key={header.id}>
                    {header.isPlaceholder
                      ? null
                      : flexRender(
                          header.column.columnDef.header,
                          header.getContext()
                        )}
                  </TableHead>
                )
              })}
            </TableRow>
          ))}
        </TableHeader>
        <TableBody>
          {table.getRowModel().rows?.length ? (
            table.getRowModel().rows.map((row) => (
              <TableRow
                key={row.id}
                data-state={row.getIsSelected() && "selected"}
              >
                {row.getVisibleCells().map((cell) => (
                  <TableCell key={cell.id}>
                    {flexRender(cell.column.columnDef.cell, cell.getContext())}
                  </TableCell>
                ))}
              </TableRow>
            ))
          ) : (
            <TableRow>
              <TableCell colSpan={columns.length} className="h-24 text-center">
                No results.
              </TableCell>
            </TableRow>
          )}
        </TableBody>
      </Table>
    </div>
  )
}

Let's see how this code works:

Imports: The component imports necessary functions and components from @tanstack/react-table and our table component installed previously @/components/ui/table.
Props: The DataTable component receives two props: columns and data. columns represents an array of column definitions, and data represents the data to be displayed in the table.
useReactTable Hook: Inside the component, the useReactTable hook is used to create a table instance. It takes data, columns, and getCoreRowModelas parameters.
Rendering Header: The component renders the table header by mapping over the header groups obtained from the table instance. For each header group, it maps over the headers and renders a TableHead component for each header.
Rendering Body: The component renders the table body by mapping over the rows obtained from the table instance. For each row, it renders a TableRow component, setting the data-state attribute based on whether the row is selected. Within each row it maps over the visible cells and renders a TableCell component for each cell.
FlexRender: The flexRender function is used to conditionally render the content of each header and cell based on the provided context.
No Results Message: If there are no rows to display, the component renders a single TableRow with a TableCell spanning all columns, displaying a "No results" message.

Displaying the Table in our First Page

Let's use what we've built so far to set up our table in src/app/page.tsx with the following code:

import { columns } from "@/app/columns";
import { DataTable } from "@/components/ui/data-table";
import getUsers from "@/services";

export default async function Home() {

  const data = await getUsers();

  return (
    <main className="flex min-h-screen flex-col items-center justify-between p-24">
      <DataTable columns={columns} data={data} />
    </main>
  );
}

This code works the following way:

Imports: The code imports the columns definition from the @/app/columns file and the DataTable component from the @/components/ui/data-table file. Additionally, it imports the getUsers function from @/services.
Fetching Data: Inside the Home function, the getUsers function is called asynchronously to fetch user data. This function makes an HTTP request to an API endpoint to retrieve the data.
Rendering: Once the data is fetched, it is passed as props to the DataTable component along with the columns definition. The DataTable component renders the table UI based on the provided data and column definitions.
Server side rendering: Next.js SSR is used. Server components allow fetching data directly within the component, enabling server-side rendering of data. In this case, getUsers is invoked directly within the component to fetch user data.

Final Result

If you've followed the steps properly, your final result should look like this:

Conclusion

Congratulations to reaching this point! You've learned how to create professional tables using Tanstack Table, Next.js and ShadcnUI. You've also learned how to connect to the JSONPlaceholder API and how to consume an API endpoint.

You can also implement pagination, filtering or sorting in the future to keep improving your skills with tables in web development. I'll create more advanced guides in the future about this topic.

Thank you so much for reading this tutorial and I hope you've learned something new!

Building a complex application or a high-performance system?

I help startups engineer scalable applications and internal tools that stay fast as they grow. See how I build scalable systems here: 👉 www.itsfranciscoluna.com

Lessons Learned from Building Hobby Explore: My First MVP Journey

Francisco Luna — Thu, 25 Apr 2024 02:04:00 +0000

After almost 3 months of hard work and dedication with the final version and another 5 with the previous ones, I've finished my first minimum viable product, Hobby Explore. This is an application that allows you to share your hobbies with other people, and you're also able to see other people's.

However, this process was far from easy. I spent countless hours building 5 different versions, refactoring code, building features and getting feedback from other developers.

This helped me to develop both my coding skills and my mindset towards building coding projects. That's why I'm sharing here the most important lessons I learned building this project!

First Lesson - Give time to yourself and your ideas

Let's say you have a great idea for a SAAS product and you develop its first version in 5 weeks. You don't like it and you think it's not a good idea to finish.

What if you gave it more time? I built the first version of Hobby Explore in 1 month, and I was thinking the same even after building new versions. It wasn't until I landed my first job and finished other clients' projects that I developed the skills to build the final production version, which I'm really proud of.

Remember, keep developing your skills and trusting your ideas. You don't know if you're going to make it with one of them.

Second Lesson - Experiment and Get Feedback

Let me tell you a fun fact. The first 3 versions of Hobby Explore were made to experiment with backend development using Express, Node.js and MongoDB to fetch mockup hobbies from the Bored API. Yes, I wasn't even thinking about creating a Social Media App. It wasn't until I shared the idea with other developers that they asked me to add the functionality to create different hobbies.

I decided to implement it, and the functionality wasn't the best thing in the world, but hey, at least I built the foundation for the MVP.

Third Lesson - Accept Opportunities and Listen to Others' Ideas

Most of the ideas for the actual MVP and the core UI/UX design were suggested by a designer. He's called Wayne, and he helped me to plan functionalities such as the tips and the user customization in Hobby Explore. Wayne, if you're reading this, you're amazing, and you did an amazing job! Always have someone who can support you throughout the development of your product.

If that's not your case, you can ask for feedback from other developers and your friends, and you can also promote the product you've been building on social media. That's also a good opportunity to improve your product as people get to know it.

Fourth Lesson - Enjoy the journey and improve as you go

Even if you're not able to monetize your product or it wasn't as good as you expected, you must feel proud of yourself. You were able to learn new technologies, overcome challenges and experiment with different functionalities and libraries.

You've also met new people and you've grown as an individual. If you were able to finish one of your best projects, then you can keep building cool stuff and eventually achieve your goals. Iterate, my friend.

Fifth Lesson - Plan and structure your projects

I cannot stress enough the importance of planning your projects. From the design to the actual functionalities and data structures. It helps you to:

Gain clarity and vision: By planning you understand better what you want to achieve and how you think you're going to achieve it.
Save time: It saves you time from making mistakes and changing data structures, functions and variables in your code in the future.
Achieve goals: By saving time and making better choices when building your project, you're likely to achieve your goals faster. A goal doesn't need to be finishing the whole product. You can use SMART goals and set goals like creating a UI component or developing an API endpoint.
Release higher-quality products: So you save time, you gain clarity, and you've achieved your goals. Now, you can also plan your tests through different edge cases to create higher-quality code and make your application more scalable and easier to maintain.

For example, I spent hours writing tests for the most important functionality of Hobby Explore, the one that allows you to create hobbies through dropzones. It wasn't easy, and I had to learn React Testing Library and Vitest, but it was worth it.

Conclusion

Building a MVP from scratch can be an exciting experience where you can improve different skills, meet new individuals and share your product with the world. That's why I shared the 5 most important lessons I learned with my application called Hobby Explore. I've covered topics from planning to creativity, inspiring others to trust their own ideas and build unique apps!

Thank you so much for reading this blog article! I hope you've learned something new and if you've built a product, let me know some of the main lessons you learned building it in the comments.

You can see my project, Hobby Explore, here.