You should embrace database seeding

Jaden Lemmon — Thu, 25 Feb 2021 03:45:00 +0000

Opinionated thoughts on why you should build a process around keeping your development environment seeded with randomized data automatically.

When you embark on a new web application, you should consider building seeders for test data in your local and development environments. Maybe this is a concept that you already know and are familiar with. Maybe you have never used anything like this at all.

I think we can all relate to the scenario of when we are trying to set up our local environment for a project and then realize the database is empty and there is no data to work with to start developing a new feature. So then we reach out to our team and ask for a database dump and one of our friends slacks it to us. We then import it into our local environment and then go about our day developing a feature.

This concept may have worked for you initially but then what happens once someone on your team modifies the database schema rendering your current dataset obsolete. Now you must start this whole process over again and retrieve a new dump.

While this is just one pain point of not using seeders, let me explain all the benefits and maybe there are some that could be useful to you during your development process.

Streamline Local Development

When using seeders, I will commonly configure an application to run seeders automatically in local development, reducing the need to reach out for database dumps or instructions on building a local database. I am a big believer in empowering your team to write quality code instead of spending time building environments. When your database schema is updated it will force developers to update the seeder alongside it causing test breakage if they don’t.

How long does it typically take a new developer to onboard onto the application you are working on? I’ve seen days pretty commonly. How much time is reduced if your entire environment can be built with a single command like npm start?

Consistent Schema Validation

When using seeders, you can then build the seeding into your CI process testing the construction of your database and that it accepts the seeded dataset without error. This allows initial validation around your database schema and that no developer has altered it in a way to cause unintended side effects.

Testing

When writing tests for particular features, you commonly will need a populated database to run a proper test. Let’s say I need to test the retrieval of an endpoint containing “projects”.

For example, the Laravel framework has coined the term “Factory” as a way to create a model instance with test data. I can easily create a few projects for my test like
Project::factory()->count(3);

Random but consistent data set

In production, users are most likely able to enter their values that will be propagated into your database. If you are using the same dataset for local development and testing you are never testing other values through your application which would result in edge case errors in production depending on what kind of logic is being used on those values.

When creating seeders I can utilize a library like faker to randomize the values in the columns. Every time I seed my database with these random values it allows me to test my database in a similar environment to production where users can enter their values. If an edge case existed to trigger an application error because of content you are increasing the likelihood of triggering such error in development.

Easy to create isolated environments for testing features

It’s easy to use with a tool like Voyage where you can deploy on-demand isolated environments to test your new features under development. Let’s say you are working on updating the database schema while you develop a feature. You do not want to keep pushing iterations to your development/staging environment making several database changes during this process.

Voyage will spin up an entirely new and isolated environment with every commit on every PR which means the database starts fresh with your seeders every time a new commit is pushed. This allows you to continually make database alterations that will not dirty up other environments.

Using Frameworks

Most well-known language frameworks provide built-in seeder functionality out of the box. Here are some helpful links to each one.

PHP / Laravel Seeders

Express / Knex.js Seeders

Python / Django Management Commands

Ruby / Rails Seeders

Dev Deployments for a Full-Stack App in Minutes

Jaden Lemmon — Wed, 03 Feb 2021 17:44:59 +0000

Creating feature deployments in Voyage for our Node, React, Postgres app.

Live demo deployment

Voyage makes it super easy to deploy a full stack application to a demo environment quickly. The only time required is typically the initial integration of Voyage into your application.

I’m going to reference an example application called ShipFinder. This app consists of a Node.js API, Create React App Front-End and Postgres database. This application is contained in a single monorepo. Voyage does allow multi-repo deployments but it does streamline the development process with a single repo. See multi-repo deploy docs.

Using Docker

If you are already using Docker within your project, you're already a step ahead. Voyage runs docker images similar to how you may have seen in a docker-compose.yml file. Each service runs in it’s own image. For most applications you will have a “primary” image that is built that contains your application code.

We will start by creating a Dockerfile in the root of our project that contains code to build our app. Since this is a Node project I will use a Node image. You may use a different Node version then what I have chosen.

Voyage does provide some presets for building simple applications. When using a preset you don’t need to have a Docker container. Only if you are building a more custom application. See the documentation for more information on presets.

Voyage Configuration

Now that we have this file we will begin to craft our Voyage config file. We will add a service for our “primary” image containing our app. All config files should begin with the “services” key at the top. Similar to how a docker-compose file may look.

services:
 app:
   context: ./
   primary: true
   exposePort: 3000

I will then expand a little bit to add some environment variables to pass to the application. These environment variables are also passed as build arguments while building the primary Docker container as well.

services:
 app:
   context: ./
   primary: true
   exposePort: 3000
   environment:
     - name: APP_ENV
       value: 'voyage'
     - name: NODE_ENV
       value: 'production'
     - name: DB_HOST
       value: 127.0.0.1

Now we will need one more service for this particular deployment. A Postgres database. It’s simple to add this as I can just pull the standard postgres docker image in like so.

database:
   image: postgres:13
   environment:
     - name: POSTGRES_USER
       value: 'voyage'
     - name: POSTGRES_PASSWORD
       value: 'voyage'
     - name: POSTGRES_DB
       value: 'voyage'

You may want to store some of these environment variables as sensitive values through the Voyage UI. You may then reference them in the config file. See environment variables documentation for more information.

Full Config File

After creating this config, your project is set to build and teardown environments on every pull request on demand. There is nothing else you need to do besides ensure this repository is enabled in Voyage!

Connecting Project

To connect your repository for on demand deployments, navigate to the “Providers” tab in the Voyage UI and ensure your git provider organization is added and press “Configure Connected Repos”.

Then just press “Connect with Voyage” on the repo you would like to connect and select “Manual” for this project. Now your repo is connected.

Next Steps

Checkout our blog for more posts like these and our documentation for additional help configuring your projects.

DEV Community: Jaden Lemmon