How to replicate a remote Postgres DB locally

Ryan Hansen — Sat, 16 May 2020 20:12:29 +0000

There have been times when I've wanted an exact copy of a database locally. If I'm writing a particularly complex database migration, it's nice to have a clone of a QA environment with sample data.

This is super easy using the postgres client tools. You can stream the remote database directly into your local one!

NOTE: I always start with a fresh database locally to avoid any synchronization errors.

#!/bin/bash
set -eo pipefail
IFS=$'\n\t'

USER=postgres
HOST=localhost
PORT=5432
DB=postgres

REMOTE_DATABASE_URL=postgres://user:pass@host:port

dropdb -U $USER -h $HOST -p $PORT $DB
createdb -U $USER -h $HOST -p $PORT $DB
pg_dump --verbose --clean --no-acl --no-owner --if-exists $REMOTE_DATABASE_URL | psql 
--set ON_ERROR_STOP=on $DB -h $HOST -p $PORT -U $USER

echo "CATS: ALL YOUR *DATABASE* ARE BELONG TO US"

Now you can test your migrations over and over, resetting as you need, from the remote database!

Becoming Homeless

Ryan Hansen — Sat, 16 May 2020 05:00:18 +0000

The other day at work I was doing something and accidentally made a folder named tilde.

ls
~ bin lib spec

Oops, well I guess I should remove that.

$ rm -r ~
rm: remove write-protected regular file .ssh/config?

Wait… what?! Oh crap, did that just delete my home directory?

$ ls ~
total 0
drwxr-xr-x 17 user group  544 Apr  18 11:45 .
drwxr-xr-x 27 user group  864 Apr  18 10:18 ..

Lucky I have rdiff-backup run weekly. I changed that to daily.

Note to self, run rm -r ./~ next time.

Man what a day.

Enforcing Java Coding Styles

Ryan Hansen — Sat, 16 May 2020 04:43:38 +0000

In any large organization there is often a desire to be able to move engineers between teams and projects. Having a consistent coding style will often make that easier. There are several reasons for this and many people have articulated them far better than I could. I'll assume you already have or want to begin using a style guide.

I'm a firm believer that if you are going to have a style guide you have to enforce it. As engineers, we automate everything. Manually spot-checking pull-requests to ensure consistency is fraught with errors. If you are going to enforce it, then you should make it as easy as possible through tooling. Engineers should not be spending their time formatting if blocks or sorting import statements. We want to enforce that style but not make it time-consuming.

The Ruby language has a pretty standard style guide. There is a great CLI tool called Rubocop that will check against this style guide for violations. But my favorite part of Rubocop is rubocop --auto-correct. This command will fix almost all of your violations for you every time.

Java has something similar. One of the most widely used style guides is from Google. You can use fmt-maven-plugin to check for violations. And just like Rubocop, it can auto-correct some of your code for you by running mvn com.coveo:fmt-maven-plugin:format.

There are several other static analysis tools in Java. Some of my favorites are maven-pmd-plugin, maven-checkstyle-plugin, and spotbugs-maven-plugin. I won't deep dive into any of these tools, but I highly recommend you take a look at them.

Getting back on track...

Ok, so we have a style guide. We have lots of tools to check it, automate auto-corrections, and analyze for other inconsistencies. So what now?

As always, your CI is a great place to enforce it!

Here is a sample bash script I use. It will return a non-zero exit code if any of the commands inside it fail. This makes it perfect to throw into any CI flow.

#!/bin/bash
set -eo pipefail

mvn fmt:check
mvn checkstyle:check
mvn pmd:check
mvn spotbugs:check

mohanpedala has a great explanation of set -eo pipefail.

Locally you can setup a git hook to run mvn com.coveo:fmt-maven-plugin:format so that you auto-correct some styles before you even push changes.

Here is a sample pom.xml with these configured...

<plugin>
    <groupId>com.coveo</groupId>
    <artifactId>fmt-maven-plugin</artifactId>
    <version>2.9</version>
    <configuration>
        <sourceDirectory>src/main/java/com/ketiko</sourceDirectory>
        <testSourceDirectory>src/test</testSourceDirectory>
        <style>google</style>
    </configuration>
    <executions>
        <execution>
            <goals>
                <goal>format</goal>
            </goals>
        </execution>
    </executions>
</plugin>
<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-pmd-plugin</artifactId>
    <version>3.12.0</version>
    <configuration>
        <verbose>true</verbose>
    </configuration>
</plugin>
<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-checkstyle-plugin</artifactId>
    <version>3.1.0</version>
    <configuration>
        <configLocation>google_checks.xml</configLocation>
        <consoleOutput>true</consoleOutput>
        <failOnViolation>true</failOnViolation>
        <failsOnError>true</failsOnError>
        <includeTestSourceDirectory>true</includeTestSourceDirectory>
        <linkXRef>false</linkXRef>
        <logViolationsToConsole>true</logViolationsToConsole>
        <violationSeverity>warning</violationSeverity>
        <suppressionsLocation>
            checkstyle-suppressions.xml
        </suppressionsLocation>
    </configuration>
</plugin>
<plugin>
    <groupId>com.github.spotbugs</groupId>
    <artifactId>spotbugs-maven-plugin</artifactId>
    <version>3.1.12</version>
</plugin>

** Update **

elmuerte brought up an excellent point in the comments that I wanted to reply to. Maven has several phases during its build process. The verify phase is described as follows:

Run any checks to verify the package is valid and meets quality criteria.

Using this you can run a single maven command in your CI, mvn verify. Simply update those plugins to run during that phase and you are all set! This is an excellent way to accomplish the same thing.

However, maven phases run in order. That means that verify runs after the test phase. I like to run the quality checks first thing in my CI without waiting for my test suite to complete.

You could just run those goals individually in your CI as well, without the bash script. When I wanted to run them locally I kept forgetting all the checks I had configured in the CI. So I created the bash script to quickly run them locally and reuse it on the CI.

DEV Community: Ryan Hansen

How to replicate a remote Postgres DB locally

Becoming Homeless

Enforcing Java Coding Styles