DEV Community: Rob Reid

CockroachDB's Online Schema Changes make modifying foreign keys safe and terror-free!

Rob Reid — Mon, 27 Feb 2023 16:02:28 +0000

In the background, CockroachDB will create a new version of your data, to ensure that existing queries can still be serviced.

This means you can make scary-looking changes without fear of downtime or customer impact 💪

CockroachDB's Online Schema Changes make modifying primary keys safe and terror-free!

Rob Reid — Thu, 23 Feb 2023 14:46:04 +0000

In the background, CockroachDB will create a new version of your data, to ensure that existing queries can still be serviced.

This means you can make scary-looking changes without fear of downtime or customer impact 💪

My favourite backend tools

Rob Reid — Thu, 24 Jun 2021 20:47:57 +0000

I spend a lot of my time as a software developer writing backend services and infrastructure and wanted to share some of the tools I use most regularly.

I'm always in the market for new tools, so please feel free to add your favourite tools in the comments!

golang

awesome

Go is a fantastic language for web server development. It's standard library is very comprehensive, meaning you can do pretty much everything you'd want to do with just the standard library (although there are plenty of packages that have already solved a lot of common challenges).

Spinning up a web server is as simple as:

package main

import (
    "fmt"
    "net/http"
)

func main() {
    http.HandleFunc("/hello", hello)

    log.Fatal(http.ListenAndServe(":8080", nil))
}

func hello(w http.ResponseWriter, r *http.Request) {
    fmt.Fprintln(w, "Hello, World!")
}

...although you should not use http.ListenAndServe when developing applications destined for production, as it does not configure sensible timeout defaults.

Go's tooling is as comprehensive as its standard library, meaning you can test, benchmark, profile, and generate coverage of your code just as easily as you wrote it.

crystal

awesome

Crystal is another great programming language for backend development. It's very concise, very declarative, and very fast.

Crystal allows a developer to write very terse code to achieve what would typically be much more verbose in other programming languages. For example, here's a web service handler using Crystal’s most well-known web framework, kemal. It accepts the id of a person from the URL path, queries a database for them using their is, and returns their first and last name as JSON:

get "/people/:id" do |env|
  id = env.params.url["id"]
  first, last = db.query_one "select first_name, last_name from person where id = $1 limit 1", id, as: { String, String }

  {"firstName": first, "lastName": last}.to_json
end

replit

Replit started out as an online IDE but thanks to the team's ridiculous efficiency in getting out new releases, it's become a completely game-changing platform.

With Replit, you can create websites, web services, games, and more, in any language you can think of (including esoteric ones). I use Replit to host long-running websites with custom domain names; all linked to GitHub. Everything's super easy and I think the world of online hosting is going to be disrupted by this awesome website.

benthos

Benthos is a declarative stream processor that makes mundane tasks fun and allows you to achieve ridiculous levels of productivity. It's under active development with new components (inputs, processors, and outputs etc.) being added regularly.

It can run anywhere and can be configured statically with YAML files or dynamically with Streams Mode. The following example consumes from 2 Kafka topics and writes messages from both into a Cassandra table:

input:
  kafka:
    addresses:
      - localhost:9092
    topics:
      - topic_one
      - topic_two
    consumer_group: my_benthos_app

output:
  cassandra:
    addresses:
      - localhost:9042
    query: INSERT INTO my_keyspace.my_table (id, event, created_at) VALUES (?, ?, ?)
    args:
      - ${! json("id") }
      - ${! json("event") }
      - ${! json("timestamp").format_timestamp() }

kubernetes

Kubernetes is a tool that's synonymous with the deployment of scalable and reliable application workloads. It's a huge tool that'll likely offer more than most people will need but it's very powerful, has a huge community, and allows you to quickly and reliably scale and serve your applications.

I use Kubernetes to run all sorts of application workloads and regularly make use of its ability to integrate with cloud providers to transparently create things like load balancers.

nomad

Nomad is an awesome alternative to Kubernetes. It's hugely scalable, allowing you to schedule many different types of workload including Docker and Podman containers, raw executables, and Java applications (to name a few).

Unlike Kubernetes, applications can (and should) be scheduled onto machines that are not part of the orchestration cluster nodes. This allows Nomad to scale to millions of running applications, whereas Kubernetes best practices recommend a limit of 150,000 pods and 300,000 containers.

I use Nomad to orchestrate IoT workloads but it can be used for many more use cases.

wrk / hey

wrk and hey are amazing HTTP benchmarking CLIs. They allow you to write very concise commands to benchmark web services, offering higher speeds and concurrency levels than alternative tools.

wrk is the fastest available (by a large margin) and can be scripted with Lua. Although it doesn't run on Windows without the use of WSL (Windows Subsystem for Linux). I tend to use wrk when I'm on my Mac and hey when I'm on Windows machines.

k6

k6 is a fantastic Open Source load testing tool that can be scripted with JavaScript.

You can configure very complex load testing scenarios with large numbers of VUs (Virtual Users) to put your web services under realistic smoke, load, stress, and soak scenarios.

I use k6 for anything more complicated than simply hammering a single endpoint.

terraform

Terraform is a de-facto standard for deploying infrastructure with code. It has official support for many cloud providers and services and can be extended with custom providers depending on your needs.

I use Terraform for managing infrastructure. It's not without its issues but once you're aware of them, it's a very productive tool that frequently leaves me in awe.

prometheus / grafana

Prometheus is a tool for collecting metric data from applications. You expose an endpoint from your application (e.g. /metrics), point Prometheus at it, and simply write metrics; all using the Prometheus client libraries that are available for most major programming languages.

Grafana is a great tool for graphing metrics. It can read data from lots of different metric/time-based backends (including Prometheus and InfluxDB etc.) and can be configured to raise alerts.

ngrok

ngrok creates "secure introspection tunnels to localhost", which - put simply - allows you to expose your machine to the outside world, even if it's behind a firewall.

The following command will expose any application running on port 8080 on your machine to the outside world. It will generate 2 unique URLs (one for HTTP and one for HTTPS):

$ ngrok http 8080

I use ngrok for testing local services against external services like applications that publish web hooks.

postman

Postman is a tool for testing web servers. It's feature-rich and allows you to test HTTP and GraphQL services with minimal effort.

You can also use Postman as a documentation and code-generation tool (create a request, then generate code for lots of different tools and programming languages).

I use Postman for testing both HTTP and GraphQL services, saving collections of request for each service for reproducible tests.

k9s

k9s is an awesome CLI for interacting with Kubernetes.

With k9s you can view pods, jump into their shells, view their logs, port forward, and delete them. You also have a lot of options for managing deployments, stateful sets, services and many other Kubernetes constructs.

grpcurl / grpcui

grpcurl and grpcui are great tools for working with gRPC services. To start, simply create a gRPC service with reflection enabled and open either of the tools. You'll now be able to call the endpoints just like you would with tools like cURL or Postman.

table plus

TablePlus is a great tool for working with a lot of database engines including Postgres, CockroachDB, MySQL, SQLServer, Cassandra, MongoDB, Redis, and more.

Having the ability to work with so many data stores makes developing any kind of database-backed service very straightforward.

datagen

A tool from me

datagen is a tool for generating lots of random relational data quickly. You write a template file using a mixture of SQL and Go templates to create multi-row DML (Data Manipulation Language), or more simply, batches of INSERT statements.

Outputs from one datagen insert (e.g, the output from a RETURNING or a follow-up SELECT statement) can be used as the input for the next, allowing you to build up related data across multiple tables.

I use datagen for generating monstrous amounts of data for testing with while developing, as the best time to find issues with database performance and integration is during development!

setcfg

A tool from me

setcfg allows you to substitute values in one YAML file with values from another YAML file like so:

Input file:

region: ~region~

credentials:
    username: ~username~
    password: ~password~

brokers:
    - broker: ~broker-1~
    - broker: ~broker-2~

subnet_cidrs: ~subnet-cidrs~

Config file:

region: eu-west-1

username: admin
password: supersecret

broker-1:
    host: https://localhost
    port: 8001

broker-2:
    host: https://localhost
    port: 8002

subnet-cidrs:
- 1.2.3.0/25
- 1.2.3.128/25

Running setcfg with these two files (setcfg -i input.yaml -e config.yaml) will result in the following, which can be piped into a file or a kubectl command, depending on the type of input file you’re using:

brokers:
- broker:
    host: https://localhost
    port: 8001
- broker:
    host: https://localhost
    port: 8002
credentials:
  password: supersecret
  username: admin
region: eu-west-1
subnet_cidrs:
- 1.2.3.0/25
- 1.2.3.128/25

I use this for working with things like Kubernetes manifests and Benthos configuration files, although it'll happily work with any YAML file.

pa55

A tool from me

pa55 is a simple CLI for generating cryptographically secure passwords and copying them to the clipboard. It can generate passwords of any length and output them in ascii, base32, base64, and hexadecimal.

I use pa55 for generating passwords for session keys, and SaaS solutions.

see

A tool from me

see is a simple CLI that does what the Linux watch command does but for any platform.

Conclusion

A great tool can not only help you to become more efficient but can also help to make mundane tasks fun. If you find a tool that you love, make sure to support it with a star on GitHub and possibly a contribution!

Crystal's Channels and the While Loop

Rob Reid — Sat, 04 Jan 2020 08:32:22 +0000

Like most, I like to learn a new programming language by doing. At the moment, I'm enjoying the catharsis of completing unimplemented Rosetta Code tasks in Crystal and vlang.

During some exploration of the Crystal programming language, I took on the Synchronous Concurrency task to pour some of my knowledge of the CSP pattern from Go, into Crystal.

The task asks you to communicate between multiple threads of execution within a process:

One of the concurrent units will read from a file named "input.txt" and send the contents of that file, one line at a time, to the other concurrent unit, which will print the line it receives to standard output. The printing unit must count the number of lines it prints. After the concurrent unit reading the file sends its last line to the printing unit, the reading unit will request the number of lines printed by the printing unit. The reading unit will then print the number of lines printed by the printing unit.

This task requires two-way communication between the concurrent units. All concurrent units must cleanly terminate at the end of the program.

My original solution to the task was as follows:

File.write("input.txt", "a\nb\nc")

lines = Channel(String).new

spawn do
  File.each_line("input.txt") do |line|
    lines.send(line)
  end
  lines.close
end

begin
  while
    line = lines.receive
    puts line
  end
rescue ex : Channel::ClosedError
end

File.delete("input.txt")

$ crystal run example.cr
a
b
c

The Crystal docs here and here are fairly light on Channels, so there was an element of trial and error involved. I knew that a call to receive against a closed channel would result in a raised Channel::ClosedError so I made use of this in a try/catch (begin/rescue) block.

I'm a big fan of Go, so much prefer to handle errors over catching exceptions. I wasn't satisfied with this solution and wanted to see if I could make Crystal's type system work to my advantage.

Crystal's type system makes use of Union types, allowing a variable to be one or more types at compile time. For example, because a is initialised in both arms of the if statement in the following example, it can be either a String or an Int32:

if true
  a = "string"
else
  a = 1
end

puts typeof(a)
# => (Int32 | String)

What's interesting about Union types in the context of Channels, is Nil. There's another version of the receive method, that instead of raising an exception when a channel is closed, it returns nil. It's called receive? and its definition can be found here.

Initially, I simply swapped out the called to receive with receive? and re-ran my code:

File.write("input.txt", "a\nb\nc")

lines = Channel(String).new

spawn do
  File.each_line("input.txt") do |line|
    lines.send(line)
  end
  lines.close
end

begin
  while
    line = lines.receive?
    puts line
  end
rescue ex : Channel::ClosedError
end

File.delete("input.txt")

I was expecting the call to receive? to block indefinitely after reading the last line from "input.txt" but was surprised to see the program output exactly what it had done in the original example:

$ crystal run example.cr
a
b
c

Confused, I added a log line into the rescue arm to see if a Channel::ClosedError exception had been thrown anyway (despite what the Crystal source told me) and re-ran:

rescue ex : Channel::ClosedError
  puts ex

$ crystal run example.cr
a
b
c

No exception!?

Then it dawned on my that the while loop must be working to truthy values and ignoring the line break!

I rewrote my code to this new expectation, removing the begin/rescue block and the line break, inlining the declaration of line, and relying on nil from the fourth call to receive?:

File.write("input.txt", "a\nb\nc")

lines = Channel(String).new

spawn do
  File.each_line("input.txt") do |line|
    lines.send(line)
  end
  lines.close
end

while line = lines.receive?
  puts line
end

File.delete("input.txt")

$ crystal run example.cr
a
b
c

Success! I've learned something new about Crystal by fumbling around in the dark.

The following demonstrates this succinctly by reading from the array elements until nil:

a = [1, 2, 3, nil, 4, 5]

while b = a.shift
  puts b
end

CockroachDB via Crystal ... In under 50 lines

Rob Reid — Mon, 30 Dec 2019 22:09:07 +0000

I've been getting rather excited about the Crystal programming language recently and I've finally gotten round to creating a very simple CockroachDB-backed Crystal API!

In this example, I'm going to create a very contrived API to manage todos. When the API is up and running, you'll be able to list the todos stored in the database, find a todo by its ID, and add new ones. These use cases lend themselves nicely to demonstrating 3 of the DB::QueryMethods.

In this post, I'll show you how to install Crystal, bring in the dependencies we'll need to create our API and walk you through the code that'll give us the following endpoints:

POST /todos

GET /todos

GET /todos/:id

Setting things up

First, install Crystal. There are a number of ways to do this listed on their install page. I use macos, so will keep things Mac-specific for this post but you'll be able to follow along using the information on their website if you're on a different platform:

$ brew update
$ brew install crystal

Check your installation by running the crystal executable:

$ crystal version
Crystal 0.32.1 (2019-12-18)

LLVM: 9.0.0
Default target: x86_64-apple-macosx

Next, create a project for our code to go into. This can all be doing using Crystal's CLI:

$ crystal init app api
    create  api/.gitignore
    create  api/.editorconfig
    create  api/LICENSE
    create  api/README.md
    create  api/.travis.yml
    create  api/shard.yml
    create  api/src/api.cr
    create  api/spec/spec_helper.cr
    create  api/spec/api_spec.cr
Initialized empty Git repository in /Users/me/crystal/learning/api/.git/
$ cd api

Finally, create a database for our application to connect to:

$ cockroach demo --nodes=1 --empty
...
root@127.0.0.1:64044/defaultdb>

Note that the terminal prompt resulting from a successful call to this command will be the address of the cluster we'll be connect to. In this case, "127.0.0.1:64044".

Using the resulting terminal prompt, we can create a table to house our data:

root@127.0.0.1:64044/defaultdb> create table "todo" (
    "id" uuid primary key default uuid_v4()::uuid,
    "value" text not null
);
CREATE TABLE

Time: 3.213ms

The components

Crystal's standard library - like the standard libraries of most other programming languages - doesn't contain specific database drivers. We'll need a Postgres driver in order to talk to CockroachDB. According to the Crystal docs, will/crystal-pg is the one to use.

Crystal's dependency manager is called Shards and in your api directory, the crystal init command would have created a file called shard.yml for you. Into that file, add the following to reference the Postgres driver as a dependency:

dependencies:
  pg:
    github: will/crystal-pg

Next, run the following to pull in the dependencies listed above. Note that it's possible to specify versions for your dependencies but for this example, the latest version will be fine:

$ shards install
Fetching https://github.com/will/crystal-pg.git
Fetching https://github.com/crystal-lang/crystal-db.git
Using pg (0.20.0)
Using db (0.8.0)

Rather than rolling my own API routing code, I'm going to use the well-known Kemal web framework. It creates macros that result in very simple and elegant API code. Bring in this dependency in the same way you brought in the database driver, by adding it to the dependencies section of your shard.yml file:

dependencies:
  kemal:
    github: kemalcr/kemal
  pg:
    github: will/crystal-pg

$ shards install
Fetching https://github.com/kemalcr/kemal.git
Fetching https://github.com/luislavena/radix.git
Fetching https://github.com/jeromegn/kilt.git
Fetching https://github.com/crystal-loot/exception_page.git
Fetching https://github.com/will/crystal-pg.git
Fetching https://github.com/crystal-lang/crystal-db.git
Using kemal (0.26.1)
Using radix (0.3.9)
Using kilt (0.4.0)
Using exception_page (0.1.2)
Using pg (0.20.0)
Using db (0.8.0)

The code

All of our code will go into src/api.cr. First, we'll want to reference our dependencies.

require "kemal"
require "db"
require "pg"
require "json"

The reference to kemal will allow us to use the fancy HTTP macros and start the server. The reference to db (a dependency automatically required by our explicit crystal-pg dependency) will allow us to make calls to the database. The reference to pg brings in the Postgres driver and provides a helper method we can use to connect to CockroachDB. The reference to json (in the standard library) will allow us to respond to callers in JSON.

Next, we open a connection to the database with the connection information we obtained when starting the CockroachDB cluster:

db = PG.connect "postgres://root@localhost:64044/defaultdb?sslmode=disable"

To keep the code clean, I create a Todo class to store the information on the todos being dealt with by the API. In this class definition, there's a constructor, which provides shortcut syntax that creates two instance variables; "id" and "value" and a JSON.mapping function, which tells Crystal how the Todo class is to be serialised to and deserialised from JSON:

class Todo
  def initialize(@id : String, @value : String)
  end

    JSON.mapping({
    id: {type: String},
    value:  {type: String},
  })
end

Kemal provides some very cool macros which are listed in the guide. I'm using a "filter" macro called "before_all", which - as you'll probably guess - gets invoked before all endpoint calls, allowing me to jump into the web call flow and inject a response header to ensure the caller always gets a "Content-Type" of "application/json":

before_all do |env|
  env.response.content_type = "application/json"
end

Onto the handlers!

POST /todos

Let's start by getting some todos into the database. We'll need a POST endpoint for that, and the "post" function created from a macro in Kemal is just the thing. Kemal captures the request and the convention for that, is |env|. From env, we can extract stuff from the JSON body.

Once inserted into the database, we'll want the returned ID, so the scalar function is perfect for this, as we'll return it from the INSERT statement. Once we've got the ID, we can build up a simple JSON object to return to the caller:

post "/todos" do |env|
  value = env.params.json["value"].as(String)
  id = db.scalar("insert into todo (value) values ($1) returning id", value).as(String)
  { "id": id }.to_json
end

GET /todos

Next, we'll create an endpoint to view all of the todos in the database. This time we don't need to capture the user's request and will use the query function to fetch all of the todos with a ResultSet, building up an array of todos before returning the JSON representation:

get "/todos" do |env|
  todos = Array(Todo).new
  db.query "select id, value from todo" do |rs|
    rs.each do
      id, value = rs.read(String, String)
      todo = Todo.new(id, value)
      todos << todo
    end
  end

  todos.to_json
end

GET /todos/:id

Finally, we'll make use of the query_one function to fetch a single user. To narrow down the search, we'll capture the ID of the todo the user wants to find and use it when querying the database:

get "/todos/:id" do |env|
  id = env.params.url["id"]
  id, value = db.query_one "select id, value from todo where id = $1 limit 1", id, as: { String, String }

  todo = Todo.new(id, value)
  todo.to_json
end

You'll notice that the todo's ID is already provided, so I could have used the scalar function and return just the value of the todo but in a real-world scenario, there will likely be more columns to return and query_one is the correct function for that.

Kemal

The only remaining thing to do now, is kick off the Kemal server. To do that, we run the Kemal.run function. By default, Kemal will listen on port 3000 but a custom port can be provided:

Kemal.run 1234

By default, Kemal will log all of the requests it receives. To disable this, run the following instead:

logging false
Kemal.run 1234

Testing

Create a todos using the POST /todos endpoint:

$ curl -X POST \
  http://localhost:1234/todos \
  -H 'Content-Type: application/json' \
  -d '{
    "value": "Productionise todo API"
}'
{"id":"e548f471-5772-473b-b7d2-39e18c57d483"}

Fetch all of the todos using the GET /todos endpoint:

$ curl -X GET \
  http://localhost:1234/todos \
  -H 'Content-Type: application/json' \
  -d '{
    "value": "Productionise todo API"
}'
[{"id":"e548f471-5772-473b-b7d2-39e18c57d483","value":"Productionise todo API"}]

Fetch the todo we created with the GET /todos/:id endpoint:

$ curl -X GET \
  http://localhost:1234/todos/e548f471-5772-473b-b7d2-39e18c57d483 \
  -H 'Content-Type: application/json' \
  -d '{
    "value": "Productionise todo API"
}'
{"id":"e548f471-5772-473b-b7d2-39e18c57d483","value":"Productionise todo API"}

Our Kemal server would have been logging these requests as well:

2019-12-30 21:31:02 UTC 200 POST /todos 2.15ms
2019-12-30 21:33:17 UTC 200 GET /todos 922.05µs
2019-12-30 21:34:06 UTC 200 GET /todos/e548f471-5772-473b-b7d2-39e18c57d483 1.52ms

I'm still learning the Crystal ropes, so will continue to blog about the language as I stumble across new things!

The Ethical Cloud

Rob Reid — Mon, 09 Dec 2019 16:29:44 +0000

On October 17th, I gave a talk at the inaugural east-cost multi-cloud conference, ESCAPE in Manhattan, organised and hosted by Cockroach Labs.

I wanted to deliver this talk for a number of reasons:

The tech industry isn't talking about these issues as much as it should be.
There are very few tech conferences where this kind of talk would be encouraged.
I know next to nothing about the subject and there's no greater kick up the arse to learn something than commit to speaking at a conference.

I work at Lush, a company that's committed to having a positive impact on the world and its inhabitants.

In my talk, I covered the environmental, data privacy, and human impacts of cloud computing. Each of the 3 sections included information on what Lush are doing in this area (limited to stuff that's relevant to the talk), along with suggestions to the attendees on what they could do.

In my opinion, this stuff is too easily ignored and too readily forgotten.

Environment

I opened my talk with the following question:

Who's in the cloud?

Around 80% of hands went up. Not unsurprising for a cloud conference! I then asked the particpants to keep their hands up if they knew:

How much of your cloud provider's energy is renewable?

Every single hand went down. No one knew how much of their cloud provider's energy usage was green. This was validation that this talk needed to happen and threw into sharp focus, the fact that the environment can be an afterthought for all of us in the tech industry.

The Internet is likely to be the biggest thing we create as a species - Greenpeace

Greenpeace are doing a lot of great work in this area, they publish reports detailing the major cloud and managed service providers and their energy impacts. To me, this quote highlights the scale of the impact the cloud will have if we don't help and push our providers to acheive 100% renewable energy soon.

As of 2019, cloud spending is around ~$200B. By 2022, that's going to increase to around $1.3T, a six-fold increase to an industry that already uses 3% of our globally available power and produces as much emissions as the entire aviation industry.

To put some figures to this, global data centre power consumption is around 100B kWh. Amazingly, 30-40% of this consumption goes to cooling to them. Google are raising the temperature in their data centres to reduce the amount of power they use on cooling. The result? They now direct around just 10% of their total power consumption to cooling, saving energy and billions of dollars in the process.

Virginia

Virginia is a major state for data centres. Many of the world's largest providers have a massive presence in the area. One provider in particular ramped up its operations by 60% in the past 2 years alone. All in a state whose available renewable energy represents just 4% of its total available energy, putting it 38th in the rankings for greenest state.

There are neighbouring states with better access to renewable energy!

Sadly, new fossil fuel infrastructure is being built to cater for this demand and it's coming in the form of the 42" Atlantic Coast Pipeline, delivering massive amounts of gas to the power-hungry data centres. This represents a huge step backwards in our critically important need to be sustainable.

Our clouds need your help.

Employees from Amazon, Google, and Microsoft are campaigning for their clouds to act faster in going green and they need our help:

Like Lush, these groups all walked out on September 20th in support of the global climate crisis strikes and we need everyone to join us, show support, and tell the world that its biggest and most influencial companies have big concerns for the future of our planet.

The 3 big cloud providers publish reports that detail their aspirations for 100% renewable energy consumption (along with their progress) and while you should try to independently validate the claims being made, they make for interesting reads:

What Lush are doing

Being picky - a few years ago, we moved all of our workloads from one cloud provider to another (in 22 days, over Christmas, the team are incredible). One of the reasons we did this was to align ourselves with a cloud provider whose environmental ethics closely matched ours.
Asking our managed service providers to do the same - we don't shy away from asking those who provide us with a managed service if they'd be prepared to switch clouds to join us. They might not be able to (and that's OK) but we think it's important to ask.
Minimising over-provisioning - we've taken a look at our infrastructure and where we're being wasteful. The outcome of this, is a project to reduce our production estate by around one third; maximising our utilisation of the cloud and having the happy side-effect of saving us a lot of money in the process.
Redesigning our applications for efficiency - we're committed to using best-in-class software to make sure that our applications run as efficiently as possible.

What you can do

Non-cloud users - consider moving into the cloud! There are massive efficiencies to being in the cloud due to the hardware available and the virtualisation technologies used. Even if your concerns aren't for the environment, allow it to be a side-effect of the decisions you make.
Choose your provider carefully - some clouds appear to be in a period of growth-at-any-cost, which is resulting in a balooning demand for fossil fuels, while others are on a commit course to being greener.
Consider the consequencies of over-provisioning your estate - beyond simply looking at it from a cost perspective. Wherever you're over-provisioning, there's power (and money) being wasted.
Speak with your managed service providers - ask them to jump to a greener cloud. They might not be able/prepared to but hopefully it'll start them thinking about the issue as well.
Optimise your cloud use - the 3 big cloud providers provide tools to help you get the best from their services, with suggestions like reducing instance sizes for workloads that never stress them:

Data privacy

I opened this segment up by asking a series of questions to gauge how much the audience knew about the implications of data privacy laws on their customers' privacy:

Who’s familiar with the 4th amendment?

Around 5% of the audience were familiar with the 4th amendment, which centres around:

The right of the people to be secure in their persons, houses, papers, and effects, against unreasonable searches and seizures.

I then asked:

Who's familiar with the Third Party Doctrine?

One hand remained in the air. Before giving this talk, I would have been among the vast majority of people with my hand in my lap. The Third Party Doctrine states that:

People who voluntarily give information to third parties such as banks, phone companies, Internet Service Providers, and email servers have no "reasonable expectation of privacy."

There have been a number of court cases seeking to establish a legal precedent and they demonstrate how challenging this law can be to navigate:

In the case of Smith v. Maryland, Michael Lee Smith was deemed not to have a legitimate expectation of privacy because he'd shared the number he was calling with the phone company, simply by dialling it. His phone record was therefore available to be used against him during his prosecution.

Don't forget that the doctrine also relates to Internet Service Providers and therefore the data we share with our cloud providers.

I then asked:

Who’s familiar with the CLOUD Act?

One hand again (a different one this time) and as with the last question, I would have been in the vast majority before reseaching for this talk.

CLOUD stands for Clarifying Lawful Overseas Use of Data and was signed into law by President Trump on March 23rd 2018 without being debated in congress. It:

Allows federal law enforcement to compel US-based technology companies via warrant or subpoena to provide requested data stored on servers regardless of whether the data are stored in the US or foreign soil.

With the CLOUD act, the legal protection of an individual's right to privacy depends on an objection from a provider (some clouds will more readily object than others). There's no direct mechanism for individuals to challenge an order under the CLOUD Act.

US courts can require production of data despite an objection, even where the laws of another nation would be violated.

Three can keep a secret if two of them are dead - Benjamin Franklin

House Bill 57

House Bill (HB) 57 was signed into law in Utah and states that:

A law enforcement agency may not obtain, without a search warrant issued by a court upon probably cause:

i. location, stored, or transmitted mobile data.

ii. data transmitted to a remote service provider.

Probable cause is key here and its remit of data covers our modern technology landscape, representing a great win for Utah and a big step in the right direction for data privacy.

So far it's only Utah and I've not ventured too far down the rabbit hole with how National Security Letters (reseach them, they're terrifying) would affect someone's privacy in the face of an objection via HB57.

Encryption-at-rest

I then asked the audience:

Who relies on cloud-provided encryption-at-rest to protect their customers’ data?

Around 5% of people had their hands up this time. Which surprised me, given encryption-at-rest is a core part of data security in the cloud.

Most people will get encryption-at-rest for free from their cloud providers. It ensures that any data stored within the cloud will be encrypted before being persisted and adds a level of security to your data in the event that it is stolen.

Unfortunately for your data (and those in my audience who weren't aware their data was being encrypted to start with), by default, encryption-at-rest will use a Data Encryption Key (DEK) that's generated by the cloud provider. If a government requests data, it'll be the data and the DEK that gets provided to them. Bye bye privacy!

Transparency reports

The 3 big cloud providers publish reports that detail the number of requests for data they've received from the goverment, along with the number of request they've responded to with customer data, or objected to:

What Lush are doing:

Keeping stored customer data to an absolute minimum - our user's privacy is a primary concern for us. Their right to privacy always trumps our need to store their data and it has done so for so long that it's part of our core work ethic.
Asking questions - we don't have all of the answers. This is a legal minefield and we're certainly not experts in navigating it! We've spoken with our cloud provider's technical and legal teams to learn more.
Considering domiciling sensitive data in countries with stricter data protection laws - as with cloud providers, some countries will more readily hand over your data when asked. This site details the countries whose data privacy laws will best protect your privacy and the privacy of your customers.
Looking at user-generated DEKs and KEKs (Key Encryption Keys, keys that are used to encrypt your DEKs, providing another level of security and an ability to change or "rotate" your keys without having to re-encrypt your data) - ensures that any data handed over to a government will be unintelligible. It takes the fight for our customer's privacy into our hands.

What can you do?

Don't store what you don't need - if you can give your customers a great experience without storing all of their information, don't store all of their information!
Understand the legal minefield - speak with your cloud provider's technical and legal teams if you have concerns; they're there to help.
Be picky - some cloud providers will more readily protect your privacy and the privacy of your customers, so consider this before deciding on your cloud partner.
Take the fight into your own hands - generate your own DEKs and KEKs to add another level of protection to your data.

Human cost

I opened up my last segment with the following question:

Who is familiar with 3TGs?

No hands up. Again, I would have had no idea what this was before researching for my talk.

It stands for the 4 registered conflict minerals; tin (cassiterite), tantalum (coltan), tungsten (wolframite), and gold. A conflict mineral is any mineral whose mining proceeds fund violent armed groups.

Mining

One of the hardest hit nations for mineral mining and the conflict their proceeds generate is The Democratic Republic of the Congo (DRC). In the DRC alone:

5.4m people have been killed since 1998 as a result of conflict funded by mineral mining.
4.5m people have been displaced and 800k people have had to seek refuge in other nations as a result of conflict.

A mind-blowing $24T worth of minerals are estimated to remain in the ground in the DRC, which shows how desperate the need for fair treatment and safe working conditions for the artisinal miners, along with a carefully controlled supply chain to prevent mining proceeds from getting into the wrong hands is.

Production

China has some of the largest technology assembly facilities in the world and as of the time of writing, workers in these facilities can be subject to appalling conditions and exploitation.

A law exists within China that stipulates that a given percentage of a workforce must be permanently employed (as opposed to temporary or "labour dispatch"). Large companies outside of China are turning a blind eye to massive violations in this law which are reducing job stability for people working in these facilities. The law states that no more than 10% of a workforce can be labour dispatch employees. In reality, they make up 50% of some workforces.

Another law exists within China to prevent people from being forced to work too many hours of overtime in any given month. This too is being completely vioated, with some workers having to work 4 hours in overtime every day to keep up with the expectations on them. Well over twice the legal limit. Not only are the overtime limits being violated but the law also states how overtime must be paid. Sadly, people are being forced to work overtime on weekdays and weekends without pay.

Following a spate of suicides in one of the world's largest assembly facilities, perhaps most famous and influential tech leader of the 21st century responded that the suicide rate in the facility was within the national average.

Should we as leaders in technology not be aspiring to give our people a safe and happy working environment by looking after their physical and mental wellbeing?

Post-production

Once a piece of hardware reaches end-of-life it becomes what's commonly referred to as e-waste and needs to be recycled.

Unfortunately, recycling isn't the fate for around 80% of e-waste, which is illegally traded or dumped. To put this into perspective, that's 50M tonnes, the area of Manhattan, the weight of 126 Empire State Buildings, or all of the commercial aircraft we've ever built, dumped. Every year. And it's worth a staggering $62B.

Worst of all, deadly amounts of mercury, lead, and beryllium are leaching into the air, soil, and groundwater in these e-waste dumps. Children are growing up being exposed to chemicals that will irriversibly affect their bodies. Agbobloshie in Ghana now ranks as the most polluted place in the world, beating even Chernobyl to the top spot.

Ghana's imports of this toxic waste are set to double in 2020.

Closing the loop

It's 13x cheaper to extract minerals from old hardware than it is to mine for new ones. And it generates 80% less emissions.

There are some amazing things happening to increase extraction efforts the world over but we're just scratching the surface:

In 2015, Apple recycled around 1 tonne of gold worth $40M and around 1,300 tonnes of copper worth $6M.
People all over Japan gave up their unused phones. The gold, silver, and bronze from these devices was enough to create the medals for the 2020 olympics. In total, around 30k of gold, over 3 tonnes of silver, and over 2 tonnes of bronze were extracted, worth a collective $3M.
Websites like e-Stewards list recyclers who are committed to never exporting hazardous materials to developing countries.

What Lush are doing

We're looking into building our own ethical tablet, considering the fair treatment and payment of everyone involved, be they producing the minerals or components, or assembling it. In doing so, we hope to:

Influence - we're sharing our experiences and trying to lead by example.
Create transparency - or as much as we're able to on the origins of materials and components, by visiting people in assembly facilities and e-waste dumps to see for ourselves.
Ask questions - we're asking the kinds of questions that people have never been asked, simply because these things have never been a priority for other companies before.
Partner up - the more people actively working on this, the better.

What can you do?

Cloud users - ask your provider what they're doing to to protect people across the whole production lifecycle. They're too wealthy to not be accountable for this.
Hardware producers - pay your people fairly and give them safe and happy conditions to work in and others will be forced to do the same. Understand your supply chain - don't rely on the goodwill of other companies. Go to see for yourself to understand the conditions that people of all ages are being forced to live and work in.
Hardware consumers - know where your products or components are coming from across the whole supply chain.
Close the loop - billions of dollars worth of minerals are dumped every year. There is more gold in 1 tonne of old phones than there is in 1 tonne of gold ore. Rather than mining for new minerals, we can recycle and reduce the amount of e-waste that gets dumped.

Closing

The Internet is growing. There are around 4.5B people online now, with around 1M new users every day. Collectively, we spend around 1.2B years online, that's 100 days per person per year.

Despite what some world leaders would have you believe, the world is warming and desperately needs help before we run out of time and completely destroy our planet.

Whether it's artisinal miners in the DRC, or people assembling hardware, people are truly suffering in ways that most of us can't imagine. And it's all because of our desire to grow the Internet and the businesses that hang from it.

It's not just about securing the cheapest price. We can't afford to blindly focus on cost while the decisions we make to extend our business reach damage the planet and the beings that inhabit it.

The is no higher-power shouldering our personal responsibility for the impact we'll have. Unless we all act now, this is on us.

It's our job to know and once we know, it's our responsibility to act. - Lush

Geo-partitioning. No column? No problem!

Rob Reid — Mon, 09 Dec 2019 16:26:31 +0000

CockroachDB's Geo-partitioning allows you to partition data using location information like country codes. If you're wanting to partition your data by-country, your data structure will likely already have the necessary components to do so. For example, the following table lends itself to being partitioned on its "country" column, meaning rows containing a country code of 'DE', 'FR', and 'UK' can be pinned to European data centres, while rows containing a country column of 'US' can be pinned to North American data centres etc:

CREATE TABLE "person" (
  id UUID NOT NULL DEFAULT uuid_v4()::UUID,
  country VARCHAR(2) NOT NULL,
  name STRING NOT NULL,
  CONSTRAINT "primary" PRIMARY KEY (country ASC, id ASC)
);

No column?

The data we have available to use might not always be in a structure that allows for straightforward partitioning. Suppose that instead of a top-level column, our country information was buried in a semi-structured JSON blob as follows:

{
    "data": {
        "country": "DE",
        "name": "Alice"
    }
}

Happily, CockroachDB allows you to include computed columns in your primary keys, meaning we can lift the "country" field out of our JSON structure and partition with that!

To spin up an example database to work with, I'll be making use of v19.2.0's updates to the demo command, which allow it to spin up an in-memory, multi-node, geo-partitioned cluster 🎉. The following command spins up an empty 6-node multi-region cluster, with 3 nodes on the east coast and 3 nodes in west Europe:

cockroach demo \
    --empty \
    --nodes 6 \
    --demo-locality=region=us-east1,az=1:region=us-east1,az=2:region=us-east1,az=3:region=europe-west1,az=1:region=europe-west1,az=2:region=europe-west1,az=3

The demo command spins up an enterprise cluster, meaning we can view the cluster in node map view to get a sense of where data is being stored.

No problem!

This time in our CREATE statement, we include a column that takes the country field out of the JSON data and creates a computed column from it:

CREATE TABLE "person" (
  id UUID NOT NULL DEFAULT uuid_v4()::UUID,
  data JSONB NOT NULL,
  country VARCHAR(2) AS ((data->'data'->>'country')::STRING) STORED,
  CONSTRAINT "primary" PRIMARY KEY (country ASC, id ASC)
);

What we're asking CockroachDB to do, is look for "data.country" in the JSON blob, lift it out as a string, create a standard column from it, and include that in the primary key. Our table from a partitioning perspective, is now equivalent to the first example.

To know we've successfully partitioned our table, let's insert some rows to observe where they're being stored once the partitions have taken effect:

INSERT INTO "person" (data) VALUES
    ('{"data": {"country": "DE", "name": "Alice"}}'),
    ('{"data": {"country": "FR", "name": "Bob"}}'),
    ('{"data": {"country": "UK", "name": "Carol"}}'),
    ('{"data": {"country": "US", "name": "Eve"}}');

Configure a partition to split the data into 2 regions, one for the US and one for Europe.

ALTER TABLE "person" PARTITION BY LIST ("country") (
    PARTITION us VALUES IN ('US'),
    PARTITION eu VALUES IN ('DE', 'FR', 'UK')
);

ALTER PARTITION us OF TABLE "person" CONFIGURE ZONE USING constraints = '[+region=us-east1]';
ALTER PARTITION eu OF TABLE "person" CONFIGURE ZONE USING constraints = '[+region=europe-west1]';

Use the following to view how data is being replicated. It'll show that your US data is being pinned to your us-east1 region, while your DE, FR, and UK is being pinned to your europe-west1 region:

SELECT start_key, replicas, replica_localities FROM [SHOW RANGES FROM TABLE person] WHERE end_key LIKE '%/PrefixEnd';

start_key	replicas	replica_localities
/"US"	{1,2,3}	{"region=us-east1,az=1","region=us-east1,az=2","region=us-east1,az=3"}
/"FR"	{4,5,6}	{"region=europe-west1,az=1","region=europe-west1,az=2","region=europe-west1,az=3"}
/"UK"	{4,5,6}	{"region=europe-west1,az=1","region=europe-west1,az=2","region=europe-west1,az=3"}
/"DE"	{4,5,6}	{"region=europe-west1,az=1","region=europe-west1,az=2","region=europe-west1,az=3"}

US data is pinned to region us-east1 (replicas 1, 2, 3), while DE, FR, and UK data is pinned to europe-west1 (replicas 4, 5, 6).

Note that this example assumes that the person table does not yet exist. If it were to exist, you wouldn't be able to update the table's primary key to include information from the JSON data without first creating a new table (with an updated primary key) and copying over the data as described here.

CockroachDB Demo Locations

Rob Reid — Mon, 09 Dec 2019 16:20:01 +0000

The cockroach demo command is my current favourite dev tool. It's instant access to an N-node CockroachDB cluster, allowing you to get up and running with an enterprise cluster straight from your dev environment, reducing feedback loops massively by giving you all the features you need to create with confidence.

If you start a cluster with cockroach demo, and don't provide any arguments for --demo-locality, you'll get the default values that are provided by the demo code.

--nodes	Regions
1	us-east1 x1
2	us-east1 x2
3	us-east1 x3
4	us-east1 x3 + us-west1 x1
5	us-east1 x3 + us-west1 x2
6	us-east1 x3 + us-west1 x3
7	us-east1 x3 + us-west1 x3 + europe-west1 x1
8	us-east1 x3 + us-west1 x3 + europe-west1 x2
9	us-east1 x3 + us-west1 x3 + europe-west1 x3

The increase is cyclical, with every additional node for the first 3 additional nodes, adding to the us-east1 region and so on.

Currently, only the us-east1, us-west1, and europe-west1 regions have locations configured, so if you'd like spin up nodes in other regions and have them represented in the cluster map, it's a very simple two-step process:

Step 1 - create your cluster with custom regions

The following spins up a 10-node cluster across 3 regions; us-central1 (4 AZs), asia-northeast1 (3 AZs), and europe-west1 (3 AZs):

cockroach demo \
--empty \
--nodes 10 \
--demo-locality=region=us-central1,az=a:region=us-central1,az=b:region=us-central1,az=c:region=us-central1,az=f:region=asia-northeast1,az=a:region=asia-northeast1,az=b:region=asia-northeast1,az=c:region=europe-west1,az=a:region=europe-west1,az=b:region=europe-west1,az=c

If you open the cluster map in the web UI now, you'll see a message that explains that not all of your regions have been assigned locations. In the knowledge that only a handful of regions come pre-loaded with location information, we can infer that the asia-northeast1 region needs to be given a location.

This can be confirmed by running the following SQL in your console session:

select * from system.locations;

localityKey	localityValue	latitude	longitude
region	europe-west1	50.448160000000000	3.818860000000000
region	us-central1	42.032974000000000	-93.581543000000000
region	us-east1	33.836082000000000	-81.163727000000000
region	us-east4	37.478397000000000	-76.453077000000000
region	us-west1	43.804133000000000	-120.554201000000000

Helpfully, the cluster map provides a link to docs that will help you assign locations to your regions. You can follow the docs, or continue to read to find out how this can be done.

Step 2 - assign localities to custom regions

Insert location information for the asia-northeast1 region by running the following SQL in your console session:

INSERT into system.locations VALUES ('region', 'asia-northeast1', 35.689487, 139.691706);

If you refresh your node map now, you'll see a 9-node cluster, with 3 regions configured in the locations you've configured.

CockroachDB Integration Testing

Rob Reid — Mon, 09 Dec 2019 16:13:46 +0000

Testing is vitally important to the success of software and software that connects to databases is no exception. There are a number of ways to test database code, ranging from mocks, to full integration tests that talk to an actual database.

When writing mocks, I've had some great experiences with Data Dog's sqlmock and I'd recommend that you check it out. In this post however, I'm going to devle into the steps involved in integration testing a CockroachDB database using docker-compose.

All code can be found in GitHub.

Test code

Firstly, let's go over the test code. It's super contrived but hopefully gets the point across. If anything's not clear, I'd be happy to discuss, send me a message on Twitter.

I use the TestMain method to bootstrap my tests, which allows me to capture command line arguments and gives my tests a logical place to setup and teardown database connections.

This post gives a lovely introduction to using Go's flags with your tests and I'll be using lessons from there to determine whether or not to run database integration tests in this example.

My simple test code uses two global variables (the horror!) and depending on your test directory structure, you might want to do this differently. One variable holds the flag value that will determine whether database integration tests are run and the other holds a reference to the database connection itself.

var (
    dbTests *bool
    db      *sql.DB
)

TestMain is an entry point for Go tests. In the following code you'll see that I'm capturing whether the caller wants to run database integration tests and setting up/tearing down the database connection if they've asked for database integration tests to be run.

func TestMain(m *testing.M) {
    dbTests = flag.Bool("db", false, "run database integration tests")
    flag.Parse()

    if *dbTests {
        setupDatabase()
    }

    exitCode := m.Run()

    if *dbTests {
        teardownDatabase()
    }

    os.Exit(exitCode)
}

The database setup code expects an environment variable called "CONN_STR" and crashes the test if it doesn't exist. Hence I wrap calls to setupDatabase in a check to *dbTests, allowing me to run unit tests that don't care about database tests (or connection strings from the environment) without having to provide this information.

The guts of setupDatabase is the database connectivity logic. A call is made to open a connection to the database and another is made to verify that the connection is open and ready:

func setupDatabase() {
    connStr, ok := os.LookupEnv("CONN_STR")
    if !ok {
        log.Fatal("connection string env var not found")
    }

    var err error
    if db, err = sql.Open("postgres", connStr); err != nil {
        log.Fatalf("error connecting to the database: %v", err)
    }

    if err = db.Ping(); err != nil {
        log.Fatalf("error pinging the database: %v", err)
    }
}

The teardownDatabase function could simply be replaced with a call to db.Close() but I preferred the consistency in the TestMain function, so decided to keep it:

func teardownDatabase() {
    db.Close()
}

Finally, I've written an integration test. It currently does nothing useful, further to logging what it's doing (running database integration tests or not):

func TestStuff(t *testing.T) {
    if !*dbTests {
        t.Skipf("not running database tests")
    }

    t.Log("write some database tests...")
}

Containerise

To be able to run our database integration tests reliably and get repeatable results, we're going to need to run 2 Docker containers, one to host an instance of CockroachDB and another to execute our tests.

Dockerfile

To start with, let's create a Dockerfile to run our tests. The below spins up a very lightweight Alpine-based container, copies our project's top-level directory into an "app" directory inside the container, and sets the working directory for the container to that directory:

FROM alpine:latest
COPY . /app
WORKDIR /app

Build

As we're using the Alpine base image, we won't have the means to run Go code (or Go tests). We'll therefore build the tests so they can simply be executed within the container, just like any other executable.

The following builds a test executable called "crdb-test" for the Alpine Linux container that when executed, will be the same as running go test ./... -v, without having to rely on the Go executable:

$ GOOS=linux go test ./... -v -c -o crdb-test

Image

Next, we need to build the Docker image. The following builds a docker image called "crdb-test", using the Dockerfile we created earlier:

$ docker build -t crdb-test .
Sending build context to Docker daemon   6.18MB
Step 1/3 : FROM alpine:latest
 ---> 5cb3aa00f899
Step 2/3 : COPY . /app
 ---> Using cache
 ---> 936e88998fa5
Step 3/3 : WORKDIR /app
 ---> Using cache
 ---> 30e76628fce3
Successfully built 30e76628fce3
Successfully tagged crdb-test:latest

At this point, you can run the Docker image and hop into it via the terminal. The following command will open up a terminal and put you straight into the container's working directory, allowing you to poke around and check that your "crdb-test" test binary is where you're expecting it to be:

$ docker run -it crdb-test:latest sh

Compose

To tie our test container and the CockroachDB container together, we'll use docker-compose. Here's the full file, which I'll break into smaller parts and describe below:

version: '3'

services:
  cockroach_test:
    image: cockroachdb/cockroach:v2.1.6
    command: start --insecure
    ports:
      - "26257:26257"
      - "8080:8080"
    volumes:
      - ./cockroach-data/roach1:/cockroach/cockroach-data
    logging:
      driver: "none"
    networks:
    - app-network

  app_test:
    image: crdb-test:latest
    environment:
      CONN_STR: postgresql://root@cockroach_test:26257/defaultdb?sslmode=disable
    command: ./crdb-test -db
    working_dir: /app
    depends_on:
    - cockroach_test
    networks:
    - app-network

networks:
  app-network:
    driver: bridge

I've named the first container (or "service") "cockroach_test". This will become the DNS name that I'll use to connect my test container to.

cockroach_test:
  image: cockroachdb/cockroach:v19.2.0
  command: start --insecure
  logging:
    driver: "none"
  networks:
  - app-network

For the image parameter, I pass the current latest version of the CockroachDB Docker image. You can substitute another version of the database for your tests.

For the command parameter, I pass start --insecure. These are the command line arguments that you'd pass to the cockroach CLI if running outside of Docker.

For the logging parameter, I pass an argument that disables logging for the CockroachDB container. This is entirely optional and just means the logging output is more test-focused. Helpful if you're trawling through test logs to figure out why a test is unexpectedly failing!

I've also configured a bridge network for the continers to talk to each other, so for the networks parameter, I pass "app-network", the name of that network.

I spin up a container to execute my test code and refer to it as "app_test".

app_test:
  image: crdb-test:latest
  environment:
    CONN_STR: postgresql://root@cockroach_test:26257/defaultdb?sslmode=disable
  command: ./crdb-test -db
  working_dir: /app
  depends_on:
  - cockroach_test
  networks:
  - app-network

For the image parameter, I refer back to the image I built earlier, "crdb-test", suffixing ":latest", the default version of a freshly minted local Docker image.

For the environment parameter, I pass "CONN_STR", the full connection string to the CockroachDB database. Note that instead of "localhost", I'm passing "cockroach_test", the name of the CockroachDB docker-compose service.

For the command parameter, I pass "./crdb-test -db". This tells docker-compose what to run from within the test container's working directory. The "-db" flag tells our test executable to run database integration tests.

For the depends_on parameter, I pass "cockroach_test", telling docker-compose that we want to wait until CockroachDB is up before trying to execute our tests.

Finally, to run everything, we call the docker-compose CLI passing the name of our docker-compose file ("crdb-test.yaml" in this case) and a few additional flags you might find helpful, including --abort-on-container-exit, which tears down all containers when one of them exists, cleaning everything up once our tests are run and passing our exit code back to the caller.

$ docker-compose --no-ansi -f crdb-test.yaml up --abort-on-container-exit --force-recreate
Recreating cockroachdb-testing_cockroach_test_1 ... 
Recreating cockroachdb-testing_cockroach_test_1 ... done
Recreating cockroachdb-testing_app_test_1       ... 
Recreating cockroachdb-testing_app_test_1       ... done
Attaching to cockroachdb-testing_cockroach_test_1, cockroachdb-testing_app_test_1
cockroach_test_1  | WARNING: no logs are available with the 'none' log driver
app_test_1        | run database tests true
app_test_1        | PASS
cockroachdb-testing_app_test_1 exited with code 0
Aborting on container exit...
Stopping cockroachdb-testing_cockroach_test_1   ... 
Stopping cockroachdb-testing_cockroach_test_1   ... done