DEV Community

Cover image for Bypass AWS_ENDPOINT to interact with any S3 compatible service from CockroachDB
Cockroach Labs profile image Fabio Ghirardello
Fabio Ghirardello for Cockroach Labs

Posted on

Bypass AWS_ENDPOINT to interact with any S3 compatible service from CockroachDB

#cockroachdb #minio #s3 #tutorial

In this blog I will demonstrate how you can trick an application or service, in our case CockroachDB, into thinking it's communicating with AWS S3 when in fact another service lays behind the scenes.

CockroachDB is perfectly capable to interact with any S3 compatible services. Following the instructions of the AWS SDK, you can add parameter AWS_ENDPOINT to your URI to tell CockroachDB what server to hit instead of the default AWS servers. This is clearly documented here.

However, this blog comes as a follow up from a customer interaction where their requirement was to NOT use AWS_ENDPOINT in their URI, yet still using their own private S3 compatible service.

Setup

Create a t2.micro or similar instance on your favorite cloud provider.
For this exercise, the instance has the below details:

OS         : Ubuntu 20.04 LTS
Public IP  : 54.83.105.95
Private IP : 10.10.81.189
Enter fullscreen mode Exit fullscreen mode

Install and Configure MinIO

MinIO is a S3 compatible service and it is my favorite choice for when I want to play locally with the S3 API. We will therefore use the MinIO Server as our custom endpoint instead of AWS S3.

Below the steps to run MinIO, also available in the official docs.

# download the binary
wget https://dl.min.io/server/minio/release/linux-amd64/minio
chmod +x minio
./minio server data
Enter fullscreen mode Exit fullscreen mode

At this point MinIO is running in insecure mode.
It will also have created the configuration folders at ${HOME}/.minio/certs.
Stop the server with Ctrl+C, then configure secure access with TLS.

# allow minio to start on port 443
sudo setcap cap_net_bind_service=+ep minio

# create the private key
cd .minio/certs
openssl genrsa -out private.key 2048
chmod 400 private.key
Enter fullscreen mode Exit fullscreen mode

Create a file named openssl.conf with the content below. Check the alt_names section carefully: in this context, fabio is the name of the s3 bucket.

[req]
distinguished_name = req_distinguished_name
x509_extensions = v3_req
prompt = no

[req_distinguished_name]
C = US
ST = VA
L = Somewhere
O = MyOrg
OU = MyOU
CN = MyServerName

[v3_req]
subjectAltName = @alt_names

[alt_names]
IP.1 = 127.0.0.1
IP.2 = 54.83.105.95
IP.3 = 10.10.81.189
DNS.1 = localhost
DNS.2 = s3.amazonaws.com
DNS.3 = fabio.s3.amazonaws.com
Enter fullscreen mode Exit fullscreen mode

Create the public cert

openssl req -new -x509 -nodes -days 730 -key private.key -out public.crt -config openssl.conf
Enter fullscreen mode Exit fullscreen mode

Setup MinIO Server to use Virtual Hosted-Style, as per here.
This is required to match AWS S3 style.

export MINIO_DOMAIN=s3.amazonaws.com
Enter fullscreen mode Exit fullscreen mode

Finally, start the MinIO Server

$ ~/minio server --console-address ":9001" --address ":443" ~/data

API: https://10.10.82.1  https://127.0.0.1     
RootUser: minioadmin 
RootPass: minioadmin 

Console: https://10.10.82.1:9001 https://127.0.0.1:9001   
RootUser: minioadmin 
RootPass: minioadmin 

Command-line: https://docs.min.io/docs/minio-client-quickstart-guide
   $ mc alias set myminio https://10.10.82.1 minioadmin minioadmin

Documentation: https://docs.min.io
WARNING: Detected default credentials 'minioadmin:minioadmin', we recommend that you change these values with 'MINIO_ROOT_USER' and 'MINIO_ROOT_PASSWORD' environment variables
Enter fullscreen mode Exit fullscreen mode

Point your browser at https://54.83.105.95:9001 to access the MinIO Console. As the self-signed certificate is invalid, the browser might not allow you to continue. In Brave Browser, and other Chrome based browsers, type thisisunsafe anywhere on the screen (not on the address toolbar) to continue and disregard the warning.

Login with minioadmin/minioadmin, which are the default login details. Once logged in, you'll see the Dashboard page, below

minio-dashboard

Using the menu on the left, create a bucket called fabio.

bucket

Then create a User fabio with any password you want. Make sure assign the consoleAdmin, readwrite and diagnostic policies to the user, so you don't run into any permission issue. Later on, you can refine these permissions.

Alt Text

Then, click on the user you just created and create a Service Account, and make sure you save the keys. In my example, the keys are:

Access Key: I6SX7TO71RZY79FXDK1T
Secret Key: 7TxSqDyrvOXCV+EWkRvZKSOXETriFik5LKNWmkQm
Enter fullscreen mode Exit fullscreen mode

Alt Text

Finally, go to Settings and change the region to us-east-1.

Alt Text

This will prompt you to RESTART the server. Do so by stopping the server, and restarting it.

Perfect, you're all set! MinIO Server is up and running in secure mode, and you've a service account to interact with it.

Next, we configure the OS.

Linux OS configuration

For CockroachDB to trust the certificate, we must add it to the system cert pool. We also need to redirect the traffic aimed at AWS servers to MinIO's.

Allow self-signed certs

In Ubuntu, this is how you go about, as per here

sudo mkdir /usr/local/share/ca-certificates/extra
sudo cp ~/.minio/certs/public.crt /usr/local/share/ca-certificates/extra/
sudo update-ca-certificates
Enter fullscreen mode Exit fullscreen mode

Routing

Edit file /etc/hosts and add these entries to route all traffic directed to Amazon AWS servers to the MinIO Server instead.
Again, fabio is the name of the bucket.

54.83.105.95 s3.amazonaws.com
54.83.105.95 s3.us-east-1.amazonaws.com
54.83.105.95 fabio.s3.amazonaws.com
Enter fullscreen mode Exit fullscreen mode

Good, we're all set from OS point of view. Next, we test with CockroachDB.

CockroachDB

Install and start CockroachDB. We will use the single-node feature as we're only trying to prove it works from a functionality perspective.

As always, the offical installation docs are here.

curl https://binaries.cockroachdb.com/cockroach-v21.1.10.linux-amd64.tgz | tar -xz && sudo cp -i cockroach-v21.1.10.linux-amd64/cockroach /usr/local/bin/
cockroach start-single-node --insecure --background
Enter fullscreen mode Exit fullscreen mode

Connect to the SQL prompt as root:

cockroach sql --insecure
Enter fullscreen mode Exit fullscreen mode

Create a user - doesn't have to be an admin -, then logout and log back in as that user

CREATE USER fabio WITH CREATEDB;
Enter fullscreen mode Exit fullscreen mode 
cockroach sql --insecure -u fabio
Enter fullscreen mode Exit fullscreen mode

As user fabio, create a simple test table and export it. Make sure you update the keys with your values!

CREATE DATABASE fabio;
USE fabio;

CREATE TABLE test AS 
  SELECT a 
  FROM generate_series(1,100) AS a;

-- notice that we don't use AWS_ENDPOINT in the URI
EXPORT INTO CSV     
  's3://fabio?AWS_ACCESS_KEY_ID=I6SX7TO71RZY79FXDK1T&AWS_SECRET_ACCESS_KEY=7TxSqDyrvOXCV+EWkRvZKSOXETriFik5LKNWmkQm'
  FROM TABLE test;
Enter fullscreen mode Exit fullscreen mode 
                              filename                             | rows | bytes
-------------------------------------------------------------------+------+--------
  export16ad4c7f75f4fe180000000000000001-n701156393427501057.0.csv |  100 |   292
(1 row)

Time: 38ms total (execution 38ms / network 0ms)
Enter fullscreen mode Exit fullscreen mode

As root, take a full cluster backup

BACKUP INTO
  's3://fabio?AWS_ACCESS_KEY_ID=I6SX7TO71RZY79FXDK1T&AWS_SECRET_ACCESS_KEY=7TxSqDyrvOXCV+EWkRvZKSOXETriFik5LKNWmkQm';
Enter fullscreen mode Exit fullscreen mode 
        job_id       |  status   | fraction_completed | rows | index_entries | bytes
---------------------+-----------+--------------------+------+---------------+--------
  701160820164395009 | succeeded |                  1 |  126 |            14 |  5920
(1 row)
Enter fullscreen mode Exit fullscreen mode

Refresh the MinIO Console to see your files

Alt Text

Awesome! The CockroachDB end-user has no clue that the engine behind the scenes is MinIO instead of AWS, and CockroachDB is happy to IMPORT, EXPORT, BACKUP and RESTORE from S3, be it AWS' or Minio's.

Reference

Top comments (0)

Read next

techpulse55 profile image

Unlocking Tube Magic AI’s Potential: Features, Pricing, and Performance Review

Techpulse -

techpulse55 profile image

Unlocking Quickpix AI's Potential: Features, Pricing, and Performance Review

Techpulse -

piyush_desai_bddc94d674ad profile image

Amazon S3 Increases Bucket Limit to 10,000 — Scalability Just Got Easier!

piyush desai -

i_am_vesh profile image

Multi-Node Kubernetes Cluster Setup with KIND

Avesh -