In this blog I will demonstrate how you can trick an application or service, in our case CockroachDB, into thinking it's communicating with AWS S3 when in fact another service lays behind the scenes.
CockroachDB is perfectly capable to interact with any S3 compatible services. Following the instructions of the AWS SDK, you can add parameter AWS_ENDPOINT
to your URI to tell CockroachDB what server to hit instead of the default AWS servers. This is clearly documented here.
However, this blog comes as a follow up from a customer interaction where their requirement was to NOT use AWS_ENDPOINT
in their URI, yet still using their own private S3 compatible service.
Setup
Create a t2.micro or similar instance on your favorite cloud provider.
For this exercise, the instance has the below details:
OS : Ubuntu 20.04 LTS
Public IP : 54.83.105.95
Private IP : 10.10.81.189
Install and Configure MinIO
MinIO is a S3 compatible service and it is my favorite choice for when I want to play locally with the S3 API. We will therefore use the MinIO Server as our custom endpoint instead of AWS S3.
Below the steps to run MinIO, also available in the official docs.
# download the binary
wget https://dl.min.io/server/minio/release/linux-amd64/minio
chmod +x minio
./minio server data
At this point MinIO is running in insecure mode.
It will also have created the configuration folders at ${HOME}/.minio/certs
.
Stop the server with Ctrl+C
, then configure secure access with TLS.
# allow minio to start on port 443
sudo setcap cap_net_bind_service=+ep minio
# create the private key
cd .minio/certs
openssl genrsa -out private.key 2048
chmod 400 private.key
Create a file named openssl.conf
with the content below. Check the alt_names
section carefully: in this context, fabio
is the name of the s3 bucket.
[req]
distinguished_name = req_distinguished_name
x509_extensions = v3_req
prompt = no
[req_distinguished_name]
C = US
ST = VA
L = Somewhere
O = MyOrg
OU = MyOU
CN = MyServerName
[v3_req]
subjectAltName = @alt_names
[alt_names]
IP.1 = 127.0.0.1
IP.2 = 54.83.105.95
IP.3 = 10.10.81.189
DNS.1 = localhost
DNS.2 = s3.amazonaws.com
DNS.3 = fabio.s3.amazonaws.com
Create the public cert
openssl req -new -x509 -nodes -days 730 -key private.key -out public.crt -config openssl.conf
Setup MinIO Server to use Virtual Hosted-Style, as per here.
This is required to match AWS S3 style.
export MINIO_DOMAIN=s3.amazonaws.com
Finally, start the MinIO Server
$ ~/minio server --console-address ":9001" --address ":443" ~/data
API: https://10.10.82.1 https://127.0.0.1
RootUser: minioadmin
RootPass: minioadmin
Console: https://10.10.82.1:9001 https://127.0.0.1:9001
RootUser: minioadmin
RootPass: minioadmin
Command-line: https://docs.min.io/docs/minio-client-quickstart-guide
$ mc alias set myminio https://10.10.82.1 minioadmin minioadmin
Documentation: https://docs.min.io
WARNING: Detected default credentials 'minioadmin:minioadmin', we recommend that you change these values with 'MINIO_ROOT_USER' and 'MINIO_ROOT_PASSWORD' environment variables
Point your browser at https://54.83.105.95:9001 to access the MinIO Console. As the self-signed certificate is invalid, the browser might not allow you to continue. In Brave Browser, and other Chrome based browsers, type thisisunsafe
anywhere on the screen (not on the address toolbar) to continue and disregard the warning.
Login with minioadmin/minioadmin
, which are the default login details. Once logged in, you'll see the Dashboard page, below
Using the menu on the left, create a bucket called fabio
.
Then create a User fabio
with any password you want. Make sure assign the consoleAdmin
, readwrite
and diagnostic
policies to the user, so you don't run into any permission issue. Later on, you can refine these permissions.
Then, click on the user you just created and create a Service Account, and make sure you save the keys. In my example, the keys are:
Access Key: I6SX7TO71RZY79FXDK1T
Secret Key: 7TxSqDyrvOXCV+EWkRvZKSOXETriFik5LKNWmkQm
Finally, go to Settings and change the region to us-east-1
.
This will prompt you to RESTART the server. Do so by stopping the server, and restarting it.
Perfect, you're all set! MinIO Server is up and running in secure mode, and you've a service account to interact with it.
Next, we configure the OS.
Linux OS configuration
For CockroachDB to trust the certificate, we must add it to the system cert pool. We also need to redirect the traffic aimed at AWS servers to MinIO's.
Allow self-signed certs
In Ubuntu, this is how you go about, as per here
sudo mkdir /usr/local/share/ca-certificates/extra
sudo cp ~/.minio/certs/public.crt /usr/local/share/ca-certificates/extra/
sudo update-ca-certificates
Routing
Edit file /etc/hosts
and add these entries to route all traffic directed to Amazon AWS servers to the MinIO Server instead.
Again, fabio
is the name of the bucket.
54.83.105.95 s3.amazonaws.com
54.83.105.95 s3.us-east-1.amazonaws.com
54.83.105.95 fabio.s3.amazonaws.com
Good, we're all set from OS point of view. Next, we test with CockroachDB.
CockroachDB
Install and start CockroachDB. We will use the single-node feature as we're only trying to prove it works from a functionality perspective.
As always, the offical installation docs are here.
curl https://binaries.cockroachdb.com/cockroach-v21.1.10.linux-amd64.tgz | tar -xz && sudo cp -i cockroach-v21.1.10.linux-amd64/cockroach /usr/local/bin/
cockroach start-single-node --insecure --background
Connect to the SQL prompt as root:
cockroach sql --insecure
Create a user - doesn't have to be an admin -, then logout and log back in as that user
CREATE USER fabio WITH CREATEDB;
cockroach sql --insecure -u fabio
As user fabio
, create a simple test table and export it. Make sure you update the keys with your values!
CREATE DATABASE fabio;
USE fabio;
CREATE TABLE test AS
SELECT a
FROM generate_series(1,100) AS a;
-- notice that we don't use AWS_ENDPOINT in the URI
EXPORT INTO CSV
's3://fabio?AWS_ACCESS_KEY_ID=I6SX7TO71RZY79FXDK1T&AWS_SECRET_ACCESS_KEY=7TxSqDyrvOXCV+EWkRvZKSOXETriFik5LKNWmkQm'
FROM TABLE test;
filename | rows | bytes
-------------------------------------------------------------------+------+--------
export16ad4c7f75f4fe180000000000000001-n701156393427501057.0.csv | 100 | 292
(1 row)
Time: 38ms total (execution 38ms / network 0ms)
As root
, take a full cluster backup
BACKUP INTO
's3://fabio?AWS_ACCESS_KEY_ID=I6SX7TO71RZY79FXDK1T&AWS_SECRET_ACCESS_KEY=7TxSqDyrvOXCV+EWkRvZKSOXETriFik5LKNWmkQm';
job_id | status | fraction_completed | rows | index_entries | bytes
---------------------+-----------+--------------------+------+---------------+--------
701160820164395009 | succeeded | 1 | 126 | 14 | 5920
(1 row)
Refresh the MinIO Console to see your files
Awesome! The CockroachDB end-user has no clue that the engine behind the scenes is MinIO instead of AWS, and CockroachDB is happy to IMPORT, EXPORT, BACKUP and RESTORE from S3, be it AWS' or Minio's.
Top comments (0)