DEV Community: Darek Dwornikowski ☁

Django with cloud sql auth proxy locally and in Cloud Run

Darek Dwornikowski ☁ — Tue, 20 Sep 2022 07:40:12 +0000

Google's Cloud SQL Auth Proxy [3] allows you to set up a connection to a GCP SQL server from wherever you are via a secured tunnel. You can then point your application to a local address SQL proxy listens on and use it like local db. At least the DB "feel" like local.

The same proxy is used in many GCP services like Google Cloud Run and GKE (if you configure it as a sidecar).

Running SQL proxy is pretty easy (example is for pre 2.0.0 version):

./cloud_sql_proxy -instances=my-gcp-project:europe-north1:mydbinstancename=tcp:127.0.0.1:20000

Where

my-gcp-project:europe-north1:mydbinstancename is the "connection" name of your managed SQL server.
tcp: directive tells the proxy where to listen locally

Now the trick is that when you run your proxy on your dev machine, or in a Docker compose for local development, you are fine with TCP. Yet, when you use Cloud Run, it expects you to connect to a UNIX socket, typically residing in this path:

/cloudsql/my-gcp-project:europe-north1:mydbinstancename/.s.PGSQL.5432

This is pretty different than our TCP connection - would that mean we have to have a sepoarate Django config for local development and deployment? Not necessarily.

If you look at a typical Django (postgres in this case) settings, you will notice it seems like it is expecting TCP connections. However, by convention when HOST starts with / the engine will treat it as a UNIX socket. One thing you need to remember is to set the PORT to empty value.

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.postgresql_psycopg2',
        'NAME': os.environ.get('DB_NAME', 'some db '),
        'USER': os.environ.get('DB_USER', 'my user'),
        'PASSWORD': os.environ.get('DB_PASS'),
        'HOST': os.environ.get('DB_HOST', '127.0.0.1'),
        'PORT': os.environ.get('DB_PORT', '20000'),
    },
}

In Cloud Run what you need to do is to set an enviromental variable DB_HOST to /cloudsql/my-gcp-project:europe-north1:mydbinstancename

This is it. Now locally you use SQL connection with TCP and with UNIX on Cloud Run. All just by changing env variables just as the Twelve Factor App [1] pattern teaches us to:)

Using go modules from private repositories in Azure DevOps Pipelines

Darek Dwornikowski ☁ — Wed, 11 Dec 2019 09:52:01 +0000

This post will explain how to use go modules that you keep in private repositories in GitHub. Sometimes you have internal modules that you do not really want to expose to the open source community. There might be several reasons for it, for example you are still working on the solution and it is not ready to see public, or it is maybe a protected intellectual property. You keep your code in a private repo then and locally go get uses your ssh keys to access the repo and download the package to the go mod cache.

However, your CI/CD tooling like Azure DevOps (ADO) do not have access to the these private repositories immediately. For that it needs to be equipped with an SSH key that it then can use to access github. This post shows how you can configure it end to end.

In TL;DC what we will do is:

generate an ssh key pair
add a public key to the github repo
upload private key to the Azure DevOps secure files
configure the Azure DevOps pipeline via YAML
have fun doing it

Generate private key pair

This is a simple step, let's generate a key pair to be used to authenticate ADO to GitHub.

ssh-keygen -t rsa -b 4096 -C "your@email.com"

When asked, save the key pair under mykey name. You will have mykey which stores the private key and mykey.pub with a public key.

Add public key to the github repo

Assuming your repo is called my-go-module, navigate to
https://github.com/{your_org}/my-go-module/settings/keysand upload the contents of mykey.pub to the deploy keys.

Upload private key to the Azure DevOps secure files

Now you need to upload the private key contents to the ADO secure files. You can find them in the Pipelines -> Library -> Secure files. Upload the mykey private file there and call it myPrivateKey.

Configure the Azure DevOps pipeline via YAML

Now we have all the things in place. Last thing to do is install the SSH key in the pipeline so that go get can use it to access github. For that we will use the InstallSSHKey task.

      - task: InstallSSHKey@0
        inputs:
          knownHostsEntry: <here you put github known host entry>
          sshPublicKey: <here you put your public key content> 
          sshKeySecureFile: myPrivateKey

First run this command and copy the line not starting with #. Paste it into the knownHostsEntry parameter. This will make sure git will not ask for adding github into the known_hosts file but it would be already there.

➜ ssh-keyscan  github.com
# github.com:22 SSH-2.0-babeld-95694f5e
# github.com:22 SSH-2.0-babeld-95694f5e
github.com ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAq2A7hRGmdnm9tUDbO9IDSwBK6TbQa+PXYPCPy6rbTrTtw7PHkccKrpp0yVhp5HdEIcKr6pLlVDBfOLX9QUsyCOV0wzfjIJNlGEYsdlLJizHhbn2mUjvSAHQqZETYP81eFzLQNnPHt4EVVUh7VfDESU84KezmD5QlWpXLmvU31/yMf+Se8xhHTvKSCZIFImWwoG6mbUoWf9nzpIoaSjB+weqqUUmpaaasXVal72J+UX2B+2RPW3RcT0eOzQgqlJL3RKrTJvdsjE3JEAvGq3lGHSZXy28G3skua2SmVi/w4yCE6gbODqnTWlg7+wC604ydGXA8VJiS5ap43JXiUFFAaQ==
# github.com:22 SSH-2.0-babeld-95694f5e

Now copy contents of the file mykey.pub to sshPublicKey parameter and finally set sshKeySecureFile to the secure file name you have chosen (like myPrivateKey).

Now this task will configure access to the private repository. You need still do to one thing before you can download the module.

      - script: |
          git config --global url."git@github.com:{yourorg}/my-go-module".insteadOf "https://github.com/{yourorg}/my-go-module"
          go build
        displayName: 'Build the binaries'

This step is needed so that go get tries to access the module with ssh not with https by default. Substitute {yourorg} with your organization name or your nickname so that it matches the URI of your module.

Resizing persistent volumes in EKS/AKS/GKE

Darek Dwornikowski ☁ — Sun, 25 Aug 2019 16:36:00 +0000

Persistent volumes (PVs) in Kubernetes is the way to abstract out block volumes used by pods in the cluster. The way pods request PVs is by creating Persistent Volume Claims (PVCs), which work like resource demand documents. Storage Class is responsible for actual implementation of how the PVs are actually handled, it can be provided by ceph volumes, or in the case of cloud providers, by EBS volumes in AWS, Managed Disks in Azure and Disks in GCP.

This post is a note made to myself, when I was checking how does the PV resize works in managed kuberentes services. Feel free to replicate the commands and see what happens by yourself.

Amazon Elastic Kubernetes Service

Let's first create the cluster fast with eksctl. Mind it takes a significant time.

eksctl create cluster -n darek -r eu-central-1

After a while you should see some nodes running.

➜  ~ kubectl get nodes 
NAME                                              STATUS   ROLES    AGE   VERSION
ip-192-168-26-190.eu-central-1.compute.internal   Ready    <none>   67s   v1.13.7-eks-c57ff8
ip-192-168-55-249.eu-central-1.compute.internal   Ready    <none>   67s   v1.13.7-eks-c57ff8

Let's see the storage class.

➜  ~ kubectl get sc gp2 -o yaml 
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"storage.k8s.io/v1","kind":"StorageClass","metadata":{"annotations":{"storageclass.kubernetes.io/is-default-class":"true"},"name":"gp2"},"parameters":{"fsType":"ext4","type":"gp2"},"provisioner":"kubernetes.io/aws-ebs"}
    storageclass.kubernetes.io/is-default-class: "true"
  creationTimestamp: "2019-08-24T09:23:35Z"
  name: gp2
  resourceVersion: "281"
  selfLink: /apis/storage.k8s.io/v1/storageclasses/gp2
  uid: d903b89e-c650-11e9-968c-0273bdd9611a
parameters:
  fsType: ext4
  type: gp2
provisioner: kubernetes.io/aws-ebs
reclaimPolicy: Delete
volumeBindingMode: Immediate

To resize volumes, storage class needs to have the allowVolumeExpansion set to true. Fortunatelly, it can be patched.

k patch sc gp2 -p '{"allowVolumeExpansion": true}'

Now, create a basic PVC of size 10Gi and storageclass set to gp2. We will us the same yaml for AKS and GKE later too, save it as pvc.yml.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: test-volume-pvc
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
  storageClassName: gp2

Create the PVC.

kubectl apply -f pvc.yml

The PVC should eventually create a PV of size 10, which is reflected in a real EBS volume created in the AWS. The name of the EBS is encoded in Source/VolumeID. You can get it by doing describe operation.

➜  kubectl describe pv/pvc-0262b546-c65a-11e9-8f74-06e2726fa54c
Name:              pvc-0262b546-c65a-11e9-8f74-06e2726fa54c
Labels:            failure-domain.beta.kubernetes.io/region=eu-central-1
                   failure-domain.beta.kubernetes.io/zone=eu-central-1b
Annotations:       kubernetes.io/createdby: aws-ebs-dynamic-provisioner
                   pv.kubernetes.io/bound-by-controller: yes
                   pv.kubernetes.io/provisioned-by: kubernetes.io/aws-ebs
Finalizers:        [kubernetes.io/pv-protection]
StorageClass:      gp2
Status:            Bound
Claim:             default/test-volume-pvc
Reclaim Policy:    Delete
Access Modes:      RWO
VolumeMode:        Filesystem
Capacity:          10Gi
Node Affinity:
  Required Terms:
    Term 0:        failure-domain.beta.kubernetes.io/zone in [eu-central-1b]
                   failure-domain.beta.kubernetes.io/region in [eu-central-1]
Message:
Source:
    Type:       AWSElasticBlockStore (a Persistent Disk resource in AWS)
    VolumeID:   aws://eu-central-1b/vol-056a0bc0115292add
    FSType:     ext4
    Partition:  0
    ReadOnly:   false
Events:         <none>

The above command shows the EBS volume name to be vol-056a0bc0115292add.
Now let's resize the PVC. Change the storage size in pvc.yml to 12Gi and do kubectl apply again.

➜  kubectl get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                     STORAGECLASS   REASON   AGE
pvc-0262b546-c65a-11e9-8f74-06e2726fa54c   12Gi       RWO            Delete           Bound    default/test-volume-pvc   gp2                     19h

Ok so the PV shows 12Gi. Let's check the size of the actual EBS volume with aws cli.

aws ec2 describe-volumes --volume-ids vol-056a0bc0115292add --query 'Volumes[0].Size'

It should now show 12.

Have in mind that in Amazon you can do resize operation on EBS only once every 6h. So next resize won't work, unless you wait for it.

You can delete the clutser by doing eksctl delete cluster -n darek.

Azure AKS

Let's now check AKS. Create a cluster in AKS. The below snippet creates a resource group darek and a cluster called darekEKS.

az group create --name darek --location westeurope
az aks create --resource-group darek --name darekEKS --node-count 2 --enable-addons monitoring --generate-ssh-keys
az aks get-credentials --resource-group darek --name darekEKS

Storage class in AKS is called default, it will provision azure managed disk.

➜  kubectl get sc
NAME                PROVISIONER                AGE
default (default)   kubernetes.io/azure-disk   13m
managed-premium     kubernetes.io/azure-disk   13m

Let's see the whole class, similarly as in EKS the resize is not enbled.

➜  kubectl get storageclasses.storage.k8s.io default -o yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"storage.k8s.io/v1beta1","kind":"StorageClass","metadata":{"annotations":{"storageclass.beta.kubernetes.io/is-default-class":"true"},"labels":{"kubernetes.io/cluster-service":"true"},"name":"default","namespace":""},"parameters":{"cachingmode":"ReadOnly","kind":"Managed","storageaccounttype":"Standard_LRS"},"provisioner":"kubernetes.io/azure-disk"}
    storageclass.beta.kubernetes.io/is-default-class: "true"
  creationTimestamp: "2019-08-24T09:31:20Z"
  labels:
    kubernetes.io/cluster-service: "true"
  name: default
  resourceVersion: "459"
  selfLink: /apis/storage.k8s.io/v1/storageclasses/default
  uid: ee21d5ae-c651-11e9-9f0b-aac03b918570
parameters:
  cachingmode: ReadOnly
  kind: Managed
  storageaccounttype: Standard_LRS
provisioner: kubernetes.io/azure-disk
reclaimPolicy: Delete
volumeBindingMode: Immediate

Let's patch it: kubectl patch sc default -p '{"allowVolumeExpansion": true}'
And create the PVC, remember to change the storageClassName to default.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: test-volume-pvc
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
  storageClassName: default

Is PVC and PV created?

➜  kubectl get pvc
NAME              STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
test-volume-pvc   Bound    pvc-eb332309-c654-11e9-a0fd-36761452f1dd   10Gi       RWO            default        26s

➜  kubectl get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                     STORAGECLASS   REASON   AGE
pvc-eb332309-c654-11e9-a0fd-36761452f1dd   10Gi       RWO            Delete           Bound    default/test-volume-pvc   default                 41s

The name of the Azure disk will be seen in Source/DiskURI.

➜ ~ kubectl describe pv
Name:            pvc-eb332309-c654-11e9-a0fd-36761452f1dd
Labels:          <none>
Annotations:     pv.kubernetes.io/bound-by-controller: yes
                 pv.kubernetes.io/provisioned-by: kubernetes.io/azure-disk
                 volumehelper.VolumeDynamicallyCreatedByKey: azure-disk-dynamic-provisioner
Finalizers:      [kubernetes.io/pv-protection]
StorageClass:    default
Status:          Bound
Claim:           default/test-volume-pvc
Reclaim Policy:  Delete
Access Modes:    RWO
VolumeMode:      Filesystem
Capacity:        12Gi
Node Affinity:   <none>
Message:
Source:
    Type:         AzureDisk (an Azure Data Disk mount on the host and bind mount to the pod)
    DiskName:     kubernetes-dynamic-pvc-eb332309-c654-11e9-a0fd-36761452f1dd
    DiskURI:      /subscriptions/6690b014-bdbd-4496-98ee-f2f255699f70/resourceGroups/MC_darek_darekEKS_westeurope/providers/Microsoft.Compute/disks/kubernetes-dynamic-pvc-eb332309-c654-11e9-a0fd-36761452f1dd
    Kind:         Managed
    FSType:
    CachingMode:  ReadOnly
    ReadOnly:     false
Events:           <none>

Now change the disk size to 12 and see. We can check the disk size with the following command.

> az disk  show --ids /subscriptions/6690b014-bdbd-4496-98ee-f2f255699f70/resourceGroups/MC_darek_darekEKS_westeurope/providers/Microsoft.Compute/disks/kubernetes-dynamic-pvc-eb332309-c654-11e9-a0fd-36761452f1dd --query 'diskSizeGb'
12

Ok works in AKS too. What is quite weird, when you do kubectl get pv it shows the new size but the PVC still shows 10Gi.

Clean resources you have created: az group delete -n darek -y

Google Kubernetes Engine

I created a zonal cluster using the CLI in my project called turnkey-cooler-31343.

gcloud config set project turnkey-cooler-31343
gcloud container clusters create darek --zone europe-north1-a --cluster-version "1.13.7-gke.19" --machine-type n1-standard-1 --num-nodes 2

And generate a kubeconfig:

gcloud container clusters get-credentials darek --zone europe-north1-a --project turnkey-cooler-31343

In GKE the default storageclass is called standard.

NAME                 PROVISIONER            AGE
standard (default)   kubernetes.io/gce-pd   3m44s

Just as in AKS and EKS the auto expand was not enabled, let's patch the storage class.

kubectl patch sc standard -p '{"allowVolumeExpansion": true}'

And create a pvc using our PVC yaml. Remember to change storageClassName to standard.

kubectl apply -f pvc.yml

And check the disk:

kubectl get pv 
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                     STORAGECLASS   REASON   AGE
pvc-f79ca9c5-c750-11e9-911b-42010aa601a8   10Gi       RWO            Delete           Bound    default/test-volume-pvc   standard                8m30s

Now check the name of the disk in GCP:

kubectl get pv/pvc-f79ca9c5-c750-11e9-911b-42010aa601a8 -o jsonpath='{.spec.gcePersistentDisk.pdName}'
gke-darek-a2e51c42-dyn-pvc-f79ca9c5-c750-11e9-911b-42010aa601a8

And the size of the disk.

➜  gcloud compute disks describe gke-darek-a2e51c42-dyn-pvc-f79ca9c5-c750-11e9-911b-42010aa601a8 --zone europe-north1-a | grep sizeGb
sizeGb: '10'

Let's now resize the disk, change the pvc.yml or patch the PVC.

➜  gcloud compute disks describe gke-darek-a2e51c42-dyn-pvc-f79ca9c5-c750-11e9-911b-42010aa601a8 --zone europe-north1-a | grep sizeGb
sizeGb: '12'

Ok all fine but as in AKS and EKS the PVC still shows 10Gi.

Clean the infra: gcloud container clusters delete darek --zone europe-north1-a.