Kubernetes-native development: sync files with Ksync

Previous post I started to set up continuous deployment for the development process. Skaffold is a handy tool for doing that.

Let's go deeper this time and imagine the next steps for making up a full-featured "local" development environment with Kubernetes. Running ruby code outside of the local machine keeps it clear and doesn't impose any resource limits.

It is the essence of the idea "Kubernetes-native development": to run all the code at a remote or local Kubernetes cluster.

The issue I try to investigate here is file synchronization. If we edit code locally and run it somewhere remotely, we could get sources out of sync. Unequal sources could cause unexpected results.

For local development with Docker it is common to use Docker volumes. The same idea could be applied here in terms of Kubernetes Persistent Volumes. I failed to use persistent volume, mapping them to the Minikube host machine. And map to local Mac directory onto the Minikube host machine.
Even if it worked well, there are other issues with that approach:

• a cluster could be remote. It's a big deal to map the local directory into the remote cluster host machine
• having additional persistent volume manifests only for developer's needs is obsessive

I decided to synchronize files instead and try out the ksync tool for Kubernetes.

Ksync speeds up developers who build applications for Kubernetes. It transparently updates containers running on the cluster from your local checkout. This enables developers to use their favorite IDEs, such as Atom or Sublime Text to work from inside a cluster instead of from outside it. There is no reason to wait minutes to test code changes when you can see the results in seconds.

If want to do something like docker run -v /foo:/bar with Kubernetes, ksync is for you!

Let's set up file sync for already known Skaffold Ruby example. We'll need the same tooling as in the previous article: kubectl, minikube, skaffold, and installed ksync.

Install ksync

Find the installation details here, or just run for MacOS:

$ curl https://ksync.github.io/gimme-that/gimme.sh | bash

Setup ksync

Then we have to initialize ksync.

$ ksync init

This command initializes ksync and installs the server component on your cluster. One part of ksync works as the entity in the Kubernetes cluster and has to be set up. See more details about ksync architecture.

Check out the Ruby example sources:

$ git clone [e-mail]:GoogleContainerTools/skaffold.git
$ cd examples/ruby

We have to enable ksync file watch with running the watch command:

$ ksync watch

Enable watching changes within ($pwd)/backend folder and sync them back and forth with /app/ directory of running containers:

We will use the specific label app.kubernetes.io/ksync: 'true' presence to determine the scope of pod container to track changes with:

$ ksync create --selector app.kubernetes.io/ksync=true --reload=false -n default $(pwd)/backend /app/

Ruby app sample

Change the k8s/deployment.yaml to add the specific tag app.kubernetes.io/ksync: 'true' to pod's containers.

apiVersion: v1
kind: Service
metadata:
  name: ruby
spec:
  ports:
  - port: 9292
    targetPort: 9292
  type: LoadBalancer
  selector:
    app: ruby
--------
apiVersion: apps/v1
kind: Deployment
metadata:
  name: ruby
spec:
  selector:
    matchLabels:
      app: ruby
  template:
    metadata:
      labels:
        app: ruby
        app.kubernetes.io/ksync: 'true'
    spec:
      containers:
      - name: ruby
        image: ruby-example
        ports:
        - containerPort: 9292
        env:
          - name: RACK_ENV
            value: "development"

And deploy the application:

$ skaffold run --tail
Generating tags...
 - ruby-example -> ruby-example:v1.1.0-118-g92f37b200-dirty
Checking cache...
 - ruby-example: Found Locally
Tags used in deployment:
 - ruby-example -> ruby-example:de8f4f77b01e508f5d862a9ef84d9f33a50d03a88225b0e1b26ddb04d948cded
   local images can't be referenced by digest. They are tagged and referenced by a unique ID instead
Starting deploy...
 - service/ruby created
 - deployment.apps/ruby created
[ruby-66d8d5758-g6m9r ruby] Puma starting in single mode...
[ruby-66d8d5758-g6m9r ruby] * Version 4.3.1 (ruby 2.7.0-p0), codename: Mysterious Traveller
[ruby-66d8d5758-g6m9r ruby] * Min threads: 0, max threads: 16
[ruby-66d8d5758-g6m9r ruby] * Environment: development
[ruby-66d8d5758-g6m9r ruby] * Listening on tcp://0.0.0.0:9292
[ruby-66d8d5758-g6m9r ruby] Use Ctrl-C to stop

This time we won't run skaffold in development mode to prevent the doubled file sync by ksync and skaffold concurrently.

File sync in action

Any changes made on local files are automatically synchronized with remote containers:

$ echo "gem 'oj'" >> backend/Gemfile

The similar ksync watch output means that file is synchronized:

INFO[0008] folder sync running                           pod=ruby-66d8d5758-g6m9r spec=massive-koala
INFO[0012] updating                                      pod=ruby-66d8d5758-g6m9r spec=massive-koala
INFO[0012] update complete                               pod=ruby-66d8d5758-g6m9r spec=massive-koala
INFO[0012] updating                                      pod=ruby-console-69674c499f-97m79 spec=massive-koala
INFO[0012] update complete                               pod=ruby-console-69674c499f-97m79 spec=massive-koala
INFO[0080] updating                                      pod=ruby-console-69674c499f-97m79 spec=massive-koala
INFO[0080] updating                                      pod=ruby-66d8d5758-g6m9r spec=massive-koala
INFO[0080] update complete                               pod=ruby-66d8d5758-g6m9r spec=massive-koala
INFO[0080] update complete                               pod=ruby-console-69674c499f-97m79 spec=massive-koala

Let check the container's Gemfile to see if changes are applied:

$ kubectl exec -it $(kubectl get pods --selector app.kubernetes.io/ksync=true | tail -n 1 | awk '{print $1}') --  cat Gemfile
source 'https://rubygems.org'

gem 'rack'
gem 'rack-unreloader' # for dynamic reloading
gem 'puma'
gem 'oj'

Then we could get inside the pod and run bundle to update Gemfile.lock dependencies lock-file:

$ kubectl exec -it $(kubectl get pods --selector app.kubernetes.io/ksync=true | tail -n 1 | awk '{print $1}') -- bundle

Using bundler 2.1.2
Using nio4r 2.5.2
Using oj 3.10.1
Using puma 4.3.1
Using rack 2.1.1
Using rack-unreloader 1.7.0
Bundle complete! 4 Gemfile dependencies, 6 gems now installed.

After bundle had finished and updated the lock-file, we got it synchronized back with a local machine!

Conclusion

Perfect, we set up the two-way file synchronization and get it working with ksync. From now, any source file updates made automatically by ruby-code, get back to a local machine.

The sync would also work well for generated files, such as rails g migration or even for initial rails project generation with rails new my_project.