DEV Community: Mitz

IntelliJ IDEs and CircleCI integration by 🌀CIclone!

Mitz — Tue, 09 Aug 2022 14:59:23 +0000

Samuraism Inc. has announced a new product: a plugin for JetBrains IDEs named CIclone [sάɪkloʊn]

https://samuraism.com/products/ciclone (Ja)

🌀CIclone?

🌀CIclone provides seamless integration of JetBrains IDEs with several CI services such as Jenkins, GitHub Actions, and CircleCI. It is a paid plugin, but the price is reasonable. In addition, it's available at no additional cost if we have a personal license with them. (and I have it :D)

I checked the CircleCI integration of the plugin because I like CircleCI and I'm working for CircleCI as an engineer (๑•̀ㅂ•́)و✧

The plugin seems super nice!

Execution result notification

When we push code changes, CircleCI starts the pipeline. Then, 🌀CIclone tells us the result by the IDE notification. We don't need to switch to other applications, but we can keep coding on the IDE after the push and can know the execution result. nice

Workflow execution results view

Even after we receive notifications, we can still stay on the IDE and check the execution status (success/failed), the job list, and even the execution logs there. nice

Jump to the failed tests

If there are some tests failed, we can jump to the test cases on the source code. nice

In addition, we can run unit tests on the IDE from there. nice

Jump to CircleCI page

It's also possible to jump to CircleCI page from the IDE. We just select the workflows or jobs on the view and click the following button, then the CircleCI site is open on the browser. nice

For example, we can see the page like this:

Trigger pipeline

We can trigger pipeline by clicking the execution button and specifying branch. nice

Rerun workflow

Selecting the executed workflow, we can rerun the workflow from the rerun button. nice

It seems super nice!

With 🌀CIclone, now I can get CircleCI notifications in the IDE, check logs, re-run, etc., without switching apps while writing code, which seems very convenient!

I don't have any Java projects right now, but I might try it with a Clojure or JavaScript project. If the language is not yet supported by 🌀CIclone, I don't think it is possible to jump to the code or execute UT, but still it is possible to see the CI results and receive notifications. I have a feeling that would be so useful!

Below are notes on setting up this project

Create a sample project: hello-ciclone

I created a Maven project with Spring Initializr. It's been a while since I've used Java!

https://github.com/bufferings/hello-ciclone

Then I wrote a simple config.yml

version: 2.1

jobs:
  build-and-test:
    docker:
      - image: cimg/openjdk:17.0.3
    steps:
      - checkout
      - run:
          name: Build
          command: mvn -B -DskipTests clean package
      - run:
          name: Test
          command: mvn test

workflows:
  sample:
    jobs:
      - build-and-test

Clicked "Set Up Project" on the CircleCI project list:

select the config.yml in the main branch:

After that, it automatically executes UT when I push code

https://app.circleci.com/pipelines/github/bufferings/hello-ciclone

I added a few tests, and some of them succeed and some others fail.

The project preparation is done.

IDEA configuration

Next is a configuration on IDEA. I got license from Samuraism at first, then activated it. But I think we can try it without license for the 30 days trial period.

Install plugin. Preferences > Plugins > Marketplace and type "ciclone" to find the plugin and install it.

After rebooting IDEA, next was to set up CircleCI integration, Preferences > Tools > CircleCI

I wanted to get a new token from CircleCI, so I clicked the "Get Token" link to open CircleCI page for token generation:

And "Create New Token" to generate a new token with some name and copied the value. Then pasted it to the config page of IDEA, and click "Import from existing project". I could see the list of my CircleCI projects:

Chose the project I created just now and it became like this. The "Max Pipeline Size" seems the history count of the pipeline to retrieve. The default is five and I think it's good for me for now.

Clicked "Test"

It works!ヽ(=´▽`=)ﾉ

🌀CIclone View

When we finish the configuration, we can see CircleCI tab at the bottom, so choosing it and click the sync button on the left top of the view to get the latest pipeline results:

Now we can use it as I wrote in the first half of this article!

CircleCI Split Config Orb!

Mitz — Thu, 21 Jul 2022 16:24:00 +0000

I created a CircleCI Orb named Split Config. This is my private fun coding :)

https://circleci.com/developer/orbs/orb/bufferings/split-config

Do you want to split your configuration file?

Have you ever thought if you could split the config.yml file of your CircleCI projects? Especially it could be bigger when the repository is a type of MonoRepo which includes multiple services in one repository.

Good news! Split Config Orb is here for you!

Split Config Orb

You can split your config.yml into multiple configuration files. This Orb merges the split configs into one YAML when CircleCI starts and execute a pipeline with the merged configuration.

Example

Let's say you have multiple services in your repo like this:

Well, it's empty but please imagine the source codes of each services are there :)

Each service has a config.yml for the service. For example. the config for service1 is as follows:

version: 2.1

jobs:
  service1-say-hello:
    docker:
      - image: cimg/base:stable
    steps:
      - checkout
      - run:
          name: "Say hello"
          command: "echo Hello, World!1"

workflows:
  service1-say-hello-workflow:
    jobs:
      - common-say-hello
      - service1-say-hello

service2:

version: 2.1

jobs:
  service2-say-hello:
    docker:
      - image: cimg/base:stable
    steps:
      - checkout
      - run:
          name: "Say hello"
          command: "echo Hello, World!2"

workflows:
  service2-say-hello-workflow:
    jobs:
      - common-say-hello
      - service2-say-hello

etc...

The Orb merges these config files and it will be like this:

version: 2.1
jobs:
  common-say-hello:
    docker:
      - image: cimg/base:stable
    steps:
      - checkout
      - run:
          name: Say hello
          command: echo Hello, World! common
  service1-say-hello:
    docker:
      - image: cimg/base:stable
    steps:
      - checkout
      - run:
          name: Say hello
          command: echo Hello, World!1
  service2-say-hello:
    docker:
      - image: cimg/base:stable
    steps:
      - checkout
      - run:
          name: Say hello
          command: echo Hello, World!2
  service3-say-hello:
    docker:
      - image: cimg/base:stable
    steps:
      - checkout
      - run:
          name: Say hello
          command: echo Hello, World!3
workflows:
  service1-say-hello-workflow:
    jobs:
      - common-say-hello
      - service1-say-hello
  service2-say-hello-workflow:
    jobs:
      - common-say-hello
      - service2-say-hello
  service3-say-hello-workflow:
    jobs:
      - common-say-hello
      - service3-say-hello

How to use

It's super simple. You just need to prepare this config.yml

version: 2.1

setup: true

orbs:
  split-config: bufferings/split-config@0.1.0

workflows:
  generate-config:
    jobs:
      - split-config/generate-config:
          find-config-regex: .*/\.circleci/config\.yml

Then the Orb finds the config files according to the regex you specified, and merge them into one YAML file. After that, the Orb executes CircleCI pipeline with the generated YAML file.

It uses the Dynamic Config feature of CircleCI to start a new pipeline. By default, the feature is disabled, so please enable the feature to get started.

Then you will see something like this:

That's all!

Are you interested in how it merges the configs?

How does it merge the configs?

The Split Config Orb uses CUE to merge the YAML files.

CUE https://cuelang.org/

CUE can import YAML file, convert them into CUE format, and merge them into one CUE config. In addition, it has a feature to export CUE config as YAML.

I used these CUE features to merge your YAML configs. CUE is so powerful but I only use a small part of it. If you want to know more about CUE or how it merges multiple CUE configs, please check CUE specification!

For more details

Please check the README in GitHub:
https://github.com/bufferings/orb-split-config

I wrote some usecases with working examples! Enjoy smaller configuration files

Rethink Scrum from a Japanese cultural perspective #RSGT2021

Mitz — Wed, 17 Feb 2021 13:18:18 +0000

I joined Regional Scrum Gathering Tokyo 2021 held on 6-8 Jan.

https://2021.scrumgatheringtokyo.org/

It was an amazing event, organized as a mixed style of online and onsite this year. I enjoyed it a lot from the online. I watched many interesting sessions. Even after the event, we gathered on Discord to watch the recorded videos and have a talk.

At the event, I talked about rethinking Scrum from a Japanese cultural perspective.

I have been working for Rakuten, Inc. for more than 10 years introducing Scrum to many teams. Last year, I was supporting a guy from oversea who joined the company at that time, he asked many questions to me and we talked a lot everyday.

Answering his questions, I gradually understood his idea. It was totally different way of thinking for me. At the same time, I also understood how I think based on the Japanese culture.

From that experience, I found the Japanese people may have a different perspective for Scrum. That's why I had this talk.

In the session, I talked about

Does Scrum look the same from Japanese culture and another culture?
Why is it so challenging to adopt Scrum in Japan?
Rethink Scrum

I'm glad if you enjoy my session video. I hope to join it onsite next year :)

Istio 1.6.0 Canary Upgrade Trial

Mitz — Sat, 23 May 2020 17:11:24 +0000

While I am playing with Istio 1.5, it has become 1.6. Time flies...

https://istio.io/news/releases/1.6.x/announcing-1.6/

Istio LTS has 3 months release cycle. And the previous LTS becomes EOL after 3 months from the new LTS is released. Therefore I think it's important how to make it easy to handle upgrade.

The good news is that Istio 1.6.0 supports Canary upgrade of Istio itself. It sounds interesting to me.

https://istio.io/blog/2020/multiple-control-planes/

So I would like to try it.

Environment

I prepared a GKE cluster.

❯ kubectl get nodes
NAME                                       STATUS   ROLES    AGE   VERSION
gke-cluster-1-default-pool-019aa731-6ztc   Ready    <none>   21m   v1.16.8-gke.15
gke-cluster-1-default-pool-019aa731-spbj   Ready    <none>   21m   v1.16.8-gke.15
gke-cluster-1-default-pool-019aa731-ztjz   Ready    <none>   21m   v1.16.8-gke.15

And prepared config.yaml file to install Istio which is a custom configuration for Istio install. I simply added Stackdriver integration, though it's not related to this article because I don't use anything of Stackdriver this time:

apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  profile: default

  addonComponents:
    prometheus:
      enabled: false

  values:
    global:
      proxy:
        tracer: "stackdriver"
      logAsJson: true
    telemetry:
      v2:
        enabled: true
        prometheus:
          enabled: false
        stackdriver:
          enabled: true
          logging: true
          monitoring: true
          # topology: true
          configOverride: {}

Canary upgrade

I followed this article:

https://istio.io/docs/setup/upgrade/

It introduces 2 types of upgrade:

Canary upgrades
In place upgrades

I will try canary upgrades this time from 1.5.4 to 1.6.0.

Install Istio 1.5

At first, I install 1.5.4:

❯ curl -L https://istio.io/downloadIstio | ISTIO_VERSION=1.5.4 sh - 

❯ cd istio-1.5.4

# I saved config.yaml with the contents I wrote above.
❯ bin/istioctl manifest apply -f config.yaml                   
- Applying manifest for component Base...
✔ Finished applying manifest for component Base.
- Applying manifest for component Pilot...
✔ Finished applying manifest for component Pilot.
  Waiting for resources to become ready...
  Waiting for resources to become ready...
  Waiting for resources to become ready...
  Waiting for resources to become ready...
  Waiting for resources to become ready...
  Waiting for resources to become ready...
- Applying manifest for component IngressGateways...
✔ Finished applying manifest for component IngressGateways.


✔ Installation complete

Then, install our favorite bookinfo app:

# Enable sidecar auto injection
❯ kubectl label namespace default istio-injection=enabled
namespace/default labeled

# Deploy Bookinfo services
❯ kubectl apply -f samples/bookinfo/platform/kube/bookinfo.yaml
service/details created
serviceaccount/bookinfo-details created
deployment.apps/details-v1 created
service/ratings created
serviceaccount/bookinfo-ratings created
deployment.apps/ratings-v1 created
service/reviews created
serviceaccount/bookinfo-reviews created
deployment.apps/reviews-v1 created
deployment.apps/reviews-v2 created
deployment.apps/reviews-v3 created
service/productpage created
serviceaccount/bookinfo-productpage created
deployment.apps/productpage-v1 created

# Deploy Gateway
❯ kubectl apply -f samples/bookinfo/networking/bookinfo-gateway.yaml
gateway.networking.istio.io/bookinfo-gateway created
virtualservice.networking.istio.io/bookinfo created

# Get IP address of LB
❯ export INGRESS_HOST=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}')

# Open the app with browser (My machine is Ubuntu)
❯ xdg-open http://$INGRESS_HOST/productpage

I can see a familiar bookinfo page.

Upgrade to Istio 1.6

Download 1.6.0:

❯ curl -L https://istio.io/downloadIstio | ISTIO_VERSION=1.6.0 sh - 
❯ cd istio-1.6.0
# I saved config.yaml with the contents I wrote above.

Currently, my Istio version is 1.5.4:

❯ bin/istioctl version                                                     
client version: 1.5.4
control plane version: 1.5.4
data plane version: 1.5.4 (7 proxies)

Run upgrade

Then, upgrade the Istio following this document:

https://istio.io/docs/setup/upgrade/

hm... though it says the command is like this:

$ istioctl install --set revision=canary

I guess I should specify my config.yaml file, because it would be deafult profile otherwise. So I checked the reference of the command:

https://istio.io/docs/reference/commands/istioctl/#istioctl-install

It has -f option, so I can try this:

❯ bin/istioctl install --set revision=canary -f config.yaml
✔ Istio core installed
✔ Istiod installed
✔ Ingress gateways installed
✔ Installation complete

And it seems worked. The result of watching pod is as follows:

❯ kubectl -n istio-system get pods -w
NAME                                    READY   STATUS    RESTARTS   AGE
istio-ingressgateway-7f76b9fc9f-ffr4j   1/1     Running   0          4m29s
istiod-6558b7b7f4-9d2dk                 1/1     Running   0          4m50s

# I ran install command here

istiod-canary-848948494f-vrpz8          0/1     Pending   0          0s
istiod-canary-848948494f-vrpz8          0/1     Pending   0          0s
istiod-canary-848948494f-vrpz8          0/1     ContainerCreating   0          0s
istiod-canary-848948494f-vrpz8          0/1     Running             0          9s
istiod-canary-848948494f-vrpz8          1/1     Running             0          10s
istio-ingressgateway-ccbb848f5-b94mh    0/1     Pending             0          0s
istio-ingressgateway-ccbb848f5-b94mh    0/1     Pending             0          0s
istio-ingressgateway-ccbb848f5-b94mh    0/1     ContainerCreating   0          1s
istio-ingressgateway-ccbb848f5-b94mh    0/1     Running             0          8s
istio-ingressgateway-ccbb848f5-b94mh    1/1     Running             0          10s
istio-ingressgateway-7f76b9fc9f-ffr4j   1/1     Terminating         0          5m37s
istio-ingressgateway-7f76b9fc9f-ffr4j   0/1     Terminating         0          5m43s
istio-ingressgateway-7f76b9fc9f-ffr4j   0/1     Terminating         0          5m44s
istio-ingressgateway-7f76b9fc9f-ffr4j   0/1     Terminating         0          5m44s

Finally, it became like this:

❯ kubectl get pods -n istio-system
NAME                                   READY   STATUS    RESTARTS   AGE
istio-ingressgateway-ccbb848f5-b94mh   1/1     Running   0          7m16s
istiod-6558b7b7f4-9d2dk                1/1     Running   0          13m
istiod-canary-848948494f-vrpz8         1/1     Running   0          7m29s

What does it mean?

It adds istiod-canary keeping the original istiod as it is.
Regarding istio-ingressgateway, it stops the original one and deploys new one.

I see... I looked into istio-ingressgateway and found it became 1.6.0:

❯ kubectl -n istio-system get pods istio-ingressgateway-ccbb848f5-b94mh -o yaml | grep image: 
    image: docker.io/istio/proxyv2:1.6.0
    image: istio/proxyv2:1.6.0

The version command shows this result:

❯ bin/istioctl version
client version: 1.6.0
pilot version: 1.5.4
istiod version: 1.6.0
data plane version: 1.5.4 (6 proxies), 1.6.0 (1 proxies)

Ah, it was like this before:

❯ bin/istioctl version                                     
client version: 1.5.4
control plane version: 1.5.4
data plane version: 1.5.4 (7 proxies)

It was separated from control plane version into 2 parts: pilot version and istiod version.

Btw, it is istio-ingressgateway which uses 1.6.0 in data plane version. It seems intended to update ingress-gateway:

https://github.com/istio/istio/issues/23923#issuecomment-630892188

At this point

Data Plane remains 1.5.4. It should be 1.5.4, even though I restart the pods. Let's try:

❯ kubectl rollout restart deployment
deployment.apps/details-v1 restarted
deployment.apps/productpage-v1 restarted
deployment.apps/ratings-v1 restarted
deployment.apps/reviews-v1 restarted
deployment.apps/reviews-v2 restarted
deployment.apps/reviews-v3 restarted

❯ bin/istioctl version   
client version: 1.6.0
pilot version: 1.5.4
istiod version: 1.6.0
data plane version: 1.5.4 (6 proxies), 1.6.0 (1 proxies)

Looks good. They keep using 1.5.4.

Upgrade Data Plane

As I tried now, each Pod still uses the previous version when we just install new istiod-canary. We need to modify namespace label in order for the Pods to start using the new version.

❯ kubectl label namespace default istio-injection- istio.io/rev=canary
namespace/default labeled

❯ kubectl get namespaces default --show-labels
NAME      STATUS   AGE   LABELS
default   Active   48m   istio.io/rev=canary

I removed istio-injection label from the default namespace, where I installed bookinfo, and added istio.io/rev=canary label. The reason why we need to remove istio-injection label from the namespace is it has higher precedence than revision label for backward compatibility.

Now they should start using new version when I restart them:

❯ kubectl rollout restart deployment
deployment.apps/details-v1 restarted
deployment.apps/productpage-v1 restarted
deployment.apps/ratings-v1 restarted
deployment.apps/reviews-v1 restarted
deployment.apps/reviews-v2 restarted
deployment.apps/reviews-v3 restarted

Wait for a while the Pods restart and check version:

❯ bin/istioctl version
client version: 1.6.0
pilot version: 1.5.4
istiod version: 1.6.0
data plane version: 1.6.0 (7 proxies)

It is updated, yay! I also checked one of the container and found it used newer version of istiod:

❯ bin/istioctl proxy-config endpoints details-v1-7bc48f795d-2fwzr --cluster xds-grpc -ojson | grep hostname
                "hostname": "istiod-canary.istio-system.svc",

Out of curiosity

I wondered whether I can write revision option in my config.yaml file rather than setting it as a command line option.

As a result, it works with this config:

apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  profile: default

  revision: "v160"

  addonComponents:
    prometheus:
      enabled: false

  values:
# omitted

At the first time, I wrote v1.6.0 but it didn't work, so I guess we can't use period as a revision name:

❯ bin/istioctl install -f config.yaml
Error: failed to apply manifests: errors occurred during operation

My thoughts

(1) How to clean up old version?

I can still see old version resources such as istiod, sidecar-injector etc. But I'm not sure how to clean them up.

❯ kubectl get pods -n istio-system
NAME                                   READY   STATUS    RESTARTS   AGE
istio-ingressgateway-ccbb848f5-b94mh   1/1     Running   0          62m
istiod-6558b7b7f4-9d2dk                1/1     Running   0          68m
istiod-canary-848948494f-vrpz8         1/1     Running   0          62m

❯ kubectl get mutatingwebhookconfigurations
NAME                                                 CREATED AT
istio-sidecar-injector                               2020-05-23T13:57:59Z
istio-sidecar-injector-canary                        2020-05-23T14:03:42Z
pod-ready.config.common-webhooks.networking.gke.io   2020-05-23T13:49:48Z

Actially, it showed errors when I tried to install another istiod after I removed the original istiod. It seems to me the validation hook keep using old istiod and shows error?

❯ bin/istioctl install -f config.yaml                               
✔ Istio core installed                                                                                                                                                           
2020-05-23T15:31:02.866368Z     error   installer       failed to create "EnvoyFilter/istio-system/metadata-exchange-1.6-v160-2": Internal error occurred: failed calling webhook "validation.istio.io": Post https://istiod.istio-system.svc:443/validate?timeout=30s: no endpoints available for service "istiod"
2020-05-23T15:31:03.008138Z     error   installer       failed to create "EnvoyFilter/istio-system/stackdriver-filter-1.6-v160-2": Internal error occurred: failed calling webhook "validation.istio.io": Post https://istiod.istio-system.svc:443/validate?timeout=30s: no endpoints available for service "istiod"
2020-05-23T15:31:03.164115Z     error   installer       failed to create "EnvoyFilter/istio-system/tcp-metadata-exchange-1.6-v160-2": Internal error occurred: failed calling webhook "validation.istio.io": Post https://istiod.istio-system.svc:443/validate?timeout=30s: no endpoints available for service "istiod"
✘ Istiod encountered an error: failed to create "EnvoyFilter/istio-system/metadata-exchange-1.6-v160-2": Internal error occurred: failed calling webhook "validation.istio.io": Post https://istiod.istio-system.svc:443/validate?timeout=30s: no endpoints available for service "istiod"
failed to create "EnvoyFilter/istio-system/stackdriver-filter-1.6-v160-2": Internal error occurred: failed calling webhook "validation.istio.io": Post https://istiod.istio-system.svc:443/validate?timeout=30s: no endpoints available for service "istiod"
failed to create "EnvoyFilter/istio-system/tcp-metadata-exchange-1.6-v160-2": Internal error occurred: failed calling webhook "validation.istio.io": Post https://istiod.istio-system.svc:443/validate?timeout=30s: no endpoints available for service "istiod"

(2) Should I keep using the name "canary"?

I used "canary" for the revision name following the tutorial. But it seems I need to keep using the name "canary" for this version.

So I feel I should use a name which I can keep using it for the version such as v160 etc?

(3) The unit isn't pod but namespace

Currently, it only supports to change namespace label for the canary upgrade. I thought it would be convenient if I can upgrade pods one by one, but hm... it might be enough good.

Summary

It is a good direction that we can deploy multiple versions of istiod and gradually change to use the new version.
I need to remember IngressGateway is upgraded at once without having multiple versions.
I'm looking forward to seeing new information about cleanup. I hope it is added in 1.7.

And I started to subscribe this issue:

https://github.com/istio/istio/issues/23889

It was fun!

Tried k8s + Istio on my laptop with k3d

Mitz — Sun, 17 Nov 2019 03:59:40 +0000

## k8s on local

Sometimes I want to run k8s on my local machine to check something. So far I've used Minikube and one bound to Docker Desktop, but I wondered there might be newer ways to do it. Then I found this thread:

https://www.reddit.com/r/kubernetes/comments/be0415/k3s_minikube_or_microk8s/

The following tools are introduced there:

Minikube
Microk8s
K3s
Kind
Desktop Docker
K3d
Kubeadm

So I tried k3d, just because I felt I like it.

## k3d

https://github.com/rancher/k3d

It seems k3d runs k3s on Docker, which is also introduced in the thread above. I was confused a little bit when I thought Docker container which is for running Docker containers... but anyway I tried it.

My laptop is Ubuntu, but it should work (I hope) on Windows & Mac because it's Docker.

Firstly, I installed it reading the README (it supports brew as well):

$ curl -s https://raw.githubusercontent.com/rancher/k3d/master/install.sh | bash

Then, I created cluster and set configuration for kubectl:

$ k3d create
$ export KUBECONFIG=$(k3d get-kubeconfig)

It really worked!

$ kubectl get pods -A
NAMESPACE     NAME                                      READY   STATUS      RESTARTS   AGE
kube-system   local-path-provisioner-58fb86bdfd-kvmdh   1/1     Running     0          3m40s
kube-system   coredns-57d8bbb86-grbbr                   1/1     Running     0          3m40s
kube-system   helm-install-traefik-4qr7t                0/1     Completed   0          3m40s
kube-system   svclb-traefik-j8c49                       3/3     Running     0          3m5s
kube-system   traefik-65bccdc4bd-vtk9r                  1/1     Running     0          3m5s

$ kubectl get svc -A                                                                                                                                                                                   
NAMESPACE     NAME         TYPE           CLUSTER-IP     EXTERNAL-IP    PORT(S)                                     AGE                                                                                
default       kubernetes   ClusterIP      10.43.0.1      <none>         443/TCP                                     5m57s                                                                              
kube-system   kube-dns     ClusterIP      10.43.0.10     <none>         53/UDP,53/TCP,9153/TCP                      5m56s                                                                              
kube-system   traefik      LoadBalancer   10.43.80.235   192.168.48.2   80:30107/TCP,443:31822/TCP,8080:31373/TCP   5m5s

It seems to use traefik for a LoadBalancer service.

## Istio?

I hit upon an idea whether Istio could run on it. Then I found this issue:

https://github.com/rancher/k3d/issues/104

It seems he successfully ran Istio on it after he turned off traefik to avoid port conflict.

Let's try it.

### Create k3d cluster without traefik

# Delete the previous cluster
$ k3d delete

# Create a cluster without traefik
$ k3d create --server-arg --no-deploy --server-arg traefik

# Generate config
$ export KUBECONFIG=$(k3d get-kubeconfig)

# Check
$ kubectl get pod,svc -A
NAMESPACE     NAME                                          READY   STATUS    RESTARTS   AGE
kube-system   pod/local-path-provisioner-58fb86bdfd-h6npn   1/1     Running   0          13m
kube-system   pod/coredns-57d8bbb86-zkjkq                   1/1     Running   0          13m

NAMESPACE     NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                  AGE
default       service/kubernetes   ClusterIP   10.43.0.1    <none>        443/TCP                  13m
kube-system   service/kube-dns     ClusterIP   10.43.0.10   <none>        53/UDP,53/TCP,9153/TCP   13m

Now I'm ready for installing Istio on it.

### Install Istio

I found the latest version was already 1.4, but I felt I would like to try 1.3 because 1.4 was released just in this month.

I downloaded Istio from here:

https://github.com/istio/istio/releases/tag/1.3.5

and I installed Istio following the steps in this page:

https://istio.io/docs/setup/install/helm/

I've already installed Helm on my laptop, and I chose the option of using helm template.

# Create a namespace for Istio
$ kubectl create namespace istio-system

# Install CRDs
$ helm template install/kubernetes/helm/istio-init --name istio-init --namespace istio-system | kubectl apply -f -

# Wait for the generation of the CRDs
$ kubectl -n istio-system wait --for=condition=complete job --all

Oh, I found the command has been changed. Previously, it was wc to check there're 23 CRDs created, but now it uses kubectl wait --for. Nice!

$ helm template install/kubernetes/helm/istio --name istio --namespace istio-system | kubectl apply -f -

I couldn't believe there's no error... This might be the first time for me to be able to install Istio without any troubles lol

kubectl get svc,pod -n istio-system
NAME                             TYPE           CLUSTER-IP      EXTERNAL-IP    PORT(S)                                                                                                                                      AGE
service/istio-galley             ClusterIP      10.43.10.191    <none>         443/TCP,15014/TCP,9901/TCP                                                                                                                   2m21s
service/istio-policy             ClusterIP      10.43.86.131    <none>         9091/TCP,15004/TCP,15014/TCP                                                                                                                 2m21s
service/istio-telemetry          ClusterIP      10.43.11.107    <none>         9091/TCP,15004/TCP,15014/TCP,42422/TCP                                                                                                       2m21s
service/istio-pilot              ClusterIP      10.43.126.19    <none>         15010/TCP,15011/TCP,8080/TCP,15014/TCP                                                                                                       2m21s
service/prometheus               ClusterIP      10.43.41.148    <none>         9090/TCP                                                                                                                                     2m21s
service/istio-citadel            ClusterIP      10.43.91.217    <none>         8060/TCP,15014/TCP                                                                                                                           2m21s
service/istio-sidecar-injector   ClusterIP      10.43.117.133   <none>         443/TCP,15014/TCP                                                                                                                            2m21s
service/istio-ingressgateway     LoadBalancer   10.43.69.0      192.168.96.2   15020:30845/TCP,80:31380/TCP,443:31390/TCP,31400:31400/TCP,15029:31842/TCP,15030:32247/TCP,15031:32685/TCP,15032:31093/TCP,15443:30499/TCP   2m21s

NAME                                          READY   STATUS      RESTARTS   AGE
pod/istio-init-crd-10-1.3.5-28hj7             0/1     Completed   0          5m40s
pod/istio-init-crd-11-1.3.5-vmwmw             0/1     Completed   0          5m40s
pod/istio-init-crd-12-1.3.5-84q77             0/1     Completed   0          5m40s
pod/istio-security-post-install-1.3.5-jb66j   0/1     Completed   0          2m21s
pod/svclb-istio-ingressgateway-ww22d          9/9     Running     0          2m21s
pod/istio-citadel-5c67db5cb-hmhvb             1/1     Running     0          2m20s
pod/prometheus-6f74d6f76d-tpjpc               1/1     Running     0          2m20s
pod/istio-policy-66d87c756b-hf4wx             2/2     Running     3          2m21s
pod/istio-galley-56b9fb859d-7jmsq             1/1     Running     0          2m21s
pod/istio-sidecar-injector-5d65cfcd79-lhh6k   1/1     Running     0          2m20s
pod/istio-pilot-64478c6886-9xm7b              2/2     Running     0          2m20s
pod/istio-telemetry-5d4c4bfbbf-g4ccz          2/2     Running     4          2m20s
pod/istio-ingressgateway-7b766b6685-5vwg5     1/1     Running     0          2m21s

Next, I tried to run a sample application on the Istio.

### Deploy bookinfo sample application

To check it actually works, I deployed bookinfo sample application included in Istio:

https://istio.io/docs/examples/bookinfo/

# Enable automatic sidecar injection
$ kubectl label namespace default istio-injection=enabled

# Deploy apps
$ kubectl apply -f samples/bookinfo/platform/kube/bookinfo.yaml

# Wait for the deployment finished for example using watch
$ kubectl get pods -w
NAME                              READY   STATUS            RESTARTS   AGE
details-v1-78d78fbddf-5db8b       0/2     PodInitializing   0          37s
reviews-v1-7bb8ffd9b6-rdgjc       0/2     PodInitializing   0          37s
ratings-v1-6c9dbf6b45-p7567       0/2     PodInitializing   0          36s
productpage-v1-596598f447-nj6wx   0/2     PodInitializing   0          36s
reviews-v3-68964bc4c8-qrhc4       0/2     PodInitializing   0          37s
reviews-v2-d7d75fff8-65f4q        0/2     PodInitializing   0          37s

# Create ingress gateway for bookinfo
$ kubectl apply -f samples/bookinfo/networking/bookinfo-gateway.yaml

After that, I confirmed External IP of LoadBalaner service:

$ kubectl get svc  -n istio-system istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}'
192.168.96.2

and opened the following URL with the IP:

http://{The IP Address}/productpage

I was surprised again that it worked!

The memory usage of the container with bookinfo was around 2GiB:

$ docker stats --no-stream
CONTAINER ID        NAME                     CPU %               MEM USAGE / LIMIT    MEM %               NET I/O             BLOCK I/O           PIDS
598bd6d07c85        k3d-k3s-default-server   52.24%              1.909GiB / 15.4GiB   12.40%              819MB / 21.7MB      1.41MB / 818MB      899

Thought it wouldn't be easier to solve problems if something happens somewhere, it seems very convenient to run k8s on local with k3d.

Micronaut PetClinic with GraalVM > 19

Mitz — Sat, 15 Jun 2019 12:29:17 +0000

Now you can run Micronaut PetClinic with the (kinda) latest GraalVM Native Image. I will explain the meaning later.

Build & Run

$ git clone https://github.com/bufferings/micronaut-petclinic.git
$ cd micronaut-petclinic

# Build Native Image. Please wait for 10-15 minutes.
$ ./mvnw package && docker build -t micronaut-petclinic .

# Start Database
$ docker-compose up -d db

# Run separately to see the log easily
$ docker-compose up app

Start up time

It starts in a few hundreds of millis.

Docker Information

The image size is 130MB.

❯ docker images micronaut-petclinic
REPOSITORY            TAG                 IMAGE ID            CREATED             SIZE
micronaut-petclinic   latest              40834e227e19        About an hour ago   130MB

and the memory usage is 125MB after using for a while

❯ docker ps -f name=micronaut-petclinic_app* -q | xargs docker stats --no-stream
CONTAINER ID        NAME                                     CPU %               MEM USAGE / LIMIT    MEM %               NET I/O             BLOCK I/O           PIDS
5c4c627ee81d        micronaut-petclinic_app_1_1350df269f2b   0.07%               124.3MiB / 15.4GiB   0.79%               158kB / 121kB       0B / 0B             11

Background

I created Micronaut PetClinic last month and it worked as Native Image with GraalVM 1.0 RC16 to start surprisingly fast speed in a few hundreds of millis.

Micronaut PetClinic

Mitz ・ May 6 '19 ・ 1 min read

#micronautfw #graalvm #java

A few days later, GraalVM released a production version 19.0.0.
https://medium.com/graalvm/announcing-graalvm-19-4590cf354df8

But unfortunately, my PetClinic doesn't work with GraalVM 19 Native Image. (´・ω・｀)

The cause is this issue:
https://github.com/oracle/graal/issues/1295

and we need this commit:
https://github.com/oracle/graal/commit/c6237d219a91c82b1954dfbfa0898ed917db09eb

The commit hasn't been included in the latest version 19.0.2 released yesterday.

Docker Image with the latest GraalVM

Then I found it works when I build GraalVM from its master branch. So I created a Docker image with the master branch:
https://hub.docker.com/r/bufferings/build-graalvm-docker

I would like to use it for a while until the commit is included in the GraalVM official release.

With Micronaut, it's so easy to create an application for Native Image, but I would like to talk about it another time (๑•̀ㅂ•́)و✧

Micronaut PetClinic

Mitz — Mon, 06 May 2019 18:14:22 +0000

I tried implementing Spring PetClinic with Micronaut instead of using Spring.

https://github.com/bufferings/micronaut-petclinic

The application uses Thymeleaf, JPA(Hibernate) and DI. But it starts in a few hundreds of millis with GraalVM native image. It's so fast!

In this picture, it started in 324ms:

I haven't yet implemented all the features and it's still very dirty, but basic features work.

How to try (not native image)

1. Clone the Project

git clone https://github.com/bufferings/micronaut-petclinic.git
cd micronaut-petclinic

2. Start PostgreSQL

This Petclinic uses PostgreSQL. You could start PostgreSQL with docker:

docker-compose up

3. Run Application

./mvnw compile exec:exec

3-2. Or you can run it from JAR

./mvnw package
java -jar target/micronaut-*.jar

4. Access PetClinic

You can then access petclinic here: http://localhost:8080/

With JAR file, it takes around 5s to start the app on my laptop.

How to try native image

Micronaut supports GraalVM native image. So I made this PetClinic run as native image.

./mvnw package && docker build -t micronaut-petclinic .

I usually have ☕ (around 10 mins on my laptop) to wait for the build finishes.

# Docker for Mac or Windows
export HOST_NAME=host.docker.internal
# Linux
export HOST_NAME=172.17.0.1

docker run --rm -p 8080:8080 -e JDBC_URL=jdbc:postgresql://${HOST_NAME}:5432/petclinic micronaut-petclinic

Then it takes around 300ms to start. (๑•̀ㅂ•́)و✧

Let's make SpringBoot app start faster

Mitz — Sat, 17 Nov 2018 01:36:32 +0000

"How Fast is Spring?"

is a session at Spring One Platform 2018. I watched the video and tried it by myself. So I introduce here what I did and the results.

I recommend you to watch this session video if you haven't yet. It's so interesting.

https://springoneplatform.io/2018/sessions/how-fast-is-spring-

Today's source code

https://github.com/bufferings/spring-boot-startup-mybench

↓I used OpenJDK 11.

❯ java --version
openjdk 11.0.1 2018-10-16
OpenJDK Runtime Environment 18.9 (build 11.0.1+13)
OpenJDK 64-Bit Server VM 18.9 (build 11.0.1+13, mixed mode)

↓You can run all the benchmarks like this. It will take a while because it runs all the benchmarks.

❯ ./mvnw clean package
❯ (cd benchmarks/; java -jar target/benchmarks.jar)

1. FluxBaseline

↓I created a project using SpringInitializr only with Reactive Web. Then, I wrote a tiny controller with WebMVC style.

@SpringBootApplication
@RestController
public class DemoApplication {

  @GetMapping("/")
  public String home() {
    return "Hello";
  }

  public static void main(String[] args) {
    SpringApplication.run(DemoApplication.class, args);
  }
}

↓The version of Spring Boot was 2.1.0.RELEASE.

    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>2.1.0.RELEASE</version>
        <relativePath/> <!-- lookup parent from repository -->
    </parent>

↓It took 2.938 ± 0.287 s/op to start.

Benchmark                                          Mode  Cnt  Score   Error  Units
MyBenchmark.case01_FluxBaseline                      ss   10  2.938 ± 0.287   s/op

Now I got a baseline to check the startup time. Let's start from here.

2. WebMVC

↓I wondered what about WebMVC, not WebFlux? So I tried it. Maybe does it just mean the comparison of Tomcat and Netty?

Benchmark                                          Mode  Cnt  Score   Error  Units
MyBenchmark.case01_FluxBaseline                      ss   10  2.938 ± 0.287   s/op
MyBenchmark.case02_Web                               ss   10  3.281 ± 0.342   s/op

WebFlux is a bit faster, isn't it?

3. spring-context-indexer

Next, I tried spring-context-indexer which seems to create component index.

        <dependency>
            <groupId>org.springframework</groupId>
            <artifactId>spring-context-indexer</artifactId>
            <optional>true</optional>
        </dependency>

↓Um... a little slower?

Benchmark                                          Mode  Cnt  Score   Error  Units
MyBenchmark.case01_FluxBaseline                      ss   10  2.938 ± 0.287   s/op
MyBenchmark.case03_WithContextIndexer                ss   10  3.063 ± 0.102   s/op

↓I checked spring.components, and found it contained only 1 component. I see... I should try with a bigger project to know the effect.

#
#Sun Nov 04 18:42:59 JST 2018
com.example.DemoApplication=org.springframework.stereotype.Component

4. Lazy Initialization

Tried Lazy Init.

@Configuration
public class LazyInitBeanFactoryPostProcessor implements BeanFactoryPostProcessor {
  @Override
  public void postProcessBeanFactory(ConfigurableListableBeanFactory beanFactory) throws BeansException {
    for (String beanName : beanFactory.getBeanDefinitionNames()) {
      beanFactory.getBeanDefinition(beanName).setLazyInit(true);
    }
  }
}

↓Here's the result. It became just a little bit faster.

Benchmark                                          Mode  Cnt  Score   Error  Units
MyBenchmark.case01_FluxBaseline                      ss   10  2.938 ± 0.287   s/op
MyBenchmark.case04_WithLazyInit                      ss   10  2.844 ± 0.129   s/op

5. NoVerify

Ran with -noverify:

Benchmark                                          Mode  Cnt  Score   Error  Units
MyBenchmark.case01_FluxBaseline                      ss   10  2.938 ± 0.287   s/op
MyBenchmark.case05_WithNoVerifyOption                ss   10  2.582 ± 0.060   s/op

It became a little faster. I don't know what it means, so I need to check it later sometime.

6. TieredStopAtLevel

Ran with -XX:TieredStopAtLevel=1:

Benchmark                                          Mode  Cnt  Score   Error  Units
MyBenchmark.case01_FluxBaseline                      ss   10  2.938 ± 0.287   s/op
MyBenchmark.case06_WithTieredStopAtLevel1Option      ss   10  1.980 ± 0.037   s/op

Uh, much faster! It took less than 2 seconds. But I don't know this flag, too. So I will check it later.

7. Specify SpringConfigLocation explicitly

Ran with -Dspring.config.location=classpath:/application.properties:

Benchmark                                          Mode  Cnt  Score   Error  Units
MyBenchmark.case01_FluxBaseline                      ss   10  2.938 ± 0.287   s/op
MyBenchmark.case07_WithSpringConfigLocationOption    ss   10  3.026 ± 0.139   s/op

Um, it became slower.

8. Turn off JMX

Ran with -Dspring.jmx.enabled=false:

Benchmark                                          Mode  Cnt  Score   Error  Units
MyBenchmark.case01_FluxBaseline                      ss   10  2.938 ± 0.287   s/op
MyBenchmark.case08_WithJmxDisabledOption             ss   10  2.877 ± 0.097   s/op

It became a little bit faster.

9. Exclude Logback

From here, I try to exclude libraries. At first, excluding Logback:

        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-webflux</artifactId>
            <exclusions>
                <exclusion>
                    <artifactId>spring-boot-starter-logging</artifactId>
                    <groupId>org.springframework.boot</groupId>
                </exclusion>
            </exclusions>
        </dependency>
        <dependency>
            <groupId>org.slf4j</groupId>
            <artifactId>slf4j-jdk14</artifactId>
        </dependency>

Here's the result:

Benchmark                                          Mode  Cnt  Score   Error  Units
MyBenchmark.case01_FluxBaseline                      ss   10  2.938 ± 0.287   s/op
MyBenchmark.case09_WithoutLogback                    ss   10  2.904 ± 0.096   s/op

mm... slightly improved?

10. Exclude Jackson

Next is Jackson

        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-webflux</artifactId>
            <exclusions>
                <exclusion>
                    <artifactId>spring-boot-starter-json</artifactId>
                    <groupId>org.springframework.boot</groupId>
                </exclusion>
            </exclusions>
        </dependency>

The result:

Benchmark                                          Mode  Cnt  Score   Error  Units
MyBenchmark.case01_FluxBaseline                      ss   10  2.938 ± 0.287   s/op
MyBenchmark.case10_WithoutJackson                    ss   10  2.789 ± 0.093   s/op

It became a little bit faster.

11. Exclude HibernateValidator

        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-webflux</artifactId>
            <exclusions>
                <exclusion>
                    <artifactId>hibernate-validator</artifactId>
                    <groupId>org.hibernate.validator</groupId>
                </exclusion>
            </exclusions>
        </dependency>

Here's the result:

Benchmark                                          Mode  Cnt  Score   Error  Units
MyBenchmark.case01_FluxBaseline                      ss   10  2.938 ± 0.287   s/op
MyBenchmark.case11_WithoutHibernateValidator         ss   10  2.857 ± 0.084   s/op

Slightly improved, too.

This is the end of library exclusion.

12. AppCDS

AppCDS (Application Class Data Sharing) was included in Oracle JDK as a commercial feature. But it became available from OpenJDK 10.

It seems AppCDS dumps information into a shared archive, so startup time becomes shorter.

Benchmark                                          Mode  Cnt  Score   Error  Units
MyBenchmark.case01_FluxBaseline                      ss   10  2.938 ± 0.287   s/op
MyBenchmark.case12_WithAppCds                        ss   10  2.957 ± 0.079   s/op

mm... it wasn't faster... then I checked articles about CDS, and found the reason.

With SpringBoot FatJAR, the libraries are out of the scope of CDS.

13. Flux with Thin Launcher

Uh, I'm sorry, but the benchmark name "Exploded" is wrong. Once I tried to explode the FatJAR, but I couldn't use CDS with the exploded JAR after all. So I switched to use Thin Launcher. Please take the benchmark name "Exploded" as "Thin Launcher".

Before using CDS, I would like to check the speed of JAR file packaged with Thin Launcher.

        <plugins>
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
                <dependencies>
                    <dependency>
                        <groupId>org.springframework.boot.experimental</groupId>
                        <artifactId>spring-boot-thin-layout</artifactId>
                        <version>1.0.15.RELEASE</version>
                    </dependency>
                </dependencies>
            </plugin>
        </plugins>

Although I used Thin Launcher to package the app, I didn't use the launch class of Thin Launcher, but specified Main class to make the startup time as fast as possible.

Benchmark                                          Mode  Cnt  Score   Error  Units
MyBenchmark.case01_FluxBaseline                      ss   10  2.938 ± 0.287   s/op
MyBenchmark.case13_Exploded                          ss   10  2.476 ± 0.091   s/op

hum, a bit faster, isn't it?

14. Thin Launcher + CDS

Now I would like to apply AppCDS to it.

Benchmark                                          Mode  Cnt  Score   Error  Units
MyBenchmark.case01_FluxBaseline                      ss   10  2.938 ± 0.287   s/op
MyBenchmark.case14_ExplodedWithAppCds                ss   10  1.535 ± 0.036   s/op

Wow! It became much faster!

15. All applied

Finally, I applied everything.

Benchmark                                          Mode  Cnt  Score   Error  Units
MyBenchmark.case01_FluxBaseline                      ss   10  2.938 ± 0.287   s/op
MyBenchmark.case15_AllApplied                        ss   10  0.801 ± 0.037   s/op

Less than 1 second! (∩´∀｀)∩yay

One more step

In the Dave's session, he mentioned "Functional Bean Definitions", tried improvements with Spring without SpringBoot and the app became much faster. I need to learn more to understand them.

Result list

Benchmark                                          Mode  Cnt  Score   Error  Units
MyBenchmark.case01_FluxBaseline                      ss   10  2.938 ± 0.287   s/op
MyBenchmark.case02_Web                               ss   10  3.281 ± 0.342   s/op
MyBenchmark.case03_WithContextIndexer                ss   10  3.063 ± 0.102   s/op
MyBenchmark.case04_WithLazyInit                      ss   10  2.844 ± 0.129   s/op
MyBenchmark.case05_WithNoVerifyOption                ss   10  2.582 ± 0.060   s/op
MyBenchmark.case06_WithTieredStopAtLevel1Option      ss   10  1.980 ± 0.037   s/op
MyBenchmark.case07_WithSpringConfigLocationOption    ss   10  3.026 ± 0.139   s/op
MyBenchmark.case08_WithJmxDisabledOption             ss   10  2.877 ± 0.097   s/op
MyBenchmark.case09_WithoutLogback                    ss   10  2.904 ± 0.096   s/op
MyBenchmark.case10_WithoutJackson                    ss   10  2.789 ± 0.093   s/op
MyBenchmark.case11_WithoutHibernateValidator         ss   10  2.857 ± 0.084   s/op
MyBenchmark.case12_WithAppCds                        ss   10  2.957 ± 0.079   s/op
MyBenchmark.case13_Exploded                          ss   10  2.476 ± 0.091   s/op
MyBenchmark.case14_ExplodedWithAppCds                ss   10  1.535 ± 0.036   s/op
MyBenchmark.case15_AllApplied                        ss   10  0.801 ± 0.037   s/op

It was really interesting. Thank you!

Spring Boot Thin Launcher and Docker

Mitz — Tue, 02 Oct 2018 12:44:32 +0000

Spring Boot Thin Launcher

Yesterday, I played with the Spring Boot Startup Bench.

Started enjoying spring-boot-startup-bench

Mitz ・ Oct 1 '18 ・ 2 min read

#springboot #java

and noticed it uses Spring Boot Thin Launcher. So today, I played with the Thin Launcher, which enables to create a thin jar.

https://github.com/dsyer/spring-boot-thin-launcher

My Source Code

is here:

https://github.com/bufferings/spring-boot-thin-sandbox

There are 3 directories.

demo-fat: Normal SpringBoot app.
demo-thin: Thin app.
demo-thin-docker: Thin app with docker.

demo-fat

This is just a simple spring boot app with web, though it has no page.

❯ cd demo-fat
❯ ./mvnw clean package
❯ ls -ahl target/demo-fat-0.0.1-SNAPSHOT.jar
-rw-rw-r - 1 bufferings bufferings 16M Sep 30 22:16 target/demo-fat-0.0.1-SNAPSHOT.jar

The size is 16MB.

❯ java -jar target/demo-fat-0.0.1-SNAPSHOT.jar
…
2018–09–30 22:29:04.617 INFO 32639 - - [ main] com.example.demo.DemoApplication : Started DemoApplication in 3.852 seconds (JVM running for 4.464)
…

demo-thin

This is also a simple spring boot app, but with Spring Boot Thin Launcher.

❯ cd demo-thin
❯ ./mvnw clean package
❯ ls -ahl target/demo-thin-0.0.1-SNAPSHOT.jar
-rw-rw-r - 1 bufferings bufferings 11K Sep 30 22:24 target/demo-thin-0.0.1-SNAPSHOT.jar

It's only 11KB. It downloads a launcher and libraries on startup, but actually it uses .m2 directory so it doesn't take so long.

❯ java -jar target/demo-thin-0.0.1-SNAPSHOT.jar
…
2018–09–30 22:30:51.689 INFO 1303 - - [ main] com.example.demo.DemoApplication : Started DemoApplication in 3.012 seconds (JVM running for 4.987)
…

demo-thin-docker

This is a thin app with Docker.

❯ cd demo-thin-docker
❯ ./mvnw clean package
❯ docker build -t demo-thin-docker .

This takes time to download all the dependencies.

❯ docker run -p 8080:8080 demo-thin-docker - thin.debug=true - thin.offline=true
…
2018–09–30 13:42:32.978 INFO 1 - - [ main] com.example.demo.DemoApplication : Started DemoApplication in 3.437 seconds (JVM running for 5.804)
…

The Dockerfile

is as follows:

FROM openjdk:11-jdk-slim
COPY docker/spring-boot-thin-wrapper-1.0.15.RELEASE.jar /thin/wrapper.jar
COPY pom.xml /thin/pom.xml
WORKDIR /thin
RUN jar cvf pom.jar pom.xml
RUN java -jar wrapper.jar \
     --thin.archive=/thin/pom.jar \
     --thin.dryrun=true \
     --thin.debug=true

FROM openjdk:11-jre-slim
COPY --from=0 /root/.m2 /root/.m2
COPY target/demo-thin-docker-0.0.1-SNAPSHOT.jar /app.jar
ENTRYPOINT ["java","-Djava.security.egd=file:/dev/./urandom","-jar","/app.jar"]

FROM

FROM openjdk:11-jdk-slim

There's no reason to use Java 11, we can use the older version :)

COPY wrapper.jar

COPY docker/spring-boot-thin-wrapper-1.0.15.RELEASE.jar /thin/wrapper.jar

I wanted to create a docker layer for the libraries. So I downloaded the wrapper.jar and put it in the repository.

It has a feature of dry-run which doesn't run the main class but just download the libraries into .m2 repository.

COPY pom.xml

COPY pom.xml /thin/pom.xml
WORKDIR /thin
RUN jar cvf pom.jar pom.xml

To create the docker layer, copy the project pom.xml. The wrapper.jar can load a pom.xml in jar file. So I create a jar file which has only the pom file.

dry-run

RUN java -jar wrapper.jar \
     --thin.archive=/thin/pom.jar \
     --thin.dryrun=true \
     --thin.debug=true

Then, execute wrapper jar with the pom.jar in a dry-run mode. If you want to know more detail, you can set --thin.trace=true.

Then all the dependencies of the pom.xml are downloaded into the /root/.m2 directory.

COPY .m2

FROM openjdk:11-jre-slim
COPY --from=0 /root/.m2 /root/.m2

I used the multi stage build, because I don't need other files except for .m2 . So I copied the .m2 directory into the new image.

COPY app.jar and run

COPY target/demo-thin-docker-0.0.1-SNAPSHOT.jar /app.jar
ENTRYPOINT ["java","-Djava.security.egd=file:/dev/./urandom","-jar","/app.jar"]

Then as usual, copy the jar and run it.

Offline Mode

To check there's no download on runtime, I used the - thin.offline=true to run docker with offline mode.

❯ docker run -p 8080:8080 demo-thin-docker - thin.debug=true - thin.offline=true

The good parts with docker

As long as we don't change the dependencies, the library layer is reused. So the build finishes in no time.

That's all for today's fun. And I'm thinking what kind of usage fits to Thin Jar and Fat Jar.

Started enjoying spring-boot-startup-bench

Mitz — Mon, 01 Oct 2018 14:47:50 +0000

Following s1p tweets, I found an interesting one this morning.

Marcin Grzejszczak

@mgrzejszczak

Wat?! @david_syer shows that you can run a @springframework app with Netty in less 0.2 seconds!!!😱😱😱

15:24 PM - 27 Sep 2018

That’s why I enjoyed this repository today.
https://github.com/dsyer/spring-boot-startup-bench

Build failed

When I tried to build it with $ ./mvnw clean install, it failed. Reading the error message, I modified one line and the build succeeded.

https://github.com/bufferings/spring-boot-startup-bench/commit/4218301f4139274abe489ddaeb09058039184da7

Strange Result

Then I ran the benchmark with $ (cd benchmarks/; java -jar target/benchmarks.jar). But it showed a strange result.

The petclinicLatest(Precompute)Thin results were too slow.

Thin?

I didn’t know what the Thin means. Then I reached to https://github.com/dsyer/spring-boot-thin-launcher and understood it’s an experimental library to make the jar file thin and download jars when it starts for the first time.

I read the source code of the benchmark module to know how it runs the target, and added --thin.debug=true to see the debug output.

java -Xmx128m -Djava.security.egd=file:/dev/./urandom \
  -XX:TieredStopAtLevel=1 \
  -jar petclinic-latest/target/petclinic-latest-1.4.2-thin.jar \
  --thin.debug=true \
  --server.port=0

SNAPSHOT?

I forgot to take a memo, but it showed jar resolution log. I noticed the petclinic-latest loaded some SNAPSHOT version libraries.

Mitsuyuki Shiiba

@bufferings

ah, ThinJarWrapper might be slower if the app depends on SNAPSHOT version?

16:35 PM - 29 Sep 2018

I guess every time it would check the SNAPSHOT version of the maven repository. So I modified the pom file to use RELEASE version.

https://github.com/bufferings/spring-boot-startup-bench/commit/17fb6be5511a928ebf26cc49d6044ce6ffdcd73d

Then…

Mitsuyuki Shiiba

@bufferings

It became underatandable speed after I changed the SNAPSHOT to RELEASE version (∩´∀`∩)

16:57 PM - 29 Sep 2018

Though it would be different from the original aim to compare the SNAPSHOT version speed, I enjoyed the repository, and I would like to understand the details of this repository especially Flux benchmarks which is on the tweet picture.

Tried Zalenium to run Selenium tests on scalable containers

Mitz — Wed, 04 Apr 2018 15:09:43 +0000

My friend, who is a great test engineer, mentioned Zalenium yesterday and I was interested in it. So I tried it.

Zalenium?

https://zalando.github.io/zalenium/

After a few hours trial, I've come to like it because

1) it's very easy to try in my local machine,
2) it dynamically starts containers to run Selenium tests,
3) and provides video recording of the tests

1) it's very easy to try in my local machine

Start Zaleinum

# Pull 2 docker images
docker pull elgalu/selenium
docker pull dosel/zalenium

# Run it!
docker run --rm -ti --name zalenium -p 4444:4444 \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -v /tmp/videos:/home/seluser/videos \
  --privileged dosel/zalenium start

(You may want to take care when you mount docker.sock, though it isn't a main topic here.)

Then you can access Live Preview page http://localhost:4444/grid/admin/live and find 2 machines are running.

Run your tests

Now you are ready to run your Selenium tests on Zalenium.

I parepared Geb script here since I like it, but you can use whatever you like.

http://www.gebish.org/

HelloZelenium.groovy

Browser.drive {
  go "http://gebish.org"

  assert title == "Geb - Very Groovy Browser Automation"

  $("div.menu a.manuals").click()
  waitFor { !$("#manuals-menu").hasClass("animating") }

  $("#manuals-menu a")[0].click()

  assert title.startsWith("The Book Of Geb")
}

This is from Geb document. I skip the explanation of this script, because I think you can easily imagine what it does.

GebConfig.groovy

driver = {
  def capabilities = new DesiredCapabilities();
  capabilities.setCapability(CapabilityType.BROWSER_NAME, BrowserType.CHROME);
  capabilities.setCapability(CapabilityType.PLATFORM_NAME, Platform.LINUX);

  def remoteWebDriverUrl = new URL("http://localhost:4444/wd/hub")
  new RemoteWebDriver(remoteWebDriverUrl, capabilities)
}

We can set configuration with GebConfig. We use RemoteWebDriver with the Zelenium Hub URL.

Full source code including import is here: https://gist.github.com/bufferings/a8980ea515a893e21a3a95955ace5dc9

Run the test

groovy HelloZalenium

Then you can see the tests are executed on the live preview page.

So easy!!

2) it dynamically starts containers to run Selenium tests,

Then I would like to run multiple tests simultaneously.

for i in {1..8};do groovy HelloZalenium & done

Then, 8 machines are up and testing! After the tests, it automatically shutdown the containers.

3) and provides video recording of the tests

By default, all the tests are recorded. We can check them on http://localhost:4444/dashboard/#

Configuration Options

In addition, we can ask Zelenium to do something by configuration for example:

driver = {
  def capabilities = new DesiredCapabilities();
  capabilities.setCapability(CapabilityType.BROWSER_NAME, BrowserType.FIREFOX);
  capabilities.setCapability(CapabilityType.PLATFORM_NAME, Platform.LINUX);
  capabilities.setCapability("screenResolution", "1280x720");
  capabilities.setCapability("tz", "Asia/Tokyo");

  def remoteWebDriverUrl = new URL("http://localhost:4444/wd/hub")
  new RemoteWebDriver(remoteWebDriverUrl, capabilities)
}

In this time, we can see FireFox is used with 1280x720 and Asiz/Tokyo timezone.

and more!

I haven't tried yet, but Zalenium has nice features more:

SauceLabs/BrowserStack/TestingBot integration
Support Kubernetes environment
etc...

Conclusion

Zelenium is so convenient and easy to run Selenium Grid with Docker. I would like to keep trying and learning it!

Access host from a docker container

Mitz — Sat, 31 Mar 2018 00:20:38 +0000

This information is as of 2018-03-31 with Docker 18.03.0-ce

I wanted to access host port from a docker container.

For example, an nginx process is running on the host machine with port 8888 open, then I would like to create a container which can curl host:8888.

I know how to discover container-to-container, but don't know how to get host IP address from a container.

Therefore, I wandered the internet.

Precondition

I mainly use Docker for Mac, but sometimes use Linux. So I want my containers to run on both platform.

TL;DR

https://github.com/bufferings/docker-access-host

Host networking mode?

I found Docker has "host networking mode".

https://docs.docker.com/network/host/

With this mode, the container seems to be able access to the host with "127.0.0.1". How easy! I found it!

So I tried it and it works on my Linux machine, but it didn't work on my Mac. Then I found:

https://docs.docker.com/network/network-tutorial-host/#prerequisites

The host networking driver only works on Linux hosts, and is not supported on Docker for Mac, Docker for Windows, or Docker EE for Windows Server.

(´・ω・｀) ok... let me find another way please.

host.docker.internal?

I found I can resolve host.docker.internal as the host IP address in the container on Docker for Mac. Great! That's what I wanted to know! But... where's Linux version of it...?

(´・ω・｀) NotFound...

In addition, I was a little bit confused there're several names:

docker.for.mac.localhost <- from 17.06.0 but now deprecated
docker.for.mac.host.internal <- from 17.12.0-ce-mac46 but now deprecated
host.docker.internal <- from 18.03.0-ce-mac59

https://docs.docker.com/docker-for-mac/release-notes

Finally, my solution is like this.

I created docker-entrypoint.sh :

HOST_DOMAIN="host.docker.internal"
ping -q -c1 $HOST_DOMAIN > /dev/null 2>&1
if [ $? -ne 0 ]; then
  HOST_IP=$(ip route | awk 'NR==1 {print $3}')
  echo -e "$HOST_IP\t$HOST_DOMAIN" >> /etc/hosts
fi

(main entrypoint)

and create Dockerfile call this file as an entry point.

COPY docker-entrypoint.sh /usr/local/bin/

ENTRYPOINT ["/usr/local/bin/docker-entrypoint.sh"]

As a result, Docker for Mac do nothing because it has "host.docker.internal" and Linux version add it into hosts file.

(๑•̀ㅂ•́)و✧

The sample code is here:
https://github.com/bufferings/docker-access-host

I hope host.docker.internal will be added for Linux near future.

If you know better solution, please let me know.