DEV Community: Chuma Takahiro

Kubernetes as OS

Chuma Takahiro — Wed, 04 Nov 2026 15:26:06 +0000

Kubernetes is increasingly incorporating the functionalities of an Operating System.

The term "OS" is multifaceted, and the essential functions vary depending on one's perspective. For software and engineers, the critical functions are those that manage hardware components like the CPU, RAM, and disk. If these are being controlled through Kubernetes configurations, then Kubernetes can be considered an OS.

Established as a Server OS

As of 2024, widely used OSes include Windows, MacOS, and Linux, with Linux having numerous derivatives like Android.

These traditional OSes provide a wide range of functions to operate a single computer.

Kubernetes, however, offers the ability to manage not just a single machine but multiple ones.

In cloud services and data center setups, it's common to have configurations that combine multiple servers. With local area networks (LAN) becoming extremely fast, distributing processing tasks across multiple servers has become a more practical way to expand processing power than merely upgrading a single machine.

At the same time, advancements in semiconductors have made even standalone machines, like personal computers, significantly more powerful. This means it is now feasible to run services, previously reserved for data centers, on a single PC.

Since Kubernetes can operate even on a single machine, it allows for the construction of a complete cloud service on a development PC — this simple success story extends directly to data center operations.

Decline of Docker

Docker is remainining an alternative at the same layer, but looking back from the future, it will likely be seen as "an implementation that existed in the early days."

With Google's rapid development of Kubernetes, Kubernetes reached critical mass and became the de facto standard. This is already a topic of the past, and there's little more to say about it.

A few years ago, before Kubernetes distributions for development PCs were available, Docker was widely used in development environments.

However, as noted earlier, having the same OS for both the data center and the development machine is important. In this context, Docker is now simply an anomaly.

Evolving as a parasite on Linux

Kubernetes is implemented on Linux, and many of its features are essentially Linux itself.

Considering Linux's history as a variant of Unix, which has gradually replaced and unified its foundations, Kubernetes can be seen as the evolved form of Unix in the long run.

While Linux replaced Unix's init with systemd, Kubernetes provides its own method for launching programs, diverging into a separate OS.

There was once a naming controversy over GNU/Linux, which made sense from an engineer's perspective. In light of this, it may be more accurate to recognize the system as GNU/Linux/Kubernetes.

For instance, even now, the lingua franca of Kubernetes remains bash, continuing its lineage from Unix.

Open-source software, compared to proprietary commercial software, offers far more opportunities for replication, supplying symbiotic components to the evolving OS—much like mitochondria in cells.

Incompatibe with Windows and Mac

Kubernetes has already become the de facto OS in the realm of multi-machine computer networks.

Since Kubernetes deliberately incorporates Linux, its compatibility with Windows and Mac is limited.

That said, there are tools available for Windows and Mac, and they aim to ensure basic functionality.

However, because Linux must be implemented as a VM, this results in a nested structure, much like a matryoshka doll. This architecture often leads to issues, particularly when attempting to use internal networks or the PC’s file system.

Problems encountered with Linux VMs on Windows or MacOS may lack productive solutions.

In the 2010s, Macs became popular among engineers, and I too used a MacBook Air at the time.

Initially, it sufficed because Unix toolchains were reasonably well integrated, but as container technologies grew in importance and incomprehensible issues increased, I eventually abandoned the Mac completely.

Replacing Linux's Functions

The previous discussion outlined Kubernetes' progression as a server OS for clustered configurations, and it has already passed its initial milestones.

Applications running on Kubernetes are essentially Linux applications, thus they aren't limited to server-specific software.

It is possible to port and run a broader range of tools on Kubernetes. Since it is primarily just a security-oriented implementation, it doesn't suffer from overheads like intermediate language runtimes, such as Java.

Although there is some effort required to port applications, the rise of container technologies in the open-source community has led to the widespread availability of Dockerfile build scripts. Many of these scripts are even provided by the development projects for various tools.

Self-destructing Tower of Babel

The main benefit of building applications specifically for Kubernetes on a single machine is reproducibility.
This approach offers a solution to the problem of Linux applications that no longer run smoothly on vanilla Linux distributions.

Software often introduces incompatible updates. It's common to encounter situations where App A and App B both require the same underlying library, but A depends on an older version while B relies on a newer one.

When there are version conflicts, one of the applications will likely fail to run.

Package managers of distributions like Ubuntu, along with their development communities, handle these incompatibilities in a unified manner. However, there are limits due to the combinatorial explosion of dependencies.

For example, I encountered an issue when trying to set up Python tools.

While Python's package manager, pip, has long allowed users to install countless libraries, the latest Ubuntu 24.04 now prevents pip installations, instead directing users to use the system's package manager, apt.

When depending on apt, you lose the flexibility to select specific library versions.

If an application's dependencies are complex, you may need to seek alternative methods to manage them.

While it's clear that using a unified package management tool reduces effort compared to juggling multiple tools, even now, some of these tools are starting to falter under the pressure.

Kubernetes apps are possible, but not yet simple

After encountering the limitations of Python package management, I decided to create packages on Kubernetes. Due to the lengthy setup process, I implemented a Tekton pipeline.

In this case, the pipeline had already been customized for development based on a version that was known to work on the server, so it operated without any issues from the start.

The container images are isolated from the host OS, and in this setup, I utilized a straightforward global installation method using pip. Using package management tools provided by programming languages like pip tends to minimize issues.

Moreover, as shown by the ability to reuse server-oriented images, container images continue to work seamlessly even when the hardware changes. Once you build an image, the traditional installation steps become completely unnecessary.

There’s a significant advantage in that you’re largely unaffected by changes to the host OS, such as during distribution upgrades.

The ability to reference directories on the host Linux is easily implemented using a simple hostPath, allowing you to handle the containers similarly to native Linux applications.

From the perspective of the completed package, it would not be incorrect to say that Kubernetes applications can already be run directly on Linux hosts.

However, as there are few tools to support the setup for using applications like desktop apps or CLIs, you will need to craft these setups yourself.

Additionally, attention is needed for scripts running inside the container.

Although they are simply bash scripts, they need to be packed within the YAML manifest.

This reflects a limitation of YAML, and it’s likely an issue that will be resolved through future extensions to the Kubernetes manifest format. However, at present, this area has not been a focus, and there has been no progress since its debut.

When using applications similarly to Linux applications, there is often a need to specify command-line options. In this case, you will need to define variables within the scripts inside the container and find a way to inject them from the environment.

Tekton provides variable substitution, which can be specified from the CLI tool tkn. While this works, specifying lengthy parameters in tkn can be cumbersome, so I created a wrapper script tkn-runner.

Moving forward, I hope to see the emergence of tools designed for directly interacting with containers from the host OS.

Since the container interface is unlikely to change significantly, the setups created now should remain usable in the future without much hassle.

The era of crafting silos

While containerization of applications has become the norm in data centers, the realm of desktop applications is still in development.

As evidenced by the introduction of the snap package manager in Ubuntu, there is a rational basis for containerizing client environments like PCs.

Each distribution has been working toward commonizing libraries, but the reality is that they have been unable to fully resolve inconsistencies between applications, leading to the necessity of using containers alongside them.

If customization is not required, it is still convenient to use the OS's standard package manager.

However, if you need to craft your own environment, you can leverage the standardized processes of Kubernetes on a Linux desktop.

Reverse Terraforming GCP Load Balancers with OpenTofu

Chuma Takahiro — Wed, 04 Nov 2026 15:06:01 +0000

GCP Load Balancers are often built using the web console or CLI, which makes backup and restoration difficult. By using OpenTofu or Terraform, you can implement Infrastructure as Code (IaC) even for existing resources.

By preparing an import.tf like the following and running tofu plan -generate-config-out=export.tf, you can export the current configuration. Since tofu plan queries GCP, you must be authenticated with a user who has appropriate access rights to GCP objects via gcloud auth application-default login.

# gcloud compute forwarding-rules list --global --uri 
import {
  id = "projects/<project-id>/global/forwardingRules/<object-id>"
  to = google_compute_global_forwarding_rule.<unique-name>
}

# gcloud compute target-https-proxies list --global --uri
import {
  id = "projects/<project-id>/global/targetHttpsProxies/<object-id>"
  to = google_compute_target_https_proxy.<unique-name>
}

# gcloud compute target-http-proxies list --global --uri
import {
  id = "projects/<project-id>/global/targetHttpProxies/<object-id>"
  to = google_compute_target_http_proxy.<unique-name>
}

# gcloud compute url-maps list --uri
import {
  id = "projects/<project-id>/global/urlMaps/<object-id>"
  to = google_compute_url_map.<unique-name>
}

# gcloud compute backend-buckets list --uri
import {
  id = "projects/<project-id>/global/backendBuckets/<object-id>"
  to = google_compute_global_backend_bucket.<unique-name>
}

# gcloud compute backend-services list --uri
import {
  id = "projects/<project-id>/global/backendServices/<object-id>"
  to = google_compute_backend_service.<unique-name>
}

# gcloud compute health-checks list --uri
import {
  id = "projects/<project-id>/global/healthChecks/<object-id>"
  to = google_compute_health_check.<unique-name>
}

# gcloud compute firewall-rules list --uri
import {
  id = "projects/<project-id>/global/firewalls/<object-id>"
  to = google_compute_firewall.<unique-name>
}

# gcloud compute ssl-certificates list --global --uri
# NOTE: TLS certificates can be used for configuration changes if managed. Unmanaged ones are hard to manage declaratively.
import {
  id = "projects/<project-id>/global/sslCertificates/<object-id>"
  to = google_compute_ssl_certificate.<unique-name>
}

# gcloud compute addresses list --global --uri
# NOTE: IP addresses are hard to manage declaratively.
import {
  id = "projects/<project-id>/global/addresses/<object-id>"
  to = google_compute_global_address.<unique-name>
}

For the id, specify the URL of the target resource obtained with gcloud compute <gcp-object-type> list. For to, specify the type corresponding to the GCP object as the prefix, and provide a <unique-name> for identification within OpenTofu.

The exported configuration file will likely not be optimal, so refactoring is necessary. You will likely want to remove many attributes that tofu specifies by default. Since the meaning of each field is not always self-evident even if concise, it's important to verify them in the reference documentation. Additionally, using for_each can help consolidate blocks where only a few parameters differ, maintaining code brevity.

The workflow consists of checking the definitions with tofu plan and applying changes with tofu apply.

Management Points

As shown in the previous examples, Load Balancers have many associated objects, which can make the configuration difficult to understand later. Bundling them by purpose in OpenTofu HCL makes management easier. Especially when building multiple clusters within a project, native tools like the Cloud Console list GCP objects by type, which can obscure dependency relationships.

Firewalls

Among Load Balancer related objects, firewall rule settings can be the most difficult to understand. There are prerequisites described in Firewall rules for load balancing. Without firewall settings, health checks will fail first. A point often not explicitly mentioned in the documentation is that actual traffic also follows these conditions. If you only permit the health check target port and don't configure settings for actual traffic on a different port, you may experience a confusing situation where the console shows everything is up, but requests time out. Since firewalls are global objects, it's almost impossible to track which cluster depends on which rule without codifying them.

Adding `depends_on` Blocks

depends_on blocks are useful for structuring code. These are not recognized automatically and must be added manually. You define them using the HCL names:

resource "google_compute_target_https_proxy" "proxy_set" {
  depends_on = [
    google_compute_url_map.url_map,
    google_compute_firewall.allow_hc_v6,
    google_compute_firewall.allow_hc_v4,
    google_compute_firewall.allow_glb_v6,
    google_compute_firewall.allow_glb_v4,
  ]
}

As in this example, there are constraints such as not being able to mix IPv6 and IPv4 or HTTPS and HTTP within a single object. As the Load Balancer configuration grows, defining dependencies allows tofu plan to detect inconsistencies before applying changes. Furthermore, there are actual dependencies between GCP objects; requesting a build in a flat manner can result in errors due to missing prerequisite objects. depends_on ensures that execution waits according to the definitions, leading to stable behavior during clean builds.

Obtaining IP Addresses

One strategy is to not manage GCP IP addresses in HCL. While they can be created with tofu, IP addresses are not resources where you can "gain" a specific desired address, so the effect of declaration is weak. If you simply obtain and lock an IP address from the Cloud Console, you can just specify it in google_compute_global_forwarding_rule.ip_address. If you want to obtain an ephemeral IP, specify IPV4|IPV6 in google_compute_global_forwarding_rule.ip_version. After startup, lock it as a static IP in the Cloud Console and update the HCL accordingly. SSL certificates are handled similarly; if you are registering and updating them on GCP via an external process, you can simply reference those objects.

Cleanup after Export

Files generated by tofu plan -generate-config-out contain many fields automatically assigned by GCP and read-only attributes that do not need management in HCL. Leaving these can make the code hard to read and may cause unnecessary drifts during future provider updates.

Specifically, you should review or remove the following fields:

id, self_link: Attributes fixed after resource creation.
creation_timestamp: The creation date and time.
project: If configured at the provider level, it doesn't need to be specified for each resource.
fingerprint: Values for preventing update conflicts, often found in backend_service. Not needed during initial import.

Consolidating common values such as project IDs and regions into locals or variables makes it easier to deploy to multiple environments.

Verification and Achieving Zero-drift

After completing the tofu apply with the import blocks, immediately run a standard tofu plan to verify that it shows "No changes".

OpenTofu compares the actual GCP configuration with the terraform.tfstate (state file), which records the state of managed resources. Since the state file plays a critical role as the "source of truth" in IaC, establishing a zero-diff state immediately after import is a fundamental prerequisite for stable long-term operation.

If there are differences, the generated HCL definitions slightly mismatch the actual configuration on GCP.

Be particularly careful if a resource recreation (Replace) occurs, such as "Plan: 1 to add, 0 to change, 1 to destroy". Recreating a Load Balancer can lead to IP address changes or service downtime. You must carefully examine the differences and adjust the HCL to match the actual configuration.

Standard Workflow Preparation

Once IaC implementation is complete, setting up a safe operational environment for the entire team is essential. The codebase should be tracked using version control systems like Git.

.tfstate: The state file (.tfstate) is the only source of truth defining the current state of your system. Even if you have the HCL, losing this file means OpenTofu can no longer recognize your existing resources. This creates a severe risk where an accidental apply might attempt to recreate or overwrite resources that are already in production, leading to potential collisions or unintended destruction, and requiring a dangerous, labor-intensive re-import process to recover.
.terraform.lock.hcl: Pin the google provider version in a required_providers block and include .terraform.lock.hcl in the repository to prevent unexpected behavior from provider specification changes.

Additionally, automatically running tofu plan during pull requests via CI/CD ensures that the impact of changes is visualized beforehand and prevents unintended infrastructure changes from being mixed into production.

Integration with Google Kubernetes Engine

This method assumes a configuration connecting from a backendService to a Service with Standalone NEG. It can connect to standard Kubernetes Services and Pods, but adding annotations to the Service and setting up to accept GCP health checks is necessary.

This approach foregoes the adoption of Gateway API. Even if you could build using Gateway API, it is possible to export the full set of GCP objects with OpenTofu. However, in reality, there are many cases where Gateway API cannot fully support requirements—for example, it has long lacked support for housing a backendBucket in the same Load Balancer. Without Gateway API, this network access layer won't be codified, necessitating the use of HCL.

Dual-stack NEGs

Network Endpoint Groups are added via annotations, but the dual-stack setting is determined by the Service spec. If the default is IPv4 only, the following setting allows the NEG to also accept IPv6:

apiVersion: v1
kind: Service
metadata:
  annotations:
    cloud.google.com/neg: '{"exposed_ports": {"443": {"name": "web-neg"}}}'
spec:
  type: ClusterIP
  ipFamilyPolicy: PreferDualStack
  ipFamilies:
  - IPv6
  - IPv4

In this case, the application within the Pod must obviously listen not only on 0.0.0.0 but also on [::]. Also, the order of ipFamilies is significant.

GKE のコンテナネイティブ負荷分散

Chuma Takahiro — Wed, 04 Nov 2026 14:42:49 +0000

GKE(Google Kubernetes Engine)は、HTTPS サービスを Kubernetes Gateway/Ingress を用いて公開できます。
非 HTTP のサービスを公開するには、外部 TCP/UDP ネットワーク負荷分散を別途構築し Service へのルーティングを設定します。

GKE は 1 クラスタで共用できますが、TCP ロードバランサは公開するポートごとに 1 つずつ追加する構成になります。
また、HTTPS サービスであっても GCP の付加機能を利用したい場合には手動構成が必要になる場合があります。

NEG の作成

リクエストを処理する Network Endpoint Group(NEG)までは、kubernetes で作成できます。

Service にcloud.google.com/negアノテーションを追加すると、GCP のスタンドアロン NEG が作成されます。
対応する Deployment などの Pod はごく普通の構成で動作します。

apiVersion: v1
kind: Service
metadata:
  name: mail
  labels:
    name: mail
  annotations:
    cloud.google.com/neg: '{"exposed_ports": {"25": {"name": "mail"}}}'
spec:
  type: ClusterIP
  selector:
    name: mail
  ports:
  - port: 25
    targetPort: mail-smtp

リソース状況は、kubectl get ServiceNetworkEndpointGroups [-o yaml <NEG name>]コマンドで確認できます。
NEG が不要になった場合にはkubectl deleteで Service を削除すると同時に削除されます。

exposed_portsアノテーションのポート番号には、ロードバランサーがアクセスする番号を指定する必要があります。
TCP ロードバランサーの場合は公開するポート番号、https ロードバランサーの場合には 443 または 80 です。

exposed_portsにはspec.ports.[].portの番号を指定するため、Pod が公開ポートと異なるポートをサーブしている場合にはtargetPortを用いて変換する必要があります。

NEG 作成エラー

なお、NEG はゾーン内で同名オブジェクトを重複作成できず、ゾーンが異なる場合には重複可能です。ゾーンはk8sクラスタのロケーションで決まります。

重複など作成エラー時には、k8s Service は作成されるものの NEG が追加されない、という分かりづらい挙動になります。
kubectl describe service <service-name>を実行することでエラーログを確認できます。Events欄の具体的なメッセージを解釈することが重要です。NEG の作成に失敗している場合にも、ログレベルはWARNINGにとどまります。

ロードバランサの作成

HTTP サービスには ingress があり、GKE ingress はコンテナネイティブ負荷分散に対応しています。
一方、TCP サービスには対応する Kubernetes サービスがなく、gcloud computeコマンドで手動構築します。

依存関係に沿って health check, backend service, TCP proxy, forwarding rule の順で作成します。

Backend service 作成

$ gcloud compute health-checks create tcp <tcp-basic-check> --use-serving-port

--use-serving-portはサービスのポート設定を参照して監視します。

$ gcloud compute backend-services create <gke-bs> \
  --global --protocol TCP \
  --global-health-checks --health-checks <tcp-basic-check> \
  --timeout 5m

TCP プロキシが global スコープのサービスであるため、backend service は--globalで作成します。

$ gcloud compute backend-services add-backend <gke-bs> --global \
  --network-endpoint-group=<mail> --network-endpoint-group-zone=asia-northeast1-a \
  --balancing-mode CONNECTION --max-connections-per-endpoint 10

backend service のリージョン(global)に加えて NEG の zone も指定する必要があります。

TCP プロキシ作成

$ gcloud compute target-tcp-proxies create <gke-lb-proxy> --backend-service <gke-bs> [--proxy-header PROXY_V1]

IP 保持のために proxy プロトコルを使いたい場合には、target-tcp-proxies作成オプションでPROXY_V1を指定します。

HTTPS プロキシ作成

HTTPS サービスは Ingress などの k8s Gateway で構築でき、基本的な構成にはgcloudによるネットワーク構築は不要です。

ロードバランサが提供する独自機能を利用したい場合には、target-tcp-proxyに換えて HTTPS 向けにtarget-https-proxy, url-maps, ssl-certificatesのセットを作成します。
HTTPS ロードバランサのオブジェクトは複雑であるため、とくに初回は Web コンソールから作成した方が手軽でしょう。

url-mapsのルーティング先がbackend-servicesを指す構成となり、NEG 経由で各 k8s Service に到達します。
health-checksとbackend-services, forwarding-rulesは TCP プロキシと共通です。

フォワーディング・ルール作成

$ gcloud compute forwarding-rules create <tcp-gke-forwarding-rule> --global --target-tcp-proxy <gke-lb-proxy> --ports 25 --address <some-static-ip>

--addressオプションは、事前に確保しておいた静的 IP を指定します。複数の転送ルールを作成する際、ポートが重複しなければ同一 IP に設定できます。

ルーティングオブジェクトの管理

target-tcp-proxy, forwarding-rules, backend-services, health-checksの 4 種は GCP 独自オブジェクトです。
これらを kubernetes のようにコードとして管理するには Terraform を利用するのが手軽です。

OpenTofu のエクスポート機能を利用することで、gcloudコマンドで作成したオブジェクトをエクスポートできます。

クラスタ移行

上の手順でグローバルの外部 TCP プロキシロードバランサを作成した場合、構築した 4 つの GCP オブジェクトのうちbackend-servicesが参照する NEG を変更するだけでクラスタ移行が完了します。
プロキシはグローバルにサービス提供しているためそのまま流用でき、IP アドレスも変わりません。

k8s の新クラスタを作成し、Service や Pod 群をデプロイ
- Service マニフェストのアノテーションにより、NEG は新クラスタのゾーンに自動作成
- クラスタを同一ゾーンに作成すると、NEG の重複エラーになる点に注意。アノテーションの編集が必要
backend-servicesの設定変更
- gcloud または terraform を用いる
- health-checksはグローバルオブジェクトであり、同一サービスならゾーンが変わっても流用できる

terraform を使う場合、backendServicesのresource.backend.groupの URI を書き換えることで、backendServices が指す NEG を変更できます。
次の例では、zones以下を新クラスタのロケーションに合わせて変更します（該当箇所のみ抜粋）。

resource "google_compute_backend_service" "tfer--mail" {

  backend {
    group = "https://www.googleapis.com/compute/v1/projects/<some-project-name>/zones/asia-northeast1-b/networkEndpointGroups/<NEG-name>"

  }

}

編集できたら、標準手順に沿ってterraform planで確認のうえterraform applyで設定できます。
この過程でhealth-checksも流用します。

ただし、構築時のgcloudオプションしだいでリージョン TCP ロードバランサも作成可能であるため、着手前に構成の特定は必須です。

リージョン TCP ロードバランサの場合には、全体的に再構築が必要で、IP アドレスも変わるため最終的に DNS 設定変更で切り替えることになります。