DEV Community: Mustafa ERBAY

Declarative Server with NixOS: A Reproducible Infrastructure

Mustafa ERBAY — Tue, 18 Aug 2026 02:02:58 +0000

NixOS's Declarative Model

NixOS is a Linux distribution that defines package management and system configuration within a single Nix expression. Packages defined in nixpkgs, their dependency trees, and build options are described via Nix expressions; these expressions are stored deterministically in the Nix store. System services, network settings, user accounts, and filesystem layout are declared in configuration.nix; when the file changes, the nixos-rebuild command can reevaluate the entire system.

This approach embraces the principle that configuration is always a data source. The same configuration.nix produces identical results on two different machines, forming the core of the Infrastructure as Code (IaC) philosophy. NixOS also tracks the exact version and configuration of every package, thereby providing technical reproducibility.

Creating a Basic Configuration File

The following configuration.nix example defines basic SSH access, an nginx service, and a postgresql database. After placing the file at /etc/nixos/configuration.nix, run the nixos-rebuild command.

{ config, pkgs, ... }:

{
  imports = [ <nixpkgs/nixos/modules/installer/scan/not-detected.nix> ];

  boot.loader.grub.device = "/dev/sda";

  networking.hostName = "demo-server";

  services.openssh.enable = true;

  services.nginx = {
    enable = true;
    virtualHosts."example.com" = {
      root = "/var/www/example";
    };
  };

  services.postgresql = {
    enable = true;
    package = pkgs.postgresql_14;
    dataDir = "/var/lib/postgresql/14/data";
    authentication = {
      enable = true;
      users = [
        { name = "admin"; password = "securePassword"; superuser = true; }
      ];
    };
  };
}

After saving this configuration file, run the following command:

sudo nixos-rebuild switch

The command compiles the packages and brings the services up as defined. The output shows the building and activating phases; an example line:

building Nix store path... done
activating configuration... done

Warning: The nixos-rebuild switch command modifies the current system configuration. In production environments, it is recommended to verify changes in a test environment before applying them.

Managing Services: PostgreSQL and Nginx

After the declarative definition, the actual state of the services is inspected with traditional systemctl commands. The systemctl output shows when the service was activated and the location of its unit file, allowing you to verify the effect of the Nix configuration on the system.

systemctl status postgresql
systemctl status nginx

Example postgresql status:

● postgresql.service - PostgreSQL RDBMS
   Loaded: loaded (/etc/systemd/system/postgresql.service; enabled)
   Active: active (running) since Thu 2023-07-10 12:34:56 UTC; 3min ago
   Main PID: 1342 (postgres)

A similar output is obtained for nginx. This step demonstrates that the parameters defined in configuration.nix are reflected on the actual system. If a misconfiguration is discovered, simply correct the relevant section in configuration.nix and rerun nixos-rebuild switch; the system will automatically activate the new version.

Reproducibility and Disk Image

To demonstrate reproducibility, copy the same configuration.nix file to another machine and run the same nixos-rebuild switch command. Additionally, NixOS’s nixos-rebuild build-vm command converts the configuration into a QEMU image that can be tested in a CI environment.

nixos-rebuild build-vm

This command produces a disk.qcow2 file under a result directory. In a CI pipeline you can run this image as follows:

- name: Test NixOS VM
  run: |
    qemu-system-x86_64 -drive file=result/disk.qcow2,format=qcow2,if=virtio \
      -m 2048 -nographic -serial mon:stdio -snapshot

Warning: The nixos-rebuild build-vm command creates a virtual‑machine image, and it is important to validate the image in a test environment before running it. Especially before moving to production, ensure the image starts the expected services.

NixOS Architecture and Package Management

NixOS’s package management is built on a pure functional model. Each package is produced by a Nix expression that defines only its inputs (source code, dependencies, build options). This approach guarantees the same output from the same inputs, so the same package on different machines shares the same hash. The nix-env -iA command manages user‑level package installations, while nixos-rebuild integrates system services.

nix-env -iA nixpkgs.htop

Package version certainty is achieved by pinning the nixpkgs channel to a specific revision. For example, the following commands lock to a particular channel version; updating the channel is done with nix-channel --update.

nix-channel --add https://nixos.org/channels/nixos-23.11 nixos
nix-channel --update

Note: Channel URLs and version numbers should be verified against the official NixOS documentation.

Security and Updates

NixOS applies security patches atomically to all packages in the system. The nixos-rebuild switch --upgrade command pulls the latest packages from the defined channels and upgrades the system. This process requires no manual intervention via systemctl; all services are restarted with the new packages.

sudo nixos-rebuild switch --upgrade

When a security vulnerability is discovered, updating the nixpkgs channel to the relevant revision is sufficient. Once the channel revision is updated, the system automatically adopts the new version.

CI/CD Integration: GitHub Actions and NixOS

Integrating declarative configuration into a CI/CD pipeline can be done by adding nix commands directly to a GitHub Actions workflow. The example workflow below builds the configuration.nix in a test VM, runs unit tests, and, if successful, stores the artifact.

name: NixOS CI
on:
  push:
    branches: [ main ]
jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - name: Install Nix
        run: |
          curl -L https://nixos.org/nix/install | sh
          . $HOME/.nix-profile/etc/profile.d/nix.sh
      - name: Build VM
        run: nixos-rebuild build-vm
      - name: Run Tests
        run: |
          qemu-system-x86_64 -drive file=result/disk.qcow2,format=qcow2,if=virtio \
            -m 2048 -nographic -serial mon:stdio -snapshot -display none \
            -monitor none -no-reboot -smp 2 -cpu host
      - name: Upload Artifact
        uses: actions/upload-artifact@v3
        with:
          name: nixos-vm-image
          path: result/disk.qcow2

This workflow shows how a declarative configuration can be automatically tested and stored as an artifact. A failed build in the CI process is quickly visible in the GitHub UI, preventing a faulty configuration from reaching production.

Rollback and Verification

NixOS simplifies rolling back to a previous system snapshot. When a configuration error is detected, the following command can be used to revert to the last successful build:

sudo nixos-rebuild switch --rollback

Warning: The --rollback option reverts to the previous configuration on the current system. The behavior and availability of this command should be verified in the official documentation for the NixOS version you are using.

After a rollback, it is recommended to check the status of the services again:

systemctl status nginx
systemctl status postgresql

If the rollback succeeds, the services should be in an active (running) state. This mechanism relies on the principle of isolating process errors and re‑evaluating only the affected layer, without needing to rebuild the entire system.

Limitations and Improvement Opportunities

The strengths of the declarative approach include clear version compatibility and explicit package dependencies. However, some situations arise where the exact version you need is not available, or special build flags are missing. In such edge‑cases, you need to customize packages using overrideAttrs or packageOverrides, which can increase the complexity of the configuration file.

{ pkgs, ... }:

{
  nixpkgs.overlays = [
    (self: super: {
      myPython = super.python3.overrideAttrs (old: {
        version = "3.10.9";
        src = pkgs.fetchFromGitHub {
          owner = "python";
          repo = "cpython";
          rev = "v3.10.9";
          sha256 = "0v1c6w7zj..."; # Real hash
        };
      });
    })
  ];
}

Another trade‑off is that the image‑building process consumes significant CPU and memory during the compilation phase, which can strain resource planning on CI servers.

Conclusion

The most robust approach to this topic is to treat claims and steps as small, verifiable pieces. Write down your assumptions and rollback plan before applying anything; then independently verify the expected outcome. When the environment, version, or conditions change, do not automatically assume the instructions still hold—re‑examine the relevant official sources. This way, decisions are based on an observable, repeatable process rather than a one‑time recipe.

Official Sources

Writing Idempotent Ansible Playbooks: Safe Change Patterns

Mustafa ERBAY — Mon, 17 Aug 2026 17:55:03 +0000

A playbook returning changed=0 on its second run is a useful signal, but it is not proof of safe automation. A different input, target state, or external service can still produce a different outcome. A more precise goal is: with the same inputs and starting state, another run should create no new side effect; when a change is required, it should be verifiable and recoverable.

State-oriented modules such as package, file, template, user, and service make that goal easier. Free-form commands, external API calls, and custom changed_when expressions put the burden of idempotence back on the playbook author.

Describe State Before Procedures

Write a task as “this state must exist,” not merely “run this command.” These tasks request the same user and directory state on every run:

- name: Ensure application group exists
  ansible.builtin.group:
    name: myapp
    system: true

- name: Ensure application user exists
  ansible.builtin.user:
    name: myapp
    group: myapp
    system: true
    create_home: false

- name: Ensure configuration directory exists
  ansible.builtin.file:
    path: /etc/myapp
    state: directory
    owner: root
    group: myapp
    mode: "0750"

Fully qualified collection names such as ansible.builtin.file remove ambiguity when multiple collections expose the same short module name. Quoting numeric file modes also avoids YAML interpretation surprises.

`--check` Is a Prediction, Not Proof

ansible-playbook site.yml --check --diff asks modules to predict changes without applying them. For modules that support it, --diff also shows file differences. This is valuable during review, but it has two important limits:

Modules do not all support check mode to the same extent.
A command that is not executed cannot reveal its real return code or external effect.

Do not read check-mode output as a guarantee of production behavior. Disable diff output when rendered content contains secrets:

- name: Render application secrets
  ansible.builtin.template:
    src: secrets.conf.j2
    dest: /etc/myapp/secrets.conf
    owner: root
    group: myapp
    mode: "0640"
  no_log: true
  diff: false

The stronger convergence test is to apply the playbook twice in a disposable environment. The first run may make required changes; the second should report no unexpected change.

`creates`, `removes`, and Honest `changed_when` for Commands

ansible.builtin.command cannot generally know whether the program it launches is idempotent. For operations represented by a filesystem artifact, creates or removes can prevent unnecessary execution and provide partial check-mode support:

- name: Inspect schema marker
  ansible.builtin.stat:
    path: /var/lib/myapp/.schema-v2
  register: schema_v2

- name: Run schema migration once
  ansible.builtin.command:
    cmd: /opt/myapp/bin/migrate --to 2
    creates: /var/lib/myapp/.schema-v2
  when: not schema_v2.stat.exists

The first task defines schema_v2 before the condition uses it. Trying to use a variable in a task's when clause while creating that variable with the same task's register is invalid: the result does not exist when the condition is evaluated.

Mark a read-only probe as unchanged explicitly:

- name: Read current application version
  ansible.builtin.command:
    cmd: /opt/myapp/bin/myapp --version
  register: myapp_version
  changed_when: false
  failed_when: myapp_version.rc != 0

changed_when: false does not make a command safe. It only changes Ansible's report. If the command has a side effect, this expression merely hides the change.

Validate a File Before Replacing It

The template module's validate option runs a validation command against a temporary file before moving it to the destination. Ansible substitutes the temporary path for %s; the command is not run through a shell.

- name: Manage nginx configuration
  hosts: web
  become: true
  tasks:
    - name: Deploy nginx configuration after syntax validation
      ansible.builtin.template:
        src: nginx.conf.j2
        dest: /etc/nginx/nginx.conf
        owner: root
        group: root
        mode: "0644"
        backup: true
        validate: /usr/sbin/nginx -t -c %s
      notify: Reload nginx

  handlers:
    - name: Reload nginx
      ansible.builtin.service:
        name: nginx
        state: reloaded

If the file is unchanged, the task does not report a change and the handler is not notified. Even when several tasks notify the same handler, it normally runs once at the end of the relevant play section. That timing matters: if a later task fails, a notified handler may not run under the default behavior. Decide deliberately whether the failure model calls for force_handlers or a controlled meta: flush_handlers; both alter execution semantics.

`block` and `rescue` Do Not Invent a Rollback

rescue runs after a task in its block enters the failed state. It does not catch every failure class, such as an unreachable host or an invalid task definition. A rollback is also fictional if no restorable artifact was created first.

- name: Deploy and verify nginx configuration
  block:
    - name: Install validated configuration
      ansible.builtin.template:
        src: nginx.conf.j2
        dest: /etc/nginx/nginx.conf
        backup: true
        validate: /usr/sbin/nginx -t -c %s
      register: deployed_config

    - name: Verify installed configuration
      ansible.builtin.command:
        cmd: /usr/sbin/nginx -t
      changed_when: false

  rescue:
    - name: Restore the backup created by template
      ansible.builtin.copy:
        src: "{{ deployed_config.backup_file }}"
        dest: /etc/nginx/nginx.conf
        remote_src: true
      when: deployed_config.backup_file is defined

    - name: Verify restored configuration
      ansible.builtin.command:
        cmd: /usr/sbin/nginx -t
      changed_when: false

This is a starting point, not a universal transaction. Service reload and traffic health checks still need environment-specific design. A rollback that has never been exercised is not reliable.

Separate False Idempotence Signals

Some playbooks look clean on a second run because they hide changes rather than prevent them. The common example is adding changed_when: false to a command with side effects. A template that embeds the current time, a random value, or a newly generated token creates the opposite problem: the template module correctly rewrites the file on every run. Fix the unstable input instead of masking the report.

state: latest is not inherently wrong, but it follows a different contract from “same inputs, same result.” When a repository publishes a new package, the task intentionally changes the host. Keep such expected drift separate from convergence tests that require pinned versions, and document the distinction.

For HTTP APIs, a filesystem creates marker may be insufficient. Use an idempotency key when the API supports one. Otherwise, read the current resource, compare it with the desired representation, and write only when they differ. If a request times out after reaching the server, query the resource before blindly repeating a POST.

When a remote marker or lock represents completion, create it atomically only after the operation and its verification succeed. A marker left by a failed operation can cause every later run to skip necessary recovery.

Measure the State After Applying It

An idempotent task result is not the same as a healthy service. A configuration file can be correct while the process cannot reach a new dependency. Add a read-only health check after the change and bound the startup window with explicit retries:

- name: Wait for local health endpoint
  ansible.builtin.uri:
    url: http://127.0.0.1:8080/health
    status_code: 200
    return_content: false
  register: health
  changed_when: false
  retries: 6
  delay: 5
  until: health.status == 200

Retries must not become an infinite wait. When health verification fails, the playbook or its calling deployment layer should state which backup is restored, how the service is reloaded, and how the restored state is verified.

Test Convergence in CI

A safe pipeline should apply at least this sequence:

Run ansible-playbook --syntax-check.
Use ansible-lint to inspect risky patterns and module usage.
Apply the playbook to an isolated test host.
Apply the same playbook again with the same inputs.
Fail on every failed result and every unexpected second-run changed result.
Keep intentional probes separate with an honest changed_when: false.

A successful first run is not enough. Evaluate second-run convergence, handler counts, rendered-file differences, and service health together.

When Does This Decision Become Invalid?

Re-test an idempotence claim when the module version, target package manager, external API behavior, template inputs, or service assumptions change. A latest package state, a time-dependent API, or a template that generates random values can make the same playbook produce a later change.

The objective is not to force every task to display changed=0. The objective is automation that reports real changes honestly, validates before replacement, measures after application, and follows a rollback path that has actually been tested.

Official References

Log Infrastructure with Loki: Label Design and Cost Control

Mustafa ERBAY — Mon, 17 Aug 2026 08:03:10 +0000

Loki deployments often become expensive in the label model before they run out of storage. Turning every user_id, request_id, pod name, or file path into an indexed label does more than make a field searchable: it creates more log streams, encourages smaller chunks, and expands the index. Cost control is therefore not a compression setting to add later. It is a schema decision that belongs before ingestion starts.

This guide does not offer a magic config.yaml to paste into every deployment. It shows how to classify fields, where to place high-cardinality metadata, how to enable retention safely, and how to verify that a change reduced operational cost instead of merely moving it.

Where Loki costs originate

Rather than indexing every word in each log line as a conventional full-text engine would, Loki organizes log streams by label sets. Entries with the same label names and values belong to the same stream. Changing one value creates a different stream. A field such as environment="prod", with a small and stable value set, fits this model. A request_id that changes for every request does not.

High cardinality creates cost in several places. First, ingesters must track more active streams. Second, sparse streams can close chunks before those chunks become usefully large, increasing the number of small objects. Third, the index and query planner must deal with a wider stream set. The impact is not limited to disk usage; memory, object-store requests, and query latency can all increase.

The useful question is not simply, "Do people search by this field?" Ask instead: "Is this field stable and low-cardinality enough to narrow stream selection effectively?" Frequent searches for a high-cardinality field do not automatically make it a good indexed label.

Indexed label, structured metadata, or log body?

The following policy is a practical starting point. Validate the final decision against your real traffic and query patterns.

Field example	Recommended location	Reason
`environment`, `cluster`, `namespace`	Indexed label	Small, relatively stable value set
`service_name`, `app`	Indexed label	Usually the first selector in a query
`level`	Measured choice: label or parsed field	Low value count, but not always needed on every stream
`pod`, `container_id`, `process_id`	Structured metadata	Operationally useful but highly cardinal
`trace_id`, `request_id`, `user_id`	Structured metadata or log body	Can change per request and explode stream count
Message, stack trace, free text	Log body	Evaluate through filtering and parsing after stream selection

Current Grafana Loki documentation recommends moving fields such as pod names and service.instance.id from index labels to structured metadata. Structured metadata attaches a field to an entry without placing it in the indexed label set or embedding it in the message. The feature requires schema v13 or newer and therefore chunk format V4. OTLP ingestion also relies on structured metadata, so do not disable it before checking your schema and ingestion path.

A query makes the separation concrete:

{service_name="checkout", environment="prod"}
  |= "timeout"
  | json
  | trace_id="0242ac120002"

The low-cardinality labels on the first line reduce the stream set. The trace_id filter is evaluated over the selected entries. You can still find a single request without creating a separate indexed stream for every request identifier.

Treat the label budget as a contract

Do not let application teams add indexed labels as an unrestricted list. Every proposed label should have an owner, an expected number of unique values, a lifetime, and at least one example query. "It is small today" is not enough; document how the value set grows with tenants, customers, processes, or pods.

A useful admission review asks four questions:

Does the value set grow over hours, days, or months?
Can service_name, environment, and cluster already select the required streams?
Can the same information live in structured metadata or the JSON body?
Which measurements will prove that the change improved stream and chunk behavior?

Loki's limits_config provides stream and query controls, but copying a universal number from a blog post is unsafe. Measure current p95 and p99 behavior, add headroom for expected growth, and introduce limits per tenant in stages. A limit does not repair a poor schema. It limits how much of the cluster that schema can consume.

Design storage schema and retention together

For new installations, the current Loki documentation recommends store: tsdb with schema: v13. Object-storage cost is not only about how many days of logs you retain; schema choices also affect how index and chunk objects are organized. When changing the schema of an existing system, preserve the old period and add a new period_config entry with an appropriate start date. Do not reinterpret existing data as if it had been written under the new schema.

The following is deliberately a partial configuration. It shows the relationships that matter for retention, not a complete production deployment:

schema_config:
  configs:
    - from: 2026-08-01
      store: tsdb
      object_store: s3
      schema: v13
      index:
        prefix: index_
        period: 24h

limits_config:
  retention_period: 720h # 30 days; derive this from business and legal needs

compactor:
  working_directory: /var/loki/compactor
  retention_enabled: true
  delete_request_store: s3

The bucket name, authentication method, and storage_config are intentionally omitted. Complete them from the current documentation for your Loki version and S3-compatible service. For a schema transition, choose the from date as part of the rollout plan. For a new installation with no existing data, use a valid date in the past as described by the schema documentation.

With TSDB or BoltDB Shipper, the Compactor applies retention. Setting retention_period without enabling retention on the Compactor does not produce the expected deletion behavior. The documentation also requires a 24-hour index period and a configured delete_request_store when retention is enabled. If you add an object-store lifecycle rule, keep it longer than Loki's retention period; deleting chunks first can leave index references to missing data.

Rollback boundary: retention deletion is irreversible. Validate it in a test tenant first, check object-store versioning or backup policy, and monitor the Compactor's deletion work. Reverting the configuration can stop future deletion. It cannot reconstruct chunks that have already been removed.

Bound query cost as well

A sound label schema does not make every query safe. Wide time ranges, weak selectors, expensive regular expressions, and excessive parallelism can cause large object-store reads. Start dashboard queries with low-cardinality selectors, restrict the time range to the operational need, and apply JSON or regex parsing only after stream selection.

If a dashboard or automation can starve other users inside a shared tenant, evaluate query fairness. Loki's query scheduler can use the X-Loki-Actor-Path header to place actors into separate subqueues within a tenant. Generate that header in a controlled Grafana data source or authentication proxy instead of trusting arbitrary end-user input. Confirm the scheduler topology and supported configuration in the documentation for the version you operate before enabling it.

Safe rollout: measure, constrain, then move

Apply a label redesign as a controlled migration rather than a cluster-wide edit:

Calculate the unique-value trend for each label over the last 24 hours and seven days.
Identify the fastest-growing fields, especially request identities and ephemeral infrastructure identifiers.
Move those fields from indexed labels to structured metadata or the log body in the collector configuration.
Release the change to one service or tenant as a canary.
Compare active streams, chunk creation, ingestion rejections, object-store requests, and query latency with the previous period.
Update saved queries and dashboards for the new field location.
Roll back the collector change if expected searches break; keep retention changes in a separate rollout.

Grafana's current documentation recommends Grafana Alloy as the primary way to send logs to Loki, and Alloy can perform label and structured-metadata transformations in the ingestion pipeline. For older Promtail-based installations, check the current support timeline and map pipeline stages to Alloy components in a separate migration plan instead of combining collector migration with label and retention changes.

Conclusion

The main control on Loki cost is not the compression level; it is the label schema that determines stream count. Keep stable, low-cardinality fields that genuinely reduce the search space as indexed labels. Move request, user, and ephemeral infrastructure identifiers to structured metadata or the log body. Treat TSDB v13, Compactor retention, and query limits as parts of the same design. Most importantly, do not call a rollout successful merely because ingestion still works. Require active-stream behavior, chunk efficiency, object-store traffic, and query latency to improve together.

Official Resources

Prometheus Alert Fatigue: Symptom-Based Alert Design

Mustafa ERBAY — Sun, 16 Aug 2026 02:57:50 +0000

In a monitoring infrastructure that generates hundreds of alerts a day, the moment engineers stop reading notifications and start muting them, system observability is completely lost. Alert fatigue is the direct consequence of writing a separate alert rule for every possible anomaly in the system. The solution is a symptom-based alert architecture that focuses not on "Why did CPU hit 90%?", but on "Can users actually get a response from the service right now?"

Symptom-based alert design makes the metrics measuring user-facing outages, latency, and error rates the primary notification triggers, rather than tracking the isolated states of subcomponents. Root causes are not used to fire alerts; instead, they serve as diagnostic dashboard metrics and log correlations once a symptom alert is triggered.

Why Do Cause-Based Alerts Fail?

A cause-based alerting strategy stems from the engineering reflex of "writing a rule for every part that could possibly break." Waking up the on-call engineer whenever disk usage reaches 80% on a server, CPU consumption exceeds a specific threshold, or Redis memory usage climbs is a textbook example of this mindset. However, high CPU usage on a server does not necessarily mean the service provided by that server is degraded; often, it simply indicates that the system is utilizing its hardware resources efficiently.

The fundamental problem created by this approach is that most alerts are non-actionable. When an on-call engineer gets paged at midnight for "Server CPU 85%", checks the system, and sees that users are experiencing zero errors and normal latency, they will completely tune out that alert channel within two weeks. Consequently, when a real database deadlock or network outage occurs, the critical alert gets lost in the noise.

The table below summarizes the operational differences between both approaches:

Criterion	Cause-Based	Symptom-Based
Trigger Focus	CPU, Memory, Disk, Thread count	HTTP 5xx rate, p99 Latency, Queue backlog
Alert Volume	High (Every component screams individually)	Low (Fires only when users are impacted)
Actionability	Ambiguous (Often requires no action)	Clear (Users cannot use the service)
Maintenance Burden	Thresholds must be constantly retuned	Tied to SLO/SLI targets, stable
Coverage	Limited to known failure modes	Catches unknown/novel failures as well

Core Principles of Symptom-Based Alerting: RED and USE

When setting up symptom-based monitoring in a distributed system, two foundational methodologies serve as your guide: RED (Rate, Errors, Duration) for request-driven services, and USE (Utilization, Saturation, Errors) for resource-driven infrastructure components. Applying these two approaches to the appropriate architectural layers cuts out unnecessary noise at the source.

The RED method should be applied to layers directly exposing APIs to users or other services. A service's request rate per second (Rate), how many of those requests fail (Errors), and how long requests take to complete (Duration) directly reflect its health. Regardless of the memory consumption of the container behind the service, if the 5xx error rate exceeds the defined Service Level Objective (SLO) boundary, you have a real symptom that must be addressed.

       +-------------------------------------------------------+
       |                  RED Method (Services)                |
       |  - Rate (RPS)                                         |
       |  - Errors (HTTP 5xx / gRPC Errors) -> ALERT           |
       |  - Duration (p95/p99 Latency)      -> ALERT           |
       +---------------------------+---------------------------+
                                   |
                                   v
       +-------------------------------------------------------+
       |             USE Method (Infrastructure Resources)     |
       |  - Utilization (CPU/RAM Usage)     -> DASHBOARD       |
       |  - Saturation (Queue/Load Average) -> WARNING/DASHBOARD|
       |  - Errors (Disk I/O, Dropped Pkts) -> DIAGNOSIS       |
       +-------------------------------------------------------+

On the infrastructure side (disks, network interfaces, database connection pools), the focus shifts to the USE method. But here is the critical distinction: Utilization alone should rarely be an urgent pager alert. The true focus must be Saturation (queuing/backlog). While 85% disk usage belongs on a dashboard, I/O requests queuing up on disk (I/O saturation) or running out of TCP sockets and dropping packets is an actionable symptom.

💡 Golden Rule: Page Only When Users Are Affected

The only condition that should wake an on-call engineer from their sleep is when the system cannot meet the promised Service Level Objective (SLO) to users, or when it is certain to fail completely in a short window (e.g., within 1-2 hours, such as a disk filling up at a rapid write rate).

Writing User-Centric Alerts with PromQL

When writing symptom-based rules in Prometheus, instead of relying on simplistic instantaneous thresholds (metric > 90), calculations should be based on rates and distributions spanning a defined time window. A well-crafted PromQL query must filter out transient spikes (jitter) and confirm the persistence of the failure.

1. Error Rate Alert

Instead of a simple error count, the error percentage relative to total traffic should be calculated. To prevent alerts from firing due to a single isolated error during low-traffic periods, a minimum request rate condition is included in the query:

groups:
  - name: service_symptom_alerts
    rules:
      - alert: HighHTTPErrorRate
        expr: |
          (
            sum(rate(http_requests_total{status=~"^5.."}[5m]))
            /
            sum(rate(http_requests_total[5m]))
          ) * 100 > 2
          and
          sum(rate(http_requests_total[5m])) > 10
        for: 3m
        labels:
          severity: critical
          tier: api
        annotations:
          summary: "API service returning high rate of 5xx errors"
          description: "Over 2% of incoming requests failed with 5xx in the last 5 minutes. Current error rate: {{ $value | printf \"%.2f\" }}%"

In this rule, the for: 3m clause prevents the on-call engineer from getting paged unnecessarily over momentary 10-second network blips; it requires the issue to persist consistently for at least 3 minutes.

2. Latency / Duration Alert

Percentiles should always be used instead of average latency. An average value masks severe slowdowns experienced by a significant fraction of users. Using the histogram_quantile function, we measure p99 or p95 latency:

      - alert: HighRequestLatencyP99
        expr: |
          histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le, service))
          > 1.5
        for: 5m
        labels:
          severity: warning
          tier: backend
        annotations:
          summary: "p99 Request latency exceeds threshold"
          description: "p99 response time for {{ $labels.service }} has been above 1.5s for 5 minutes. Current value: {{ $value | printf \"%.2f\" }}s"

3. Fast-Depleting Resources (Multi-Window Burn Rate)

For disk space, instead of a static 85% threshold, we use the predict_linear function to estimate when the disk will run out of space based on its current write rate. This ensures no alerts are generated at night for a disk that will take a month to fill, while fast action is taken for a disk that will fill up in 2 hours:

      - alert: DiskFillingFast
        expr: |
          predict_linear(node_filesystem_free_bytes{mountpoint="/"}[4h], 4 * 3600) < 0
          and
          node_filesystem_free_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"} < 0.2
        for: 15m
        labels:
          severity: critical
          tier: infra
        annotations:
          summary: "Root filesystem will fill within 4 hours"
          description: "Root disk on instance {{ $labels.instance }} will be completely exhausted within 4 hours at the current fill rate."

Filtering Noise with Alertmanager: Routing, Grouping, and Inhibition

Even when Prometheus rules are designed around symptoms, a major infrastructure failure can still trigger hundreds of alerts simultaneously. When a core backbone switch goes down, receiving separate "Service Unreachable" alerts for 40 servers and 200 containers behind it represents peak noise. This is where Alertmanager's grouping and inhibition capabilities become indispensable.

Setting up proper inhibit_rules in Alertmanager prevents downstream cascading alerts from flooding your notification channels when a primary outage occurs.

# alertmanager.yml
global:
  resolve_timeout: 5m

route:
  group_by: ['alertname', 'cluster', 'service']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  receiver: 'default-slack'
  routes:
    - match:
        severity: critical
      receiver: 'pagerduty-critical'
      continue: true
    - match:
        severity: warning
      receiver: 'slack-warnings'

inhibit_rules:
  # If the host is completely down, mute service alerts on that host
  - source_match:
      alertname: 'NodeDown'
    target_match_re:
      alertname: 'InstanceDown|HighHTTPErrorRate|HighMemoryUsage'
    equal: ['node', 'instance']

  # If the database is unreachable, mute latency alerts for backend services depending on it
  - source_match:
      alertname: 'PostgresDead'
    target_match:
      tier: 'backend'
    equal: ['environment']

Here, group_wait: 30s waits 30 seconds after the first alert fires to bundle other related alerts in the same group into a single consolidated notification. Meanwhile, repeat_interval: 4h controls how frequently an unresolved issue re-notifies the channel, avoiding unnecessary spam.

ℹ️ Label Matching in Inhibition Rules

The equal parameter in the inhibit_rules block is critical. If you apply a global inhibition rule without verifying that the source and target alerts share the same instance or cluster label, a failure on Host A could accidentally silence a legitimate alert on Host B.

Actionable Alerts and Runbook Architecture

When an alert triggers, the message displayed on the on-call engineer's screen should not merely state a status—it must serve as a concrete remediation guide. Every minute an engineer spends searching for documentation asking "What does this alert mean and what should I do?" directly inflates your Mean Time to Recovery (MTTR).

Every alert rule must include two essential links: a focused Grafana Dashboard URL to inspect the relevant service metrics and a Runbook URL outlining initial mitigation steps.

      - alert: OrderProcessingLagSpike
        expr: |
          rabbitmq_queue_messages_ready{queue="order_processing"} > 5000
        for: 10m
        labels:
          severity: critical
          service: order-pipeline
        annotations:
          summary: "Order processing queue is backing up"
          description: "The number of unprocessed orders in the queue has exceeded 5000 for 10 minutes. Consumer services may be falling behind."
          runbook_url: "https://wiki.internal/ops/runbooks/order-queue-lag"
          dashboard_url: "https://grafana.internal/d/orders/order-pipeline-metrics?var-queue=order_processing"

A well-designed runbook should answer these 4 questions directly and concisely:

What symptom occurred? (e.g., Order queue is backing up; users cannot receive order confirmations.)
What is the initial verification step? (e.g., Check consumer pod logs for OOM kills or connection timeouts.)
What is the temporary mitigation? (e.g., Scale up consumer pods: kubectl scale deployment order-consumer --replicas=10)
Root cause diagnosis: (e.g., Inspect payment gateway response times, check for PostgreSQL lock contention.)

Alert Hygiene and Lifecycle Management

Building a symptom-based monitoring infrastructure is not a one-off project; it requires continuous maintenance. Evolving architectures, newly added microservices, or refactored components can quickly render old alert rules obsolete.

       +-------------------------------------------------------+
       |                  Weekly Alert Review                  |
       |  - Which top 5 alerts fired most in the last 7 days?  |
       |  - Which alerts required ZERO action?                 |
       +---------------------------+---------------------------+
                                   |
                                   v
       +-------------------------------------------------------+
       |              Classification and Improvement           |
       |  - Non-actionable      -> DELETE or move to Dashboard |
       |  - Incorrect threshold -> Adjust PromQL / Time window |
       |  - Genuine symptom     -> Update Runbook              |
       +-------------------------------------------------------+

To maintain alert hygiene, incorporate these rules into your team's operational routines:

The Zero-Action Rule: Any alert that fired more than 5 times in the past month without requiring a single code change, service restart, configuration fix, or incident triage must be deleted immediately or converted into a dashboard metric.
Silencing Discipline: Temporary silences created in the Alertmanager UI must always include an expiration (TTL) and a link to an active tracking ticket or task. Indefinite silences are the leading cause of blind spots in production.
Alert Review Meetings: During weekly on-call handovers, review the total alert count and noise ratio. Any rule that woke up the on-call engineer unnecessarily should be revised immediately via a pull request.

Wrapping Up

Achieving sustainable observability with Prometheus and Alertmanager isn't about writing more alert rules; it's about monitoring the right architectural layer with the right methodology. Instead of drowning in the noise of thousands of cause-based rules, implementing symptom-based alerting grounded in RED and USE principles restores confidence for on-call teams.

Keeping infrastructure metrics (CPU, RAM, connection counts) on Grafana dashboards as diagnostic aids when symptoms fire—while reserving pages exclusively for moments when the user experience is genuinely compromised—is the foundation of operational resilience.

Official Resources

Introduction to GitOps: Pull-Based Deployment Model with Argo CD

Mustafa ERBAY — Sun, 16 Aug 2026 00:01:36 +0000

What is GitOps?

GitOps defines the principle of managing infrastructure and application configurations by using a Git repository as the single source of truth. This approach minimizes human error through declarative definitions and automated synchronization. The core advantage of GitOps is that every change is fully traceable and reversible by the entire team; when a commit is rolled back, the system can automatically revert to its previous desired state.

ℹ️ Why GitOps?

GitOps combines change auditing with Git's robust version control mechanism, significantly simplifying auditing, compliance, and rollback processes.

How Does Argo CD Work?

Argo CD is a controller that monitors Git repositories and automatically pulls and applies application manifests to a Kubernetes cluster. In the Pull-Based model, Argo CD defines the target repository and path via an Application custom resource; the control loop checks the repository every 3 minutes (180 seconds) by default and initiates a synchronization if differences are detected. This process displays real-time sync and health statuses directly in the Argo CD UI.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: demo-app
spec:
  project: default
  source:
    repoURL: https://github.com/example/demo
    targetRevision: HEAD
    path: manifests
  destination:
    server: https://kubernetes.default.svc
    namespace: default
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

In the Application manifest above, setting spec.syncPolicy.automated.prune: true ensures that resources deleted from the Git repository are also automatically removed from the Kubernetes cluster. Setting selfHeal: true ensures that when drift from the state defined in Git is detected within the live cluster, Argo CD automatically reverts those changes to keep the cluster state matched with Git.

What is the Pull-Based Deployment Model?

The Pull-Based deployment model refers to the control plane (controller) pulling and applying configurations from a Git repository independently of manual user interaction. In this model, instead of push commands like kubectl apply, an operator like Argo CD periodically reads the repository and automatically synchronizes differences. The pull approach simplifies firewall rules because it requires only an outbound connection and eliminates the need to expose an external listener within your internal network, significantly reducing security risks.

Argo CD Installation Steps

Installation can be done using Helm charts or kubectl manifests. Here is a simple example using kubectl. The first step is to create the argocd namespace and apply the necessary manifests.

⚠️ Production Environment Warning

The kubectl apply command below installs Argo CD with cluster administrator (cluster-admin) privileges and is a non-HA (High Availability) setup, which is not recommended for production environments. For production, installing via Helm and pinning to a specific Argo CD version is recommended. Additionally, rather than applying manifests directly from https://raw.githubusercontent.com, downloading, inspecting, and then applying them is much safer.

# Create namespace
kubectl create namespace argocd

# Apply Argo CD manifests
# --server-side and --force-conflicts flags are recommended due to CRD size limits.
kubectl apply -n argocd --server-side --force-conflicts -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

Next, configure a LoadBalancer or Ingress to expose the argocd-server service externally. Once the installation is complete, the admin password is stored directly inside a secret, allowing you to log in to the UI.

ℹ️ Port Forwarding for Local Testing

This command is used to access the Argo CD UI from your local machine and should not be used in production environments. The terminal session will remain attached while the command runs.

# Port forward for UI access (for local testing)
kubectl port-forward svc/argocd-server -n argocd 8080:443

After installation, you can authenticate via the CLI using argocd login; this allows you to bypass the UI in script-based automations. For Argo CD v1.9 and later, the initial admin password is stored in the argocd-initial-admin-secret secret under the password key.

⚠️ Security Warning: Change the Default Password

It is strongly recommended to change the default admin password immediately after your first login. You can also delete the argocd-initial-admin-secret secret once the password has been updated.

# Retrieve the admin password (for Argo CD v1.9+)
ADMIN_PASSWORD=$(kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d)
echo "Argo CD Admin Password: $ADMIN_PASSWORD"

# Authenticate via CLI (--insecure flag may be required for local testing)
argocd login localhost:8080 --username admin --password "$ADMIN_PASSWORD" --insecure

Application Deployment Workflow

The Mermaid diagram below shows how a Git commit reaches Kubernetes through Argo CD. This workflow includes the detect change → synchronize → health check → report steps. Logs are recorded at each phase.

In a production-like environment, changes can be inspected beforehand using the dry-run option during sync. In the Argo CD CLI, running argocd app sync --dry-run executes a client-side dry-run by default. For a server-side dry-run, additional configuration or an alternative approach may be needed.

To reproduce this workflow in a lab environment, simply create an Application manifest and replace the repoURL and path fields with your own test repository. After synchronization, you can verify that the deployment succeeded by running kubectl get pods -n <namespace>.

Best Practices and Common Pitfalls

Best Practice 1 – Declarative Manifests: Make manifest files as parameter-driven as possible; overlays based on kustomize or helm allow you to manage environment differences (dev, staging, prod) within a single repository. Common Pitfall 1 – Direct Pushes: Direct push operations like kubectl apply defeat the entire purpose of GitOps and stop Git from being the single source of truth; route all changes exclusively through Git.

Best Practice 2 – Automated Self-Heal: The syncPolicy.automated.selfHeal feature automatically remediates drift (state divergence) without requiring manual intervention. Common Pitfall 2 – Prune Disabled: Failing to clean up deleted resources automatically leads to resource leakage over time; enabling prune: true eliminates this risk.

Best Practice 3 – RBAC Configuration: Using the default admin account in production creates a serious security vulnerability; define project-based roles instead and grant access only to the necessary namespaces. Common Pitfall 3 – Overly Broad Permissions: Granting broad cluster-admin roles can cause a single faulty commit to impact the entire cluster; strictly enforce the principle of least privilege.

Conclusion

Argo CD puts the pull-based model of GitOps into practice, making Kubernetes deployments consistent and observable. Declarative manifests, automated self-healing, and granular RBAC controls ensure the long-term sustainability of this approach. Because the setup and core workflow are straightforward, teams can quickly transition to production; however, if security and resource management practices are neglected, the benefits of GitOps can quickly erode. The next natural step is to integrate your existing CI pipeline with Argo CD and introduce post-sync automated testing and canary deployment strategies.

Official Resources

Lightweight Kubernetes with k3s: Scaling from Single Node

Mustafa ERBAY — Sat, 15 Aug 2026 13:46:34 +0000

Introduction

While working on a production ERP system, we needed a fast and flexible container management system. To meet this need, we chose k3s. k3s is a lightweight Kubernetes distribution. In this article, we will explain how to scale from a single node to multiple nodes using k3s.

Lightweight Kubernetes with k3s

k3s is an open-source project supported by the Cloud Native Computing Foundation (CNCF). k3s is lighter and consumes fewer resources compared to traditional Kubernetes distributions, making it ideal for resource-constrained environments or edge computing scenarios. Therefore, k3s is ideal for use in small and medium-sized projects.

Single Node Deployment with k3s

Deploying k3s as a single node is quite simple. The first step is to create a server and install k3s on it. After installing k3s, you can create a cluster and run containers on this cluster.

# To install the k3s server
# WARNING: Running scripts downloaded from the internet can pose a security risk.
# In production environments, it is recommended to review the script or use alternative installation methods.
curl -sfL https://get.k3s.io | sh -

# To check the status of the k3s cluster
k3s kubectl get nodes

This command installs the k3s server and loads helper programs like kubectl, crictl, ctr. The Kubeconfig file is written to /etc/rancher/k3s/k3s.yaml, and the k3s kubectl command automatically uses this file.

Scaling to Multiple Nodes

Scaling from a single server node to multiple nodes is particularly easy with k3s by adding worker (agent) nodes. This allows you to increase the capacity of your cluster.

Adding Worker Nodes

To add worker nodes, you first need to create these nodes. Then, you can install the k3s agent on them and have them join the existing k3s server cluster.

To have worker nodes join the cluster, you need to obtain a token from your k3s server node. This token is found in the /var/lib/rancher/k3s/server/node-token file on the server node.

# Get the token from the server node (e.g., if the server IP is 192.168.1.100)
sudo cat /var/lib/rancher/k3s/server/node-token

# To install the k3s agent on worker nodes and join the cluster
# Replace <SERVER_IP> with the IP address of your k3s server and <NODE_TOKEN> with the token you obtained above.
# WARNING: Running scripts downloaded from the internet can pose a security risk.
# In production environments, it is recommended to review the script or use alternative installation methods.
curl -sfL https://get.k3s.io | K3S_URL=https://<SERVER_IP>:6443 K3S_TOKEN=<NODE_TOKEN> sh -

This command configures the new node as a k3s agent and registers it with the specified server. After the agent joins successfully, you can verify that the new worker node is visible in the cluster by running k3s kubectl get nodes from your server node.

Security

When creating a lightweight Kubernetes deployment with k3s, it is crucial not to neglect security. The first step is to secure your servers. Then, you can secure your k3s cluster.

Server Security

To secure your servers, you first need to keep them updated regularly. Then, you can protect them with a firewall.

# To update your servers
sudo apt update && sudo apt full-upgrade -y

# To protect your servers with a firewall (e.g., using UFW)
# WARNING: Before enabling the firewall, ensure that you have opened the necessary ports for Kubernetes (e.g., 6443, 10250),
# or cluster communication may be disrupted.
sudo ufw enable

k3s Security

To secure your k3s cluster, it is essential to keep your k3s version up to date, as updates include security patches and improvements. To update k3s, you can rerun the installation script with the same configuration options or manually update the binary.

Additionally, you can apply Kubernetes Network Policies to control traffic between pods. k3s includes a built-in network policy controller by default. Network policies allow you to restrict communication between pods, enabling a zero-trust network model.

# To check the k3s version
k3s --version

# To update k3s (preserving the existing configuration)
# WARNING: Before updating, it is recommended to back up your data and update server nodes before agent nodes.
# Also, avoid large version jumps by updating one minor version at a time.
curl -sfL https://get.k3s.io | sh -

# To create an example network policy (this only creates a template, you need to define its content)
# This command alone does not provide security; the content of the NetworkPolicy YAML file is important.
# kubectl create networkpolicy <policy-name> --from=<source> --to=<target> --port=<port>
# For example, to create a policy that denies all traffic between pods by default:
# kubectl apply -f default-deny-all.yaml
# (default-deny-all.yaml content: https://docs.k3s.io/security/hardening-guide#network-policies)

Conclusion

In this article, we explained how to create a lightweight Kubernetes deployment using k3s and scale it from a single node to multiple nodes by adding worker nodes. k3s is a lightweight and flexible Kubernetes distribution ideal for small and medium-sized projects. When creating a lightweight Kubernetes deployment with k3s, do not neglect security; take measures such as regular updates and network policies for both server and cluster security. The next step is to deploy this distribution in a production environment and use it in a real project.

Official Resources

Setting Up Your Own Tailscale Control Server with Headscale

Mustafa ERBAY — Sat, 15 Aug 2026 08:09:42 +0000

Introduction

Headscale is an open-source application of the Tailscale control server, allowing users to set up their own Tailscale-compatible control servers. This provides more flexibility in managing your mesh-VPN network and customizing security settings. In this article, you will learn the steps to set up your own Tailscale control server with Headscale.

What is Headscale?

Headscale is an open-source application of the Tailscale control server. Tailscale is a modern VPN solution built on WireGuard, enabling users to securely access remote devices. While Tailscale's managed service does not offer a self-hosted control server option, Headscale provides an open-source alternative for self-hosting and hobbyist users.

Setting Up Your Own Tailscale Control Server with Headscale

To set up your own Tailscale control server with Headscale, follow these steps:

Step 1: Install Headscale

To install Headscale, you first need a server or virtual machine. After setting up your server or virtual machine, you can install Headscale. Headscale is officially available on Docker Hub and can be installed using Docker.

In production environments, it is recommended to use a specific version instead of the 'latest' tag. The following command pulls the latest stable version of Headscale:

docker pull headscale/headscale:0.23.0 # Example version, check the current stable version

Warning: Instead of the 'latest' tag, it is recommended to always use the current stable version number (e.g., 0.23.0) from Headscale's official GitHub release page when pulling the Docker image. This helps avoid unexpected changes.

Step 2: Configure Headscale

After installing Headscale, you can configure it using the following commands. This command runs Headscale on port 8080 by default and mounts its configuration to the /etc/headscale directory.

First, you need to create and edit the Headscale configuration file (e.g., config.yaml) on your server. Headscale looks for the configuration file in /etc/headscale, $HOME/.headscale, or the current working directory. You can download an example configuration file from the Headscale GitHub repository.

# Example: Create the configuration directory
mkdir -p /var/lib/headscale # Or your preferred directory

# Example: Download the current configuration example from Headscale's GitHub repository
# Replace HEADSCALE_VERSION with the current stable version (e.g., 0.23.0)
# wget -O /var/lib/headscale/config.yaml https://github.com/juanfont/headscale/raw/v${HEADSCALE_VERSION}/config-example.yaml
# nano /var/lib/headscale/config.yaml # Edit the configuration file

Then, start Headscale with Docker:

docker run -d --name headscale \
  -p 8080:8080 \
  -v /var/lib/headscale:/etc/headscale \
  headscale/headscale:0.23.0 # Example version, use the current stable version

This command runs Headscale on port 8080 and mounts its configuration from /var/lib/headscale to /etc/headscale in the container.

Step 3: Install Tailscale

After installing Headscale, you can install the Tailscale client. Tailscale has different installation methods for various platforms. For more information, refer to the Tailscale documentation.

Step 4: Connect Tailscale to Headscale

After installing the Tailscale client, you can connect it to your Headscale server using the following command:

tailscale up --login-server http://<HEADSCALE_SERVER_ADDRESS_OR_IP>:8080

This command connects the Tailscale client to your Headscale server. Replace <HEADSCALE_SERVER_ADDRESS_OR_IP> with the actual IP address or domain name of your Headscale server. For example, if your Headscale server is running at myheadscale.example.com, the command would be tailscale up --login-server http://myheadscale.example.com:8080.

Managing Your Own Tailscale Control Server with Headscale

After setting up your own Tailscale control server with Headscale, you can use Headscale to manage your mesh-VPN network. Headscale provides full control over the control plane, in addition to many features found in Tailscale's managed service.

Security Settings

Headscale offers similar security options to Tailscale and provides full control over your environment. For example, with Headscale, you can use your own SSL/TLS certificates and manage TLS configuration.

Network Management

Headscale provides various tools for managing your network. For example, with Headscale, you can manage access control lists (ACLs), control node registration, and customize your network security settings.

Conclusion

Setting up your own Tailscale control server with Headscale provides more flexibility and control in managing your mesh-VPN network and customizing security settings. In this article, you learned the steps to set up your own Tailscale control server with Headscale. Headscale is an open-source application of the Tailscale control server, allowing users to self-host their own Tailscale-compatible networks.

Official Resources

Syncthing for Cloudless Synchronization: Conflicts and Versions

Mustafa ERBAY — Sat, 15 Aug 2026 03:06:51 +0000

Introduction

While working on a production ERP, I experienced issues with file synchronization. Instead of storing files in cloud services, we preferred to store them on our own servers. However, this led to some problems with file synchronization. To address these issues, we decided to use Syncthing. Syncthing is a tool that provides cloudless synchronization. In this article, I will discuss the advantages and disadvantages of using Syncthing, as well as what can be done regarding conflicts and versions.

Advantages of Syncthing

Syncthing has several advantages when it comes to file synchronization. Firstly, it is not dependent on cloud services; your data is stored only on your devices, and there is no central server. This provides more control over file storage and synchronization. Additionally, Syncthing's source code is open, and it is licensed under the Mozilla Public License 2.0 (MPLv2), making it verifiable and reliable for developers.

Disadvantages of Syncthing

Syncthing also has some disadvantages. The initial setup and configuration, especially regarding network settings, can be somewhat complicated at first. Moreover, for Syncthing to synchronize files between devices, these devices need to be on the same local network or an internet-connected network. If devices are on different networks, direct connection may require port forwarding or the use of relay servers, which can result in lower performance.

Conflicts and Versions

Syncthing supports file versioning. This feature allows tracking the history of changes made to a file and can store older versions in a .stversions folder using strategies like "Simple File Versioning."

In the event of a conflict, Syncthing does not automatically resolve it; instead, it preserves both conflicting files. When a file is modified simultaneously on multiple devices and the contents differ, Syncthing recognizes the conflict and renames one of the files to filename.sync-conflict-<date>-<time>-<changingDeviceID>.<extension>. Typically, the file with the older modification time is marked as the conflict and renamed. This allows the user to have both versions and manually decide which one is the "best" version.

syncthing --version

The above command will display Syncthing's version.

Settings and Configuration

When you start Syncthing for the first time, it automatically generates a device ID and a default configuration file. Syncthing does not use a central account system; devices recognize and pair with each other through unique device IDs.

syncthing generate

The above command generates a default configuration file (config.xml) and cryptographic keys for Syncthing. Warning: If Syncthing is already running or there is an existing configuration file, this command will create a new one and overwrite the existing file. To update or change an existing configuration, it's usually preferable to use Syncthing's web-based graphical user interface (GUI) or edit the config.xml file directly.

Conclusion

Syncthing is a cloudless, open-source, and secure file synchronization tool. It has advantages and disadvantages. It supports file versioning and manages conflicts by creating sync-conflict files, allowing the user to manually resolve them. To use Syncthing, there's no need to create a central account; instead, device IDs are used for pairing, and the configuration file is automatically generated on the first run or can be manually generated with the syncthing generate command. The next step is to start using Syncthing and synchronize files by pairing devices and configuring folders.

Official Resources

Code Review Culture: Small PRs, Fast Flow, Low Friction

Mustafa ERBAY — Sat, 15 Aug 2026 01:10:08 +0000

Code Review Culture

Code review is a critical process that enables a software team to collaborate and produce high-quality code. Building a solid code review culture helps boost team collaboration and efficiency. Small PRs (Pull Requests) and a fast flow are essential to making the review process much more effective. In this post, we will explore the importance of code review culture and what you can do to maintain a fast flow through small PRs.

Code reviews impact not just code quality, but also the way a team collaborates. Establishing a healthy review process helps build mutual trust among team members. Furthermore, code reviews help onboard new members quickly and accelerate their learning curve.

Small PRs

Small PRs are vital for keeping the code review process efficient. They allow the code to be split into bite-sized pieces, making it possible to review each part thoroughly on its own. This improves overall code quality and makes bugs significantly easier to catch. Small PRs get reviewed faster, generate better feedback, and are much easier to revert if something goes wrong.

To create small PRs, pick an isolated part of the codebase and turn it into its own dedicated pull request. For instance, small changes such as refactoring a single function or introducing a small feature should each live in their own PRs.

# Example of creating a PR
def calculate_area(length, width):
    return length * width

# Modified code
def calculate_area(length, width):
    if length < 0 or width < 0:
        raise ValueError("Length and width must be greater than zero")
    return length * width

Fast Flow

A fast flow is crucial for making the code review process efficient. It ensures that code gets reviewed and approved without unnecessary delays. This enhances communication across the team and raises the overall quality of the code.

To achieve a fast flow, you can automate parts of the code review process. For example, GitHub Actions is commonly used to automate review checks and CI pipelines on GitHub.

# Example GitHub Actions workflow
name: Code Review
on:
  pull_request:
    types: [opened, synchronize]
jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout code
        uses: actions/checkout@v4 # Current and recommended version is v4.
      - name: Run code review
        run: |
          # Code review commands
          echo "Code review completed successfully"

Low Friction

Low friction is key to maintaining an efficient code review process. Minimizing friction helps code get reviewed and approved swiftly, improving communication across the team and elevating code quality.

To achieve low friction, it is essential to keep the code review process open, clear, and transparent. For instance, you can establish clear documentation outlining the review guidelines and make it easily accessible to everyone on the team.

# Code Review Process
## Step 1: Code Review
The code review phase is essential for improving code quality.
## Step 2: Code Approval
The approval phase ensures the code meets standards before merging.

Conclusion

A strong code review culture is a vital practice that helps software teams collaborate effectively and produce high-quality code. Small PRs, fast flow, and low friction are the core pillars of an efficient review process. In this post, we covered why code review culture matters and how keeping PRs small enables continuous flow. Structuring the review process in an open and transparent manner strengthens team communication and consistently elevates the quality of your code.

Official Resources

Ending SSH Key Chaos: Setting Up Your Own SSH Certificate Authority

Mustafa ERBAY — Fri, 14 Aug 2026 13:46:24 +0000

Ending SSH Key Chaos: Setting Up Your Own SSH Certificate Authority

SSH keys often create conflicts and revision problems when managing server access; you can eliminate this chaos by setting up your own SSH Certificate Authority (CA).

Why Does SSH Chaos Occur?

Manually distributing SSH keys leads to inconsistencies and the problem of not removing old keys.

Duplicate keys: When the same key is copied to multiple servers, access remains even if an employee's account is deleted.
Revocation delay: It can take hours to collect all keys when an employee leaves, creating a security vulnerability.

These problems can be reduced by creating a central signing authority.

ℹ️ Why CA?

CA signs multiple host certificates with a single root private key; clients only need to trust the CA's public key, thus eliminating individual key management.

What is an SSH CA and How Does it Work?

An SSH CA is a certificate model supported by OpenSSH, similar to X.509 certificates but in its own format. Host or user certificates are generated using ssh-keygen -s.

Root key: The CA's private key is stored in one place and has very limited access.
Signed certificates: The certificate given to the server carries the CA's signature and matches files like HostCertificate. The certificate given to the user is trusted by the server via the TrustedUserCAKeys directive and can have user-specific access restrictions via files like AuthorizedPrincipalsFile.

This setup creates a trust chain: the client accepts the CA public key, verifies the host certificate signed by the CA, and grants access.

Workflow Summary Graph

Creating the CA Root Key

The root key should be created in a high-security directory and only accessible by root or via sudo privileges.

# Create a 4096-bit RSA key
sudo ssh-keygen -t rsa -b 4096 -f /etc/ssh/ca_key -N "" -C "My SSH CA"
# Save the public key to /etc/ssh/ca_key.pub
sudo chmod 600 /etc/ssh/ca_key
sudo chmod 644 /etc/ssh/ca_key.pub

WARNING: Using an empty password (-N "") for the CA private key poses a security risk. In production environments, it's strongly recommended to protect the CA private key with a password and store the key offline or in a Hardware Security Module (HSM).

The ssh-keygen output typically includes (output not shown):

Private key: /etc/ssh/ca_key
Public key: /etc/ssh/ca_key.pub

A secure backup of the root key should be kept offline, such as on a USB drive or in an HSM.

Signing Host Certificates

Each server has a host key (e.g., ssh_host_rsa_key); this key needs to be certified with the CA's signature.

# Sign the host key on the server
# Replace ${HOSTNAME} and ${HOSTNAME}.example.com with your server names.
sudo ssh-keygen -s /etc/ssh/ca_key -I ${HOSTNAME} -h -n ${HOSTNAME},${HOSTNAME}.example.com /etc/ssh/ssh_host_rsa_key.pub

This command creates a certificate named ssh_host_rsa_key-cert.pub in the same directory. Add this certificate file to the HostCertificate line in the SSH daemon's sshd_config file:

# In /etc/ssh/sshd_config
HostKey /etc/ssh/ssh_host_rsa_key
HostCertificate /etc/ssh/ssh_host_rsa_key-cert.pub

WARNING: Incorrect configurations in sshd_config can disrupt SSH access. It's recommended to back up the file before making changes and, if possible, perform a dry run or validation step.

Restart the daemon after saving the changes:

sudo systemctl restart sshd

Client-Side: Making the CA Public Key Trusted

On client machines, the CA public key is added to ~/.ssh/known_hosts or /etc/ssh/ssh_known_hosts.

# Add the CA public key (global)
# Replace *.example.com with your domain.
sudo mkdir -p /etc/ssh
sudo tee -a /etc/ssh/ssh_known_hosts <<EOF
@cert-authority *.example.com $(cat /etc/ssh/ca_key.pub)
EOF

# For a personal user (optional, the above global configuration is sufficient)
# Replace *.example.com with your domain.
# This command adds the CA public key to the user's known_hosts file with the @cert-authority directive.
tee -a ~/.ssh/known_hosts <<EOF
@cert-authority *.example.com $(cat /etc/ssh/ca_key.pub)
EOF

This line accepts all host certificates under *.example.com signed by the CA.

User Certificate (Optional)

If you also want to manage individual user keys, you can create a user certificate with the same CA:

# Replace user01 and +52w with your username and validity period.
ssh-keygen -s /etc/ssh/ca_key -I user01 -n user01 -V +52w ~/.ssh/id_rsa.pub

This command produces a file named id_rsa-cert.pub. On the server, to make this user certificate trusted, you need to specify the CA's public key in the TrustedUserCAKeys directive in sshd_config.

Example sshd_config configuration:

# In /etc/ssh/sshd_config
TrustedUserCAKeys /etc/ssh/ca_user_keys.pub

Here, /etc/ssh/ca_user_keys.pub should contain the CA's public key that signed the user certificates.

Deployment and Management Strategies

When deploying SSH CA to multiple servers, the following practical tips are useful:

Strategy	Description	Advantage	Disadvantage
Manual deployment	Copy files to each server using `scp`/`rsync`	Simple, low infrastructure need	High error potential, not scalable
Ansible Playbook	Use `copy` and `lineinfile` modules for automation	Repeatable, version control	Requires Ansible setup
GitOps (Flux/ArgoCD)	Store CA files in a Git repo and deploy via CI/CD	Change tracking, easy rollback	Requires CI/CD pipeline

Example Ansible task:

- name: Deploy SSH CA and host certificates
  hosts: all
  become: true
  tasks:
    - name: Copy CA public key
      copy:
        src: ca_key.pub
        dest: /etc/ssh/ca_key.pub
        mode: '0644'
    - name: Ensure HostCertificate is configured
      lineinfile:
        path: /etc/ssh/sshd_config
        regexp: '^HostCertificate'
        line: "HostCertificate /etc/ssh/ssh_host_rsa_key-cert.pub"
    - name: Restart sshd service
      service:
        name: sshd
        state: restarted

This playbook deploys the CA public key and host certificate in one step.

Security and Maintenance

The CA's private key should never be transferred over the network; it should only be copied with physical access or stored in an HSM.

Key rotation: Regularly rotating (e.g., every 6-12 months) the CA key and re-signing old certificates can enhance long-term security.
Revocation: OpenSSH uses Key Revocation Lists (KRLs) to revoke keys or certificates. KRLs are binary files created with ssh-keygen -k. The list of revoked keys (revoked_keys) is specified in sshd_config with the RevokedKeys directive.

WARNING: The ssh-keygen -R command removes a key from known_hosts, it does not create a revocation list. Use ssh-keygen -k for key revocation.

Conclusion

Setting up your own SSH Certificate Authority eliminates key conflicts, centralizes access management, and simplifies revocation processes.

Step 1: Create the CA root private key in a secure location and protect it with a password.
Step 2: Sign server host keys with the CA and add the HostCertificate directive to sshd_config.
Step 3: Make the CA public key a trusted source on client machines using the @cert-authority directive.
Step 4: Scale deployment with automation (Ansible, GitOps).

This setup enhances both security and operational efficiency; planning for CA rotation and revocation strategies is recommended as a next step.

Official Resources

Embedding Selection for Turkish: Semantic Search with Multilingual

Mustafa ERBAY — Fri, 14 Aug 2026 07:53:01 +0000

What is Embedding and Why is it Important?

Embedding is a function that converts text into a high-dimensional vector; sentences with the same meaning are located close to each other in this vector space. In semantic search systems, the similarity between query and document vectors directly determines the quality of the search. Comparing millions of documents in a few milliseconds is only possible with the right embedding selection. In this section, I will summarize the basic logic of embedding and its role in search performance.

How to Select Embedding for Turkish in Multilingual Models?

When selecting a multilingual model for Turkish, first, look at the model's vocabulary and pre-training dataset's representation of Turkish. If the model tokenizes Turkish characters (ç, ğ, ı, ö, ş, ü) correctly, it produces consistent vectors for the same word in different forms (e.g., "kredi" vs "krediği"). Using the sentence-transformers package from HuggingFace, you can download and test a model with just a few lines of code.

from sentence_transformers import SentenceTransformer

model_name = "sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2"
model = SentenceTransformer(model_name)

sentences = [
    "Müşteri siparişini onayladı.",
    "Müşteri siparişini onayladı."
]
embeddings = model.encode(sentences, normalize_embeddings=True)
print(embeddings.shape)  # (2, 384)

This code snippet generates 384-dimensional vectors for two identical sentences and, with normalize_embeddings=True, the cosine similarity directly approaches 1.0. In a real environment, running different models with the same test set and comparing cosine similarity values is the primary criterion for the selection process.

Impact of Language Features and Tokenization

Turkish, being an agglutinative language, has long word forms; tokenization errors can negate this. For example, BPE-based tokenizers split the word "çalıştırma" into multiple tokens, while a word-based tokenizer might keep it as a single token. This difference can produce vectors of different lengths and, consequently, different similarity scores. When selecting a model, consider how the tokenizer splits words into subword units like sentence-piece or WordPiece, especially for long sentences, to increase consistency.

The example below compares the same sentence tokenized by two different tokenizers:

python -c "from transformers import AutoTokenizer;
t1 = AutoTokenizer.from_pretrained('bert-base-multilingual-cased');
t2 = AutoTokenizer.from_pretrained('xlm-roberta-base'); # Using XLM-R to demonstrate different tokenization behavior
print('BERT tokens:', t1.tokenize('çalışma ortamı güvenli mi?'));
print('XLM-R tokens:', t2.tokenize('çalışma ortamı güvenli mi?'))"

The output shows that BERT splits the sentence into 6 parts using WordPiece-based tokenization, while XLM-R splits it into 5 parts. This difference can affect the sentence's position in the vector space, particularly critical in retrieval-augmented generation (RAG) scenarios where correct tokenization is key.

Model Comparison: mBERT vs XLM-R vs LaBSE vs Sentence-Transformer

Model	Size (Parameters)	Output Dimension	Turkish Pre-training Data
`bert-base-multilingual-cased`	110M	768	104-language Wikipedia
`xlm-roberta-base`	~270M	768	100+ language CommonCrawl
`LaBSE`	~800M	768	Texts in 109 languages
`paraphrase-multilingual-MiniLM-L12-v2`	~22M	384	Multilingual texts (50+ languages)

The parameter counts in the table are approximate and based on information from model cards or related publications. In a production environment, a lighter model like MiniLM offers a balance between low latency and sufficient accuracy. This decision is made based on the need to run on CPU-only servers or edge devices.

💡 Model Selection: Memory and Latency

Smaller models typically require less RAM, while larger models demand more. For instance, some MiniLM models can use under 50 MB of memory for inference, while larger models may require hundreds of MB or even GB of memory. Memory usage depends on the model size, precision used (FP32, FP16, INT8), and batch size. Measure latency using time python script.py and compare results directly.

Performance Measurement and Evaluation Methodology

When measuring performance, two primary metrics are used: Recall@k and Mean Reciprocal Rank (MRR). Recall@k gives the rate of finding the correct document within the first k results; MRR provides an average score by taking the reciprocal of the correct document's rank. A sample evaluation pipeline includes:

Preparing a suitable-sized Turkish query-document test set.
Generating model embeddings for each query and searching for the corresponding document in the FAISS index.
Retrieving the top k results and comparing them with the ground truth.

Below is a simple evaluation code using faiss:

import faiss, numpy as np
from sentence_transformers import SentenceTransformer

model = SentenceTransformer("sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2")
queries = [...]          # Query sentences
docs = [...]             # Document sentences
doc_embeddings = model.encode(docs, normalize_embeddings=True)
index = faiss.IndexFlatIP(doc_embeddings.shape[1]) # IndexFlatIP is used for inner product similarity
index.add(np.array(doc_embeddings, dtype='float32'))

def recall_at_k(k=10):
    correct = 0
    for q, true_id in zip(queries, range(len(queries))):
        q_emb = model.encode([q], normalize_embeddings=True)
        _, I = index.search(np.array(q_emb, dtype='float32'), k)
        if true_id in I[0]:
            correct += 1
    return correct / len(queries)

print("Recall@10:", recall_at_k(10))

This script prints the Recall@10 value directly to the terminal; saving the output and comparing it with different models makes the decision process objective. In a real project, adding GPU acceleration and reporting measurement time can further enhance the evaluation.

Production Environment: Embedding Service and Query Optimization

In a production environment, serving the embedding calculation via a REST or gRPC API provides a scalable solution. The following Docker-Compose example runs a sentence-transformers service with uvicorn:

version: "3.8"
services:
  embed-api:
    image: python:3.11-slim
    working_dir: /app
    volumes:
      - ./:/app
    command: >
      sh -c "pip install sentence-transformers fastapi uvicorn &&
             uvicorn embed_api:app --host 0.0.0.0 --port 8000" # Uvicorn command line arguments
    ports:
      - "8000:8000"

# embed_api.py
from fastapi import FastAPI
from pydantic import BaseModel
from sentence_transformers import SentenceTransformer

app = FastAPI() # Creating a FastAPI application
model = SentenceTransformer("sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2")

class TextPayload(BaseModel): # Using Pydantic BaseModel
    text: str

@app.post("/embed")
def embed(payload: TextPayload):
    vec = model.encode([payload.text], normalize_embeddings=True)[0].tolist()
    return {"embedding": vec}

This service can handle batch requests to process multiple queries simultaneously; monitoring CPU usage with top or htop is crucial to determine resource limits. For query optimization, configuring an index like FAISS IVF+PQ provides millisecond response times for large collections. Additionally, adding a caching layer (e.g., Redis) to store embeddings for frequently queried texts can significantly reduce latency.

curl -X POST http://localhost:8000/embed -H "Content-Type: application/json" \
     -d '{"text":"How does semantic search work in Turkish texts?"}'

This command returns an embedding in response to a single HTTP request; measuring the response time with time curl ... can verify performance targets.

Conclusion

Selecting an embedding for Turkish text depends on the model's representation of Turkish in its pre-training dataset, tokenizer compatibility, and resource constraints in the production environment. MiniLM-like lightweight models offer a balance between low latency and sufficient accuracy, while larger models like LaBSE provide higher recall but require more memory and GPU resources. By measuring Recall@k and MRR with a real test set, you can objectively determine the most suitable model for your needs. The next step involves integrating the selected model into a CI/CD pipeline and monitoring its performance continuously with tools like Prometheus and Grafana.

Official Resources

Transcribing Turkish with Whisper: Local Audio‑Text Pipeline

Mustafa ERBAY — Fri, 14 Aug 2026 02:36:17 +0000

Whisper model and Turkish support

The Whisper model is an open‑source, multilingual speech‑to‑text system released by OpenAI in September 2022. Because it supports 99 languages, it can be used out of the box without any language‑specific fine‑tuning, and it includes Turkish. The large variant has 1.55 billion parameters.

Whisper’s architecture uses an encoder‑decoder Transformer. This design provides robustness to various accents and background noise, allowing it to turn recordings from low‑quality microphones into meaningful text. The training dataset consists of 680 k hours of multilingual, multitask supervised audio collected from the web.

ℹ️ Audio quality

The model performs best with mono audio files sampled at 16 kHz. Lower sampling rates or stereo audio can affect transcription accuracy.

Environment setup and dependency management

To use Whisper you need Python 3.8–3.11 and the pip package manager. First make sure the ffmpeg tool is installed on your system, as Whisper relies on ffmpeg for audio preprocessing.

# Ubuntu or Debian based systems
sudo apt update && sudo apt install ffmpeg
# macOS (Homebrew)
brew install ffmpeg
# Windows (Chocolatey)
choco install ffmpeg

Then install the Whisper package with pip:

pip install -U openai-whisper

If you have a CUDA‑capable GPU, installing a compatible PyTorch build for GPU‑accelerated inference is recommended. For example, for CUDA 11.8 you can use the following command:

pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118

If no GPU is available, use the following command to install a CPU‑optimized PyTorch build:

pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cpu

A virtual environment (venv) provides an isolated Python environment; creating one with python -m venv .venv && source .venv/bin/activate runs independently of system packages and prevents dependency conflicts.

Preparing audio files and preprocessing

Although Whisper supports many audio formats, it is recommended that files be in WAV or FLAC format with a 16 kHz sampling rate and a single channel (mono). You can perform this conversion with ffmpeg:

ffmpeg -i input.mp3 -ar 16000 -ac 1 output.wav

Normalizing the volume of recordings captured in noisy environments can be done as an optional preprocessing step with external tools like sox. For example, the command sox output.wav normalized.wav gain -n can normalize the average RMS level. This helps reduce the risk of the model missing words in low‑volume sections.

Whisper processes audio internally in 30‑second chunks. For very long files (e.g., several hours) or systems with limited memory, splitting the file into smaller segments can help manage memory usage. For instance, you can split a file into 10‑minute pieces using ffmpeg:

ffmpeg -i normalized.wav -f segment -segment_time 600 -c copy part_%03d.wav

Transcription steps with Python

import whisper
import pathlib

# Load the large model variant.
# Model weights are typically downloaded to ~/.cache/whisper.
model = whisper.load_model("large")
audio_path = pathlib.Path("part_000.wav")    # A pre‑processed chunk
result = model.transcribe(str(audio_path), language="tr")
print(result["text"])

The snippet above shows that load_model downloads the model weights, and the transcribe method, with language="tr" specified, disables automatic language detection and locks it to Turkish. The result dictionary contains the raw transcript under the text key and timestamps in the segments list.

Each entry in the segments list is structured like {"id":0, "seek":0, "start":0.0, "end":5.12, "text":"Hello world", "tokens":[...]}, providing a clear view of which words were spoken during a specific time interval.

# Combine multiple segments from a single file
full_text = " ".join([s["text"] for s in result["segments"]])
print(full_text)

Creating a loop for multiple audio parts can be done easily with for part in pathlib.Path('.').glob('part_*.wav'):; this aggregates a large collection of audio files into a single transcript.

Post‑processing: text cleaning and timestamps

The transcription output usually includes punctuation and case conversion, but additional text cleaning and formatting can improve the end‑user experience. For example, you can replace double spaces with a single space and capitalize the first character of a sentence using: result["text"].replace(" ", " ").strip().capitalize().

Timestamps are available as start and end fields in result["segments"]. Converting them to SRT format can be done with a simple loop using the srt library:

from pathlib import Path
import srt

def to_srt(segments):
    entries = []
    for i, seg in enumerate(segments, 1):
        entry = srt.Subtitle(index=i,
                             start=srt.timedelta(seconds=seg["start"]),
                             end=srt.timedelta(seconds=seg["end"]),
                             content=seg["text"].strip())
        entries.append(entry)
    return srt.compose(entries)

srt_content = to_srt(result["segments"])
Path("output.srt").write_text(srt_content, encoding="utf-8")

This code saves each segment as a subtitle line, enabling media players to synchronize it directly.

Performance monitoring and debugging

torch.cuda.is_available() checks whether a GPU is present. If it returns True, you can create a GPU‑enabled model with model = whisper.load_model("large", device="cuda"). When no GPU is available, the device="cpu" option automatically runs a CPU‑optimized execution.

To measure processing time you can use the time module: start = time.time(); model.transcribe(...); elapsed = time.time() - start. This quantifies performance differences across audio lengths and model variants.

⚠️ Memory constraints

In a CPU‑only environment the large model can consume a substantial amount of RAM (roughly 3.9 GB up to 10 GB). If memory is insufficient, you may need to switch to smaller model variants such as medium or small to avoid out‑of‑memory errors.

Pipeline visualization

This diagram shows the flow from an audio file to the final text as a single‑direction pipeline; each box takes the output of the previous step and feeds it into the next.

Conclusion

Processing Turkish audio files with Whisper locally makes it possible to generate high‑accuracy transcripts when proper preprocessing and model selection are applied. Clear command and configuration examples at each pipeline stage create a reusable template and enable rapid integration into production environments.

The next step is to add this pipeline to a CI/CD workflow to provide an automated transcription service; an example GitHub Actions file runs the whisper command inside a Docker container, offering a scalable solution.

DEV Community: Mustafa ERBAY

Declarative Server with NixOS: A Reproducible Infrastructure

NixOS's Declarative Model

Creating a Basic Configuration File

Managing Services: PostgreSQL and Nginx

Reproducibility and Disk Image

NixOS Architecture and Package Management

Security and Updates

CI/CD Integration: GitHub Actions and NixOS

Rollback and Verification

Limitations and Improvement Opportunities

Conclusion

Official Sources

Writing Idempotent Ansible Playbooks: Safe Change Patterns

Describe State Before Procedures

--check Is a Prediction, Not Proof

creates, removes, and Honest changed_when for Commands

Validate a File Before Replacing It

block and rescue Do Not Invent a Rollback

Separate False Idempotence Signals

Measure the State After Applying It

Test Convergence in CI

When Does This Decision Become Invalid?

Official References

Log Infrastructure with Loki: Label Design and Cost Control

Where Loki costs originate

Indexed label, structured metadata, or log body?

Treat the label budget as a contract

Design storage schema and retention together

Bound query cost as well

Safe rollout: measure, constrain, then move

Conclusion

Official Resources

Prometheus Alert Fatigue: Symptom-Based Alert Design

Why Do Cause-Based Alerts Fail?

Core Principles of Symptom-Based Alerting: RED and USE

Writing User-Centric Alerts with PromQL

1. Error Rate Alert

2. Latency / Duration Alert

3. Fast-Depleting Resources (Multi-Window Burn Rate)

Filtering Noise with Alertmanager: Routing, Grouping, and Inhibition

Actionable Alerts and Runbook Architecture

Alert Hygiene and Lifecycle Management

Wrapping Up

Official Resources

Introduction to GitOps: Pull-Based Deployment Model with Argo CD

What is GitOps?

How Does Argo CD Work?

What is the Pull-Based Deployment Model?

Argo CD Installation Steps

Application Deployment Workflow

Best Practices and Common Pitfalls

Conclusion

Official Resources

Lightweight Kubernetes with k3s: Scaling from Single Node

Introduction

Lightweight Kubernetes with k3s

Single Node Deployment with k3s

Scaling to Multiple Nodes

Adding Worker Nodes

Security

Server Security

k3s Security

Conclusion

Official Resources

Setting Up Your Own Tailscale Control Server with Headscale

Introduction

What is Headscale?

Setting Up Your Own Tailscale Control Server with Headscale

Step 1: Install Headscale

Step 2: Configure Headscale

Step 3: Install Tailscale

Step 4: Connect Tailscale to Headscale

Managing Your Own Tailscale Control Server with Headscale

Security Settings

Network Management

Conclusion

Official Resources

Syncthing for Cloudless Synchronization: Conflicts and Versions

Introduction

`--check` Is a Prediction, Not Proof

`creates`, `removes`, and Honest `changed_when` for Commands

`block` and `rescue` Do Not Invent a Rollback