DEV Community: Konrad Kądzielawa

Building a Local IT Assistant with Ollama, Open WebUI and a Custom Model Router

Konrad Kądzielawa — Mon, 29 Jun 2026 07:42:42 +0000

Running a local LLM is easy. Turning it into something useful for daily IT administration is a different story.

This article is not only about installing Ollama and Open WebUI in Docker. There are already many tutorials for that. What I wanted to build was a local assistant that could actually help with SysOps tasks: reading logs, explaining errors, preparing safe Bash commands, writing Ansible playbooks, reviewing YAML files, and helping with Docker-related troubleshooting.

My goal was to prepare a local AI setup that could later be moved to a dedicated Actina machine and used as a small internal IT assistant.

The interesting part was not the installation itself.

The interesting part was what happened after the installation.

I had to solve a few practical problems:

how to choose a default model,
how to route coding-related prompts to a better model,
how to avoid manually switching models in the UI,
how to make Open WebUI start with the right assistant,
how to make the whole setup survive a VM restart,
and how to keep the models and data persistent.

This is Part 1 of the setup.

Final goal

The final idea was simple:

I wanted to open Open WebUI, select one assistant, and use it for IT administration.

Under the hood, however, I wanted different models to be used depending on the task.

For example:

General chat, logs, CSV analysis, explanations  -> qwen3:8b
Bash, Python, Docker, Ansible, YAML, errors     -> qwen2.5-coder:7b

So instead of manually switching between models, I created a custom Pipe Function in Open WebUI.

From the user perspective, it looked like one model:

AUTO Local IT Assistant

But internally it worked as a simple model router.

Environment overview

The local setup was based on Docker Compose.

The project directory looked like this:

/home/kkadzielawa/ai-local
├── docker-compose.yml
├── .env
└── data
    ├── ollama
    └── open-webui

The main containers were:

ollama
open-webui

The local models were:

qwen3:8b
qwen2.5-coder:7b

The idea was:

qwen3:8b           -> general local IT assistant
qwen2.5-coder:7b  -> coding, scripting, Docker, Ansible, YAML

At first, I thought that setting a default model would be enough.

It was not.

Problem 1: A default model is not enough

The first approach was to configure Open WebUI with a default model.

In Docker Compose, this can be done using environment variables like this:

environment:
  - DEFAULT_MODELS=qwen3:8b
  - DEFAULT_PINNED_MODELS=qwen3:8b,qwen2.5-coder:7b

This works, but only solves one problem:

Which model should be selected by default?

It does not solve the more important problem:

Which model is best for this specific prompt?

For example, this prompt is probably fine for a general model:

Explain what should be checked before upgrading a Debian server in production.

But this one should go to a coding-oriented model:

Write an Ansible playbook that checks hostname, uptime, OS version and free disk space.
Read-only tasks only. Do not modify the system.

I did not want to remember which model to select every time.

I wanted Open WebUI to make that decision automatically.

So I needed a custom layer between the user prompt and the actual model.

Problem 2: Tools, Pipelines or Functions?

This was one of the first confusing parts.

Open WebUI has several places that may look similar at first:

Workspace -> Tools
Settings  -> Pipelines
Admin     -> Functions

My first assumption was that this kind of logic should be created as a tool or pipeline.

But that was not the right direction.

A model router is not a normal tool called by the model during a conversation. It is not something like a calculator, web search or custom API tool.

What I needed was something that behaves like a model in the UI, receives the user request, decides what to do with it, and then forwards it to the selected real model.

The right place was:

Admin Settings
-> Functions
-> New Function

And the right type of function was a Pipe Function.

Problem 3: Open WebUI showed an Example Filter, but I needed a Pipe

When creating a new function, Open WebUI may show an example based on a filter.

That can be misleading.

For this use case, a filter was not what I needed.

A filter can modify input or output, but I wanted to create something that appears as a selectable model and routes the request to another model.

So the function had to be based on:

class Pipe:

not:

class Filter:

This was an important detail.

The custom function was named:

AUTO Local IT Assistant

The function had its own configurable valves:

Default Model: qwen3:8b
Coder Model: qwen2.5-coder:7b
Show Selected Model: false

This allowed me to change the underlying models later without rewriting the whole function.

The routing logic

The first version of the router was intentionally simple.

It checked the user message and looked for keywords or patterns suggesting that the prompt was related to coding, scripting or infrastructure automation.

Examples of keywords:

ansible
playbook
yaml
bash
python
powershell
docker compose
terraform
traceback
exception
stack trace

Examples of patterns:

docker ...
systemctl
journalctl
apt install
def function_name(...)
class Something
#!/bin/bash
- hosts:
tasks:

If the prompt matched coding or automation-related patterns, the function routed it to:

qwen2.5-coder:7b

Otherwise, it used:

qwen3:8b

This is not an advanced agent.

It is a simple and predictable router.

And that was exactly what I wanted at this stage.

For a local IT assistant, predictable behavior is often better than clever behavior that is hard to debug.

Adding a safety-oriented system prompt

Because this assistant is meant for IT administration, I did not want it to randomly suggest destructive commands.

So I added a system prompt with practical safety rules.

The assistant should:

- answer in Polish,
- classify commands as read-only, system-changing or risky,
- suggest diagnostics before making production changes,
- avoid destructive commands without backups,
- explain Bash, Python and Ansible snippets,
- prefer safe troubleshooting steps first.

This is a very small addition, but it changes the behavior a lot.

For example, instead of immediately suggesting a dangerous command, the assistant should first propose read-only diagnostics:

df -h
free -m
uptime
systemctl status docker
journalctl -u docker --no-pager -n 100

For me, this is one of the most important parts of building a local SysOps assistant.

The model should not only answer.

It should answer in a way that matches how administrators actually work with production systems.

Problem 4: The router existed, but Open WebUI still showed qwen3

After creating the Pipe Function, I wanted Open WebUI to start with the router selected by default.

So I changed the Docker Compose environment variables to something like this:

environment:
  - DEFAULT_MODELS=auto-local-it-assistant
  - DEFAULT_PINNED_MODELS=auto-local-it-assistant,qwen3:8b,qwen2.5-coder:7b

At first, it looked like it did not work.

Open WebUI still showed:

qwen3:8b

This was confusing, because the configuration seemed correct.

The reason was simple but easy to miss:

Open WebUI can remember the model selected in an existing chat or user session.

So the fix was not another Docker Compose change.

The fix was:

1. Hard refresh the browser with CTRL + F5
2. Start a new chat
3. Search for "auto" in the model selector
4. Select AUTO Local IT Assistant

After that, the router worked as expected.

The lesson here was important:

DEFAULT_MODELS affects the default state,
but existing chats or user sessions may still remember the old model.

This is a very practical troubleshooting point.

Without knowing this, it is easy to waste time restarting containers, changing YAML files or assuming the Pipe Function is broken.

Problem 5: Containers must start after VM reboot

Another issue was more operational.

This setup was running on a VM, and I did not want the local assistant to disappear after a reboot.

The solution had two parts.

First, Docker and containerd should start with the system:

sudo systemctl enable docker.service
sudo systemctl enable containerd.service

Second, the containers should have a proper restart policy.

In docker-compose.yml:

services:
  ollama:
    image: ollama/ollama:latest
    container_name: ollama
    volumes:
      - ./data/ollama:/root/.ollama
    restart: unless-stopped

  open-webui:
    image: ghcr.io/open-webui/open-webui:main
    container_name: open-webui
    volumes:
      - ./data/open-webui:/app/backend/data
    restart: unless-stopped

I used:

restart: unless-stopped

instead of:

restart: always

because I prefer this behavior for a local admin tool.

If I manually stop the container, I usually mean it. I do not want Docker to bring it back immediately against my decision.

To verify the restart policy:

docker inspect -f '{{.HostConfig.RestartPolicy.Name}}' ollama
docker inspect -f '{{.HostConfig.RestartPolicy.Name}}' open-webui

Expected result:

unless-stopped
unless-stopped

Then the real test:

sudo reboot

After the VM comes back:

docker ps

Both containers should be running again.

Problem 6: Models and data must survive container recreation

Another thing that matters a lot with Ollama and Open WebUI is persistence.

I did not want downloaded models or Open WebUI data to live only inside containers.

That is why I used local directories mounted into the containers:

volumes:
  - ./data/ollama:/root/.ollama

and:

volumes:
  - ./data/open-webui:/app/backend/data

This gives me a clean project structure:

/home/kkadzielawa/ai-local
├── docker-compose.yml
├── .env
└── data
    ├── ollama
    └── open-webui

It also makes the setup easier to back up or move to another machine:

cd ~
tar -czf ai-local-backup-$(date +%F).tar.gz ai-local

This was important because the environment was prepared with a future migration in mind.

The VM was only the first step.

The target was a dedicated machine for local AI-assisted administration.

Testing the assistant

I tested the router with a few different prompts.

A general SysOps question:

Explain what should be checked before upgrading a Debian server in production.

Expected model:

qwen3:8b

A Bash-related prompt:

Write a Bash script that checks CPU, RAM, disks, IP addresses and whether Docker is running.

Expected model:

qwen2.5-coder:7b

An Ansible prompt:

Write an Ansible playbook that checks hostname, uptime, OS version and free disk space.
Read-only tasks only.

Expected model:

qwen2.5-coder:7b

A YAML troubleshooting prompt:

Check this docker-compose.yml and explain why Open WebUI cannot reach Ollama.

Expected model:

qwen2.5-coder:7b

A log analysis prompt:

Analyze this log and tell me what looks suspicious.

Expected model:

qwen3:8b

At this stage, the routing does not need to be perfect.

It needs to be predictable and useful.

That is enough for Part 1.

What I learned

The biggest lesson from this setup is:

Local AI is not only about running a model.
It is about building the workflow around the model.

Ollama and Open WebUI give you a very good starting point.

But the real value starts when you customize the environment for your own use case.

In my case, the most important improvements were:

- one visible assistant in the UI,
- automatic routing between a general model and a coding model,
- a safety-oriented system prompt,
- persistent model and Open WebUI data,
- container autostart after VM reboot,
- a structure that can be moved to another machine later.

The most useful parts were not the obvious installation steps.

The most useful parts were the small issues found along the way:

- a model router is not a normal Tool,
- the correct Open WebUI feature is a Function,
- the function should be a Pipe, not a Filter,
- Open WebUI may remember the previous model in an existing chat,
- DEFAULT_MODELS may look broken until a new chat is created,
- restart policies should be verified, not assumed,
- persistent volumes are mandatory if the setup is meant to survive recreation.

These are the details that usually do not appear in a clean installation guide.

But these are exactly the details that matter when building something usable.

What comes next

This is only the first part of the project.

The current setup gives me a local IT assistant with simple model routing.

But there are many possible next steps.

The most obvious one is adding a vision model.

That would allow the assistant to handle screenshots, UI errors, diagrams or monitoring views.

For example:

screenshot or image -> vision model
code or automation  -> coder model
general IT question -> general model

Another important step is RAG.

I want the assistant to work with local documentation, internal notes, procedures, PDFs, CSV exports and troubleshooting guides.

That would require adding an embedding model and a local knowledge base.

Possible future improvements:

1. Add a vision model for screenshots and UI troubleshooting
2. Add a local knowledge base with RAG
3. Add an embedding model for document search
4. Improve the routing logic
5. Add monitoring for Ollama and Open WebUI
6. Add backups for models and Open WebUI data
7. Add reverse proxy and HTTPS
8. Restrict access to VPN or internal LAN only
9. Build more specialized assistants for specific IT tasks

So this is not a finished platform.

It is the base layer.

The first working version of a local SysOps assistant.

And for me, this is the most interesting part of local AI: not just running a model, but slowly turning it into a practical internal tool.

Post-Mortem: Diagnosing and Fixing Ceph Quorum Breakdown (Time Drift) in a Hybrid Proxmox VE Cluster

Konrad Kądzielawa — Wed, 25 Mar 2026 21:42:53 +0000

Key Takeaways & Conclusions:

Root Cause: A Time Drift between the physical Proxmox host and nested/external virtual machines exceeded Ceph's strict 0.05s tolerance limit.
Impact: The time mismatch caused a continuous authorization loop (electing/probing) in the local Ceph Monitor (MON), corrupting its local RocksDB database and triggering a cascading failure of the dependent OSD.0 daemon.
Resolution: Enforced rigorous time synchronization via Chrony and QEMU Guest Agent, followed by a destructive purge and rebuild of the corrupted MON daemon using the surviving quorum.
Verification: Chaos Engineering tests (rados bench) confirmed that with a min_size = 2 replication policy, the cluster successfully survives a hard node power-off without dropping client I/O.

1. The Architecture and The Incident

Building a Ceph cluster typically requires dedicated bare-metal servers. My home lab topology, however, is a hybrid 3-node setup designed to simulate a full cluster using limited hardware:

admin (Physical Proxmox host, IP: 192.168.0.250)
admin-02 (Nested VM 103 on the main host, IP: 192.168.0.251)
admin-03 (External VM running on a separate Ubuntu desktop, IP: 192.168.0.252)

The incident began when the cluster state dropped to HEALTH_WARN. Despite the underlying Corosync cluster and network layer (ICMP) functioning perfectly across the 192.168.0.0/24 subnet, the main physical node (admin) dropped out of the Ceph quorum:

root@admin:~# ceph -s
  cluster:
    id:     25f2d0a2-d64b-42a2-b93d-99a560571e0e
    health: HEALTH_WARN
            1/3 mons down, quorum admin-02,admin-03
            1 osds down
            1 host (1 osds) down

2. Debugging & Log Analysis

To find the root cause, I inspected the systemd journals for the failed Object Storage Daemon (osd.0) located on the physical admin host.

root@admin:~# journalctl -u ceph-osd@0.service -n 50 --no-pager
Mar 25 21:08:21 admin ceph-osd[3147]: failed to fetch mon config (--no-mon-config to skip)
Mar 25 21:08:21 admin systemd[1]: ceph-osd@0.service: Main process exited, code=exited, status=1/FAILURE

The OSD drive itself was physically healthy, but it crashed on startup because it could not retrieve the Cluster Map from the local Monitor (ceph-mon).

Next, I checked the local Monitor service on the admin node:

root@admin:~# journalctl -u ceph-mon@admin.service -n 30 --no-pager
Mar 25 21:49:55 admin ceph-mon[1377]: 2026-03-25T21:49:55.526+0100 7c00b3b756c0 -1 mon.admin@0(probing) e3 get_health_metrics reporting 1 slow ops, oldest is auth(proto 0 30 bytes epoch 0)
Mar 25 21:50:50 admin ceph-mon[1377]: 2026-03-25T21:50:50.528+0100 7c00b3b756c0 -1 mon.admin@0(electing) e3 get_health_metrics reporting 1 slow ops, oldest is auth(proto 0 30 bytes epoch 0)

The logs revealed the local MON was stuck in an endless electing/probing state, constantly dropping cryptographic authorization packets (auth slow ops). According to Ceph's official clock synchronization documentation, Monitors require clocks to be synchronized within 0.05 seconds. Because admin-02 and admin-03 were virtual machines, their CPU clock cycles drifted away from the physical host's hardware clock, effectively locking the admin node out of the quorum and corrupting its metadata database.

3. The Action Plan: Fixing Time Drift & Rebuilding

To stabilize the cluster, I had to enforce strict timekeeping on the virtual nodes and reconstruct the corrupted Monitor.

Table: Remediation Steps per Node

Node	Ceph Role	Root Cause	Implemented Solution
admin (Physical)	MON, OSD.0	Corrupted RocksDB	Destroyed and recreated local MON daemon.
admin-02 (VM 103)	MON, OSD.1	Hypervisor Time Drift	Installed QEMU Guest Agent for hardware sync.
admin-03 (Ext. VM)	MON, OSD.2	OS-level Time Drift	Deployed Chrony to force strict NTP sync.

Step 1: Enforcing Strict NTP on the External VM (`admin-03`)

Standard systemd-timesyncd is too sluggish for Ceph. I replaced it with chrony and forced an immediate clock step on the external Ubuntu VM:

root@admin-03:~# sudo apt-get update && sudo apt-get install chrony -y
root@admin-03:~# sudo chronyc makestep
root@admin-03:~# chronyc tracking
Reference ID    : C292FB72 (host-194-146-251-114.virtuaoperator.pl)
Stratum         : 2
System time     : 0.000000000 seconds slow of NTP time

The output confirmed a 0.000000000 seconds deviation.

Step 2: Enabling QEMU Guest Agent (`admin-02`)

For the nested Proxmox node, time had to be injected directly from the physical hypervisor. I enabled the QEMU Guest Agent flag, performed a hard power cycle of VM 103, and verified the communication channel:

# Executed on the physical host (admin)
root@admin:~# qm set 103 -agent 1
root@admin:~# qm shutdown 103
root@admin:~# qm start 103
root@admin:~# qm agent 103 ping

Step 3: Reconstructing the Corrupted Monitor (`admin`)

With the time drift neutralized, I had to fix the broken ceph-mon daemon on the physical host. Attempting to repair a broken RocksDB is often futile. Since the other two nodes maintained a healthy quorum and an intact Cluster Map, I simply purged the broken node and recreated it:

root@admin:~# pveceph mon destroy admin
root@admin:~# pveceph mon create
root@admin:~# systemctl restart ceph-osd@0.service

Immediately after the restart, osd.0 successfully authenticated with the fresh Monitor, and the cluster began the peering process, eventually returning to HEALTH_OK.

4. Verification: Chaos Engineering

You can't trust a storage cluster until you break it while it's under load. To prove the fix worked, I initiated a continuous benchmark write on the ceph-storage pool:

root@admin:~# rados bench -p ceph-storage 120 write --no-cleanup

While the benchmark was running at ~55 MB/s, I executed a brutal sudo systemctl poweroff on the external admin-03 node.

Here is what happened in the benchmark logs:

  38       16       419       403   42.4168         0            -     1.12533
  39       16       419       403   41.3292         0            -     1.12533
2026-03-25T22:31:25.378550+0100 min lat: 0.334379 max lat: 3.14551 avg lat: 1.12533
  40       16       419       403   40.2959         0            -     1.12533
  41       16       419       403   39.3131         0            -     1.12533
  42       16       439       423   40.2816         8     0.399663      1.5659
  43       16       470       454   42.2282       124     0.593991      1.5049
  44       16       489       473   42.9956        76     0.755278     1.47177

The Result:

Seconds 38-41: The moment the node died, the cluster momentarily paused client I/O. Throughput dropped to 0 MB/s, and latency spiked.
Second 42+: The CRUSH map recalculated the topology. Because the pool was configured with min_size = 2, the two surviving nodes authorized the writes. The cluster automatically resumed operations, quickly catching up and hitting speeds of 124 MB/s.

Once the test concluded, I cleaned up the benchmark data:

root@admin:~# rados -p ceph-storage cleanup

Conclusion: Time synchronization is the absolute lifeblood of Ceph. Never trust default virtual machine clocks in a production or simulated lab environment. By implementing chrony and QEMU hardware sync, the cluster is now fully immune to node-level failures.

Proxmox & Ceph in a Home Lab: Building a Cluster from a Single Drive & Surviving the CLI Battle

Konrad Kądzielawa — Mon, 23 Mar 2026 19:58:21 +0000

Building a Ceph cluster on Proxmox VE in a home lab is a fantastic proving ground. The catch is that Ceph is inherently a powerful Enterprise-grade system – and by default, it demands entire, raw physical drives (raw block devices) for its OSD (Object Storage Daemon) daemons.

But what if your "IT Bunker" only has one physical server with a single extra 111 GB SSD, and you want to simulate a full-fledged 3-node cluster? My architecture looked like this:

admin (Physical Proxmox host)
admin-02 (Nested VM on the main server)
admin-03 (VM... running remotely on a separate desktop PC)

Here is the story of how I "cheated" Ceph's strict mechanics, sliced a single drive into pieces, bypassed hypervisor GUI errors, and ultimately achieved the coveted HEALTH_OK status.

Challenge 1: Handing an LVM Partition over to Ceph

My plan was to split the physical sdc (111.8 GB) drive on the main server into three 35 GB logical volumes using LVM (Logical Volume Manager). Here is the lsblk output from the main host:

sdc                            111.8G LVM2_member disk 
├─vg_ceph_lab-ceph_admin          35G             lvm  
└─vg_ceph_lab-ceph_admin02        35G             lvm

The first node (admin) took its piece without a hitch. The trouble started when I tried to add an OSD for the second node (admin-02) via the main host's CLI:

root@admin:~# pveceph osd create /dev/vg_ceph_lab/ceph_admin
unable to get device info for '/dev/dm-6'

The Solution: Proxmox and Ceph hate shared LVM paths and standard partitions when creating an OSD. Since admin-02 is a virtual machine, I leveraged the built-in QEMU/KVM Virtual Machine Manager (qm) tool. Instead of fighting Ceph daemons on the host, I attached the virtual volume directly into the admin-02 VM, bypassing the filesystem layer entirely:

qm set 103 -scsi2 /dev/vg_ceph_lab/ceph_admin02

Inside the admin-02 VM, the OS saw new, "physical", and raw block hardware. A quick click on Create OSD in the node's Proxmox GUI, and the drive initialized perfectly!

Challenge 2: An External Host and a Rebellious virt-manager

That left the third node: admin-03. The problem? This VM physically resided on my Linux Desktop PC across the room. The qm set command was useless here because the sliced LVM volume was on the server in the rack.

In a cluster architecture, Ceph doesn't care where the physical storage comes from. I decided to ditch the server's LVM for this node and create a new virtual disk locally on my PC. But trying to add a disk via the virt-manager GUI threw a hardware incompatibility error: pc-q35-noble. The graphical interface completely refused to cooperate.

The Solution: When the GUI throws weird exceptions, the terminal is your best friend. I fell back on native tools from the libvirt and QEMU stack.

First, I manually generated the disk file using qemu-img:

sudo qemu-img create -f qcow2 /var/lib/libvirt/images/admin-03-ceph.qcow2 35G

Next, I hard-attached the generated image to the offline VM configuration using virsh:

sudo virsh attach-disk Proxmox-Lab /var/lib/libvirt/images/admin-03-ceph.qcow2 vdb --config --subdriver qcow2

Upon booting admin-03, its internal Proxmox system recognized the /dev/vdb drive. Adding the third OSD in the Proxmox GUI worked flawlessly.

Finale: Pools, Replication, and Green Status

After adding the third OSD, I had a "Frankenstein" cluster: drives supplied by raw LVM from a physical server mixed with a remote qcow2 virtual disk. Did Ceph care? Not at all.

The final step was to create a Ceph Pool—the logical organizational layer where the actual data lands:

Size: 3 (Every data block is replicated in 3 copies).
Min. Size: 2 (The cluster allows I/O writes as long as at least 2 out of 3 nodes are active).

CLI verification (on the main node) showed exactly what I had been fighting for all evening:

root@admin:~# ceph -s
  cluster:
    id:     25f2d0a2-d64b-42a2-b93d-99a560571e0e
    health: HEALTH_OK

  services:
    mon: 3 daemons, quorum admin,admin-02,admin-03 (age 6m)
    mgr: admin(active, since 60m), standbys: admin-02, admin-03
    osd: 3 osds: 3 up (since 6m), 3 in (since 6m)

Let's check the CRUSH (Controlled Replication Under Scalable Hashing) algorithm tree using the ceph osd tree command to confirm the cluster topology:

root@admin:~# ceph osd tree
ID  CLASS WEIGHT   TYPE NAME         STATUS  REWEIGHT  PRI-AFF
-1         0.10258 root default                      
-3         0.03419     host admin                     
 0   hdd  0.03419         osd.0          up   1.00000  1.00000
-5         0.03419     host admin-02                  
 1   hdd  0.03419         osd.1          up   1.00000  1.00000
-7         0.03419     host admin-03                  
 2   hdd  0.03419         osd.2          up   1.00000  1.00000

As you can see, Ceph's intelligent engine sees my crafted nodes as separate physical hosts and correctly distributes the traffic, providing me with a single, unified rbd (RADOS Block Device) storage pool labeled ceph-storage in Proxmox.

Conclusion

Home lab experiments quickly expose the gap between dry documentation theory and actual implementations in simulated environments. This deployment taught me two key lessons:

Understanding how to pass storage resources through the hypervisor (like the qm set tool in PVE) is a game changer when Ceph insists on block access and you only have LVM partitions available.
When GUI software (like desktop virt-manager in my case) throws motherboard configuration errors, going back to the basics and pure CLI (virsh, qemu-img) lets you bypass those issues in just a few quick commands.

With a fully functioning, green cluster, I am ready for the next level: Chaos Engineering. I plan to disconnect drives while VMs are running just to see if my Ceph setup is truly as resilient as the documentation promises!

DNS Troubleshooting on Windows: When the Internet Works but the Intranet Doesn’t

Konrad Kądzielawa — Fri, 20 Mar 2026 12:10:01 +0000

One of the more frustrating problems in IT is when a user says:

“The internet works, but this one internal site doesn’t.”

At first, it sounds simple. Public websites load fine. Email works. Teams works. But the internal app still refuses to open, or it loads painfully slowly.

In situations like that, the application itself is not always the real problem. Quite often, the issue is one layer lower: DNS.

That is also why I think DNS is worth understanding in a practical way, not just as a protocol you read about once and then forget. In day-to-day admin work, name resolution problems show up more often than people expect.

A very typical scenario

Let’s take a neutral example.

A user tries to open:

http://intranet-app01/portal

What they see is something like this:

the page does not open at all, or takes too long to load,
public websites still work,
nslookup intranet-app01 returns an IP address,
but the internal application still behaves as if something is wrong.

At that point, it is very tempting to say:

“If nslookup works, DNS is fine.”

I used to think that too in simpler cases. But in real environments, that conclusion is often too optimistic.

Why `nslookup` helps — but does not always tell the full story

nslookup is still a useful tool. It is quick, simple, and good for checking what a DNS server returns for a given name.

The catch is that a successful nslookup result does not always mean the client is resolving names cleanly in real use.

Microsoft’s own DNS client troubleshooting guide shows why this matters. If a machine has multiple DNS servers configured and some of them are unreachable, Windows may still eventually resolve the name — but only after retries and timeouts. From the user’s point of view, that can look like “the app is slow” or “the site is broken,” even though DNS eventually returns an answer.

So when I troubleshoot DNS now, I try not to stop at:

“Does the name resolve?”

A better question is:

“How does it resolve, from where, and how long does it take?”

How Windows actually resolves a name

One detail I find especially useful from Microsoft’s troubleshooting article is that Windows does not immediately send every query to a DNS server.

The client typically goes through these steps:

check the local DNS cache
check the Hosts file
query the DNS server

That means a “correct” answer is not always coming from the place you think it is.

Sometimes the client is using:

an old cached entry,
a negative cache result,
or a static entry in the Hosts file.

So even when name resolution appears to work, it is still worth asking whether you are looking at live DNS behavior or just local state.

A simple workflow I usually follow

When an internal hostname behaves strangely, I try not to jump between random fixes. I usually go through a short set of checks in the same order.

It does not solve every case, but it helps narrow things down quickly.

1. Check the client’s DNS configuration

I usually start with:

ipconfig /all

This gives me a quick view of:

which DNS servers are configured,
in what order,
and whether anything looks off right away.

If I see multiple DNS servers, I want to know whether all of them are still reachable. Microsoft notes in its DNS troubleshooting guide that unreachable DNS servers can introduce noticeable delays because the client keeps trying before moving on.

That is one of those things that can make an application feel broken even though the underlying name eventually resolves.

2. Compare the default result with a specific DNS server

Next, I usually run:

nslookup intranet-app01
nslookup intranet-app01.corp.example 192.0.2.53

The first command tells me what happens with the machine’s default DNS configuration.

The second helps remove some ambiguity, because I am querying a specific DNS server directly.

According to the official nslookup documentation, you can provide the DNS server explicitly in non-interactive mode, which makes this a useful comparison step.

3. Test the same name with `Resolve-DnsName`

After that, I often repeat the test in PowerShell:

Resolve-DnsName -Name intranet-app01.corp.example
Resolve-DnsName -Name intranet-app01.corp.example -Server 192.0.2.53

Microsoft’s Resolve-DnsName documentation describes it as functionally similar to nslookup, and I find it really useful when I want clearer, more structured output.

It is especially helpful when I want to compare answers from:

the default client path,
and a specific DNS server I actually trust for that zone.

This does not automatically fix anything, of course, but it helps answer a more precise question:

“Is the issue really DNS, or is it the path the client is taking to get the answer?”

4. Inspect the local DNS cache

If the results still feel inconsistent, I check the local resolver cache:

ipconfig /displaydns

Microsoft’s ipconfig reference explains that /displaydns shows the DNS resolver cache, including entries loaded from the local Hosts file and recently resolved records.

I like this step because it helps explain a lot of confusing cases.

For example:

maybe the client is not querying DNS at all anymore,
maybe it is holding on to an old result,
maybe there is a local entry that makes the test misleading.

5. Flush the cache if I suspect stale local state

If I think the cache might be part of the problem, I clear it:

ipconfig /flushdns

Again, this is documented by Microsoft in the ipconfig command reference.

I would not treat this as a magic fix, because it is not. But it is a reasonable troubleshooting step when I want to remove stale local state before testing again.

6. Check whether the short name is the real issue

This is probably one of the most common things behind “it works for some people but not for others.”

Users often type:

http://intranet-app01

instead of:

http://intranet-app01.corp.example

That difference matters more than it seems.

Microsoft’s DNS troubleshooting guide includes examples showing how the DNS suffix search list can affect resolution time and behavior. If the suffix list is long, badly ordered, or just not aligned with the environment, short-name resolution can become slow or inconsistent.

So sometimes the problem is not that DNS is “down.”

Sometimes the problem is simply that:

the short name is ambiguous,
the client is trying the wrong suffixes first,
or the right answer only shows up after too many failed attempts.

What usually turns out to be wrong

In many environments, DNS is not completely broken.

It is just slow, inconsistent, or slightly misleading.

In practice, I have found that the most common causes are things like:

the primary DNS server is unreachable,
DNS servers are configured in a bad order,
the client is using a cached result,
the Hosts file contains an override,
users rely on short names instead of FQDNs,
the DNS suffix search list is too long or poorly ordered,
or the application is simply less tolerant of delays than a browser is.

That is why I try not to stop at “nslookup works.”

That statement may be true and still not explain the user’s experience.

The question I actually want to answer

When I troubleshoot something like this, I am not really asking:

“Does the name resolve?”

I am usually asking:

“How is the client resolving it, what is it talking to first, and where is the delay happening?”

That small shift in thinking helps a lot.

It turns troubleshooting from random guessing into a more predictable workflow.

A quick checklist I like to keep in mind

Here is the short version:

1. ipconfig /all
2. nslookup shortname
3. nslookup FQDN <specific_DNS_server>
4. Resolve-DnsName -Name FQDN
5. Resolve-DnsName -Name FQDN -Server <specific_DNS_server>
6. ipconfig /displaydns
7. ipconfig /flushdns
8. test the FQDN instead of the short name
9. review DNS server order and suffix search list
10. only then move deeper into the application itself

Final thought

A lot of DNS issues do not look like DNS issues at first.

They look like browser problems, application issues, random network instability, or “something weird on this one computer.”

But sometimes the underlying cause is much simpler than that:

the client is asking the wrong place, in the wrong order, for too long.

That is why I try to troubleshoot DNS methodically instead of jumping straight into random fixes. It saves time, and it usually makes the actual problem easier to explain — both to yourself and to the user who just wants the internal site to work again.

Building a Zero-Maintenance SysOps Knowledge Base with React and the Dev.to API

Konrad Kądzielawa — Wed, 18 Feb 2026 12:25:24 +0000

After spending nearly a decade navigating different areas of IT—from building complex e-commerce frontends to managing the underlying Linux servers and infrastructure—I decided it was time to completely rebuild my personal portfolio.

My old website was full of typical "frontend fireworks": heavy animations, flashy graphics, and complex layouts. But as my career naturally shifted towards IT Systems Administration and infrastructure management, that style no longer resonated with my daily work. I wanted a space that felt like home to a SysOps engineer: raw, minimalist, and terminal-inspired.

However, as an admin, I had a strict set of requirements for this new platform:

Zero maintenance: I don't want to deal with database backups or traditional CMS panels.
No local Markdown files: Pushing a commit to GitHub just to fix a typo in a blog post is an overkill that breaks the writing flow.
Single Source of Truth: I want to write technical notes once and have them automatically distribute to my community profile and my personal website.

The solution? Using the Dev.to API as a free, headless CMS for a React Single Page Application.

Here is a technical breakdown of how I designed this architecture

1. Fetching the Data (The API Layer)

Dev.to provides a fantastic, mostly undocumented perk for developers: a public, read-only API that returns your articles in JSON format.

Instead of hardcoding data, my React app fetches the article list directly from my Dev.to profile upon initialization. To prevent unnecessary network requests and ensure a 0ms load time when navigating back and forth, I implemented a simple in-memory cache outside the component lifecycle.

// Global cache for instant UI rendering
let cachedArticles: DevToArticle[] | null = null;

export function MainAdmin() {
  const [articles, setArticles] = useState<DevToArticle[]>(cachedArticles || []);
  const [loading, setLoading] = useState(cachedArticles === null);

  useEffect(() => {
    if (!cachedArticles) {
      fetch('[https://dev.to/api/articles?username=kkadzielawa](https://dev.to/api/articles?username=kkadzielawa)')
        .then(response => response.json())
        .then(data => {
          setArticles(data);
          cachedArticles = data; // Cache the response
          setLoading(false);
        })
        .catch(console.error);
    }
  }, []);

  // ... render logic
}

2. Dynamic Tag Filtering

Since the Dev.to API returns an array of tags (tag_list) for each article, I didn't need to build a custom taxonomy system. I simply extract all unique tags dynamically from the fetched payload and generate a terminal-like filter menu [ #linux ], [ #docker ], etc.

3. The "Optimistic UI" Trick

One downside of the /articles list endpoint is that it doesn't return the full Markdown body of the posts. To get the content, you have to make a second request to /articles/{id} when the user clicks on a specific note.

To prevent the annoying "layout shift" and loading spinners, I used React Router's state to pass the title of the article from the main list directly to the reader component.

<Link 
  to={`/note/${article.id}`} 
  state={{ title: article.title }} // Pass the title forward!
>
  {article.title}
</Link>

Thanks to this, the note viewer immediately renders the title and structure, while the actual Markdown body stream loads discreetly in the background.

4. Rendering Markdown with Syntax Highlighting

Once the API returns the body_markdown, it needs to be safely parsed into HTML. Since this is a SysOps blog, code snippets must look flawless. I combined react-markdown with react-syntax-highlighter using the VS Code Dark Plus theme for a native IDE feel.

import ReactMarkdown from 'react-markdown';
import { Prism as SyntaxHighlighter } from 'react-syntax-highlighter';
import { vscDarkPlus } from 'react-syntax-highlighter/dist/esm/styles/prism';

// Inside the render function:
<ReactMarkdown
  components={{
    code({ node, inline, className, children, ...props }: any) {
      const match = /language-(\w+)/.exec(className || '');
      return !inline && match ? (
        <SyntaxHighlighter
          style={vscDarkPlus as any}
          language={match[1]}
          PreTag="div"
          {...props}
        >
          {String(children).replace(/\n$/, '')}
        </SyntaxHighlighter>
      ) : (
        <code className="inline-code" {...props}>
          {children}
        </code>
      );
    }
  }}
>
  {content}
</ReactMarkdown>

Conclusion

By treating Dev.to as a completely touchless, headless CMS, I achieved a perfect Separation of Concerns. Dev.to handles the database, image hosting, SEO, and the writing experience. My React application acts purely as a customized, terminal-themed presentation layer.

My personal portfolio updates automatically. That's the exact kind of automation every IT Admin loves.

Now, whenever I want to drop some technical meat (like a new Linux configuration or securing a server environment), I just hit "Publish" on Dev.to, and it's instantly live.

Check out the live result at kadzielawa.dev.

Server-Side Rendering (SSR) vs. Client-Side Rendering (CSR): The Fascinating World of Page Rendering

Konrad Kądzielawa — Fri, 01 Dec 2023 15:28:31 +0000

The choice between Server-Side Rendering (SSR) and Client-Side Rendering (CSR) is not just a technological decision. It's like choosing between classic art and modern cinema - each has its unique characteristics and hidden treasures to discover.

Server-Side Rendering (SSR): Classical Elegance

Advantages of SSR:
SEO Magic: With SSR, your website is like a well-written book, easy to find and read by search engines.
Fast First Impression: SSR delivers content quickly, like an express courier.

Disadvantages of SSR:
Heavier Server Load: SSR is a bit like working full-time for the server - it requires continuous attention and resources.

SSR is like visiting an elegant, classical art gallery, where each artwork is carefully prepared and presented with attention to detail. In SSR, every website is like a finely crafted painting, ready to be admired immediately upon entry. This approach combines solidity and reliability, offering users and search engines complete, ready-to-consume content. As a result, SSR pages are like well-tailored suits: stylish, professional, and always in good taste, perfectly suited for the world of SEO and making a fast first impression.

Discover the secrets of SSR.

Client-Side Rendering (CSR): Modern Theater of Interaction

Advantages of CSR:
Mastery of Interaction: CSR is like a director who adapts the scene to the audience's mood - dynamic and full of surprises.
Server Resource Savings: It's like remote work for the server - fewer loads, more flexibility.

Disadvantages of CSR:
SEO Puzzle: For search engines, CSR pages can sometimes be like a complicated puzzle.
CSR is nothing but entering a modern theater of interaction, where each scene is created live, in response to the viewer's actions. CSR is like a dynamic, improvised performance in which the website's content unfolds and evolves in real-time in the user's browser. It's a world where interactivity and adaptability play the lead roles, and the website is constantly in motion, like a crowd in a bustling city center. CSR brings what is most modern and exciting in web development – a site that not only responds but dances to the user's rhythm.

Immerse yourself in the world of CSR.

Blending Dimensions: Hybrid Rendering Techniques

Using hybrid rendering methods is like the art of culinary fusion, where the chef combines traditional ingredients with modern cooking techniques to create surprising and complex dishes. In the world of web development, hybrid approaches blend the solidity and reliability of SSR with the flexibility and dynamism of CSR, creating unique "dishes" in the digital space.

Innovation in technology: Tools like Next.js for React are redefining the frameworks in which developers create web experiences. By combining the fast loading and SEO optimization offered by SSR with the reactivity and rich interactions of CSR, hybrid approaches create platforms that are both efficient and engaging.

Interesting IT fact: Using hybrid approaches in web design is like balancing between two worlds. Developers can decide which elements of the website are best rendered on the server-side and which on the client-side, similar to how a chef decides when to combine classic techniques with modern ones for the best effect.

Conclusion: Hybrid rendering techniques open up new possibilities in web development, allowing for the creation of websites that combine the best features of both worlds. Just as in culinary art, where the combination of old and new techniques creates amazing dishes, in web design, combining SSR and CSR creates unforgettable digital experiences.

Summary
The choice between SSR and CSR is not just a technical decision. It's a choice between two different ways of storytelling on the internet - one old and proven, and the other new and exciting. Understanding their unique features and benefits allows for the creation of better, more engaging websites.

Programming in 2033 – Vision of the Future of Software Development

Konrad Kądzielawa — Fri, 01 Dec 2023 15:20:47 +0000

In the last decade, the world of technology has undergone a metamorphosis, and software development has become a part of an exciting future. In 2033, the work of programmers is full of extraordinary experiences, seeming like something out of fantasy from years past, which we could only dream of not too long ago. Let's contemplate what their reality looks like... let's delve into this future, discovering the latest innovations and futuristic trends in technology.

Code-Generating Algorithms:
In the year 2033, programmers have become supervisors and process engineers, having long delegated some tasks to advanced code-generating algorithms. Thanks to advanced tools, programmers exclusively use a thought interface to communicate with artificial intelligence systems that create and optimize code according to their intentions.

#Example: A programmer thinks about a function to add two numbers. Text = "Create a function that adds two numbers." AI.create_code(text)

Discover how AI is changing programming.

The World of Cybersecurity:
In an era full of advanced hackers and AI, the importance of cybersecurity has reached its zenith. Programmers are now cyber-wizards, using tools based on neural algorithms to defend our digital worlds against constantly evolving threats.

Cybersecurity in 2033 is not just about fighting threats; it's also about the art of predicting potential attacks through predictive analysis.

Explore the latest trends in cybersecurity.

The Ubiquity of Clouds and Quantum Computing:
A key element of a programmer's work in 2033 is cloud computing and solutions based on quantum computing. Programmers use such computers to solve problems that were once thought impossible to solve in a traditional way. Serverless and microservices solutions have become the norm, requiring programmers to have the skills to manage complex, distributed systems.

Get to know the future of the cloud.

Virtual and Augmented Reality in Everyday Work:
Programmers in 2033 use VR and AR technologies not only for design but also as their daily work tools. Using thought interfaces, they design three-dimensional software models that can be "touched" and edited in virtual space.

In the world of VR, programmers work within their projects, moving through virtual spaces to solve problems and test applications.

Immerse yourself in the world of VR/AR for programmers.

The Blossoming of Soft Skills:
In an era where machines handle technical details, soft skills have become a key asset for programmers. Communication, creativity, and the ability to work in teams are essential for developing innovative solutions.

Programmers participate in simulations of customer interactions to refine their soft skills and understand user needs.

Learn more about soft skills in IT.

Never-ending Education:
In 2033, learning knows no bounds. Programmers are eternal students, using interactive learning interfaces that enable them to quickly grasp the latest technologies and concepts.

#Example: An interactive AI-based learning interface. AI.learn("New_Technology") AI.connect_with_mentor("Name Surname")

Check out the best educational platforms for programmers.

Conclusion:
The work of programmers in 2033 is an exciting journey into the future, full of interactions with intelligent algorithms, quantum computing, virtual worlds, and continuous learning. It's a world where humans and machines collaborate to create innovative solutions that transform the face of technological reality. Artificial intelligence, cloud solutions, VR/AR, and cybersecurity play crucial roles. This evolution of the profession demonstrates how adaptability and the ability to learn are essential in a dynamically changing world of technology.

DEV Community: Konrad Kądzielawa

Building a Local IT Assistant with Ollama, Open WebUI and a Custom Model Router

Final goal

Environment overview

Problem 1: A default model is not enough

Problem 2: Tools, Pipelines or Functions?

Problem 3: Open WebUI showed an Example Filter, but I needed a Pipe

The routing logic

Adding a safety-oriented system prompt

Problem 4: The router existed, but Open WebUI still showed qwen3

Problem 5: Containers must start after VM reboot

Problem 6: Models and data must survive container recreation

Testing the assistant

What I learned

What comes next

Post-Mortem: Diagnosing and Fixing Ceph Quorum Breakdown (Time Drift) in a Hybrid Proxmox VE Cluster

1. The Architecture and The Incident

2. Debugging & Log Analysis

3. The Action Plan: Fixing Time Drift & Rebuilding

Step 1: Enforcing Strict NTP on the External VM (admin-03)

Step 2: Enabling QEMU Guest Agent (admin-02)

Step 3: Reconstructing the Corrupted Monitor (admin)

4. Verification: Chaos Engineering

Proxmox & Ceph in a Home Lab: Building a Cluster from a Single Drive & Surviving the CLI Battle

Challenge 1: Handing an LVM Partition over to Ceph

Challenge 2: An External Host and a Rebellious virt-manager

Finale: Pools, Replication, and Green Status

Conclusion

DNS Troubleshooting on Windows: When the Internet Works but the Intranet Doesn’t

A very typical scenario

Why nslookup helps — but does not always tell the full story

How Windows actually resolves a name

A simple workflow I usually follow

1. Check the client’s DNS configuration

2. Compare the default result with a specific DNS server

3. Test the same name with Resolve-DnsName

4. Inspect the local DNS cache

5. Flush the cache if I suspect stale local state

6. Check whether the short name is the real issue

What usually turns out to be wrong

The question I actually want to answer

A quick checklist I like to keep in mind

Final thought

Further reading

Building a Zero-Maintenance SysOps Knowledge Base with React and the Dev.to API

1. Fetching the Data (The API Layer)

2. Dynamic Tag Filtering

3. The "Optimistic UI" Trick

4. Rendering Markdown with Syntax Highlighting

Conclusion

Server-Side Rendering (SSR) vs. Client-Side Rendering (CSR): The Fascinating World of Page Rendering

Server-Side Rendering (SSR): Classical Elegance

Client-Side Rendering (CSR): Modern Theater of Interaction

Blending Dimensions: Hybrid Rendering Techniques

Programming in 2033 – Vision of the Future of Software Development

Step 1: Enforcing Strict NTP on the External VM (`admin-03`)

Step 2: Enabling QEMU Guest Agent (`admin-02`)

Step 3: Reconstructing the Corrupted Monitor (`admin`)

Why `nslookup` helps — but does not always tell the full story

3. Test the same name with `Resolve-DnsName`