DEV Community: Mr Say Nothing

Systemd Service Not Starting? How to Fix It

Mr Say Nothing — Mon, 14 Sep 2026 16:49:12 +0000

A systemd service that won't start is almost never mysterious. Run systemctl status <unit>, then read the last 50 journal lines for that unit with journalctl -u <unit> -n 50 --no-pager — between the two, one of five causes is usually named outright: a bad path, a missing binary, wrong permissions, an SELinux/AppArmor denial, or a unit file syntax error. The failure reason is in the log; the fixes below are just pattern matching against it.

How do I see why a systemd service failed?

Status first, journal second:

systemctl status myapp.service
journalctl -u myapp.service -n 50 --no-pager

status gives you the state (inactive (dead), failed (exit-code), activating (auto-restart)) and the last few log lines. The journal gives you the full story: stdout, stderr, and systemd's own complaints about the unit.

If the unit failed before and you want the record of that run, add -b for the current boot or --since today:

journalctl -u myapp.service -b --no-pager

For the full toolkit — boots, priorities, following output live — see the journalctl cheat sheet. It's the same muscle memory.

One more pair worth knowing:

systemctl list-units --failed
systemctl reset-failed myapp.service

The first lists every red unit on the box. The second clears the failed state after you've fixed it — cosmetic, but it stops status pages from crying wolf.

What are the most common causes?

After the log names the symptom, the cause is almost always one of these five:

Log says	Likely cause	Fix
`status=203/EXEC`	Wrong `ExecStart=` path or missing interpreter	Absolute path, `chmod +x`, check shebang
`status=203/EXEC` on a script	Script has CRLF line endings or bad shebang	`dos2unix script.sh`, fix first line
`status=1/FAILURE`, no app log	Working directory or env var missing	Set `WorkingDirectory=`, add `Environment=`
`Permission denied`	User can't read files or bind the port	Fix ownership; ports below 1024 need `AmbientCapabilities=CAP_NET_BIND_SERVICE` or root
`Unit is masked`	Someone ran `systemctl mask`	`systemctl unmask myapp.service`
Unit file edit "does nothing"	Daemon not reloaded	`systemctl daemon-reload`

The 203/EXEC family deserves a special mention because it's the one that eats afternoons. Systemd does not use your shell to launch ExecStart=. That means:

# Wrong — no shell, ~ never expands, no PATH lookup
ExecStart=~/app/run.sh

# Right
ExecStart=/opt/app/run.sh

And the script itself must be executable and start with a real shebang (#!/bin/bash or #!/usr/bin/env bash). A script that runs fine from your terminal but fails with 203 under systemd is nearly always one of: not executable, CRLF endings, a shebang pointing nowhere, or a relative path.

Why does it start manually but not on boot?

The classic ordering bug. If the log shows failures right after boot but the unit starts fine when you systemctl start it by hand, your service is losing a race — it's reaching for the network, a mounted disk, or a database before that thing exists.

The fix is to declare dependencies instead of hoping:

[Unit]
After=network-online.target postgresql.service
Wants=network-online.target

[Service]
ExecStartPre=/usr/bin/test -f /opt/app/config.toml

After= orders the start; Wants= makes systemd actually bring the dependency up. network-online.target only works if the network-wait service is enabled on your distro, so check systemctl is-enabled NetworkManager-wait-online.service (or the systemd-networkd equivalent). The ExecStartPre= guard is a cheap, honest way to fail loudly with a readable message instead of a stack trace.

A second variant: the service starts on boot but immediately dies. Look for things your interactive environment had that boot doesn't — PATH differences, a virtualenv, a HOME. Set what you need explicitly:

[Service]
Environment="PATH=/opt/app/venv/bin:/usr/bin"
User=appuser
WorkingDirectory=/opt/app

Why is my service not logging to the journal?

If journalctl -u shows nothing, check three things in order:

StandardOutput= and StandardError= in the unit — they must be journal (the default) or journal+console. Someone may have set them to null.
The app writes to a file instead of stdout. Systemd only captures stdout/stderr; log-file writers bypass the journal entirely. Either point the app at stdout or read the file.
Storage limits dropped old lines: journalctl --disk-usage, and SystemMaxUse= in /etc/systemd/journald.conf if the journal is choking.

For debugging the start itself, nothing beats dropping a shell into the boot-time context:

[Service]
ExecStart=/bin/bash -c 'exec /opt/app/bin/server 2>&1'

Or, for genuine mysteries, run the exact ExecStart= command as the unit's user in a shell — most environment differences surface in the first ten seconds.

How do I make it restart automatically after a crash?

Default policy is Restart=no: a crashed service stays dead, and you find out from a user. Fix that per service:

[Service]
Restart=on-failure
RestartSec=5

Setting	Restarts when
`no` (default)	Never
`on-failure`	Non-zero exit, signal, timeout
`always`	Any exit, even clean
`on-watchdog`	Watchdog timeout only

Pair Restart=on-failure with a start-rate guard so a crashing loop doesn't hammer the box: StartLimitIntervalSec= and StartLimitBurst= in the [Unit] section. Five failures in sixty seconds should page a human, not spin a CPU.

If you're wiring this up as a scheduled job rather than a daemon, weigh cron vs systemd timer first — timers give you journal logging and dependency ordering for free, which is exactly what this article keeps reaching for.

The 60-second checklist

systemctl status <unit> — read the state and last lines.
journalctl -u <unit> -n 50 --no-pager — find the real error.
203/EXEC? Fix path, shebang, permissions. Permission denied? Fix user and file ownership.
Fails only at boot? Add After=/Wants=network-online.target and an ExecStartPre= guard.
Change a unit file? systemctl daemon-reload && systemctl restart <unit>.
Add Restart=on-failure so the next crash announces itself instead of hiding.

Most "systemd is complicated" moments reduce to the error was in the journal the whole time. Read it before you edit the unit file, not after.

GGUF Quantization: Which Level Should You Use?

Mr Say Nothing — Sun, 13 Sep 2026 16:39:46 +0000

Pick Q4_K_M by default; go Q6_K or Q8_0 when you have VRAM to spare and need the last few percent of quality. GGUF quantization shrinks a model's weights from 16 bits to fewer — Q4_K_M stores roughly 4.85 bits per weight, so a 7B model drops from ~14 GB to ~4.1 GB with perplexity typically less than 1% worse than the original. The question "which gguf quantization to use" has a boringly stable answer that most of the drama online obscures. Below: what quantization actually does to weights, how much quality each level costs, how to do the size math yourself, and one command to measure the damage on your own hardware instead of trusting a stranger's benchmark.

What does GGUF quantization actually do?

A model is trained in 16-bit floating point (FP16 or BF16): every one of its billions of weights is a 2-byte number. Quantization compresses each weight into fewer bits. The naive way — round every weight to a 4-bit integer — destroys small-but-important values, so GGUF uses two tricks:

Block scaling. Weights are grouped into blocks (usually 32), and each block gets its own scale factor. The 4-bit values are offsets within that block, so a wide range of magnitudes survives.
Importance-aware k-quants. The "K" in Q4_K_M means super-blocks of scales, plus treating attention and feed-forward layers differently from each other, because they tolerate compression unequally.

The "I" family (IQ4_XS and friends) goes further with information-theoretic codebooks borrowed from image compression. Same idea, fancier encoding: fewer bits per weight at similar quality, at the cost of slightly slower inference on some backends.

One clarification that prevents most confusion: quantization changes only the stored weights. Architecture, tokenizer, and context handling are untouched. A Q4 file and a Q8 file of the same model are the same model, wearing different coats.

Q4 vs Q8: is higher quantization better?

Yes, technically; no, perceptually. Using llama.cpp's own perplexity runs on Llama models as the reference: Q8_0 lands within ~0.02% of FP16 — for any practical purpose, lossless. Q6_K is near-indistinguishable. Q4_K_M gains roughly 1–2% perplexity, Q4_0 a bit more, and Q2_K is where coherent answers start falling apart on small models.

Two rules the numbers imply:

Model size buys quantization headroom. A 70B model survives Q2/Q3 far better than a 7B model does, because larger models are more redundant. Quantizing a 7B to Q2 is amputation; quantizing a 70B to Q3 is tailoring.
The quality floor moves with the task. Chat tolerates Q4. Exact code generation, math, and RAG over precise documents expose quantization noise sooner. If a Q4 model keeps writing subtly wrong code, test the same model at Q6_K before you blame the model.

Level	Bits/weight	Size vs FP16	Quality loss	Use it when
Q2_K	~3.4	~21%	Severe on sub-13B	Nothing else fits, large models only
Q3_K_M	~3.9	~25%	Noticeable	Tight VRAM, ≥14B models
Q4_K_S	~4.6	~29%	Small	Q4_K_M won't fit and it's close
Q4_K_M	~4.85	~30%	~1% perplexity	The default. Best quality/size trade
Q5_K_M	~5.7	~35%	~0.5%	VRAM available, quality-critical tasks
Q6_K	~6.6	~41%	Near-nil	Code/math, still fits comfortably
Q8_0	~8.5	~53%	Effectively none	Reference runs, fine-tune bases
IQ4_XS	~4.3	~27%	≈Q4_K_M	Q4_K_M slightly too big, backend supports i-quants

How much VRAM does each level need?

Do the size math yourself instead of memorizing tables — it is one line:

size_GB ≈ (bits_per_weight × params) / 8
# 8B model @ Q4_K_M: 4.85 × 8 / 8 ≈ 4.9 GB
# 8B model @ Q8_0:   8.50 × 8 / 8 ≈ 8.5 GB

Then add the parts the formula leaves out: the KV cache (grows with context length — a few hundred MB to multiple GB), activations, and compute buffers. Practical margin: a "4.9 GB" model wants a 6 GB card at 4k context, and flash-attention plus KV-cache quantization to stay there at 16k. Weights are the headline, not the whole bill.

Which GGUF quantization should you use?

Decision order, no exceptions worth memorizing:

Compute your context + KV budget first, then weights. Context you can't fit is worse than quality you can't measure.
Default to Q4_K_M. It is the community's default for a reason — roughly 1% perplexity for 70% of the size. Every registry, including Ollama's, ships it as the baseline.
Step up to Q6_K when the task punishes noise: code, math, extraction, anything you feed to a pipeline unattended.
Use Q8_0 only for reference — A/B tests, quantization-damage measurement, or a fine-tune base. As a daily driver it mostly buys warmth in your VRAM sensors.
Go below Q4 only under duress, and only on large models. Test with a known-hard prompt before trusting it.

If you are choosing which file to download on Hugging Face, prefer a single Q4_K_M.gguf over sharded splits unless the uploader only ships the latter — fewer moving parts. And if you are choosing where to run it, the engine choice is separate: see llama.cpp vs Ollama for that axis.

How do you measure quantization damage yourself?

Benchmarks differ; your prompt is constant. Build llama.cpp once, download two levels of the same model, and measure both perplexity (lower is better) and tokens/second:

git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp
cmake -B build && cmake --build build --config Release -j

huggingface-cli download bartowski/Meta-Llama-3.1-8B-Instruct-GGUF \
  Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf Meta-Llama-3.1-8B-Instruct-Q8_0.gguf \
  --local-dir models

# perplexity on a wiki-text chunk (lower = closer to the original model)
./build/bin/llama-perplexity -m models/Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf -ngl 99
./build/bin/llama-perplexity -m models/Meta-Llama-3.1-8B-Instruct-Q8_0.gguf  -ngl 99

# and speed on the same hardware
./build/bin/llama-bench -m models/Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf -ngl 99

The Q4_K_M number will be a few hundredths of a point worse than Q8_0 and the file will be ~40% smaller. If your downstream task can't tell the difference — and for most, it can't — you have your answer without reading anyone's leaderboard.

Does quantization hurt privacy or local-only claims?

No — it is arithmetic on weights, entirely offline, and the quantized file is just a smaller container of the same parameters. Running a Q4_K_M locally leaks exactly as much (or little) as running the full-precision model locally: nothing leaves the machine. The privacy-relevant variable is where inference runs, not the bit width. The usual caveats about model provenance apply equally to every quant level: a stolen-base "uncensored" fine-tune in Q8 is not safer than the same weights in Q4. For the file-format side of this, see how to run GGUF models locally.

Which level should you pick?

Q4_K_M, and stop reading forums about it. Upgrade to Q6_K for precision-hungry work if VRAM allows, keep one Q8_0 around for A/B comparisons, and treat anything under Q4 as an emergency ration for large models only. The one mistake worth avoiding is symmetric: worrying about Q4-vs-Q5 while ignoring context length, which wrecks more local setups than any quant ever did.

Pipeline test

Mr Say Nothing — Sun, 13 Sep 2026 11:57:49 +0000

Validation body, leaving now.