DEV Community

Kengah Ireneaus
Kengah Ireneaus

Posted on

sysk a CLI health monitor in bash and python, how i built it

#python #bash #linux #automation

Sysk architecture

Design notes on Sysk, a small system health monitor.

Intro

Sysk is a command-line system health monitor built in Bash and Python.

Bash collects machine data, handles flags, and drives the user interface. Python runs the inference and decision engines.

This article records the design of v1: what the system does, which decisions shaped it, and where those decisions cost something.

System design

The pipeline is four stages:

collect --> infer --> decide --> print
Enter fullscreen mode Exit fullscreen mode

v1 collects data across five modules:

  • cpu
  • disk
  • memory
  • thermal
  • sound

Each module presented a different reading problem. The rest of the article follows the pipeline above.

1. Data collection (Bash)

Decision 1: read kernel files instead of running helper programs

The main collection decision was to read kernel-exported files rather than treat programs such as top or free as the source of truth.

Advantages

  • fewer extra binaries, so fewer runtime dependencies
  • one read model per module
  • data taken from the same place the kernel already exposes
  • no wait for a helper tool to start and format output

Disadvantages

  • you parse kernel-formatted text yourself
  • each file has its own layout, so the learning curve is steep

Files and directories used

/proc/meminfo          -> memory
/proc/cpuinfo          -> cpu identity
/proc/loadavg          -> load average
/proc/stat             -> cpu accounting
/proc/uptime           -> uptime

/sys/block             -> disks
/sys/class/thermal     -> thermal zones and cooling
/sys/class/hwmon       -> hardware monitors (fans, sensors)
Enter fullscreen mode Exit fullscreen mode

This rule is not absolute. Hardware identity and sound did not fit it cleanly. dmidecode, smartctl, and pactl are still dependencies. Sound, in particular, is read through pactl rather than a kernel file.

Finding the right file for each module was messy. Once the path was known, the output looked cryptic at first glance, but it was stable enough to parse.

Decision 2: many scripts, one entry point

Reading files raised the next question: how should the project be structured?

Each major function became its own script, sourced from a single entry point. That was easy to say and immediately produced four portability questions:

  • How does the program find itself, no matter where it is launched from?
  • How do scripts share variables without hard-coding absolute paths?
  • In what order should scripts be sourced so that dependencies exist before they are used?
  • What does “runs on another machine” actually require?

Those questions produced the portable layout.

1. Resolve the application path

export SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd -P)"
Enter fullscreen mode Exit fullscreen mode

This is the usual “where am I?” idiom. A pure string-manipulation version exists, but the cd / pwd -P form follows symlinks and is the one Sysk uses.

2. Export directory names from that root

export SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
export XDG_CONFIG_HOME=${XDG_CONFIG_HOME:-${SCRIPT_DIR}/.config}
export DUMPS_PATH=${DUMPS_PATH:-${SCRIPT_DIR}/logs}
export CACHE_PATH=${CACHE_PATH:-${SCRIPT_DIR}/.cache}
export CONFIG_DIR="$CACHE_PATH/sysk"
export ENGINE_DIR=${ENGINE_DIR:-${SCRIPT_DIR}/engine}
export RULE_DIR="$ENGINE_DIR/.rules"
export RESULT_DIR="$ENGINE_DIR/results"
export DATE="$(date +%Y_%m_%d)"
Enter fullscreen mode Exit fullscreen mode

Defaults can be overridden by the environment. Paths are derived from SCRIPT_DIR instead of being pasted in as /home/kit/....

3. Source scripts in a fixed order

core -> device scripts -> hardware scripts
Enter fullscreen mode Exit fullscreen mode 
for script in "$CORE_DIR"/*.sh; do
    source "$script"
done

for dir in "$DEVICE_DIR"/*; do
    for script in "$dir"/*.sh; do
        source "$script"
    done
done

for script in "$HW_DIR"/*.sh; do
    source "$script"
done
Enter fullscreen mode Exit fullscreen mode

Core files are numbered so the source order is obvious:

00_error.sh
01_privileged.sh
02_check_deps.sh
03_install_deps.sh
04_cleanup.sh
05_parse_flags.sh
tui.sh
Enter fullscreen mode Exit fullscreen mode
  • 00_error.sh — named error codes
  • 01_privileged.sh — root / sudo handling
  • 02_check_deps.sh — missing binaries
  • 03_install_deps.sh — install those binaries
  • 04_cleanup.sh — cleanup and traps
  • 05_parse_flags.sh — getopts
  • tui.sh — text interface

Privileged work

Some reads (dmidecode, parts of disk data) need elevated permission. A one-line sudo "$@" was not enough. The questions were:

  • what if the user is already root?
  • what if sudo is missing?
  • what if the user refuses permission?
  • how do you drop sudo again after the privileged step?

01_privileged.sh answers those with small predicates:

is_root() {
    [[ $EUID -eq 0 ]] && return "$ERR_SUCCESS"
    return "$ERR_FAILURE"
}

is_sudo_available() {
    command -v sudo >/dev/null 2>&1 && return "$ERR_SUCCESS"
    return "$ERR_FAILURE"
}

is_sudo_active() {
    sudo -n true >/dev/null 2>&1 && return "$ERR_SUCCESS"
    return "$ERR_FAILURE"
}

kill_sudo() {
    is_sudo_active && {
        sudo -k
        return "$ERR_SUCCESS"
    }
    printf 'sudo not active, aborting...\n' >&2
    return "$ERR_BAD_USAGE"
}
Enter fullscreen mode Exit fullscreen mode

Checking and installing dependencies

Even with a file-first design, Sysk still depends on a few programs: dmidecode, smartctl, pactl, and jq.

The flow is:

check dependencies -> get permission -> install missing ones
Enter fullscreen mode Exit fullscreen mode
  1. Store the required names in an array.
  2. Run command -v on each name; keep the missing ones.
  3. Ask for permission.
  4. Detect the package manager and install the missing set.
  5. Confirm the set is present. 
declare -r PACKAGE_MANAGERS=(
    apt dnf pacman zypper emerge
)

os_pkg_manager=""

get_pkg_manager() {
    for pkg in "${PACKAGE_MANAGERS[@]}"; do
        if command -v "$pkg" >/dev/null 2>&1; then
            os_pkg_manager=$pkg
            return 0
        fi
    done
    printf 'unknown package manager, aborting...\n' >&2
    exit 1
}
Enter fullscreen mode Exit fullscreen mode

Collection per module

Each module binds the files it cares about and hides the parsing behind a writer.

CPU

readonly CPU_INFO_FILE="/proc/cpuinfo"
readonly LOAD_INFO_FILE="/proc/loadavg"
readonly CPU_USAGE_FILE="/proc/stat"
readonly UPTIME_INFO_FILE="/proc/uptime"
Enter fullscreen mode Exit fullscreen mode

Memory

readonly MEM_INFO_FILE="/proc/meminfo"
Enter fullscreen mode Exit fullscreen mode

Disks

readonly DRIVE_DIR="/sys/block"

mapfile -t drives < <(
    find "$DRIVE_DIR"/* -maxdepth 1 -printf '%f\n' | grep -Ev '(^loop|ram|zram)'
)
Enter fullscreen mode Exit fullscreen mode

Thermal

readonly SYSTEM_THERMAL_ZONE_PATH="/sys/class/thermal/thermal_zone"
readonly SYSTEM_FAN_INFO_PATH="/sys/class/hwmon"
Enter fullscreen mode Exit fullscreen mode

Sound — collected through pactl, not /proc.

Storing collected data

Snapshots are JSON, written with jq and a per-module filter file:

write_memory_json() {
    local MEM_CONFIG="$CONFIG_DIR/memory_$DATE.json"
    mkdir -p "$CONFIG_DIR"

    jq -n \
        --arg total_mem "$(get_total_mem)" \
        --arg available_mem "$(get_available_mem)" \
        -f "$FEATURE_DIR/build_memory.jq" \
        > "$MEM_CONFIG"
}
Enter fullscreen mode Exit fullscreen mode

Example thermal snapshot:

{
  "average_temp": 31.87,
  "thermal_zones": 3,
  "zone_temparatures": {
    "acpitz": "29.80",
    "x86_pkg_temp": "38.00"
  },
  "cooling": {
    "fan_zones": "2",
    "fan_speed": "fan1_input=\"0\"\nfan2_input=\"0\""
  }
}
Enter fullscreen mode Exit fullscreen mode

Problem. The JSON shape is not the same across modules. Some values are scalars, some are objects, some are nested. That leaked into inference and into the rule files.

Fix for v1.

  • each module has its own rule file
  • each module has its own build_*.jq filter
  • a source field in the YAML tells the engine where to read
  • a resolve function walks that path

A uniform snapshot schema is still future work.

2. Inference and decision (Python)

Inference pipeline:

load data -> load rules -> set status per field -> write result
Enter fullscreen mode Exit fullscreen mode

The engine is a Python script. It uses the standard library plus PyYAML, and it reads the same environment variables Bash exported (CACHE_PATH, RULE_DIR, RESULT_DIR, DATE).

Loading 

def load_rules(rule_path: Path) -> dict:
    with rule_path.open(encoding="utf-8") as rule:
        return safe_load(rule)

def load_json(data_path: Path) -> Any:
    try:
        with data_path.open(encoding="utf-8") as data:
            return load(data)
    except (FileNotFoundError, OSError, JSONDecodeError) as err:
        print(f"[ERROR] failed to load {data_path}: {err}", file=sys.stderr)
        sys.exit(1)
Enter fullscreen mode Exit fullscreen mode

Rules live in engine/.rules as YAML. Each field names the value it wants through source:

module: cpu
check_interval: 10
system_vars:
  cpu_cores: "auto"

fields:
  - name: cpu_usage_percent
    source: "usage"
    type: "percentage"
    base_value: 100
    warning_multiplier: 0.83
    critical_multiplier: 0.90
    unit: "%"

  - name: load_avg5
    source: "load_avg.1"
    type: "load"
    base_value: "cores"
    warning_multiplier: 1
    critical_multiplier: 1.5
    unit: ""
Enter fullscreen mode Exit fullscreen mode

The collector JSON could not be renamed. The mapping therefore lives in the rules, not in a hard-coded Python dictionary.

Decision 3: a path language instead of one parser per module

source is a small path:

  1. a bare key for a top-level field — usage
  2. dots for nesting — core_info.cores_usage
  3. an integer segment for a list index — load_avg.1

resolve walks that path:

def resolve(data: dict, source: str) -> Any:
    current = data
    for obj in source.split("."):
        if obj.isdigit():
            current = get_list_value(current, int(obj))
        else:
            current = get_dict_value(current, obj)
    return current
Enter fullscreen mode Exit fullscreen mode

The helpers catch KeyError, IndexError, and TypeError, print on stderr, and exit. resolve itself stays a straight loop.

Because cpu, memory, and thermal still do not share one value shape, evaluate dispatches to a small private helper per module (_evaluate_cpu, and so on) and merges the results into one dictionary. v1 loads the three modules it already has rules for. Disk and sound collection exist; their inference rules are not finished in the same way.

The public main in inference.py is only orchestration: check environment variables, check input paths, create RESULT_DIR if needed, evaluate, write result_{DATE}.json.

Decision engine 

load result -> find warning and critical fields -> write a cause log if needed
Enter fullscreen mode Exit fullscreen mode

The decision script does not re-parse /proc. It reads today’s result file.

The decision I made for v1: if every field is OK, exit 0 and write nothing extra. If any field is not OK, write cause_{DATE}.log and return a non-zero status (90) so Bash can tell an alert from a clean run.

3. Flags, display, and the entry point

v1 uses short flags only, parsed with getopts:

parse_args() {
    while getopts ':hvrm:' opt; do
        case "$opt" in
            h) usage; exit 0 ;;
            v) printf '%s version %s\n' "$PROGNAME" "$VERSION"; exit 0 ;;
            # ...
        esac
    done
}
Enter fullscreen mode Exit fullscreen mode

Action flags such as -h and -v print and exit inside the parser. Setting flags only store values. The TUI runs only when the session is a terminal and quiet mode is off.

Main procedure

All of the pieces meet in one function:

check files and directories
parse flags
install missing dependencies
collect snapshots
run inference
run decision
show the TUI unless -q was passed
exit
Enter fullscreen mode Exit fullscreen mode 
main() {
    check_req_files || exit "$ERR_FAILURE"
    check_req_dirs  || exit "$ERR_FAILURE"

    parse_args "$@"
    install_missing_deps

    write_cpu_json &
    write_memory_json
    write_thermal_json
    write_disk_json
    write_sound_json

    mkdir -p "$DUMPS_PATH"
    python3 "$ENGINE_DIR/inference.py" && python3 "$ENGINE_DIR/decision.py"

    display_tui || exit "$ERR_FAILURE"
    exit "$ERR_SUCCESS"
}

main "$@"
Enter fullscreen mode Exit fullscreen mode

Decision 4: collect in the background where it hurts

The first runs collected modules one after another. CPU sampling waits about one second to compute usage, so the whole collect stage felt slow.

write_cpu_json & starts that slow job in the background so the other writers can proceed. That made the collect stage feel almost instant.

This only stays correct if the entry point waits for the background job before inference starts. Otherwise Python can open a CPU file that is not finished yet. A wait after the writers is part of this design, not an optional extra.

What v1 is not

The project is small on purpose.

  • no long options
  • no previous-day comparison of result files
  • no uniform JSON schema across modules
  • disk and sound are collected; they are not fully inferred like cpu, memory, and thermal
  • the TUI is a banner and a status list, not a full-screen toolkit

Dated result files are already named result_{DATE}.json so a later version can compare runs without changing the collector.

Layout 

sysk/
├── lib/
│   ├── core/          # 00_error.sh ... 05_parse_flags.sh, tui.sh
│   ├── devices/
│   └── hw/
├── engine/
│   ├── .rules/        # cpu.yml, memory.yml, thermal.yml
│   ├── inference.py
│   ├── decision.py
│   └── results/
└── sysk               # entry point
Enter fullscreen mode Exit fullscreen mode

That is the architecture of v1: files in, JSON snapshots, YAML rules, a small path walker, a verdict, and a Bash wrapper that the user actually types. Many features could still be added. The shape above is what is running now.

if you are interested in the project you can check it out at https://github.com/4kit1-glitch/sysk

thanks for being along with me in my journey

Top comments (0)