DEV Community

Cover image for How I Built a 100% Offline Local Toolchain Analytics Daemon in Python 3.10
Ruesch Manny
Ruesch Manny

Posted on

How I Built a 100% Offline Local Toolchain Analytics Daemon in Python 3.10

⚑ TL;DR β€” Ready-to-Deploy Automation Blueprints:


If you read the EULAs of modern development tools, you'll notice a disturbing trend: your operating system, your IDE, your AI assistants, and your CLI tools are all harvesting your usage data. They call it "telemetry" for product improvement, but for engineering teams, it's a massive leak of intellectual property and developer behavioral data.

I needed a way to track toolchain usage, IDE adoption, and local productivity metrics for my team without sending a single byte of data to a third-party server.

In this guide, I'll show you how to build a Local Toolchain Analytics & Productivity Engineβ€”a lightweight, privacy-first background daemon written entirely in Python 3.10+.


1. The Bottleneck: Why Commercial Solutions Fail Engineering Teams

There are plenty of time-tracking and developer analytics tools on the market (WakaTime, RescueTime, etc.). However, they all suffer from the same fundamental flaws when applied to strict enterprise environments:

  1. The Privacy Nightmare: They require constant internet connectivity to stream your active window titles, file names, and branch data to a centralized SaaS.
  2. Exorbitant Per-Seat Pricing: Paying $10-$20/month per developer just to know how often they use VS Code vs. JetBrains is an unjustifiable expense.
  3. Fragile Integrations: They rely on brittle IDE plugins rather than OS-level process monitoring, meaning CLI tools and standalone AI clients (like Claude desktop) often go untracked.

We don't need a bloated Electron app. We need a low-overhead, zero-dependency Python daemon that runs silently in the background, aggregates metrics into a local SQLite database, and exports structured JSON payloads for internal BI tools like Grafana or Metabase.


2. The Architecture: A Privacy-First Telemetry Engine

To ensure zero external telemetry and sub-1% CPU overhead, the architecture must be strictly local and asynchronous. Here is the blueprint:

  • The Observer: An asyncio event loop using the psutil library to poll active processes every 60 seconds.
  • The Sanitizer: A Python 3.10 match/case router that filters out OS noise and categorizes development tools (e.g., categorizing code, idea, and nvim under IDE).
  • The Aggregator: A local SQLite database running with PRAGMA journal_mode=WAL for lock-free, concurrent batch writes.
  • The Exporter: A Pydantic-powered microservice interface that dumps strictly typed JSON payloads for BI ingestion.

3. The Code & Logic

Let's dive into the core engine. We leverage Python 3.10's structural pattern matching and Pydantic for robust data validation.

The Data Models

First, define the schemas. Using Pydantic guarantees that our local database schemas remain perfectly aligned with the JSON exports expected by our BI tools.

from pydantic import BaseModel, Field
from datetime import datetime
from typing import Optional

class ProcessSnapshot(BaseModel):
    timestamp: datetime = Field(default_factory=datetime.utcnow)
    process_name: str
    category: str
    duration_seconds: int = 60
    metadata: Optional[dict[str, str]] = None

class AggregatedExport(BaseModel):
    date: str
    total_ide_minutes: int
    total_cli_minutes: int
    top_tools: list[str]
Enter fullscreen mode Exit fullscreen mode

The Background Observer

This is the heartbeat of the daemon. We use psutil to fetch active processes. To prevent CPU spikes, we don't track every process switch; we sample the active environment at fixed intervals.

import asyncio
import psutil
import sqlite3

# Standardize tool names using Python 3.10+ pattern matching
def categorize_process(proc_name: str) -> str | None:
    match proc_name.lower():
        case "code" | "idea" | "pycharm" | "nvim":
            return "IDE"
        case "iterm2" | "alacritty" | "terminal" | "tmux":
            return "CLI"
        case "docker" | "podman":
            return "Infrastructure"
        case _:
            return None  # Ignore OS noise

async def poll_toolchain_usage(db_path: str):
    conn = sqlite3.connect(db_path, isolation_level=None)
    conn.execute("PRAGMA journal_mode=WAL")
    conn.execute("""
        CREATE TABLE IF NOT EXISTS tool_usage (
            timestamp TEXT,
            process_name TEXT,
            category TEXT,
            duration_seconds INTEGER
        )
    """)

    while True:
        active_tools = []
        for proc in psutil.process_iter(['name']):
            try:
                name = proc.info['name']
                category = categorize_process(name)

                if category:
                    active_tools.append((name, category))
            except (psutil.NoSuchProcess, psutil.AccessDenied):
                continue

        # Batch write to SQLite
        if active_tools:
            cursor = conn.cursor()
            timestamp = datetime.utcnow().isoformat()
            records = [(timestamp, name, cat, 60) for name, cat in active_tools]
            cursor.executemany(
                "INSERT INTO tool_usage VALUES (?, ?, ?, ?)", 
                records
            )

        # Sleep for the polling interval
        await asyncio.sleep(60)
Enter fullscreen mode Exit fullscreen mode

Structured JSON Export

Your internal dashboards need clean data. We run an export job that rolls up SQLite data and validates it via Pydantic before dropping it into a pre-configured export directory.

import json

def export_daily_metrics(db_path: str, export_path: str):
    conn = sqlite3.connect(db_path)
    cursor = conn.cursor()

    # Aggregate today's metrics
    cursor.execute("""
        SELECT category, SUM(duration_seconds) / 60 
        FROM tool_usage 
        WHERE date(timestamp) = date('now')
        GROUP BY category
    """)
    results = cursor.fetchall()

    payload = {
        "date": datetime.utcnow().strftime("%Y-%m-%d"),
        "total_ide_minutes": 0,
        "total_cli_minutes": 0,
        "top_tools": []
    }

    for category, minutes in results:
        if category == "IDE":
            payload["total_ide_minutes"] = minutes
        elif category == "CLI":
            payload["total_cli_minutes"] = minutes

    # Validate and export
    export_data = AggregatedExport(**payload)
    with open(export_path, "w") as f:
        json.dump(export_data.dict(), f, indent=2)
Enter fullscreen mode Exit fullscreen mode

4. Deployment & Performance

Running a background script endlessly requires a few production safeguards:

  • Process Management: Do not run this inside a raw tmux session. On Linux, wrap the CLI in a systemd service with Restart=always. On macOS, use a launchd plist.
  • Resource Throttling: Ensure psutil handles AccessDenied exceptions gracefully. System-level processes will reject polling attempts. Catching these prevents the daemon from crashing.
  • Data Retention: SQLite files can grow indefinitely. Add a daily cleanup routine via an asyncio scheduled task to DELETE FROM tool_usage WHERE timestamp < datetime('now', '-30 days').

5. Conclusion & Turnkey Package

By keeping telemetry strictly local, you protect your intellectual property while still gaining critical insights into team productivity, IDE migration trends, and AI tool adoption. You can easily take the architecture and code snippets above and expand them into a fully-fledged monitoring tool.

If you want to skip the boilerplate, avoid OS-specific polling edge-cases, and deploy immediately, you can grab the complete, production-ready codebase. The turnkey package includes the full Python 3.10+ CLI tool, systemd/launchd configuration templates, automated test suites, and pre-built Grafana JSON schemas.

Grab the complete pipeline here:


πŸ› οΈ D'autres outils MR Studio qui pourraient vous intΓ©resser :

Top comments (0)