DEV Community

Cover image for Building a Predictive Memory Policy Engine: From Sampling to Graceful Degradation
GLA LOCAL AI SYSTEMS
GLA LOCAL AI SYSTEMS

Posted on • Edited on

Building a Predictive Memory Policy Engine: From Sampling to Graceful Degradation

#ai #github #infrastructure #discuss

Building a Predictive Memory Policy Engine: From Sampling to
Graceful Degradation

The Problem

You deploy a Node.js inference service. It runs fine for hours. Then memory creeps up. RSS balloons. The container OOMKills at 3 AM. Your users get 502s. You wake up to PagerDuty.
The usual fix? Set a memory limit and restart when it hits 80%. That works until it doesn't. Memory doesn't grow linearly. Workloads spike. Batch inference doubles heap in seconds. A fixed threshold is a blunt instrument.

We needed something smarter. Something that watches, learns, and acts before the crash.
What We Built
A memory policy engine that does four things in a loop:

Sample — Read heap, RSS, system RAM, swap, GPU VRAM, cgroup limits
Learn — Build per-workload baselines and detect anomalies
Predict — Forecast memory usage 60 seconds ahead
Act — Apply tiered responses from logging to graceful degradation
Enter fullscreen mode Exit fullscreen mode

All local. No external dependencies. Works on a Raspberry Pi, a Kubernetes pod, a mobile device, or a desktop GPU workstation.
Scale: From 2GB IoT to 128GB+ Servers

The engine auto-detects its environment and adapts. We tested across six device classes:
Table

Device Class    Typical RAM Use Case    Default Warn    Default Critical
IoT / Edge  2–4 GB    Embedded inference, sensors 70% 85%
Mobile (iOS/Android)    4–8 GB    On-device ML, camera apps   75% 90%
Browser / WASM  2–8 GB    Client-side inference   75% 90%
Desktop / Workstation   16–64 GB  Local GPU inference 80% 95%
Container / Pod 4–32 GB   K8s microservices   80% 92%
Server / Bare Metal 64–128+ GB    Production model serving    85% 95%
Enter fullscreen mode Exit fullscreen mode

A 2GB Raspberry Pi and a 128GB inference server run the same engine. The defaults shift. The logic stays identical.
The Architecture
plain

┌─────────────────────────────────────────────────────────────┐
│                    MEMORY POLICY ENGINE                      │
├─────────────────────────────────────────────────────────────┤
│                                                              │
│  ┌─────────────┐    ┌─────────────┐    ┌─────────────┐    │
│  │   SAMPLER   │───→│   LEARNER   │───→│   DECIDER   │    │
│  │  (5s loop)  │    │  (patterns) │    │ (thresholds)│    │
│  └─────────────┘    └─────────────┘    └─────────────┘    │
│         │                  │                  │              │
│         ▼                  ▼                  ▼              │
│  ┌─────────────────────────────────────────────────────┐  │
│  │              WEIGHTED SCORING ENGINE                 │  │
│  │  heap=1.0  rss=0.8  sys=0.6  cgroup=1.2  gpu=0.7   │  │
│  └─────────────────────────────────────────────────────┘  │
│                              │                             │
│                              ▼                             │
│  ┌─────────────────────────────────────────────────────┐  │
│  │              TIERED ACTION PIPELINE                │  │
│  │  warn → throttle → drop → kill → emergency        │  │
│  └─────────────────────────────────────────────────────┘  │
│                              │                             │
│                              ▼                             │
│  ┌─────────────────────────────────────────────────────┐  │
│  │              EVENT EMITTER (20+ events)             │  │
│  │  memory:sampled  memory:warn  memory:critical      │  │
│  │  memory:throttle  memory:leak_detected             │  │
│  └─────────────────────────────────────────────────────┘  │
│                                                              │
└─────────────────────────────────────────────────────────────┘
Enter fullscreen mode Exit fullscreen mode

Sampling: Reading Every Memory Source
Most tools read process.memoryUsage() and call it done. We read five sources:
Table
Source What It Tells You Platform
Node.js heap V8 heap used / total All
RSS Resident set size All
System RAM os.totalmem() / freemem() All
Swap /proc/meminfo, vm_stat, wmic Linux, macOS, Windows
GPU VRAM nvidia-smi, rocm-smi, Metal Linux, macOS
Cgroup memory.limit_in_bytes, memory.max Linux containers

JavaScript

// What a sample looks like
const sample = {
  timestamp: Date.now(),
  heapUsedPct: 45.2,
  rssPct: 62.1,
  sysUsedPct: 58.3,
  cgroupPct: 71.4,
  gpuVramPct: 33.0,
  swapUsedPct: 12.0,
  deviceType: 'container'
};
Enter fullscreen mode Exit fullscreen mode

The engine auto-detects whether it is running in a container, on a mobile device, an IoT board, a server, or a browser. Each context gets different default thresholds.
Mobile Device Support
Mobile is not an afterthought. iOS and Android have different memory pressure signals:
Android
Detected via ANDROID_ROOT or TERMUX_VERSION. The engine knows Android's low-memory killer (LMK) can terminate processes before the system OOMs. Thresholds are conservative:
bash


export MEMORY_POLICY_MODE=adaptive
export MEMORY_MOBILE_LOW_MODE=true
export MEMORY_SAMPLE_INTERVAL_MS=5000
Enter fullscreen mode Exit fullscreen mode

iOS
Detected via PLATFORM_NAME=iphoneos or the presence of /System/Library/CoreServices/SystemVersion.plist. iOS sends didReceiveMemoryWarning to the app delegate. The engine treats this as an early warn signal:
bash

export MEMORY_POLICY_MODE=adaptive
export MEMORY_WARN_PCT=75
export MEMORY_CRITICAL_PCT=90
Enter fullscreen mode Exit fullscreen mode

On both platforms, the engine reduces sampling frequency if the device is thermal-throttling and avoids SQLite writes during low-power mode to preserve battery.
Weighted Scoring: Not All Memory Is Equal
Heap pressure is more urgent than swap pressure. Cgroup limits are harder than system limits. We weight each metric:
plain

┌────────────────────────────────────────┐
│         WEIGHTED SCORING               │
├────────────────────────────────────────┤
│  heap     × 1.0  =  45.2              │
│  rss      × 0.8  =  49.7              │
│  sys      × 0.6  =  35.0              │
│  cgroup   × 1.2  =  85.7              │
│  gpuVram  × 0.7  =  23.1              │
├────────────────────────────────────────┤
│  weightedSum / totalWeight =  63.4%   │
│                                        │
│  Thresholds: warn=80%  critical=92%    │
│  Level: normal                         │
└────────────────────────────────────────┘
Enter fullscreen mode Exit fullscreen mode

The weighted percentage is what drives the level decision. Not any single metric in isolation.
Learning: Building a Baseline Per Workload
Static thresholds fail because workloads differ. A model serving API idles at 30% heap. A batch embedding job idles at 70%.
The engine learns an EWMA baseline per device type and mode:
JavaScript


// After 100 samples on a container running adaptive mode//
const baseline = {
  avgHeapPct: 42.3,
  avgRssPct: 51.7,
  avgSysPct: 38.1,
  sampleCount: 100,
  lastUpdated: 1720454400000
};
Enter fullscreen mode Exit fullscreen mode

If the baseline is already high (avg weighted > 60%), thresholds tighten automatically. If the baseline is low and stable (> 50 samples), thresholds loosen. The engine adapts to your workload, not a global average.
Anomaly patterns are tracked separately. Every warn or critical sample feeds an anomaly record. If three critical anomalies occur, the critical threshold drops by 10 points.
Prediction: Linear Regression 60 Seconds Ahead
In predictive mode, the engine runs linear regression on the last N samples:
plain


slope = (nΣxy − ΣxΣy) / (nΣx² − (Σx)²)
intercept = (Σy − slope·Σx) / n

futureHeap = slope · futureX + intercept
Enter fullscreen mode Exit fullscreen mode

If the forecast shows heap crossing critical in the next 60 seconds, the level is bumped to critical immediately. The response happens before the crash, not after.
JavaScript

// Predictive status output//
{
  forecastWindowMs: 60000,
  predictedHeapPct: 94.2,
  timeToCritical: 'imminent',
  confidence: 0.85
}
Enter fullscreen mode Exit fullscreen mode

Tiered Actions: From Log to Emergency
Actions are configurable per level. A typical setup:
Table
Level Default Actions What Happens
warn log, alert Emit event, log metrics, notify tracker
critical throttle, drop, alert Shrink inference context, shed non-critical requests
Throttle
JavaScript

// Context size: 8192 → 4096
// Batch size: 32 → 16
// GPU layers: 40 → 30
Enter fullscreen mode Exit fullscreen mode

The engine talks to the inference config layer. It reduces batch size, context window, and GPU offload layers proportionally. Not a hard stop — a controlled slowdown.
Drop
Non-critical requests are rejected for 60 seconds. The queue drains. Memory pressure eases. Critical inference jobs keep running.
Kill
If the supervisor module is present, leaking services are restarted. Not the whole process — just the identified leak source.
Emergency
The nuclear option:
JavaScript

// Zero GPU layers, minimum threads, force GC//
process.env.GPU_LAYERS = '0';
process.env.CONTEXT_SIZE = '2048';
process.env.NUM_THREADS = '1';
if (global.gc) global.gc();
Enter fullscreen mode Exit fullscreen mode

This is gated behind RBAC. Only admin, system, or owner roles can trigger emergency. A confirmation event is emitted for audit.
Device-Aware Defaults
The engine detects its environment and sets sensible defaults:
plain

┌─────────────┬──────────┬──────────┬─────────────┐
│ Device      │ Warn %   │ Critical │ Sample Int. │
├─────────────┼──────────┼──────────┼─────────────┤
│ IoT         │ 70       │ 85       │ 10s         │
│ Mobile      │ 75       │ 90       │ 5s          │
│ Container   │ 80       │ 92       │ 5s          │
│ Server      │ 85       │ 95       │ 5s          │
│ Desktop     │ 80       │ 95       │ 5s          │
│ Browser     │ 75       │ 90       │ 5s          │
└─────────────┴──────────┴──────────┴─────────────┘
Enter fullscreen mode Exit fullscreen mode

IoT devices get conservative thresholds and slower sampling to minimize flash wear. Servers get headroom because they have swap and monitoring infrastructure.
All defaults are overridable via environment variables. The engine never assumes it knows better than the operator.
Leak Detection
Memory leaks are not threshold events. They are growth rate events.
The engine tracks heap growth over the last hour:
JavaScript

const growthRate = ((last.heapUsed - first.heapUsed) / first.heapUsed) * 100;
// per hour
Enter fullscreen mode Exit fullscreen mode

If growth exceeds the configured threshold (default 10%/hour for servers, 5%/hour for IoT), thresholds tighten by 10 points. The engine assumes a leak is in progress and becomes more aggressive.
Event-Driven Integration
The engine extends EventEmitter. Every state transition emits a named event:


JavaScript

const policy = new MemoryPolicyEngine();

policy.on('memory:warn', ({ sample, thresholds }) => {
  // Send to your alerting system
  pagerDuty.trigger('memory-warn', sample);
});

policy.on('memory:throttle', ({ sample }) => {
  // Log to your metrics backend
  prometheus.gauge('inference_throttled').set(1);
});

policy.on('memory:leak_detected', ({ growthRate }) => {
  // Page the on-call if growth rate is extreme
  if (growthRate > 20) {
    slack.post('#alerts', `Memory leak: ${growthRate.toFixed(1)}%/hour`);
  }
});

policy.startSampler();
Enter fullscreen mode Exit fullscreen mode

No polling. No polling your polling. Just events when things change.
DevOps Usage Patterns
Pattern 1: Kubernetes Sidecar
Deploy the engine as a sidecar in your inference pod. Mount a shared volume for SQLite state. The sidecar samples, learns, and emits memory:throttle events. Your main container listens and adjusts batch size via an internal API.
yaml

# Conceptual — adapt to your stack
containers:
  - name: inference
    env:
      - name: MEMORY_POLICY_MODE
        value: predictive
      - name: MEMORY_ACTION_CRITICAL
        value: throttle,drop,alert
  - name: memory-policy
    env:
      - name: MEMORY_DATA_DIR
        value: /shared/memory
Enter fullscreen mode Exit fullscreen mode

Pattern 2: IoT Edge Device
On a Raspberry Pi running embedded inference:
bash

export MEMORY_POLICY_MODE=adaptive
export MEMORY_IOT_FLASH_AWARE=true
export MEMORY_SAMPLE_INTERVAL_MS=30000
export MEMORY_ACTION_WARN=log
export MEMORY_ACTION_CRITICAL=throttle,emergency
Enter fullscreen mode Exit fullscreen mode

The engine slows sampling to 30 seconds, learns the edge workload baseline, and throttles inference before the Pi freezes.
Pattern 3: GPU Workstation
On a local CUDA workstation:
bash


export MEMORY_GPU_VRAM_WARN_PCT=85
export MEMORY_GPU_VRAM_CRITICAL_PCT=95
export MEMORY_THROTTLE_GPU_LAYERS_REDUCTION_PCT=25
Enter fullscreen mode Exit fullscreen mode

VRAM pressure triggers GPU layer reduction before the CUDA OOM killer strikes.

Pattern 4: Mobile App (iOS/Android)
In a React Native or native mobile app running on-device inference:
bash

export MEMORY_POLICY_MODE=adaptive
export MEMORY_MOBILE_LOW_MODE=true
export MEMORY_WARN_PCT=75
export MEMORY_CRITICAL_PCT=90
export MEMORY_SAMPLE_INTERVAL_MS=10000
Enter fullscreen mode Exit fullscreen mode

The engine respects low-memory warnings, reduces SQLite writes during thermal throttling, and never triggers kill/emergency without confirmation.
Building On Top: Custom Actions
The action pipeline is extensible. Override executeActions or listen to events:
JavaScript

policy.on('memory:critical', async ({ sample }) => {
  // Your custom logic
  await scaleDownWorkers();
  await notifyTeam();
  await writePostMortemStub(sample);
});
Enter fullscreen mode Exit fullscreen mode

The engine provides the signal. You provide the response.
Offline-First, Cloud-Optional
Every feature works without internet. SQLite persistence is local. Learning is local. Events are local. If you enable federation, the engine emits memory:federation_offload_requested — but only if you explicitly opt in via

INTERNET_ENABLED=true
.
No phoning home. No telemetry. No external API dependencies.
What We Learned

Fixed thresholds are a liability. A workload-aware baseline reduces false positives by an order of magnitude.
Prediction beats reaction. Linear regression on 60 seconds of samples catches spikes before they become outages.
Tiered actions prevent overreaction. Logging, throttling, dropping, and emergency are separate controls. Use the lightest tool that solves the problem.
Device context matters. An 80% threshold is generous on a server and fatal on an IoT board or mobile device.
Events over polling. Event-driven integration keeps your monitoring stack simple and responsive.
Scale is not an excuse. The same engine runs on a 2GB Pi and a 128GB server. The defaults shift. The logic stays identical.
Enter fullscreen mode Exit fullscreen mode

Try It
The engine is a single Node.js module. No build step. No external services.
JavaScript

const { MemoryPolicyEngine } = require('./memoryPolicy');

const engine = new MemoryPolicyEngine({
  runtimeContext: {
    userId: 'service-account',
    tenantId: 'production',
    sector: 'inference'
  }
});

engine.on('memory:warn', console.warn);
engine.on('memory:critical', console.error);
engine.startSampler();
Enter fullscreen mode Exit fullscreen mode

Tune thresholds. Watch it learn. Let it predict.

Available for discussions!

GLAY SYSTEMS
Absolute sovereignty...

Top comments (0)