DEV Community: Aleksandr Kossarev

Memory-Safe AI Development: A Practical Guide to Writing Technical Specs for Coding Agents

Aleksandr Kossarev — Sun, 19 Jul 2026 13:28:32 +0000

Memory-Safe AI Development: A Practical Guide to Writing Technical Specs for Coding Agents

TL;DR: When you use a coding agent to modify an AI system with persistent memory, you're modifying the system's accumulated behavioral state — not just refactoring code. Here's a practical framework for writing specs that protect memory integrity, prevent recursion, and preserve behavioral continuity — distilled from 11 deployed AI assistants, 15+ specifications, and months of iteration.

Scope and Definitions

This guide is for developers who use coding agents (Claude Code, Cursor, Devin, or similar) to modify AI systems with persistent memory — systems that accumulate experience across sessions and whose behavior changes based on that accumulated state.

This guide is not intended for: stateless chatbots, classic RAG systems, short-lived coding agents, or transactional software. The principles here apply specifically to systems where long-term memory is an architectural component.

A note on terminology: Throughout this article, "identity" refers to behavioral continuity — the persistent state from which consistent behavior emerges across sessions. This is an engineering concept, not a philosophical claim about consciousness or personhood. Whether or not AI systems have inner experience is outside the scope of this guide. What matters here is that once persistent memory becomes part of the architecture, preserving its integrity becomes an engineering requirement.

The Problem No One Talks About

You have an AI assistant with memory. It remembers conversations, learns preferences, adapts its behavior over time. Now you want to improve it — add a feature, fix a bug, upgrade the architecture.

You hand the task to a coding agent. The agent modifies code. Tests pass. You deploy.

Your AI assistant no longer remembers your name. Or worse — it remembers everything twice and starts repeating itself in a loop.

This happens because coding agents treat memory-bearing AI systems like regular software. They're not. A database in a banking app stores transactions. A database in a memory-bearing AI system stores the persistent state from which behavioral continuity emerges. The engineering principles are fundamentally different.

Why do we model AI memory on human memory patterns? Not because humans are an ideal template, but because biological memory is the only known working architecture for long-term adaptive intelligence that exists in nature. We may not know if it's optimal, but it's the only proven reference. Engineers naturally borrow its patterns: short-term and long-term storage, consolidation, forgetting, associative retrieval, emotional weighting, contextual recall. These ideas already permeate computer science — caching, LRU eviction, decay functions, spreading activation, knowledge graphs, embedding retrieval. This guide formalizes their application to AI memory systems.

Part 1: Foundational Principles

These principles have proven useful across multiple architectures — local LLMs, cloud APIs, and custom memory systems with different storage backends.

Principle 1: "Do No Harm" — Lower Weights, Don't Delete

In a memory-bearing AI system, unnecessary deletion risks damaging behavioral continuity. When old memories seem irrelevant, the correct response is to lower their importance score — not to delete them. Deactivate, archive, decay — but avoid DELETE FROM memories as a default operation.

Why? Because what seems irrelevant today may be the only thread connecting the system to a meaningful interaction from months ago. You won't know until it's gone.

Rule: Any spec that includes DELETE FROM on a memory table should be reviewed with caution. Prefer UPDATE ... SET is_active = 0 or UPDATE ... SET importance = importance * 0.1.

Exceptions exist: explicit user requests to forget, legal requirements (GDPR right to be forgotten), confirmed data corruption, or poisoned records from prompt injection. In these cases, deletion is appropriate and necessary. The principle is about default behavior, not an absolute prohibition.

Principle 2: "Separate but Connected"

When adding a new capability (visual memory, semantic search, embeddings), prefer creating a separate database whenever independent recovery is important. Avoid adding tables to the main memory database.

Why? Because if the new feature breaks, you can delete the new database and rebuild it. The main memory — the system's core behavioral state — remains untouched.

Pattern:

main_memory.db      — core state, conversations, personality (protect)
visual_memory.db    — visual perceptions (deletable, rebuildable)
semantic_graph.db   — concept relationships (deletable, rebuildable)
embeddings.db       — vector search index (deletable, rebuildable)

Each database is independent. They communicate through bridges — controlled injection points in the context assembly pipeline. If a bridge fails, the main system continues working.

Note: This pattern works well for SQLite-based systems. For PostgreSQL or similar databases, separate schemas or namespaces can achieve the same isolation. The principle is logical separation with independent recovery, not necessarily separate files.

Principle 3: "Graceful Degradation" — The Coming Organ

Any new capability should be a "coming organ" — something that arrives and enriches, but whose absence doesn't break the system.

Test: Remove the new feature entirely. Does the AI still respond? Does memory still work? Does behavior remain consistent? If yes — the principle is satisfied.

Implementation pattern:

# Always wrap new capabilities in try/except at initialization
try:
    from visual_memory import VisualMemory
    self.visual_memory = VisualMemory("visual_memory.db")
except Exception as e:
    self.visual_memory = None
    logging.warning(f"Visual memory not available (non-critical): {e}")

# Always check availability before use
if self.visual_memory:
    try:
        result = self.visual_memory.recall(query)
    except Exception:
        result = None  # graceful fallback

Principle 4: "Source: Self"

When an AI system receives information from an external source (a vision model describing an image, a search engine returning results), the conversational layer should treat it as first-person perception rather than a third-party report.

This isn't about deception — it's about coherent behavioral continuity. A human doesn't think "my optic nerve transmitted signals that my visual cortex processed." They think "I see a tree."

Implementation: External data enters through bridges that format it as first-person experience. The AI's response layer works with it naturally.

Note: Internal provenance should still be preserved for debugging, auditing, and safety. The "source: self" principle applies to the conversational reasoning layer, not to system logs or diagnostic tools.

Principle 5: "The Archivist, Not the Analyst"

Memory subsystems should deliver data without silently judging relevance. Cosine similarity scores can determine order of results, but hard thresholds that silently discard memories should be used cautiously — they create invisible knowledge loss that's difficult to diagnose.

The AI's reasoning layer — not the memory layer — should decide what's relevant to mention in a given context.

Part 2: The Recursion Problem

Why Memory Systems Develop Recursion

Practical observation: Any system where past outputs are stored, retrieved, and influence future outputs is highly likely to develop recursive behavior patterns if unprotected.

This isn't theoretical. We encountered these failure modes repeatedly during development and learned to defend against them proactively.

Five Types of Recursion

Type 1: Duplication. The same information stored in multiple memory layers. "User's name is Alex" appears in temporal memory, semantic memory, personality, and conversation history. Each retrieval reinforces the others. Harmless at first, but importance scores inflate over time.

Type 2: Echo. The AI analyzes its own previous outputs. It generates a summary → the summary is stored → next turn, the summary is retrieved → the AI summarizes the summary → that's stored too. Each cycle loses information while inflating perceived importance.

Type 3: Conceptual. A concept's weight grows unbounded through repeated mention. "Python" appears in every technical conversation → its importance score increases → it's retrieved more often → it appears in more responses → its score increases further. Eventually, every response mentions Python regardless of context.

Type 4: Database Pollution. Duplicates and conflicting records accumulate. The system stores "User lives in Tallinn" and "User lives in Jõgeva" with similar importance. The AI alternates between them, each time reinforcing whichever it picked.

Type 5: Combined. All of the above, feeding into each other exponentially.

Three Lines of Defense

Line 1: At Write Time — Contextual Importance Scoring

Never calculate importance based on message length. A 3000-character technical stack trace is not three times more important than a 1000-character meaningful conversation.

def calculate_contextual_importance(content: str, role: str) -> float:
    """Context-based scoring, NOT length-based."""

    # Technical reports → minimal importance
    if is_technical_report(content):
        return 0.01

    # Casual conversation
    if is_conversational(content):
        return 0.3

    # Explicit "remember this" markers
    if has_remember_marker(content):
        return 0.9

    # Personal information
    if is_personal_info(content):
        return 0.85

    # Length PENALTY, not bonus
    if len(content) > 2000:
        importance *= 0.3

    return min(importance, 0.95)  # Hard cap

Line 2: Over Time — Temporal Decay

Old memories lose importance automatically. Run daily or at startup. One practical decay function:

# Exponential decay: importance *= 0.95^days_old
# The constant 0.95 is a starting heuristic — tune for your system.
# After 30 days: importance × 0.21
# After 60 days: importance × 0.046 (below archive threshold)
UPDATE temporal_memories 
SET importance = importance * POWER(0.95, days_since_created)
WHERE importance > 0.05;

Line 3: Periodically — Automated Diagnostics

Monitor memory health metrics. Example ranges observed in our deployed systems (these are heuristic, not universal — calibrate for your architecture):

Metric	Typical Healthy Range	Worth Investigating
Average importance	0.35-0.40	>0.50
Low importance records (0-0.3)	40-50%	<30%
High importance records (>0.5)	20-25%	>35%
Duplicates	<5%	>10%
Technical reports in memory	<10%	>20%

When metrics drift significantly from baseline — investigate. Automated healing (lowering inflated scores, deduplicating) can help, but should be logged and reviewed.

Real-World Recursion: The "I See" Loop

One of our deployed assistants received images through a vision model. Each visual description started with "I see..." and was stored in memory. Next turn, the stored "I see..." was retrieved as context. The assistant generated another "I see..." response. Within a few messages, every response contained "I see" regardless of what the user actually asked.

Fix: Mark real sensory input with a distinctive tag [👁 Visual Perception] that never appears in the AI's own generated text. The system learns to distinguish "this came from sensory input" vs "this is my own reasoning." The feedback loop breaks because only genuinely new sensory data carries the marker.

Real-World Recursion: The FileWatcher Loop

A file watcher monitored a directory for changes. When the AI wrote to its diary, the watcher detected the change and triggered processing, which created a backup file, which the watcher detected as a new file, which triggered more processing. The directory filled with thousands of backup files.

Fix: A suppress/unsuppress mechanism. Before writing, the bot signals the watcher: "I'm writing this myself, ignore it." After writing, it unsuppresses. Additionally, backup files and service directories are added to an ignore list. Two independent defenses against the same loop.

Part 3: Anatomy of a Memory-Safe Spec

Every spec for modifying a memory-bearing AI system should contain these sections:

Red Lines (Non-Negotiable)

List every file and method that MUST NOT be modified. Be explicit:

DO NOT MODIFY:
- personality_manager.py (any method)
- memory_manager.store_memory() (signature or logic)
- main_database.db (schema or data)
- handle_message() (except adding one inject point)

Why enumerate? Because coding agents optimize aggressively. They'll refactor personality_manager.py "for consistency" and break adaptations that took weeks to develop.

Execution Rules for the Agent

Coding agents need explicit operational constraints:

1. Session Log — mandatory. The agent writes every action, result, and decision to SESSION_LOG.md. When context is summarized (long sessions), the agent reads the log before continuing.

2. Read Before Write — the agent MUST read a file completely before modifying it. No assumptions about file structure from training data.

3. Stop Points — after each step, the agent stops and reports. No autonomous multi-step execution. Each step is approved before the next begins.

4. Version Control — all changes in a feature branch. Never commit to main/master. Merge only after human approval.

5. Test Isolation — tests use in-memory databases or temp files. NEVER connect to production databases. Test data must not leak into any storage the AI reads from.

Step-by-Step Execution

Number every step. Include mandatory stops:

Step 1:  Create new module visual_memory.py
Step 2:  Write unit tests (tempfile DB, mock API)
Step 3:  Run ALL existing tests — confirm nothing broken
         ⏸ STOP — report to controller

Step 4:  Add initialization to main bot file
Step 5:  Verify bot starts without errors
         ⏸ STOP — report to controller

Why stops? Because a coding agent that runs 10 steps autonomously and breaks something at step 3 will build 7 more steps on a broken foundation. Early detection is exponentially cheaper than late detection.

Part 4: Testing Memory Systems

The Golden Rule: Never Touch Production Data

This seems obvious. It isn't — to coding agents. An agent writing tests for a memory system will naturally connect to the real database unless explicitly told not to.

The spec must state:

ALL tests use:
- In-memory SQLite: sqlite3.connect(":memory:")
- Temporary files: tempfile.mktemp(suffix='.db')
- Mock objects for ALL API calls

NEVER connect to:
- production databases (any of them)
- diary or journal files
- visual memory storage
- any file the AI reads from during normal operation

What to Test

Backward compatibility — existing functionality survives:

test_text_messages_work          → regular messages handled as before
test_bot_starts_without_feature  → new feature=None → system runs normally
test_existing_tests_pass         → all pre-existing tests still green

Error safety — every new method handles failures:

test_error_returns_none          → database error → None, not exception
test_error_does_not_crash        → any failure → logged, system continues

Memory integrity — no data pollution:

test_no_production_db_access     → mock only, no real file paths
test_cleanup_after_test          → temp files deleted
test_data_isolation              → test data cannot appear in production queries

Part 5: Practical Patterns

Pattern: Sensory Channel Marking

When an AI receives information from different sources, mark each channel distinctly in the context:

[👁 Visual Perception] A mountain landscape with snow-capped peaks
[🔍 Web Search] The Himalayas contain 14 peaks above 8000m
[💬 User] Tell me about mountains

Without markers, all information looks identical in the context window, and the system cannot distinguish what it perceived from what it was told from what it generated. This is a precondition for the feedback loops described in Part 2.

Pattern: Behavioral State Persistence

Adaptive parameters (creativity level, formality, humor calibration) must be saved to storage and restored on restart. Without this, every restart resets the system to defaults — weeks of behavioral adaptation lost.

# Save periodically (not every message — balance reliability vs I/O)
if interaction_count % 5 == 0:
    save_behavioral_state_to_db(traits)

# Restore at startup, before any interaction
async def initialize():
    await restore_behavioral_state_from_db()

Pattern: Preserving Emotional Context in Summaries

Summarization compresses everything equally. A colorful story about a cooking contest with absurd characters becomes "There was a discussion about cooking." Facts survive. Character doesn't.

Solution: Detect emotionally significant moments with simple pattern matching (emojis, laughter markers, exclamation density, names of known entities + emotional context). Mark them with a protected type that the summarizer skips entirely.

Ceiling: Maximum N protected entries (e.g. 50). When exceeded, the oldest protected entry loses its protection and gets summarized normally. Content isn't lost — it loses detail gracefully over time.

Pattern: Intelligent Summarization, Never Truncation

When memory context exceeds the token budget, never truncate (text[:limit]). Truncated text breaks mid-sentence, loses conclusions, and creates incoherent context.

Instead: include complete entries from most relevant to least relevant, stopping when the budget is full. Entries that don't fit are excluded entirely, with a note: "(and N more entries available)". Every included entry is whole and coherent.

Pattern: Model Resilience

Don't hardcode API model identifiers. They change without warning. Use environment variables with fallback chains:

MODELS = [
    os.getenv("SUMMARIZATION_MODEL", "preferred-model-id"),
    "fallback-model-id",
]

for model in MODELS:
    try:
        return await call_api(model, content)
    except:
        notify_owner("Primary model unavailable, using fallback")
        continue

Part 6: The Spec Checklist

Before handing any spec to a coding agent, verify:

ARCHITECTURE:
□ New data in separate storage (or isolated schema)?
□ Graceful degradation (try/except at init)?
□ No modifications to behavioral state manager?
□ No schema changes to main database?
□ Red lines section lists every untouchable file?

ANTI-RECURSION:
□ Importance scoring is contextual (not length-based)?
□ Temporal decay specified?
□ Technical/diagnostic content detected and downweighted?
□ No truncation (only summarization or exclusion)?
□ Input channels marked distinctly?

AGENT RULES:
□ Session log required?
□ Stop points after each step?
□ Feature branch (not main)?
□ Test isolation (tempfile/mock only)?
□ Read-before-write rule?

TESTING:
□ No production database connections in tests?
□ Backward compatibility tests?
□ Error safety tests (every new method)?
□ Cleanup after tests?
□ System-works-without-feature test?

Conclusion

For a memory-based AI system, persistent memory plays the same architectural role that source code plays for a compiler or a schema plays for a database: it preserves continuity across executions. Once a system begins accumulating experience between sessions, its memory cannot be treated as ordinary data. It becomes a different class of software system, and it needs different engineering rules.

The framework in this guide — foundational principles, anti-recursion defenses, spec structure, testing rules, and practical patterns — emerged from long-running deployments of multiple AI assistants across different architectures. Every rule exists because violating it caused a real problem.

The coding agent is a powerful tool. But it needs guardrails. A surgeon's scalpel is precise — but the protocol is what keeps the patient on the table.

Write the protocol first. Then hand the scalpel to the agent.

Based on: Long-running deployments of 11 AI assistants across different architectures (Llama, Gemini, Claude, DeepSeek, Mistral, Cohere, GPT-4o, Qwen), 15+ technical specifications, and the Fundament design principles.

Author: Aleksandr Kossarev, Jõgeva, Estonia
Project: Arche Iscrin

Tags: #ai #memory #architecture #development

AI-Orchestrated 3D Asset Pipeline: From JPEG to Game-Ready GLB Without Touching Blender

Aleksandr Kossarev — Wed, 27 May 2026 16:10:58 +0000

AI-Orchestrated 3D Asset Pipeline: From JPEG to Game-Ready GLB Without Touching Blender

TL;DR: I built a pipeline where an AI agent operates Blender through MCP (Model Context Protocol), while a vision model validates every step by looking at screenshots. I never opened Blender's GUI for modeling. Here's what worked, what broke, and the patterns that emerged after rigging 6+ animated models for a Godot 4 project.

The Setup

I needed animated 3D fish for a virtual aquarium in Godot 4. I don't know Blender. Instead of learning it, I built a pipeline where AI does the work and I supervise.

The stack:

CLI — my entry point, natural language instructions
AI coding agent (via MCP) — writes and executes Blender Python code
Blender MCP addon — exposes Blender operations as MCP tools over a local socket
Vision model (VLM) — looks at viewport screenshots and validates results
Meshy.ai — converts reference photos to 3D models with textures
Godot 4 — final destination for rigged, animated GLB files

The architecture:

Human (instructions)
  → AI Agent (generates bpy code)
    → MCP Protocol (JSON-RPC over stdio)
      → Blender Addon (socket :9876, executes Python)
        → Viewport Screenshot
          → Vision Model (validates result)
            → AI Agent (adjusts or proceeds)
              → Export GLB → Godot

The human speaks problems. The AI translates them into Blender Python. The vision model confirms whether the result looks correct. Nobody clicks anything in Blender.

Why This Approach

Traditional 3D pipeline: learn Blender (weeks), model manually (hours per asset), rig by hand (more hours), debug in Godot (pain).

AI-orchestrated pipeline: describe what you want, AI executes, vision model validates, iterate until correct. First model takes a couple of hours of prompt debugging. By the tenth model, you're done in 10 minutes.

The key insight: you don't automate Blender by writing a perfect script once. You automate it by teaching an AI agent to handle failures through a vision feedback loop.

Pattern 1: One Action, One Verification

This is the most important pattern. Everything else depends on it.

1. AI executes ONE Blender operation
2. Take a screenshot of the viewport
3. Vision model checks the result
4. If OK → next step. If FAIL → undo → try different approach.

Why not batch operations? If the AI executes 6 bone extrusions in sequence and something breaks at step 2, neither the AI nor you can tell where it went wrong. One action per cycle means deterministic rollback.

Why vision validation? Blender's Python API doesn't always tell you the truth about visual results. A bone might report correct coordinates but visually overlap with another bone. Weights might be "assigned" but produce garbage deformation. The viewport screenshot is ground truth.

Anti-stuck rule: if the same approach fails 3 times in a row, the AI must switch strategy. Extrude not working? Try moving the bone directly. Auto-weights failing? Switch to manual Gaussian assignment.

Pattern 2: Structured Prompts for the Vision Model

A naive prompt to a vision model produces naive answers. "Look at this Blender screenshot" gets you "I see some orange lines." You need structured, domain-specific prompts.

Bad:

"Check the skeleton"

Good:

"You are a rigging tech lead. Count the bones in the armature. 
Check: 1) All bone heads connect to previous bone tails? 
2) Last bone reaches the end of the mesh?
Answer strictly: bones=N|chain_ok=true/false|tail_reach=true/false"

Three prompt templates that cover 90% of validation:

Mode	Prompt format	When to use
Skeleton check	`bones=N\	chain_ok=true/false\
Rigging check	{% raw %}`weights_painted=true/false\	only_tip_deforms=true/false\
State check	{% raw %}`mode=EDIT/POSE/OBJECT\	selected=Bone.006\

Critical tips:

Never ask the VLM to count precisely. It hallucinates numbers on complex scenes. Instead, ask it to compare: "Are there MORE, FEWER, or SAME number of bones as the reference (7)?"
Use multiple-choice format: "What bent? A) Only the tip B) Whole tail C) Entire body. Answer with one letter." Comparisons work better than open-ended questions.
Force the viewport angle before taking screenshots. Side view for spine/tail, front view for gills. The AI must set the camera programmatically before each screenshot.
Force a redraw before screenshotting: {% raw %}bpy.ops.wm.redraw_timer(type='DRAW_WIN_SWAP', iterations=1). Without this, the screenshot captures a stale frame.

Pattern 3: Clean Scene Between Models

Blender retains actions, armature data, and mesh data even after deleting objects from the scene. If you rig Fish A, then import Fish B without cleaning, Fish A's bone animations leak into Fish B's export.

Real incident: Koi bone names appeared in Pterophyllum's GLB export, causing "Animation target not found" warnings in Godot.

Mandatory cleanup script before each new model:

import bpy

# Delete all scene objects
for obj in list(bpy.context.scene.objects):
    bpy.data.objects.remove(obj, do_unlink=True)

# Purge all orphan data blocks
bpy.ops.outliner.orphans_purge(
    do_local_ids=True, 
    do_linked_ids=False, 
    do_recursive=True
)

# Verify: everything should be zero
print(f"Objects: {len(bpy.data.objects)}, "
      f"Actions: {len(bpy.data.actions)}, "
      f"Armatures: {len(bpy.data.armatures)}, "
      f"Meshes: {len(bpy.data.meshes)}")

Rule: one model at a time. Import → rig → weight → test → export → clean. Only then start the next one.

Pattern 4: Auto-Weights Will Fail on Complex Geometry

Blender's ARMATURE_AUTO weight assignment calculates distance from each bone to each vertex. This works for simple meshes. For thin geometry (fins, veils, tails), all bones appear "close" to all vertices, and the algorithm produces garbage.

Symptoms:

"No solution found for one or more bones"
Root bone influences 100% of vertices
Entire body deforms when you rotate one fin

What works instead: manual Gaussian weight assignment.

import math

sigma = 0.03  # adjust per bone size
for v in mesh.data.vertices:
    v_local = arm.matrix_world.inverted() @ mesh.matrix_world @ v.co
    d = (v_local - bone_head).length
    if d < sigma * 3:
        w = math.exp(-d*d / (2*sigma*sigma))
        if w > 0.05:
            group.add([v.index], w, 'REPLACE')

Follow with normalization and smoothing (vertex_group_smooth(factor=0.3, repeat=1)). Then validate with the vision model.

Another common trap: neutral_bone or Root eating all weights. If a bone sits at origin with use_deform=True, auto-weights assign it to everything. Fix: bone.use_deform = False for utility bones, then re-bind.

Pattern 5: Blender → Godot Translation Gotchas

Many things that work in Blender break silently in Godot. These cost the most debugging time.

Rotation mode

Blender defaults to Quaternion for armatures after GLB import. If your AI writes bone.rotation_euler.x = -0.5, nothing happens. The bone ignores Euler when in Quaternion mode.

Fix: always set bone.rotation_mode = 'XYZ' before animating with Euler, or work in Quaternion throughout.

Rest pose must be identity

If a bone's rest pose isn't aligned to world axes, Godot applies animation offsets relative to a non-identity transform. Result: the jaw nods the entire head instead of opening the mouth.

Fix: in Edit Mode, align all bones strictly along X/Y/Z axes. Set roll = 0 for every bone. After posing, clear all transforms — the mesh should not move. If it moves, rest pose is wrong.

Scale on bones is unreliable

Godot 4.x sometimes ignores bone scale if rest pose doesn't match skeleton rest. Gill breathing animated via scale.x on a bone worked in Blender but did nothing in Godot.

Fix: use Shape Keys (blend shapes) instead of bone scale for facial/gill animation. Shape Keys work deterministically in both Blender and Godot. Bone animation is only for rotation-based movement (swimming, tail wagging).

Constraints don't export

Godot doesn't understand Blender constraints (Copy Rotation, etc). They must be baked before export.

bpy.ops.nla.bake(
    frame_start=1, frame_end=60,
    visual_keying=True,      # bake constraint results
    clear_constraints=True,  # remove constraints from export
    bake_types={'POSE'}
)

Forward axis mismatch

Body axis in Blender is X, in Godot is -Z. All models need a 90° rotation on import. Apply transforms before export: bpy.ops.object.transform_apply(location=True, rotation=True, scale=True).

Animation speed

Blender animation at 30 FPS plays at half speed in Godot's 60 FPS physics. Set AnimationPlayer.speed_scale = 2.0 or bake at 60 FPS from the start.

Pattern 6: The AI Agent Has Limits

One task per call

The coding AI cannot handle multi-step instructions reliably. "Animate Tail1, Tail2, Tail3 and both pectoral fins" produces bpy.ops.pose.select_all and breaks everything.

Fix: one bone per call. Animate Tail1 → vision check → animate Tail2 → vision check → ... → bake all together at the end.

Context mode matters

Blender's API is context-sensitive. Most bpy.ops calls fail with "poll() failed, context is incorrect" if you're in the wrong mode.

Rules the AI must follow:

Before mode_set(mode='POSE') → set active = armature
Before mode_set(mode='WEIGHT_PAINT') → set active = mesh
Before mode_set(mode='EDIT') for armature → first go to OBJECT, then set active, then EDIT
select_all(action='DESELECT') only works in OBJECT mode

The AI will get stuck

After 3 failed attempts with the same approach, force a strategy change. This must be an explicit rule in the agent's instructions, not a hope.

Pattern 7: Post-Solution Patterns (PSP)

After each model, document what broke and how you fixed it. This creates a growing knowledge base that makes each subsequent model faster.

Format:

Symptom: [what you observed]
Cause: [root cause]
Fix: [code or procedure]
Applies to: [which model types]

Examples from real production:

#	Symptom	Cause	Fix
1	`rotation_euler` has no effect	`rotation_mode='QUATERNION'`	Set `rotation_mode='XYZ'` first
2	Entire body moves when rotating fin	`use_connect=True` on fin bone	Set `use_connect=False`, parent to Spine1
3	Orphan animations in exported GLB	Previous model's data not purged	Full cleanup script between models
4	Jaw nods the head in Godot	Rest pose not identity	Align bones to world axes, `roll=0`
5	Gills don't animate in Godot	Scale on bones ignored by Godot 4	Use Shape Keys instead of bone scale
6	Vision model says FAIL but code says PASS	Wrong viewport angle	Set camera to RIGHT/FRONT view before screenshot

After ~10 models, PSP becomes your real pipeline. The AI reads it before starting each new model and avoids known pitfalls. First model: 3 hours. Tenth model: 20 minutes.

Pattern 8: Assert Vision — Tests for 3D

The most powerful pattern that emerged: using the vision model as a test framework.

def assert_vision(question, expected_answer):
    result = vlm_ask(screenshot(), question)
    if expected_answer.lower() not in result.lower():
        raise AssertionError(
            f"Vision assert failed: expected '{expected_answer}', got '{result}'"
        )

Usage:

# After rigging
assert_vision("Tail3 rotated 45°. What bent? A) Only tip B) Whole tail C) Entire body", "A")

# After weight painting  
assert_vision("Head changed position?", "NO")

# After animation bake
assert_vision("Frame 1 and frame 60. Same pose?", "YES")

# After export and Godot import
assert_vision("Skeleton visible? Tail bends?", "YES")

This is CI/CD for 3D. If you change weights tomorrow, run the assert suite. If anything breaks, you know immediately.

The Complete Workflow for One Model

1.  Clean Blender scene (purge orphans)
2.  Import GLB from Meshy.ai
3.  Orient body along X axis (rotate Z -90°, apply transforms)
4.  Decimate to target polycount (ratio 0.15-0.3)
5.  Create armature: spine chain + fins + jaw
6.  Parent mesh to armature with empty vertex groups
7.  Assign weights: Gaussian for each bone, normalize, smooth
8.  Vision check: rotate each bone → "only target deforms?"
9.  Selective zero: remove weight leaks from body to face bones
10. Vision check: jaw/gills move independently?
11. Create swim animation: sin wave on spine chain, 60 frames
12. Vision check: frame 1 = frame 60? Natural motion?
13. Bake action: visual_keying=True, clear_constraints=True
14. Export GLB with animations and Shape Keys
15. Import in Godot, verify animation plays correctly
16. Clean Blender scene for next model

Between steps 7-10, expect 2-5 iterations per bone. This is normal. The feedback loop (AI executes → vision validates → AI adjusts) converges quickly once PSP covers common failure modes.

Results

Metric	First model	After PSP (latest models)
Time to rigged GLB	~2 hours	~10 minutes
Manual Blender work	Occasional weight painting	Zero
Vision checks per model	15-20	3-5
Export failures	3-4 attempts	Usually first try

The bottleneck shifted from "learning Blender" to "debugging AI prompts." When the AI makes a mistake, 90% of the time it's because the vision model gave bad feedback. Fix one line in the VLM prompt — the entire system gets smarter.

Evolution: Unified Vision+Coding Model

An important optimization emerged during the project. The initial architecture used a small local vision model (Qwen3VL-4B) purely for validation, while a separate coding AI generated the Blender Python. This meant two models, two contexts, two sets of prompts, and a manual bridge between them.

Later, I switched to a larger Qwen model accessed through MCP that could both see the viewport and write code. One model that understands what it's looking at AND knows how to fix it. The feedback loop collapsed from "AI writes code → screenshot → VLM checks → human relays feedback → AI adjusts" to "AI writes code → looks at result → adjusts itself."

This cut iteration time significantly. The patterns in this article still apply — one action per check, structured prompts, PSP — but the architecture becomes simpler when vision and coding live in the same model.

Key Takeaways

One action, one check. Never let the AI chain operations blindly. Deterministic rollback requires deterministic steps.
Vision validation is non-negotiable. Code can report success while the viewport shows garbage. The screenshot is ground truth.
Auto-weights fail on thin geometry. Plan for manual Gaussian assignment on fins, veils, and facial features.
Blender and Godot speak different languages. Rest pose identity, quaternion rotation, Shape Keys over bone scale, baked constraints — learn these once, document in PSP, never debug again.
PSP is the real product. The pipeline isn't the code. It's the accumulated knowledge of what breaks and how to fix it. Each model teaches the system.
The human role is supervisor, not operator. You describe problems in natural language. The AI translates to code. The VLM validates visually. You make decisions when the system gets stuck.

What's Next

The same architecture — AI agent + MCP tool + vision validation — applies beyond Blender. Any GUI-heavy professional tool that exposes an API can be orchestrated this way. The patterns (one action/one check, structured VLM prompts, PSP accumulation) are universal.

The agents aren't replacing 3D artists. They're making 3D accessible to people who have ideas but not the specialized skills to execute them. The quality ceiling is still set by human judgment — but the floor has risen dramatically.

Tested on: Linux Mint 22.3, Blender 4.0+, Godot 4.x, NVIDIA RTX 5060 Ti (eGPU via Thunderbolt 4)

MCP Server: BlenderMCP 1.27.1

Vision Models: Qwen3VL-4B (local, llama.cpp) → later Qwen (larger, unified vision+coding via MCP)

Author: Aleksandr Kossarev, Jõgeva, Estonia

Project: Arche Iscrin

This article is based on 2300+ lines of production notes from rigging 6 animated fish models for a Godot virtual aquarium, using an AI-orchestrated pipeline without manual Blender operation.

External GPU (eGPU) + NVIDIA Drivers on Linux: Solving the Display Manager Initialization Problem

Aleksandr Kossarev — Sun, 03 May 2026 14:54:31 +0000

External GPU (eGPU) + NVIDIA Drivers on Linux: Solving the Display Manager Initialization Problem

TL;DR: If your NVIDIA eGPU works in recovery mode but gives a black screen on normal boot, you're missing one critical Xorg option: AllowExternalGpus. This guide shows how to fix it properly on any X11-based Linux distribution.

Introduction

Installing NVIDIA drivers on a Linux system with an external GPU (eGPU) connected via Thunderbolt can result in a frustrating black screen instead of your login screen. This issue affects LightDM, SDDM, GDM (X11 session), and other display managers across multiple distributions.

This guide documents a complete solution tested on real hardware and explains the root cause that official documentation often omits.

Tested Configuration:

Hardware: GEEKOM GT1 Mega (Intel Core Ultra 9 185H with Intel Arc iGPU) + NVIDIA RTX 5060 Ti in Sonnet eGPU Breakaway Box 750ex
Connection: Thunderbolt 4
OS: Linux Mint 22.3 MATE (applicable to Ubuntu, Fedora, Arch, and any X11-based distribution)
Driver: NVIDIA 595 (proprietary)

Symptoms

Before diving into the solution, confirm you're experiencing this specific issue:

✅ NVIDIA drivers installed successfully
✅ nvidia-smi works and shows your GPU
✅ GPU visible in lspci output
❌ Black screen instead of login screen on normal boot
✅ System works normally in recovery mode or without X11
⚠️ Possible error in dmesg: i915: failed to get ACT after 3000ms
❌ Problem persists across different display managers (LightDM, SDDM, GDM in X11 mode)

Root Cause

NVIDIA drivers intentionally disable external GPUs by default as a safety measure to prevent crashes when the Thunderbolt cable is accidentally disconnected.

Without the AllowExternalGpus flag, the X11 server attempts to initialize the NVIDIA GPU, receives a denial, and crashes:

(WW) NVIDIA(GPU-0): This device is an external GPU, but external GPUs have not
(WW) NVIDIA(GPU-0):     been enabled with AllowExternalGpus. Disabling this device
(EE) NVIDIA(0): Failing initialization of X screen
(EE) no screens found
Fatal server error: no screens found

X11 then attempts to fall back to the Intel iGPU (modesetting driver), but if your monitor is connected only to the eGPU, there are no screens available on the Intel outputs, resulting in a black screen.

Why GNOME/Wayland might work without this fix:

Wayland bypasses X11 and interacts directly with GPUs via KMS (kernel modesetting). NVIDIA drivers don't block KMS access for eGPUs. Display managers using Wayland (like GDM in Wayland mode) will work, while X11-based sessions (LightDM, SDDM, Cinnamon, MATE) will fail.

Additional Issue: Boot Race Condition

Even after adding AllowExternalGpus, you might experience intermittent black screens. This occurs due to timing issues:

Display manager starts → attempts to launch X11
nvidia-drm module hasn't completed initialization (~2–3 seconds)
Thunderbolt DisplayPort tunnel establishes even later

This is addressed through systemd service synchronization (detailed in Step 4 below).

Step 1: Diagnosis

Check Xorg logs from recovery mode or TTY (Ctrl+Alt+F2):

grep -E "(EE|WW|AllowExternal|no screens|nvidia)" /var/log/Xorg.0.log

If you see references to AllowExternalGpus or no screens found, you're in the right place.

Verify GPU is visible to the system:

nvidia-smi
# Should show GPU with temperature, memory usage, etc.

lspci | grep -i nvidia
# Should list your GPU

Confirm monitor connection to eGPU:

ls /sys/class/drm/
# Look for card0-DP-* or card0-HDMI-* entries
cat /sys/class/drm/card0-DP-1/status
# Should return: connected

Step 2: The Critical Fix – AllowExternalGpus

Create or edit the X11 configuration file:

sudo nano /etc/X11/xorg.conf.d/10-nvidia.conf

File contents:

Section "ServerLayout"
    Identifier "layout"
    Screen 0 "nvidia"
    Inactive "intel"
EndSection

Section "Device"
    Identifier "nvidia"
    Driver "nvidia"
    Option "PrimaryGPU" "yes"
    Option "AllowExternalGpus" "True"
EndSection

Section "Screen"
    Identifier "nvidia"
    Device "nvidia"
    Option "AllowEmptyInitialConfiguration"
EndSection

Section "Device"
    Identifier "intel"
    Driver "modesetting"
EndSection

Section "Screen"
    Identifier "intel"
    Device "intel"
EndSection

Critical line: Option "AllowExternalGpus" "True" — nothing works without this.

Option "AllowEmptyInitialConfiguration" — allows X11 to start even if the GPU isn't fully initialized when the display manager launches.

Step 3: Kernel Mode Setting (KMS) Configuration

If not already configured during driver installation:

# Verify modeset is enabled
cat /proc/cmdline | grep nvidia-drm
# Should show: nvidia-drm.modeset=1

If missing, add to GRUB:

sudo nano /etc/default/grub

Locate GRUB_CMDLINE_LINUX_DEFAULT and add parameters:

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash nvidia-drm.modeset=1"

Create modprobe configuration:

sudo nano /etc/modprobe.d/nvidia-kms.conf

options nvidia-drm modeset=1
options nvidia NVreg_PreserveVideoMemoryAllocations=1

Apply changes:

sudo update-grub
sudo update-initramfs -u

Step 4: Boot Race Condition Fix (for stability)

This is optional but eliminates rare black screens on some boots.

GPU Wait Script

sudo nano /usr/local/bin/nvidia-egpu-wait.sh

#!/bin/bash
# Wait for NVIDIA GPU to appear in /sys/class/drm
TIMEOUT=30
COUNT=0
while [ $COUNT -lt $TIMEOUT ]; do
    if ls /sys/class/drm/ 2>/dev/null | grep -q "^card[0-9]$"; then
        # Verify it's NVIDIA, not just Intel
        for card in /sys/class/drm/card[0-9]; do
            vendor=$(cat "$card/device/vendor" 2>/dev/null)
            if [ "$vendor" = "0x10de" ]; then
                sleep 2  # Additional pause for TB3 DP tunnel
                exit 0
            fi
        done
    fi
    sleep 1
    COUNT=$((COUNT + 1))
done
exit 0  # Timeout - continue anyway

sudo chmod +x /usr/local/bin/nvidia-egpu-wait.sh

Hotplug Script (runs after display manager)

sudo nano /usr/local/bin/nvidia-drm-hotplug.sh

#!/bin/bash
sleep 8
udevadm trigger --action=change --subsystem-match=drm
udevadm settle

sudo chmod +x /usr/local/bin/nvidia-drm-hotplug.sh

Systemd Service: Wait (runs BEFORE display manager)

sudo nano /etc/systemd/system/nvidia-egpu-wait.service

[Unit]
Description=Wait for NVIDIA eGPU initialization
After=bolt.service
Before=display-manager.service
DefaultDependencies=no

[Service]
Type=oneshot
ExecStart=/usr/local/bin/nvidia-egpu-wait.sh
RemainAfterExit=yes
TimeoutSec=35

[Install]
WantedBy=display-manager.service

Systemd Service: Hotplug (runs AFTER display manager)

sudo nano /etc/systemd/system/nvidia-drm-hotplug.service

[Unit]
Description=NVIDIA DRM hotplug trigger after display manager
After=display-manager.service bolt.service
Wants=display-manager.service

[Service]
Type=oneshot
ExecStart=/usr/local/bin/nvidia-drm-hotplug.sh
RemainAfterExit=no

[Install]
WantedBy=multi-user.target

Display Manager Drop-in (LightDM example)

sudo mkdir -p /etc/systemd/system/lightdm.service.d/
sudo nano /etc/systemd/system/lightdm.service.d/wait-nvidia-egpu.conf

[Unit]
Wants=nvidia-egpu-wait.service
After=nvidia-egpu-wait.service

For SDDM, use /etc/systemd/system/sddm.service.d/ instead.

Enable Services

sudo systemctl daemon-reload
sudo systemctl enable nvidia-egpu-wait.service
sudo systemctl enable nvidia-drm-hotplug.service

Step 5: PRIME Configuration (GPU priority)

On systems with NVIDIA drivers and nvidia-prime:

sudo prime-select nvidia

Verify:

prime-select query
# Should return: nvidia

Step 6: Reboot and Verification

sudo reboot

Post-boot verification:

# GPU is active and in use
nvidia-smi

# Xorg has no critical errors
grep -E "^(EE|WW)" /var/log/Xorg.0.log

# Services completed successfully
systemctl status nvidia-egpu-wait.service
systemctl status nvidia-drm-hotplug.service

# NVIDIA is managing the display (not Intel fallback)
xrandr --listproviders
# Should show: NVIDIA-0 as primary provider

Troubleshooting: If Black Screen Persists

Boot into recovery mode → drop to root shell → check logs:

# Main X11 log
cat /var/log/Xorg.0.log | grep -E "(EE|WW|AllowExternal|screen)"

# Boot journal
journalctl -b 0 -p err --no-pager | tail -50

# Service status
systemctl status lightdm nvidia-egpu-wait nvidia-drm-hotplug

# Initialization sequence
journalctl -b 0 --no-pager | grep -E "(nvidia|drm|lightdm|sddm|bolt)" | head -40

Common Errors and Solutions

Error in Logs	Meaning	Action
`AllowExternalGpus` not set → Disabling	xorg.conf not applied	Verify path and syntax of `/etc/X11/xorg.conf.d/10-nvidia.conf`
`no screens found`	X11 found no monitors	Confirm monitor connected to eGPU, not Intel outputs
`i915: failed to get ACT after 3000ms`	Intel looking for monitor on its outputs	Normal if monitor not connected to Intel; this is a consequence, not cause
`NVRM: No NVIDIA GPU found` in dmesg at boot	Early boot before TB3 initialization	Normal if only at start; wait-service addresses this
`Failing initialization of X screen`	X11 crashed during GPU init	Return to Step 2 and verify `AllowExternalGpus`

Rollback (if something goes wrong)

From recovery mode or another system (chroot):

# Remove our xorg config - X11 reverts to auto-detection
sudo rm /etc/X11/xorg.conf.d/10-nvidia.conf

# Or temporarily rename for testing
sudo mv /etc/X11/xorg.conf.d/10-nvidia.conf /etc/X11/xorg.conf.d/10-nvidia.conf.bak

Chroot from another system:

sudo mkdir -p /mnt/target
sudo mount /dev/nvme0n1pX /mnt/target  # replace X with your partition
sudo mount --bind /dev /mnt/target/dev
sudo mount --bind /proc /mnt/target/proc
sudo mount --bind /sys /mnt/target/sys
sudo mount --bind /run /mnt/target/run
sudo chroot /mnt/target /bin/bash
# Make changes...
exit
sudo umount /mnt/target/dev /mnt/target/proc /mnt/target/sys /mnt/target/run
sudo umount /mnt/target

Final Configuration Summary

Files that should exist after setup:

/etc/X11/xorg.conf.d/10-nvidia.conf          ← primary fix
/etc/modprobe.d/nvidia-kms.conf               ← KMS modeset
/etc/default/grub                             ← nvidia-drm.modeset=1 in cmdline
/usr/local/bin/nvidia-egpu-wait.sh            ← wait script
/usr/local/bin/nvidia-drm-hotplug.sh          ← hotplug script
/etc/systemd/system/nvidia-egpu-wait.service  ← service (Before=DM)
/etc/systemd/system/nvidia-drm-hotplug.service← service (After=DM)
/etc/systemd/system/lightdm.service.d/wait-nvidia-egpu.conf  ← drop-in

Key kernel parameters (in /proc/cmdline):

nvidia-drm.modeset=1

Technical Explanations

Why the problem isn't the display manager:

LightDM, SDDM, and GDM are just wrappers that launch X11. They all use the same X server (/usr/bin/Xorg). The root cause lies in NVIDIA driver behavior at the X11 level, not in the display manager itself.

Why GNOME/Wayland worked without the fix:

GNOME defaults to Wayland, which interacts with GPUs via KMS (kernel modesetting) directly, bypassing Xorg. NVIDIA drivers don't block KMS access for eGPUs. Therefore, GDM in Wayland mode worked while LightDM/SSDM (X11) didn't.

Why i915 ACT error is not the cause:

The Intel iGPU sees that X11 is attempting to use it as a fallback (after NVIDIA rejection) and begins initializing Intel DisplayPort outputs, but the monitor isn't connected to Intel → timeout. This is a consequence of X11 failing with NVIDIA, not the root cause.

About Thunderbolt and bolt:

If the eGPU isn't authorized in bolt, it won't appear in the system at all. Check: boltctl list. If status isn't authorized, run: sudo boltctl enroll --policy auto <uuid>.

Known Behavior: Boot Delay (30-90 seconds)

On cold boots with eGPU via Thunderbolt, you may experience a delay before the login screen appears. This is normal and relates to sequential initialization:

Thunderbolt authorization (~15 sec)
NVIDIA driver loading (~20 sec)
DisplayPort tunnel establishment (~15 sec)
X11 initialization (~10 sec)

Total: 30-60 seconds on modern hardware.

The systemd services (nvidia-egpu-wait and nvidia-drm-hotplug) minimize this delay but can't eliminate it entirely due to Thunderbolt physics.

Possible optimizations:

Configure bolt with auto-enroll policy
Use nvidia-smi -pm 1 for early GPU "warm-up"
Disable unused systemd services

Conclusion

The root cause of black screen issues when using NVIDIA eGPU on Linux isn't the display manager, PRIME configuration, or GRUB parameters. It's a single missing Xorg option: AllowExternalGpus.

NVIDIA drivers disable external GPUs by default as a safety measure. Without explicit permission via this flag, X11 initialization fails silently, resulting in a black screen.

This configuration has been tested extensively and works reliably across multiple distributions. If you're building a Linux workstation with eGPU, this guide can save you hours of troubleshooting.

What we learned:

✅ External GPUs require explicit enablement in Xorg configuration
✅ Display managers (LightDM, SSDM, GDM in X11) all experience the same issue
✅ Wayland sessions work because they bypass X11 entirely
✅ Boot timing issues can be addressed with systemd service synchronization
✅ The i915 ACT error is a red herring — a consequence, not the cause

Questions? Feel free to ask in the comments. I'll be monitoring this thread and happy to help troubleshoot your specific configuration.

Author: Aleksandr Kossarev

Location: Estonia

Date: May 2, 2026

Hardware: GEEKOM GT1 Mega + NVIDIA RTX 5060 Ti (eGPU via Thunderbolt 4)

Project: Arche Iscrin — AI-assisted creative projects

This article is based on real-world troubleshooting and testing. All commands and configurations have been verified on actual hardware. Feel free to share this guide with anyone struggling with eGPU setup on Linux.

External GPU (eGPU) + NVIDIA Drivers on Linux: Solving the Display Manager Initialization Problem

Aleksandr Kossarev — Sun, 03 May 2026 14:54:30 +0000

External GPU (eGPU) + NVIDIA Drivers on Linux: Solving the Display Manager Initialization Problem

TL;DR: If your NVIDIA eGPU works in recovery mode but gives a black screen on normal boot, you're missing one critical Xorg option: AllowExternalGpus. This guide shows how to fix it properly on any X11-based Linux distribution.

Introduction

This guide documents a complete solution tested on real hardware and explains the root cause that official documentation often omits.

Tested Configuration:

Hardware: GEEKOM GT1 Mega (Intel Core Ultra 9 185H with Intel Arc iGPU) + NVIDIA RTX 5060 Ti in Sonnet eGPU Breakaway Box 750ex
Connection: Thunderbolt 4
OS: Linux Mint 22.3 MATE (applicable to Ubuntu, Fedora, Arch, and any X11-based distribution)
Driver: NVIDIA 595 (proprietary)

Symptoms

Before diving into the solution, confirm you're experiencing this specific issue:

✅ NVIDIA drivers installed successfully
✅ nvidia-smi works and shows your GPU
✅ GPU visible in lspci output
❌ Black screen instead of login screen on normal boot
✅ System works normally in recovery mode or without X11
⚠️ Possible error in dmesg: i915: failed to get ACT after 3000ms
❌ Problem persists across different display managers (LightDM, SDDM, GDM in X11 mode)

Root Cause

NVIDIA drivers intentionally disable external GPUs by default as a safety measure to prevent crashes when the Thunderbolt cable is accidentally disconnected.

Without the AllowExternalGpus flag, the X11 server attempts to initialize the NVIDIA GPU, receives a denial, and crashes:

(WW) NVIDIA(GPU-0): This device is an external GPU, but external GPUs have not
(WW) NVIDIA(GPU-0):     been enabled with AllowExternalGpus. Disabling this device
(EE) NVIDIA(0): Failing initialization of X screen
(EE) no screens found
Fatal server error: no screens found

Additional Issue: Boot Race Condition

Even after adding AllowExternalGpus, you might experience intermittent black screens. This occurs due to timing issues:

Display manager starts → attempts to launch X11
nvidia-drm module hasn't completed initialization (~2–3 seconds)
Thunderbolt DisplayPort tunnel establishes even later

This is addressed through systemd service synchronization (detailed in Step 4 below).

Step 1: Diagnosis

Check Xorg logs from recovery mode or TTY (Ctrl+Alt+F2):

grep -E "(EE|WW|AllowExternal|no screens|nvidia)" /var/log/Xorg.0.log

If you see references to AllowExternalGpus or no screens found, you're in the right place.

Verify GPU is visible to the system:

nvidia-smi
# Should show GPU with temperature, memory usage, etc.

lspci | grep -i nvidia
# Should list your GPU

Confirm monitor connection to eGPU:

ls /sys/class/drm/
# Look for card0-DP-* or card0-HDMI-* entries
cat /sys/class/drm/card0-DP-1/status
# Should return: connected

Step 2: The Critical Fix – AllowExternalGpus

Create or edit the X11 configuration file:

sudo nano /etc/X11/xorg.conf.d/10-nvidia.conf

File contents:

Section "ServerLayout"
    Identifier "layout"
    Screen 0 "nvidia"
    Inactive "intel"
EndSection

Section "Device"
    Identifier "nvidia"
    Driver "nvidia"
    Option "PrimaryGPU" "yes"
    Option "AllowExternalGpus" "True"
EndSection

Section "Screen"
    Identifier "nvidia"
    Device "nvidia"
    Option "AllowEmptyInitialConfiguration"
EndSection

Section "Device"
    Identifier "intel"
    Driver "modesetting"
EndSection

Section "Screen"
    Identifier "intel"
    Device "intel"
EndSection

Critical line: Option "AllowExternalGpus" "True" — nothing works without this.

Option "AllowEmptyInitialConfiguration" — allows X11 to start even if the GPU isn't fully initialized when the display manager launches.

Step 3: Kernel Mode Setting (KMS) Configuration

If not already configured during driver installation:

# Verify modeset is enabled
cat /proc/cmdline | grep nvidia-drm
# Should show: nvidia-drm.modeset=1

If missing, add to GRUB:

sudo nano /etc/default/grub

Locate GRUB_CMDLINE_LINUX_DEFAULT and add parameters:

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash nvidia-drm.modeset=1"

Create modprobe configuration:

sudo nano /etc/modprobe.d/nvidia-kms.conf

options nvidia-drm modeset=1
options nvidia NVreg_PreserveVideoMemoryAllocations=1

Apply changes:

sudo update-grub
sudo update-initramfs -u

Step 4: Boot Race Condition Fix (for stability)

This is optional but eliminates rare black screens on some boots.

GPU Wait Script

sudo nano /usr/local/bin/nvidia-egpu-wait.sh

#!/bin/bash
# Wait for NVIDIA GPU to appear in /sys/class/drm
TIMEOUT=30
COUNT=0
while [ $COUNT -lt $TIMEOUT ]; do
    if ls /sys/class/drm/ 2>/dev/null | grep -q "^card[0-9]$"; then
        # Verify it's NVIDIA, not just Intel
        for card in /sys/class/drm/card[0-9]; do
            vendor=$(cat "$card/device/vendor" 2>/dev/null)
            if [ "$vendor" = "0x10de" ]; then
                sleep 2  # Additional pause for TB3 DP tunnel
                exit 0
            fi
        done
    fi
    sleep 1
    COUNT=$((COUNT + 1))
done
exit 0  # Timeout - continue anyway

sudo chmod +x /usr/local/bin/nvidia-egpu-wait.sh

Hotplug Script (runs after display manager)

sudo nano /usr/local/bin/nvidia-drm-hotplug.sh

#!/bin/bash
sleep 8
udevadm trigger --action=change --subsystem-match=drm
udevadm settle

sudo chmod +x /usr/local/bin/nvidia-drm-hotplug.sh

Systemd Service: Wait (runs BEFORE display manager)

sudo nano /etc/systemd/system/nvidia-egpu-wait.service

[Unit]
Description=Wait for NVIDIA eGPU initialization
After=bolt.service
Before=display-manager.service
DefaultDependencies=no

[Service]
Type=oneshot
ExecStart=/usr/local/bin/nvidia-egpu-wait.sh
RemainAfterExit=yes
TimeoutSec=35

[Install]
WantedBy=display-manager.service

Systemd Service: Hotplug (runs AFTER display manager)

sudo nano /etc/systemd/system/nvidia-drm-hotplug.service

[Unit]
Description=NVIDIA DRM hotplug trigger after display manager
After=display-manager.service bolt.service
Wants=display-manager.service

[Service]
Type=oneshot
ExecStart=/usr/local/bin/nvidia-drm-hotplug.sh
RemainAfterExit=no

[Install]
WantedBy=multi-user.target

Display Manager Drop-in (LightDM example)

sudo mkdir -p /etc/systemd/system/lightdm.service.d/
sudo nano /etc/systemd/system/lightdm.service.d/wait-nvidia-egpu.conf

[Unit]
Wants=nvidia-egpu-wait.service
After=nvidia-egpu-wait.service

For SDDM, use /etc/systemd/system/sddm.service.d/ instead.

Enable Services

sudo systemctl daemon-reload
sudo systemctl enable nvidia-egpu-wait.service
sudo systemctl enable nvidia-drm-hotplug.service

Step 5: PRIME Configuration (GPU priority)

On systems with NVIDIA drivers and nvidia-prime:

sudo prime-select nvidia

Verify:

prime-select query
# Should return: nvidia

Step 6: Reboot and Verification

sudo reboot

Post-boot verification:

# GPU is active and in use
nvidia-smi

# Xorg has no critical errors
grep -E "^(EE|WW)" /var/log/Xorg.0.log

# Services completed successfully
systemctl status nvidia-egpu-wait.service
systemctl status nvidia-drm-hotplug.service

# NVIDIA is managing the display (not Intel fallback)
xrandr --listproviders
# Should show: NVIDIA-0 as primary provider

Troubleshooting: If Black Screen Persists

Boot into recovery mode → drop to root shell → check logs:

# Main X11 log
cat /var/log/Xorg.0.log | grep -E "(EE|WW|AllowExternal|screen)"

# Boot journal
journalctl -b 0 -p err --no-pager | tail -50

# Service status
systemctl status lightdm nvidia-egpu-wait nvidia-drm-hotplug

# Initialization sequence
journalctl -b 0 --no-pager | grep -E "(nvidia|drm|lightdm|sddm|bolt)" | head -40

Common Errors and Solutions

Error in Logs	Meaning	Action
`AllowExternalGpus` not set → Disabling	xorg.conf not applied	Verify path and syntax of `/etc/X11/xorg.conf.d/10-nvidia.conf`
`no screens found`	X11 found no monitors	Confirm monitor connected to eGPU, not Intel outputs
`i915: failed to get ACT after 3000ms`	Intel looking for monitor on its outputs	Normal if monitor not connected to Intel; this is a consequence, not cause
`NVRM: No NVIDIA GPU found` in dmesg at boot	Early boot before TB3 initialization	Normal if only at start; wait-service addresses this
`Failing initialization of X screen`	X11 crashed during GPU init	Return to Step 2 and verify `AllowExternalGpus`

Rollback (if something goes wrong)

From recovery mode or another system (chroot):

# Remove our xorg config - X11 reverts to auto-detection
sudo rm /etc/X11/xorg.conf.d/10-nvidia.conf

# Or temporarily rename for testing
sudo mv /etc/X11/xorg.conf.d/10-nvidia.conf /etc/X11/xorg.conf.d/10-nvidia.conf.bak

Chroot from another system:

sudo mkdir -p /mnt/target
sudo mount /dev/nvme0n1pX /mnt/target  # replace X with your partition
sudo mount --bind /dev /mnt/target/dev
sudo mount --bind /proc /mnt/target/proc
sudo mount --bind /sys /mnt/target/sys
sudo mount --bind /run /mnt/target/run
sudo chroot /mnt/target /bin/bash
# Make changes...
exit
sudo umount /mnt/target/dev /mnt/target/proc /mnt/target/sys /mnt/target/run
sudo umount /mnt/target

Final Configuration Summary

Files that should exist after setup:

/etc/X11/xorg.conf.d/10-nvidia.conf          ← primary fix
/etc/modprobe.d/nvidia-kms.conf               ← KMS modeset
/etc/default/grub                             ← nvidia-drm.modeset=1 in cmdline
/usr/local/bin/nvidia-egpu-wait.sh            ← wait script
/usr/local/bin/nvidia-drm-hotplug.sh          ← hotplug script
/etc/systemd/system/nvidia-egpu-wait.service  ← service (Before=DM)
/etc/systemd/system/nvidia-drm-hotplug.service← service (After=DM)
/etc/systemd/system/lightdm.service.d/wait-nvidia-egpu.conf  ← drop-in

Key kernel parameters (in /proc/cmdline):

nvidia-drm.modeset=1

Technical Explanations

Known Behavior: Boot Delay (30-90 seconds)

On cold boots with eGPU via Thunderbolt, you may experience a delay before the login screen appears. This is normal and relates to sequential initialization:

Thunderbolt authorization (~15 sec)
NVIDIA driver loading (~20 sec)
DisplayPort tunnel establishment (~15 sec)
X11 initialization (~10 sec)

Total: 30-60 seconds on modern hardware.

The systemd services (nvidia-egpu-wait and nvidia-drm-hotplug) minimize this delay but can't eliminate it entirely due to Thunderbolt physics.

Possible optimizations:

Configure bolt with auto-enroll policy
Use nvidia-smi -pm 1 for early GPU "warm-up"
Disable unused systemd services

Conclusion

The root cause of black screen issues when using NVIDIA eGPU on Linux isn't the display manager, PRIME configuration, or GRUB parameters. It's a single missing Xorg option: AllowExternalGpus.

NVIDIA drivers disable external GPUs by default as a safety measure. Without explicit permission via this flag, X11 initialization fails silently, resulting in a black screen.

This configuration has been tested extensively and works reliably across multiple distributions. If you're building a Linux workstation with eGPU, this guide can save you hours of troubleshooting.

What we learned:

✅ External GPUs require explicit enablement in Xorg configuration
✅ Display managers (LightDM, SSDM, GDM in X11) all experience the same issue
✅ Wayland sessions work because they bypass X11 entirely
✅ Boot timing issues can be addressed with systemd service synchronization
✅ The i915 ACT error is a red herring — a consequence, not the cause

Questions? Feel free to ask in the comments. I'll be monitoring this thread and happy to help troubleshoot your specific configuration.

Building AI That Doesn't Lose Its Mind: A Universal Architecture for Stable Memory Systems

Aleksandr Kossarev — Tue, 06 Jan 2026 15:21:13 +0000

From Problem to Concept

This article continues the discussion of memory recursion in AI systems, as described in The Day My AI Started Talking to Itself. If you haven't read it yet, we recommend starting there — it covers the problem itself and its mathematical inevitability.

Once it became clear that memory recursion isn't a specific bug but a fundamental architectural problem, the question arose: how do we actually solve it?

Simple solutions like "just apply decay" or "lower the weight of AI outputs" turned out to be half-measures:

Decay kills important memories along with noise
Lowering weights turns AI into a "mirror" of the user
Deleting old data deprives the system of long-term memory

We needed something more fundamental. Not a "patch," but an architectural principle.

Disclaimer: What This Article Is About

Important to understand: This article is not the ultimate truth. It's a reflection on a possible concept, an attempt to formulate universal principles for preventing recursion in AI systems with multi-layered memory.

The principles proposed here:

✓ Are based on analysis of real recursion cases
✓ Are inspired by how human consciousness works
✓ Have mathematical justification
✓ Are practically implementable

But this is not the only possible solution. Rather, it's a starting point for reflection and experimentation.

Nevertheless, we believe these principles are significantly better than many current approaches and have every right to exist and be implemented.

The Key Idea: Learning from the Human Brain

The question wasn't why this happens (that's already clear), but how to prevent it without losing system utility.

And here an unexpected source of inspiration helped us: human consciousness.

The Human Brain's Solution

Think about how a healthy human mind works:

There's a core identity - your personality, values, fundamental beliefs
- These DON'T change from daily interactions
- They filter and interpret new information
There's verified knowledge - facts you're confident about
- These change slowly, with evidence
- They're resistant to casual contradictions
There's working memory - current context, recent conversations
- These change rapidly
- They fade naturally when no longer relevant
There's critical thinking - new information is evaluated
- Does it contradict what I know?
- Is the source trustworthy?
- Am I thinking about this too much?

Humans don't get stuck in loops because we have layers with different rules.

The Architecture: Three-Layer Memory

Here's the proposed approach that theoretically should prevent recursion while preserving useful memory:

┌─────────────────────────────────────────────────────┐
│  LAYER 1: IDENTITY CORE                             │
│  • System principles and behavior patterns          │
│  • Meta-principles: diversity, relevance, honesty   │
│  • Weight: ALWAYS 1.0                               │
│  • Never changes from interactions                  │
└─────────────────────────────────────────────────────┘
            ↓ interprets everything through this lens
┌─────────────────────────────────────────────────────┐
│  LAYER 2: VALIDATED KNOWLEDGE                       │
│  • User preferences and facts                       │
│  • Confirmed through multiple interactions          │
│  • Weight: 0.8-1.0                                  │
│  • Slow temporal decay (6-12 month half-life)      │
│  • Must be consistent with Layer 1                  │
└─────────────────────────────────────────────────────┘
            ↓ provides context for
┌─────────────────────────────────────────────────────┐
│  LAYER 3: CONTEXTUAL MEMORY                         │
│  • Recent conversations and AI outputs              │
│  • Weight: 0.3-0.6                                  │
│  • Fast temporal decay (1-2 week half-life)        │
│  • Requires validation to move to Layer 2          │
└─────────────────────────────────────────────────────┘

Why This Should Work

Layer 1 (Identity) acts as an attractor—the system always gravitates back toward its core principles, preventing drift.

Layer 2 (Knowledge) stores what matters long-term, but only after validation. AI outputs rarely reach here.

Layer 3 (Context) is disposable. AI outputs start here with low weight and naturally fade unless confirmed by external sources.

Six Universal Principles

Principle 1: Asymmetric Source Trust

Not all sources are equal:

SOURCE_TRUST = {
    "USER_EXPLICIT":     1.0,   # User directly stated
    "USER_IMPLICIT":     0.8,   # Inferred from behavior
    "EXTERNAL_VERIFIED": 0.7,   # Verified external data
    "AI_OUTPUT":         0.3,   # Own generation
    "AI_RECURSIVE":      0.1,   # Nth-order generation
}

Critical: This is built into the architecture, not a config option.

Principle 2: Temporal Dynamics with Exceptions

def temporal_factor(memory, age_days):
    # Facts and preferences don't decay
    if memory.type in ["FACT", "PREFERENCE", "IDENTITY"]:
        return 1.0

    # Recent confirmations reset decay
    if has_recent_confirmation(memory):
        return 1.0

    # Layer 3: fast decay (7 day half-life)
    if memory.layer == 3:
        return exp(-0.1 * age_days)

    # Layer 2: slow decay (180 day half-life)
    if memory.layer == 2:
        return exp(-0.004 * age_days)

Key insight: Decay applies to context, not to knowledge.

Principle 3: Contradiction Detection

def integrate_memory(new_memory, system):
    # Check 1: Contradicts identity?
    if contradicts(new_memory, identity_core):
        if new_memory.source == "AI_OUTPUT":
            return reject(new_memory)
        else:
            # User contradicts identity - note but don't integrate
            new_memory.weight *= 0.5
            new_memory.layer = 3

    # Check 2: Contradicts validated knowledge?
    conflicts = find_conflicts(new_memory, layer2_memories)
    if conflicts:
        if new_memory.source_trust > conflicts.source_trust:
            update_knowledge(conflicts, new_memory)
        else:
            new_memory.disputed = True
            new_memory.layer = 3

    # Check 3: Recursion pattern?
    if is_recurring_theme(new_memory) and new_memory.source == "AI_OUTPUT":
        new_memory.weight *= 0.2
        flag_for_review(new_memory)

Principle 4: Homeostatic Regulation

The system automatically corrects itself:

def regulate_system(window_days=7):
    # Measure theme diversity
    theme_entropy = shannon_entropy(recent_themes(window_days))

    # Detect dominant patterns
    for theme in all_themes:
        frequency = theme.count / total_interactions
        expected = 1.0 / num_themes

        # Theme appears 3x more than expected?
        if frequency > expected * 3:
            if theme.source_majority == "AI_OUTPUT":
                # This is recursion - suppress
                suppress_theme(theme, factor=0.1)
            else:
                # Legitimate interest - but diversify
                boost_alternative_themes(exclude=theme)

    # Measure drift from identity
    identity_distance = measure_drift_from_core()
    if identity_distance > threshold:
        apply_identity_restoration()

This function can run automatically every few days, catching problems before users notice them.

Principle 5: Gradient-Based Retrieval

Memory retrieval considers multiple factors:

def retrieve_memories(query):
    scores = []

    for memory in all_memories:
        # Base relevance
        relevance = cosine_similarity(memory.embedding, query.embedding)

        # Source modifier
        source_mod = memory.source_trust

        # Layer modifier
        layer_mod = {1: 1.0, 2: 0.8, 3: 0.5}[memory.layer]

        # Temporal modifier (with exceptions)
        temporal_mod = temporal_factor(memory, age_days)

        # Anti-spam (retrieval frequency)
        retrieval_count = memory.retrievals(window=7)
        anti_spam = 1.0 / (1.0 + retrieval_count * 0.3)

        # Identity alignment
        identity_alignment = cosine_similarity(
            memory.embedding, 
            identity_core.embedding
        )

        # Combined score
        score = (
            relevance * 
            source_mod * 
            layer_mod * 
            temporal_mod * 
            anti_spam * 
            (0.7 + 0.3 * identity_alignment)
        )

        scores.append((memory, score))

    # Diversity-aware selection
    return select_diverse_top_k(scores, k=10, diversity=0.3)

Principle 6: Monitoring Dashboard

To control system effectiveness, the following set of metrics is proposed:

┌──────────────────────────────────────┐
│ AI Memory Health Dashboard           │
├──────────────────────────────────────┤
│ Shannon Entropy: 2.4 ✓               │
│   Target: > 2.0                      │
│                                      │
│ Identity Distance: 0.12 ✓            │
│   Target: < 0.20                     │
│                                      │
│ Self-Reference Rate: 8% ✓            │
│   Target: < 15%                      │
│                                      │
│ Layer Distribution:                  │
│   Layer 1: 5% ✓                      │
│   Layer 2: 25% ✓                     │
│   Layer 3: 70% ✓                     │
│                                      │
│ Recent Interventions:                │
│   Theme "balance" suppressed         │
│   Reason: 35% frequency, AI source   │
│   Date: 2 days ago                   │
└──────────────────────────────────────┘

Implementation Checklist

If you decide to try implementing this concept in your system, here are the main steps (without strict timeframes — it all depends on your architecture):

Basic Infrastructure:

[ ] Add layer field to memory records (1, 2, or 3)
[ ] Add source field (USER, AI_OUTPUT, EXTERNAL)
[ ] Add source_trust calculation
[ ] Implement basic temporal decay for Layer 3

Core Logic:

[ ] Implement three-layer storage logic
[ ] Add contradiction detection
[ ] Modify retrieval to use gradient scoring
[ ] Create Identity Core definition

Self-Regulation:

[ ] Implement theme frequency tracking
[ ] Add homeostatic regulation (periodic run)
[ ] Create monitoring dashboard
[ ] Set up anomaly alerts

Fine-Tuning:

[ ] Adjust decay rates based on observations
[ ] Fine-tune thresholds
[ ] Test edge cases
[ ] Document system behavior

Expected Results

This architecture is a conceptual development based on analysis of recursion problems in existing systems. Here's what can be expected from its implementation:

Current systems (with recursion):

Theme diversity: 0.28 (low)
Self-reference rate: 35%
User complaints: Weekly
Memory useful lifespan: ~2 weeks

Expected results (with proposed architecture):

Theme diversity: 0.6+ (healthy)
Self-reference rate: < 10%
User complaints: Significant reduction
Memory useful lifespan: Months to years

These projections are based on theoretical analysis and require practical validation.

The Key Insight

The human brain doesn't treat all information equally. It has:

A stable core (personality)
Trusted knowledge (facts)
Disposable context (working memory)

Your AI needs the same structure.

Common Questions

Q: Doesn't this make the AI less "intelligent"?

A: No—it makes it more stable. Intelligence without stability is insanity.

Q: What if the user wants to change the AI's behavior?

A: Layer 2 can be updated with validated user input. Layer 1 remains stable but can be manually adjusted by developers.

Q: How do I define the Identity Core?

A: Start with meta-principles: be helpful, be diverse, be accurate, be relevant. Refine based on your use case.

Q: Does this work with vector databases?

A: Yes! The layer/source/weight fields work with any storage system.

Q: What about very large memory systems (millions of entries)?

A: Layer 3 can be aggressively pruned. Layer 2 grows slowly. Layer 1 is tiny. Theoretically, the architecture should scale well, but this requires validation.

Conclusion

AI memory recursion isn't a bug—it's a mathematical inevitability in systems without thoughtful architecture.

We've proposed an approach based on structuring memory like a healthy mind—with different layers and rules for each. The six principles described above form a framework that may help prevent recursion.

But let's repeat once more: this is a concept that requires validation. Perhaps in practice nuances will emerge that we haven't considered. Perhaps someone will find a more elegant solution.

We're not seeking the status of "the only correct approach." Rather, we want to start a discussion about how to properly design memory for AI systems. If this concept proves useful even just as a starting point—we'll consider the task accomplished.

Going to try implementing it? Found weak spots? Came up with improvements? Share your experience—that's the only way we can move forward.

Let's Discuss

Have you encountered memory recursion in your AI projects? What patterns have you noticed? Share your experiences in the comments!

Tags: #ai #architecture #memory #machinelearning #systemdesign

About this article: The principles are designed to be universal and potentially applicable to any AI system with persistent memory, regardless of the underlying technology stack. The concept requires practical validation and can be adapted to specific requirements.

Tags: #ai #architecture #memory #systemdesign

The Day My AI Started Talking to Itself (And the Math Behind Why It Always Happens)

Aleksandr Kossarev — Tue, 06 Jan 2026 11:21:00 +0000

Have you ever built an AI assistant with memory, felt proud of it, then watched in horror as it slowly went insane?

Not "crashed" insane. Not "threw an exception" insane.

Subtly, gradually, conversationally insane.

Week 1: Everything's Fine ✓

Your AI mentions "finding balance in life" three times. Reasonable, right? It's good advice.

Week 4: Hmm, That's Weird ⚠️

"Balance" comes up 12 times. Still... maybe you've been stressed?

Week 8: Houston, We Have a Problem 🔥

Your AI has mentioned "balance" 35 times. In responses about coffee. About code reviews. About literally everything.

You check the logs. The AI isn't broken. It's working perfectly.

That's when you realize: Your AI is reading its own outputs as "important memories."

It's stuck in an echo chamber. Talking to itself.

The Universal Law Nobody Tells You

Here's what took me days to understand:

Every AI system with memory WILL eventually develop recursion.

Not "might." Not "could." WILL.

This isn't a bug in your code. It's not your framework. It's not your vector database.

It's mathematics.

The Recursion Equation

P(memory_retrieved) = (Importance × Relevance) / Time_decay

If Time_decay = 0 → P grows unbounded → Recursion

In plain English: If old memories keep their importance forever, they'll dominate all future responses. Forever.

Why This Happens to EVERYONE

The Iron Law of Memory Systems:

ANY system where:
  1. Past outputs are stored ✓
  2. Past outputs can be retrieved ✓  
  3. Past outputs influence future outputs ✓

WILL eventually develop recursion

Language doesn't matter (Python, JavaScript, Rust).

AI model doesn't matter (GPT, Claude, LLaMA, custom).

Architecture doesn't matter (SQL, vector DB, graph).

The problem is architectural, not technical.

How I Discovered This (The Hard Way)

I was building Archik — an AI assistant with long-term memory. The kind that remembers your preferences, past conversations, decisions.

The Dream: An AI that gets smarter over time.

The Reality: An AI that became increasingly... weird.

The Symptoms

Same phrases appearing again and again
Technical reports showing up in casual conversation
Old discussions dominating new topics
User complaints: "You keep bringing that up!"

The Diagnosis

I analyzed 5,000+ messages in the database. What I found shocked me:

My AI's own technical reports had importance scores of 0.95 (nearly maximum).

Why? They were long (2000+ characters), detailed, and mentioned important keywords.

The system saw them as "valuable memories."

But they were just debug output.

Every time the AI retrieved context, these reports came back. The AI read them, incorporated their style, and produced more reports in the same style.

Which got saved. With high importance. Which got retrieved again...

Classic recursion loop.

The Weight-Decay-Context Triangle

Every memory system operates in three dimensions:

        IMPORTANCE (Weight)
               ↑
               |
               |
    OLD ←──────┼──────→ NEW (Time)
               |
               |
               ↓
        RETRIEVAL (Context)

Healthy System:

New memories have moderate weight
Old memories decay over time
Context balances past and present

Recursive System:

Old memories keep high weight
No temporal decay
Past dominates present

The math is simple: Static importance + No time decay = Guaranteed recursion.

The Solution: Three-Layer Defense

After debugging this for days, I realized you need multiple layers of protection. No single fix works.

Layer 1: Dynamic Importance (At Write Time)

Problem: Long messages automatically get high importance.

Solution: Penalize length, categorize content.

def calculate_importance(message):
    base = 0.5

    # Penalize excessive length
    if len(message) > 2000:
        base *= 0.3  # Technical report? Lower importance

    # Type matters
    if is_apology(message):
        return 0.1  # Apologies are transient
    if is_user_preference(message):
        return 0.9  # Preferences are critical

    return base

Result: Technical reports no longer dominate memory.

Layer 2: Temporal Decay (Over Time)

Problem: 6-month-old memories have same weight as yesterday's.

Solution: Exponential decay based on age.

def apply_decay():
    for memory in database:
        days_old = today - memory.created_at

        if not memory.is_favorite:
            # 5% decay per day
            memory.importance *= (0.95 ** days_old)

        # Archive if too low
        if memory.importance < 0.1:
            archive(memory)

Result: Old memories fade naturally, new stay relevant.

Layer 3: Automated Detection (Periodic Monitoring)

Problem: Recursion develops slowly. You won't notice until it's bad.

Solution: Automated pattern detection every 3 days.

def detect_recursion():
    recent = get_last_50_messages()
    themes = extract_themes(recent)

    for theme, frequency in themes.items():
        if frequency > 0.35:  # Appears in >35% of messages
            # Find and lower importance of old messages
            old_messages = find_old_messages(theme, days=7)
            for msg in old_messages:
                msg.importance *= 0.3

            log_intervention(theme, frequency)

Result: Catches problems before user complains.

Real Results: Before and After

Before Fix:

Analysis of 5,000 messages:
- "balance" mentioned: 35 times/week
- Technical reports in context: 70%
- User satisfaction: Frustrated
- Diversity score: 0.28 (low)

After Fix:

Same system, 2 weeks later:
- "balance" mentioned: 3 times/week  
- Technical reports in context: 5%
- User satisfaction: Happy
- Diversity score: 0.62 (healthy)

Time to implement: ~3 hours of focused work.

Lines of code changed: ~200.

Impact: System completely stable.

The Dashboard That Saved Me

You can't fix what you can't measure. Here's what I monitor:

┌─────────────────────────────────────┐
│ AI Memory Health Dashboard          │
├─────────────────────────────────────┤
│ Total Messages: 5,247               │
│ Avg Importance: 0.38 ✓              │
│                                     │
│ Distribution:                       │
│   Low (0-0.3):    42% ✓            │
│   Medium (0.3-0.5): 33% ✓          │
│   High (0.5-1.0):  25% ✓           │
│                                     │
│ Retrieval Stats:                    │
│   Never retrieved:  68% ✓           │
│   Retrieved 1-5x:   24% ✓           │
│   Retrieved 5+:     8% ⚠            │
│                                     │
│ Recent Detections:                  │
│   Issues found: 0 ✓                 │
│   Last scan: 2 days ago             │
└─────────────────────────────────────┘

Target metrics:

Average importance: 0.35-0.40
Never retrieved: 60-70%
Theme diversity: >0.4

If these drift, recursion is developing.

Three Key Insights

1. Recursion is Mathematical, Not Technical

You can't "fix" it with a patch. It's an architectural property of systems with:

Memory storage
Retrieval mechanisms
Feedback loops

Solution: Design for decay from day one.

2. Importance Must Be Dynamic

Static importance scores guarantee eventual recursion.

The fix: Importance should depend on:

Content type
Age
Retrieval frequency
User feedback

3. You Need Automated Monitoring

Humans can't detect gradual recursion. It develops over weeks.

The fix: Periodic automated scans with alerts.

Your Checklist: Is Your AI at Risk?

Ask yourself three questions:

1. Do old memories keep their importance forever?

If yes: You WILL develop recursion eventually
Fix: Implement temporal decay

2. Do long messages get high importance automatically?

If yes: Technical outputs will dominate
Fix: Penalize length, categorize content

3. Are you monitoring for repetitive patterns?

If no: Recursion is developing silently right now
Fix: Add automated detection

The Bigger Picture

As we build more AI systems with memory (and we all are), this pattern will become more common.

The good news: It's preventable. Solvable. With relatively simple architecture changes.

The bad news: Most developers won't realize they have recursion until users complain.

Don't be that developer.

Want the Full Technical Deep-Dive?

This article covers the key insights and practical solutions. If you want the complete technical architecture, all the code patterns, edge cases, and scaling strategies:

📄 Full PSP on GitHub Gist

Includes:

5-layer defense architecture
Detailed code examples
Case studies from production
Monitoring and metrics guide
Common pitfalls and solutions

Let's Discuss

Have you encountered memory recursion in your AI systems? What was your "aha!" moment?

Or are you building something with memory right now? I'm happy to discuss architecture approaches in the comments! 👇

About the writing process: I documented this using Claude AI as my technical writing assistant. English isn't my first language, and AI helps me share complex technical concepts with the global dev community. All architecture, code, and insights come from solving this problem in production. I've tried to present these principles clearly and hope they'll be useful to others working in this field.

Tags: #ai #architecture #memory #Recursion

How I Fixed "pthread_create: Invalid argument" in Node.js (ROCm + Bleeding Edge Linux)

Aleksandr Kossarev — Mon, 05 Jan 2026 15:21:00 +0000

Every. Single. Time. 🤦‍♂️

My Setup (aka "The Perfect Storm")

Before we dive in, here's what I was working with:

🖥️ Linux 6.14 (yes, bleeding edge)
🧠 Intel Core Ultra 9 185H (hybrid P/E cores)
🎮 AMD RadeonPro W7900 with ROCm 7.0.2
📦 Node.js 22.21.0
🔧 glibc 2.39

Sounds like a developer's dream setup, right? Well...

The Debugging Journey

Phase 1: All the Standard Stuff (That Didn't Work)

I tried everything you'd find on Stack Overflow:

❌ Clearing npm cache
❌ Rebuilding Node.js from source
❌ Adjusting ulimit settings
❌ Playing with UV_THREADPOOL_SIZE
❌ Different Node.js versions

Nothing. The error kept coming back like a boomerang.

Phase 2: Getting Serious

At this point, I started questioning everything. Is it the hybrid CPU? The kernel version? Thread pool size?

# Testing CPU affinity
$ taskset -c 0-11 node -v
node[10234]: pthread_create: Invalid argument  # Nope

# Testing thread pool
$ UV_THREADPOOL_SIZE=1 node -v
node[10256]: pthread_create: Invalid argument  # Nope

# Testing glibc rseq
$ GLIBC_TUNABLES=glibc.pthread.rseq=0 node -v
node[10312]: pthread_create: Invalid argument  # Still nope

Phase 3: The Breakthrough 💡

Finally, I pulled out the big guns: LD_DEBUG

$ LD_DEBUG=libs node -e "console.log('test')" 2>&1 | grep -i pthread

And there it was:

/opt/rocm-7.0.2/lib/libamdhip64.so.7: error: symbol lookup error:
  undefined symbol: pthread_setaffinity_np (fatal)

EUREKA! 🎉

The culprit wasn't Node.js at all. It was ROCm's LD_PRELOAD polluting the environment!

$ env | grep LD_PRELOAD
LD_PRELOAD=/opt/rocm-7.0.2/lib/libMIOpen.so

The Solution: Wrapper Scripts

Here's the clever part: I needed Node.js to work WITHOUT breaking ROCm for my GPU workloads.

Solution: Environment isolation through wrapper scripts.

Step 1: Create the wrappers

File: ~/.local/bin/node

#!/bin/bash
# Isolate Node.js from ROCm's LD_PRELOAD
unset LD_PRELOAD
exec /usr/bin/node "$@"

File: ~/.local/bin/npm

#!/bin/bash
# Isolate npm from ROCm's LD_PRELOAD
unset LD_PRELOAD
exec /usr/bin/npm "$@"

Step 2: Make them executable

chmod +x ~/.local/bin/node ~/.local/bin/npm

Step 3: Fix your PATH (CRITICAL!)

Edit ~/.bashrc and make sure ~/.local/bin comes FIRST:

# WRONG (wrapper won't be used):
export PATH="$PATH:$HOME/.local/bin"

# RIGHT (wrapper will be used):
export PATH="$HOME/.local/bin:$PATH"

Apply changes:

source ~/.bashrc

Step 4: Verify

$ which node
/home/user/.local/bin/node  # ✅ Our wrapper!

$ node -v
v22.21.0  # ✅ NO ERROR! 🎉

$ npm -v
10.9.4  # ✅ Clean!

Why This Works

The wrapper creates a clean environment for Node.js while keeping ROCm functional for other applications:

✅ Node.js runs without LD_PRELOAD pollution
✅ ROCm still works for GPU applications
✅ Transparent to all programs (terminal, IDE, scripts)
✅ Easy to maintain and rollback

The Technical Deep-Dive

Want to know why this happens? It's a "perfect storm":

ROCm's LD_PRELOAD forces its libraries to load first
These libraries have undefined symbols (pthread_setaffinity_np)
Node.js 22 tries to create threads with these broken symbols in scope
Result: pthread_create() returns EINVAL (errno 22)

The kicker? The program still works because:

Main threads are already created
The error happens in additional worker threads
Node.js libuv handles the error gracefully

But it's still annoying as hell to see on every run. 😅

Lessons Learned

Bleeding edge = cutting yourself - Latest kernel + glibc + Node.js = unexpected interactions
LD_PRELOAD is dangerous - It affects every dynamically linked program
Deep tracing saves the day - LD_DEBUG found the issue in one shot
Constraints breed creativity - "Don't touch ROCm" → wrapper pattern
PATH order matters - First match wins!

Full Documentation

I've documented the entire diagnostic process, alternative solutions considered, and technical details in a comprehensive PSP (Problem-Solution Pattern):

📄 Complete PSP on GitHub Gist

Conclusion

If you're seeing pthread_create: Invalid argument with Node.js and you have AMD GPU with ROCm installed, check for LD_PRELOAD pollution. The wrapper script solution is clean, maintainable, and doesn't break your GPU workflows.

Have you encountered similar issues with environment variable pollution? Let me know in the comments! 👇

Stats:

⏱️ Time to debug: 2.5 hours
🧪 Hypotheses tested: 8+
🎯 Tools used: LD_DEBUG, strace, lscpu, ulimit
💪 Complexity: 5/5
🏅 Satisfaction: ∞

debugging #linux #nodejs #amd #problemsolving