DEV Community

Abhishek Banerjee
Abhishek Banerjee

Posted on Originally published at Medium on

Building & Operating an AI Second Brain- Part 2

#machinelearning #ai #rags #secondbrain

Welcome to the 24-Hour Intensive Masterclass on AI Knowledge Architecture.

Traditional approaches to artificial intelligence treat AI as an ephemeral chat box: you ask a question, receive an answer, close the session, and lose all context. This masterclass establishes a paradigm shift moving from ephemeral query-time Retrieval-Augmented Generation (RAG) to an incrementally compiled, persistent, self-maintaining Knowledge Base (LLM Wiki).

Part 1 -https://medium.com/@abhishekninja2018/building-operating-an-ai-second-brain-part-1-542ffad72ec1?sharedUserId=abhishekninja2018

Traditional Ephemeral AI Chat The AI Second Brain Paradigm
┌───────────────────────────────┐ ┌───────────────────────────────────┐
│ User Query ──> LLM ──> Output │ │ Raw Knowledge (PDFs, Notes, Web) │
│ (Session Ends = Reset) │ └─────────────────┬─────────────────┘
└───────────────────────────────┘ │ (Agent Ingests)
                                                             ▼
                                           ┌───────────────────────────────────┐
                                           │ Compiled Markdown Wiki & Graph │
                                           │ (Compounding Knowledge Base) │
                                           └─────────────────┬─────────────────┘
                                                             │ (Query / Synthesis)
                                                             ▼
                                           ┌───────────────────────────────────┐
                                           │ Persistent, Context-Aware Outputs│
                                           └───────────────────────────────────┘
Enter fullscreen mode Exit fullscreen mode

DAY 2: STRUCTURING, AGENT HARNESSES, AUTOMATION & ADVANCED MASTERY

Module 2.1: Schema Engineering & The Two-Layer System Architecture

1. System Prompt Contracting (CLAUDE.md / AGENTS.md)

The core behavior, rules, and schema of your Second Brain are governed by a single root contract file: CLAUDE.md. Instead of typing instructions every session, Claude reads CLAUDE.md automatically upon bootup.

Interactive Initialization Prompt

Run this single-question interview prompt to generate your personal CLAUDE.md:

You are setting up my second brain. Interview me ONE question at a time to build my profile. Ask about: 1) Who I am and what I do, 2) My goals for this year, 3) How I want you to communicate with me, 4) My primary domains of interest, and 5) My active projects. Wait for my answer before asking the next question. When finished, write a comprehensive CLAUDE.md file at the vault root structured with markdown headers.
Enter fullscreen mode Exit fullscreen mode

Canonical CLAUDE.md Template

Below is the standard production template for CLAUDE.md:

# VAULT SYSTEM CONTRACT & RULES

## Core Directives
1. NOTHING IS INGESTED UNTIL IT IS LINKED. Every new page MUST link to at least 2 existing concept pages.
2. RAW IS IMMUTABLE. Never edit files in raw/ or archive/.
3. CONTRADICTIONS ARE RECORDED, NEVER OVERWRITTEN. Document opposing viewpoints with dates and source citations.
4. EVERY CLAIM HAS A CITATION. Cite raw sources using `[[source-title]]`.

## Directory Structure
- `raw/`: Incoming raw source documents.
- `wiki/sources/`: Summaries of ingested raw files.
- `wiki/concepts/`: Atomic core ideas (one concept per file).
- `wiki/entities/`: People, organizations, products, and software tools.
- `wiki/synthesis/`: High-level thematic overviews.
- `projects/`: Active execution workspaces.
- `archive/`: Processed raw files.

## Page Templates & Frontmatter
All wiki pages MUST include valid YAML frontmatter:
---
type: concept # Options: source, concept, entity, synthesis
title: "Page Title"
created: YYYY-MM-DD
tags: [domain/subdomain]
---
Enter fullscreen mode Exit fullscreen mode

2. The Two-Layer System Architecture

To maintain order, we enforce a strict separation between Knowledge and Action :

┌─────────────────────────────────────────────────────────────────────────┐
│ TWO-LAYER SYSTEM ARCHITECTURE │
├────────────────────────────────────┬────────────────────────────────────┤
│ THE WIKI LAYER │ THE PROJECT LAYER │
│ (What You Know) │ (What You Do) │
├────────────────────────────────────┼────────────────────────────────────┤
│ • Densely linked concept graph │ • Isolated subfolders per project │
│ • Long-term memory & research │ • Structured 4-stage pipeline: │
│ • Maintained automatically by AI │ Inputs -> Process -> Outputs -> │
│ • Non-linear, compounding network │ Feedback │
│ • Scope: Broad & permanent │ • Scope: Scoped & goal-oriented │
└────────────────────────────────────┴────────────────────────────────────┘
Enter fullscreen mode Exit fullscreen mode

When executing a specific task (e.g., launching a podcast episode, writing a report), scope down to the specific Project folder. This keeps the LLM’s context window pristine, focusing 100% of its working memory on the project deliverables while drawing necessary facts from the Wiki.

Module 2.2: Graph Mechanics, Typed Links & Open Knowledge Format (OKF v0.1/v0.2)

1. Graph Physics & Topology

A Second Brain is an explicit graph network (G = (V, E)), where (V) represents Markdown pages (nodes) and (E) represents Markdown wikilinks ([[link]]) (edges).

  • Hubs: Highly connected central pages representing major domains.
  • Bridges: Nodes connecting two disparate clusters.
  • Orphans: Isolated pages with zero inbound or outbound links. An unlinked page is effectively invisible to retrieval. 
[Concept: Machine Learning] ──(relates_to)──> [Concept: Neural Networks]
                   │ │
             (utilized_by) (defined_in)
                   ▼ ▼
       [Entity: Claude Code] [Source: Karpathy Talk]
Enter fullscreen mode Exit fullscreen mode

2. Typed Links

Instead of simple unstructured links, we introduce explicit relationship predicate semantics into link syntax:

  • Format: [[Target Page|predicate:relationship]]
  • Examples: [[DeepSeek R1|supports:System 2 Thinking]], [[Vector RAG|contradicts:Compounding Wiki]].

3. Open Knowledge Format (OKF v0.1/v0.2)

Published by Google Cloud, OKF formalizes the LLM-Wiki pattern into an open, vendor-neutral standard:

  • Pure Markdown + Frontmatter: Portable across any operating system or editor.
  • Required type Field: Every document specifies its concept type (source, concept, entity, table, runbook).
  • Interoperable Bundles: Allows wikis compiled by one AI agent (e.g., Claude) to be seamlessly read by another agent (e.g., Gemini or Codex) without translation layers.

Module 2.3: Subagents, Reusable Agent Skills & Slash Command Orchestration

As a vault grows past hundreds of files, relying on a single generalist prompt causes performance degradation. We deploy specialized Subagents and Skills.

1. Specialized Subagents

Subagents are dedicated agent configurations with restricted, role-specific prompts and tools:

  1. ingestor: Converts raw/ inputs into linked wiki pages.
  2. linker: Scans existing concept pages to discover and add missing wikilinks.
  3. graph-analyst: Measures graph density, identifies orphans, hubs, and structural clusters.
  4. reviewer: Generates weekly/monthly synthesis digests.
  5. curator: Identifies stale or duplicate pages and proposes archival.
  6. researcher: Executes multi-step queries across the vault, citing source paths. 
┌──────────────────────────────┐
                        │ CLAUDE CODE HARNESS │
                        └──────────────┬───────────────┘
                                       │
         ┌──────────────────┬──────────┴───────────┬──────────────────┐
         ▼ ▼ ▼ ▼
┌─────────────────┐ ┌───────────────┐ ┌─────────────────┐ ┌──────────────────┐
│ subagent: │ │ subagent: │ │ subagent: │ │ subagent: │
│ ingestor │ │ linker │ │ graph-analyst │ │ reviewer │
│ (Reads raw/) │ │ (Adds links) │ │ (Audits topology│ │ (Synthesizes) │
└─────────────────┘ └───────────────┘ └─────────────────┘ └──────────────────┘
Enter fullscreen mode Exit fullscreen mode

2. Reusable Skills & Slash Commands

A Skill is a saved, repeatable workflow definition stored inside .claude/skills//SKILL.md. A Slash Command is a lightweight trigger exposing that skill.

Core Command Reference Table:

  • /ingest – Processes new raw files into the wiki.
  • /link – Scans the vault and adds missing wikilinks.
  • /lint – Audits frontmatter, broken links, and structural errors.
  • /ask – Queries the vault, answering strictly from internal notes.
  • /health – Outputs a health metric dashboard.
  • /changed-my-mind – Traces historical evolution or shifts in thinking.

Module 2.4: Automated Maintenance, Linting & Self-Healing Health Loops

A Second Brain stays alive because the maintenance cost is transferred to automated AI background jobs.

1. The Audit & Linting Protocol (/lint)

Periodic health checks preserve data integrity:

  • Broken Link Detection: Identifies wikilinks pointing to non-existent target files.
  • Orphan Identification: Flags concept pages with zero inbound links.
  • Contradiction Resolution: Surfaces pages where new sources conflict with older statements, ensuring both are recorded with dates rather than silently overwritten.
  • Schema Validation: Verifies that all YAML headers contain required fields (type, created, tags).

2. Deterministic Vault Health Scripts

We execute local Python scripts to measure vault health without consuming API tokens:

# scripts/vault_stats.py
import os, re
from pathlib import Path

wiki_dir = Path("./wiki")
notes = list(wiki_dir.rglob("*.md"))
links, orphans = 0, 0

for note in notes:
    content = note.read_text(encoding="utf-8")
    found_links = re.findall(r"\\[\[(.*?)\\]\]", content)
    links += len(found_links)
    if len(found_links) == 0:
        orphans += 1

print(f"Total Pages: {len(notes)}")
print(f"Total Wikilinks: {links}")
print(f"Orphan Pages: {orphans} ({orphans/len(notes)*100:.1f}%)")
print(f"Link Density: {links/len(notes):.2f} links/page")
Enter fullscreen mode Exit fullscreen mode

3. Setting Up Unattended Cron Automation

Configure Claude Desktop or a system cron job to run maintenance overnight:

Task Name: Daily Vault Maintenance
Schedule: Every day at 03:00 AM
Prompt:
1. Read CLAUDE.md.
2. Ingest any files currently sitting in raw/ into wiki/.
3. Run /lint to repair broken links and tag orphan pages.
4. Append an audit line to wiki/log.md.
5. Commit changes to Git with message "Automated overnight maintenance".
Enter fullscreen mode Exit fullscreen mode

Module 2.5: Retrieval Mastery, Synthesis, Query Patterns & Enterprise Use Cases

1. Progressive Disclosure Query Patterns

When querying a Second Brain of hundreds of pages, do NOT dump the whole vault into context. Follow the Progressive Disclosure Retrieval Pattern :

Step 1: Read index.md (Catalog Map) ──> Step 2: Identify Candidate Pages
                                                         │
  Step 4: Synthesize Answer <── Step 3: Deep Read Targeted Concept Files
Enter fullscreen mode Exit fullscreen mode
  1. Catalog Read: The agent reads wiki/index.md first to scan summaries and node titles.
  2. Targeted Drill-Down: The agent opens only the specific 3 to 5 relevant Markdown files.
  3. Synthesis & Citation: The agent drafts a response, citing specific concept pages and raw source paths.
  4. Filing Answers Back: High-value answers, comparisons, or new analyses are written back into wiki/synthesis/ as new concept pages ensuring your explorations compound for future queries.

2. Four Transformative Enterprise Use Cases

  • Use Case 1: Stakeholder Memory Vault: Ingesting Slack threads, emails, and meeting notes to build detailed profiles per stakeholder their preferences, objections, and past approvals.
  • Use Case 2: Frictionless Side-Project Context: Ingesting decisions and TODOs after every session so you can resume work instantly without spending an hour remembering where you left off.
  • Use Case 3: Zero-Loss Team Onboarding: Preserving institutional memory when senior staff depart by capturing architecture decisions and rationale in accessible Markdown wikis.
  • Use Case 4: Solution Memory Repository: Documenting complex bug fixes once so you can query past solutions immediately when similar errors resurface.

Module 2.6: Security, Local Data Sovereignty & Capstone Project Execution

1. Data Sovereignty & Security Rules

  • Keys, Not Prompts: Prompt instructions like “do not reveal secrets” are suggestions, not security controls. Enforce security via file system permissions, API key scoping, and read-only flags.
  • Local-First Air-Gapped Stack: For total privacy and intellectual property protection, deploy a 100% offline local stack:
  • Editor: Obsidian (local Markdown files on disk).
  • Inference Engine: Ollama / Local Models (running open weights like Llama 3 or DeepSeek locally).
  • Agent Framework: Nous Research Hermes Agent or Claude Code running in isolated Docker containers. 
┌─────────────────────────────────────────────────────────────────────────┐
│ AIR-GAPPED LOCAL SECOND BRAIN │
├──────────────────┬───────────────────────────┬──────────────────────────┤
│ LOCAL STORAGE │ LOCAL INFERENCE │ CONTAINERIZED HARNESS │
│ Obsidian Vault │ Ollama / Llama 3 / │ Hermes Agent in │
│ (Markdown files) │ DeepSeek Local Weights │ Docker Sandbox │
└──────────────────┴───────────────────────────┴──────────────────────────┘
Enter fullscreen mode Exit fullscreen mode

Capstone Execution Checklist

Participants will now execute their end-to-end Capstone Project:

  • [x] Initialize Directory Structure: Scaffold raw/, wiki/, projects/, archive/, prompts/, and scripts/.
  • [x] Configure Obsidian & MCP Bridge: Link Obsidian to Claude Desktop via the Local REST API plugin and Model Context Protocol.
  • [x] Execute System Interview: Generate a customized root CLAUDE.md contract via interactive prompt.
  • [x] Ingest First Sources: Ingest 3 diverse sources (a web article via Web Clipper, a PDF paper, and a voice transcript) into raw/.
  • [x] Run Agent Ingestion & Linking: Execute /ingest and /link to generate atomic concept pages, sources, entities, and explicit [[wikilinks]].
  • [x] Verify Graph Topology: Open Obsidian’s visual Graph View to inspect nodes, clusters, and link density.
  • [x] Perform Vault Querying & Synthesis: Run /ask to execute a complex cross-document query, verify citations, and file the output into wiki/synthesis/.
  • [x] Automate Daily Maintenance: Schedule an unattended nightly cron loop to auto-ingest raw files, run /lint, and log activity.

Conclusion & Next Steps

You now possess a complete, autonomous, compounding AI Second Brain. As you feed it daily notes, articles, and project decisions, your vault will grow denser and more intelligent over time ensuring that your past learning compounds for the rest of your career.

Need High-Impact Technical Content for Your Engineering Team?

I partner with developer-tooling startups, SaaS platforms, and engineering teams to translate complex infrastructure, agentic systems, and backend architecture into publication-grade technical writing.

Whether you need deep-dive architecture essays, hands-on developer tutorials, or technical counter-narratives:

Top comments (0)