Building Your Second Me: A Practical Framework for Encoding Yourself into an AI Agent
What Karpathy started with a personal wiki, this article turns into a buildable system.
Andrej Karpathy once wrote about the idea of a "second self" — an AI model trained on your writing, your decisions, your reasoning patterns, your voice. A system that, when asked a question you would be asked, gives an answer you would give. Not a chatbot. A mirror.
He started building it with a personal wiki: a continuously updated document of his thinking, preferences, opinions, and mental models. It was a fascinating experiment in self-modeling. But a wiki is only half the problem. Text is explicit. What about the stuff you do but never write down?
This article picks up where Karpathy left off. It combines behavioral recording, think-aloud extraction, and chain-of-thought distillation into a practical, week-by-week framework for building your own Second Me — an AI agent that thinks like you, defaults like you, and can represent you when you are not in the room.
Why a Wiki Is Not Enough
A personal wiki captures your declared knowledge: what you believe, how you approach problems, what your values are. It is excellent for identity and values alignment.
But it misses your tacit knowledge: the things you know how to do but could not easily explain. The judgment call you make in two seconds. The pattern you recognize but cannot describe. The instinct that guides your fingers before your brain catches up.
Karpathy himself hinted at this gap when he noted that his wiki was "only the beginning." The real challenge — and the real prize — is encoding the tacit layer. That is what this framework is designed to do.
The Architecture of a Second Me
Before diving into methods, let us define what we are building. A functional Second Me has four layers:
| Layer | What It Encodes | Source |
|---|---|---|
| Identity | Values, beliefs, communication style | Personal wiki, essays, writing samples |
| Reasoning | How you think through problems | Think-aloud recordings, decision logs |
| Skills | Domain-specific expertise and workflows | Behavioral recordings, process documentation |
| Intuition | Pattern recognition and gut calls | Chain-of-thought distillation, RLHF feedback |
The wiki feeds Layer 1. This article focuses on Layers 2, 3, and 4.
Phase 1: Behavioral Recording and Think-Aloud Extraction
Timeline: Week 1-2 | Effort: Medium | Output: Raw reasoning corpus
The Principle
The Think-Aloud Protocol comes from cognitive psychology. The idea is simple: ask someone to narrate their thinking out loud while performing a task. No editing. No structuring. Just talk.
What comes out is extraordinary. The expert will say things they did not know they knew: "This approach feels wrong but I cannot explain why yet — let me think... Oh, because if X breaks, there is no fallback." That "feel" is tacit knowledge escaping in real time.
Week 1: Recording Setup
Pick your recording triggers. Do not try to record everything — that is unsustainable. Instead, identify the high-value moments:
- Decisions with stakes: when you make a consequential choice (tool selection, architecture decision, hiring call, writing a difficult email)
- Problem-solving sessions: debugging, design sessions, research synthesis
- Expertise demonstrations: tasks where you have deep domain knowledge
- Edge case handling: moments where you deviate from standard practice because of context
Set up minimal friction. Use a headset microphone and a one-button recording tool (Otter.ai, macOS Voice Memos, or Tactiq for Google Meet). The moment you make recording inconvenient, you stop doing it.
The recording prompt. Before each session, say out loud:
"I am going to work on [task]. Here is what I know so far. Here is what I am unsure about. Here is what I am going to try first and why."
This framing forces you to externalize your starting model before you begin — which makes the later delta (what you learned) explicit.
Week 2: Transcription and Annotation
Transcribe everything. Use Whisper or Otter to batch-transcribe your recordings. Do not try to clean the transcript as you go — speed matters more than perfection at this stage.
Mark the gold. Go through transcripts and flag these patterns:
- Hypothesis statements: "I think this is an X problem because..."
- Rule-of-thumb moments: "Generally I would do it this way, but in this case..."
- Intuition calls: "Something about this feels off"
- Pattern recognition: "This looks like the Y pattern I saw before"
- Uncertainty signals: "I am not sure about this part, let me think..."
- Revision moments: "Wait, actually, I was wrong — the real issue is..."
This annotation is tedious but irreplaceable. It is the difference between raw data and training signal.
What You Get at the End of Week 2
- 10-20 hours of transcribed reasoning
- 200-500 annotated reasoning snippets organized by domain
- A clear picture of your decision-making patterns across contexts
Phase 2: Building the Skill Library
Timeline: Week 3-4 | Effort: High | Output: Structured skill manifests
The Skill Manifest Format
Each skill in your Second Me should have a structured manifest — not a procedure, but a decision-making model. The difference matters: a procedure tells you what to do; a decision-making model tells you how you decide what to do.
## Skill: [Name]
### What it looks like when done well
[Description of the outcome you produce]
### Trigger conditions
[When does this skill activate? What signals?]
### My default approach (unconscious first step)
[What do I instinctively reach for first?]
### My decision tree
[How do I evaluate options? What questions do I ask?]
### Edge cases I handle differently
[The exceptions I have learned, and why]
### Signals I use to know I am done
[What "done" looks like to me]
### Common mistakes I have seen (and avoid)
[The wrong turns others make that I now avoid]
### Related skills
[What this connects to]
How to Build Each Manifest
For each high-frequency skill you identified in Phase 1:
Step 1 — Pull relevant recordings. Find all the think-aloud segments where this skill appeared.
Step 2 — Identify the unconscious first step. Every expert has a reflex. Find yours. What is the first thing you check, say, or do when this type of problem appears? This is often the hardest part to articulate and the most valuable to capture.
Step 3 — Map the decision tree. Work backward from your conclusions. What options did you consider? Why did you reject the ones you rejected? What information did you prioritize?
Step 4 — Surface the edge cases. Your intuition is most differentiated from a novice in edge cases. When do you deviate from the obvious answer? What have you learned from failure that is not in any book?
Example: Skill Manifest for "Code Review"
## Skill: Code Review
### What it looks like when done well
PRs that ship are correct, readable, and leave the codebase in better shape than I found it. I approve fast when it is good and block firmly when it is not.
### Trigger conditions
Any PR that touches shared code, introduces new dependencies, or affects data integrity. I scan everything but deep-review selectively.
### My default approach
1. Read the diff with fresh eyes (never review immediately after reading the ticket)
2. Ask: what is the simplest version of this? Is this it?
3. Look for what is NOT in the diff (tests, error handling, rollback)
4. Check for premature optimization
5. Comment with a question before a judgment ("what happens if X?" is better than "this is wrong")
### Edge cases I handle differently
- Senior engineer PR: review for design, not line-by-line correctness. Trust but verify.
- Urgent hotfix: risk-aware review. What is the blast radius? Is there a faster safe path?
- First PR from a new team member: generous with coaching, firm on standards.
- Architecture change: slow way down. Understand the why before the what.
### Signals I use to know I am done
- I can explain the change to someone who has not read the code
- I have left comments that help the author, not just criticize
- I have flagged at least one thing that genuinely improves the code
### Common mistakes I avoid
- Bikeshedding (arguing about style when substance is wrong)
- Reviewing the implementation instead of the problem-solution fit
- Approving because I am tired of the thread
### Related skills
Debugging, technical writing, mentoring
Build 5-10 such manifests in Month 1. You do not need to capture everything — you need the things you do most often and care about most.
Phase 3: Chain-of-Thought Distillation
Timeline: Month 2 | Effort: Medium | Output: Reasoning chain dataset
The goal of this phase is to take your implicit reasoning (captured in think-aloud recordings) and turn it into explicit reasoning chains that an LLM can learn from.
A reasoning chain is not just your conclusion. It is your path from input to conclusion, including the information you prioritized, the options you considered, the leap you made when the obvious answer was wrong, and the uncertainty you held rather than resolving prematurely.
Building a Reasoning Chain Dataset
Pick 50-100 decisions from your recordings that have clear inputs, observable outputs, and recorded reasoning. For each case, structure the chain:
## Reasoning Chain: [Case Title]
### Context
[The problem as it presented itself]
### Initial hypothesis
[What I first thought was going on]
### Information I gathered
[The signals I looked for and what they told me]
### Options I considered
[What alternatives I weighed]
### The decisive factor
[The thing that tipped the balance]
### The leap
[The non-obvious connection or intuition I acted on]
### What I held uncertain
[The things I chose not to resolve yet]
### Final decision
[The conclusion and why]
### Outcome (post-hoc)
[What actually happened — especially useful when I was wrong]
The outcome field is critical. When you were wrong, the reasoning chain is even more valuable. The mistake reveals the boundary of your model — exactly where your intuition stops being reliable. Include both successes and failures. The failures are the most honest data.
Three Ways to Inject Into Your Second Me
Once you have 50+ structured reasoning chains, you have three deployment options:
1. Few-shot prompting (easiest, start here)
When [type of problem], here is how I reason through it:
[Insert 3-5 relevant reasoning chains as examples]
Now, here is a new problem: [current situation]
What would I conclude, and why?
2. Fine-tuning (most powerful)
Fine-tune a 7B-8B model (Llama 3.1, Qwen 2.5) on your reasoning chains. This requires more technical setup but produces a genuinely autonomous Second Me that can reason in your voice without prompt scaffolding.
3. Retrieval-augmented reasoning (most scalable)
Store chains in a vector database (Chroma, Weaviate, or Qdrant). When your Second Me encounters a new problem, retrieve the most similar past reasoning chains and use them as context. This is the approach that scales best over time and lets you add new chains without retraining.
Phase 4: Closing the Loop with Continuous Feedback
Timeline: Month 3+ | Effort: Ongoing | Output: Self-improving agent
A Second Me that does not learn from interaction is a snapshot. Here is how to keep it alive.
The Feedback Journal
After any significant interaction where your Second Me was used (or where you made a decision you would have asked it about):
- Did it give an answer you would have given? If yes, mark it. If no, write why not.
- Was its reasoning close to yours? Even if the conclusion matched, was the path similar?
- What did it miss? This is the most valuable feedback — what context or intuition did it lack?
- What would you add to its knowledge base?
Keep this as a daily practice. Five minutes per significant interaction. Over time, this becomes your RLHF dataset — the raw material for fine-tuning future iterations.
Quarterly Skill Refresh
Every quarter, review your skill manifests against your actual behavior. Your intuition evolves. New patterns emerge. Old ones break. Your Second Me should track that evolution. A Second Me that stops learning is not a second you — it is a fossil.
The Karpathy Alignment: What His Wiki Got Right
Karpathy is personal wiki approach had one thing exactly right: identity and voice are the foundation. Before you can encode how you think, your Second Me needs to know who it is.
The wiki gives you:
- Your values (what you care about)
- Your voice (how you communicate)
- Your perspective (how you see the world)
The behavioral recording + reasoning chain distillation gives you:
- Your process (how you actually work)
- Your judgment (how you actually decide)
- Your intuition (the pattern recognition you cannot articulate)
Together, they are a complete picture. The wiki is the soul. The reasoning chains are the operating manual.
A 12-Week Implementation Map
| Week | Focus | Output |
|---|---|---|
| Week 1 | Start recording (3 high-value sessions) | 3-5 hours raw audio |
| Week 2 | Transcribe and annotate first batch | 50+ annotated snippets |
| Week 3-4 | Build 5-10 skill manifests | Structured skill library |
| Month 2 | Build 50+ reasoning chains | Reasoning chain dataset |
| Month 2-3 | Inject into prompts / begin fine-tuning | Functional Second Me v0.1 |
| Month 3+ | Start feedback journal | Continuous improvement loop |
| Quarterly | Skill manifest refresh | Living knowledge base |
What You Are Actually Building
This is not a chatbot with your personality. It is not a digital clone or a productivity hack.
It is a thinking partner that has been trained on your thinking — one that can hold your context, mirror your reasoning, and be a credible proxy when you are not in the room.
The closer it gets to actually thinking like you, the more useful it becomes. And the more useful it becomes, the more you refine it. It is a flywheel.
Start with one recording. One skill manifest. One reasoning chain.
The Second Me does not get built in a weekend. It gets built in the margins of your actual work, week by week, as you go.
Tags: ai productivity selfimprovement llm personaldevelopment
Top comments (0)