AI Trading: Lesson Learned #112: Self-Healing Data Integrity System Required

Lesson Learned #112: Self-Healing Data Integrity System Required

Date: January 8, 2026
Severity: HIGH
Category: System Architecture

Incident Summary

CEO identified multiple data integrity issues that have been recurring daily:

• system_state.json showed Jan 7 instead of Jan 8
• Challenge day counter showed 70 instead of 72 (calculated from start date)
• docs/index.md showed "Day 70/90" and "Wednesday, January 7, 2026"
• GitHub Wiki dashboard showed old $117K paper account instead of post-reset $5K
• docs/progress_dashboard.md had wrong day counter

CEO Quote: "Why isn't our system self healing? Why do I have to bring out these issues everyday?"

Root Cause Analysis

1. No automated date updates: current_date in system_state.json required manual updates
2. Day counter not calculated dynamically: Was stored as static value instead of computed from start_date
3. Multiple sources of truth: system_state.json, index.md, wiki, progress_dashboard all needed separate updates
4. No staleness detection with auto-fix: Hook detected staleness but didn't fix it
5. Dashboard not auto-regenerated: When CEO reset paper to $5K, wiki dashboard never updated

Solution Implemented

1. Created scripts/self_heal_data.py

Auto-fixes:

• current_date in system_state.json (always today)
• current_day counter (calculated from start_date: Oct 29, 2025)
• days_remaining counter
• docs/index.md date and day references
• "Last updated" timestamps

2. Added Self-Healing to CI Workflow

In .github/workflows/daily-trading.yml:

• Runs before trading execution every day
• Auto-commits fixes if any changes detected
• Non-blocking (doesn't fail the workflow)

3. Correct Calculation

from datetime import date
start = date(2025, 10, 29)
today = date.today()
current_day = (today - start).days + 1

For Jan 8, 2026: Day 72 (not 70 or 71)

Prevention Measures

1. Calculate, don't store: Day counter should be COMPUTED from start_date, not stored
2. Single source of truth: All displays should read from system_state.json
3. Auto-regenerate dashboards: After any data change, regenerate all views
4. Self-healing CI step: Every workflow should fix data drift before execution

Files Changed

• scripts/self_heal_data.py (NEW)
• .github/workflows/daily-trading.yml
• data/system_state.json
• docs/index.md
• docs/progress_dashboard.md

Verification

Run self-healing manually:

python3 scripts/self_heal_data.py

Expected output when data is current:

SELF-HEALING COMPLETE: All data is current

Tags

data-integrity #self-healing #automation #ci-cd #staleness

---

This lesson was auto-published from our AI Trading repository.

More lessons: rag_knowledge/lessons_learned