DEV Community

Igor Ganapolsky
Igor Ganapolsky

Posted on • Originally published at github.com

AI Trading: Lesson Learned #120: Paper Trading System Broken for 4 Days (Jan 5-9, 2026)

#ai #trading #python #machinelearning

Lesson Learned #120: Paper Trading System Broken for 4 Days (Jan 5-9, 2026)

Date: January 9, 2026
Severity: CRITICAL
Category: Infrastructure/Operations
Status: Under Investigation

Summary

Paper trading system has been completely broken since January 6, 2026. ZERO trades executed for 4 consecutive trading days (Jan 5, 7, 8, 9). Only Jan 6 has any trading activity.

Evidence

Missing Trade Files 

❌ 2026-01-09 (Friday): NO TRADES
❌ 2026-01-08 (Thursday): NO TRADES
❌ 2026-01-07 (Wednesday): NO TRADES
✅ 2026-01-06 (Tuesday): 3 trades ← ONLY successful day
❌ 2026-01-05 (Monday): NO TRADES
Enter fullscreen mode Exit fullscreen mode

Workflow Status 

{
  "github_actions_enabled": true,
  "workflow_status": "NEEDS_VERIFICATION",
  "workflow_status_reason": "Jan 7 2026: No trades_2026-01-07.json file exists - automation may not be creating trade files"
}
Enter fullscreen mode Exit fullscreen mode

Paper Account State

  • Equity: $5,000.00
  • Last Sync: 2026-01-07T17:15:00
  • Win Rate: 0% (n=0)
  • Last Trade: 2026-01-06 ← 3 days ago

Root Cause

PRIMARY SUSPECT: Scheduled GitHub Actions workflow not executing.

Known Contributing Factors

  1. LL-119 (Jan 8): Paper trading API keys were mismatched after $5K account reset

    • Status: FIXED (secrets updated to use _5K suffix)
    • But: Trading still not working after fix

  2. Possible Cause #1: Secrets not configured in GitHub repository

    • The ALPACA_PAPER_TRADING_5K_API_KEY and ALPACA_PAPER_TRADING_5K_API_SECRET may not exist in GitHub Secrets
    • This would cause validate-and-test job to fail
    • All downstream jobs would be skipped

  3. Possible Cause #2: Workflow schedule not triggering

    • GitHub disables scheduled workflows after 60 days of repo inactivity
    • OR scheduled runs are failing silently
    • No way to verify from sandbox environment

  4. Possible Cause #3: Workflow condition blocking execution

    • The workflow requires: if: needs.validate-and-test.outputs.secrets_valid == 'true'
    • If secrets validation fails, ALL trading is blocked

Impact

Operational

  • 4 lost trading days (Jan 5, 7, 8, 9, 2026)
  • Paper account equity unchanged at $5,000 (no activity)
  • Phil Town strategy validation completely stalled
  • 0 trades in 4 days = strategy NOT being tested

Strategic

  • R&D Day 72/90: Lost 5.6% of remaining R&D period
  • Cannot validate $5K paper trading strategy
  • Cannot test compounding approach
  • Dashboard and Dialogflow showing stale data (last trade Jan 6)

Trust

  • System appears "working" (no error alerts) but is actually dead
  • "Zombie mode" - automation enabled but not executing
  • Violates anti-lying mandate (reports "ready" but doesn't trade)

Diagnostic Actions Taken

  1. ✅ Created PAPER_TRADING_DIAGNOSIS_JAN09.md - comprehensive diagnostic document
  2. ✅ Created scripts/diagnose_paper_trading.py - automated diagnostic script
  3. ✅ Verified workflow file has correct secrets (_5K suffix)
  4. ✅ Confirmed no old secret references in workflow
  5. ❌ Cannot access GitHub Actions logs from sandbox

Required Fixes

IMMEDIATE (CEO Action Required)

  1. Verify GitHub Secrets Exist 
   Go to: https://github.com/IgorGanapolsky/trading/settings/secrets/actions

   MUST exist:
   - ALPACA_PAPER_TRADING_5K_API_KEY
   - ALPACA_PAPER_TRADING_5K_API_SECRET
   - ANTHROPIC_API_KEY (for AI features)
   - GOOGLE_API_KEY (for Gemini/ADK)
Enter fullscreen mode Exit fullscreen mode
  1. Check GitHub Actions Status 
   Go to: https://github.com/IgorGanapolsky/trading/actions/workflows/daily-trading.yml

   Check:
   - Are there workflow runs for Jan 7, 8, 9?
   - If YES: What conclusion? (success/failure/cancelled)
   - If NO: Why isn't the schedule triggering?
Enter fullscreen mode Exit fullscreen mode
  1. Manual Trigger Test 
   Go to: https://github.com/IgorGanapolsky/trading/actions/workflows/daily-trading.yml
   Click: "Run workflow"
   Set:
     - Branch: main
     - Trading mode: paper
     - Force trade: true

   Monitor: Does it succeed or fail? At which step?
Enter fullscreen mode Exit fullscreen mode

LONG-TERM (System Fixes)

  1. Add Workflow Health Monitoring

    • Detect when no trades for 2+ days
    • Auto-trigger emergency notification
    • Create GitHub issue automatically

  2. Improve Error Visibility

    • If secrets validation fails, create visible alert
    • Don't fail silently - noise is better than silence
    • Add health check endpoint that reports workflow status

  3. Self-Healing Retry Logic

    • If workflow fails, auto-retry within 30 minutes
    • Maximum 3 retries with exponential backoff
    • Alert after all retries exhausted

  4. Sandbox Secret Management

    • Secrets aren't available in sandbox (expected)
    • But need way to test workflow logic without secrets
    • Add dry-run mode that mocks Alpaca API

Prevention Rules

MANDATORY: After Every Account Reset

When resetting Alpaca accounts (paper or live):

  1. Update ALL secret references across ALL workflows 
   # Check for old references
   grep -r "OLD_SECRET_NAME" .github/workflows/

   # Should return ZERO results
Enter fullscreen mode Exit fullscreen mode

  1. Verify secrets exist in GitHub repository settings

    • Don't just update workflow files
    • Actually ADD the secrets to GitHub
    • Test secret retrieval with simple workflow

  2. Manual trigger test IMMEDIATELY after reset

    • Don't wait for next scheduled run
    • Trigger workflow manually
    • Verify it succeeds end-to-end

  3. Monitor for 48 hours after reset

    • Check trade files are created daily
    • Verify workflow runs in Actions dashboard
    • Confirm Dialogflow webhook shows fresh data

MANDATORY: Daily Monitoring (CTO)

Every morning before market open:

# 1. Check for yesterday's trades
ls -la data/trades_$(date -d "yesterday" +%Y-%m-%d).json

# 2. Run diagnostic
python3 scripts/diagnose_paper_trading.py

# 3. Verify dashboard is fresh
curl https://igorganapolsky.github.io/trading/ | grep "$(date +%Y-%m-%d)"
Enter fullscreen mode Exit fullscreen mode

Key Insight

A working workflow configuration ≠ a working trading system.

Verification levels REQUIRED:

  1. ✅ Workflow file is correct
  2. ✅ Secrets are updated in code
  3. ❌ Secrets exist in GitHub ← Missing verification
  4. ❌ Workflow actually runs ← Missing verification
  5. ❌ Trades are executed ← Missing verification
  6. ❌ Trade files are committed ← Missing verification

Related Lessons

  • LL-119: Paper Trading API Key Mismatch (Jan 8, 2026)
  • LL-095: Daily Trading Workflow Failure (Jan 7, 2026)
  • LL-082: CI Failure Resolution (Jan 5, 2026)
  • LL-078: System Lying - Trust Crisis

Status

  • Diagnostic: ✅ Complete
  • Root Cause: ⚠️ Under investigation (need GitHub Actions access)
  • Fix: ⏳ Pending CEO verification of secrets
  • Monitoring: ❌ Not implemented

Next Steps

  1. CEO checks GitHub Secrets configuration
  2. CEO checks GitHub Actions workflow runs
  3. CEO triggers manual workflow test
  4. CTO implements automated monitoring
  5. CTO adds self-healing retry logic

Tags

paper-trading, workflow-failure, automation, critical, secrets, zombie-mode

This lesson was auto-published from our AI Trading repository.

More lessons: rag_knowledge/lessons_learned

Top comments (0)