Why I Run Every Code Snippet Through Two Validation Gates Before Publishing
🎁 Free resource: AI Publishing Checklist — 7 steps to ship a technical ebook with Python (free, no email required) · Full pipeline + 10 scripts: germy5.gumroad.com/l/xhxkzz (pay what you want, min $9.99)
There's a hidden contract in every technical ebook:
"The code in this book was tested before you paid for it."
Most authors violate it without meaning to. They write the code from memory, paste it without running it, and ship. You buy it. By chapter 6, the imports from chapter 2 don't resolve.
I built a system to make that impossible.
The full validation system is part of the AI Publishing Pipeline at germy5.gumroad.com/l/xhxkzz — $12.99, includes all validation scripts.
The Problem With "I Tested This Locally"
"Locally" means:
- Your specific Python version (3.11? 3.12? 3.13?)
- Your specific installed packages
- Your specific environment variables
- Your specific file system layout
When a reader runs your code in their environment, those assumptions break. The script fails with ModuleNotFoundError or FileNotFoundError and the reader assumes they did something wrong.
They didn't. You did — by shipping code that was tested only in one environment.
Gate 1: AST Parsing
The first gate catches syntax errors before any execution happens.
import ast
def validate_syntax(code: str, chapter_id: str) -> bool:
"""
Gate 1: Catch syntax errors at parse time.
Fast, safe, no execution side effects.
"""
try:
tree = ast.parse(code)
return True
except SyntaxError as e:
print(f"[{chapter_id}] Syntax error at line {e.lineno}: {e.msg}")
print(f" Offending text: {e.text}")
return False
What this catches:
- Missing closing parentheses
- Incorrect indentation
- Invalid f-string syntax
- Python 2 syntax in a Python 3 script
- Unclosed brackets, strings, or block structures
What this doesn't catch:
- Wrong imports (
import nonexistent_modulepasses AST) - Runtime errors (division by zero, missing files)
- Logic errors (script runs but does the wrong thing)
AST parsing is the fast filter. Cheap to run, catches the obvious failures before you pay for an execution.
Gate 2: Subprocess Isolation
The second gate actually runs the script.
import subprocess
import tempfile
import os
def validate_execution(code: str, chapter_id: str, timeout: int = 30) -> bool:
"""
Gate 2: Execute the script in a clean, isolated environment.
Catches import errors, runtime exceptions, and infinite loops.
"""
with tempfile.TemporaryDirectory() as tmpdir:
script_path = os.path.join(tmpdir, "test_script.py")
with open(script_path, "w", encoding="utf-8") as f:
f.write(code)
try:
result = subprocess.run(
["python3", script_path],
capture_output=True,
timeout=timeout,
cwd=tmpdir, # clean working directory
text=True
)
if result.returncode != 0:
print(f"[{chapter_id}] Non-zero exit code: {result.returncode}")
print(f" stderr: {result.stderr[:500]}")
return False
return True
except subprocess.TimeoutExpired:
print(f"[{chapter_id}] Script exceeded {timeout}s timeout — possible infinite loop")
return False
What this catches:
-
ModuleNotFoundError— import that doesn't exist -
FileNotFoundError— path that doesn't exist in the clean environment -
AttributeError,TypeError,ValueError— runtime exceptions - Infinite loops (via timeout)
- Non-zero exit codes from
sys.exit(1)or unhandled exceptions
Key design decisions:
-
tempfile.TemporaryDirectory()— each script runs in a fresh directory with no leftover files from previous scripts -
cwd=tmpdir— the working directory is the temp dir, so relative paths fail loudly -
timeout=30— catches infinite loops without hanging the pipeline -
capture_output=True— stderr is captured and logged, not lost
The State Machine Integration
These two gates integrate into the chapter state machine:
from enum import Enum
class ChapterState(Enum):
PENDING = "pending"
RUNNING = "running"
DONE = "done"
NEEDS_REVIEW = "needs_review"
def validate_chapter(chapter):
"""Run both gates. Chapter only reaches DONE if both pass."""
chapter.state = ChapterState.RUNNING
save_state(chapter)
code = extract_code_block(chapter.content)
# Gate 1
if not validate_syntax(code, chapter.id):
chapter.state = ChapterState.NEEDS_REVIEW
save_state(chapter)
return False
# Gate 2
if not validate_execution(code, chapter.id):
chapter.state = ChapterState.NEEDS_REVIEW
save_state(chapter)
return False
# Both gates passed
chapter.state = ChapterState.DONE
save_state(chapter)
return True
A chapter can only be in DONE state if both gates returned True. There is no override path. There is no "I'll fix it in the next edition."
Failure Rates in Practice
In my pipeline runs, roughly 2–3 chapters out of 10 fail on first pass.
Common failure modes by frequency:
| Failure | Gate | Frequency | Fix |
|---|---|---|---|
Missing import (import nonexistent) |
Gate 2 | ~40% of failures | Adjust prompt to use stdlib only |
| Script expects input file | Gate 2 | ~25% | Prompt to create sample data inline |
| Infinite loop in demo code | Gate 2 (timeout) | ~15% | Add exit condition to prompt |
| Syntax error in generated code | Gate 1 | ~10% | Re-generate chapter |
| Non-zero exit for illustrative error | Gate 2 | ~10% | Wrap in try/except in prompt |
The pipeline doesn't eliminate failures. It catches them before you ship.
What This Doesn't Catch
Be honest about limitations:
Silent semantic failures. A script that exits with code 0 but prints the wrong output passes both gates. If the chapter is demonstrating bubble sort and the generated code implements insertion sort — both gates pass, but the content is wrong.
Environment-specific modules. If the script imports boto3 or tensorflow and the reader doesn't have those installed, Gate 2 passes on my machine but fails on theirs.
Dependency on external services. A script that makes a real API call will fail if the API is down or the key is invalid.
For these cases, the pipeline flags chapters for manual review and includes instructions in the prompt to keep scripts self-contained with stdlib only.
The Result: A Guarantee You Can Actually Make
After running both gates, you can honestly say:
"Every code snippet in this ebook ran — in a clean subprocess — and returned exit code 0. That's not a claim. It's the precondition for publishing."
That's a guarantee worth making.
The complete validation system (both scripts + state machine integration) is included in the pipeline at germy5.gumroad.com/l/xhxkzz — $12.99, 30-day refund.
Top comments (0)