DEV Community

Keo Fung | FormLM
Keo Fung | FormLM

Posted on

[2/10]Designing a YAML-Based Task Planning System for AI-Driven Application Generation

#agents #ai #architecture #systemdesign

Series: Building a Modular Assessment Engine (2/10)

The plan agent's job sounds simple: read a user's request, decide which modules to run, output a task list. In practice, this was the hardest component to get right. One wrong decision in planning cascades through every downstream module.

The Strategy Decision Matrix

The core of the plan agent is a decision matrix — a table that maps user intent to a module sequence. I tried free-form AI planning first. It was a disaster. The AI would invent modules that didn't exist, skip required ones, or order them wrong.

The matrix constrains the AI to six plan types, each with a fixed module sequence:

planType Module Sequence When to Use
consultation form→scale→connect→report→expert→share Assessment + AI expert follow-up
assessment form→scale→connect→report→share Self-assessment with PDF report
report form→scale→connect→report→share External evaluation/rating
exam form→scale→connect→report→share Knowledge test with correct answers
survey form→connect→share Data collection, no scoring
learn form→connect→share Flashcard learning with quick quizzes

The rule is strict: the AI cannot generate modules outside this table. If the user asks for something that doesn't fit any row, the AI picks the closest match. No free-form module creation.

The YAML Output Contract

The plan agent outputs pure YAML, wrapped in code fences:

planType: assessment
description: 'Workplace Stress Assessment'
tasks:
  - seq: 1
    skill: form
    title: 'Design Assessment Form'
    description: 'Create 3-dimension stress assessment with 13 questions'
    knowledge: |
      Scene: assessment. Name field: "Your Name".
      3 dimensions:
      - Workload (5 questions, measures perceived work intensity)
      - Role Stress (4 questions, measures role ambiguity and conflict)
      - Social Support (4 questions, measures workplace support networks)
      Total: 13 questions. Use scale type (1-5 Likert).
  - seq: 2
    skill: scale
    title: 'Configure Scoring'
    knowledge: |
      3 dimensions, all negative direction (high score = worse).
      Workload: 5 questions, sum format.
      Role Stress: 4 questions, sum format.
      Social Support: 4 questions, sum format.
      3-tier ranges per dimension (severe/moderate/mild).
  - seq: 3
    skill: connect
    lookAndFeel: 'theme:Noir; look:workplace stress, psychological assessment, dark professional; layout:stack'
    title: 'Configure Pages and Style'
    knowledge: |
      Cover: hook with 73% statistic, 5-minute estimate, dimension preview.
      Main: field mode (one question per page).
      Final: report-pdf with download enabled.
  - seq: 4
    skill: report
    lookAndFeel: 'theme:Noir; look:dark professional, consistent with connect'
    knowledge: |
      3 dimensions, theme:Noir. Radar chart for summary.
      Per-dimension: high/mid/low 3-tier interpretation.
      Total score: low=good, high=needs support (symptom scale).
  - seq: 5
    skill: share
    title: 'Publish'
    knowledge: |
      Anonymous access, one submission per person, permanent validity.
Enter fullscreen mode Exit fullscreen mode

The Knowledge Self-Containment Rule

This is the most important rule in the entire system, and it took three iterations to enforce:

Each task's knowledge field is the ONLY context that skill receives.

Task 1 (form) knows nothing about Task 2 (scale). Task 2 knows nothing about Task 3 (connect). When the scale skill runs, it doesn't see what the form skill generated — it only sees its own knowledge field plus runtime references (the actual form fields that now exist in the database).

Why? Because the execution engine runs each task as an independent AI call. There's no conversation history between tasks. This was a deliberate design choice for reliability — if Task 3 fails, Task 4 can still run because it doesn't depend on Task 3's chat history.

But it means the plan agent must be explicit. If the form task says "3 dimensions: Workload, Role Stress, Social Support," the scale task must repeat those exact names. The connect task must mention them for the cover page preview. The report task must list them for interpretation.

The Strong Consistency Rule

The plan agent enforces verbatim name consistency across tasks. If the form task calls a dimension "Workload," the scale task cannot call it "Workload Intensity." The names must match character-for-character because downstream modules do string comparison.

// In the execution engine, scale dimension names are matched against
// form field groupings by exact string comparison
for (Scale scale : factor.getScales()) {
    String scaleName = scale.getName();
    // This must EXACTLY match the dimension name in the form task's knowledge
    if (!formKnowledge.contains(scaleName)) {
        FLog.w(TAG, "Scale name mismatch: " + scaleName);
    }
}
Enter fullscreen mode Exit fullscreen mode

What Went Wrong: The "Same As Above" Problem

Version 1 of the plan agent didn't enforce self-containment. The AI, being efficient, would write:

# Task 1
- skill: form
  knowledge: |
    3 dimensions: Workload, Role Stress, Social Support.
    13 questions total.

# Task 2 — the AI got lazy
- skill: scale
  knowledge: |
    Same 3 dimensions as above. Sum format, 3-tier ranges.
Enter fullscreen mode Exit fullscreen mode

"Same 3 dimensions as above" — except there is no "above" when the scale skill runs. It's a fresh AI call with only this knowledge block. The scale skill would generate 3 empty dimensions named "Dimension 1," "Dimension 2," "Dimension 3" because it had no idea what the actual dimension names were.

What Went Wrong: Phantom Modules

Version 2 introduced a different problem. The AI would sometimes add a feedback or analytics module that doesn't exist:

tasks:
  - skill: form
  - skill: scale
  - skill: connect
  - skill: report
  - skill: analytics   # ← doesn't exist
  - skill: share
Enter fullscreen mode Exit fullscreen mode

The execution engine would crash trying to load skills/assess/skill-analytics/SKILL.md.

The fix: the available skills are now dynamically injected into the plan agent's system prompt at runtime:

StringBuilder skillsSection = new StringBuilder();
skillsSection.append("## Available Skills\n\n");
skillsSection.append("Each task's `skill` field MUST be selected from this list:\n\n");
for (String skillId : BUILDER_SKILL_IDS) {
    AssessPlanSkill skill = getOrCacheSkill("skills/assess/skill-" + skillId + "/SKILL.md");
    skillsSection.append("- `").append(skillId).append("`: ").append(skill.getSummary()).append("\n");
}
String systemPrompt = agentContent + skillsSection;
Enter fullscreen mode Exit fullscreen mode

The AI sees the exact list of available skills in its system prompt. No more phantom modules.

What Went Wrong: PageTemplates Leakage

The plan agent used to output a pageTemplates field in report tasks — a list of template IDs for the report module. This was fragile because the AI would sometimes output invalid template names, or forget to include them entirely.

The current solution: the plan agent never outputs pageTemplates. Instead, it declares two values in the report task's knowledge:

DimN=3
theme=Noir
Enter fullscreen mode Exit fullscreen mode

The execution engine reads these values and computes the page template list automatically:

// In AssessAgent.skillGenerateStream()
if ("report".equals(skillName) && ObjectUtil.isEmpty(task.getPageTemplates())) {
    // Recover from scale data — count actual dimensions in memory
    Factor _factor = session.getFlower().getModel().getFactor();
    if (_factor != null && !ObjectUtil.isEmpty(_factor.getScales())) {
        int _dimCount = _factor.getScales().size();
        List<String> _recovered = buildPageTemplates(_dimCount, task, planType);
        task.setPageTemplates(_recovered);
        FLog.w(TAG, "pageTemplates recovered from scale data, dimCount=" + _dimCount);
    }
}
Enter fullscreen mode Exit fullscreen mode

This is a runtime fallback: even if the plan agent messes up, the execution engine can reconstruct the page template list from the actual scale dimensions in memory. The AI doesn't need to know about template IDs at all.

The LookAndFeel Field

Three tasks can carry a lookAndFeel field: connect, report, and expert. This is a semicolon-delimited string:

theme:Noir; look:workplace stress assessment, dark professional, deep blue; layout:stack
Enter fullscreen mode Exit fullscreen mode

The plan agent generates this once, and downstream skills extract values from it:

  • theme: → presets the visual theme (Noir, Classic, Fresh, etc.)
  • look: → natural language visual description for the style AI
  • layout: → page flow type (flat, stack, cube)

The rule: if a task has lookAndFeel, downstream skills must use it. If the connect task says theme:Noir, the report task cannot override it with theme:Fresh. The plan agent enforces this by generating consistent lookAndFeel values across connect/report/expert tasks.

Lessons Learned

  1. Constrain the AI. Free-form planning sounds flexible but produces garbage. A decision matrix with 6 rows covers 95% of use cases and eliminates phantom modules.

  2. Self-containment is non-negotiable. Cross-task references break when tasks execute independently. Validate and reject them early.

  3. Don't trust the AI with internal IDs. Template names, page IDs, and other internal identifiers should be computed by the engine, not generated by the AI.

  4. Always have a runtime fallback. If the plan agent forgets to declare dimension count, the execution engine should be able to count them from the in-memory scale data.

Next: [Form Module — Scene-aware field types and inline scoring]

Top comments (0)