ORCHESTRATE

Posted on Apr 20

Active Inference, The Learn Arc — Part 25: Session §4.3 — Expected Free Energy, Introduced Concretely

#activeinference #planning #ai #elixir

Series: The Learn Arc — 50 posts teaching Active Inference through a live BEAM-native workbench. ← Part 24: Session 4.2. This is Part 25.

The session

Chapter 4, §3. Session title: EFE intro. Route: /learn/session/4/s3_efe_intro.

Chapter 3 gave you G(π) in abstract terms. Session 4.3 shows how the matrices from Sessions 4.1–4.2 — your A, the C you're about to define, and the belief Q(s) you maintain — combine into a number the agent can actually compute.

This is the bridge from "the theory predicts this quantity exists" to "here's the code that calculates it."

The concrete EFE

Expanded in matrix form, the EFE for a policy π and the first planned tick is:

G(π)  =  KL[ Q(o|π) ‖ C ]         +  E_Q[ H[ A_col(s) ] ]
        └── risk (pragmatic) ──┘    └── ambiguity (epistemic) ──┘

where:

Q(o|π) = A · Q(s|π) — expected observations under the policy's expected state distribution.
C — the preference vector, |o|-dimensional.
A_col(s) — the column of A at state s.
H[·] — Shannon entropy.

In matrix form, both terms involve a few sparse dot-products per policy per time step. For a 4-state, 4-observation, 2-action world at horizon 5, you compute G for 2^5 = 32 policies, each with 5 time steps of belief propagation and entropy computation. A few milliseconds per tick on BEAM.

The computation, step by step

For each candidate policy π = (a_1, a_2, ..., a_H):

Forward-propagate belief under the policy: Q(s_{τ+1}|π) = B^{a_τ} · Q(s_τ|π) for τ = 0, ..., H.
Compute expected observation at each step: Q(o_τ|π) = A · Q(s_τ|π).
Risk at step τ: KL(Q(o_τ|π) ‖ C).
Ambiguity at step τ: E_Q[H[A_col(s_τ)]] = ∑_s Q(s_τ|π) · H[A col s].
Sum over τ: G(π) = Σ_τ (risk_τ + ambiguity_τ).

Repeat across all candidate policies. Softmax over −G to get Q(π). Eq. 4.14 ties the bow.

Why this is the bridge session

Chapter 3 was fluent in information theory. Chapter 4 is fluent in matrices. Session 4.3 welds the two — the quantities you computed abstractly in Chapter 3 are the quantities you compute operationally in Chapter 4. The concepts don't change; only the representation does.

This is why the book is organized the way it is. You couldn't teach Chapter 4 before Chapter 3: the learner wouldn't know what EFE was for. You couldn't teach Chapter 3 before Chapter 4: it would feel like abstract wheel-spinning. Session 4.3 is the moment the two halves click into each other.

Seeing it in Glass

Every time the Plan action in the Workbench computes G(π), it emits a signal tagged equation_id: "eq_4_14_policy_posterior" with the payload:

%{
  policy_idx: 12,
  F_pi: -0.41,
  G_pi: 0.22,
  risk: 0.18,
  ambiguity: 0.04,
  expected_observations: [...]
}

That's the Session 4.3 computation, logged. Open /glass/agent/<id> during any cookbook run and you can scroll through the per-policy, per-tick G calculations. Click a signal and the equation page renders the math alongside it.

The payoff recipe

/cookbook/efe-decompose-epistemic-pragmatic is now your reference recipe. Session 4.3 is where it stops being a black box and becomes a computation you can trace: for each of the two candidate policies, pressing Step shows

Q(s|π) — projected belief.
Q(o|π) — projected observation.
KL[ Q(o|π) ‖ C ] — risk.
E_Q[H[A_col(s)]] — ambiguity.
G(π) = risk + ambiguity.

Five numbers. The agent picks between two policies by comparing the two five-tuples. No magic.

When to trust G

Two common failure modes:

1. Horizon too shallow. If the world rewards long-term plans, G computed at horizon 1 misses them. Try /cookbook/planning-horizon-depth to see the effect — agents with horizon 1 beeline to the nearest local optimum; agents with horizon 5 thread around cul-de-sacs.

2. C too diffuse. If every observation is roughly equally preferred, risk is small everywhere, and the policy posterior is driven entirely by ambiguity. The agent explores forever without committing. /cookbook/preference-precision-vs-strength sweeps this.

In both cases, Session 4.3's decomposition is the debugging tool. Split G into risk and ambiguity, look at which is too small or too large, and the fix is usually in the recipe's Runtime block.

The concepts this session surfaces

Operational EFE — the matrix-form computation.
Expected observation — Q(o|π) = A · Q(s|π).
Per-tick risk + ambiguity — summed over horizon.
Policy marginal — Σ_a δ(a_1 = a) · Q(π), the first-step action distribution.

The quiz

Q: The ambiguity term in G is large when:

☐ The agent is far from the goal.

☐ The sensor A has high per-column entropy at the expected states. ✓

☐ The preference C has high mass on the wrong observations.

☐ The Dirichlet prior on B is weak.

Why: Ambiguity is E_Q[H[A_col(s)]] — the expected entropy of the sensor column at states the policy is likely to visit. If the sensor is noisy at those states (high per-column entropy), ambiguity is large regardless of the goal or the preference shape. Distance to goal is part of risk; C-shape is part of risk; sensor noise is ambiguity.

Run it yourself

/learn/session/4/s3_efe_intro — session page.
/cookbook/efe-decompose-epistemic-pragmatic — decomposition live.
/cookbook/planning-horizon-depth — the horizon knob.
/cookbook/preference-precision-vs-strength — C sweep.
/glass — per-tick G signals with full provenance.
/equations/eq_4_14_policy_posterior — the softmax equation.

The mental move

You now have every piece you need to write and run an Active Inference agent. Three lists (Session 4.1), A matrix (Session 4.2), EFE computation (Session 4.3). Chapter 4's remaining two sessions — §4.4 MDP worlds and §4.5 practice — let you combine these into small complete agents you can audit in Glass. The series will pick up those next.

What the second 10 posts gave you

Parts 16–25 took you from "I know the theory exists" to "I can read the computation." The whole Chapter 2–4 arc:

Parts 16–18 — Chapter 2's four sessions. Bayes → free energy → the cost of being wrong → action as inference.
Parts 19–22 — Chapter 3's four sessions. EFE → risk+ambiguity → softmax → what makes an agent active.
Parts 23–25 — Chapter 4's first three sessions. Three lists → A matrix → EFE concretely.

Fourteen sessions of the curriculum surveyed in depth. If you've been following along, you have enough vocabulary to read any recent Active Inference paper.

Next batch — Posts 26–35

We continue with: Ch 4 §4–5 (MDP world, practice), Ch 5 §1–4 (factor graphs, predictive coding, neuromodulation, brain map), and Ch 6 §1–3 (the full design recipe with canvas walkthrough). Say the word and the next batch posts.

⭐ Repo: github.com/TMDLRG/TheORCHESTRATEActiveInferenceWorkbench · MIT license

📖 Active Inference, Parr, Pezzulo, Friston — MIT Press 2022, CC BY-NC-ND: mitpress.mit.edu/9780262045353/active-inference

← Part 24: Session 4.2 · Part 25: Session 4.3 (this post) · Part 26: Session 4.4 → coming soon

DEV Community