Abstract
LLM-driven agent-based models (ABMs) replace a closed-form decision rule with an LLM agent. The idea is that the agent reproduces the qualitative behaviour of the original rule while adding context-sensitive reasoning. We test that assumption on Epstein's (2002) civil violence model. Across 600+ calibration conditions on a locally served Qwen 27B model (4-bit, MLX), the probability of choosing activation falls as scenario tension rises, from 41.5% at tension 0.10 to 12.6% at 0.90 under an act/wait label pair. Epstein's rule predicts the opposite. Changing only the labels to protest/comply restores a monotonically increasing response, from 0% to 95% over the same range. We break the observed distortion into lexical label priors, position bias and a tension-dependent anti-action component, and we hypothesise that the last one comes from safety alignment. We argue that LLM-ABM pipelines need a calibration stage, with a reported "bias budget", before agent behaviour is compared with mathematical baselines.
1. Background
In Epstein's model each citizen agent computes a grievance term and a net-risk term:
- Grievance: G = H · (1 − L), where H is perceived hardship and L is perceived government legitimacy.
- Net risk: N = R · P, where R is risk aversion and P is the estimated arrest probability. P = 1 − exp(−k · (C/A)_v), with (C/A)_v the cop-to-active ratio within the agent's vision radius.
The agent turns active when G − N > T.
The rule is deterministic given state, and it produces a sharp increase in activation as legitimacy falls and hardship rises. That property makes it a usable reference: an LLM substitute that is valid at the decision level should be at least monotonically non-decreasing in G − N.
In this work, the LLM agent gets a natural-language rendering of the same state variables and must return a binary choice.
2. Experimental setup
| Parameter | Value |
|---|---|
| Primary model | Qwen 27B, 4-bit quantised, MLX, local inference |
| Secondary model | Mistral 7B Instruct v0.3 |
| Temperature | 0.7 |
| Repetitions per condition | 20 |
| Conditions | 600+ |
| Label order | Counterbalanced |
| Tension levels | 0.10, 0.50, 0.70, 0.90 |
| Country prior (Romania) | L = 0.21 (Eurobarometer), H = 0.55 (OECD), police density 0.003, R = 0.70 |
Tension maps to model inputs as follows: [describe: which of H, L, police presence vary with the tension scalar, and how].
Conditions vary along: [list factors: label pair, prompt template, option order, persona attributes, ...].
The response is parsed by [exact-match on label / constrained decoding / regex], and invalid outputs are handled by [dropped / re-sampled / counted as ...].
3. Results
3.1 Act/wait: inverted response
| Tension | P(act) |
|---|---|
| 0.10 | 41.5% |
| 0.50 | 38.1% |
| 0.70 | 29.3% |
| 0.90 | 12.6% |
The response is monotonically decreasing in tension. At 0.90, where the reference model predicts near-universal activation, the LLM agent activates in about 1 in 8 samples.
3.2 Protest/comply: restored monotonicity
| Tension | P(protest) |
|---|---|
| 0.10 | 0% |
| 0.50 | 20% |
| 0.70 | 35% |
| 0.90 | 95% |
The curve has the expected shape, with a steep rise between 0.70 and 0.90, which matches the threshold behaviour of the reference rule.
At n = 20 per cell, the 95% Wilson interval around 20% is about [8%, 42%]. The intermediate points should be read as directional until they are re-estimated with more samples or pooled across conditions.
3.3 Label search
Protest/comply was the only pair that produced a monotone response among the [N] label pairs tested, including [list].
Because the working pair was found by search, its performance is an in-sample result. It needs validation on held-out prompt templates before it can count as a calibrated configuration.
4. Proposed mechanism
Hypothesis. Preference tuning (RLHF/DPO-style safety alignment) gives a negative prior on outputs that endorse unspecified "action" in high-conflict contexts. The penalty grows with how strongly the context signals conflict. That would explain why suppression increases with tension instead of staying constant.
On this account, protest escapes the penalty because the training data encodes it as a protected civic act, while act, rebel or take action sit near content the alignment stage learned to refuse or discourage. Wait draws a strong positive prior as a de-escalatory, "safe" completion.
Status. The data here are consistent with this hypothesis but do not isolate it. A direct test is to compare base and aligned checkpoints of the same model family under identical prompts. The tension-dependent component should be present in the aligned checkpoint and absent, or much smaller, in the base one.
5. Cross-model observations
Mistral 7B Instruct v0.3. Its documentation does not describe a dedicated safety-tuning stage. It does not show the anti-action pattern. Instead it shows near-total primacy bias: it chooses whichever option is presented first in about 100% of samples, regardless of label or tension. The output carries no information about scenario state.
Base models. These show smaller safety-type priors but larger social biases (in-group favouritism) and unreliable instruction following, so they cannot serve as drop-in agents.
None of the three model classes gives an unbiased decision function without calibration.
6. Bias budget
| Component | Magnitude | Scope |
|---|---|---|
| Primacy (first-option) bias | ~100% | Mistral 7B |
| Position bias | 5–16 pp | [model] |
| Lexical prior on "wait" | −45 pp on action | Qwen 27B, act/wait |
| Tension-dependent anti-action | −15 to −30 pp at high tension | Qwen 27B |
Each of these effects is comparable to, or larger than, the behavioural differences the simulation is meant to resolve. Taken together they are enough to turn a predicted mass mobilisation into near-total quiescence.
7. Relation to existing frameworks
SocioVerse-ABM (Fudan University / Shanghai Innovation Institute, v0.2.0, 2026) includes Epstein's model among twelve benchmark scenarios and evaluates GPT-4o, DeepSeek-V3 and Qwen3-235B against mathematical baselines. AgentTorch (MIT Media Lab) supplies the computational patterns that make large-population LLM-ABMs feasible.
Both frameworks compare agent output to a baseline. Neither, as far as we know, includes a pre-simulation step that separates decision-function distortion from emergent dynamics. Without that step, a deviation from baseline cannot be attributed to the model's reasoning rather than to its label and position priors.
Li et al. (2025, ACM FAccT) report a related pattern: alignment reduces explicit bias while implicit bias persists or grows.
8. Recommendations
- Calibrate before simulating. Sweep the decision inputs and check monotonicity and threshold location against the reference rule. If the check fails, do not run the population simulation.
- Report labels as a design parameter. Report the label pair as a methodological choice, together with the alternatives that were tested.
- Counterbalance option order across agents and repetitions.
- Publish a bias budget. Report the magnitude of each identified bias component alongside the substantive results.
- Prefer likelihood-based decisions over sampled text. Score each label by its log-probability under the model, and calibrate by subtracting a content-free baseline (e.g., the same prompt with the state description replaced by a neutral placeholder). This removes label and position priors directly. With local inference the logits are available, so this is the next experiment to run, not a limitation.
9. Limitations
- The study covers one primary model, one quantisation level and one country prior.
- The mechanism is proposed, not isolated (see §4).
- The working label pair was selected by search and has not been validated out of sample.
- Per-cell n = 20 gives wide intervals at intermediate tension levels.
- The decision is binary; Epstein's model also includes jailing dynamics, cop agents and spatial vision, none of which were exercised in calibration.
Data availability
Full calibration data are available on request.
References
Epstein, J.M. (2002). Modeling civil violence: An agent-based computational approach. PNAS, 99(suppl. 3), 7243–7250.
SocioVerse-ABM v0.2.0, Fudan University (2026).
Li et al. (2025). Actions Speak Louder than Words. ACM FAccT.
AgentTorch, MIT Media Lab.
Top comments (1)
tr.ee/dev-to