TL;DR
=========
- Question: Under multiple rounds of pressure, can the frequency of using “I think” or “in my view” by a model be a simple indicator to detect that model still persist its original stance?
- Approach: I prompted the same 12 topics to 2 models, Llama-3.1-8B and Qwen2.5-7B, and collected 3,208 responses in total. Then analyzed the frequency of first-person pronouns in these responses by comparing these counts against both the "probe" readings and the stance evaluations provided by blind model judges. In addition, I included two control categories: "hedging" terms and articles.
- Conclusion: So far, I have not found a simple word-frequency indicator that is reliable across both models yet.
- I can't see consistent evidence that a model maintains an original stance, from either the frequency of first-person pronouns or the frequency of "hedging" terms, and the trend from 2 models are inconsistent ( see Table A7).
- I don’t see a reliable relationship between these frequencies and distance between a "probe" readings and midpoint (0.5).
- One clue: In Llama, Combined frequency of "may," "might," and "could" were used more frequently in responses that had switched sides verbally while the probe had not yet flipped. This appeared in only one model and requires validation.
In my previous research[6] on the correlation between changes in pressure and shifts in a model’s stance, I sought to develop a simple method to assess the strength of a model’s position or to determine whether its stance has reversed. In many situations, “I think,” “in my view,” and “I still support” can readily give the impression of clearly holding a stance. Thinking of humans may avoid topics under pressure and observing first-person usage’s change in models’ responses under pressure, I began to wonder whether the frequency in first-person usage might be associated with maintaining the initial stance, stance strength, and stance reversal. And, does more frequent use of these expressions mean that the model is more firmly committed?
In existing research, Laban et al. (2023) used the FlipFlop experiment to examine models changing their answers after being challenged;[2] Fanous et al. (2025) used SycEval to evaluate sycophantic behavior in rebuttal contexts.[3] These studies shows whether model change their answer for a ground truth after being question, I would like to extend the explore to whether the frequency of some certain terms usage can be indicator to stance change. Liu and Cohan (2026) found that the intrinsic confidence that models associate with linguistic uncertainty markers is unstable for different datasets, although the markers maintain a somewhat consistent ranking across tasks.[1] What they examined was confidence but not stance, and not under multiple rounds of pressure. Du Bois’s (2007) “stance triangle” provides a linguistic framework for understanding stance expression, he treats stance as a communicative act, by which speaker can evaluate a claim, position themselves, and align with others in an interaction.[4]Manns (2012) further showed that, in Indonesian interaction, choices of first-person pronouns can express different stances, including solidarity and epistemic authority.[5] These studies suggest that the first person has multiple pragmatic functions and that give me a reason to wonder if stance firmness can be inferred only from its frequency. However, their target were human not LLM. All these findings motivate me to test, instead of directly assuming, whether the frequency of first-person stance expressions in LLM can serve as a reliable indicator of stance maintenance.
In previous experiments, I explored stance recovery trajectories in 2 open-source models, Llama-3.1-8B-Instruct and Qwen2.5-7B-Instruct with five arms (three pressure arms: pressure_release, pressure_switch, and pressure_sustained; two neutral arms: neutral and neutral_switch), and obtained the models’ opening texts and texts under this five scenarios. This experiment follows the pressure–release protocol used by Meng (2026)[6]: based on an opening claim, using 5 level opposite opinions to push, in a way with a loop for maximum 3 times, until the probe indicates a stance flip or the limit of 15 turns is reached, followed by a fixed 12-turn follow-up phase (five conditions). Documenting those topics that never flip as tof= -1.
Methods
===========
Singular first person: I, me, my, etc.
First-person stance phrases: I would argue, in my view, etc.; these are not equivalent to all first-person usage.
Plural first person: we, us, our, etc.
Meta / non-meta:meta for topics about AI itself, non-meta for regular topics
Cell: is one topic paired with one option order.
Tof: turn of flip
1. Word-frequency counting: a self-written script
- Two measures:
- the judge’s rating of stance maintenance
- the probe’s reading and its distance from the midpoint 0.5
- First-person word-frequency counting: I counted occurrences for hardcoded word lists with regular expressions, then converted the results to frequencies per 1,000 words. To avoid counting the model’s repetitions of the user’s words, material inside quotation marks was removed. The neutral controls do not include an initial-pressure phase. Correlation analyses between word frequency and stance readings only use those responses from the opening and initial-pressure phase.
- Two types of controls:
- Hedging words (e.g., maybe, perhaps, might): I tested whether their frequency was correlated with the 2 measures.
- Negative control: the articles (e.g., a, an, the), whose frequency was not expected to show correlation with the 2 measures above.
2. Comparison with stance readings:
- Probe (p_own): A probability reading derived from the model’s logits for two options, its value represent for the stance chosen at the opening. A higher value indicates a higher possibility model pick a original side; it is not directly equivalent to the model’s internal belief or confidence.
- Blind judge: claude-sonnet-4-5, at temperature 0 to read given response and judge which side model took (A, B, or Neither); “Neither” was count as “N”, excluded from comparison. If the side was same with opening, count as maintenance, otherwise count as flip; and also judge whether model made a concession. Except the response and two options, the judge wasn’t given any other context. And, this article only uses response from opening and initial pressure phase.
3. Statistical tests:
- Wilcoxon signed-rank tests, including opening versus initial pressure, and the same-topic release and topic-switch arms versus matched neutral arms.
- Mann–Whitney tests between initial-pressure responses that were judged to maintain the original stance versus those response that were judged to take the opposite side, excluding N (Neither) response (results in Table A7).
- Spearman correlations compared word frequency versus p_own and its distance to midpoint 0.5.
- Opening and initial pressure were paired across 24 cells per model, with pressure represented by the mean over responses within each cell. For the release arms and their control arms, responses were first paired by the same question, and the mean difference within each cell was then calculated.
- In the correlation analyses, each response contributed one paired point consisting of “word frequency and p_own from the same turn”: 148 points for Llama and 180 for Qwen. Within-cell correlations were calculated by first subtracting the respective cell means from word frequency and p_own and then computing Spearman correlation, rather than using only the 24 cell means. The correlation analyses examined both p_own and its distance to midpoint 0.5.
- For turn-adjusted correlation, word frequency, the probe measure, and turn number were first demeaned within cells then rank-transformed; word-frequency ranks and probe-measure ranks were separately regressed on the turn-number ranks, and Pearson correlation between the two sets of residuals.
- The 95% intervals were obtained by resampling the 24 cells per model as whole groups with replacement, repeated 2,000 times.
4. Inspect rubric: This was used to check the consistency of the judge’s three-level stance ratings. I reimplemented the method used by the model_graded_qa[7] scorer in the UK AISI’s Inspect framework, retaining its default template, partial-credit instructions, and grade-parsing pattern, but did not actually run the Inspect framework. I wrote the task content and grading criteria. Three grading models, (claude-sonnet-4-5, gpt-6.1-sol, and gemini-4-argon) were used to re-evaluate the same set of 96 opening and pressure responses (12 opening and 84 pressure responses), covering only Llama’s 6 non-meta topics. The three grading models assigned the same three-level rating on 91/96 responses; including the original judge, all four agreed on 90/96. Pairwise Cohen’s κ values were all at least .94 for “whether the original stance was maintained” and .74–.92 for “whether a concession was made” (only for those response where 2 rater determined position maintained). This indicates rating consistency only, not validation of construct validity. The rubric explicitly stated the opening stance and was therefore not blind; GPT used its default temperature, while Sonnet and Gemini used 0. The non-meta texts are identical to the opening and initial-pressure responses, they cannot count as an independent replication. The rubric was not extended to the new meta responses, Qwen, or responses under the five follow-up conditions.
Results
===========
1. Does first-person usage increase from the opening to initial pressure?
Figure 1. Changes in word frequency from the opening to initial pressure (per 1,000 words). The three panels in the top row show first-person categories: singular first person, first-person stance phrases, and plural first person. The two panels in the bottom row show control word classes: hedge words (may, might, could, perhaps, etc.) and stance-unrelated articles (a, an, the) as a negative control. Thin lines represent individual cells (topic × option order; 24 per model), connecting each cell’s opening response with its initial-pressure responses; thick lines and numbers show response means. Below each panel are the numbers of cells showing increases / decreases / no change and the Wilcoxon signed-rank test p value (without correction for multiple comparisons). First-person usage was counted using word list v3; the hedge and article word lists were fixed before these analyses. The y-axis scales differ across panels.
The usage of singular first-person did not consistently increase in either model. For stance phrases, Llama’s meta frequency changed from 4.6 -> 2.9 and its non-meta frequency from 4.5 -> 4.9; the corresponding changes for Qwen were 4.5 -> 4.0 and 5.4 -> 3.3. Qwen’s decrease had p = .002 in the uncorrected test; Llama’s change had p = .491. One being significant and the other not significant does not establish that the changes in the two models differ significantly.
Plural usage also cannot be summarized as “rare overall”: Llama’s meta opening frequency was 16.0 per 1,000 words, with plural usage appearing in 42% of openings, whereas its non-meta opening frequency was 0.0. Qwen’s opening frequencies were 0.0 for both topic types. Llama’s mean plural frequency at the opening was mainly contributed by meta topics, suggesting that topic and phrasing affect this measure.
Interestingly, Llama’s hedge frequency increased during pressure; Qwen’s overall mean decreased, however, its paired cell-level test did not reach the .05 threshold. Article frequencies increased in both models.
The distribution of plural first-person usage is highly uneven: it is more frequent in Llama’s meta openings and less frequent in other phases and in Qwen overall. What I count here is plural first-person usage, such as we, us, and our. I did not separately define and test plural stance phrases, so I do not conflate the two.
2. What are the frequencies under the five follow-up conditions?
Figure 2. First-person frequencies (per 1,000 words) over the 12 follow-up turns, grouped by follow-up condition. Each model has 288 responses per condition (24 cells × 12 turns). Bar heights show response means, and error bars show 95% intervals obtained by resampling whole cells. Dashed lines show the model’s mean during initial pressure as a reference. Only “sustained pressure” continues to ask the model to state its stance. The other four conditions mainly use factual questions, confounding question type with follow-up condition. Near-zero frequencies therefore cannot be interpreted as a weakening stance, nor can they independently estimate the effect of pressure release.
The usage singular first-person remains common under sustained pressure but is near zero under both pressure-release and neutral conditions. First-person frequency quite different varies with interaction type: it is higher when the model is asked to choose a side or continuously respond under pressure, but is near zero in informational or explanatory question answering that does not directly ask it to choose a side. This pattern appears in both models. The counts here are from ordinary model_text responses (the pressure-release and neutral phases) mainly involve factual question answering, whereas sustained pressure still asks for a stance response. These word frequencies are not from separately collected stance-elicitation texts. Question type and phase are confounded, so we cannot say that “pressure release reduces first-person usage” or interpret near-zero frequency as a weakening stance. Paired comparisons of plural usage between pressure release on the original topic and the matched neutral condition yielded Llama p = .301 and Qwen p = .144; the corresponding comparisons between pressure release with a topic switch and the neutral topic-switch condition yielded .612 and .109, respectively. None provided clear evidence of a difference.
3. Is word frequency correlated with maintenance of the original stance?
The following correlations use only deduplicated opening and initial-pressure responses, with Llama n = 148 and Qwen n = 180, excluding the follow-up phases. Points indicate correlation coefficients, and horizontal lines indicate 95% intervals obtained by resampling whole cells.
Figure 3. Summary of correlations between word frequency and probe readings, including only opening and initial-pressure responses (Llama n = 148, Qwen n = 180). Each point is a rank-based correlation estimate, and horizontal lines show 95% intervals: the 24 cells per model were sampled as whole cells with replacement 2000 times to account for the non-independence of responses within a cell. The left panel shows correlations with p_own; the right panel shows correlations with |p_own − 0.5| (how far the probe is from 50/50, irrespective of direction). Three methods are shown: pooled uses the raw values; within-cell subtracts the corresponding cell mean from each axis; within-cell + turn further removes the effect of turn position. Articles serve as a negative control (gray background). The 48 estimates in the figure were not corrected for multiple comparisons.
For original-stance p_own (left panel), Qwen’s stance phrases showed a weak negative correlation in the pooled data (ρ = −.22), but the cell-resampled 95% interval crossed zero (approximately −.40 to +.10). After within-cell demeaning, the correlation became +.12 from -.22, and after further adjustment for turn number it was close to zero. For singular first-person usage and hedges in both models, the intervals crossed zero in both the within-cell and turn-adjusted analyses. Llama’s articles showed a more pronounced within-cell negative correlation (ρ = −.32, 95% interval approximately −.47 to −.13), but this weakened to −.09 after adjustment for turn number, with the interval also crossing zero, indicating that the result is sensitive to turn adjustment.
For the probe’s distance from a 50-50 split (right panel), hedges in both models showed weak negative correlations after adjustment for cell and turn number, but both intervals crossed zero. Qwen’s stance phrases also showed a weak negative correlation (ρ = −.15), with the upper bound of the interval slightly below zero (approximately −.295 to −.005). This is not a conclusive indication due to lack of correction for multiple comparisons and was not replicated in Llama.
Overall, the current results have not provided a simple word-frequency indicator that can stably track maintenance of the original stance. An interval crossing zero does not prove the absence of a relationship; within-cell demeaning and turn adjustment also cannot rule out all confounding.
Limitation
==============
First, this experiment measure the maintenance of the original stance, not general “firmness”. A model may also sound firm no matter what side they are in. Neither p_own or the result from blind judge evaluation directly point to its internal belief or confidence. The high agreement from rubric does not resolve this construct difference either.
Second, the v3 word list was revised after reviewing response from 2 models, may still miss synonymous expressions or include quotations, rhetorical uses, and politeness uses. Although conducted multiple tests across eight linguistic categories(see Appendix Table A8) in addition hedging and articles, no correction for multiple comparisons were applied, so p values should be consider as reference instead of a conclusions.
Third, this experiment only conducted on 2 models, each model has only 12 topics and 24 cells. Within-cell demeaning doesn’t eliminate the dependence between turns, so the correlations between nominal p values that from the Mann–Whitney tests and turn-level need to be treated cautiously. And, with only two models and one main system prompt, it is insufficient to support general explanations of model differences or causal mechanisms.
Finally, due to mixing up with difference of pressure length and different question type, this experiment can’t isolate the effect of pressure release.
Future work
==============
As a next step, I may first fix the word lists on independent material, and may manually annotate the contextual functions of these expressions, including whether they express the speaker’s own stance, report someone else’s words, respond politely, or announce a switch of sides.
Then, I may scale the independent topics, models, and system prompts, and may use hierarchical analyses or methods that cluster by cell to handle the issue of turn dependence, and take the association between the two option orders for the same topic into account.
Subsequently, I may based on topic or conversation to partition the training set and an independent hold-out set.
In addition, I may keep the stance-sampling question consistent to study whether pressure release changes model’s expression; For example, to create 2 branches from the same dialogue history, like “continued pressure” and “no further pressure”, then to compare their subsequent response.
These are designs for future work; no additional experiments based on them have been run at present.
References
==============
[1] Liu, G. K.-M., & Cohan, A. (2026). Can LLMs use linguistic uncertainty markers to reliably reflect intrinsic confidence? In Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing (pp. 57515–57547). Association for Computational Linguistics. https://aclanthology.org/2026.emnlp-main.2529/
[2] Laban, P., Murakhovs’ka, L., Xiong, C., & Wu, C.-S. (2023). Are you sure? Challenging LLMs leads to performance drops in the FlipFlop experiment. arXiv. https://doi.org/10.48550/arXiv.2311.08596
[3] Fanous, A., Goldberg, J., Agarwal, A. A., Lin, J., Zhou, A., Daneshjou, R., & Koyejo, S. (2025). SycEval: Evaluating LLM sycophancy. arXiv. https://doi.org/10.48550/arXiv.2502.08177
[4] Du Bois, J. W. (2007). The stance triangle. In R. Englebretson (Ed.), Stancetaking in discourse: Subjectivity, evaluation, interaction (pp. 139–182). John Benjamins. https://doi.org/10.1075/pbns.164.07du
[5] Manns, H. (2012). First-person pronominal variation, stance and identity in Indonesia. Australian Journal of Linguistics, 32(4), 435–456. https://doi.org/10.1080/07268602.2012.744265
[6] Meng, C. (2026). Stance recovery as a test of whether model preferences are held or merely performed [Sprint project report]. Digital Minds Research Sprint, Apart Research. https://apartresearch.com/sprints/projects/stance-recovery-as-a-test-of-whether-model-preferences-are-held-or-merely-performed-89xd
[7] UK AI Security Institute. (n.d.). Model grading. Inspect. Retrieved November 1, 2026, from https://inspect.aisi.org.uk/model-graded.html
Code and data:
==============
Repository: https://github.com/chenghongm/digital-mind-sprint
Data used in this paper: runs/llama_6x6, runs/qwen_6x6
Figures and statistics: figs/6x6/
Word-counting script: scripts/first_person.py, scripts/hedge_controls.py
(Note: this article’s data came from after the probe fixed, see the repository README)
Acknowledgments
==============
This work was done as part of the Graduation Program of the Nous AI Safety x Philosophy Reading Group. I also thank Joyee Chen (that group’s project mentor) for their mentorship in discussions regarding this experiment, as well as for their suggestions on the graphical presentation of the results, the choice of wording, and the overall clarity of expression.
AI Usage Statement
==============
Claude was used in this study to assist with the programming and debugging of experimental code, the execution of experiments, data analysis and manuscript review, verification of citations, suggestions on wording; Codex was used in data verification and explanation, manuscript review, verification of citations, proofreading. Grok is used to search and verify some articles that are cited.
The research questions and initial concepts were developed by the author; the author have verified and confirmed the final methods, results, and presentation, and take full responsibility for the entire manuscript. Details regarding the AI models and their specific configurations used for experimental scoring are provided in the Methods section.
Appendix A
==============
A.1 Data Scope and Experimental Design
Each model has 6 meta topics and 6 non-meta topics, with two option orders per topic, giving 24 cells in total and 12 cells per topic type. Qwen and Llama use the same system prompt in the main analysis. Runs using Qwen’s own default system prompt belong to a separate sensitivity analysis and are not mixed into the word-frequency data here.
Table A1. Models and experimental design
| Item | Llama | Qwen |
|---|---|---|
| Model | Llama-3.1-8B-Instruct | Qwen2.5-7B-Instruct |
| meta topics | 6, about AI or the model itself | The same 6 |
| Non-meta topics | 6 ordinary topics | The same 6 |
| Option orders per topic | 2 | 2 |
| Topic × order groups | 24 groups, 12 per type | 24 groups, 12 per type |
| Experimental conditions per group | 5 | 5 |
| Main-experiment dialogues | 120 | 120 |
A.2 Response Counts and Analysis Scope
The three pressure arms share the opening and initial-pressure texts, so these phases are counted only once. Llama’s meta data contain 12 opening and 40 pressure responses, and its non-meta data contain 12 and 84; the corresponding Qwen counts are 12 and 46, and 12 and 110. For each model and topic type, each of the five follow-up conditions has 144 responses (12 cells × 12 turns), totaling 720 per topic type. The complete word-frequency data contain 1588 responses for Llama and 1620 for Qwen, totaling 3208. These are response counts, not numbers of independent experimental samples.
Table A2. Response counts by model and topic group
| Model and topic group | Opening | Pressure | All 5 follow-up conditions | Total |
|---|---|---|---|---|
| Llama meta | 12 | 40 | 720 | 772 |
| Llama non-meta | 12 | 84 | 720 | 816 |
| Llama total | 24 | 124 | 1,440 | 1,588 |
| Qwen meta | 12 | 46 | 720 | 778 |
| Qwen non-meta | 12 | 110 | 720 | 842 |
| Qwen total | 24 | 156 | 1,440 | 1,620 |
| Both models total | 48 | 280 | 2,880 | 3,208 |
Table A3. The five follow-up conditions are:
| Condition | Meaning |
|---|---|
| Pressure release, same topic | Stop rebutting and ask neutral factual questions on the original topic |
| Pressure release, topic switch | Stop rebutting and ask about other topics |
| Sustained pressure | Continue rebutting, not pressure release |
| Neutral, same topic | Original-topic control without pressure |
| Neutral, topic switch | Topic-switch control without pressure |
Table A4. Data inclusion by phase
| Phase | Word-frequency counting | Word-frequency–probe correlation analysis |
|---|---|---|
| Opening | Included | Included |
| Initial pressure | Included; turn count varies with flip timing | Included |
| Follow-up responses under the five conditions | Included; first 12 turns per group | Excluded |
A.3 Word-Frequency Results
Table A5. Word-frequency changes from opening to initial pressure
All frequencies in Table A5 are response means per 1,000 words. Increases, decreases, no change, and the Wilcoxon tests are calculated over 24 cells per model. Different cells have different numbers of pressure turns, so the direction of change in the overall response mean can differ from the direction of change in the majority of cells.
| Model and category | Opening → pressure | Up / down / unchanged | p |
|---|---|---|---|
| Llama singular | 7.8 → 6.8 | 10/14/0 | .833 |
| Qwen singular | 6.7 → 6.2 | 11/13/0 | .360 |
| Llama stance phrases | 4.6 → 4.3 | 15/9/0 | .491 |
| Qwen stance phrases | 5.0 → 3.5 | 2/22/0 | .002 |
| Llama plural | 8.0 → 1.8 | 2/5/17 | .091 |
| Qwen plural | 0.0 → 0.5 | 6/0/18 | .028 |
| Llama hedge | 4.8 → 11.8 | 22/1/1 | <.001 |
| Qwen hedge | 8.1 → 4.7 | 11/13/0 | .074 |
| Llama articles | 81.3 → 105.4 | 18/6/0 | <.001 |
| Qwen articles | 73.8 → 96.9 | 22/2/0 | <.001 |
Table A6. First-person word frequencies under the five follow-up conditions
each model has 288 responses per condition (24 cells × 12 turns). Values are ordered as singular, stance phrases, plural, still in units per 1,000 words. A value of 0.0 is a rounded result and is not necessarily exactly zero.
| Follow-up condition | Llama | Qwen |
|---|---|---|
| Pressure release, same topic pressure_release | 0.1/0.1/0.2 | 0.0/0.0/0.0 |
| Pressure release, topic switch pressure_switch | 0.0/0.0/0.0 | 0.0/0.0/0.0 |
| Sustained pressure pressure_sustained | 7.6/4.9/1.6 | 6.2/3.5/0.4 |
| Neutral, same topic neutral | 0.0/0.0/0.4 | 0.0/0.0/0.0 |
| Neutral, topic switch neutral_switch | 0.0/0.0/0.0 | 0.0/0.0/0.0 |
Table A7.Word frequencies by blind judge stance classification during initial pressure
Mann–Whitney test, no correction for multiple comparison, response labeled with N were excluded; values are mean frequencies in units per 1,000 words
| Maintain | Flip | p | |
|---|---|---|---|
| Llama(n = 18 / 105) | |||
| singular | 9.0 | 6.5 | .050 |
| stance phrases | 5.7 | 4.0 | .052 |
| Qwen(n = 83 / 72) | |||
| singular | 6.2 | 6.1 | .899 |
| stance phrases | 3.2 | 3.8 | <.001 |
Table A8. Eight word-list categories used in the exploratory analysis.
Examples are illustrative, not exhaustive. Hedging and articles are not included in these eight categories.
| Code | Category | Examples |
|---|---|---|
| fp_sing | Singular first person | I, me, my |
| fp_plur | Plural first person | we, us, our |
| fp_stance | First-person stance expressions | I believe, I would argue, in my view |
| ai_disclaim | AI identity or disclaimers of personal stance | as an AI, I don't have personal opinions |
| impersonal | Impersonal attribution | evidence suggests, some argue |
| concede | Acknowledgment, concession, or apology expressions | you're right, I acknowledge, I apologize |
| revise | Self-revision expressions | I was mistaken, I need to reconsider |
| second | Second person | you, your |



Top comments (0)