The bigger aim is to use TRIBE v2 to improve my speaking. Before that can work,
I need to know whether the model captures anything about emotion, attention or
social connection — or only the sound of the words.
Why TRIBE v2
It maps video, audio and text onto brain activity with state-of-the-art
accuracy, and it trains with modality dropout (p = 0.3), so it still
predicts sensibly when a modality is missing — useful when all I have is audio.
One constraint shaped everything below. The amygdala — the region an emotion
study most wants — sits in subcortex, and the paper reports subcortical accuracy
is "two to three folds" lower than cortical. So the most relevant region is
also the least reliably predicted, and this whole analysis runs on the cortical
surface instead.
Focus question: can I find a brain region whose predicted activity lets me
say, reliably, "this passage is emotional"?
Data
- one of my own Toastmasters speeches (7.5 min)
- two Toastmasters World Championship finalists — the 2025 winner and the 2017 third place (7.8 and 8.1 min)
Audio only, 16 kHz mono.
Architecture in one paragraph
- Three modalities, four frozen encoders: Llama-3.2-3B for text, w2v-BERT-2.0 for audio, and V-JEPA2-giant plus DINOv2-large for video.
- Each is compressed to 384 dimensions by a trainable linear layer, concatenated into a 3 × 384 = 1152-dimensional vector
- passed through an 8-layer transformer over 100-second windows so the modalities can exchange information
- A final linear readout maps those 1152 dimensions onto 20,484 cortical vertices at 1 Hz.
[ Input Video ] ───► [ Visual Track ] ───► DINOv2 + V-JEPA2 ───┐
│ ──► [ Unified Transformer ] ──► [ Cortical fMRI Prediction ]
[ Audio Track ] ───► Wav2Vec-BERT ───┤
[ Text Track ] ───► LLaMA 3.2 ───┘
Method — two angles
Step 0 was standing up inference on an AWS g6.xlarge (one L4, 24 GB) —
enough for all backbones, since features are cached per modality so only one
encoder is resident at a time.
First approach — exploratory. Look at the parcels the literature ties to
social and emotional inference — temporo-parietal junction (PGi) and middle
temporal gyrus (TE1a) — and check whether they rise during passages I know are
emotional.
Second approach — statistical. Hand-label every speech by ear on two
independent dimensions, affect (emotional / neutral) and discourse (narrative /
argument). Compute loudness, pitch variability and speech rate per second. Fit
all of them together against each of the 360 Glasser parcels, and test each
contrast by shuffling which segments carry which label, 10,000 times.
Labels are just Audacity label-track exports — start, end, label:
15.069025 28.128847 neutral
29.803183 46.881412 neutral
56.257694 73.001056 emotional
86.060878 97.781231 neutral
125.575211 147.006713 emotional
Results
The exploratory pass died early
TPJ and MTG turn out to track language regions at r = 0.80–0.90 — their
apparent prominence is inherited, not independent. Nothing cleanly marked jokes
landing, arguments convincing, or emotional build-ups.
Is that bad? Not for the model — TPJ genuinely is part of the language network,
so this may be biologically correct. It is bad for my question: if TPJ is
80–90% predictable from language cortex, it cannot serve as independent evidence
that something social or emotional happened.
Twelve region groups over 450 seconds of my own speech. Red is above this
clip's average, blue below. The auditory and language rows move; the emotion
rows (interoceptive insula, vmPFC) sit nearly flat throughout.
Emotion: zero parcels
Once loudness, pitch and speech rate were accounted for, no parcel showed a
reliable emotional-vs-neutral difference in any of the three speeches.
More telling is which predictors actually moved the predictions:
| predictor | mean |β| across 360 parcels |
|---|---|
| speech rate | 0.036 |
| loudness | 0.034 |
| emotion label | 0.021 |
Two things I can measure from the waveform in ten lines of numpy move the
predicted brain activity more than what I heard as emotional content.
Separately, a PCA across region groups showed a single component explaining
55–59% of the variance — twelve "networks" with roughly three degrees of
freedom between them. Regions move together far more than they move apart.
Discourse: 42 parcels, and why I don't believe them
Narrative-vs-argument looked more promising — 42 parcels survived in two of the
three speeches. Then I looked at which ones: primary motor, premotor,
somatosensory, superior parietal, visual cortex. Nothing to do with discourse.
The reason showed up in a whole-brain check. In those two speeches the entire
cortex shifted between narrative and argument blocks:
| speech | global shift (narr − argu) | % of all 360 parcels negative | "significant" parcels |
|---|---|---|---|
| 2025 champion | −0.044 | 79% | 91 |
| 2017 third place | −0.079 | 86% | 114 |
| my speech | +0.031 | 41% | 0 |
The two competition speeches show a brain-wide offset; mine doesn't. And the
"significant" parcels appear exactly where the offset appears. They aren't
regions that respond to storytelling — they're simply the quietest parcels,
where a brain-wide shift was big enough to clear the noise bar first. My own
speech, with no such shift, produced nothing at all, which is what exposed the
artifact.
I tested and rejected the obvious causes: slow drift over the clip (r = −0.04
and −0.19 against time), loudness (r = +0.09, +0.03) and speech rate
(r = −0.08, −0.17). Where the offset comes from is still open.
Conclusion
I could not find an emotion-triggered signal in TRIBE v2's predicted brain
activity. Possible reasons, most defensible first:
- Not enough data. I planted effects of known size into the real predictions and raised them until they cleared the significance bar. That took an effect about 1.3× a parcel's own natural fluctuation — larger than everything the region normally does on its own. Three speeches cannot reach that, so this is a null from lack of power, not evidence of absence.
- Emotion may simply be subtler in predicted activity than auditory and linguistic responses, which dominate everything measured here.
- The model predicts a normative response. It has no listener state — the same audio always produces the same prediction — while real emotional reactions vary enormously between people.
- Time resolution is poor. The haemodynamic response is slow and stays elevated after a peak, so sharp emotional moments smear into their surroundings. There is one sample per second but only a few seconds of real temporal precision.
- My labels may be too coarse. "Emotional" spans humour, pathos and urgency, which plausibly engage different systems and may cancel when pooled into one condition.
The thing I'd actually tell someone else
The most useful thing I learned wasn't a finding. It was this:
If the whole brain moves a little, a statistical test will hand you a short
list of regions and call them significant.
In one speech that list was 114 regions long. Every one passed a permutation
test. They looked like a result — until I checked the other 246 parcels and
found they had all shifted the same way. The 114 weren't the regions that
responded; they were just the quietest ones, where a small brain-wide nudge
was easiest to detect. The test was working correctly. It was answering "is this
region different?" when the honest question was "is this region different from
the rest of the brain?"
So: when a result appears, look at what didn't make the list too. If
everything moved, nothing did.
Top comments (0)