DEV Community: Jui-Hung Yuan

I tried to find emotion in my own speech using TRIBE v2

Jui-Hung Yuan — Fri, 14 Aug 2026 10:54:06 +0000

The bigger aim is to use TRIBE v2 to improve my speaking. Before that can work,
I need to know whether the model captures anything about emotion, attention or
social connection — or only the sound of the words.

Why TRIBE v2

It maps video, audio and text onto brain activity with state-of-the-art
accuracy, and it trains with modality dropout (p = 0.3), so it still
predicts sensibly when a modality is missing — useful when all I have is audio.

One constraint shaped everything below. The amygdala — the region an emotion
study most wants — sits in subcortex, and the paper reports subcortical accuracy
is "two to three folds" lower than cortical. So the most relevant region is
also the least reliably predicted, and this whole analysis runs on the cortical
surface instead.

Focus question: can I find a brain region whose predicted activity lets me
say, reliably, "this passage is emotional"?

Data

one of my own Toastmasters speeches (7.5 min)
two Toastmasters World Championship finalists — the 2025 winner and the 2017 third place (7.8 and 8.1 min)

Audio only, 16 kHz mono.

Architecture in one paragraph

Three modalities, four frozen encoders: Llama-3.2-3B for text, w2v-BERT-2.0 for audio, and V-JEPA2-giant plus DINOv2-large for video.
Each is compressed to 384 dimensions by a trainable linear layer, concatenated into a 3 × 384 = 1152-dimensional vector
passed through an 8-layer transformer over 100-second windows so the modalities can exchange information
A final linear readout maps those 1152 dimensions onto 20,484 cortical vertices at 1 Hz.

[ Input Video ] ───► [ Visual Track ] ───► DINOv2 + V-JEPA2 ───┐
                                                               │ ──► [ Unified Transformer ] ──► [ Cortical fMRI Prediction ]
                     [ Audio Track ]  ───► Wav2Vec-BERT     ───┤
                     [ Text Track ]   ───► LLaMA 3.2        ───┘

Method — two angles

Step 0 was standing up inference on an AWS g6.xlarge (one L4, 24 GB) —
enough for all backbones, since features are cached per modality so only one
encoder is resident at a time.

First approach — exploratory. Look at the parcels the literature ties to
social and emotional inference — temporo-parietal junction (PGi) and middle
temporal gyrus (TE1a) — and check whether they rise during passages I know are
emotional.

Second approach — statistical. Hand-label every speech by ear on two
independent dimensions, affect (emotional / neutral) and discourse (narrative /
argument). Compute loudness, pitch variability and speech rate per second. Fit
all of them together against each of the 360 Glasser parcels, and test each
contrast by shuffling which segments carry which label, 10,000 times.

Labels are just Audacity label-track exports — start, end, label:

15.069025   28.128847   neutral
29.803183   46.881412   neutral
56.257694   73.001056   emotional
86.060878   97.781231   neutral
125.575211  147.006713  emotional

Results

The exploratory pass died early

TPJ and MTG turn out to track language regions at r = 0.80–0.90 — their
apparent prominence is inherited, not independent. Nothing cleanly marked jokes
landing, arguments convincing, or emotional build-ups.

Is that bad? Not for the model — TPJ genuinely is part of the language network,
so this may be biologically correct. It is bad for my question: if TPJ is
80–90% predictable from language cortex, it cannot serve as independent evidence
that something social or emotional happened.

Twelve region groups over 450 seconds of my own speech. Red is above this
clip's average, blue below. The auditory and language rows move; the emotion
rows (interoceptive insula, vmPFC) sit nearly flat throughout.

Emotion: zero parcels

Once loudness, pitch and speech rate were accounted for, no parcel showed a
reliable emotional-vs-neutral difference in any of the three speeches.

More telling is which predictors actually moved the predictions:

predictor	mean \|β\| across 360 parcels
speech rate	0.036
loudness	0.034
emotion label	0.021

Two things I can measure from the waveform in ten lines of numpy move the
predicted brain activity more than what I heard as emotional content.

Separately, a PCA across region groups showed a single component explaining
55–59% of the variance — twelve "networks" with roughly three degrees of
freedom between them. Regions move together far more than they move apart.

Discourse: 42 parcels, and why I don't believe them

Narrative-vs-argument looked more promising — 42 parcels survived in two of the
three speeches. Then I looked at which ones: primary motor, premotor,
somatosensory, superior parietal, visual cortex. Nothing to do with discourse.

The reason showed up in a whole-brain check. In those two speeches the entire
cortex shifted between narrative and argument blocks:

speech	global shift (narr − argu)	% of all 360 parcels negative	"significant" parcels
2025 champion	−0.044	79%	91
2017 third place	−0.079	86%	114
my speech	+0.031	41%	0

The two competition speeches show a brain-wide offset; mine doesn't. And the
"significant" parcels appear exactly where the offset appears. They aren't
regions that respond to storytelling — they're simply the quietest parcels,
where a brain-wide shift was big enough to clear the noise bar first. My own
speech, with no such shift, produced nothing at all, which is what exposed the
artifact.

I tested and rejected the obvious causes: slow drift over the clip (r = −0.04
and −0.19 against time), loudness (r = +0.09, +0.03) and speech rate
(r = −0.08, −0.17). Where the offset comes from is still open.

Conclusion

I could not find an emotion-triggered signal in TRIBE v2's predicted brain
activity. Possible reasons, most defensible first:

Not enough data. I planted effects of known size into the real predictions and raised them until they cleared the significance bar. That took an effect about 1.3× a parcel's own natural fluctuation — larger than everything the region normally does on its own. Three speeches cannot reach that, so this is a null from lack of power, not evidence of absence.
Emotion may simply be subtler in predicted activity than auditory and linguistic responses, which dominate everything measured here.
The model predicts a normative response. It has no listener state — the same audio always produces the same prediction — while real emotional reactions vary enormously between people.
Time resolution is poor. The haemodynamic response is slow and stays elevated after a peak, so sharp emotional moments smear into their surroundings. There is one sample per second but only a few seconds of real temporal precision.
My labels may be too coarse. "Emotional" spans humour, pathos and urgency, which plausibly engage different systems and may cancel when pooled into one condition.

The thing I'd actually tell someone else

The most useful thing I learned wasn't a finding. It was this:

If the whole brain moves a little, a statistical test will hand you a short
list of regions and call them significant.

In one speech that list was 114 regions long. Every one passed a permutation
test. They looked like a result — until I checked the other 246 parcels and
found they had all shifted the same way. The 114 weren't the regions that
responded; they were just the quietest ones, where a small brain-wide nudge
was easiest to detect. The test was working correctly. It was answering "is this
region different?" when the honest question was "is this region different from
the rest of the brain?"

So: when a result appears, look at what didn't make the list too. If
everything moved, nothing did.

Event-Driven AI Orchestration: Local-First Smart Home Automation

Jui-Hung Yuan — Tue, 10 Mar 2026 14:25:07 +0000

In my previous blog, I built a remote MCP server connected to Claude so I could control my smart light bulb by just sending a chat message. It worked! But looking back, the architecture was a glorious overkill.

All of this, for two people who just want to dim the lights in the evening 🫠..

Around the same time, I had been hearing a lot about the trend of “Skills over MCP” and the hype around OpenClaw. OpenClaw is popular because it’s a local-first AI assistant — the whole orchestration runs on your own devices, no cloud infrastructure needed, just enough flexibility to build something genuinely useful. That sounded like exactly the right learning opportunity.

So I decided to build an OpenClaw-inspired personal assistant for smarthome control — simpler, local, and tailored to just what I needed.

Spoiler⚠️: it worked out well. My partner and I can now control the light bulb from Slack, and we can schedule light adjustments via chat. I am obsessed with healthy lifestyle. So naturally, I scheduled the light to slowly dim itself over the evening. It’s very peaceful. The only unintended side effect is that when we have guests over, the room gets darker and darker as the night goes on, and somehow they always end up leaving earlier than planned. I feel a little bad about it. But also — they’re getting more sleep now, so really, I’m doing them a favor.

OpenClaw basics

To understand how OpenClaw works, I reimplemented its 4 core components with the help of Claude — purely for learning. No shortcuts, no copy-paste. Just building each piece by hand until it clicked.

Beauty is in the eye of the beholder — or as the teacher in Spy x Family would say, 🕺ELEGANTO🕺. In the following sections, I'll share what I learned from each component, the key insights that surprised me, and what I find elegant about the design.

1. Memory system

OpenClaw keeps the assistant’s memory as simple Markdown files on disk. The assistant’s personality and tone live in SOUL.md, user preferences in USER.md, and notable conversations get appended to a daily log. During a conversation, the agent can selectively write to these files, and the system prompt guides which information goes where. It’s not over-engineered — just enough structure to make memory feel real and 🕺ELEGANTO🕺.

🤓 In my implementation, this is exposed via two tools: memory_search (hybrid keyword + vector search over past logs) and memory_write (append or overwrite a memory file).

2. Skill registry

Skills give the agent new capabilities in a Markdown-based format — like a cheatsheet it can pull up on demand. What I find 🕺ELEGANTO🕺 is the progressive disclosure mechanism. At startup, the agent only sees a one-line summary of each available skill. When it needs to use one, it calls describe_skill to load the full cheatsheet — actions, parameters, examples — and that doc gets injected into the system prompt for the rest of the session. This keeps the context lean until it’s actually needed, and the same pattern naturally extends to revealing new tools or even new skills over time.

🤓 In my implementation, each skill is a folder under skills/ with a SKILL.md file (frontmatter = summary, body = full docs) and a Python script that exposes an execute(action, params) function. The agent calls describe_skill first, then execute_skill. My light bulb skill lives in src/smarthome/agent/skills/light-control.

3. Scheduled jobs (CRON and Heartbeat)

Beyond just responding to messages, OpenClaw can also proactively run tasks on a schedule. CRON-style jobs fire at specific times, while the Heartbeat is better suited for background monitoring — it batches tasks together so they can share context and inform each other’s results.

🤓 My implementation is closer to CRON: the scheduler wakes up on a regular interval, checks if any task’s scheduled time falls within the elapsed window, and fires it directly — no LLM call involved. That works 🕺ELEGANTO🕺 enough for simple device commands like dimming the light at 9pm. If I wanted smarter tasks (e.g. checking my calendar and sending a notification), it would need to trigger an actual agent turn. Tasks are managed via the schedule_task tool (add/remove/list) and persist in SCHEDULE.md across restarts.

4. Channel Adapter

OpenClaw supports many messaging apps — WhatsApp, Telegram, Slack, Discord, and more. The channel adapter is the translator between a specific app and the agent: it normalizes incoming messages into a unified format, and formats the agent’s response back into whatever the app expects. It also acts as the authentication boundary — rather than managing per-user credentials like MCP does, OpenClaw’s approach is perimeter-based: “as long as you can reach me, I trust you.”

🤓 I only implemented Slack, which supports both direct messages and @mentions in channels. One part I find particularly 🕺ELEGANTO🕺 is the color picker: when the user wants to change the bulb color, a show_palette action returns a Slack dropdown block. The agent loop intercepts this block and passes it directly to the Slack adapter, which renders it as an interactive dropdown — no extra roundtrip needed. The user picks a color, and it triggers set_color directly.

Key learning

After building this, my biggest takeaway is that all these components — memory, skills, scheduling — work as well as they do because of how good modern LLMs are at tool calling. Once you wrap the right functionality as tools, the agent just... uses them. Memory feels persistent, skills feel modular, scheduling feels effortless. Especially the skill system: because you can add a new folder with a Markdown file and a Python script and the agent picks it up automatically — that expandability is truly 🕺ELEGANTOOOO🕺.

One more thing I want to share: I used Claude Code throughout this build, and it went better when I slowed down at the start. My first instinct was to clone the OpenClaw repo, dump it into context, and ask Claude to plan everything at once. That didn’t go well — I didn’t know enough about OpenClaw myself to tell if the plan made sense, and Claude can’t read your mind about what trade-offs matter to you. What actually helped was spending time upfront to clarify scope and intent together, before handing off any implementation. The clearer the brief, the better the output. That’s not specific to Claude — it’s just good collaboration.

Quick start for everyone

# Clone the repo
git clone https://github.com/jui-hung-yuan/smarthome-mcp-lab
cd smarthome-mcp-lab

# Add your Anthropic API key
 mkdir -p ~/.smarthome
 echo 'ANTHROPIC_API_KEY=sk-...' >> ~/.smarthome/.env

# Seed memory files (optional but recommended)
 mkdir -p ~/.smarthome/memory
 echo "# Memory" > ~/.smarthome/memory/MEMORY.md
 echo "# User Preferences" > ~/.smarthome/memory/USER.md

# Run with mock bulb (no hardware needed)
uv run python -m smarthome.agent --mock

# Run with real bulb — add `TAPO_USERNAME`, `TAPO_PASSWORD`, `TAPO_IP_ADDRESS` to `~/.smarthome/.env`, then:
uv run python -m smarthome.agent

# Run as a Slack bot — add `SLACK_BOT_TOKEN`, `SLACK_APP_TOKEN`, `SLACK_SIGNING_SECRET` to `~/.smarthome/.env`, then:
uv run python -m smarthome.agent --slack --mock   # mock bulb
uv run python -m smarthome.agent --slack          # real bulb

Thanks for reading! 🙏 This was an 🕺ELEGANTO🕺 project to build and to write about. If you've built something similar, or have thoughts on the architecture, I'd love to hear from you — drop a comment below or open an issue on the repo. Always happy to exchange ideas.

From Local to Cloud: What I Learned Building a Remote MCP Server on AWS for Smart Home Control

Jui-Hung Yuan — Fri, 20 Feb 2026 14:21:59 +0000

I wanted to tell Claude to turn off my bedroom light. Not just from my laptop at home — but from anywhere.

What started as "let's try MCP" became "why does OAuth keep failing" and "how do cloud services reach devices behind my router".

This post walks through the architecture decisions and the stuff that tripped me up. Not every choice was obvious, and some things only made sense after I'd already built them wrong once.

The full code is here.

Why I Built This

I recently attended a session organized by the Berlin AWS User Group about Amazon Bedrock AgentCore (shoutout to the organizers — the session was really helpful!!). I'd been using MCP for a while but never built one myself. When I looked around, most posts covered the concept and building local MCP servers — not much about deploying remote MCP servers to the cloud or what actually trips you up when you build one. So I picked a concrete use case: control my TAPO smart light bulb via Claude, and build the whole thing on AWS.

The goal was simple and deliberately small. One bulb. A handful of tools. But with enough real infrastructure to actually learn from.

Phase 1: Local Prototype with FastMCP

I started with FastMCP and Claude Desktop. Getting a working local MCP server took one afternoon.

The developer experience is genuinely impressive. You define a tool like this:

@app.tool()
async def turn_on() -> str:
    """Turn on the TAPO smart light bulb."""
    bulb = await get_bulb()
    await bulb.turn_on()
    return "Light turned on"

That's it. FastMCP reads your function and auto-generates everything Claude needs — the tool name, description, and input schema. You write the function. FastMCP handles the rest.

Claude Desktop runs the FastMCP server as a local subprocess. The full path is:

It worked. I could chat with Claude and control my light. But then I left home.

The Turning Point

Claude Desktop's MCP only works with local subprocesses. The moment you close your laptop or step outside, it's gone. I wanted the Claude web app to work too — partly because it's more convenient, partly because building the remote version is where the real learning happens.

That's a fundamentally different problem.

Claude web app needs a publicly accessible MCP server with proper authentication.
MCP server somehow needs to reach a device on your local network.

This is where AWS comes in.

The Architecture

Each component has a specific job:

Cognito: Handles user login.
AgentCore Gateway: The MCP server.
Lambda: Serverless handler (No always-on server to manage).
IoT Core: Cloud-to-device message broker.
Local Bridge: Runs at home and controls the bulb.

Key Lesson 1: Why AWS IoT Core (and not just Lambda → HTTP)?

AWS IoT Core is a managed cloud service that acts as a message broker between cloud services and physical devices. It uses MQTT — a lightweight protocol designed for devices with unreliable connections — to route messages through a publish/subscribe model.

Think of it like a radio station: Lambda broadcasts on a channel, and any device tuned to that channel receives the message.

The obvious question: why not have Lambda call the local bridge directly over HTTP?

1. NAT Traversal — Lambda can't reach your home network

Lambda lives in AWS. Your home bridge lives behind your router. Your router blocks all inbound connections — Lambda has no address to call. To make direct HTTP work, you'd need a static IP and port forwarding (security risk), a VPN tunnel (operational overhead), or a reverse tunnel like ngrok (fragile, costs money).

IoT Core flips the direction. Your local bridge reaches out to AWS and holds a persistent MQTT connection open. Lambda publishes to a topic, IoT Core delivers it over that already-open connection. Your home network never needs a public address.

2. Cost and efficiency — MQTT is event-driven, HTTP requires polling

With HTTP, if Claude wants to know the current brightness, Lambda would need to make a request every time — or poll regularly to keep state fresh. That's an HTTP call (and cost) for every status check.

With MQTT, the bridge reports state changes to IoT Core's Device Shadow automatically. Claude asks for brightness? The Shadow answers instantly from cache. No new request needed. The bridge only sends updates when something actually changes.

3. Resilience — MQTT handles disconnections automatically

HTTP assumes both sides are reliably reachable. If your bridge restarts or your internet hiccups during a Lambda call, the request just fails.

MQTT is designed for unreliable connections. If your bridge goes offline, IoT Core queues messages. When it reconnects, pending commands are delivered automatically. No retry logic to write, no state to track manually.

Key Lesson 2: FastMCP vs. AgentCore Gateway

Amazon Bedrock AgentCore Gateway is a fully managed service that turns your backend functions into an MCP-compliant server that AI clients like Claude can talk to. It handles OAuth authentication, protocol translation, and tool discovery — so you only write business logic.

Think of it as the bouncer and translator standing between Claude and your Lambda: it checks credentials, speaks MCP fluently, and routes the right instructions inward.

Here's how it compares to FastMCP as an MCP hosting approach:

Aspect	Winner	Why
Tool Schema	Tie🤝	FastMCP auto-generates from decorators (better DX). AgentCore requires explicit JSON (~70 lines for 4 tools), but makes the contract reviewable.
Hosting	AgentCore🏆	FastMCP needs a persistent runtime (container, EC2, Fargate) — costs money even when idle. AgentCore is serverless — Lambda only runs when invoked.
Auth Support	Tie🤝	Both handle OAuth well now (FastMCP 2.11+ added JWT, OAuth proxy, full OAuth server).

Key Lesson 3: OAuth — The Part That Actually Took Time

If I'm honest, the infrastructure was the easy part. Authentication is where I spent most of my debugging time.

The problem started with how I set up my Cognito configuration. It defaulted to the client_credentials flow — machine-to-machine (M2M) auth where a service exchanges a client ID and secret directly for a token. No login page, no user interaction.

That works fine for service-to-service communication. But the Claude web app is a browser-based client. It needs to redirect the user to a login page, have them authenticate, and receive an authorization code back — the authorization_code flow. These are fundamentally different OAuth patterns. It took me a while to figure out I was using the wrong flow entirely.

Again think of it like a nightclub with a strict bouncer. The bouncer (AgentCore Gateway) doesn't know you personally, but trusts the ID checker down the street (Cognito). You walk down to Cognito, prove who you are, and Cognito gives you a wristband. You bring that wristband back to the bouncer at /auth_callback, and now you're in.

That callback address is the key. The authorization_code flow exists precisely because Claude (the browser client) needs a human to authenticate interactively. The code is the shop's way of receiving confirmation from Cognito without the user handing over their password directly.

Once I understood that distinction, I knew what to fix: create a separate Cognito app client configured for authorization_code flow with the correct callback URL.

There was a second, subtler issue. When the AgentCore Gateway's resource metadata (/.well-known/oauth-protected-resource) doesn't specify a scope, Claude falls back to Cognito's OIDC discovery endpoint (/.well-known/openid-configuration), which advertises standard scopes: openid, email, phone, profile. But my Cognito app client only allowed openid, smarthome-gateway/read, and smarthome-gateway/write. Claude requesting email and phone caused an invalid_scope error. The fix: explicitly configure the allowed scopes on the client to match exactly what Claude will request.

Neither of these issues had anything to do with MCP itself. They were pure OAuth configuration problems. But you can only diagnose them if you understand the handshake well enough to know which step is failing.

Real-World Performance

Once everything was wired up, I measured actual round-trip times from Claude's perspective:

Tool	Round-trip
`get_status`	1,131 ms
`turn_on`	3,367 ms

get_status only hits the IoT Device Shadow — no trip to the physical bulb.
turn_on goes the full path: Lambda → IoT Core → bridge → bulb → confirmation back. Three seconds is noticeable but acceptable for a chat experience with the light switch. For anything latency-sensitive, you'd want to think harder about this.

The monthly cost for 1,000 tool calls across all services: $0.07.

What's Next

A few things I want to clean up:

Terraform for infrastructure provisioning. Right now I have five boto3 scripts that need to be run in a specific order. It works, but it's tedious. Terraform would make this reproducible and shareable.
Dockerize the local bridge to run on a Raspberry Pi or similar edge device, so it doesn't depend on a laptop being on.
More device types. The architecture already supports it — BaseDevice is an abstract interface, and the DeviceRegistry manages multiple devices. Adding a smart plug or thermostat is mostly a new implementation, not a new architecture.

Closing Thoughts

I started this project to understand MCP better. I ended up learning more about AWS IoT, OAuth flows, and serverless architecture than I expected. That's usually the sign of a good learning project — the stated goal was an excuse to dig into something real.

A few things I'd tell someone starting this from scratch:

Get the local FastMCP version working first. It takes an afternoon and gives you immediate feedback. Only then add the AWS layer.
The OAuth debugging will take longer than the infrastructure. Learn the authorization_code vs client_credentials distinction before you start configuring Cognito — it'll save you hours.
AgentCore Gateway is genuinely easy to set up compared to what I expected. The tool schema verbosity is real, but it's a one-time cost.
IoT Core is the right tool for this specific problem. The NAT traversal alone justifies it.

If you're building something similar or have questions about any of the architectural decisions, I'd love to hear from you in the comments. And if you spot something I got wrong — even better.

→ Full code on GitHub