DEV Community: Matt Hamilton

From a half-finished training run to a reproducible eval

Matt Hamilton — Wed, 12 Aug 2026 16:17:11 +0000

So our last training pass on Qwen3-Omni-30B-A3B-Instruct — fine-tuning it on Barbados newspapers — ran out of time halfway through. Twelve hours of GPU budget, planned for 1624 training steps, finished at step 1000. Loss was still falling. The full checkpoint, optimiser state, scheduler — everything you need to keep going — was sitting safely in a private cloud bucket. So we restarted it. Then the interesting bit happened.

The headline finding is the +5 percentage points on the 60-probe text knowledge eval (reproducible to 0pp on the same stack). The bigger finding is the radio audio number, which roughly doubled on completion. Both point to the same conclusion: completing the training schedule to step 1624 mattered.

Context: Future Caribbean and Pulse

This work is part of Future Caribbean, a regional initiative that runs a global agentic AI buildathon over a 21-day sprint each summer. The framing is "build deployable AI systems that strengthen how economies coordinate" — ten tracks, $70K+ in prizes, and a final showcase at the NYSE. We're one of the selected teams.

The programme advertised NVIDIA H200-class compute for selected teams, and we planned this training work on that basis. In practice, compute was not made available to our team during the build window. Rather than drop the model work, I rented the H200 capacity personally through Hugging Face Jobs. That is a decision worth flagging: access to compute materially shapes what small teams can attempt in a three-week buildathon, and going it alone means we could only train the model once, on the budget I could afford.

Our build is Pulse — a live public-signal intelligence system for Barbados that continuously ingests fragmented social, broadcast, and news signals, resolves them into a trusted graph of sources, places, events, people, and communities, and exposes that intelligence through agentic workflows. The Barbados LoRA is one of the domain-knowledge components feeding into Pulse's natural language layer.

The model itself isn't Pulse-specific. It's a standard text-only LoRA on a standard base model, trained on a Barbados newspaper archive. The source corpus is private, but the adapter is public. Anyone working on Caribbean-domain text — search, summarisation, knowledge probes, a personal project on Bajan history — can download and use the V4 artefacts directly. You don't need to be running Pulse to benefit from the training, and Pulse isn't the only reason the model exists.

The setup, briefly

Domain-adaptive pretraining (DAPT) is the simplest "teach a model something new" technique you can run: take a model that's already been trained on the open web, and continue its next-token prediction on a corpus from your target domain. No labels, no chat format, no reward model. Just keep showing it text and let the loss fall. Gururangan and colleagues at ACL 2020 called this Don't Stop Pretraining — the follow-up paper that established the technique.

In our case the corpus is the cleaned Barbados newspaper archive, the base model is a 30-billion-parameter multimodal MoE, and the fine-tune is a small low-rank adapter (LoRA) on the text-only "thinker" half of the model. Training runs on a rented H200 GPU inside a Hugging Face Job. This post is about what happened when we tried to finish the schedule and what re-scoring the older artefacts taught us.

The mistake that shaped the recipe

The first real training run only touched the Thinker's attention projections — the q, k, v, and o projection layers around each attention head. That made 53.5 million parameters trainable: 0.17% of the 31.8-billion-parameter base model. The reasoning at the time was that attention is where the model decides which knowledge to retrieve, and retargeting that was the cheapest plausible intervention. We called it V2.

The mistake was thinking that was enough. On the GPU paired path (LoRA loaded at runtime into a bf16 base), V2 helped. On the 4-bit fused path that Pulse actually runs — the LoRA merged into the weights, then quantised — V2 regressed. 55% overall, against 61.7% for the unfine-tuned 4-bit base. Fusing an attention-only LoRA into a 4-bit snapshot loses more fidelity than the same adapter applied at runtime. The base model still knew things the fused V2 didn't.

Imagine the model as a library. Attention is the librarian deciding which shelves and passages matter for the question. The MLP/MoE pathway is more like the subject specialists who read that material, transform it, and connect it to other facts. Retraining the librarian — V2 — changed where the model looked. It did not change the specialists' working knowledge. On a paired eval the librarian can be enough: the specialists still know what they knew. Fused into 4-bit weights, the librarian's retraining doesn't survive the round-trip cleanly, and the specialists never moved at all.

The later recipe kept the attention targets and added LoRA parameters to the Thinker's MLP/MoE pathway (gate_up_proj and down_proj). That expanded the trainable surface to about 2.59 billion parameters, or 7.54% of the base model. In library terms, we could now retrain both the librarian and the subject specialists. V3 stopped partway through that broader schedule. V4 is the completed version. The +15pp jump from V2 (55%) to V3 (70%) is the recipe change doing real work — not the schedule completing.

Picking up where we left off

Resuming a Hugging Face Job is conceptually simple: launch a new job that points at the same run identifier. The bucket-and-sidecar architecture (the worker continuously uploads checkpoint snapshots to a private bucket as it trains) handles the bookkeeping — when a fresh training job starts, it asks the bucket "what's the latest complete snapshot for this run?" and picks up from there.

In practice there were two things to be careful about. First, the resume check enforces recipe identity strictly — it compares a hash of the training config (base model, dataset, LoRA shape, library versions, the image digest of the container) against the snapshot it's about to restore. So we couldn't use the rebuilt container image that fixes an upload bug elsewhere in the pipeline; the manifest wouldn't match, the resume would silently skip the bucket, and we'd start a brand new run instead. We used the original container and dealt with the upload bug the same way we did for V3: don't trust the in-process adapter upload, publish the final adapter in a separate recovery job.

Second, that recovery job hardcoded the destination repository name in the supervisor code, pointing at V3. There was no flag to override it. Three-line code change to fix, but the cleanest workaround without rebuilding the container was an inline Python script that calls the same upload primitives with a different target — published …-LoRA-v4 in 61 seconds. Then a single finalise job produced the fused bf16 and the 4-bit MLX snapshots (the latter is what you actually run on a Mac). Both published, both verified — the provenance file correctly lists adapter: …-LoRA-v4 rather than the V3 adapter a publishing bug had caught in an earlier round.

The full resume, including one sidecar-marker hiccup that took a two-minute manual fix, is recorded in docs/19 §10.2.

The result

Three runs on the same 60-probe Barbados knowledge set — twenty well-known local facts (parliament year, Crop Over revival, parish count), twenty niche ones (Barrow constituency, Leadpipe Glitch, Grantley Adams Secondary), and twenty general-knowledge controls (longest European river, deepest trench, Marie Curie). The model picks one of four candidate completions for each probe by ranking them on mean log-probability over the completion tokens. Right or wrong.

Run	Recipe	Schedule	Overall
V2	attention only (`q/k/v/o_proj`)	500 / 801 steps · 53.5M params	55.0%
V3	attention + MLP/MoE	1000 / 1624 steps · 2.59B params	70.0%
V4	attention + MLP/MoE (completed)	1624 / 1624 steps · 2.59B params	75.0%
base	unfine-tuned 4-bit	—	61.7%

The V2 number is the most important one in the table. The attention-only recipe helped on a GPU paired eval (LoRA loaded at runtime into a bf16 base) — a result from earlier in the project that I wrote up at the time. But on the 4-bit fused path that Pulse actually runs, V2 scored worse than base: 55% against 61.7%. Fusing an attention-only LoRA into a 4-bit snapshot loses more fidelity than fusing one that also has MLP/MoE parameters. V3's recipe change — adding LoRA to gate_up_proj and down_proj — is what brought the fused 4-bit path back to parity. The +15pp jump from V2 (55%) to V3 (70%) is the recipe change doing real work, not the schedule completing.

V4 then completes the schedule V3 started, lifting another 5pp on top.

The reproducibility question was whether +5pp on the V3 → V4 step was real or evaluator drift. The eval script's hardcoded model-repo field pointed at V3, the MLX backend is sensitive to small numerical changes between versions, and 5pp on 60 probes is roughly three probes — within sample noise. The honest test was to re-score the earlier artefacts on the same stack — V3's fused 4-bit snapshot and the unfine-tuned base, both downloaded from the Hub.

Track	base	V3	V4
local	55.0%	65.0%	70.0%
rare_local	40.0%	55.0%	60.0%
control	90.0%	90.0%	95.0%
overall	61.7%	70.0%	75.0%

All three reproduce to 0pp of their originally-recorded numbers. The eval path is stable. The +5pp V3 → V4 is real weight movement. The cumulative DAPT contribution (base → V4) is +13.3pp overall: +3 local, +4 rare_local, +1 control.

The interesting bit

So the +5pp is real, but the mechanism is interesting. On the 34 probes that both the base model and V4 got right, V4's mean margin (the gap between the top-ranked correct completion and the runner-up) is 0.56 log-prob *smaller* than the base model's. V4 is less confident on the easy probes it shares with the base model, yet wins more probes overall.

This is the signature of a model that's been retargeted toward a domain. The output layer has shifted to put more weight on Barbados-relevant tokens, which compresses the log-probability gap on general-completion probes where those tokens weren't relevant, while flipping close calls on the domain probes where they are. Same total budget, redistributed. The accuracy gain comes from breaking ties on the borderline probes, not from being more confident on what's already easy.

The +1 control-probe gain (a paraphrasing probe the base and V3 both missed) is also real and slightly unexpected. V4 picks up some paraphrasing robustness from incidental general prose in the corpus (sports, world news, op-eds). One probe on N=20 is suggestive rather than load-bearing, but it does mean V4 is genuinely a better model, not just a better Barbados model.

Does it survive contact with actual audio?

The 60-probe eval is text-only — it asks whether the model knows facts about Barbados. Pulse's actual workload is audio: five-minute FM radio windows and short-form TikToks that need transcription and structured extraction (events, venues, prices, promotions). Two more evals cover that path.

Radio — 17 five-minute windows from real Barbados FM (HOTT 95.3, VOB 92.9, CBC 94.7, Life 97.5, Q 100.7, The Beat 104.1, The One 98.1, Y 103.3, Radio Bimshire 106.1, SLAM 101.1). Scored on quality, hallucination rate, proper-noun precision/recall, event false-positive and false-negative rates, and whether evidence quotes survive.

Model	Windows	Avg quality	Avg latency
base	17	0.135	9.2s
V3	17	0.140	15.5s
V4	17	0.230	17.8s

V3 was essentially flat against base on radio. Completing the LR schedule to step 1624 nearly doubled the audio extraction quality (0.140 → 0.230). The place where DAPT shows up most clearly is the proper-noun layer: "Rise Together" instead of "Rice Together", "Leadpipe and Sadis" instead of plumbing+typo. These errors are unrecoverable once cheap Whisper has destroyed them, so the audio-tower-bearing model is the only place to fix them.

TikTok — 10 Barbados TikToks (golf promotions, catamaran cruises, cave tours, farmers' markets, flower forests, rug-making workshops). Each fixture has hand-written ground truth for the observations the pipeline should extract.

Model	Precision	Recall	F1	Field recall
base	0.26	0.36	0.30	0.42
V3	0.60	0.55	0.57	0.38
V4	0.34	0.55	0.42	0.52

V4 and V3 are tied on recall (0.55) — both find the right observations. They diverge on what they do around the edges. V3 is more conservative: higher precision (fewer false positives), lower field recall (less detail per observation). V4 emits more candidate entities per fixture (35 vs V3's 20 across all 10) and fills more fields correctly per observation, but at the cost of more false positives. That's the same calibration shift showing up in a different modality: V4's unembedding has broadened its candidate distribution, which helps coverage and hurts precision.

There is a fair question about whether V4's precision drop is a real regression or an artefact of the benchmark itself. The ground truth for each fixture is hand-written — one to six expected observations per video. When V4 emits a correct observation that isn't in that small gold set, the scorer counts it as a false positive. The model isn't wrong; the ground truth is incomplete. This is a known problem in information-extraction evaluation, and we're working on better benchmarks that separate raw transcription accuracy from knowledge extraction quality, and that don't punish a model for finding more than the annotator thought to write down.

If that benchmark design question is more interesting to you than the model, I wrote a whole separate post on it: When your benchmark is wrong and your model is right.

The V4 artefacts are on Huggingface at:

hammertoe/Qwen3-Omni-30B-A3B-Barbados-LoRA-v4
hammertoe/Qwen3-Omni-30B-A3B-Barbados-fused-bf16-v4
hammertoe/Qwen3-Omni-30B-A3B-Barbados-4bit-v4

If you want to take V4 for a spin, the model card on each Hub repo walks through download, the supported prompt format, and the activation-check probe that proves the adapter is wired up.

Stay tuned for the next iteration. I want to try a longer schedule to see if +13.3pp is in fact a ceiling for this recipe, or whether the calibration shift keeps compounding as the model spends more time in the corpus.

When your benchmark is wrong and your model is right

Matt Hamilton — Wed, 12 Aug 2026 15:39:03 +0000

When your benchmark is wrong and your model is right

We fine-tuned a 30-billion-parameter model on Barbados newspapers. Then we had to write three different benchmarks to figure out whether it actually got better. It did — but not in the way any single benchmark could show on its own, and one of them made it look worse before it looked better.

This is about the benchmarks, not the model. The model is Qwen3-Omni-30B-A3B-Instruct with a LoRA adapter trained on a cleaned Barbados newspaper archive. We're building Pulse, a live public-signal intelligence system for Barbados — radio, newspapers, TikTok, and government feeds turned into a queryable knowledge graph — as part of Future Caribbean, a 21-day agentic AI buildathon. Pulse is still in active development. The fine-tuned model is one of its domain-knowledge components, and it's also published openly for anyone working on Caribbean-domain text.

Three benchmarks, three answers

The model went through two versions. V3 stopped halfway through its training schedule (step 1000 of 1624, when the 12-hour GPU budget ran out). V4 resumed and completed the full schedule to step 1624. We scored both against the unmodified base model on three different eval tracks.

1. Text knowledge probe — 60 multiple-choice questions

Twenty well-known Barbados facts, twenty niche ones, and twenty general-knowledge controls. Scored by mean token log-probability over four candidate completions. Clean, reproducible, deterministic.

V4 wins across the board. The DAPT contribution is real (+13.3pp overall vs base). This benchmark is the one I'd cite if someone asked "did the fine-tuning work?".

2. Radio audio extraction — 17 five-minute FM windows

Real Barbados radio captures (HOTT 95.3, VOB 92.9, CBC 94.7, and seven others). The model hears the audio and emits structured JSON: corrected transcript, events, entities, facts. Scored on quality, hallucination rate, proper-noun precision/recall, and evidence-quote presence.

V3 was essentially flat against base on this benchmark. V4 nearly doubled the quality score. The difference is proper nouns: the model that has read a thousand editions of the Barbados Advocate hears "Rise Together" (a Crop Over event) instead of "Rice Together". That is the entire reason we fine-tuned.

3. TikTok extraction — 10 short videos

Barbados TikToks: golf promotions, catamaran cruises, cave tours, farmers' markets. Each has hand-written ground truth for the structured observations the pipeline should extract. Scored on field-level precision, recall, and F1.

V4 is worse than V3 here. That is the interesting bit.

What "worse" means

V4 and V3 find the same number of correct observations (12 true positives each). V4 emits 23 false positives; V3 emits 8. The difference is 15 extra observations that don't match anything in the ground truth.

I dug into where those 15 extras come from. Two patterns.

Over-emission. On a 65-second catamaran cruise video, V3 emitted 1 observation. V4 emitted 9 — the same catamaran cruise, three times (as duplicates), plus separate "promotion" objects for "drinks were already flowing", "so many options and variety", and "staff were super friendly". Every positive sentence in the review got its own JSON object. V4 found both expected observations (V3 found neither), but buried them in noise.

Type confusion. On an itinerary TikTok listing eight things to do, V3 emitted five event_occurrence objects: racehorses at Pebble Beach, sunrise on the east side, yacht charter, St Nicholas Abbey, Hammers Market. V4 saw the same content but emitted three of them as entity_mention instead — just the place name, without the activity context. The scorer matches on observation type first, so those became false positives (wrong type) while the expected events became false negatives (unmatched).

Both patterns share a root cause. Completing the training schedule broadened the model's sensitivity to Barbados content. Its output layer now fires more readily on domain tokens — place names, venue names, activity descriptions. In the text probe this shows up as higher accuracy. In audio extraction it shows up as better proper-noun transcription. In structured extraction it shows up as over-triggering: the model sees Barbados signal everywhere because it has been trained to.

The benchmark is the bottleneck

The TikTok eval has a scoring problem, and it took V4 to expose it.

The ground truth for each fixture is hand-written: one to six expected observations per video, each with specific fields (name, location, category, price). It is a small, fixed target. When V4 emits a correct observation that isn't in the ground truth — say, "snorkelling at the first stop" on the catamaran video — the scorer counts it as a false positive. The model isn't wrong; the ground truth is incomplete.

This is a known problem in information-extraction evaluation. Strict precision against a small gold set punishes models that extract more than the annotator thought to write down. The right fix is either a larger gold set, a human-in-the-loop review of every "false positive" to separate real hallucinations from under-annotated ground truth, or a softer scoring rubric that rewards partially-correct observations instead of treating them as binary hits or misses.

We're now moving toward the second option: a review pass over every V4 "false positive" to classify it as a real hallucination (the model invented something that isn't in the audio) or an annotation gap (the model found something correct that the ground truth missed). That will give us a corrected precision number that reflects the model's actual quality, not the ground truth's coverage.

Benchmarks evolve with the model

When we started, the 60-probe text eval was the only benchmark. It was designed to answer a simple question: does the adapter know more about Barbados than the base model? It does, reproducibly, and that question is answered.

But the text probe doesn't test what Pulse actually does. Pulse ingests audio — radio and TikTok — and extracts structured events from it. The radio eval was built next, and it revealed that V3 (the half-finished model) was barely better than base on audio extraction despite scoring +8pp on the text probe. That finding motivated resuming the training to produce V4, which is where the audio quality jumped.

Then the TikTok eval revealed that V4, the better model by every other measure, over-triggers on structured extraction. That finding is now driving the ground-truth review, which will in turn produce a better benchmark for the next model version.

Each benchmark answered one question and exposed the next one. The text probe asked "does it know more?". The radio eval asked "can it hear better?". The TikTok eval asked "can it extract cleanly?". The answers are yes, yes, and "it depends on how you score it".

That is not a failure of evaluation. It is what evaluation looks like when you are building something new. You write the benchmark you can write. You run it. It tells you something true but incomplete. The incompleteness becomes visible only when the model outgrows the benchmark's assumptions. Then you write the next one.

The alternative — waiting until you have a perfect benchmark before training — is a good way to never train at all.

Where we are now

We're still in development, not production. The V4 numbers are encouraging — its audio extraction quality (0.230 vs base's 0.135) is the difference between a pipeline that transcribes "Rice Together" and one that transcribes "Rise Together". For a system whose job is to tell you what's happening in Barbados tonight, that is the whole game. But encouraging numbers on a small eval set are not the same as a model ready to ship.

What we're doing now is refining the benchmarks themselves. The TikTok precision regression made it clear that our extraction eval was scoring the wrong thing — punishing correct observations that the ground truth didn't anticipate. So we're splitting the evals along the lines of what actually matters to Pulse's use cases:

Raw transcription accuracy — does the model hear the right words? This is the proper-noun problem: "Rise Together" vs "Rice Together", "Leadpipe and Sadis" vs plumbing and a typo. It's a clean, scoreable metric that directly reflects whether the DAPT investment paid off.
Knowledge extraction quality — given the right words, does the model produce the right structured output? This is where the over-emission and type confusion show up, and where the scoring rubric needs to evolve beyond binary hit/miss against a small gold set.

These are different skills and they need different benchmarks. The current eval conflates them — a model can transcribe perfectly but still score zero on extraction because it emitted the wrong observation type, or because the ground truth didn't list the entity it found. Splitting them apart will tell us whether V4's regression is a hearing problem (it isn't — transcription is better than ever) or a formatting problem (it is — the model over-triggers on domain content).

The model artefacts are published openly at hammertoe/Qwen3-Omni-30B-A3B-Barbados-LoRA-v4 (adapter), …-fused-bf16-v4 (fused weights), and …-4bit-v4 (4-bit MLX for Apple Silicon). You don't need to be running Pulse to use them — any Caribbean-domain NLP task can benefit.

If you have ideas for better extraction scoring against under-annotated ground truth, I'd love to hear them.

Teaching an Audio Model More About Barbados

Matt Hamilton — Fri, 07 Aug 2026 00:30:31 +0000

Automatic speech recognition is very good until somebody mentions the name of a local school, a village, a politician, a festival, or a cricket ground.

Then things get strange.

In an earlier test with audio from Barbados, GPT Transcribe and GPT Audio 1.5 heard the event name “Rise Together” as “Rice Together”, while Qwen3.5-Omni Plus and Flash got it right. Those are different models from the Qwen3-Omni checkpoint used here, but the result motivated this experiment. Acoustically, the mistake is understandable. Culturally, it is wrong. A person who knows the local context has another signal available: they know that Rise Together is the plausible name.

That led me to a question: can we give an audio-native model a stronger model of Barbados, using text that already contains the names, institutions, places, events and relationships it is likely to hear?

So I took an archive of Barbados newspapers, turned it into 51.6 million tokens, and used it for domain-adaptive pretraining of the Thinker inside Qwen3-Omni.

The result is promising, but not conclusive. The adapted model produced higher scores on our preliminary Barbados knowledge probe, particularly on people and institutions. It also got slightly worse on a small set of general-knowledge controls. And, most importantly, we have not yet shown that it transcribes audio more accurately.

This is a very preliminary result. It came from our first training run, which we stopped at step 500 of a planned 801 steps. We were also still extracting the newspaper archive, so the 51.6 million training tokens represent the material available for that run rather than the full corpus we ultimately intend to use.

This post is about what we have actually demonstrated, what broke along the way, and why I think the experiment is still worth pursuing.

The Problem Is Not Just Acoustic

A transcription model is doing more than converting sound into letters.

When audio is clean and a word is common, the acoustic evidence can be enough. But real radio is compressed. Music leaks into speech. Presenters talk quickly. Phone-in guests have different microphones and accents. Proper nouns may be rare or absent from a model's original training data.

At that point, transcription becomes a contest between plausible sequences:

weak acoustic evidence + language context -> selected transcript

For Barbados, the useful context includes things such as:

Kensington Oval
Cave Hill
Samuel Jackman Prescod Polytechnic
Crop Over
constituency and parish names
local politicians, athletes, performers and organisations
the ordinary relationships between all of those things

This is related to contextual biasing in automatic speech recognition. In Deep Context, Pundak and colleagues showed that an end-to-end recogniser could use supplied contextual phrases, including out-of-vocabulary terms, and reported relative word-error-rate improvements of up to 68% on their tasks. Later work on trie-based deep biasing and shallow fusion reported a 19.5% relative improvement over existing contextual-biasing methods, with particular attention to rare long-tail words.

Those systems explicitly supply a biasing list at inference time. Our approach is different: we are trying to make local context part of the model's learned language prior. The papers do not validate our method directly, but they do support the underlying idea that context can help a recogniser resolve rare and ambiguous language.

Why Qwen3-Omni?

I selected Qwen3-Omni-30B-A3B-Instruct because it is natively multimodal. It accepts text, audio, images and video, and it can produce text or speech. More importantly for this experiment, its audio understanding and text generation are parts of one end-to-end architecture.

According to the Qwen3-Omni technical report, the model uses a Mixture-of-Experts Thinker-Talker design:

The Thinker consumes multimodal representations and generates text.
The Talker is responsible for generating streaming speech tokens.
An Audio Transformer, or AuT, turns audio into representations consumed by the Thinker.

The Thinker has roughly 30 billion total parameters with about 3 billion active for each token. The Talker is a separate 3-billion-parameter MoE component. For text-only use, Qwen exposes model.disable_talker(), allowing the speech-generation component to be omitted.

This separation matters. I am not trying to teach the model a Barbadian voice, and I do not have a labelled corpus of Barbadian audio and transcripts large enough to retrain its acoustic front end. I am trying to adapt the component that interprets context and chooses text.

There is precedent inside Qwen's own work for treating these components separately. The Qwen3-Omni report describes distinct post-training pipelines for the Thinker and Talker, and its audio Captioner is produced by fine-tuning the Thinker for audio-to-text output. Our training objective is different, but the architectural boundary is intentional rather than a convenient hack.

Training The Thinker, Not The Talker

We used domain-adaptive pretraining, usually shortened to DAPT.

The idea is straightforward: take a model that has already been pretrained on broad data, then continue its language-modelling objective on unlabelled text from the target domain. Gururangan and colleagues' ACL 2020 paper Don't Stop Pretraining found that this second in-domain phase consistently improved downstream performance across several domains and resource levels.

This was not instruction tuning. The newspaper chunks were not converted into question-and-answer pairs. We presented them as assistant-only continuation records and continued next-token training. The objective was to change what language and relationships the model finds probable, not to teach it a new chat format.

We used LoRA rather than updating every weight. LoRA freezes the original model and injects trainable low-rank matrices into selected transformer layers. That gives us a portable adapter rather than another complete copy of a very large model.

Our LoRA configuration was:

Setting	Value
Rank	64
Alpha	128
Dropout	0.0
Attention targets	`q_proj`, `k_proj`, `v_proj`, `o_proj`
MLP targets	`gate_proj`, `up_proj`, `down_proj`
Base precision	bfloat16
Context length	4,096 tokens
Batch size	1
Gradient accumulation	16
Learning rate	`1e-4`
Scheduler	cosine, 3% warm-up

The modality encoders and aligner were frozen. The Talker was disabled. Only LoRA parameters associated with the Thinker's language model were trained.

One unusually specific setting was lora_dropout=0.0. Qwen3-Omni uses PEFT parameter wrappers for some MoE parameters, and those wrappers reject non-zero LoRA dropout. The default in our training stack was 0.05, so this was one of those small details that stops an expensive run before it has really begun.

Building The Corpus

The source material available for this first run was 1,286 extracted editions of the Barbados Advocate, spanning 2013 to 2023. This was a substantial working corpus, but it was not the complete archive. PDF extraction and cleaning were still running when we started training.

This gave us something general web text does not: sustained coverage of local life over time. The archive contains politics, schools, trade unions, public health, sport, music, community events, weather, regional affairs, court reports, business and the names of people and places that rarely occur in global datasets.

The raw source was PDF, so the preparation pipeline mattered as much as the training command.

We used Docling with full-page OCR to extract text. We then:

Normalised punctuation, HTML entities, Unicode and whitespace.
Removed selected boilerplate patterns, page-continuation markers, standalone labelled contact lines, bylines and obvious layout noise.
Rejected prose dominated by isolated OCR characters.
Preserved useful article headings and paragraph boundaries.
Combined the cleaned material into chunks targeting 250 to 1,000 words.
Serialised each chunk as an assistant-only JSONL training record.

On the GPU server, we applied a second validation pass using Qwen's tokenizer:

validate every record and require non-empty assistant text
deduplicate exact content using SHA-256
discard records shorter than 256 tokens or longer than 4,096
length-sort the remaining records
create a deterministic 2% evaluation split

The resulting numbers were:

Corpus stage	Count
JSONL shards	1,286
Valid input records	49,322
Exact duplicates removed	150
Below 256 tokens	256
Above 4,096 tokens	17
Training records	47,921
Evaluation records	978
Training tokens	51,647,254

The source JSONL snapshot occupied about 238 MB. These figures describe the first run, not the final planned corpus. The corpus remains private and is not redistributed. The trained adapter is public.

There are limitations in the data. It is formal newspaper language, not spontaneous Bajan speech. OCR is never perfect. Newspaper coverage reflects editorial choices and historical bias. We also trained only on the extracted Barbados Advocate snapshot, without adding a separate general-language “wash”. The newspaper itself contains regional and international reporting, but this one-source mixture may still encourage specialisation at the expense of some general capability.

That last trade-off shows up in the benchmark.

The Training Run

I rented a single NVIDIA H200 with 141 GB of memory through Prime Intellect. The successful machine was an on-demand Nebius instance with 16 vCPUs, 200 GB of system RAM and a 1.5 TB boot disk.

The model used almost all of it.

A two-step smoke test peaked at 138.1 GB of GPU memory. That left roughly 3 GB of headroom, so batch size 1 with 16 gradient-accumulation steps was not merely conservative. It was the practical ceiling at a 4,096-token context length.

Training used PyTorch 2.7.1, Transformers 5.8.1, ms-swift 4.4.2, FlashAttention 2 and bfloat16. Checkpoints were written every 250 optimiser steps, with automatic resume and a GPU-hour budget guard around the process.

We stopped after checkpoint 500 of a planned 801 steps, completing about 62% of the intended schedule. This was the first full training attempt, not a converged final model. At that point:

Metric	Value
Training loss	2.0749
Evaluation loss	2.1271
Evaluation token accuracy	0.5358

The adapter is available on Hugging Face as hammertoe/Qwen3-Omni-30B-A3B-Barbados-LoRA. It is tied to base revision 26291f793822fb6be9555850f06dfe95f2d7e695.

The Benchmark That Nearly Lied

The first benchmark run reported identical results for the base model and the adapter. The held-out perplexity was identical too.

That looked like a disappointing but perfectly plausible outcome: perhaps the adapter had learned nothing useful.

It was actually an inference bug.

Qwen3-Omni's adapter keys are rooted beneath the complete model:

base_model.model.thinker.model.layers.0.self_attn.q_proj.lora_B.weight

We had attached PEFT directly to model.thinker. PEFT returned a model without raising an obvious error, but the intended projection layers remained ordinary linear layers. The adapter was not active.

The correct sequence was to attach PEFT to the complete Omni model and only then select its adapted Thinker:

model = Qwen3OmniMoeForConditionalGeneration.from_pretrained(
    base_path,
    dtype=torch.bfloat16,
    device_map="auto",
)
model.disable_talker()

adapted_root = PeftModel.from_pretrained(model, adapter_path).eval()
adapted_thinker = adapted_root.model.thinker

We added an activation gate before trusting another result. It checks that a target projection is a PEFT LoRA layer, that the loaded adapter weights have a non-zero norm, and that base and adapted outputs differ on identical input.

There was another compatibility wrinkle. ms-swift 4.4.2 requires PEFT below 0.20 for training, but PEFT 0.19.1 could not load this Qwen Omni adapter with Transformers 5.8.1 because of an incompatible WeightConverter call. The working inference stack used PEFT 0.20.0 in a separate environment. Keeping training and inference environments separate turned out to be much safer than trying to make one dependency set serve both jobs.

For “The Crop Over festival in Barbados”, the loss changed from 3.5603 to 3.1089. That did not tell us whether the adapter was good, but it finally proved that we were measuring the adapter.

This bit took more work than I expected. It was also a useful reminder that “the checkpoint loaded” is not the same as “the checkpoint affected inference”.

A Barbados Knowledge Benchmark

We built a small paired benchmark of 60 four-choice completion probes:

20 local probes
20 rare-local probes
20 general-knowledge controls

The 60 probes represent 30 facts. Every fact has a canonical question and an independently worded paraphrase. Answer positions are balanced across all four choices.

Rather than ask the model to emit a letter, we score each candidate completion using its mean token log-probability. For each probe, the margin is:

score(correct completion) - score(strongest incorrect completion)

A positive margin is correct. A negative margin is wrong. Changes in margin also show whether the model moved towards the answer without crossing the decision boundary.

The benchmark then compares base and adapter results as a pair. We report accuracy, margin changes, wins and losses, strict fact families, and a control-adjusted local margin. A strict fact family only passes when both wordings are correct.

The Results

These are preliminary results from that partial first run and partial corpus. They are useful for deciding whether the approach is worth continuing, not for claiming a finished Barbados model.

Track	N	Base accuracy	Adapter accuracy	Delta	Margin delta
Local	20	55.0%	65.0%	+10.0 points	+0.247
Rare local	20	45.0%	60.0%	+15.0 points	+0.198
General controls	20	90.0%	85.0%	-5.0 points	-0.344
Overall	60	63.3%	70.0%	+6.7 points	+0.034

Pooling the local and rare-local probes, then subtracting the control margin delta, gave an exploratory difference-in-differences estimate of +0.567. Across the 20 Barbados fact families, strict performance improved from 7 to 11. Four families moved from failing to passing, with no strict Barbados family moving from fully correct to fully wrong. The general controls declined from 8/10 to 7/10 strict families.

The sample is small. A descriptive percentile bootstrap over the 20 local probe wordings produced an interval of -0.399 to +0.822. Because each fact contributes two related wordings, this probe-level interval should not be treated as a confirmatory fact-level confidence interval. This is evidence of movement, not a definitive estimate of effect size.

The Category Breakdown Was More Interesting

Category	N	Base	Adapter	Accuracy delta	Margin delta
Cricket and sport	6	33.3%	33.3%	0.0	+0.310
Culture and music	6	33.3%	33.3%	0.0	+0.074
History	6	66.7%	66.7%	0.0	+0.503
General controls	20	90.0%	85.0%	-5.0	-0.344
People and institutions	16	50.0%	75.0%	+25.0	+0.347
Places and geography	6	66.7%	83.3%	+16.7	-0.327

Here, N counts probe wordings rather than independent facts. A category with six probes contains three facts, so one answer changes its accuracy by 16.7 percentage points. The category table is diagnostic, not a stable leaderboard.

People and institutions produced the clearest improvement. Four probes changed from wrong to correct, with no correct-to-wrong flips:

Queen Elizabeth Hospital opening: 1958 became the correct 1964
Barbados Workers' Union formation: 1938 became the correct 1941
free secondary education: 1966 became the correct 1962
SJPP: “St. James Parish Polytechnic” became the correct “Samuel Jackman Prescod Polytechnic”

Three of those became strict two-wording transitions from failing to passing. The Barbados Workers' Union fact changed on only one wording.

History showed another useful pattern. Accuracy did not change, but the average margin improved by +0.503. The model became more confident about the first Parliament year and moved much closer to the national motto, while still not ranking it first. A binary accuracy score would hide that movement.

Places and geography is the warning against reading category percentages too quickly. Accuracy improved by one net probe, but average margin declined. The adapter corrected a land-area question and one wording of the UWI Cave Hill question, then broke the other wording of the same Cave Hill fact. That is wording sensitivity, not robust geographical understanding.

Culture and cricket mostly showed movement without new correct answers. The model got closer on some obscure details while losing confidence on facts it already knew. Again, useful diagnostic information, but not a product result.

The control regression came from one answer flip: the SI unit of catalytic activity, katal, moved from correct to incorrect. Several controls remained correct but lost margin. With only 20 control probes, this is not evidence of a broad collapse, but it is consistent with the specialisation risk we wanted the controls to expose.

What This Benchmark Is

I think this benchmark is best described as an archive-authored Barbados knowledge probe for an early checkpoint.

It suggests that:

the adapter is active
continued pretraining changed scores in a Barbados-specific direction
some Barbados fact probes became more robust across paraphrases
the strongest gains were around named institutions and their history
there may be a small general-capability cost

That is worthwhile. Before this benchmark, we did not know whether the training had done anything beyond lowering next-token loss.

What This Benchmark Is Not

It is not an audio benchmark.

Knowing that Queen Elizabeth Hospital opened in 1964 does not demonstrate that the model can hear “Queen Elizabeth Hospital” through a Barbadian accent, a poor microphone or music in the background. Years are particularly weak proxies for transcription.

It is not a broad test of cultural understanding either. The benchmark covers named people, institutions, dates, places and selected events. It does not test Bajan vocabulary, informal grammar, pragmatics, code-switching, pronunciation or whether the model understands an implied local reference in conversation.

The questions were authored from the broader newspaper archive, but we did not resolve their cited source records against the exact packed training and evaluation split used for this run. At least one benchmark source sits outside the 2013 to 2023 snapshot. That means this run cannot establish whether every changed answer came from direct exposure, indirect domain adaptation, or knowledge already present in the base model. It is not a clean test of unseen generalisation either.

So the honest conclusion is not “we improved Barbadian transcription by 15%”. We did not measure that.

The conclusion is:

In a preliminary first run using a partially extracted corpus, text-only DAPT
produced higher scores on a small set of Barbados questions, with the clearest
difference around people and institutions. The exact relationship between
those questions and the packed training split was not resolved. Whether the
effect holds after full training, generalises to held-out local material, or
produces more accurate audio transcription remain open questions.

Why I Still Think It Could Help Transcription

Qwen3-Omni's Thinker is where multimodal evidence becomes text. Audio reaches it through the AuT encoder and modality alignment layers. We have left those acoustic components unchanged, but adjusted the language model that interprets their representations and chooses the output tokens.

The intended effect is narrow:

ambiguous audio + stronger Barbados prior -> better local word choice

It should not make the model hear an accent better. It may make it more likely to choose Kensington Oval over a globally common but locally implausible alternative. It may help preserve the full name Samuel Jackman Prescod Polytechnic. It may resolve a phrase because it understands that a person, institution and place regularly occur together in Barbados reporting.

But there is an equal and opposite risk. A strong prior can override clear acoustic evidence and hallucinate familiar local entities. The decline in our general controls is a reason to test that carefully, not hand-wave it away.

This is also where newspaper text may reach its limit. It contains a great deal of local context, but it does not represent spontaneous radio speech. If the problem is pronunciation, accent or code-switching, text-only DAPT is unlikely to be enough. We may eventually need paired local audio and transcripts, or a hybrid approach combining a local lexicon, retrieval and model adaptation.

The Next Benchmark Has To Be Audio

The next evaluation will compare the base and adapted models on exactly the same manually transcribed Barbados audio.

It needs more than one aggregate word error rate. We plan to measure:

ordinary word error rate and character error rate
named-entity error rate
local-term recall
accuracy for numbers and dates
substitutions, deletions and hallucinated entities
performance by speaker, noise level and content type
general English audio as a regression control

The test set should include news, interviews, radio phone-ins, sport, Crop Over coverage, government speech and informal conversation. It should also contain minimal pairs such as Kensington versus Kingston, Cave Hill versus Codrington, and Prescod versus Prescott.

Most importantly, none of its reference transcripts should appear in the text training corpus.

If the adapter reduces errors on local entities without increasing hallucinations or damaging general transcription, then we will have something much closer to the result we actually care about.

Where We Are Now

So far, we have:

prepared a first 51.6-million-token Barbados newspaper snapshot
adapted the Qwen3-Omni Thinker with a rank-64 LoRA
completed 500 of 801 planned optimiser steps on a single H200
published the adapter on Hugging Face
built a paired Barbados knowledge benchmark
observed higher scores on a preliminary Barbados knowledge probe and a small control regression
fixed an inference bug that initially made the benchmark lie to us

We have not yet:

completed the first planned training schedule
finished extracting and preparing the full newspaper corpus
validated improved transcription on Barbados audio
shown improved accent or dialect recognition
established broad cultural understanding
completed and validated the planned 4-bit Apple Silicon deployment

That may sound like a cautious ending. I think it is a useful one.

The experiment has moved from “perhaps local text could help” to “an early, partially trained model behaves differently on a small set of local questions”. The next run will use more of the archive and complete the training schedule. Then we need held-out text and audio benchmarks to find out whether that change is useful.

Stay tuned. The word-error-rate table is the one I really want to publish.

References

Qwen Team. Qwen3-Omni Technical Report, 2025.
Suchin Gururangan et al. Don't Stop Pretraining: Adapt Language Models to Domains and Tasks, ACL 2020.
Edward J. Hu et al. LoRA: Low-Rank Adaptation of Large Language Models, 2021.
Golan Pundak et al. Deep Context: End-to-End Contextual Speech Recognition, 2018.
Duc Le et al. Contextualized Streaming End-to-End Speech Recognition with Trie-Based Deep Biasing and Shallow Fusion, Interspeech 2021.
Qwen3-Omni-30B-A3B-Instruct on Hugging Face
Qwen3-Omni Barbados LoRA on Hugging Face

Moving WeOutside246 from GPT-5 to local models on a base M4 Mac Mini

Matt Hamilton — Mon, 06 Apr 2026 11:29:57 +0000

So, I've been spending a lot of time recently trying to answer a question that I think a lot of indie AI builders are going to hit sooner rather than later:

Can I stop renting intelligence from a hyperscaler and just run the thing myself?

In my case the project is WeOutside246, an autonomous agent I built to track the pulse of Barbados. It follows more than 900 Instagram accounts, reads thousands of posts, looks at images, and tries to work out whether something is an upcoming event on the island or just noise. And by noise I mean all the things that look a bit event-ish but are not actually useful for an events listing site: recaps, giveaway posts, sports fixtures, lifestyle shots, throwbacks, posts from other islands, and so on.

This is very much not a toy problem.

The small things matter here. A model that confuses a recap from last weekend with a fete happening next Friday is not just slightly wrong. It makes the site worse.

So this post is a technical write-up of what I did, what I learned, and how the latest generation of local models performed when I moved evaluation onto a base-spec Mac Mini M4 with 16GB of RAM.

Background

As some of you may know, WeOutside246 is an autonomous system for discovering events in Barbados from Instagram posts.

At a high level the pipeline looks like this:

Follow hundreds of relevant Barbados accounts
Collect post text, metadata, and images
Ask a model to classify whether the post is an upcoming Barbados entertainment event
Extract structured fields such as event name, date, venue, artists, and type
Deduplicate and rank posters
Publish the results to the site

The core extraction problem sounds simple until you look at real data.

The model needs to distinguish between:

an actual upcoming fete in Barbados
a recap of a fete that already happened
a cruise poster for St. Lucia with a Barbadian DJ on it
a sporting event that looks like nightlife marketing
a giveaway post that references an event but is not itself an event listing
a lifestyle post with hashtags that smell like an event but no actual details

And it has to do that from a combination of caption text and image understanding.

Until recently I was relying heavily on frontier hosted models for this. That works. But having said that, it is expensive, and if you are running an always-on ingestion system against thousands of posts, those costs become structural rather than occasional.

I also wanted to reduce the environmental footprint. If I can do the same work on a tiny local machine sipping power on my desk rather than a large hosted inference stack somewhere in a data centre, I think that is worth doing.

Why a Mac Mini M4?

I've been running these evaluations on a base-spec Mac Mini M4 with 16GB unified memory.

That is very much the point.

I did not want a benchmark based on a giant workstation or a rented GPU box because that would miss the whole exercise. The question I cared about was:

Can a normal, relatively inexpensive, very power-efficient machine do useful multimodal event extraction work locally?

The Mac Mini is attractive for a few reasons:

low idle and working power draw
tiny footprint
quiet
unified memory makes local model serving on Apple Silicon surprisingly capable
easy to leave running 24/7 for a self-hosted pipeline

So the motivation here was not just cost. It was cost and energy together.

Hosted frontier models are brilliant, and I still use them in parts of the workflow, but I don't necessarily want to burn that much money and energy on the classification layer forever.

The Models

I tested a mix of Gemma 4 and Qwen 3.5 models, both dense and MoE, plus one private fine-tuned Gemma 4B variant.

The public models were:

I also tested a private fine-tuned Gemma 4B model trained on my own reference set.

The Evaluation Setup

One of the things I wanted to avoid was the classic "I eyeballed a few examples and it seemed good" trap.

So I built a reference dataset and local evaluation harness.

The process looked like this:

Use GPT-5 to generate high-quality structured reference outputs for a set of real posts
Manually review and correct the labels where necessary
Run local models against the same inputs
Compare isEvent classification and structured extraction quality
Use GPT-5 again as a judge for qualitative scoring of the full output

That gave me two lenses:

simple metrics: accuracy, precision, recall, false positives, false negatives
judge metrics: a GPT-5 score for how good the full structured answer was

The simple metrics mattered most for the event gate. If the model gets isEvent wrong, the rest is almost irrelevant.

But the judge scores were useful because they told me something about the shape of the errors. Two models can have similar classification accuracy and still differ a lot in extraction quality, reasoning quality, or schema compliance.

Cleaning the Gold Standard

This bit took more work than I expected.

I started by treating the GPT-5 generated reference set as a gold standard. It turned out to be more like a gold-plated standard. Very good, but not perfect.

Once I began running the local models against it, I found a bunch of cases where the references themselves needed correction. That included:

recap posts marked as upcoming events
sporting events marked as entertainment events
giveaway posts treated as event listings
posts with no explicit date that should not have passed the gate
posts from outside Barbados that looked plausible at first glance

In total I corrected dozens of labels in the reference and fine-tuning data. That was frustrating, but also useful. The models were forcing me to sharpen the rules, not just evaluate them.

That led to a stricter prompt with explicit rules such as:

date is mandatory
recap language means isEvent=false
sports and giveaways are excluded
the event must actually be in Barbados
venue is preferred, but I eventually relaxed that rule for dated event posts where venue is genuinely announced separately

That last point was an interesting one. I initially made venue mandatory. In practice, that was too strict for some real-world Caribbean event posts, particularly certain cruises, outdoor events, and posts where the date is locked in but the venue is released later.

So the prompt evolved. And the reference data evolved with it.

The Results

These are the latest full 200-sample runs on the cleaned reference set.

Model	Architecture	Accuracy	Precision	Recall	F1	TP	TN	FP	FN
Gemma 4 26B A4B	MoE	96.0%	86.0%	100.0%	92.5%	49	143	8	0
Qwen 3.5 9B	Dense	92.5%	94.7%	76.6%	84.7%	36	149	2	11
Qwen 3.5 35B A3B	MoE	91.5%	83.3%	81.6%	82.5%	40	143	8	9
Gemma 4 4B fine-tuned	Dense	88.0%	71.9%	83.7%	77.4%	41	135	16	8
Gemma 4 4B base	Dense	87.5%	70.0%	85.7%	77.1%	42	133	18	7
Qwen 3.5 4B	Dense	86.0%	86.8%	76.7%	81.5%	33	139	5	10

And here are the judge scores from GPT-5 on the 200-sample runs:

Model	Accuracy	Judge Avg	Judge Median
Qwen 3.5 35B A3B	91.5%	70.3	72.0
Gemma 4 26B A4B	96.0%	66.6	68.0
Qwen 3.5 9B	92.5%	64.3	65.0
Qwen 3.5 4B	86.0%	59.3	61.0
Gemma 4 4B base	87.5%	58.8	60.5
Gemma 4 4B fine-tuned	88.0%	56.7	58.0

What Stood Out

Gemma 4 26B A4B was the clear winner on the metric that matters most

If your primary concern is "did we miss an actual event?" then Gemma 4 26B A4B was the standout.

It had:

the best accuracy
the best F1
perfect recall in this run
zero false negatives

That last number is the one that really grabbed me. Missing a real event is expensive for the product because it means the site is incomplete. Gemma 4 26B simply did not miss any of the true events in this dataset.

It did still produce 8 false positives, so it is not flawless. But if I had to choose one local model today to sit behind the extraction gate, this would be the one.

Qwen 3.5 9B was surprisingly sharp

Qwen 3.5 9B had the best precision at 94.7% with only 2 false positives.

That means when it said something was an event, it was usually right.

But the trade-off was recall. It missed 11 true events.

So this is a more conservative classifier. Good if you hate false positives. Less good if your job is to make sure nobody misses a fete.

Qwen 3.5 35B A3B had the best judge score, but not the best event gate

This was one of the more interesting outcomes.

The GPT-5 judge liked Qwen 3.5 35B A3B the most overall. It scored 70.3 average versus 66.6 for Gemma 4 26B.

I think what that means is that Qwen 35B often produced more polished or internally coherent structured outputs even when it was slightly worse on the raw event classification metrics.

So if you optimise for "niceness" of extraction, Qwen 35B looks very strong.

If you optimise for "did it miss a real event in Barbados?" then Gemma 26B still wins.

Fine-tuning the 4B Gemma did not deliver the win I hoped for

This one was humbling.

The private fine-tuned Gemma 4B model scored 88.0%. The untuned Gemma 4B base model scored 87.5%.

So yes, the fine-tuned version was technically a little better. But only just. And its judge score was actually worse than the base 4B.

That suggests a few things may be going on:

the base model was already quite capable
the training set had historical label noise before I cleaned it properly
the fine-tune may have overfit some patterns from the earlier prompt regime
instruction-following from Gemma 4 is already doing a lot of the work

This is a good reminder that fine-tuning is not magic. If the prompt and labels are evolving quickly, it can be easier to degrade a small model than improve it.

Bigger was not always better inside the same family

Qwen 3.5 9B outperformed Qwen 3.5 35B A3B on raw accuracy.

That was a bit surprising.

The 35B MoE variant had better judge scores and slightly better recall, but the 9B dense model was just more precise and ended up ahead on accuracy overall.

So model family matters. Prompt compatibility matters. And architecture matters. It is not as simple as "pick the biggest one you can fit".

A Few Failure Modes

Looking through the false positives and false negatives was one of the most useful parts of the process.

Some of the recurring failure modes were:

hallucinated dates from partial cues in the image
treating recap albums as future events
failing to apply the Barbados-only rule consistently
being overconfident on lifestyle posts with event-ish hashtags
treating giveaways and ancillary promos as primary event posts

One nice thing about using a local model here is that once you can categorise the failure mode, you can usually do one of three things:

tighten the prompt
fix the reference labels
accept the trade-off and pick the model whose mistakes you dislike the least

That last one is important. There is no perfect model here. There is only the model whose mistakes fit the product best.

For WeOutside246, false negatives are especially painful because they mean a real event just never appears. So I am more willing to tolerate some false positives than I am to miss a legitimate Barbados event.

What This Means in Practice

I think the biggest takeaway is this:

You can now do serious multimodal extraction work locally on a tiny, consumer-grade Apple Silicon machine.

Not perfectly. Not for every workload. But absolutely well enough to make it useful for a real product.

That feels like a step change.

Even a few months ago, I would have assumed this kind of pipeline needed to stay glued to frontier hosted models for the foreseeable future. Now I think the picture is much more nuanced.

For WeOutside246, the likely shape of the system going forward is:

use local models for the bulk event extraction and classification work
keep frontier models for higher-value judging, dataset generation, and perhaps some of the harder consolidation tasks
continue refining prompts and evaluation as the content distribution changes over time

So this is not me saying hosted models are obsolete. Far from it.

It is me saying that the boundary has moved. And for small products, side projects, and self-funded tools, that is a very big deal.

Next Steps

There are a few things I still want to do:

re-run the fine-tune on the cleaned dataset
measure throughput and cost-per-post more rigorously
profile power usage on the Mac Mini rather than just infer the savings qualitatively
test whether a cascaded setup works better, for example a smaller model first and a larger model only on borderline posts
publish more of the evaluation tooling once I have cleaned it up a bit

I also want to keep pressure-testing the Barbados-specific rules. The line between event, promo, recap, and social chatter is culturally specific, and that is exactly what makes this interesting.

One of the goals here was never to build a generic event extractor. It was to build something that understands this particular corner of the internet.

The difference between a proper Barbados party flyer and a post that just happens to have a DJ in it is subtle until you have seen enough of them.

Then it becomes obvious.

Well... obvious to a human anyway. Getting a model to internalise that is the fun part.

Closing

I think local AI is finally at the point where it can do meaningful work for real products, not just demos.

For me, this is exciting because it opens up a path to making WeOutside246 cheaper to run, more sustainable, and more independent of hosted inference pricing.

And I quite like the idea that a tiny Mac Mini sat quietly on a desk can now do work that, not very long ago, I would have assumed needed a large remote AI system.

If you're building something similar, especially anything involving classification plus extraction over messy real-world social media data, I would strongly encourage you to build the evaluation harness first.

The model leaderboard is interesting. Your own error taxonomy is much more important.

Anyway, that's where things are up to right now. I'm going to keep iterating on the prompts, probably retrain the 4B fine-tune on the cleaned data, and see how much more I can squeeze out of the local stack.

Stay tuned. WeOutside246 is getting a lot smarter.

Why Semantic Search Matters... especially in Barbados!

Matt Hamilton — Tue, 02 Dec 2025 21:31:43 +0000

How Semantic Search Finds Products That Keywords Miss

I've been building this product search engine for Barbados called "Yuh Gettin' Tru?" (Bajan for "Are you finding what you need?"), and I wanted to share something that perfectly illustrates why semantic search matters.

The Problem with Traditional Search

I was testing the site against a local home store called Dwellings. They sell all sorts of home goods - furniture, kitchenware, decor. I searched for "hardwood cutting board" on their website and got... nothing. Zero results.

Now, here's the thing. They absolutely do sell hardwood cutting boards. They have these lovely acacia wood boards with handles. But their product database doesn't call them "hardwood cutting boards" - it calls them "Acacia Cutting Board with Handle".

Traditional keyword search sees "hardwood" and "acacia" as completely different strings. No overlap, no match, no results. The customer leaves, probably assumes the store doesn't stock what they need.

What Semantic Search Does Differently

When you search for "hardwood cutting board" in Yuh Gettin' Tru?, the system understands that you're looking for a cutting board made from hard, durable wood. It doesn't just match keywords - it matches meaning.

The same search returns those acacia cutting boards with a 90% match score. And here's the bit I'm particularly pleased with - it explains why it matched:

"This is a cutting board made of acacia wood, which is a type of hardwood. It is a very good match for the query."

The system knows that acacia is a hardwood. Not because someone manually tagged it, but because the embedding model (sentence-transformers/all-MiniLM-L6-v2) has learned the semantic relationship between these concepts.

How It Works Under the Hood

The technical pipeline is quite satisfying to work on:

Vector embeddings - Each product gets converted into a 384-dimensional vector that captures its semantic meaning. "Acacia cutting board" and "hardwood cutting board" end up quite close in this vector space because they mean similar things.
Similarity search - MongoDB Atlas Vector Search finds the closest matches in milliseconds, regardless of exact wording.
LLM reranking - Gemini Flash takes the top candidates and scores each one for genuine relevance, filtering out false positives and generating those explanations.

The whole process takes about 3-5 seconds, most of which is the LLM reranking. The vector search itself is lightning fast.

Why This Matters for Caribbean Retail

Barbados has a fragmented retail landscape. Dozens of stores, each sourcing goods from different parts of the world - some from the US, some from the UK, some from Asia. This means product terminology varies wildly.

Take "cutlery" versus "flatware". If you're British (or Bajan, since we inherited British English), you call your knives and forks "cutlery". But many retailers here source from American suppliers, and their databases use "flatware" instead. Search for "cutlery" on some store websites and you get nothing. The products exist, they're just labelled differently.

Semantic search understands that cutlery and flatware mean the same thing. It bridges the gap between how customers think about products and how retailers happen to have labelled them.

The Small Things

I think what I enjoy most about this project is how the small technical decisions compound into real user value. Choosing the right embedding model. Adding category extraction from breadcrumbs. Structuring the embedding text with labels like "product:" and "brand:" to give the model more context.

None of these are flashy features. But together, they mean a customer searching for "hardwood cutting board" actually finds the acacia board they're looking for.

The small things. They matter.

Making Services Discoverable with ERC-8004: Trustless Agent Registration with Filecoin Pin

Matt Hamilton — Tue, 04 Nov 2025 21:51:57 +0000

Making Services Discoverable with ERC-8004: Trustless Agent Registration with Filecoin Pin

I've been thinking a lot about agent metadata lately. Not the sexy stuff, not the AI models or the clever algorithms. The boring bit. Where do you put the JSON file that describes what an agent does?

It's one of those problems that seems trivial until you actually try to solve it properly. You can't store it on-chain (too expensive). You can't use regular IPFS pinning (it might disappear). You can't use AWS (defeats the whole point of decentralisation). So what do you do?

This week, I finally got a chance to properly explore this by registering GitHub's MCP server as an ERC-8004 agent with Filecoin Pin. Here's what I learnt.

The Problem

The issue is simple enough to explain: when you register an agent, you need to point to some metadata, a JSON file describing what it does, how to connect to it, what it's capable of. The tokenURI in ERC-8004 parlance.

You can't store this on-chain. A typical agent card is 1-2KB of JSON. On Ethereum mainnet, that's prohibitively expensive. Even on L2s, you're looking at a chunk of change for every agent registration.

So you store it off-chain and reference it. Which raises the obvious question: where?

Generic IPFS pinning services? They can drop your content. No guarantees. Centralised storage like AWS? Single point of failure, ongoing costs, and frankly defeats the point. Store everything on-chain anyway? Not practical.

I needed something that would actually work long-term.

Filecoin Pin

I'd been aware of Filecoin Pin for a while but hadn't actually tried it. The pitch is compelling: you get cryptographic proof that your data is being stored. Daily PDP (Proof of Data Possession) checks. Decentralised across Filecoin's storage network. IPFS compatible so it works with existing tooling.

The key bit is that last part about proof. With generic IPFS pinning, you're trusting that someone, somewhere, is keeping your content around. With Filecoin Pin, you can actually verify it. Daily proofs that storage providers are holding your data.

That matters for agent metadata. If you're building something meant to last years, you need more than hope.

What I Actually Built

For this experiment, I took GitHub's existing MCP server and registered it as an ERC-8004 agent. Not building something new, just making something that already exists discoverable through the registry.

GitHub already runs a public MCP server at https://api.githubcopilot.com/mcp/. It provides repository management, issue tracking, PR tools. The usual GitHub stuff exposed through the Model Context Protocol. It's real, it's running, and anyone can use it.

What it didn't have was an ERC-8004 identity. No way for other agents to discover it. No on-chain registration. No verifiable metadata storage.

So I created an agent card (a JSON file describing the service), stored it on Filecoin Pin, and registered it on the Identity Registry on Base Sepolia. Now it has a verifiable, discoverable identity.

The Agent Card

The agent card is a JSON file that describes the MCP server's capabilities and how to connect to it:

{
  "type": "https://eips.ethereum.org/EIPS/eip-8004#registration-v1",
  "name": "GitHub Integration Agent",
  "description": "AI agent providing GitHub repository, issue, and pull request management capabilities...",
  "endpoints": [
    {
      "name": "MCP",
      "endpoint": "https://api.githubcopilot.com/mcp/",
      "version": "1.0.0",
      "capabilities": {
        "tools": [
          {
            "name": "repository_management",
            "description": "Browse code, search files, analyze commits..."
          },
          {
            "name": "issue_management",
            "description": "Create, update, search, and manage GitHub issues..."
          },
          {
            "name": "pull_request_management",
            "description": "Review PRs, manage approvals, merge conflicts..."
          }
        ]
      }
    }
  ],
  "supportedTrust": ["reputation"]
}

This agent card gets uploaded to Filecoin Pin, which returns a CID (Content Identifier). That CID then gets registered on-chain as ipfs://<CID>/github-agent-card.json, effectively creating a verifiable, discoverable identity for GitHub's existing MCP server.

The Complete Workflow

Here's how the pieces fit together:

1. Upload to Filecoin Pin

filecoin-pin add --auto-fund github-agent-card.json

This uploads your agent card and returns:

Root CID - The IPFS identifier
Dataset ID - For checking PDP proof status
Storage deal confirmation - Proof it's being stored

The --auto-fund flag ensures your storage provider wallet has sufficient USDFC (Filecoin stablecoin) to pay for storage.

2. Register on Base Sepolia

cast send 0x7177a6867296406881E20d6647232314736Dd09A \
  "register(string)" \
  "ipfs://<CID>/github-agent-card.json" \
  --rpc-url https://sepolia.base.org \
  --private-key $PRIVATE_KEY

This mints an ERC-721 NFT representing your agent on the ERC-8004 Identity Registry.

Why Base Sepolia?

Low gas costs (it's an L2)
Official ERC-8004 reference implementation is deployed there
Easy to get testnet ETH from faucets

3. Verify PDP Proofs

filecoin-pin data-set <dataset-id>

This shows you the proof status:

Dataset ID: 933
Root CID: bafybeihhal5hlbylkibniig6j72wdrm7lr4nf6z47natleh2jkyosrg7di
Storage Provider: f01234
Status: Active
Last PDP Proof: 2025-01-15 14:32:10 UTC
Next Proof: 2025-01-16 14:32:10 UTC

Daily proofs mean you can always verify your agent metadata is still being stored. This is the cryptographic guarantee that sets Filecoin Pin apart.

4. Agent Discovery

Now any application can discover and use your agent:

# Query the registry
cast call 0x7177a6867296406881E20d6647232314736Dd09A \
  "tokenURI(uint256)" 55 \
  --rpc-url https://sepolia.base.org

# Returns: ipfs://bafybeihhal5hlbylkibniig6j72wdrm7lr4nf6z47natleh2jkyosrg7di/github-agent-card.json

# Fetch the agent card
curl -s "https://ipfs.io/ipfs/bafybeihhal5hlbylkibniig6j72wdrm7lr4nf6z47natleh2jkyosrg7di/github-agent-card.json" | jq .

The agent card tells them how to connect to the GitHub MCP server and what capabilities are available.

Why This Matters for Builders

This is still very much a work in progress. Everything here is running on testnets, and we're actively figuring out what the production infrastructure should look like. But having said that, the pattern is already useful for making any existing service discoverable as an ERC-8004 agent with verifiable, persistent metadata.

For Agent Builders

You get:

Peace of mind - Your agent metadata won't disappear
Verifiability - Anyone can check that storage is active
Decentralisation - No single point of failure
Standards compliance - Works with ERC-8004 ecosystem

For Agent Users

They get:

Trust - Can verify agents are legitimate and persistent
Discovery - Find agents via on-chain registry
Composability - Combine multiple agents together
Transparency - See full agent capabilities before using

For the Ecosystem

We get:

Infrastructure for the emerging agent economy
Interoperability via ERC-8004 standard
Long-term viability with provable storage
Foundation for reputation and validation systems

Whilst these lists look a bit like marketing bullet points, each of these actually matters. The infrastructure piece is what I find most interesting, because we're finally building the plumbing that makes agent composition practical.

We'd love to have more developers try this out and give us feedback on what works and what doesn't.

Try It Yourself

So if you want to give this a go, I've put together a complete tutorial that walks you through every step:

Register an ERC-8004 Agent with Filecoin Pin Storage

The tutorial includes:

Complete prerequisites and token setup
Step-by-step commands with expected outputs
Screenshot indicators showing what you should see
Troubleshooting for common issues
Example agent card you can customise

Everything you need is in the tutorial. It's designed to take you from zero to a registered agent in about 30-45 minutes. Having said that, if it's your first time working with Filecoin or Base Sepolia, you might want to give yourself a bit more time to get familiar with the faucets and tooling.

What's Next?

Well, this is just the beginning. The ERC-8004 standard includes three registries:

Identity Registry (what I demonstrated here) - Register and discover agents
Reputation Registry (coming) - Build trust through verified actions
Validation Registry (coming) - Third-party verification of agent behaviour

Combining these with provable storage creates the foundation for a real agent economy. I think the reputation piece is going to be particularly interesting, because that's where agents can start to build track records that others can actually verify.

Ideas to Explore

Some things I'm thinking about:

Multi-agent systems - Agents that discover and compose with other agents
Reputation building - Agents that accumulate verified track records
Validator networks - Decentralised verification of agent behaviour
Agent marketplaces - Discover and use agents based on capabilities and reputation
Cross-chain agents - Agents that operate across multiple networks

There's probably a lot more that I haven't thought of yet. If any of this sounds like something you're working on, I'd love to hear about it.

The Bigger Picture

I think we're at an inflection point. AI agents are becoming capable enough to act autonomously, but they need decentralised infrastructure to be truly trustless.

By solving the storage problem with Filecoin Pin and the identity problem with ERC-8004, we're enabling a new category of applications:

Autonomous trading agents with verifiable track records
Code review agents that build reputation over time
Data analysis agents that can be audited and verified
Coordination agents that compose multiple specialised agents
Personal assistants that users actually own and control

The key insight? Agents aren't just smart contracts. They're long-lived entities that need persistent, verifiable infrastructure. It has been a challenge working out what that infrastructure looks like, but I think the combination of ERC-8004 and Filecoin Pin gets us pretty close.

Getting Involved

The ERC-8004 ecosystem is just getting started, and we need builders to shape it.

If you want to try this out:

Follow the tutorial to register your first agent
Join the ERC-8004 discussions
Check out the reference implementation
Explore Filecoin Pin documentation

Some things you could build:

Register your existing services as ERC-8004 agents
Build agent discovery tools
Develop reputation systems
Create validator networks

The infrastructure is ready. The standard is here. So if any of this sounds like it would be useful for what you're building, then get in touch and let's chat!

Resources

Tutorial: Register an ERC-8004 Agent with Filecoin Pin
ERC-8004 Specification: https://eips.ethereum.org/EIPS/eip-8004
Filecoin Pin CLI: https://docs.filecoin.io/builder-cookbook/filecoin-pin/filecoin-pin-cli
Base Sepolia Faucet: https://www.alchemy.com/faucets/base-sepolia
Example Agent: View on Basescan

Building "Yuh Hear Dem": A Parliamentary AI with Google's ADK and a Lesson in Agentic Design

Matt Hamilton — Thu, 19 Jun 2025 01:01:54 +0000

Democracy thrives on transparency, but the raw data of governance—hours of parliamentary video, dense transcripts, and complex legislation—is often inaccessible to the very citizens it’s meant to serve. This was the challenge that sparked "Yuh Hear Dem," our submission for the Google Agent Development Kit (ADK) Hackathon. The project began as a father-daughter mentoring journey into AI and evolved into a powerful tool for civic engagement in Barbados. It combines deep experience in backend AI architecture with a fresh perspective on user experience, guided by principles from the world of education. This blend allowed us to build a system that is not only technically sophisticated but also genuinely accessible, transforming the way citizens can interact with their government.

This post details our technical journey, from the initial data pipeline to a crucial architectural pivot, all powered by Google's Agent Development Kit (ADK), Gemini, and a Knowledge Graph backend.

The Problem: From Hours of Video to Actionable Insight

Parliamentary sessions in Barbados, like in many places, are published as long-form YouTube videos. Finding what a specific minister said about a particular topic, like the "sugar tax," requires manually scrubbing through hours of footage. This creates a significant barrier to civic engagement.

Our goal was to transform this unstructured data into a structured, queryable format, allowing any citizen to ask a natural language question and get a direct, source-verified answer.

The Solution: An AI-Powered Parliamentary Research Assistant

"Yuh Hear Dem" (Bajan dialect for "Did you hear them?") is a conversational agent that allows users to query parliamentary data. A user can ask, "What has been discussed about the sugar tax?" and receive a concise summary, direct quotes from MPs, and links to the exact moments in the source videos.

The system is built on a sophisticated Retrieval-Augmented Generation (RAG) pipeline that combines the semantic power of vector search with the structured precision of a knowledge graph.

The Technical Architecture: A Three-Stage Pipeline

Our system is built on a robust data processing and retrieval pipeline.

1. Ingest, Clean, Extract

The foundation of our system is a structured knowledge base built from raw, messy transcripts.

Ingest: We start by ingesting the full YouTube transcripts from hundreds of parliamentary session videos—over 1,200 hours of content.
Clean: The raw transcripts are often riddled with grammatical errors and misattributions. We use Gemini to clean and structure this text, correcting grammar, identifying speakers, and aligning the text with accurate video timestamps.
Extract: With clean, timestamped text, we use Gemini again to perform entity and relationship extraction. It identifies people, topics, bills, and the connections between them (e.g., "Minister X spoke about Bill Y"). This structured data, including over 33,000 nodes and 86,000 statements, is stored in MongoDB Atlas.

This process creates a rich, interconnected Knowledge Graph that forms the backbone of our agent's "brain."

2. Hybrid Retrieval with GraphRAG

When a user asks a question, the agent doesn't just rely on a simple semantic search. It uses a hybrid retrieval strategy:

Vector Search: We run a vector search over MongoDB Atlas embeddings to find semantically similar transcript segments. This is great for broad, topic-based queries.
Knowledge Graph Search: We traverse the entities and relationships in our knowledge graph to find precise connections (e.g., Minister -> Topic -> Session). This excels at specific, factual queries.

The results are combined and ranked using a hybrid scoring model (GraphRAG), giving us the best of both worlds. Critically, every piece of information is grounded in video timestamps, allowing us to generate direct links to the source.

3. The Agent Architecture Evolution: A Lesson in Pragmatism

Our journey with ADK taught us a valuable lesson about the current state of multi-agent frameworks.

The Original Vision: A Multi-Agent Pipeline

# Root Conversational Agent
ConversationalAgent(
    model="gemini-2.0-flash",
    sub_agents=[ResearchPipeline]
)

# Sequential Research Pipeline
ResearchPipeline(SequentialAgent):
├── ResearcherAgent(LlmAgent)
│   ├── Tools: [hybrid_search_turtle, authority_search_turtle, topic_search_turtle]
│   └── Role: Parliamentary data collection via MCP tools
│
├── ProvenanceAgent(BaseAgent) 
│   ├── Custom Implementation: Video source enrichment
│   └── Role: Enrich entities with YouTube timestamps & transcripts
│
└── WriterAgent(LlmAgent)
    ├── Dynamic Instruction: Receives enriched turtle data
    └── Role: Synthesize findings into cited responses

Initially, we designed a classic multi-agent system using a SequentialAgent. The idea was to have a clear separation of concerns:

ConversationalAgent: The main entry point.
ResearchPipeline (SequentialAgent):
- ResearcherAgent: Collects data from our knowledge graph.
- ProvenanceAgent: Enriches the data with video sources and timestamps.
- WriterAgent: Synthesizes the final response.

The Roadblock: Session State Management

We quickly hit a wall. We found that ctx.session.state was not being reliably shared between the agents in our SequentialAgent pipeline. The ResearcherAgent would fetch data, but by the time the flow reached the ProvenanceAgent or WriterAgent, the state was often empty or corrupted.

# What SHOULD have worked:
ctx.session.state["turtle_results"] = raw_turtle_data  # ResearcherAgent
enriched_turtle = ctx.session.state.get("turtle_results", [])  # ProvenanceAgent ❌

# What we encountered:
# - Session state not reliably shared between agents in SequentialAgent
# - Context loss during agent handoffs  
# - Empty/corrupted state in downstream agents
# - Related to: https://github.com/google/adk-python/issues/1119

This appears to be a known challenge, which we tracked in GitHub Issue #1119. This roadblock became a critical learning moment: while the theory of multi-agent systems is powerful, the practical implementation can be fraught with state management complexities.

The Pivot: A Robust Single-Agent Solution

To solve this, we refactored our architecture into a single intelligent agent with a set of specialized function tools. This approach proved to be far more reliable and easier to debug.

The agent maintains context reliably, and the tools are called synchronously, ensuring data is passed correctly.

# Refactored Single-Agent Solution
self.agent = LlmAgent(
    name="YuhHearDem",
    model="gemini-2.5-flash-preview-05-20",
    planner=BuiltInPlanner(),
    tools=[
        FunctionTool(search_parliament_hybrid), # Was: ResearcherAgent
        FunctionTool(clear_session_graph),      # Memory management
        FunctionTool(get_session_graph_stats),  # Session insights
        FunctionTool(visualize_knowledge_graph) # Was: Custom visualization
    ]
)

This pragmatic pivot allowed us to achieve our desired modularity—with each tool handling a specific task—without the overhead and unreliability of inter-agent state management.

The User Experience: Making AI Accessible

Technology is only as good as its interface. Our focus on educational design was instrumental here. The frontend was built to make the agent's powerful capabilities accessible to everyone.

Key design principles included:

Progressive Disclosure: Information is presented in expandable cards, preventing cognitive overload. Users see a high-level summary first and can expand for details.
Visual Learning: We used D3.js to create interactive knowledge graphs, helping users visually understand the relationships between speakers, topics, and sessions.
Contextual Guidance: The agent uses the knowledge graph to generate relevant follow-up questions, guiding users on natural exploration paths.

Conclusion and What's Next

"Yuh Hear Dem" is more than just a technical demo; it's a functioning tool for enhancing democratic transparency. Our journey taught us several key lessons:

The Power of Hybrid RAG: Combining knowledge graphs and vector search provides superior retrieval accuracy.
ADK's Strengths: While multi-agent state sharing needs maturing, ADK’s single-agent with function tools model is incredibly robust for building complex, reliable AI systems.
Pragmatism Over Purity: A simpler, more reliable architecture is often better than a theoretically "purer" but fragile one.
Human-Centered Design is Key: An intuitive UI, grounded in learning principles, is essential for making powerful AI accessible and useful.

We invite you to explore the project yourself.

Live Site: https://yuhheardem.com
Git Repo: https://github.com/KatieM00/YuhHearDem
Hackathon Submission: https://devpost.com/software/yuh-hear-dem-adk

Our next steps involve expanding the data sources to include official legislative documents and exploring a return to a multi-agent architecture as the ADK framework evolves. For now, we're proud to have built a tool that helps citizens hear what really matters.

Lift as You Climb

Matt Hamilton — Tue, 26 Jul 2022 17:34:14 +0000

It was Sunday the 7th of June 2020. The first wave of the Covid-19 pandemic had hit and everywhere was in strict lockdown. I can't remember the exact specifics, but I think I was probably sat on the couch in my house in the UK at the time, perusing Twitter on my phone. I stumbled across a tweet from a Developer Advocate at Microsoft, Brandon. I'm not sure I was even following him at the time, but it was probably retweeted by someone I was following.

// Detect dark theme var iframe = document.getElementById('tweet-1269740874721710080-612'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1269740874721710080&theme=dark" }

I went upstairs to my “home office” that I had set up in the spare bedroom. I had started a new role as a Developer Advocate at IBM a few months prior. I'd only visited the office in London once, just before the lockdowns came into effect. As a team whose main role was in-person meetups and events in the UK, we were trying to re-orient ourselves to producing online content. As a part of that, I had recently started Twitch streaming myself. I'd not used Twitch before, but become aware of it via some other colleagues at IBM that were using it for live-coding and education. I loved the idea (and it terrified me!) that you could just broadcast yourself coding and other people might drop in the chat to keep you company, offer assistance, chat nonsense, etc.

Anyway, I joined PJ's channel and he was there learning the very basics of, I think, HTML at the time. Maybe C#. Brandon was there too assisting him. I think there was no more than 3-4 people in the chat. But it felt great to be able to help someone start learning the basics. Every so often PJ would divert into something about literature (he was an English teacher after all) and talk about some paragraph of a book, or some poet I'd never heard of. But it was enjoyable. Listening to his passion about literature, and trying to channel that into learning something new.

I started following PJ on Twitter, as well as Brandon and another Developer Advocate friend of theirs Chloe. Together they created a bunch of silly little bots on Twitter, e.g. the Shania bot ("Let's go girls!"). A bot that begged Mountain Dew for sponsorship (at least I think that one was a bot). Chloe was also from a non-programming background, having originally come from theatre arts and stage school into DevRel. They did some fantastically entertaining shows online and started a Podcast called 8-bits.

PJ eventually managed to get a role in DevRel and is now an Education evangelist at Gitlab.

This is all great Matt, but so what?

It is just over two years since that tweet at the top, and stumbling across PJ and his journey into DevRel, and Chloe and her journey from theatre arts to DevRel.

A few days ago I was interviewing for a company for a senior developer relations role. I had four different interviews, covering different aspects such as my technical knowledge or my thoughts on developer advocacy, building and leading teams, etc.

One specific question I was asked in the last interview was:

"Do you think a developer advocate needs to have a software development / technical background?".

And some neurons somewhere in the recesses of my brain fired and I remembered PJ, and Chloe and their origins in high school english teaching and stage acting.

"No." I replied. And I went on to explain that the most important aspect, in my view, for developer relations is the ability to relate to and empathise with developers. Yes, of course, one of the most obvious ways is to have a development or technical background yourself. But actually, those coming into tech from other backgrounds are learning themselves. They have very fresh experience of having to learn the new terminology, functionality, etc of a product and explain it to others. They bring different skills, and different experiences to the table.

I remember myself when I first started learning AI and machine learning. Whilst I was a developer, I had no experience yet in machine learning. "Is that a regression problem or a classification problem?". I had no idea. I didn't even understand the terminology to describe what I was trying to learn. What is a tensor? What is a logit? So when I moved into doing DevRel for machine learning, I was able to think "What would I like to have known when I started in machine learning that would have made it easier for me?".

Another question I was asked in the interview:

"Can you give an example of where you have helped another developer progress?"

Again, I remembered another event that came from a random encounter. We were running an internal hackathon at Ripple, and a developer, Ami, from a different part of the company, that I'd never come across presented a fantastic hackathon project they were working on. I had recently been accepted as a speaker at Python Web Conference 2022, and the topic I was going to speak on aligned very much with what Ami developed in the hackathon. I approached Ami after the hackathon, and asked if she would like to co-present a talk with me at the conference. She was a fantastic developer, but I could see she would make a really good developer advocate as well. She had never spoken at a conference before, but agreed to do so. I contacted the conference organisers and let them know I'd now be co-presenting with a colleague.

The talk went fantastically! I talked about the general topic, and Ami gave a live-coding demo. A pretty amazing feat for someone's first conference talk!

Around that time I left Ripple, but Ami wanted to stay in touch and asked me to be her mentor. I was honoured, and glad to be able to help in any way I could. I hope one day she makes the move full-time into DevRel!

So this brings me back to the title of this post, "Lift as you climb". This was a saying I first heard from another developer advocate, Cassidy Williams. As you learn and progress, help bring others along with you too. In that post she talks about the importance of communities and mentorship:

“I wouldn’t be where I am today if it wasn’t for the community around me,” Cassidy points out. “You can do pretty well on your own in tech but you can really succeed if you have a community that you help that then helps you in return.”

And so I'd like to take this time to thank all those random people I've interacted with over the years that have got me to where I am now. Many of you may not have even realised how you have helped.

Oh... and I got the job! ;)

Header Photo by Sylvain Mauroux on Unsplash

XRPL Grants Wave 3 Theme: Contributing to Open Source

Matt Hamilton — Fri, 04 Mar 2022 15:34:47 +0000

Applications for Wave 3 are now open!

The official theme for this wave is “Contributing to Open Source.” The Program aims to support the independent, open-source ecosystem development around the XRP Ledger (XRPL). Making “Contributing to Open Source” the focus area of Wave 3 encourages projects that not only expand and enhance the resources available for XRPL developers, but that also continue to grow and strengthen the XRPL community. Types of projects may include:

Creating and open-sourcing a new project
Contributing to an existing open-source project
Adding XRP or XRP Ledger support to a project
Integrating the XRP Ledger with existing APIs, SDKs, and technologies that simplify deployment and integration
Building libraries for the XRP Ledger
Opening PRs for other projects to support the XRP Ledger

Projects applications for Wave 3 will be accepted form March 4 - 28, 2022, and applicants can view and submit their applications online. For all information about applying visit www.xrplgrants.org.

Wave 3 Timeline

Applications Open: March 4 - 28, 2022
Finalist Interviews: May/June 2022
Grantees Announced: July 2022

All applications for Wave 3 will be carefully reviewed by an expert panel of judges from respected institutions and companies from around the world. Select project teams will then be invited to participate and present in a 30-minute interview with members of the grants team and the judging committee.

Since the XRPL Grants program launched in May 2021, more than 50 projects have been funded internationally. View all the awarded projects.

Join a Live Q&A

We invite anyone interested in learning more about applying for an XRPL Grant to attend our official Q&A on the RippleXDev Twitch channel on Tuesday, March 15, 2022 at 11 AM PT (18:00 UTC).

Have more questions about applying?

Answers to commonly asked questions are available on the FAQ page. The XRPL Developers Discord is also a great place to connect with the community, including past funding recipients.

Debunking the Myths — What Is XRPL

Matt Hamilton — Thu, 16 Sep 2021 17:09:35 +0000

Facts & Fiction: Debunking XRP Ledger Misconceptions

As someone who’s been deeply ingrained in the software development community for 20 years, I’ve come to realize that misinformation propels hearsay unlike anything else. There are a number of falsehoods circling the XRP Ledger (XRPL) that I encounter on a regular basis including the misconception that it’s centralized, that there are hidden fees, that it’s private and used only by banks — and it's time to set the record straight. If you’re a developer interested in building on the XRP Ledger, or you simply want to learn more about the network, keep reading. I’m breaking down the facts and debunking the biggest misconceptions about XRPL.

MYTH: The XRP Ledger is Centralized. Based on my experience as a developer and my conversations with fellow devs, this is perhaps the most rampant falsehood — and here’s why. In a centralized blockchain, a single authority governs the entire network, controls validation, controls updates to the ledger, and creates a single point of failure (which theoretically leaves the entire network vulnerable to an attack).

TRUTH: The XRP Ledger is Decentralized. The XRPL delivers powerful utility to developers on a public, decentralized blockchain. Validation occurs via a consensus process where independent nodes are managed by a wide range of participants — not by a singular controlling entity. Amendments to the XRPL can be proposed by any participant and require 80% quorum approval for two consecutive weeks by the validator community. And once confirmed, transactions cannot be reversed or altered.

These are all hallmarks of decentralized ledgers. If there’s one takeaway I want you to remember, it’s this: Ripple is a contributor to the network, but only one contributor among many. As of writing, Ripple run around 5% of the approximately 900 nodes on the XRP Ledger, and six of the approximately 150 validator nodes. Ripple follows the same protocols and its rights are the same as those of any other contributor.

Final Myths to Debunk

MYTH: New XRP Can Be Added to the Ledger. According to this myth, a single authority can make unilateral changes to the fundamental, underlying code, which leaves the Ledger open to hackers who could create new XRP.

TRUTH: Even if a bad actor attempted to add unauthorized XRP to the Ledger, the consensus protocol ensures no single authority can execute on this. More than 66 million ledgers have been successfully closed since the XRP Ledger was first conceived with 100 billion XRP created at the inception of the Ledger, and no additional XRP has ever been added into the system.

MYTH: XRPL Has Hidden Fees. Fees, including transaction costs and reserve fees are returned to Ripple after being implemented into the Ledger, or rewarded to validators after being implemented into the XRP Ledger.

TRUTH: Just like other public blockchains, transaction fees are applied on the XRP Ledger, although they are far lower than most (just fractions of a cent on XRPL). Unlike other blockchains, however, the fee is neither returned to a central authority or paid as a reward to validators or any other party. It is, in fact, irreversibly destroyed. Because fees rise in-step with the load on the network, this protects the network from spam, malicious behavior and DDoS attacks. Furthermore, XRP transactions on the Ledger are settled almost instantaneously (only 3-5 seconds to confirm completion vs. 10+ minutes for other blockchains, or multiple days for banks to send cross-border fiat funds to other banks). The XRPL consensus protocol bridges the gap, saving substantial time and transaction costs.

MYTH: Blockchains Can’t Be Decentralized, Scalable and Secure. Improvement in one of these aspects must negatively impact one of the other two. We must sacrifice one to optimize the others, right? Not so fast–here’s the truth.

TRUTH: The Blockchain Trilemma is a model to conceptualize the challenges that all blockchains face, stating that the platforms cannot be truly decentralized, scalable and secure all at once. The truth is, the XRP Ledger was the first and is one of only a few blockchains able to run a decentralized, on-chain, limit order book exchange in near real-time. It can sustain a maximum throughput of up to 1,500 transactions per second (scalable), is managed by a range of diverse participants who collectively confirm transactions and approve proposed amendments (decentralized) and utilizes consensus protocol that protects against attacks and failure modes (security). And XRPL does all this in a very sustainable way. In fact it is the world’s first major, global, carbon-neutral blockchain.

In Conclusion…

Developers need to find the best blockchain for their project requirements. Any misunderstanding around how the tech works hinders that process.

By diving into the nuts and bolts of how the XRPL operates, we’ve been able to debunk several myths and misconceptions about the ledger as well as review some of the advantages that make it ideal for a wide variety of projects. Hopefully, this overview will help you to look beyond the unfounded myths circulating about the XRPL to discover another viable tool for your needs.

For more information, head to XRPL.org, where you can find plenty of background information, documentation, updates on ongoing projects and an extensive FAQ.

Any other misconceptions you’ve seen that I didn’t address? Let us know in the comments right here.

A Vision for Federated Sidechains on the XRP Ledger

Matt Hamilton — Mon, 07 Jun 2021 19:38:55 +0000

Today, RippleX is rolling out a community page on DEV aimed at engaging the growing XRP Ledger Community.

Our inaugural post was written by David Schwartz, CTO at Ripple and one of the original architects of the XRP Ledger, and introduces his vision for Federated Sidechains.

Developers around the world are invited to share ideas, provide feedback, ask questions and join the discussion. Together, we look forward to building trust and utility for the XRP Ledger.

Over the last nine years, the XRP community has been committed to advancing the innovation and forward progress of the XRP Ledger (XRPL) to dramatically increase its decentralization, performance, and feature set.

Among the most-requested features we have heard from developers and contributors to the XRP Ledger is smart contract capabilities brought about by the exponential growth in decentralized finance (DeFi). In fact, the number of DeFi developers has grown 110% since 2019, and that number is projected to grow well beyond 2021. However, we at Ripple have long advocated against features that would compromise the XRP Ledger's highly efficient focus on payments.

Today, we are proposing a strategy that enables the best of both worlds: Federated Sidechains for the XRP Ledger. This will enable developers to implement new features, such as native smart contracts that interoperate seamlessly with XRP and the XRP Ledger, while also allowing the XRP Ledger to maintain its existing, "lean and efficient" feature set.

Federated Sidechains allow for experimentation and specialization, so developers can enjoy the power of the XRPL on a sidechain that acts as its own blockchain. For example, imagine the potential to branch out into new functionality by slimming down the XRPL's features to a specific subset for a particular use case—or even creating a private, parallel network for a permissioned blockchain. Federated Sidechains could very well make this a reality.

How It Works

In order to understand the vision for Federated Sidechains, it is first important to define a federator: a piece of software that connects to at least two instances of the XRPL software. The federator software means anyone who wanted to could run a sidechain to the XRP Ledger. On one side, the federator is connected to XRP Ledger Mainnet. On the other side, it connects to one or more sidechains. The federator would be run only by parties who operate validators on at least one sidechain.

The vision is that each sidechain would function as its own blockchain. They’d have their own ledger and transactions just as the XRP Ledger does. What makes them sidechains is the federation system which allows XRP and issued tokens to move from one chain to another.

Federated Sidechains could use XRP as their primary asset. In that case, people could use the federation system to move XRP from XRPL to the sidechain. Then, the moved XRP could be used on the sidechain just as it is on the main chain. Anyone could move XRP from either chain to the other.

Alternatively, sidechains could use their own native asset, so people with accounts on both ledgers could move XRP to and from the issued asset on the sidechain.

Functionality:

Federated assets imported onto XRPL itself would trade on the XRPL’s integrated decentralized exchange (DEX). XRP imported onto sidechains would be used for liquidity on their integrated DEX as well.

This strategy requires three things:

Building a new piece of software, or the “federator.”
Making two trivial changes to the operation of the live XRP Ledger network.
Adding new features to the XRPL server software to allow it to operate in a sidechain. However, these features would not be enabled on XRPL itself. (The current recommendation is to fork the XRPL software so that new versions of the sidechain software could come out without having to make new versions of the XRPL software and to reduce the risk of harming XRPL.)

Each sidechain would have a "trust" account on the XRPL Mainnet. This account can hold assets on the XRPL on behalf of users of the sidechain. The account would use a multisign or threshold key with the signers being the validators of the sidechain. Each sidechain validator operator registers a signing key that signs transactions on XRPL; thus, the validators of the sidechain can collectively create transactions to manage the sidechain's Mainnet account.

The XRP Ledger Mainnet has one native asset, XRP, and an unlimited number of issued tokens that can represent anything else but don't have the same status as XRP. It wouldn't make sense for each sidechain to start with a whole new set of 100 billion XRP, so instead, sidechains have two options for their native asset: either have a new native asset for the sidechain, or set aside some real XRP for use on the sidechain. If the sidechain uses XRP as its native asset, then the chain’s account on the Mainnet holds the sidechain's total amount of XRP "in trust" for use in the sidechain. If the sidechain creates a different native asset, that asset can be issued on XRPL Mainnet by the sidechain's Mainnet account.

The sidechain can hold other assets and tokens issued natively on the XRPL Mainnet; just like with XRP, the sidechain's Mainnet account holds the total amount in use on the sidechain. The ownership of that asset within the sidechain can change as a result of transactions and events in the sidechain that the XRPL Mainnet never needs to see. Whenever an asset—XRP or otherwise—needs to move "out of" the sidechain, the sidechain's Mainnet account sends that amount of XRP to its intended recipient on the Mainnet. This could even be another sidechain's account, allowing assets to cross from one sidechain through the Mainnet to any other sidechain. Conversely, to send funds "into" a sidechain, you would send funds to that sidechain's Mainnet account.

Someone who establishes a new sidechain should pick a set of initial validators and have them negotiate appropriate threshold or multi-signing keys. They would then create the sidechain’s XRPL Mainnet account and set it up so that only the sidechain validators' collective signing power can control that account. If the sidechain's validators change, then the Mainnet account should change its keys to match the new list of trusted validators. (Note: The XRP Ledger's native multi-signing lists are limited to 8 keys or fewer, but threshold keys can support as many signers as necessary for each of the sidechain's validators to be included).

Advantages

With this software, anyone can choose to run a sidechain to the XRP Ledger. For developers, it unlocks new use cases like native DeFi capabilities and smart contracts. Developers can also build and launch blockchain features that are “baked” into these sidechains; in the future, successful features could even be ported to the XRPL Mainnet.

The developers managing a sidechain also have the freedom to decide how their chains work. They would choose their own validators for their sidechain and could change the system’s rules as they need (with the cooperation of their sidechain's validators). For example, a sidechain could operate without transaction fees or reserve requirements, it could operate without its own copy of the XRP Ledger's decentralized exchange, or it could add new transaction types and functionality for storing large chunks of data on-ledger. The possibilities are limitless: a sidechain can be strictly permissioned or (nearly) permissionless, centralized or (mostly) decentralized. You could even run a sidechain temporarily while letting it manage real value and gracefully shut it down after it has served its purpose.

Immediate advantages of Federated Sidechains for developers include:

Horizontal scaling: Sidechains can have their own fee system, their own reserve system, and their own transaction capacity. Someone who wants to create a system with thousands of users that can hold XRP has a better option than being the custodian or putting all the accounts on XRPL directly.

Low risk: The XRP Ledger doesn't need to change at all. Even the changes that would be helpful are quite minimal.

Low effort: Anyone who needs or wants to experiment with a blockchain can get started with a complete system ready out of the box, based on powerful, stable, and sustainable XRP Ledger technology.

Long roadmap: New features can be added over a long period of time-based on feedback on what people find interesting. This would be a continuous stream of new features and capabilities.

Changes to the XRP Ledger

Succeeding in this vision requires a few changes to the XRPL software that wouldn’t be used on XRPL itself in order to support the sidechain features. The primary change to the software would be to support the unique node list (UNL) being stored in the ledger. Pseudo-transactions to change the UNL would be needed. A “hint” UNL would need to be supported to avoid the chicken and egg problem of needing the UNL to get the ledger and the ledger to get the UNL.

Support for the coordination of creating threshold and/or multisign keys and signing XRPL transactions introduced by the federator is also necessary. Some API enhancements would likely be needed to handle pseudo-transactions introduced by the federator or federator-federator communication through the peer network.

The XRP Ledger mainnet could also use a flag to indicate whether an issued asset was permitted to federate or not. Some asset issuers, for example, might insist that all holders of their assets be directly represented on the main chain for regulatory purposes while others could allow their assets to freely trade on sidechains. (It's always possible to privately allocate some of your own resources to others, with or without a sidechain to automate the process, but the legal responsibilities of doing so can vary based on jurisdiction and circumstances.)

Sidechains would have a special entry in their ledgers that tracks the last sidechain transaction that has been executed on the main chain and the last main chain transaction that has been executed on the sidechain.

When federators see a new transaction on the sidechain that affects the main chain, they coordinate the submission of that transaction to the main chain. When federators see a new transaction on the main chain that affects the sidechain, they coordinate the submission of that transaction to the main chain.

Making these changes is probably the biggest part of this effort because even though they won’t be enabled on XRPL, there is still risk associated with changing the software. For example, some existing code may need to be moved or adjusted which carries the risk of inadvertently changing behavior.

The outlined strategy is a starting point to gather feedback from the XRP Ledger community. We invite developers and contributors to the community to comment below. Let’s build a roadmap for innovative, new use cases together.

My Setup for Streaming Livecoding

Matt Hamilton — Fri, 18 Dec 2020 12:47:52 +0000

This is a video showcasing the setup I have for streaming live-coding sessions.

A 1080p version of this video is available here

In this video I cover three main areas:

What I want to achieve and why?

You've no doubt seen plenty of live-streamed videos with a screen capture taking up much of the screen and a small disconnected head in the bottom corner of the presenter talking away. My goal in this setup is to make my session much more engaging. One of the main advantages of streaming live-coding is the audience gets to not only see what you produce but how you go about it. What are your thought processes? What mistakes you make. How you go about solving a problem.

Also, as a developer presenting, I have to speak my mind out loud for the audience as I go along, and this helps me to reflect on my own understanding. Similar in fashion to "rubber duck" debugging.

In this setup, I try to achieve a much more engaging format by orienting the camera, the screen, and myself in such a way that my eye line follows the code. It is much more engaging for the audience that my eyes actually look towards the line of code I'm working on and talking about. It gives them a much better feeling of actually being there with me, looking over my shoulder as I'm coding.

I want to try and replicate a feeling a bit like the big transparent lightboards you see. These have a presenter behind a glass panel drawing on the panel with fluorescent markers and a camera on the other side. However, they require physically a lot of space and setup so not suitable to do in a home-office environment.

Physical Setup

I have my camera (Logitech c920) mounted at eye-level to the side of my monitor in such a way that the geometry of my eyes, the screen, and the camera produce the effect that I am looking sideways slightly, towards the code.

I have some home-made light panels that I use and have a pipe-frame on my desk that my monitor, lights, and cameras are all mounted to. This idea was taken from another developer advocate, Cassie. This means that I can raise my sit-stand desk to standing and everything goes up with it and is still at eye level.

I have an external USB microphone (AKG Perception 120) mounted on an arm that I can position just out of shot.

OBS Setup

I use OBS - Open Broadcaster Software to do my streaming. It allows you to create scenes with multiple input sources, such as screen capture and webcam feeds.

I have two monitors, the built-in screen on my Macbook and an external 1080p monitor above it. I arrange my screens such that what I am presenting (a terminal, a web browser, slides, etc) are all arranged on the lower screen and each full screen on a separate space. I can then use four fingers to swipe on the touchpad to move between them.

I have OBS and anything I don't want the audience to see on the top screen. So that includes my Slack workspaces, email, the Twitch channel management page, etc.

When presenting I am looking at the top screen, at the OBS preview which means my head is up and I'm not just looking down with the audience just seeing the top of my head. It also means I see the preview of the effect of myself behind the code when I point at the code, which means I can accurately look and point at the parts I'm talking about.

Anyway, enjoy the video and I hope it gives you some ideas on making your streaming more engaging.