DEV Community: Jhin Lee

Local speech with llamadart: giving Llama Village a voice

Jhin Lee — Wed, 14 Oct 2026 03:03:34 +0000

Beyond chat with llamadart · Part 3

Pip’s scarf is missing. I wanted Dash to be able to fly over to Mo and ask about it out loud.

Dash: “Mo, have you seen Pip’s red scarf?”

Mo: “Yes, it's in the reeds beside the pond.”

I tried that exchange with local models. For now, it runs in a small command-line program outside Llama Village. The question is a synthetic recording, and Mo gets the scarf’s location in his prompt. The game does not have voice controls yet.

The existing village. I’d like to let the player speak to these characters.

Two ways for Mo to listen

ASR is speech recognition. TTS is text to speech. Both routes finish with TTS. Concept illustration.

Via text: Qwen3-ASR transcribes the recording, a dialogue model answers, and Qwen3-TTS speaks. Having a transcript helps when “Pip” is misheard.

Direct audio: Qwen2.5-Omni receives the recording and writes an answer. This example has no separate transcript. The GGUF conversion accepts audio but does not generate it, so I still use TTS for Mo’s reply.

For the comparison, both routes use the same 3B dialogue model. In the first route it receives text; in the second it also loads its audio projector. This is one tested multimodal route, not a claim that all multimodal models need separate TTS.

I also tried a smaller pair for the text route: Qwen3.5 0.8B for dialogue and Qwen3-TTS 0.6B Base for speech. Qwen3-ASR 0.6B still handles recognition. The smaller pair is the default below; staged keeps the original 3B/1.7B pair for comparison. Qwen3.5 is used for text here, not audio input.

Try it on your Mac

The complete example is below. You need macOS, Dart 3.10.7 or newer, and about 2.31 GB for the five small-model files, plus space for Dart packages and the native runtime. I tested on an M4 Max with Dart 3.13.5 and llamadart 0.11.0. The commands use macOS say, afconvert, and afplay; they are not a phone or browser demo.

1. Create the project

Run this in a terminal. The Dart program uses pinned ModelSource.huggingFace references. Each engine’s load() uses llamadart’s download manager to fetch missing files and reuse its cache. There is no separate curl script. The first run needs a network connection and takes longer; subsequent generation uses the downloaded models locally.

Copy the setup commands

mkdir -p llama-voice-demo/bin
cd llama-voice-demo
cat > pubspec.yaml <<'YAML'
name: llama_voice_demo
publish_to: none
environment:
  sdk: ^3.10.7
dependencies:
  llamadart: 0.11.0
YAML
dart pub get

2. Add the complete program

Stay in llama-voice-demo and paste this block. It creates bin/voice.dart. The program loads one engine at a time and disposes it before loading the next.

Copy the full Dart program

cat > bin/voice.dart <<'DART'
import 'dart:io';

import 'package:llamadart/llamadart.dart';

const params = ModelParams(
  contextSize: 4096,
  numberOfThreads: 4,
  numberOfThreadsBatch: 4,
);

const sources = {
  'asr': (
    'ggml-org/Qwen3-ASR-0.6B-GGUF',
    '928ab958557df9aa2ef1c93e0e83c7ad0933fae2',
  ),
  'omni': (
    'ggml-org/Qwen2.5-Omni-3B-GGUF',
    '75f1b73b657a50f5092502799457ccb4a4a1f9df',
  ),
  'dialogue': (
    'unsloth/Qwen3.5-0.8B-GGUF',
    '6ab461498e2023f6e3c1baea90a8f0fe38ab64d0',
  ),
  'tts': (
    'ggml-org/Qwen3-TTS-12Hz-1.7B-Base-GGUF',
    'ca27d74bc954b73dadab5b71ca265d87fc861a7c',
  ),
  'smallTts': (
    'mradermacher/Qwen3-TTS-12Hz-0.6B-Base-GGUF',
    '0762b31e9cd99274d49dca4e895bd81863134f08',
  ),
};

ModelSource model(String family, String file) {
  final (repo, revision) = sources[family]!;
  return ModelSource.huggingFace(
    repoId: repo,
    revision: revision,
    filePath: file,
  );
}

Future<void> main(List<String> args) async {
  if (args.isEmpty ||
      args.length > 3 ||
      (args.length > 1 && !['small', 'staged', 'audio'].contains(args[1]))) {
    stderr.writeln(
      'Usage: dart run bin/voice.dart question.wav '
      '[small|staged|audio] [voice-reference.wav]',
    );
    exitCode = 64;
    return;
  }
  final question = args[0];
  final small = args.length == 1 || args[1] == 'small';
  final directAudio = args.length > 1 && args[1] == 'audio';
  final reference = args.length == 3 ? args[2] : null;
  for (final path in [question, if (reference != null) reference]) {
    if (!await File(path).exists()) {
      throw ArgumentError('Audio file not found: $path');
    }
  }

  String? transcript;
  if (!directAudio) {
    final asr = await SpeechToTextEngine.load(
      SpeechToTextModel(
        model('asr', 'Qwen3-ASR-0.6B-Q8_0.gguf'),
        projector: model('asr', 'mmproj-Qwen3-ASR-0.6B-Q8_0.gguf'),
        adapter: const Qwen3AsrAdapter(),
      ),
      params: params,
    );
    try {
      final result = await asr.transcribeOnce(
        SpeechToTextRequest(
          audio: SpeechAudioFileInput(question),
          contextPrompt: 'Mo, Pip, Dash, Llama Village',
        ),
      );
      transcript = result.text.trim();
      if (transcript.isEmpty) throw StateError('No speech recognized');
      stdout.writeln('Dash: $transcript');
    } finally {
      await asr.dispose();
    }
  }

  final llm = await LlamaEngine.load(
    LlamaModel(
      small
          ? model('dialogue', 'Qwen3.5-0.8B-Q4_K_M.gguf')
          : model('omni', 'Qwen2.5-Omni-3B-Q4_K_M.gguf'),
      projector: directAudio
          ? model('omni', 'mmproj-Qwen2.5-Omni-3B-Q8_0.gguf')
          : null,
    ),
    params: params,
  );
  final reply = StringBuffer();
  try {
    if (directAudio && !(await llm.capabilities).supportsAudio) {
      throw StateError('This model/runtime cannot accept audio');
    }
    await for (final chunk in llm.create(
      [
        const LlamaChatMessage.fromText(
          role: LlamaChatRole.system,
          text:
              'You are Mo, a friendly baker llama. Dash is visiting you. '
              "Known fact: Pip's red scarf is in the reeds beside the pond. "
              'Answer the visitor in one short sentence in English. '
              'No stage directions. Do not invent facts you do not know.',
        ),
        if (transcript != null)
          LlamaChatMessage.fromText(role: LlamaChatRole.user, text: transcript)
        else
          LlamaChatMessage.withContent(
            role: LlamaChatRole.user,
            content: [LlamaAudioContent(path: question)],
          ),
      ],
      enableThinking: false,
      params: const GenerationParams(maxTokens: 100, temp: 0, seed: 42),
    )) {
      reply.write(chunk.text);
    }
  } finally {
    await llm.dispose();
  }

  final text = reply.toString().trim();
  if (text.isEmpty) throw StateError('The model returned no reply');
  stdout.writeln('Mo: $text');
  final tts = await TextToSpeechEngine.load(
    TextToSpeechModel(
      small
          ? model('smallTts', 'Qwen3-TTS-12Hz-0.6B-Base.Q4_K_M.gguf')
          : model('tts', 'Qwen3-TTS-12Hz-1.7B-Base-Q4_K_M.gguf'),
      projector: small
          ? model('smallTts', 'Qwen3-TTS-12Hz-0.6B-Base.mmproj-Q8_0.gguf')
          : model('tts', 'mmproj-Qwen3-TTS-12Hz-1.7B-Base-Q8_0.gguf'),
      adapter: const Qwen3TtsAdapter(),
    ),
    params: params,
  );
  try {
    final audio = await tts.synthesizeOnce(
      TextToSpeechRequest(
        text: text,
        language: 'en',
        seed: 42,
        speakerReference: reference == null
            ? null
            : SpeechAudioFileInput(reference),
      ),
    );
    await File('reply.wav').writeAsBytes(audio.toWavBytes());
    stdout.writeln('Saved reply.wav');
  } finally {
    await tts.dispose();
  }
}
DART

3. Ask Mo a question

say -o question.aiff "Mo, have you seen Pip's red scarf?"
afconvert -f WAVE -d LEI16@16000 -c 1 question.aiff question.wav
dart run bin/voice.dart question.wav small
afplay reply.wav

This creates a synthetic question so you can try the whole path without a microphone package or a sample download. You can replace question.wav with your own short WAV. The program prints the transcript and answer, then saves reply.wav in the project directory. It does not record a microphone.

The main calls inside the program are:

// ASR: a recording becomes text.
final result = await asr.transcribeOnce(
  SpeechToTextRequest(
    audio: SpeechAudioFileInput(question),
    contextPrompt: 'Mo, Pip, Dash, Llama Village',
  ),
);

// TTS: the generated answer becomes a WAV file.
final audio = await tts.synthesizeOnce(
  TextToSpeechRequest(text: text, language: 'en', seed: 42),
);
await File('reply.wav').writeAsBytes(audio.toWavBytes());

These excerpts sit inside the loaded engines’ try/finally blocks in the full program. Between them, llm.create answers using the transcript and Mo’s supplied fact.

Try the same question as audio

Change the mode to download and load Omni’s audio-capable model and projector, plus the original 1.7B TTS pair:

dart run bin/voice.dart question.wav audio
afplay reply.wav

That route skips ASR and sends LlamaAudioContent(path: question) to the dialogue model. The program checks supportsAudio before asking it to answer. The files used by this route total about 5.13 GB; keeping all three modes cached takes about 7.44 GB.

In the game, Mo should only receive facts he is allowed to reveal. DecisionEngine could choose an allowed intention before dialogue generation. A friendly persona prompt does not enforce those rules.

Make it sound like Mo

A possible production workflow: prepare a voice, save a reference, and reuse it for new dialogue. Concept illustration.

I’d like Mo to sound warm and unhurried. Dash could be brighter and quicker. A short reference clip lets several characters share one TTS model while sounding different. When the player speaks as Dash, they already provide his voice.

The full program accepts an optional third argument. Use small for the 0.8B/0.6B pair, or staged for the original 3B/1.7B pair:

dart run bin/voice.dart question.wav small mo-reference.wav
afplay reply.wav

Use a short, clear recording you have permission to use. For my test I made a synthetic reference with the macOS Daniel voice. Inside TextToSpeechRequest, the program passes it as speakerReference: SpeechAudioFileInput(reference).

Qwen’s voice-design workflow also lets you create a voice during development, then use the resulting clip with a Base model. Players would download the synthesis model and reference, without needing the voice-design model. I have not tested that workflow here.

A reference gives Mo a voice, but the tested API has no instruction such as “say this nervously.” That is not just a missing Dart parameter. Qwen’s 1.7B CustomVoice and VoiceDesign models support instruction control; the Base model used here does not offer that same mode. Exposing it would need a compatible model, native runtime support, and a Dart API. Changing the dialogue prompt only changes the words.

Smaller models, smaller downloads. Faster conversations?

I tested the smaller pair on the same M4 Max (64 GB), with llamadart 0.11.0. Both routes kept Qwen3-ASR 0.6B and used the same synthetic question and Daniel voice reference.

Text route	Model files	Complete turn, median	Peak process RSS
Omni 3B + TTS 1.7B	4.61 GB	4.27 s	3.86 GiB
Qwen3.5 0.8B + TTS 0.6B	2.31 GB	5.83 s	2.51 GiB

Half the download, about 35% less peak RSS, but a slower turn in this pilot. The smaller dialogue model gave a longer reply:

3B: “Yes, Pip's red scarf is in the reeds beside the pond.”

0.8B: “Yes, I saw it! It was tucked inside the reeds by the pond.”

The 0.8B model also ignored the one-sentence instruction. More words mean more speech to generate. These are complete application turns with different replies, so I also tested the components separately:

Loaded component	Generation time, median	Peak process RSS
Omni 3B dialogue	171 ms	2.57 GiB
Qwen3.5 0.8B dialogue	96 ms	1.08 GiB
Qwen3-TTS 1.7B	1.78 s	3.85 GiB
Qwen3-TTS 0.6B	1.78 s	2.48 GiB

For TTS, both models spoke exactly “Yes, Pip's red scarf is in the reeds beside the pond.” The 1.7B clips lasted 4.16–4.56 seconds; the 0.6B clips lasted 4.80–6.48 seconds. The smaller model’s synthesis times ranged from 1.78 to 2.53 seconds. Smaller did not mean a consistently shorter wait.

Did Mo say the right thing?

Both dialogue models supplied the location and colour correctly when asked. But when I asked “Who put the scarf in the pond?”, both blamed Pip. That fact was never supplied, even with “Do not invent facts you do not know” in the prompt. The smaller model’s “I saw it” also adds an unsupported claim about how Mo knows.

All six measured TTS clips transcribed back to the intended sentence with Qwen3-ASR. That is an intelligibility check, not a listening score for naturalness, emotion, or similarity to Daniel. I would still listen to more lines before choosing a character voice.

I would use the smaller pair for this prototype’s storage budget. I would keep the game’s knowledge rules outside the prompt, and measure real conversations before claiming a latency win.

Test conditions and model files

This is a small development-machine pilot, not a device benchmark. Other development workloads were active; the village renderer was not. The complete-turn test alternated the two setups, discarded the first turn for each, then measured three fresh-process turns per setup. Baseline turns ranged from 3.76 to 4.41 seconds; smaller-pair turns ranged from 5.82 to 5.91 seconds. Timing includes model loading and disposal, but excludes process startup, downloads, recording, and playback. Peak RSS was sampled every 100 ms and is the largest observed peak across the three runs. It is process memory, not total GPU or device memory.

Component tests loaded each model in its own process and discarded one warm-up. Dialogue used three questions: the scarf’s location, who put it there, and its location plus colour. TTS repeated the same sentence three times with the same reference. Component times exclude loading and disposal; memory uses the process high-water RSS. Both dialogue models used temperature 0, seed 42, a 100-token limit, thinking disabled, a 4096-token context, and four threads. TTS used seed 42; its waveforms still varied between calls.

The dialogue weights are Q4_K_M; audio projectors are Q8_0. The new files are Unsloth’s Qwen3.5 0.8B conversion and mradermacher’s Qwen3-TTS 0.6B Base conversion. The exact revisions and matching projector filenames are pinned in the program above. A GGUF file made for a different TTS runtime is not automatically interchangeable.

Let the player read while Mo gets ready

This TTS API returns the complete audio before playback. Its progress events are not playable audio chunks. I’d show Mo’s reply as text immediately, then play it when synthesis finishes.

For a first game version, I’d use push-to-talk, stop recording while Mo speaks, and discard an old reply if the player starts another turn. Text should remain available when voice is switched off or a model cannot load. I still need to measure speech alongside rendering before adding this to the game.

For model pairs and platform requirements, see the speech recognition guide and speech synthesis guide. Partial live captions use a different recognition path from this file-based example.

Local embeddings with llamadart: how llamas share secrets

Jhin Lee — Mon, 12 Oct 2026 18:18:49 +0000

Beyond chat with llamadart · Part 2

Llama Village, where five llamas share a world without sharing everything they know.

Pip’s red scarf is missing. Mo knows where it is. He also knows how it got there—and would rather keep that part to himself.

That gives the village a small mystery. It gives the game a harder question: when two llamas talk, what should the listener actually learn?

I introduced the game in Llama Village: a virtual world powered by local AI with llamadart. Here, we will follow one part of its machinery: local embeddings help compare what a llama said with facts that might have been shared.

The destination is a change you can inspect: a llama acquires information, the game records who told them, and they may still doubt it. Dialogue, embeddings, and game rules each have a different job.

A secret starts with one llama

The game stores the scarf’s location and Mo’s accident as separate facts. At the start, Mo knows both. Knowing where the scarf is does not automatically mean knowing who knocked it into the pond.

Before checking a conversation, the game narrows the possibilities: facts the speaker knows and the listener does not. It compares each fact with that speaker’s lines, not every line from both characters.

An embedding model has learned text representations from training data. It maps each line and fact to a numeric vector; cosine similarity compares those vectors. Similar wording and related meaning can bring vectors closer, but the score is not a yes/no test for identical meaning, truth, or whether a statement implies a fact. The game uses each fact’s highest similarity across the speaker’s lines.

AI-generated overview: compare meaning, check evidence, then record who said it. A conceptual illustration, not a captured game trace.

The implementation uses EmbeddingGemma through llamadart and caches vectors by text. It applies title: none | text: to both lines and facts. This walkthrough reproduces that game choice. Google’s model card recommends task-specific prefixes for new applications, including a sentence-similarity query prompt; changing the format requires reevaluating the scores and thresholds.

The words do not have to match

June and Bramble talking about poems in a recorded playthrough. This shows the dialogue system, not the scarf exchange illustrated below.

For a small walkthrough, compare the stored fact “Pip’s red scarf is lying in the pond reeds” with the illustrative line “I saw Pip’s scarf tangled in the reeds by the pond.” They describe related information using different words. The line is written for this explanation, not quoted from the recording.

The input is text. The embedding output is a numeric vector. The application then calculates a similarity score; the model does not return “June learned the scarf’s location.” That decision comes later.

Keep the same embedding model and compatible settings for both texts. Equal vector lengths alone do not guarantee compatible representations.

Follow the text through llamadart

With a suitable embedding engine loaded, the core calls look like this. The full loading and comparison example is at the end.

final facts = [
  "Pip's red scarf is lying in the pond reeds.",
];
final factVectors = await engine.embedBatch(
  facts.map((text) => 'title: none | text: $text').toList(),
  normalize: true,
);
final lineVector = await engine.embed(
  "title: none | text: I saw Pip's scarf tangled "
  "in the reeds by the pond.",
  normalize: true,
);

The game’s configured embedding artifact is embeddinggemma-300M-Q8_0.gguf. For another model, follow its input-format requirements. The llamadart embedding guide covers loading, backend support, and input limits.

For normalized vectors, a dot product gives cosine similarity. In the game, keyword evidence adds a ranking bonus of 0.3, and the best six candidates go to the outcome model. These are game-specific choices, not standard embedding thresholds.

I ran the companion example using published llamadart 0.11.0 and the local embeddinggemma-300M-Q8_0.gguf artifact on macOS arm64. It returned 768-dimensional vectors. Here are selected cosine similarities from that run; these are illustrative inputs run through the real model, not dialogue captured during gameplay.

Input line	Compared with	Cosine similarity
I saw Pip’s scarf tangled in the reeds by the pond.	Scarf location	0.8449
I did not knock the scarf into the pond.	Mo’s accident	0.6085
The sky looks beautiful today.	Scarf location	0.3137

The denial is the revealing case: it remains similar to the accident fact despite saying the opposite. These scores belong to this artifact and formatting; they are not universal cutoffs or probabilities.

Who decides whether the meaning matches

Embeddings do not first detect a topic and then prove matching meaning. They produce one representation and a similarity score. To ask “did this speaker actually reveal that fact?”, Llama Village sends the conversation and shortlisted facts to a separate generative outcome model. That model is prompted to return the IDs whose content was clearly said aloud.

Illustrative alternatives, not a recorded exchange: mentioning the same scarf and pond does not make a denial a confession. AI-generated illustration; no similarity scores are implied.

For the illustrative denial “I did not knock the scarf into the pond,” the intended interpretation is that the accident was not confessed. The outcome model must interpret the wording and context to reach that conclusion; it can be wrong. Its JSON constrains the shape and allowed IDs of the response, not whether the interpretation is correct.

The code then combines the model’s proposed revelations with keywords and similarity. The location secret requires “scarf” and a location term. The accident secret requires “scarf” and a responsibility term such as “knock.” Crucially, “I did not knock the scarf into the pond” still matches those substring groups. They do not understand the negation.

There is a real weakness in the current game rule. For a secret, matching keywords plus similarity ≥ 0.5 can accept a transfer even when the outcome model does not claim it. Feeding the measured denial score of 0.6085 into the game’s actual keywordHit and acceptTransfer functions returned accepted=true with claimed=false. This is a rule-level reproduction, not a complete gameplay run: the fact must still be eligible and reach the shortlist.

The current rules therefore do not guarantee matching meaning. A stricter design would remove that similarity-only fallback for secrets and require an explicit revelation judgment with supporting dialogue.

One possible next step is llamadart’s DecisionEngine, introduced with Laya in Can a local AI model learn to play Tetris?. Here, the choice would be about Mo’s words: given the dialogue and a candidate fact, does he reveal it, deny it, or leave it unclear?

Embeddings shortlist → DecisionEngine judges → game applies the rules.

Only a “revealed” decision would let the fact proceed through the game’s knowledge-transfer checks; similarity alone would no longer accept it. This is a proposed extension, not the current implementation. A structured answer can still be wrong: we would need to evaluate Laya on denial, speculation, quotation, and ambiguous dialogue—and potentially tune a dedicated head—before claiming it improves on the current outcome model.

Current secret transfer check	Required evidence
Keywords	Every keyword group matches
Additional support	Outcome model claims it or similarity ≥ 0.5
Known weakness	The similarity branch can bypass an unclaimed revelation

June hears the story and still doubts it

Look at June’s inspector: Mo’s scarf accident is marked “heard from Mo; maybe untrue.” The screenshot shows a stored outcome, not a captured embedding score or proof of the exact dialogue that produced it.

After accepting a transfer, the game records the teller and separates having heard information from believing it. Belief depends on the teller’s belief and the listener’s trust rules. That distinction lets a conversation change the village without making every listener accept every story.

Give your app the same separation

The reusable pattern is: narrow candidates with application state, use embeddings to find plausible matches, and give consequential state changes a separately evaluated decision. Llama Village demonstrates the pattern and a failure mode: a similarity fallback can undo the distinction between finding related text and accepting a revelation.

Keep sources and provenance alongside vectors. Rebuild cached vectors when the model or its input formatting changes. Choose an embedding model that fits the runtime, and check backend support after loading.

Pip’s missing scarf makes the technical question visible: did someone say where it was, admit how it got there, deny it, or just mention the pond? Test those separately. A convincing conversation and a correct state update are two different things to evaluate.

For the rest of the world, including Dash, the Berry Festival, and the storybook ending, see the Llama Village article. Next in this series: local speech.

Complete embedding comparison example

This example was analyzed and executed with Dart 3.13.5 and published llamadart 0.11.0 on macOS arm64. Create a Dart console project, pin llamadart: 0.11.0 in its pubspec.yaml, run dart pub get, and save the code as bin/village_embeddings.dart. Use a local EmbeddingGemma Q8_0 GGUF and run the command below. The two prefixes intentionally reproduce the game’s format. This compares five fixed facts; it does not implement character eligibility, the outcome model, or knowledge updates.

dart run bin/village_embeddings.dart \
  /absolute/path/to/embeddinggemma-300M-Q8_0.gguf \
  'title: none | text: ' 'title: none | text: '

import 'package:llamadart/llamadart.dart';

Future<void> main(List<String> args) async {
  if (args.length != 3) {
    throw ArgumentError(
      'Usage: dart run village_embeddings.dart model.gguf '
      '"query prefix" "document prefix" (use "" for no prefix)',
    );
  }

  const facts = [
    'Pip\'s red scarf is lying in the pond reeds.',
    'Mo knocked Pip\'s red scarf into the pond reeds by accident and never told her.',
    'Bramble writes anonymous love poems to June.',
    'Pip\'s red scarf has gone missing from her hut.',
    'June made up the rumour that Mo\'s bread made Bramble sick.',
  ];
  final engine = await LlamaEngine.load(
    LlamaModel(ModelSource.path(args[0])),
  );
  try {
    if (!(await engine.capabilities).supportsEmbeddings) {
      throw StateError('The active backend does not support embeddings.');
    }
    // Index these once; reuse them for subsequent searches.
    final vectors = await engine.embedBatch(
      facts.map((text) => '${args[2]}$text').toList(),
      normalize: true,
    );
    for (final query in [
      'I saw Pip\'s scarf tangled in the reeds by the pond.',
      'I did not knock the scarf into the pond.',
      'The sky looks beautiful today.',
    ]) {
      final vector = await engine.embed(
        '${args[1]}$query',
        normalize: true,
      );
      final ranked = [
        for (var i = 0; i < facts.length; i++)
          (text: facts[i], score: dot(vector, vectors[i])),
      ]..sort((a, b) => b.score.compareTo(a.score));
      print('\n$query — ${vector.length} dimensions');
      for (final hit in ranked.take(3)) {
        print('${hit.score.toStringAsFixed(4)}  ${hit.text}');
      }
    }
  } finally {
    await engine.dispose();
  }
}

double dot(List<double> a, List<double> b) {
  if (a.length != b.length) {
    throw ArgumentError('Vector dimensions differ.');
  }
  var result = 0.0;
  for (var i = 0; i < a.length; i++) {
    result += a[i] * b[i];
  }
  return result;
}

Implementation references: candidate selection and acceptance and embedding formatting.

DecisionEngine: Teaching local AI to play Tetris

Jhin Lee — Sat, 10 Oct 2026 00:23:42 +0000

Beyond chat with llamadart · Part 1

A piece is falling. There are a few places it could land. Somewhere in the background, a local AI model is choosing one.

If it takes too long, the game carries on without it.

The tuned player in an actual browser run: 51 lines cleared using WebGPU, Q8_0, and mixed candidates. This illustration is separate from the recorded benchmarks below.

Try the browser demo · Open the example

This opens Beyond chat with llamadart, a five-part series about cross-platform local AI for Dart and Flutter. We’ll explore decisions, embeddings, speech, images, and an application that connects them.

Tetris gives the experiment a visible consequence: the model picks a landing, and the board changes.

The game already has a useful heuristic. What does adding a decision model buy us? Tetris lets us compare different questions, training, and the cost of waiting for an answer.

Give the model a smaller job

The planner builds the options without AI. From the active piece’s current pose, it tries rotations, horizontal shifts, and a hard drop, rejecting blocked paths.

Each candidate is a Move: a reachable landing, the key sequence that gets there, and the resulting board measurements. Duplicate landings keep the shortest discovered sequence.

The game simulates each landing to count cleared lines, holes, stack height, and bumpiness. describeCompact() turns those consequences into the text sent to Laya.

The model chooses among descriptions of these planned moves. It does not generate the keys.

Three illustrative landings, with B shown as an example answer. Laya receives descriptions, not board images. Game code validates and executes the move.

Gravity keeps running. A target can become unreachable before the answer arrives, and the game records those failures.

Where Jev comes in

TypeSafe’s Jev gives this approach a useful vocabulary. TypeSafe calls it System One: provide a state and typed questions, then use the structured answers directly in code.

Laya is a separate open-weight model with a Jev-compatible interface. Llamadart follows Jev’s vocabulary and Laya’s request format; this experiment runs Laya locally.

The results here measure Laya and its Tetris-tuned head, not Jev.

Make your first decision with llamadart

A ModernBERT encoder runs through llama.cpp, and a decision head produces the answer. Each question takes an encoder pass without generating prose.

One question through the local pipeline. DecisionEngine prepares the tokens and decodes the model’s scores. The game continues running while it awaits the result.

ModernBERT reads the question, options, and state together. The head scores the positions marking each option; the decoder turns those scores into probabilities. Tuning changes the head while the encoder stays frozen.

Load a model and ask for a move

Add llamadart to a Dart project with dart pub add llamadart, save the following as bin/decision_walkthrough.dart, and run dart run bin/decision_walkthrough.dart on a supported native target. The documented decision path is validated on macOS; consult the support matrix for other targets. The first run downloads a roughly 421 MB encoder and 106 MB head. This standalone example prints one decision; it does not launch the Tetris game.

import 'package:llamadart/llamadart.dart';

Future<void> main(List<String> args) async {
  final useTunedHead = args.contains('--tuned');
  const repo = 'fr0stbit3/laya-gguf';
  const revision = 'ce2afdc0a8766af56a29a22dcf4a781e1f5c7d3c';

  final decisions = await DecisionEngine.load(
    DecisionModel(
      encoder: ModelSource.huggingFace(
        repoId: repo,
        revision: revision,
        filePath: 'laya-Q8_0.gguf',
      ),
      head: ModelSource.huggingFace(
        repoId: useTunedHead ? 'leehack/laya-tetris-head' : repo,
        revision: useTunedHead
            ? '465546a595ee2e8e3b212b8cb16829205d5dfab6'
            : revision,
        filePath: useTunedHead
            ? 'laya-head-tetris.safetensors'
            : 'laya-head.safetensors',
      ),
    ),
    params: const DecisionModelParams(
      device: ComputeDevice.auto,
      threads: 4,
    ),
  );

  try {
    // Illustrative descriptions, not a captured game turn.
    const candidates = <String, String>{
      'A': 'columns 1 to 2: clears zero lines, no new holes, '
          'stack 5 rows, bumpiness 8',
      'B': 'columns 5 to 6: clears one line, no new holes, '
          'stack 3 rows, bumpiness 4',
      'C': 'columns 9 to 10: clears zero lines, adds one hole, '
          'stack 6 rows, bumpiness 10',
    };

    final result = await decisions.systemOne(
      state: {'game': 'tetris', 'piece': 'O'},
      questions: {
        'move': DecisionQuestion.choice(
          'Which placement of the falling Tetris piece is best?',
          criteria: candidates,
        ),
      },
    );

    final ChoiceAnswer answer = result.choices['move']!;
    final String selectedDescription = candidates[answer.choice]!;

    print('Choice: ${answer.choice}');
    print('Probabilities: ${answer.probabilities}');
    print('Candidate: $selectedDescription');
  } finally {
    await decisions.dispose();
  }
}

DecisionModel identifies the encoder and head. load prepares their files and owns both resources. systemOne answers the named questions about one state; criteria maps each candidate label to its description.

Read the typed answer

result.choices['move'] returns a ChoiceAnswer. Its choice is the selected label, and probabilities contains every option’s probability. Here is an illustrative rendering of those fields, not captured output; your run may choose a different option.

{
  "choice": "B",
  "probabilities": {
    "A": 0.15,
    "B": 0.75,
    "C": 0.10
  }
}

The example uses candidates[answer.choice] to recover the selected description. In the game, that same lookup leads back to a Move with its landing and planned keys. The tuned player chooses from the probabilities itself so it can break ties randomly; ChoiceAnswer.choice uses the first highest-probability option. A probability of 0.75 is not a 75% guarantee of success.

From B to button presses

The mapping stays in game memory:

B → original candidate → Move {
  landing,
  planned keys,
  board measurements
}

Before pressing anything, the game checks that it is still controlling the same piece. Gravity has continued during inference, so it runs the planner again and finds a route to B’s landing from the current pose. It queues that route and presses one action at the configured interval.

If the piece has already locked, the answer is discarded. If the landing is now blocked, the game records the failure and hard-drops. The execution code handles those cases.

The API supports three kinds of question:

Kind	Result	Example
Choice	A selected option and option probabilities	Which landing should we use?
Score	An expected value across ordered levels	How urgent is this request?
Noul	Probability of a yes/no statement	Does this move clear a line?

“Noul” keeps the existing Jev and Laya vocabulary. A score can fall between levels, and a probability is not a boolean until the application applies a rule.

A chat model could also choose a move. There is no matched chat-model benchmark here, so this experiment cannot establish an advantage over LLMs.

Two settings to keep separate

Setting	What it changes	Options
Player	How a move is selected	Human, heuristic, random, Laya judge, checklist, base choice, tuned choice
Candidates	Which planned moves are considered	3 best + 3 random, 6 best, 6 random, all generated legal moves

The heuristic player considers all generated legal moves. Base choice accepts at most six, so “all” falls back to the mixed shortlist for that player. Tuned choice handles larger sets through knockout rounds. The comparisons below use the mixed shortlist unless stated otherwise.

First result: one line

Give the base head six candidates and ask it to choose. In the short recorded benchmark, that player clears an average of one line. Random choice also clears one.

Asking “is this a good move?” about each candidate brings the mean to 18 lines.

A narrower checklist reaches 35 lines: ask whether a move clears a line and whether it creates holes, then subtract the second probability from the first.

The game already knows whether a move clears a line or creates a hole. The checklist tests how well Laya interprets those facts from text; an application could read the computed values directly. Six candidates can mean twelve model questions.

That improvement costs work. In this implementation, batching the questions into a call does not turn them into one encoder pass.

All three receive a shuffled shortlist: the heuristic’s three best moves plus three random remaining moves. Could training make the single-choice approach useful too?

Let the heuristic be the teacher

Only the decision head is tuned; the encoder stays frozen. Run dart run bin/decision_walkthrough.dart --tuned to try the pinned Tetris head with the same encoder. One sample decision is a smoke check; the gameplay comparison comes later.

The dataset has 16,000 training questions and 2,000 validation questions from other games. Labels come from the heuristic; tied best moves share the target probability.

On the held-out questions, the base head scored 30.5% accuracy. A random pick scored 27.5%; the task can have multiple equally best options, so this is not simply one chance in six.

The published head reaches 75.7%. Eight documented 12-epoch runs ranged from 75.5% to 78.5%. The training recipe includes the full results.

This measures imitation of the heuristic, not optimal play. MPS training was not bit-for-bit repeatable.

A useful control: the base head still played roughly randomly when given the tuned question format. New wording alone did not explain the improvement.

Train a head of your own

Training happens in the Python notebook; DecisionEngine loads and runs the resulting head. From the prepared example/laya_tetris project, generate the seeded dataset:

dart run bin/make_dataset.dart dataset

Then run training/laya_head_tuning.ipynb using the documented Python environment. The notebook keeps the encoder frozen, trains the head, evaluates held-out questions, and writes laya-head-tetris.safetensors. Its embedded configuration lets llamadart load it without a separate config file.

In compare_heads.dart, replace the tunedHead: source with ModelSource.path('training/laya-head-tetris.safetensors'), adjusting the path to your exported file. For the first walkthrough, use the same source as DecisionModel.head. Keep the matching encoder; the Dart API runs the head after Python training.

Compare two heads on one encoder

For an app that switches between base and tuned players, load the encoder once and attach both heads. The essential calls are below. The complete runnable comparison includes the pinned model sources, request, main() invocation, and cleanup, including partial loading failures. Save it as bin/compare_heads.dart in the same Dart project and run dart run bin/compare_heads.dart.

// Excerpt from compare_heads.dart; setup and cleanup are in the file.
final engine = await LlamaEngine.load(
  LlamaModel(encoder),
  params: const DecisionModelParams().encoderModelParams,
);
final base = await DecisionEngine.attach(engine, head: baseHead);
final tuned = await DecisionEngine.attach(engine, head: tunedHead);

final baseResults = await base.systemOneBatch([request]);
final tunedResults = await tuned.systemOneBatch([request]);
print(baseResults.single.choices['move']!.probabilities);
print(tunedResults.single.choices['move']!.probabilities);

DecisionRequest packages a state and its questions for systemOneBatch. Results retain request order; .single reads this one-request comparison. For a larger comparison, supply several requests to each head. Each question still needs its own encoder pass. Attached heads borrow the encoder, so dispose the heads first, then the encoder. In a game, keep them loaded across turns.

Back to the board

The tuned choice player clears 48 lines and reaches the 150-piece cap in every game. Then the plain heuristic clears 56.

Recorded turn-based results on M4 Max, Metal, Q8_0. Model and random rows use 3 best + 3 random candidates; the heuristic () considers all legal moves. These are descriptive means without per-seed intervals. Source: Tetris benchmark.*

The model learned something useful. The teacher still leads.

For six candidates, the tuned player needs just one question.

The heuristic considers all legal landings; the model’s shortlist already includes heuristic-selected moves. This is not a model learning Tetris unaided.

The board does not wait

In a live game, decision quality includes arriving on time. A stronger answer that arrives after the piece locks cannot help that move.

Jev’s documented hosted API involves a network request. Running locally removes that round trip and its variability. Whether it is faster overall still depends on the device, model, and workload; this experiment does not include a matched Jev benchmark.

That gives task-specific training a practical purpose. A small local model does not have to be the strongest general model to be useful. It needs to make sufficiently good decisions within the application’s deadline. Here, tuning improves the single-choice player while retaining the same encoder and decision-head architecture.

One six-option, 175-token question took about 20 milliseconds on Metal in the recorded M4 Max measurements.

Observed ranges across three runs per setting, using Q8_0 and a fresh engine for each setting. Model preparation is excluded; these are question timings, not full game-turn latency.

More candidates mean more work. With all legal moves, the tuned player runs knockout rounds in groups of up to six.

Across 40 documented real-time games, the tuned head averaged 52.3 lines with the mixed shortlist and 51.5 with all candidates. Every answer arrived before the piece locked.

These live games and the earlier capped tests have different conditions. Their scores belong to separate experiments.

Does clearing lines prove it works

It proves the player can clear lines. The integration needs other checks.

The validation record compares the implementation with Laya 0.3.5. A 24-question fixture showed no changed decisions across the measured configurations.

A broader 187-question set caught a Q8_0 CPU yes/no probability changing from 0.694 to 0.457, crossing the 0.5 threshold. F32 had no changed decisions on that set.

A smaller model representation needs its own evaluation on the questions your app asks.

Game tests cover rules, planning, candidate selection, and loading failures. Reference checks cover numerical agreement. Neither is replaced by a high game score.

Take the pattern into your app

Load once at application startup, reuse the model for decisions, and dispose it when the owning feature closes. Keep model preparation out of the game’s per-piece loop.

The decision-model guide covers typed keys, loading options, and runtime support when you move beyond this example.

Outside Tetris, the same pattern might choose a support category, route a request, or select among actions allowed by a simulation. Use clear descriptions and keep the available actions bounded by application state.

The current Laya format has a 512-token sequence limit and a smaller shared question-and-option budget. Large descriptions can be truncated. Confidence is derived from the returned distribution; it is not a calibrated guarantee of correctness.

Try changing the player

If the only goal were to build the strongest player in this comparison, the heuristic would be the obvious starting point.

The progression from one line to 48 makes local specialization worth exploring. Training improved this narrow task without requiring a larger inference architecture. The practical question is whether that quality is useful within the target device’s latency budget, with a simple heuristic beside it to keep the comparison honest.

Llamadart provides a way to explore that through Dart and Flutter, with a native llama.cpp path and a compatible WebGPU path. The measured M4 Max results should stay attached to that hardware and setup rather than becoming a claim about every phone or browser.

Try the Tetris demo, switch between the base and tuned players, and watch both the board and the decision panel. The failed moves are part of what makes the experiment worth looking at.

Next in the series: working with local embeddings.

Complete head comparison

Save this as bin/compare_heads.dart in your Dart project and run dart run bin/compare_heads.dart. The candidates are illustrative, as in the first walkthrough.

import 'package:llamadart/llamadart.dart';

Future<void> main() async {
  const repo = 'fr0stbit3/laya-gguf';
  const revision = 'ce2afdc0a8766af56a29a22dcf4a781e1f5c7d3c';
  await compareHeads(
    encoder: ModelSource.huggingFace(
      repoId: repo,
      revision: revision,
      filePath: 'laya-Q8_0.gguf',
    ),
    baseHead: ModelSource.huggingFace(
      repoId: repo,
      revision: revision,
      filePath: 'laya-head.safetensors',
    ),
    tunedHead: ModelSource.huggingFace(
      repoId: 'leehack/laya-tetris-head',
      revision: '465546a595ee2e8e3b212b8cb16829205d5dfab6',
      filePath: 'laya-head-tetris.safetensors',
    ),
    request: DecisionRequest(
      state: {'game': 'tetris', 'piece': 'O'},
      questions: {
        'move': DecisionQuestion.choice(
          'Which placement of the falling Tetris piece is best?',
          criteria: {
            'A': 'columns 1 to 2: clears zero lines, no new holes, stack 5 rows, bumpiness 8',
            'B': 'columns 5 to 6: clears one line, no new holes, stack 3 rows, bumpiness 4',
            'C': 'columns 9 to 10: clears zero lines, adds one hole, stack 6 rows, bumpiness 10',
          },
        ),
      },
    ),
  );
}

// Illustrative candidate descriptions, not a captured game turn.
Future<void> compareHeads({
  required ModelSource encoder,
  required ModelSource baseHead,
  required ModelSource tunedHead,
  required DecisionRequest request,
}) async {
  final engine = await LlamaEngine.load(
    LlamaModel(encoder),
    params: const DecisionModelParams().encoderModelParams,
  );

  DecisionEngine? base;
  DecisionEngine? tuned;
  try {
    base = await DecisionEngine.attach(engine, head: baseHead);
    tuned = await DecisionEngine.attach(engine, head: tunedHead);

    final baseResults = await base.systemOneBatch([request]);
    final tunedResults = await tuned.systemOneBatch([request]);

    print('Base: ${baseResults.single.choices['move']!.probabilities}');
    print('Tuned: ${tunedResults.single.choices['move']!.probabilities}');
  } finally {
    try {
      await tuned?.dispose();
    } finally {
      try {
        await base?.dispose();
      } finally {
        await engine.dispose();
      }
    }
  }
}

Llama Village: a virtual world powered by local AI with llamadart

Jhin Lee — Thu, 08 Oct 2026 22:34:10 +0000

Llama Village in motion. The characters’ dialogue is generated by an on-device model.

Watch the full trailer on GitHub

I wanted to try building a 3D game with flutter_scene. I was already working on llamadart, so I thought it would be fun to use local AI in the game too. My starting idea was fairly loose: make it look good, keep it simple, and put Dash, the Flutter mascot, in a scene with some llamas.

I wanted to show different kinds of AI working together, including multiple characters acting on their own. But I did not have a game concept I liked yet.

Finding a reason for the AI

I spent quite a while on the visuals. I tried a hand-drawn look, asked for something closer to Borderlands, and eventually switched to a softer film style because the earlier renders kept looking glitchy. Meanwhile, we tried several game ideas. I was not happy with how they played.

These were two of my messages in the Claude Code session:

“Btw the game isn't fun.”

“What's the point of ai?”

Eventually I asked for a village where the llamas would live and interact on their own. I wanted to give them personalities, see their thoughts and conversations, and control Dash to interact with them. That was the idea we went ahead with.

The village at golden hour, with the bakery in the centre and the festival stage in the foreground.

Who lives here?

Pip, Mo, June, Bramble, and Clover each have a personality, needs, relationships, goals, and things they know. They make plans, walk around the village, and stop to talk. A missing scarf, a bread rumour, a secret crush, and the upcoming Berry Festival give them things to care about.

June and Bramble talking. Each line is written from the speaker’s situation and the conversation so far.

They do not all know the same things. In the inspector below, June knows that Mo knocked Pip’s scarf into the pond because she heard it from Mo. Other entries are things she saw herself. The game keeps track of both the information and how she got it.

This is what I mean by a multi-agent system in this game. Each llama has its own knowledge, goals, and relationships. They use the same underlying language model, with different context for each character.

The inspector shows what June knows and who told her.

How a conversation changes the village

I wanted to try more than one kind of AI. The language model writes dialogue and thoughts. Embeddings help compare the meaning of the dialogue with facts the listener could have learned. An optional Laya decision model helps choose casual conversation topics from the options the game allows.

Movement, needs, knowledge tracking, and endings are handled in code. After a conversation, the model reports which facts were mentioned and suggests changes to mood and friendship. The game checks the fact claims before accepting them.

A llama knows → they talk → the game checks → a neighbour learns.

Secrets require explicit keyword evidence; embedding similarity alone is not enough.

For example, imagine one llama passing along news. The speaker needs to know it first. The model writes the exchange, and the game checks whether the words support transferring that information. If accepted, it becomes something the listener knows and may discuss later.

The llamas can keep walking and going about their day while the model is busy writing.

Enter Dash

You play as Dash. Click a llama and he flies over to visit. The game generates four things he could say while he is on the way. I specifically wanted selectable options here, so you can play without stopping to type a message.

Dash can tell Pip the bread rumour is false, or start another rumour about Mo.

The game chooses the available intentions and facts; the model words the options for the character. You can encourage someone, help them, pass along news, or stir up gossip. The llama’s reaction and the resulting changes to trust or mood are governed by rules, while the model gives the reply its voice.

Passing along a correction can help. Gossip can leave a llama believing something false. Those changes carry into later interactions and count toward the ending.

Making the village in 3D

I had a lot of fun building this with flutter_scene. I kept asking for changes to the characters, especially making the llamas look different from one another and getting Dash’s face right. Then came the weather, animals, music, and cutscenes for skipping the night.

Fireflies and lamps around the pond at night.

Clover and Bramble heading home during the storm.

The models and the 3D renderer share the same machine, which makes performance a concern. The scene needs to keep running while text is generated. Some work can happen ahead of time, like writing Dash’s options during his flight.

Llamadart provides the cross-platform local AI foundation. The game shown here is the macOS preview, running inference locally with the model files already on the machine. Bringing the same experience to other devices still requires working through their memory, inference, and rendering budgets.

When I started thinking about mobile, I even asked whether we could make a version without an LLM. That is still an open design question. Generating everything live has a cost, and I would want to reconsider how much of it a mobile version needs.

A festival, then a storybook

I also wanted the game to have a goal and different endings based on what Dash did. The Berry Festival gives the week a destination. The llamas perform, a winner is chosen, and the ending reflects the relationships, rumours, and trust built up during play. The winner and ending are determined by game rules.

Mo wins the Golden Bell in this recorded playthrough.

Afterward, the game turns the week into a picture book. A journal records notable events, conversations, rumours, and Dash’s visits. The model uses a digest of those events to write the pages, alongside pictures captured during play.

The first day, retold using the game’s journal and a picture captured during play.

This was something I asked for after the game had taken shape: a fairy-tale book about what happened during the run. It is a nice way to look back at conversations and events you may have missed while following Dash.

Where I would take it next

I would like to try a lighter version that works well on more devices. Llamadart’s cross-platform support gives me a starting point, but this game still needs its own performance work and testing on each platform.

For this version, I got to build the experiment I wanted: several characters talking and acting in the same world, with local models doing different jobs. I also spent far more time adjusting llama faces and village lighting than the original idea suggested.

If you try it, I would be interested to hear whether you prefer watching the llamas or getting involved as Dash.

Explore Llama Village on GitHub · Watch the preview trailer · Explore llamadart

Writing assistance: this article was drafted with AI assistance using my development notes and project documentation. Gameplay images and animation are from Llama Village.

Making cross-platform local AI easier with llamadart

Jhin Lee — Thu, 08 Oct 2026 00:52:40 +0000

My writing assistant could talk to Gemini. The missing piece was using it without a connection.

That was the reason I started llamadart: I wanted an offline mode for flights, unreliable internet, and environments where outbound API access was unavailable. In my first article, I focused on the work needed to get there: native binaries, Dart build hooks, and model chat templates.

Getting a response from a local model is a satisfying milestone. It also leaves you with a surprisingly long list of application problems.

Where does the model come from? What happens if someone cancels its download? Where does the conversation live? Can they switch models without losing the one that already works?

Those questions are a useful way to explain how llamadart has changed. The project now does more, but the interesting part of 0.11 is how much of the ordinary integration work fits into a smaller, clearer API.

Let's use a simple writing-assistant flow to walk through it: download a model, ask it to rewrite a paragraph, then ask for a shorter version. This is an illustrative app flow, not a claim about a particular model's writing quality.

Before the first answer

Imagine adding an “Offline mode” button to that application. Pressing it cannot immediately make a large model appear on the device.

The app needs to explain the download, show progress, and recover when the user closes the screen. Later launches should reuse the file rather than download it again. These are the details a user encounters before they ever see a generated word.

The native packaging work from the first article still helps here. App developers do not need a local C++ toolchain for the common setup; the package's build hook resolves the native runtime assets. Model files are a separate concern.

In 0.11, the model-loading entrypoint brings those files together. A LlamaModel describes the model and an optional multimodal projector. Its ModelSource can be a local path, a URL, or a Hugging Face reference.

The change is small enough to show directly. With source and params already defined, the old setup looked like this:

// 0.10
final engine = LlamaEngine(LlamaBackend());
try {
  await engine.loadModelSource(source, modelParams: params);
  // Use the engine.
} finally {
  await engine.dispose();
}

Now the ready engine comes back from the load:

// 0.11
final engine = await LlamaEngine.load(
  LlamaModel(source),
  params: params,
);
try {
  // Use the engine.
} finally {
  await engine.dispose();
}

The important difference is where ownership begins. If the new load throws, the engine and backend it created are already disposed. If it succeeds, the caller owns the ready engine.

Progress and cancellation are still things the app needs to present. The loader exposes them through onProgress and download options, so they can belong to the same operation as loading the model. The lifecycle guide covers that operation in detail.

From a prompt to a conversation

Once the model is ready, our writing assistant can ask for a rewrite. Then comes the follow-up: “Make it shorter.”

That short instruction depends on the earlier exchange. The app needs to keep the paragraph and the first answer in the conversation.

ChatSession handles that history. It already existed before 0.11; the new convenience methods make the common case easier to read. Send a message with session.send, then get the completed response through reply.text.

Here is a complete native Dart starting point:

import 'package:llamadart/llamadart.dart';

Future<void> main() async {
  final engine = await LlamaEngine.load(
    LlamaModel(
      ModelSource.parse(
        'hf://unsloth/SmolLM2-135M-Instruct-GGUF/'
        'SmolLM2-135M-Instruct-Q2_K.gguf',
      ),
    ),
    params: const ModelParams(contextSize: 1024, gpuLayers: 0),
  );

  try {
    final session = ChatSession(
      engine,
      systemPrompt: 'You help rewrite text concisely.',
    );
    final reply = await session.send(
      'Rewrite this: We are writing to let you know '
      'that the meeting will start at nine.',
    );
    print(reply.text);

    final shorter = await session.send('Make it shorter.');
    print(shorter.text);
  } finally {
    await engine.dispose();
  }
}

The tiny model keeps this example approachable. Choosing a model that follows your instructions well is a separate step, and you should evaluate it with your own inputs.

For a live chat UI, session.create still streams the response. Reading its text has changed from chunk.choices.first.delta.content to chunk.text. That is a small improvement you notice every time you connect the stream to a widget.

There is a limit to conversation memory. ChatSession trims older turns to fit the context budget while keeping the system prompt. If your app already owns and edits transcripts, engine.create lets you supply the full message list yourself.

These examples follow the tagged 0.10 and 0.11 documentation.

The model picker changes the problem

Now imagine the writing assistant has a model picker. Someone wants to try a different model while the current one is available.

An app could unload the current model first and then start downloading its replacement. On a slow connection, that leaves the user waiting without either model ready.

On native targets, setModel keeps the current model available while the replacement files resolve or download. If that stage fails or is cancelled, the old model stays loaded. Only after the files are ready does replacement begin.

This has a boundary worth understanding: a failure during the later load can leave nothing loaded. Web runtimes also unload before fetching the replacement. The API gives the app a defined lifecycle, rather than a guarantee that every switch succeeds.

The same ownership question appears when leaving the chat screen. A Flutter engine should live in a service, provider, or state object, not inside build(). Dispose it when that owner is finished. Desktop app exit needs its own cleanup path where documented.

These choices are less visible than a new model demo, but they determine whether the feature behaves sensibly when someone uses the rest of the app.

More runtimes without pretending they are identical

The project has also grown beyond its original llama.cpp path.

Version 0.7 added LiteRT-LM support. Version 0.8 separated Apple runtime companions from the core package, keeping pure Dart consumers free of a Flutter SDK constraint. Version 0.10 added preview image generation through an opt-in stable-diffusion.cpp runtime.

That creates more possibilities, along with more differences the application must respect.

For our writing assistant, a useful question is whether to enable an image attachment button. The answer should come from the loaded model and runtime. In 0.11, engine.runtime identifies that runtime, and await engine.capabilities reports supported operations.

Device selection follows the same principle. ModelParams.device provides a shared CPU, GPU, or NPU selection. An explicit request runs there or fails as unsupported; auto preserves the runtime's default.

There are still platform limits. Android llama.cpp Vulkan is experimental and device-dependent. Web support is experimental. Automatic tool loops reject the pinned LiteRT-LM runtimes because they cannot reliably report token-limit truncation. The support matrix is the place to check a deployment.

The payoff for app code is a common way to ask what is available, and a clear failure when a requested operation cannot be supported.

Try one useful offline feature

To run the example, add llamadart: ^0.11.0 and run dart pub get or flutter pub get. The package requires Dart 3.10.7 or newer, and Flutter 3.38.0 or newer for Flutter apps.

Flutter iOS and macOS apps using the GGUF Apple companion should pair it with llamadart_llama_cpp_flutter: ^0.0.21. Other runtimes have their own companions and requirements; follow the installation guide.

The first setup needs a connection for missing runtime assets and model files. Native model downloads are cached for reuse. Once the required assets are present, inference can run locally; any remote tools or cloud services you add still need their own connections.

For an existing app, read the migration guide. Older model-loading methods remain deprecated until 1.0, but other API and lifecycle changes need attention.

Start with one feature: a rewrite, a summary, or a conversation. Get it working, then try cancelling, switching models, and leaving the screen. Those interactions tell you much more about the integration than a successful first prompt.

The Dart and Flutter examples are there to build on. I would welcome feedback about the parts that still make that journey harder than it needs to be.

Why I Built llamadart: Offline Local LLM Inference for Dart/Flutter

Jhin Lee — Sun, 15 Feb 2026 02:30:51 +0000

I built a desktop AI-powered writing assistant, and cloud inference with Gemini worked great.

But I needed one more mode: offline AI.

I wanted the app to work on flights, unstable connections, and restricted environments where outbound API access is unavailable. That pushed me toward local inference with llama.cpp and GGUF models.

The biggest friction was native packaging. Cross-platform C++ setup is powerful, but it usually means extra work before product work: toolchains, linker/runtime quirks, per-architecture builds, and CI maintenance.

So I started llamadart with one practical goal:

Make local LLM inference in Dart/Flutter usable without forcing app developers to manually build llama.cpp first.

What llamadart is

llamadart is a Dart/Flutter package for local GGUF inference on top of llama.cpp, with a Dart-first API and streaming support.

Core references:

Package: https://pub.dev/packages/llamadart
Source: https://github.com/leehack/llamadart
Main docs: https://github.com/leehack/llamadart/blob/main/README.md

Architecture overview

High-level structure:

App Layer (Flutter UI / CLI / API server)
        |
        v
LlamaEngine (stateless orchestration)
        |
        +--> ChatSession (stateful history + context trimming)
        |
        +--> ChatTemplateEngine (detect/render/parse/tool grammar)
        |
        v
LlamaBackend abstraction
        |
        +--> NativeLlamaBackend (isolate + FFI)
        |
        +--> Web backend (bridge runtime path)

Implementation links:

LlamaEngine: https://github.com/leehack/llamadart/blob/main/lib/src/core/engine/engine.dart
ChatSession: https://github.com/leehack/llamadart/blob/main/lib/src/core/engine/chat_session.dart
ChatTemplateEngine: https://github.com/leehack/llamadart/blob/main/lib/src/core/template/chat_template_engine.dart
LlamaBackend interface: https://github.com/leehack/llamadart/blob/main/lib/src/backends/backend.dart
Native backend: https://github.com/leehack/llamadart/blob/main/lib/src/backends/llama_cpp/llama_cpp_backend.dart
Web backend: https://github.com/leehack/llamadart/blob/main/lib/src/backends/webgpu/webgpu_backend.dart

Build hook deep dive: how native binaries are delivered

This is the part I cared about most.

llamadart uses Dart build hooks (official docs: https://dart.dev/tools/hooks). Build hooks are executed automatically by the Dart SDK during run, build, and test to compile or download native assets.

For FFI context, Dart also documents this under C interop: https://dart.dev/interop/c-interop#build-hooks

Hook entry point

The hook implementation is here:

https://github.com/leehack/llamadart/blob/main/hook/build.dart

build.dart reads target metadata from hook input (BuildInput) and reports resolved assets through BuildOutputBuilder as CodeAsset entries.

Related official APIs:

Hooks package: https://pub.dev/packages/hooks
BuildInput and BuildOutputBuilder: https://pub.dev/documentation/hooks/latest/hooks/
Code assets package: https://pub.dev/packages/code_assets
CodeAsset API: https://pub.dev/documentation/code_assets/latest/code_assets/CodeAsset-class.html

What happens at build/run time

The hook runs a deterministic local-first pipeline:

Resolve target platform and architecture from hook input.
Map (os, arch) to a concrete release asset filename (for example, libllamadart-linux-x64.so).
Try local packaged binary (third_party/bin/...) first.
If missing, try cache (.dart_tool/llamadart/binaries/...).
If cache miss, download from GitHub Releases.
Copy to hook output directory with standardized runtime filename.
Report the asset as DynamicLoadingBundled() so Dart can bundle and load it.

On Apple targets, the hook also handles binary thinning when needed (lipo) so the reported artifact matches the active architecture.

How GitHub is used

llamadart uses GitHub Releases as the binary distribution layer.

In the hook, the release URL is pinned by a llama.cpp tag constant:

const _llamaCppTag = 'b8011';
const _baseUrl =
    'https://github.com/leehack/llamadart/releases/download/$_llamaCppTag';

That gives stable, tag-based binary resolution instead of "latest" behavior.

The automation that produces those release assets is here:

https://github.com/leehack/llamadart/blob/main/.github/workflows/llama_cpp_auto.yml

That workflow builds artifacts across target OS/arch matrices, uploads release assets, regenerates bindings when needed, and updates the pinned hook tag.

Runtime FFI resolution

At runtime, FFI symbols are linked to the hook-provided asset via annotations like:

@Native: https://api.dart.dev/dart-ffi/Native-class.html
@DefaultAsset: https://api.dart.dev/dart-ffi/DefaultAsset-class.html

In llamadart, generated bindings use this default asset pattern:

https://github.com/leehack/llamadart/blob/main/lib/src/backends/llama_cpp/bindings.dart

That avoids per-platform dynamic library path logic in app code.

Why this matters in practice

This hook + release model gives a practical DX benefit:

Package users avoid manual local llama.cpp build steps in the common case.
First build can fetch the correct binary for the target.
Subsequent builds use cache/local artifacts.
App code stays focused on model lifecycle and inference logic, not native packaging scripts.

A quick note on dinja

While implementing chat-template compatibility, I ran into another gap: GGUF models expose Jinja templates, but template behavior differs significantly across model families.

I could not find a Dart Jinja package tuned for this multi-template LLM use case, so I built dinja (source: https://github.com/leehack/dinja).

In llamadart:

Template handlers render prompts with dinja.
JinjaAnalyzer does AST-level capability checks (system role, tool-call shape, typed content, thinking tags).
ChatTemplateEngine routes handlers and supports custom overrides/registration.

Relevant links:

Handlers directory: https://github.com/leehack/llamadart/tree/main/lib/src/core/template/handlers
Jinja analyzer: https://github.com/leehack/llamadart/blob/main/lib/src/core/template/jinja/jinja_analyzer.dart
Template engine: https://github.com/leehack/llamadart/blob/main/lib/src/core/template/chat_template_engine.dart

Minimal usage example

import 'dart:io';
import 'package:llamadart/llamadart.dart';

Future<void> main() async {
  final engine = LlamaEngine(LlamaBackend());

  try {
    await engine.loadModel('path/to/model.gguf');

    final session = ChatSession(
      engine,
      systemPrompt: 'You are a helpful writing assistant.',
    );

    await for (final chunk in session.create([
      LlamaTextContent('Rewrite this paragraph in a concise style.'),
    ])) {
      stdout.write(chunk.choices.first.delta.content ?? '');
    }
  } finally {
    await engine.dispose();
  }
}

Closing

I built llamadart because my app needed both online model quality and reliable offline execution, without turning native build management into the main project.

If you are building AI features in Dart/Flutter and need local inference, I would love feedback:

How to use GoLang in Flutter Application - Golang FFI

Jhin Lee — Wed, 20 Dec 2023 02:35:14 +0000

GoLang is a very powerful language that is easy to use. Writing a GoLang program and compiling it to a binary file is very easy. Golang is very useful in writing high-performance and low-latency applications, but it needs improvement in GUI. Flutter is a compelling cross-platform UI framework. It allows you to write a beautiful application effortlessly. Mixing GoLang and Flutter is an excellent idea to write a high-performance and beautiful application.

In this article, we will see how to compile a GoLang-based library and create a Flutter plugin package for a Flutter application.

What is FFI?

FFI stands for Foreign Function Interface. It is a way to call functions written in other languages from Dart. FFI helps use libraries written in other languages in a Flutter application. Flutter supports FFI for Android, iOS, Linux, macOS, and Windows. In this article, we will see how to use FFI.

In Dart, the programming language used to write Flutter applications, we can use the dart:ffi package to use FFI. The dart:ffi package is used to call functions written in other languages from Dart. We can use the dart:ffi package to call functions written in C, C++, or Rust. In this article, we will see how to use the dart:ffi package to call functions in GoLang.

Create a plugin project for Flutter

The Flutter plugin can contain Dart and platform-specific code written in Kotlin, Swift, Objective-C, or GoLang. We can use a Flutter plugin to add functionality to a Flutter application.

The below command will create a Flutter plugin project named native_add. The native_add project will support Android, iOS, Linux, macOS, and Windows. The native_add project will have a lib folder that contains the Dart code and an example folder that contains a Flutter application to test the plugin.

 $ flutter create --platforms=android,ios,macos,windows,linux --template=plugin_ffi native_add
 $ cd native_add

Create a GoLang library

The plugin project we created in the previous step will have a src folder. The src folder will have a cpp folder that contains the C++ code. We will create a GoLang library in the src folder. The GoLang library will have a function to add two numbers.

First, we will create a go.mod file. The go.mod file defines the module name and the GoLang version. The go.mod file is used to manage dependencies. We will not use any dependencies in this article.

// go.mod file
module go_code

go 1.21.3

Second, we will create a sum.go file. The sum.go file will have a function to add two numbers. The sum.go file will also have a function called enforce_binding. The enforce_binding function enforces the binding in the ios application. The Dart code will not use the enforce_binding function, but it was the trick I found to make the library linked correctly for iOS.

Above each function we want to expose, we will add a comment //export <function_name>. The //export <function_name> comment exposes the function to other languages. We must also use the type C.int instead of int in the function signature. The C.int type is used to make the function compatible with other languages.

// sum.go file
package main

import "C"

//export sum
func sum(a C.int, b C.int) C.int {
  return a + b
}

//export enforce_binding
func enforce_binding() {}

func main() {}

Compile the GoLang library for Android

Since we are compiling the GoLang library for Android, we must use cross-compilation. Modern Android devices mostly use arm64 or x86_64 architectures. We will compile the GoLang library for both arm64 and x86_64 architectures.

export ANDROID_OUT=../android/src/main/jniLibs
export ANDROID_SDK=$(HOME)/Library/Android/sdk
export NDK_BIN=$(ANDROID_SDK)/ndk/23.1.7779620/toolchains/llvm/prebuilt/darwin-x86_64/bin

# Compile for x86_64 architecture and place the binary file in the android/src/main/jniLibs/x86_64 folder
CGO_ENABLED=1 \
GOOS=android \
GOARCH=amd64 \
CC=$(NDK_BIN)/x86_64-linux-android21-clang \
go build -buildmode=c-shared -o $(ANDROID_OUT)/x86_64/libsum.so .

# Compile for arm64 architecture and place the binary file in the android/src/main/jniLibs/arm64-v8a folder
CGO_ENABLED=1 \
GOOS=android \
GOARCH=arm64 \
CC=$(NDK_BIN)/aarch64-linux-android21-clang \
go build -buildmode=c-shared -o $(ANDROID_OUT)/arm64-v8a/libsum.so .

Now we can use the libsum library in the Dart code using FFI.

DynamicLibrary.open('libsum.so');

Compile the GoLang library for iOS

We need some extra steps to cross-compile the Golang library and use it for iOS. iOS only supports static libraries. So, we need to compile the GoLang library as a static library. We also need to modify the podspec file to use the static library in the Flutter plugin. Another thing to note is that we need to create xcframework for iOS. The xcframework is used to support both simulator and device architectures.

# build_ios.sh file
#!/bin/sh
export GOOS=ios
export CGO_ENABLED=1
# export CGO_CFLAGS="-fembed-bitcode"
# export MIN_VERSION=15

SDK_PATH=$(xcrun --sdk "$SDK" --show-sdk-path)

if [ "$GOARCH" = "amd64" ]; then
    CARCH="x86_64"
elif [ "$GOARCH" = "arm64" ]; then
    CARCH="arm64"
fi

if [ "$SDK" = "iphoneos" ]; then
  export TARGET="$CARCH-apple-ios$MIN_VERSION"
elif [ "$SDK" = "iphonesimulator" ]; then
  export TARGET="$CARCH-apple-ios$MIN_VERSION-simulator"
fi

CLANG=$(xcrun --sdk "$SDK" --find clang)
CC="$CLANG -target $TARGET -isysroot $SDK_PATH $@"
export CC

go build -trimpath -buildmode=c-archive -o ${LIB_NAME}_${GOARCH}_${SDK}.a

The above script will create a static library for iOS. We must use the script to build the static library for both simulator and device architectures.


# Compile for x86_64 simulator architecture
GOARCH=amd64 \
SDK=iphonesimulator \
LIB_NAME=libsum \
./build_ios.sh

# Compile for arm64 simulator architecture
GOARCH=arm64 \
SDK=iphonesimulator \
LIB_NAME=libsum \
./build_ios.sh

# Compile for arm64 device architecture
GOARCH=arm64 \
SDK=iphoneos \
LIB_NAME=libsum \
./build_ios.sh

The above script will create three static libraries. Before creating the xcframework, we must combine the two static libraries for the simulator architecture. The lipo command can combine the two static libraries.

lipo \
-create \
libsum_arm64_iphonesimulator.a \
libsum_amd64_iphonesimulator.a \
-output libsum_iphonesimulator.a

Now we can create the xcframework using the below command.

xcodebuild -create-xcframework \
-output ../ios/libsum.xcframework \
-library ios-arm64/libsum.a \
-headers ios-arm64/libsum.h \
-library ios-simulator/libsum.a \
-headers ios-simulator/libsum.h

The libsum.xcframework will be created in the ios folder.

We need to modify the podspec file to use the xcframework. We must add the lines below to the podspec file under the ios folder.

s.public_header_files = 'Classes/**/*.h'
s.vendored_frameworks = 'libsum.xcframework'

After some trial and error, I found that we need to use the library in the Classes folder to link it properly. So, we create binding.h and EnforceBinding.swift files under the Classes folder.

The binding.h file will have the below code.

void enforce_binding();

The EnforceBinding.swift file will have the below code.

public func dummyMethodToEnforceBundling() {
    enforce_binding() // disable tree shaking
}

Now, we can use the libsum library in the Dart code using FFI.

// The libsum library is statically linked in the native_add so we only need to load native_add.
DynamicLibrary.open('$native_add.framework/native_add');

Create a binding in Dart using `ffigen`

The ffigen tool generates the Dart code to use the GoLang library. We will create a ffigen.yaml file to configure the ffigen tool. The config.yaml file will have the below code.

# Run with `flutter pub run ffigen --config ffigen.yaml`.
name: NativeLibrary
description: Bindings to `src/sum.h`.
output: 'lib/generated_bindings.dart'
headers:
  entry-points:
    - 'src/libsum.h'
preamble: |
  // ignore_for_file: always_specify_types
  // ignore_for_file: camel_case_types
  // ignore_for_file: non_constant_identifier_names
  // ignore_for_file: unused_field
  // ignore_for_file: unused_element
comments:
  style: any
  length: full

Now, we can run the ffigen tool to generate the Dart code.

flutter pub run ffigen --config ffigen.yaml

The ffigen tool will generate the generated_bindings.dart file under the lib folder based on the libsum.h file in src folder that was generated by the go build command in the previous step.

Use the GoLang library in Dart

Now, we can use the GoLang library in the Dart code. We can use the DynamicLibrary.open method to load the library. We can use the NativeLibrary class to call the functions in the GoLang library.

// lib/native_add.dart
import 'dart:ffi';
import 'dart:io';

import 'generated_bindings.dart';

int sum(int a, int b) => _bindings.sum(a, b);

const String _libName = 'native_add';

/// The dynamic library in which the symbols for [NativeAddBindings] can be found.
final DynamicLibrary _dylib = () {
  if (Platform.isMacOS || Platform.isIOS) {
    return DynamicLibrary.open('$_libName.framework/$_libName');
  }
  if (Platform.isAndroid || Platform.isLinux) {
    return DynamicLibrary.open('libsum.so');
  }
  if (Platform.isWindows) {
    return DynamicLibrary.open('$_libName.dll');
  }
  throw UnsupportedError('Unknown platform: ${Platform.operatingSystem}');
}();

/// The bindings to the native functions in [_dylib].
final NativeLibrary _bindings = NativeLibrary(_dylib);

Now, we can use the sum function in the Dart code from the Flutter project that uses this plugin.

import 'package:native_add/native_add.dart' as native_add;
native_add.sum(1, 2);

Conclusion

In this article, we checked how to use GoLang in a Flutter application. We created a GoLang library and compiled it for Android and iOS. We also created a Flutter plugin to use the GoLang library in a Flutter application. We used the ffigen tool to generate the Dart code to use the GoLang library. We also checked how to use the GoLang library in the Dart code.

I also created a sample project to demonstrate using GoLang in a Flutter application. You can find the sample project in the below link. The example also contains the script to compile the GoLang library for MacOS.
https://github.com/leehack/flutter_golang_ffi_example
It's still missing for Windows and Linux but it should be much easier than for Android and iOS. Like Android or iOS, we must cross-compile and link the GoLang library for Windows and Linux in the CMake file.

Due to the lack of information, I had to spend a lot of time figuring out how to use GoLang in a Flutter application. This article will help you to use GoLang in a Flutter application.

Let's build a full-stack dart app

Jhin Lee — Thu, 20 Jul 2023 20:33:57 +0000

I've been working with Flutter for a while and love it. It's an excellent tool for building cross-platform applications. But what if you want to build a full-stack application? We will explore how to build a full-stack application using Flutter and Dart.

The complete example code: https://github.com/leehack/fullstack_dart_todo

What will we build?

The todo application with Flutter and Dart! We will use Flutter for the client and Dart Frog for the server. Flutter will display the UI, and Dart Frog will handle the business logic and data storage.

Why full-stack in Dart?

The most significant benefit of using Dart for full-stack development is that you can use the same language on both the client and server. It means that you don't have to learn two different languages, which makes it easier to get started with full-stack development. Of course, you can also share your code between the client and server, which is a huge benefit.

Project Structure

We will use the mono-repo structure for this project. We will create three packages: backend, frontend, and shared. shared package will contain all the code that is shared between the client and server. backend package will contain all the code for the server(Dart Frog) and frontend package will contain all the code for the client(Flutter).

Shared package (packages/todo)

todo_entity.dart - The Todo data model with the business logic.
todo_repository.dart - The repository interface. The backend will implement it for storing data. The frontend will implement it for communicating with the backend.

Backend package (packages/backend_frog)

routes/todos/index.dart - The rest api handlers for the route https://address.com/todos. It handles the GET and POST requests.
routes/todos/[id].dart - The rest api handlers for the route https://address.com/todos/[id]. It handles each todo item's GET/PUT/DELETE requests.
lib/data/in_memory_todo_repository.dart - The repository implementation of the in-memory database.
routes/todos/_middleware.dart - Dependency Injection of the in-memory database that handlers will use.

Frontend package (packages/frontend_flutter)

data/remote_todo_repository.dart - The repository implementation of the remote database (rest api client).
providers/todo_list_provider.dart - The riverpod provider handles business logic and data storage.
presentation folder - The UI code for widgets.
pages/todo_page.dart - The todo page displays the todo list and form.

Mono-repo setup

We will use melos to manage the mono-repo. Melos is a tool for managing Dart projects with multiple packages.

Install melos cli



    dart pub global activate melos

Create a project folder and initialize the melos workspace



    mkdir fullstack_dart_todo
    cd fullstack_dart_todo
    mkdir packages

Add the following to the pubspec.yaml file:



    name: remote_todo

    environment:
      sdk: '>=2.18.0 <3.0.0'

Add the melos as a development dependency:



    dart pub add melos --dev

Add the following to the melos.yaml file:



    name: remote_todo

    packages:
      - packages/*

    scripts:
      analyze:
        exec: Dart analyze .

Now you can run the melos bs for bootstrapping the mono-repo. Bootstrapping has two primary roles:

Installing all package dependencies (internally using pub get).
Locally linking any packages together.

Every time creating a new package, you must run the melos bs command. If you clone the project from GitHub, you must run the melos bs command.

Create the backend package with the Dart Frog

We'll use the Dart Frog framework for the backend. Dart Frog is built on top of shelf and mason and is inspired by many tools, including remix.run, next.js, and express.js.

We need to install the dart_frog_cli first.



    dart pub global activate dart_frog_cli

Now we are ready to create the backend project.



    cd packages
    dart_frog create backend_frog
    cd backend_frog

Start the dev server:



    dart_frog dev

By default, it opens the web server at http://localhost:8080. You can see the default page. Thanks to the hot reload, you can see the changes immediately.

Create the frontend package with the Flutter

We'll use Flutter for the frontend. Flutter is Google's UI toolkit for building beautiful, natively compiled mobile, web, and desktop applications from a single codebase.

Let's create the Flutter project under the packages folder.



    cd packages
    flutter create frontend_flutter
    cd frontend_flutter

Create the pure Dart shared package

The shared package will contain all the code that is shared between the client and server. We will create the pure dart package since it has no UI code.

Let's create the library project under the packages folder.



    cd packages
    dart create -t package todo
    cd todo

To make the backend and frontend projects use the shared package, we must add the dependency to each project's pubspec.yaml file (backend_frog, frontend_flutter).



    dependencies:
      todo: ^0.0.1

Once we run the melos bs command, the shared package will be linked to the backend and frontend project.

So, now we have everything ready to start the coding!

Global Architecture

The shared package has the Todo data model and the repository interface. The backend and frontend packages implement the repository interface. The backend implements it as the in-memory database, and the frontend implements it as the rest api client. The Todo entity is the data model with the business logic that will be referenced in both the backend and frontend packages.

Backend

The backend is a simple CRUD application that exposes REST APIs. It uses the in-memory database for storing data. The in-memory database is a simple List that stores the Todo data model. The backend has two routes: https://address.com/todos and https://address.com/todos/[id]. The first route handles the GET and POST requests. The second route handles each todo item's GET/PUT/DELETE requests.

Routes

Dart Frog uses directory-based routing. Each route is a directory with the *.dart file. index.dart represents the root route of the path. For example, the routes/todos/index.dart represents the https://address.com/todos route. The routes/todos/[id].dart represents the https://address.com/todos/[id] route.

Handlers for /todos in routes/todos/index.dart ```dart

Future onRequest(RequestContext context) async {
return switch (context.request.method) {
HttpMethod.get => await _get(context),
HttpMethod.post => await _post(context),
_ => Response(statusCode: HttpStatus.notFound)
};
}

Future _get(RequestContext context) async {
final todos = await context.read().fetchAll();
final json = [for (var todo in todos) todo.asMap];
return Response.json(body: json);
}

Future _post(RequestContext context) async {
final json = await context.request.json() as Map;
final todo = await context.read().add(Todo.fromMap(json));
return Response.json(body: todo.asMap);
}


* Handlers for `/todos/[id]` in `routes/todos/[id].dart`
  ```dart


  Future<Response> onRequest(RequestContext context, String id) async {
    return switch (context.request.method) {
      HttpMethod.get => await _get(context, id),
      HttpMethod.put => await _put(context, id),
      HttpMethod.delete => await _delete(context, id),
      _ => Response(statusCode: HttpStatus.notFound)
    };
  }

  Future<Response> _get(RequestContext context, String id) async {
    final todo = await context.read<TodoRepository>().getById(id);

    return Response.json(body: todo.asMap);
  }

  Future<Response> _put(RequestContext context, String id) async {
    final json = await context.request.json() as Map<String, dynamic>;
    final todo = Todo.fromMap(json);
    await context.read<TodoRepository>().updateById(todo);
    return Response();
  }

  Future<Response> _delete(RequestContext context, String id) async {
    final json = await context.request.json() as Map<String, dynamic>;
    final id = json['id'] as String;
    await context.read<TodoRepository>().deleteById(id);
    return Response();
  }

Each handler does the simple thing. Just call the repository methods and return the response. The repository is injected into the handler using the dependency injection with the middleware.

Dependency Injection with the middleware

Dart Frog supports the dependency injection natively. We can inject the repository into the handler using the middleware. The middlewares are also directory based. For example, routes/todos/_middleware.dart is the middleware for the routes/todos route. The _middleware.dart file is a particular file name that represents the middleware for the directory.



InMemoryTodoRepository? _todoData;

Handler middleware(Handler handler) {
  return handler.use(
    provider<TodoRepository>(
      (context) => _todoData ??= InMemoryTodoRepository(),
    ),
  );
}

The code above injects the InMemoryTodoRepository into the provider so the handler can access the repository. Thanks to the middleware with the dependency injection, we can easily use it from the handler as below.



await context.read<TodoRepository>()

Repository with in-memory database

I said it's a database, but it's just a simple List. It's not an actual database. It's just an example. The repository interface is in the shared package. The backend implements it as the in-memory database repository. The repository contains the List of Todo data model.



  class InMemoryTodoRepository implements TodoRepository {
    /// Local data
    List<Todo> data = [];

    /// Get List of [Todo]
    List<Todo> get todos => data;

    @override
    Future<Todo> getById(String id) async {
      return data.firstWhere((todo) => todo.id == id);
    }

    /// Add a new [todo] and return [Todo] with new id
    @override
    Future<Todo> add(Todo todo) async {
      final newTodo = todo.copyWith(id: const Uuid().v4());
      data = [...data, newTodo];
      return newTodo;
    }

    @override
    Future<void> deleteById(String id) async {
      data = [
        for (final t in data)
          if (t.id != id) t
      ];
    }

    @override
    Future<List<Todo>> fetchAll() async {
      return data;
    }

    @override
    Future<void> updateById(Todo todo) async {
      data = [
        for (final t in data)
          if (t.id == todo.id) todo else t
      ];
    }
  }

As you can see, it has some CRUD methods for mutating the List<Todo> data.

Frontend

The Frontend app is written in Flutter. It uses the Riverpod for state management and dependency injection. Same as the backend architecture, we store the implemented repository in the provider so the todoListProvider can access the repository. The todoListProvider is the riverpod provider that handles the business logic of the todo list.

Presentation and pages

There are two folders for UI code: presentation and pages. The presentation folder contains the code for widgets. The pages folder contains the UI code for pages. The pages/todo_page.dart is the main page that displays the todo list and form using the TodoListView and TodoTextField widgets in the presentation folder. main.dart is the entry point of the Flutter application that displays the TodoPage.

In the body of the TodoPage, it renders a column with the TodoListView and TodoTextField widgets.



const Column(
  children: [
    Padding(
      padding: EdgeInsets.all(16.0),
      child: TodoTextField(),
    ),
    Expanded(child: SelectionArea(child: TodoListView())),
  ],
),

The TodoTextFiled Widget is a simple text field that handles the input and submits the todo item. When the user submits the text field, it's calling todoListProvier's add method.



class _TodoTextFieldState extends ConsumerState<TodoTextField> {
  final controller = TextEditingController();
  final focusNode = FocusNode();

  @override
  Widget build(BuildContext context) {
    return TextField(
      decoration: const InputDecoration(
        hintText: 'Enter a new todo',
      ),
      focusNode: focusNode,
      controller: controller,
      onSubmitted: (value) async {
        await ref.read(todoListProvider.notifier).add(
              Todo(id: '', title: value),
            );
        controller.clear();
        focusNode.requestFocus();
      },
    );
  }
}

The TodoListView Widget is a simple list view that displays the todo list. It watches the todoListProvider and rebuilds the list view when the todo list is changed. When the user clicks the checkbox, it calls todoListProvier's toggle method.



class TodoListView extends ConsumerWidget {

  const TodoListView({super.key});

@override

  Widget build(BuildContext context, WidgetRef ref) {

    return switch (ref.watch(todoListProvider)) {

      AsyncData(:final value) => ListView.builder(

          padding: const EdgeInsets.all(16),

          itemCount: value.length,

          itemBuilder: (context, index) => TodoItem(todo: value[index]),

        ),

      AsyncLoading() => const Center(child: CircularProgressIndicator()),

      _ => const Center(child: Text("Error")),

    };

  }

}

class TodoItem extends ConsumerWidget {

  const TodoItem({super.key, required this.todo});

final Todo todo;

@override

  Widget build(BuildContext context, WidgetRef ref) {

    return Row(

      mainAxisAlignment: MainAxisAlignment.spaceBetween,

      children: [

        Text(todo.title),

        Checkbox(

          value: todo.isDone,

          onChanged: (value) {

            ref.read(todoListProvider.notifier).toggle(todo);

          },

        ),

      ],

    );

  }

}

Business logic with the Riverpod

The todoListProvider is the riverpod's AsyncNotifier provider that handles the business logic of the todo list. It calls the RemoteTodoRepository's methods for communicating with the backend. When it's initialized, it calls the fetchAll method of the TodoRepository to fetch all the Todo items from the backend and cache them. The add and toggle methods call the add and updateById methods of the TodoRepository. It updates the backend todo list and renews the local cache with the latest todo items in the provider.



@riverpod

class TodoList extends _$TodoList {

  @override

  FutureOr<List<Todo>> build() {

    return ref.read(todoRepositoryProvider).fetchAll();

  }

Future<void> add(Todo todo) async {

    state = await AsyncValue.guard(() async {

      await ref.read(todoRepositoryProvider).add(todo);

      return ref.read(todoRepositoryProvider).fetchAll();

    });

  }

void toggle(Todo todo) async {

    state = await AsyncValue.guard(() async {

      await ref

          .read(todoRepositoryProvider)

          .updateById(todo.copyWith(isDone: !todo.isDone));

      return ref.read(todoRepositoryProvider).fetchAll();

    });

  }

}

Rest API Client

RemoteTodoRepository is the rest api client for communicating with the backend, which is implementing the TodoRepository interface. It uses the http package for making the HTTP request. It's calling the http package's get, post, put, and delete methods for communicating with the backend.



class RemoteTodoRepository implements TodoRepository {

  @override

  Future<Todo> add(Todo todo) async {

    final client = http.Client();

    final response = await client.post(

      Uri.parse("$baseURL/todos"),

      headers: {

        'Origin': 'http://localhost:8080',

        'Content-Type': 'application/json',

      },

      body: jsonEncode(todo.asMap),

    );

    final json = response.body;

<span class="k">return</span> <span class="n">Todo</span><span class="o">.</span><span class="na">fromMap</span><span class="p">(</span><span class="n">jsonDecode</span><span class="p">(</span><span class="n">json</span><span class="p">));</span>



}

@override

  Future<void> deleteById(String id) async {

    final client = http.Client();

    await client.delete(

      Uri.parse("$baseURL/todos/$id"),

      headers: {

        'Origin': 'http://localhost:8080',

        'Content-Type': 'application/json',

      },

    );

  }

@override

  Future<List<Todo>> fetchAll() async {

    final client = http.Client();

    final response = await client.get(

      Uri.parse("$baseURL/todos"),

      headers: {

        'Origin': 'http://localhost:8080',

        'Content-Type': 'application/json',

      },

    );

    final json = response.body;

<span class="k">return</span> <span class="p">[</span><span class="k">for</span> <span class="p">(</span><span class="kd">final</span> <span class="n">todo</span> <span class="k">in</span> <span class="n">jsonDecode</span><span class="p">(</span><span class="n">json</span><span class="p">))</span> <span class="n">Todo</span><span class="o">.</span><span class="na">fromMap</span><span class="p">(</span><span class="n">todo</span><span class="p">)];</span>



}

@override

  Future<Todo> getById(String id) async {

    final client = http.Client();

    final response = await client.get(

      Uri.parse("$baseURL/todos/$id"),

      headers: {

        'Origin': 'http://localhost:8080',

        'Content-Type': 'application/json',

      },

    );

    final json = response.body;

<span class="k">return</span> <span class="n">Todo</span><span class="o">.</span><span class="na">fromMap</span><span class="p">(</span><span class="n">jsonDecode</span><span class="p">(</span><span class="n">json</span><span class="p">));</span>



}

@override

  Future<void> updateById(Todo todo) async {

    final client = http.Client();

    await client.put(

      Uri.parse("$baseURL/todos/${todo.id}"),

      headers: {

        'Origin': 'http://localhost:8080',

        'Content-Type': 'application/json',

      },

      body: jsonEncode(todo.asMap),

    );

  }

}

Summary

We've built a full-stack application with Flutter and Dart. We used Flutter for the client and Dart Frog for the server. Flutter displayed the UI, and Dart Frog handled the business logic and data storage. We used the mono-repo structure for the project. We created three packages: backend, frontend, and shared. shared package contained all the code that is shared between the client and server. backend package contained all the code for the server(Dart Frog) and frontend package contained all the code for the client(Flutter).

We would need to run performance benchmarks to use dart as the backend language before we decided to build a production application, but the development experience was great. I liked the basic tooling of the dart_frog. It's simple and easy to use, like Flutter. I also liked the dependency injection with the middleware. I could build the full-stack application with the Dart Frog without spending too much time learning the framework.

The complete code example is available at https://github.com/leehack/fullstack_dart_todo

How to use Firebase Emulator suite for Flutter

Jhin Lee — Thu, 29 Jun 2023 14:38:25 +0000

The Firebase provides an emulator suite for the local dev environment. The suite allows us to run the Firebase on the local machine. So, we can locally use Cloud Firestore, Realtime Database, Cloud Storage for Firebase, Authentication, Firebase Hosting, Cloud Functions (beta), Pub/Sub (beta), and Firebase Extensions (beta) for the app we are working on.

More detail can be found at https://firebase.google.com/docs/emulator-suite

Pre-Requisite

The Flutter project is initialized for the Firebase: https://firebase.google.com/docs/flutter/setup

Install the Emulator Suite

Setup the Emulator Suite with this command under the project directory:



firebase init emulators

Start the Emulator Suite



firebase emulators:start

Run the Flutter app using the Emulators

Add this lines of code in the main() in main.dart



void main() async {
  WidgetsFlutterBinding.ensureInitialized();
  // Initialize Firebase
  await Firebase.initializeApp(
    options: DefaultFirebaseOptions.currentPlatform,
  );
  // Use Firebase Emulators
  if (kDebugMode) { // Only for debug mode.
    try {
      final emulatorHost = defaultTargetPlatform == TargetPlatform.android
          ? "10.0.2.2"
          : "localhost";
      FirebaseStorage.instance.useStorageEmulator(emulatorHost, 9199);
      FirebaseFirestore.instance.useFirestoreEmulator(emulatorHost, 8080);
    } catch (e) {
      // ignore: avoid_print
      print(e);
    }
  }
  // Start Flutter Application
  runApp(const MyApp());
}

The code example makes the app use FireStorage and Firestore Emulators in debug mode. It also provides Web UI(http://localhost:4000) for easy inspection.

How to screenshot a widget in the flutter

Jhin Lee — Wed, 23 Nov 2022 00:32:06 +0000

Define a global key

GlobalKey previewContainer = GlobalKey();

Wrap the screenshot area with RepaintBoundary

FittedBox(
    fit: BoxFit.contain,
    child: RepaintBoundary(
        key: previewContainer,
        child: Container( // The widget to capture
            width: 500,
            height: 500,
            color: Colors.blue,
        ),
    ),
)

Screenshot for the widget

Future<Uint8List> _capture() async {
    RenderRepaintBoundary boundary = previewContainer.currentContext
        ?.findRenderObject() as RenderRepaintBoundary;
    final image = await boundary.toImage(pixelRatio: 2);
    final byteData = await image.toByteData(format: ImageByteFormat.png);
    return byteData!.buffer.asUint8List();
}

Journey to the clean architecture for my flutter app

Jhin Lee — Mon, 31 Oct 2022 19:15:31 +0000

Motivation

When I started my flutter project two years ago, Android Jetpack architecture inspired me a lot. DataBinding between ViewModel and View using LiveData that comes from the data layer passing through the repository and so on, as I mentioned in another posting

(I cannot find this diagram anymore from the official google document. They added the domain layer as an optional recommendation: link)

My project folder structure was something like this:

- lib
  - repositories
    - a_repository.dart
    - b_repository.dart
  - models
    - a_model.dart
    - b_model.dart
  - services
    - a_service.dart
    - b_service.dart
  - ui
    - pages
      - pageA
        - a_page.dart
        - a_viewmodel.dart
      - pageB
        - b_page.dart
        - b_viewmodel.dart
    - widgets
      - a_widget.dart
      - b_widget.dart
  - main.dart
  - DI.dart

DI.dart wasn't truly doing the dependency injection. Still, I was initializing all global dependencies as singletons to inject them into the page-scoped ViewModels when initializing the page.

The problem with this approach was that the ViewModel held too much business logic. If I needed to use similar logic for another page, I had to reimplement the logic in the ViewModel. Eventually, I started adding ViewModels in a shared folder and reusing them directly from the widgets, creating another instance of the ViewModel. It reduced the code duplication, but it made it look messy. Whenever I tried to add a new feature, it was confusing where I should put the new codes.

Clean architecture

After some research, I decided to apply the clean architecture I've used for a golang backend project. The clean architecture was quite famous for the flutter project, too, since it's a recommended architecture by BLoC, a popular state management library in dart.

Layer-First structure

I started to restructure the directory based on the layer. (At this moment, I also decided to use riverpod instead of provider)

- lib
  - data
    - remote
      - repositories
        - a_repository_impl.dart
        - b_repository_impl.dart
      - models
        - a_model.dart
        - b_model.dart
      - services
        - a_service.dart
        - b_service.dart
      - data_provider.dart
  - domain
    - entities
      - a_entity.dart
      - b_entity.dart
    - repositories
      - a_repository.dart
      - b_repository.dart
    - usecases
      - featureA
        - get.dart
        - get_many.dart
      - featureB
        - get.dart
        - get_by_filter.dart
    - providers
      - feature_a_usecase_provider.dart
      - feature_b_usecase_provider.dart
  - presentation
    - states
      - a_state_provider.dart
      - b_state_provider.dart
    - pages
      - pageA
        - a_page.dart
      - pageB
        - b_page.dart
    - widgets
      - a_widget.dart
      - b_widget.dart
  - main.dart

The repositories in the domain layer are abstract classes that interface between the domain and the data layer. The data_provider.dart initialize the service and repository providers. The providers in the domain layer are initializing the usecases using repository providers defined in data_provider.dart. The states in the presentation layer only access entities and usecase providers in the domain layer.

The model is a service-specific model that holds entity getter(mapper)

// a_model.dart
class ModelA {
    ModelA({this.name});
    final name;

    EntityA get entity => Entity(name: name);
}

The service can locally or remotely access the raw data source

// a_service.dart
abstract class ServiceA {
    ModelA get(String a);
}
class ServiceAImpl implements ServiceA {
    ModelA get(String a) {
        return ModelA(name: "test");
    }
}

The implementation of the repository that accesses the service

// a_repository_impl.dart
class RepositoryAImpl implements RepositoryA {
    RepositoryAImpl(this.service);
    final ServiceA service;
    EntityA get(String id) {
        return service.get(id).entity;
    }
}

Initialize services and repositories(Services are private)

// data_provider.dart
final _serviceAProvider = Provider<ServiceA>((_) => ServiceAImpl());
final repositoryAProvider = Provider<RepositoryA>(
    (ref) => RepositoryAImpl(ref.watch(_serviceAProvider)),
);

Domain entity that application uses

// a_entity.dart
class EntityA {
    EntityA({this.name});
    final name;
}

Interfacing between domain and data layer

// a_repository.dart
abstract class RepositoryA {
    EntityA get(String id);
}

The usecase can access the repository

// featureA/get.dart
abstract class GetA {
    EntityA execute(String id);
}
class GetAImpl implements GetA {
    GetAImpl(this.repo)
    final RepositoryA repo;

    EntityA execute(String id) {
        return repo.get(id)
    }
}

The states in the presentation layer access the usecase through the provider

// feature_a_usecase_provider.dart
final getAProvider = Provider<GetA>((ref) => 
    GetImpl(ref.watch(repositoryAProvider)),
);

States are globally shared and accessible from widgets

// a_state_provider.dart
final stateAProvider = StateNotifierProvider<EntityA>((ref){
    return StateANotifier(EntityA(), ref);
});
class StateANotifier extends StateNotifier<EntityA> {
    StateANotifier(super.state, this.ref)
    final Ref ref;

    void fetch(String id) {
        state = ref.watch(getAProvider).execute(id);
    }
}

Widget accesses the states

// a_widget.dart
class WidgetA extends ConsumerWidget {
  @override
  Widget build(BuildContext context, WidgetRef ref) {
    final stateA = ref.watch(stateAProvider);
    return InkWell(
      onTap: () => ref.read(stateAProvider.notifier).fetch(),
      child: Text('${stateA.name}'),
    );
}

After the massive refactoring of the architecture (Even though it looks like there are a lot of boilerplate codes), I was pretty satisfied. I don't need to think twice about where to put which logic. Every business logic goes to the usecase and entity. If the view needs to update the state, I create the StateNotifierProvider. The Widgets only focus on the viewing part. The separation gives better testability and lets the codes keep the single responsibility rule.

Feature-First structure

After the refactoring, the only problem I had was that it was not easy to navigate the source code. When implementing a feature, I wished all related codes were grouped. Since the codes are split based on the layer, It was pretty easy to lose my focus.
So, I decided to do another refactoring for a feature-first structure.

- lib
  - features
    - featureA
      - domain
        - entity
          - a_entity.dart
        - repository
          - a_repository.dart
        - usecases
          - get.dart
          - get_many.dart
      - data
        - repositories
          - a_repository_impl.dart
        - models
          - a_model.dart
        - services
          - a_service.dart
        - data_provider.dart
      - providers
        - feature_a_usecase_provider.dart
    - featureB
      - domain
        - entity
          - b_entity.dart
        - repository
          - b_repository.dart
        - usecases
          - get.dart
          - get_many.dart
      - data
        - repositories
          - b_repository_impl.dart
        - models
          - b_model.dart
        - services
          - b_service.dart
        - data_provider.dart
      - providers
        - feature_b_usecase_provider.dart
  - presentation
    - states
      - a_state_provider.dart
      - b_state_provider.dart
    - pages
      - pageA
        - a_page.dart
      - pageB
        - b_page.dart
    - widgets
      - a_widget.dart
      - b_widget.dart
  - main.dart

For the code-wise, nothing changed. Some might wonder why I didn't put the presentation in the feature directory. I still don't have a conclusion on this, but I thought the UI should use different domains, and I didn't want to cross-reference between features. I didn't specify above, but I put the shared codes in the core feature. It's the only exception that other features can reference.

Conclusion

I don't have any good recommendations for a small project or team if the clean architecture is a good choice or not since it requires some boilerplate codes. But It gives us a good separation between layers and domains, making the code more testable. It can take time to write the pattern every time adding a new feature, but the time can be easily compensated by focusing on implementing a single domain feature. There's no perfect architecture or solution. One day, we need to refactor our code base. A library we've been using probably doesn't fit our needs anymore or introduces security vulnerabilities. The well-structured architecture allows quickly switching a library or refactoring a part of your code.

References

Journey to the riverpod for my flutter app.

Jhin Lee — Mon, 31 Oct 2022 01:59:38 +0000

In this article, I'd like to talk about how I ended up using Riverpod as a state management system for my flutter app.

Provider with MVVM
get_it for global dependencies
Riverpod

Provider with MVVM

When I started the flutter project as a newbie two years ago, I wanted to build something similar to the MVVM pattern I used to use for Android application development. After some research, I found Provider the most straightforward option. (I wasn't even thinking of state management at that time)

So, I was doing something like this at the entry point of the app:

final serviceA = ServiceA();
final serviceB = ServiceB();
MultiProvider(
  providers: [
    Provider<RepoA>(create: (_) => RepoA(service:serviceA)),
    Provider<RepoB>(create: (_) => RepoB(service:serviceA)),
    Provider<RepoC>(create: (_) => RepoC(service:serviceB)),
  ],
  child: MaterialApp(),
)

I've been initializing the services and repositories at the application launch time and adding them into the Provider using the MultiProvider. It allowed the ViewModels of each page to access the repositories. (Yes, one view, one ViewModel)

class ViewModelA with ChangeNotifier {
  ViewModelA(this.repo) {
    init();
  }
  final RepoA repo;
  String title;

  Future<void> init() async {
    title = "test";
    notifiyListeners();
  }
}

// Wrap the page with ChangeNotifierProvider
class PageA extends StatelessWidget {
  @override
  Widget build(BuildContext context) {
    return ChangeNotifierProvider(
      create: (_) => new ViewModelA(Provider.of<RepoA>(context, listen: false)),
      child: PageView(),
  );
}

I could access the' ViewModel' in any widgets used in the PageView using the Selector.

Selector<ViewModelA, String>(
  selector: (_, vm) => vm.title,
  builder: (_, data, __) {
    return Text('title: ${data}');
  }
)

It was annoying to write the boilerplate codes of the Selector every time I needed to use variables in the ViewModel, but I helped to reduce unnecessary rendering. So far, It's not bad.

After growing the code bases with many repositories and services, I realized that every time I add more repositories in the global provider scope with the MultiProvider, it creates the cascading tree in the widget hierarchy. It made debugging super tricky with the Widget Inspector tool.

One day, I decided to remove MultiProvider and convert all global dependencies as singletons. Technically they were already singletons but just living in the Provider scope anyway.

get_it for global dependencies

After some research, I found the get_it looked promising.
I replaced MultiProvider with getIt singletons at the application start:

final serviceA = ServiceA();
final serviceB = ServiceB();
getIt.registerSingleton<RepoA>(RepoA(service:serviceA));
getIt.registerSingleton<RepoB>(RepoB(service:serviceA));
getIt.registerSingleton<RepoC>(RepoC(service:serviceB));

And the View and ViewModel like this:

class ViewModelA with ChangeNotifier {
  // This allows passing mocked repository to the ViewModel for testing purposes
  ViewModelA(RepoA? repo): _repo = repo ?? GetIt.get<RepoA>() {
    init();
  }
  final RepoA _repo;
  String title;

  Future<void> init() async {
    title = "test";
    notifiyListeners();
  }
}

class PageA extends StatelessWidget {
  @override
  Widget build(BuildContext context) {
    return ChangeNotifierProvider(
      create: (_) => new ViewModelA(),
      child: PageView(),
  );
}

Now, I don't see the cascading tree in the widget inspector, and much more comfortable debugging the widgets. I also liked it because the widget doesn't need to know what dependencies the ViewModel needs. Thanks to the GetIt can give it to me without context. But I still have a slow application start problem.

A few months later, I wanted to tackle the start-up time issue and realized I might have a few options.

Use lazy singleton with GetIt.
Go for another approach.

I could have converted the global dependencies to lazy singleton with GetIt so that it doesn't need to initialize all dependencies at the application start. However, It would still keep in the memory until the app completely shut down. I wanted something that could automatically initiate and de-initiate based on the need.
So another round of research started!

Riverpod

In the meantime, I also was working on a react project that uses the React Query, and even though it's not a client-side state management library, I found it a simple and nice way of managing states. I wanted to have something similar but for flutter. Finally, the Riverpod caught my eye. The developer of the Provider developed Riverpod as a successor of the Provider. (Later, I found Recoil is the one I've been looking for in the React app for client state management, and it has a lot of similarities with Riverpod)

As a first step, I converted all singleton services and repositories into Riverpod providers. I also defined abstract classes for better testability.

abstract class ServiceA {}
class ServiceAImpl implements ServiceA {}
abstract class ServiceB {}
class ServiceBImpl implements ServiceB {}

abstract class RepoA {
  Future<List<Entity>> getMany();
}
class RepoAImpl implements RepoA {
  RepoAImpl(this.service)
  final ServiceA service;

  @override
  Future<List<Entity>> getMany() {
    return service.getMany();
  };
}
abstract class RepoB {}
class RepoBImpl implements RepoB {
  RepoBImpl(this.service)
  final ServiceA service;
}
abstract class RepoC {}
class RepoCImpl implements RepoC {
  RepoCImpl(this.service)
  final ServiceB service;
}

final serviceAProvider = Provider<ServiceA>((_) => ServiceAImpl());
final serviceBProvider = Provider<ServiceB>((_) => ServiceBImpl());

final repoAProvider = Provider<RepoA>((ref) => RepoAImpl(ref.watch(serviceAProvider)));
final repoBProvider = Provider<RepoB>((ref) => RepoBImpl(ref.watch(serviceAProvider)));
final repoCProvider = Provider<RepoC>((ref) => RepoCImpl(ref.watch(serviceBProvider)));

I also started getting rid of the ViewModels. Since I can now define a state as a provider that is accessible from any widgets, the ViewModel was too heavy as a state. This way, I was able to write more reusable codes.

Create state providers

// Simple future state provider
final stateAProvider = FutureProvider<List<EntityA>>((ref) {
  return ref.watch(repoAProvider).getMany();
});

// State provider with notifier that contains methods.
final stateBProvider = StateNotifierProvider<EntityB>((ref) {
  return StateBNotifier(EntityB(), ref.watch(repoAProvider));
});

class StateBNotifier extends StateNotifier<ThemeMode> {
  StateBNotifier(super.state, this.repo);
  final RepoA repo;

  void fetch() {
    state = repo.getMany();
  }
}

I can access the providers from widgets.

class WidgetA extends ConsumerWidget {
  @override
  Widget build(BuildContext context, WidgetRef ref) {
    final stateB = ref.watch(stateBProvider);
    return InkWell(
      onTap: () => ref.read(stateBProvider.notifier).fetch(),
      child: Text('${stateB.name}'),
    );
}

With the Riverpod, I could make the state lighter and more reusable. I often create a container widget that holds the state that wraps the pure widgets. This way, the design widget is reusable as a design widget without business context. Providers and widgets are all unit-testable by simply mocking the repository or other dependency of the Provider.

DEV Community: Jhin Lee

Local speech with llamadart: giving Llama Village a voice

Two ways for Mo to listen

Try it on your Mac

1. Create the project

2. Add the complete program

3. Ask Mo a question

Try the same question as audio

Make it sound like Mo

Smaller models, smaller downloads. Faster conversations?

Did Mo say the right thing?

Let the player read while Mo gets ready

Local embeddings with llamadart: how llamas share secrets

A secret starts with one llama

The words do not have to match

Follow the text through llamadart

Who decides whether the meaning matches

June hears the story and still doubts it

Give your app the same separation

Complete embedding comparison example

DecisionEngine: Teaching local AI to play Tetris

Give the model a smaller job

Where Jev comes in

Make your first decision with llamadart

Load a model and ask for a move

Read the typed answer

From B to button presses

Two settings to keep separate

First result: one line

Let the heuristic be the teacher

Train a head of your own

Compare two heads on one encoder

Back to the board

The board does not wait

Does clearing lines prove it works

Take the pattern into your app

Try changing the player

Complete head comparison

Llama Village: a virtual world powered by local AI with llamadart

Finding a reason for the AI

Who lives here?

How a conversation changes the village

Enter Dash

Making the village in 3D

A festival, then a storybook

Where I would take it next

Making cross-platform local AI easier with llamadart

Before the first answer

From a prompt to a conversation

The model picker changes the problem

More runtimes without pretending they are identical

Try one useful offline feature

Why I Built llamadart: Offline Local LLM Inference for Dart/Flutter

What llamadart is

Architecture overview

Build hook deep dive: how native binaries are delivered

Hook entry point

What happens at build/run time

How GitHub is used

Runtime FFI resolution

Why this matters in practice

A quick note on dinja

Minimal usage example

Closing

How to use GoLang in Flutter Application - Golang FFI

What is FFI?

Create a plugin project for Flutter

Create a GoLang library

Compile the GoLang library for Android

Compile the GoLang library for iOS

Create a binding in Dart using ffigen

Use the GoLang library in Dart

Conclusion

Let's build a full-stack dart app

What will we build?

Why full-stack in Dart?

Project Structure

Shared package (packages/todo)

Backend package (packages/backend_frog)

Frontend package (packages/frontend_flutter)

Create a binding in Dart using `ffigen`