Three AI features in a fitness app: on-device tool calling, pose analysis, and a coach that renders its own UI

Ananth H — Fri, 28 Aug 2026 15:40:19 +0000

an2tha / onerep

A simple, yet effective working tracking solution

OneRep

The open-source personal fitness OS.
Training, nutrition, recovery, progress, and an AI coach, all running on your own infrastructure.

Explore the docs »

Try it · Join the iOS beta · Self-host it · Contribute

Why OneRep?

Most workout apps are a single feature wearing a subscription. OneRep is the whole stack: training, nutrition, recovery, and progress, backed by an AI coach, a REST API, an MCP endpoint, and data export, all self-hostable on infrastructure you control.

Built around a single idea: your fitness data should belong to you.

(back to top)

What it is

One React codebase shipping as a web app, a PWA, and Capacitor apps for iOS and Android. Convex holds the database, the sync, the auth, the crons, and every server-side integration; there is no API server in between. Self-hosting brings up the whole thing with one script.

Daily dashboard, training, nutrition with…

View on GitHub

After paying for 4 different apps, that all hid the basics behind a paywall, I
grew tired and just made my own. I wanted this app to be as feature-rich and
low-cost as it can be.

OneRep is the result: one React codebase shipping as
web, PWA, and Capacitor apps for iOS and Android, with Convex holding the
database, auth, and crons. This post covers the three AI features that were
hardest to get right.

1. Needle 2 on the device

Needle 2 is a 45M-parameter tool-calling model at two bits a weight. 14 MB of
engine, 13 MB of weights, about 28 MB of peak session RAM. It runs on the CPU
and needs no network once the files are on disk.

It has no free-text mode. The context declares which tools may be called, and
every turn returns JSON: either a list of calls, or the empty call meaning
nothing declared here serves this request.

const needle = await createNeedleSession({
  baseUrl: "/needle",
  system: "date: 2026-08-26 Wed 14:30; locale: en-GB; device: phone",
  minConfidence: 0.6,
})

needle.toolbox.register(
  defineTool({
    name: "log_food",
    description: "Add a food to today's diary",
    input: z.object({ name: z.string(), grams: z.number().positive() }),
    execute: (input) => logFood(input),
  }),
)

const { calls, stop } = await needle.run("200g of chicken breast")

Inference

Three backends sit behind one interface. All of them call the same four C
functions against a single process-global engine.

graph TD
    S["NeedleSession.complete()<br/>calls serialised through one queue"]
    S --> R{"NeedleRuntime<br/>backend: auto"}
    R -->|"Capacitor, iOS"| A["NeedlePlugin.swift<br/>libneedle.a, ios-arm64, 14.2 MB<br/>needle2.cact linked as needle_weights"]
    R -->|"Capacitor, Android"| B["NeedlePlugin.kt via JNI<br/>libneedle.a, android-arm64, 20.7 MB"]
    R -->|"browser"| C["Web Worker<br/>needle.js 62 KB + needle.wasm 333 KB<br/>needle2.cact 13.7 MB fetched"]
    A --> E
    B --> E
    C --> E
    E["needle_init / needle_complete / needle_reset / needle_free<br/>process-global engine, no handle argument"]
    E -->|"NeedleTurn: calls[] or the empty call, plus confidence"| S

The queue is required, not defensive. needle_reset takes no handle and returns
nothing, so two overlapping complete() calls do not race on a missing mutex.
They interleave inside one KV cache and return each other's arguments.

The loop itself is a two-node LangGraph, so it can be interrupted before the
node that writes. This is the compiled graph, printed by
graph.getGraph().drawMermaid():

%%{init: {'flowchart': {'curve': 'linear'}}}%%
graph TD;
    __start__([<p>__start__</p>]):::first
    model(model)
    tools(tools)
    __end__([<p>__end__</p>]):::last
    __start__ --> model;
    model -.-> tools;
    model -.-> __end__;
    tools -.-> model;
    tools -.-> __end__;
    classDef default fill:#f2f0ff,line-height:1.2;
    classDef first fill-opacity:0;
    classDef last fill:#bfb6fc;

Two nodes, because Needle's contract is two moves. There is no "should I answer
in prose" edge to draw. The model conditional routes to END on a failed
turn, on the empty call, and on confidence < minConfidence. The tools
conditional routes to END when approval was declined, when the turn type is
respond, and at the step ceiling.

// A destructive call with no `approve` wired is declined, not run.
const needsApproval = session.toolbox.anyDestructive(turn.calls)
const approved = options.approve
  ? await options.approve(turn.calls, turn)
  : !needsApproval
if (!approved) return { stop: "declined" }

Tool errors are fed back into the next turn rather than thrown, which is how
"that preset does not exist" becomes a search instead of a crashed screen.

Tool scoping

Past five declared tools, Needle switches to retrieval. Schemas are embedded
once at init, the query is embedded per turn, and only the five best-scoring
tools enter the grammar. An unselected tool is unreachable, not merely unlikely.

The app has 50 quick actions. Measured against the real engine:

prompt	all 50 declared	scoped to one family
`delete my push day preset`	empty call, confidence 1.0	fires at 0.71 to 1.0
`take me to my shopping list`	`add_grocery_item("shopping list")`	`show_grocery_list()`, 0.86
`I drank a glass of water`	0.67	0.92

Across twelve representative prompts, scoped toolboxes picked the expected tool
14 times out of 15. The same prompts against all fifty succeeded roughly half
the time. Tools are therefore grouped into five families and each screen
registers only the family it is about.

const session = await needleQuickActions({
  navigate,
  scopes: ["food"],          // 12 tools competing, not 50
  confirm: (calls) => askUser(calls),
})

Three argument-design rules came out of the same testing:

// 1. No tool takes an id. Arguments may only carry values evidenced by the
//    input, and nobody says "k57d8...". Tools take names.
input: z.object({ preset: nameArg("the workout preset") })

// 2. No example values in argument descriptions. `e.g. "Push A"` made the
//    model send preset: "Push A" for a user whose preset is called Push Day.
//    Examples belong in the tool description.

// 3. Enums wherever a string gets filled in anyway. `meal` was a free string,
//    so "log a pot of greek yoghurt" returned meal: "greek yoghurt".
meal: z.enum(["breakfast", "lunch", "dinner", "snack"])

Fine-tuning

The stock checkpoint handles single calls well and multi-step chains poorly. The
missing behaviour is sequence: look the food up, then log what came back; read
the diary, then remove the entry you found.

Training data is generated through OpenRouter and validated against a schema
before it is written. Rows that fail validation are dropped and counted rather
than repaired.

export OPENROUTER_API_KEY=sk-or-...
python generate.py --count 600 --out chains.jsonl

Each row is one query, the reasoning, the answer chain, and the declared tools:

{
  "query": "log the greek yoghurt I had this morning",
  "reasoning": "...",
  "answers": [
    { "name": "search_food", "arguments": { "query": "greek yoghurt" } },
    { "name": "log_food", "arguments": { "name": "Greek yoghurt", "grams": 170 } }
  ],
  "tools": ["<at most five schemas>"]
}

The tools declared per row mirror the families the app puts in front of the
model. Training on all fifty while inference sees five teaches a retrieval
problem that does not exist at run time.

prepare.py cuts each example down to five schemas, keeping the tools the
answer names plus distractors to fill the slate:

python prepare.py more-data.jsonl prepared.jsonl

This step is load-bearing. The raw rows inline fifteen schemas each, which
renders as a 5,000-token prompt. The trainer masks everything but the answer and
truncates at --max-len, so the answer falls off the end and the loss is
exactly zero.

The LoRA runs on a Colab GPU:

needle finetune prepared.jsonl \
  --epochs 2 \
  --batch-size 8 \
  --lora-rank 16 \
  --max-len 2048 \
  --out onerep-lora.pkl

needle build checkpoints/needle2.pkl --lora onerep-lora.pkl --out needle2-onerep.cact

The epoch count sets the cosine schedule, so stopping early is not equivalent to
training for fewer epochs. If the first two step lines print 0.0000, the mask
is empty and the prepare step did not work.

Deployment is a file copy:

cp ~/Downloads/needle2-onerep.cact scripts/needle2-finetune/
bun run needle:tuned      # copies it to apps/mobile/public/needle/

Measure the adapter against the weights it replaces before shipping, with
negation cases included. One earlier adapter read "half a chicken breast"
correctly and also logged greek yoghurt for "I skipped lunch today".

2. Form coach

Film up to three angles of a set and get back a technique analysis. The pose
pipeline runs entirely on the device; only numbers and five small stills are
uploaded.

graph TD
    A["clip, sampled at SAMPLE_FPS = 12"]
    A -->|"letterbox → Float32Array [1, 3, 448, 448]"| B
    B["yolo11n_pose_448_fp32.onnx, 11 MB<br/>10.6 ms/frame, wasm SIMD, 1 thread"]
    B -->|"[1, 56, N] → argmax over N anchors"| C
    C["inverse letterbox<br/>17 COCO keypoints (x, y, score)"]
    C -->|"cocoToH36m + cropScale → [1, T, 17, 3], T ≤ 243"| D
    D["motionbert_lite_int8.onnx<br/>2D→3D lift, attends over the whole window"]
    D -->|"[1, T, 17, 3] world landmarks"| E
    E["One Euro filter<br/>minCutoff 0.5, beta 6, dCutoff 1"]
    E -->|"body-framed, camera-independent"| F
    F["rep detection over distance signals<br/>hip_to_ankle · wrist_to_shoulder · wrist_to_hip"]
    F -->|"FormCoachCapture: reps[], timeline, 5 stills"| G["upload"]

Two model choices are worth recording.

The detector stays fp32. Quantizing it to int8 makes it ten times slower,
206 ms a frame against 21 ms, because onnxruntime has no fast int8 convolution
kernel on wasm and dequantizes on every inference. It runs once per frame, so it
stays fp32 at 11 MB. The lifter is quantized, because it runs once per clip and
shrinking it is what gets a 64 MB graph under Cloudflare Pages' 25 MiB per-file
cap.

The detector is traced at 448, not 640. Half the compute, 10.6 ms a frame
against 21.8 ms, with a mean per-joint disagreement of 32 mm on a deadlift.

Averaging reps was the main defect

The original pipeline collapsed every rep in a set into one canonical rep and
measured that. This is the least useful summary available: a lifter whose third
rep collapses still has a respectable mean, and the fault disappears.

Nothing is averaged now. Every reading is taken per rep and reported as a
spread:

{
  mean: 96.4,
  min: 78.2,
  max: 104.1,
  spreadAcrossReps: 25.9,
  perRep: [104.1, 101.7, 78.2, 95.4, 92.6]   // in performance order
}

The type carries the same rule:

export type FormCoachCapture = {
  /** Every rep from every angle, whole and unaveraged. */
  reps: KinematicRep[]
  /**
   * All reps averaged into one, on pre-2026 captures only. The client no
   * longer produces this and nothing reads it. Declared so an older capture
   * blob still parses.
   */
  canonical?: KinematicFrame[]
}

The prompt states the consequence directly: a fault lives in the worst rep, and
a set whose first rep is fine and whose last rep is not is a more useful finding
than either rep alone.

Geometry, then judgement

The kinematics module knows nothing about any exercise. Every function answers
"what is this angle" or "how far is this joint off that line". No function names
a correct value, a threshold, or a movement. The model is given thirteen
measurement tools over the capture and decides which ones the exercise turns on.

The prompt section that did the most for output quality is the list of what the
data cannot show:

- There are no landmarks along the spine between the shoulders and the hips.
  Spinal flexion, lumbar rounding and butt wink are NOT measurable. Do not
  comment on them, even to say they look fine.
- Bar position, bar path and grip width are invisible.
- The skeleton ends at the ankles. Ankle dorsiflexion, foot angle and heel
  lift are NOT measurable.
- The depth axis is the least reliable in single-camera estimation.

Without it, the model wrote confident and entirely invented paragraphs about
spinal position, derived from a shoulder-to-hip line.

3. Coach

The cloud coach answers questions, generates its own UI, and can make real
writes to the user's data.

Context

buildCoachWorkspace runs roughly two dozen indexed queries in parallel and
serialises the result into the user turn. Because it is stringified with no
formatter, an unbounded payload is both a cost problem and a truncation risk. A
budget function trims it:

/** Roughly 15k tokens, leaves room for the system prompt, history, and reply. */
export const MAX_WORKSPACE_CHARS = 60_000

Trim steps are ordered lowest-value first and applied until the payload fits.
Ids and names are never trimmed, because operations reference presets and
recipes by id: cutting them would break the model's ability to act rather than
narrow what it knows. Fields that were actually cut get named in a truncated
array so the model can hedge its claims.

Memory eviction follows four rules, in order:

Anything the user typed is never evicted.
Injury, safety, medical, allergy and constraint memories are never evicted.
Weekly digests are capped at 14 independently of the overall ceiling.
Everything else goes oldest first, down to 60.

The agent graph

Tool-using features run through runOpenAiAgent in convex/ai/provider.ts.
This is the compiled graph, printed verbatim by graph.getGraph().drawMermaid():

%%{init: {'flowchart': {'curve': 'linear'}}}%%
graph TD;
    __start__([<p>__start__</p>]):::first
    agent(agent)
    tools(tools)
    extract(extract)
    finalize(finalize)
    __end__([<p>__end__</p>]):::last
    __start__ --> agent;
    extract --> __end__;
    finalize --> __end__;
    agent -.-> tools;
    agent -.-> extract;
    agent -.-> finalize;
    tools -.-> agent;
    tools -.-> finalize;
    classDef default fill:#f2f0ff,line-height:1.2;
    classDef first fill-opacity:0;
    classDef last fill:#bfb6fc;

The two conditional edges:

.addConditionalEdges(
  "agent",
  (state) => {
    const last = lastAiMessage(state.messages)
    if (last.tool_calls?.length) return "tools"
    return finishReasonOf(last) === "stop" ? "extract" : "finalize"
  },
  ["tools", "extract", "finalize"],
)
.addConditionalEdges(
  "tools",
  (state) => (state.steps >= maxSteps ? "finalize" : "agent"),
  ["agent", "finalize"],
)

finalize exists because a run that ends on a tool call at the step ceiling, or
mid-sentence at the token ceiling, never produced the schema-shaped answer.
Throwing there discards a transcript that already contains the measurements. So
it re-asks once with the tools still bound but tool_choice: "none", and double
the token budget.

UI generation

Coach replies are not prose. The model returns a JSON object constrained by a
schema, and the useful part is uiBlocks.

type CoachChatResult = {
  reply: string              // one orienting sentence, 8 to 20 words
  uiBlocks: CoachUiBlock[]   // at most 3
  operations: CoachOperation[]
  artifacts: CoachArtifact[]
}

There are six block types. Five are fixed shapes:

type CoachUiBlock =
  | { type: "card"; label: string; title: string; detail: string }
  | { type: "stat_group"; title: string; stats: CoachUiStat[] }
  | { type: "checklist"; title: string; items: ChecklistItem[] }
  | { type: "goal"; title: string; detail: string; durationDays: number; tasks: TaskDraft[] }
  | { type: "action_row"; title: string; actions: Array<{ label: string; action: CoachUiAction }> }
  | { type: "interactive_card"; /* below */ }

The sixth is a composable canvas rather than a template. The model arranges
primitives in whatever order fits the request:

type CoachInteractiveElement =
  // presentation
  | { type: "text"; text: string; emphasis?: "quiet" | "strong" }
  | { type: "section"; title: string; detail?: string }
  | { type: "divider"; label?: string }
  | { type: "key_value"; items: Array<{ label: string; value: string; detail?: string }> }
  | { type: "progress"; label: string; value: number; max: number; unit?: string }
  | { type: "list"; style: "bullet" | "number" | "timeline"; items: ListItem[] }
  | { type: "metric_group"; metrics: Metric[] }
  // input
  | { type: "stepper"; id: string; label: string; value: number; min: number; max: number; step: number }
  | { type: "range"; id: string; label: string; value: number; min: number; max: number; step: number }
  | { type: "choice"; id: string; label: string; value: string; options: string[] }
  | { type: "rating"; id: string; label: string; value: number; max: number }
  | { type: "toggle"; id: string; label: string; detail?: string; value: boolean }

Nothing the model emits is trusted. normalizeCoachUiBlocks walks the array,
clamps every string to a per-field length, coerces every number into a range,
and returns null for anything it does not recognise. Nulls are filtered out,
so an unknown element type degrades to a missing element rather than a thrown
render.

if (type === "section") {
  const title = clampText(row.title, 64)
  if (!title) return null
  return {
    type,
    title,
    ...(clampText(row.detail, 140) ? { detail: clampText(row.detail, 140) } : {}),
  }
}

The prompt sets a word budget per field, and states the consequence so the model
does not read brevity as laziness:

BE SHORT. This is read on a phone, usually mid-session. Every field has a
budget: block and card titles 6 words, card and stat detail one sentence of
20 words, checklist and list items 12 words, operation summaries 15 words.
Exceeding these is a worse answer, not a more thorough one.

Making generated UI actionable

A card of numbers is read-only. What makes interactive_card worth the schema
is the submit contract, which lets a control the model placed drive the values
that get written.

submit?: {
  type: "log_nutrition"
  label: string
  name: string
  meal: string
  calories: number            // macros for ONE baseQuantity
  protein: number
  carbs: number
  fat: number
  quantityControlId?: string  // the id of a stepper or range in `elements`
  baseQuantity?: number       // what the macros above correspond to
  mealControlId?: string      // the id of a choice in `elements`
  assumptions: string[]
}

The client keeps live control state keyed by element id, derives a scale factor,
and multiplies:

function scaleFor(controlId?: string, baseQuantity?: number) {
  if (!controlId) return 1
  const current = values[controlId]
  const initial = initialValues[controlId]
  if (typeof current !== "number") return 1
  const base =
    typeof baseQuantity === "number" ? baseQuantity
    : typeof initial === "number" && initial > 0 ? initial
    : 1
  return Math.max(0, current / base)
}

Pressing the button emits an ordinary operation, so a generated card writes
through the same validation, application, and undo path as anything else:

await onSubmit({
  type: "log_nutrition",
  confirmation: "auto",
  summary: `Log ${block.submit.name}`,
  assumptions: block.submit.assumptions,
  warnings: [],
  name: block.submit.name,
  meal: typeof selectedMeal === "string" ? selectedMeal : block.submit.meal,
  calories: Math.round(block.submit.calories * factor),
  protein: Math.round(block.submit.protein * factor * 10) / 10,
  carbs: Math.round(block.submit.carbs * factor * 10) / 10,
  fat: Math.round(block.submit.fat * factor * 10) / 10,
})

The whole path, from model turn to a row in the diary:

graph TD
    M["model turn, response_format: json_schema"]
    M -->|"{ reply, uiBlocks[], operations[] }"| N
    N["normalizeCoachUiBlocks()<br/>clampText / clampNumber / clampInteger per field"]
    N -->|"unrecognised type → null → filtered"| B
    B["CoachUiBlock[], sliced to 3"]
    B --> R["React renderer, one branch per element type"]
    R -->|"stepper · range · choice · rating · toggle"| V
    V["values: Record&lt;elementId, number or string or boolean&gt;"]
    V -->|"scaleFor(quantityControlId, baseQuantity)"| O
    O["CoachOperation: log_nutrition, macros × factor"]
    O --> A["validate → apply → recordUndoableAction()"]

The photo-logging flow uses this end to end. The meal parser returns detections
matched against the food database, and the prompt instructs the model to build
an interactive_card from the matched entries, with a quantity stepper and its
assumptions stated, rather than answering in prose.

Writes are proposals

The model never touches the database. It emits typed operations, each carrying
four mandatory fields:

type CoachOperationMeta = {
  confirmation: "auto" | "confirm"
  summary: string
  assumptions: string[]
  warnings: string[]
}

Idempotency is a run row keyed by request id. A completed run replays its stored
result; a run marked running within the last 60 seconds is rejected.

recordUndoableAction is exported rather than private, because the REST API and
the MCP endpoint write through the same feed. An agent that logs a meal over
HTTP appears in the same list, behind the same undo button, as one that logged
it through chat.

That's the whole system. If you have questions about any part of it, or you've solved something here differently, leave a comment. I'd like to hear it.

DEV Community: Ananth H