DEV Community: HYEONGSEOB JO

My Picnic Robot Memorized the Table, Not My Words

HYEONGSEOB JO — Sun, 25 Oct 2026 15:27:22 +0000

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 3: Slow and Steady

What I Built

In Week 2, fine-tuning worked: one demo per item, relabeled with the ways people actually ask, taught my open robot model (SmolVLA) new phrasings (31% to 80%) for the price of a coffee. It still packed one item at a time.

So this week I pushed the same idea toward the laziest picnic request of all: say one sentence, "grab the cream cheese and the salad dressing", and walk away while the robot puts both in the basket, in that order. If fine-tuning could teach it how I talk, could it teach it a two-item task?

I built the pieces around the model to make that a single request:

Piece	What it does
Request splitter	"grab the cream cheese and the salad dressing" becomes two steps, in the order named
Step checker	The simulator reports when an item is in the basket, and the instruction moves on to the next item
SmolVLA	Does one step at a time from two camera images and the arm state
Scripted demo generator	Writes training demos from the simulator's object positions, no teleoperation

It did not go to plan, and the reasons turned out to be the interesting part.

Demo

"Pick up the cream cheese ..." then, once it is in, "pick up the salad dressing ...". Neither item is the one this scene was trained on (bbq sauce). The cream cheese lands at step 95, the dressing at step 371 (20 steps per second).

The honest number: this is 1 success in 80 two-item episodes. The rest of this post is why.

Finding 1: the robot memorized the table, not the sentence

Every LIBERO-Object scene has six groceries, but the model only ever learned one task per scene: "pick up {that scene's item} and place it in the basket". It scores 96% on that benchmark. So I asked for each of the six items in each of the ten scenes.

Published SmolVLA	Success
The scene's own item	10/10
Any other item on the same table	1/50

When it failed on another item, it lifted the scene's own item instead in 24 of 26 episodes. It was not reading the item name; it was recognizing the table. I am not the first to see this: VLA or IL? by @fn8211 shows the same with π0.5 and summarizes LIBERO-PRO and LIBERO-Plus, which found that fine-tuned models keep the same trajectory when the instruction changes. This is the same failure in a 450M model, measured on every item of every scene.

A single sentence for two items ("put both the cream cheese and the butter in the basket") failed 0/3: the cream cheese, that scene's item, went in every time; the butter never did.

This also explains Week 2. There, every new phrasing still named the scene's own item, so the table alone gave the right answer. Two items in one scene is the first time the item name has to matter.

Finding 2: breaking the shortcut is not the same as reading

LIBERO-Object has no demos of "pick the other item", so I generated them. A short script reads the object positions from the simulator, moves the gripper over the asked item, grasps it, and places it in the basket. The training set had single picks of all six items in every scene (8 per item), chains of two to four items, and a few original demos; no teleoperation.

	Published	Fine-tuned on the scripted demos
Scene's own item	10/10	3 to 5 of 10 (3k, 6k, 9k steps)
Other items	1/50	3 to 5 of 50
Wrong lifts that were the scene's own item	24 of 26	12 of 40 (3k), 16 of 41 (6k)

The fine-tune broke the habit: wrong picks mostly stopped going to the scene's own item. But they did not go to the asked item either. I checked whether a wrong pick at least shared the asked item's kind (box, carton, can, bottle): 6 of 40 and 10 of 41, against 7.6 expected by chance. When it is wrong, the item name plays no role.

Removing a shortcut did not create the skill it was standing in for. The model stopped recognizing the table, and nothing took its place.

Two items, in order

Model (same 80 episodes: 40 trained item orders, 2 table layouts)	Both items, in order	First item
Fine-tuned on the first scripted chains (6k)	3/80
Fine-tuned on the scripted demos above (9k)	1/80	15/80

With picking the right item at about 10%, two in a row is about 1%. The step checker and splitter work; the bottleneck is grounding one item at a time.

The lazy payoff

	Effort
What I wanted	One sentence, then walk away
What works today	One sentence, then watch: the robot gets the order right about 1 time in 80
What the pipeline saves already	No teleoperation: 657 training demos written by a script in under an hour on the rented machine's CPUs

The data side is lazy in the good way. The robot side is not there yet, and I would rather show that than a cherry-picked clip.

How I Built It

Part	Choice
Model	`lerobot/smolvla_libero`, 450M parameters, LeRobot 0.6.1
Simulation	LIBERO-Object on robosuite and MuJoCo
Demos	Scripted pick-and-place from simulator object poses, human pace, failed attempts discarded
Fine-tuning	Whole model (vision encoder frozen), batch 32, lr 2.5e-5, 9,000 steps, about 75 minutes
Evaluation	Every item in every scene (60 episodes), two-item chains (80 episodes), failure kinds from the lift height of every object
Hardware	1x A100 80 GB on RunPod

The request splitter is the smallest part:

def split_request(request: str) -> list[str]:
    text = request.lower()
    names = {item: item for item in ITEMS} | SHORT_NAMES
    found = []  # (start, end, item)
    # Longest names first, so "cream cheese" is taken before "cheese".
    for name in sorted(names, key=len, reverse=True):
        for match in re.finditer(rf"\b{re.escape(name)}\b", text):
            if not any(match.start() < end and start < match.end() for start, end, _ in found):
                found.append((match.start(), match.end(), names[name]))
    items = []
    for _, _, item in sorted(found):
        if item not in items:
            items.append(item)
    return items

split_request("grab the cream cheese and the salad dressing") returns ["cream cheese", "salad dressing"]; each item becomes "pick up the {item} and place it in the basket", the sentence the model was trained on.

Limits: all numbers are from one training layout per scene (state 46) or two (46 and 47); simulation only; one training run per setting.

Code

johyeongseob / hacktoberfest-2026

Hacktoberfest 2026 projects and experiments focused on open-weight AI, VLA, and Physical AI.

Hacktoberfest 2026

A month-long exploration of open-weight AI, VLA, and Physical AI.

Goals

Explore open-weight AI models
Build VLA/Physical AI projects
Participate in DEV Challenges
Document experiments and learnings

Projects

hacktoberfest-2026/
├── dev-challenge/          # Hacktoberfest DEV Challenge entries
│   ├── weekend-challenge/  # Shared Workbench Reset Assistant (local open-weight VLM)
│   ├── week-1/             # Picnic Packing with SmolVLA (paraphrased instructions in LIBERO)
│   └── week-2/             # Teaching the Picnic Robot My Words (SmolVLA fine-tune on a rented GPU)
├── global-hack-week/       # Global Hack Week: Hacktoberfest challenge write-ups
│   ├── 01-registration.md  # Registration challenges
│   └── github/             # GitHub challenges (Codespaces, repo management, Copilot)
├── devrelay/               # DevRelay usage notes and reviewed agent session transcripts
└── models/                 # Shared local model weights; excluded from Git

DEV Challenge entries

Folder	Theme	Project	Post
weekend-challenge	Build for a Friend: solve a real problem for someone you know	Shared Workbench Reset Assistant: a local open-weight

…

View on GitHub

Where	What
`week-3/datagen`	Scripted demo generator and recorder
`week-3/eval`	Step-by-step rollout, request splitter, every-item-in-every-scene test
`week-3/train`	Training-set builder, normalization fix, training script

Why Does Open Innovation Matter?

Open piece	What it made possible
Open weights	I could fine-tune the model and inspect every checkpoint
Open simulator and benchmark	The simulator knows where every object is, so a script could write demos no one recorded
Open data	The original LIBERO demos are there to mix back in against forgetting
Open training code	Reading LeRobot's training loop let me check what fine-tuning changes, down to the normalization statistics
Open discussion	Others had already measured the same shortcut, so I knew what I was looking at

A closed model would have given me a success rate. Open pieces let me find out why the robot fails.

What's Next

Grounding is the missing skill, and a detector already has it. Next I want to mark the asked item on the camera image (first with the simulator's own boxes, then with an open-vocabulary detector) and see whether SmolVLA learns to follow the mark.

Notes from the Bench

Things I tried along the way that did not make the main story.

1. My scripted demos had hidden pauses. The script moved the gripper in proportion to the remaining distance, and some targets were out of reach: the fingers stopped against a tall bottle or the basket floor, and the script kept nudging until its step limit. The average pace still matched the human demos (about 146 steps per item), so success rates and episode lengths looked fine.

A "steady" version ends a move once the gripper stops getting closer. After retraining on it, "picked up nothing" dropped from 34 to 18 of 60 at 3,000 steps (Fisher's exact test, p ≈ 0.006; the published model: 22). The steady set also had more single picks (about 450 vs 168), so this is the likely cause, not an isolated one. @ankk98 saw the same in teleop data: idle footage teaches a policy to linger. Slow and steady, it turns out, has to be steady.

My Agent Session

I built this with Claude Code over a week. The session below is a curated slice: the diagnosis, the scripted demos, the hidden pauses, the grounding check, and the one two-item success.

Two picnic items in one sentence: a shortcut, hidden pauses, and grounding (curated, translated)

You

[Note: condensed English translation of a Claude Code session held in Korean, October 19-26, 2026. Messages are selected and shortened; wording is translated, not verbatim.]

Week 3 is Slow and Steady. The laziest picnic request: one sentence, two items, walk away. Can SmolVLA put two LIBERO-Object items in the basket, one after the other? I want the hardest version I can actually finish.

Agent

First, what the published model does today. Two tests in one LIBERO-Object scene: one sentence for both items (LIBERO-10 style), and the item sentences one at a time, switching when the simulator says the first item is in the basket.

▸ chain_rollout.py scene 1: 'put both the cream cheese and the butter in the basket', then step ...

Input

scene 1: 'put both the cream cheese and the butter in the basket', then step by step

Output

both modes 0/3: the cream cheese (that scene's item) goes in every time, the butter never

You

Is it the butter, or is it the order?

Agent

Ask for every item in every scene, one at a time. If the model reads the item name, other items should work about as well as the scene's own.

▸ run_cross_items.sh 10 scenes x 6 items, one episode each

Input

10 scenes x 6 items, one episode each

Output

own item 10/10, any other item 1/50

And when it fails on another item, it lifts the scene's own item in 24 of 26 episodes. It recognizes the table, not the sentence. LIBERO-Object pairs each scene with exactly one item, so the table alone always gave the right answer.

You

There are no demos of picking the other item. Can we make them without teleoperation?

Agent

The simulator knows every object's pose, so a short script can move the gripper over the asked item, grasp it, and place it in the basket. Fixes along the way: aim at the object center (small boxes were missed from 1 cm above), hold the gripper orientation, special settings for tall bottles, and separate spots in the basket so a second item does not land on the first.

▸ run_scripted_all.sh every item in every scene, 5 training states each

Input

every item in every scene, 5 training states each

Output

most items above 90%; ketchup 8 -> 25 of 30 after the tall-bottle settings

You

Record chains of two to four items and fine-tune. Plan the item orders so some are only ever used for testing.

Agent

The first fine-tune was worse than the published model. compare_demos.py showed why: the scripted demos took about 300 steps per item, twice the human demos' 146. A faster pace matched the human one, and after retraining: 3 of 80 two-item episodes in order. In the 10 episodes where the second item went in, the model had put the order's second item in first. The bottleneck is choosing the item, not chaining.

You

Then teach it to choose. Same table, every item.

Agent

Added single picks of all six items in every scene, so the table alone no longer gives the answer. Result: other items rose a little (1/50 -> 4-8/50), but the scene's own item fell from 10/10 to 0/10 at the first checkpoint.

You

Why would it get worse at what it already knew?

Agent

First suspect: lerobot-train replaces the checkpoint's normalization statistics with the new dataset's. The scripted data had an action mean of -0.34 on z instead of -0.09 and a 20x narrower orientation spread.

▸ use_base_stats.py keep smolvla_libero's mean and std, retrain

Input

keep smolvla_libero's mean and std, retrain

Output

own item 2/10 at 3k steps either way: not the cause

You

Why does the second video stop in the middle?

Agent

Measured frame-to-frame change per second: the arm sits nearly still for about five seconds over a tall item and again over the basket. The script moves in proportion to the remaining distance, and some targets are out of reach (fingers against a bottle or the basket floor), so it nudged until its 120-step limit. The average pace still matched the human demos, so success rates and lengths never showed it.

▸ scripted_pick.py --pace steady end a move once the gripper stops getting 1 mm closer for 5 steps

Input

end a move once the gripper stops getting 1 mm closer for 5 steps

Output

longest pause in 15 samples: 0.2-0.5 s (was 5-6 s)

You

Retrain on the steady demos.

Agent

Same normalization, steady data. 'Picked up nothing' dropped from 34 to 18 of 60 at 3,000 steps (Fisher p = 0.006), back to the published model's level (22). Caveat: this run also had more single picks, so it is the likely cause, not an isolated one.

You

When it picks the wrong item, which item does it pick?

Agent

Every rollout now records how high every object on the table was lifted. Wrong lifts that were the scene's own item: 24 of 26 before fine-tuning, 12 of 40 after. The shortcut is broken. But does a wrong pick at least share the asked item's kind (box, carton, can, bottle)? 6 of 40 and 10 of 41, against 7.6 by chance. When it is wrong, the name plays no role.

Breaking the shortcut did not create grounding.

You

Today's minimum goal: one real two-item success.

Agent

▸ run_chain_eval.sh steady 9k checkpoint, 40 trained orders x 2 layouts, videos on

Input

steady 9k checkpoint, 40 trained orders x 2 layouts, videos on

Output

2 items: all in 1/80 (scene 3: cream cheese at step 95, then salad dressing at step 371)

Neither item is the scene's own (bbq sauce). One honest success, with the instruction switching exactly when the first item landed.

You

Would mixing the original LIBERO demos back in stop the forgetting?

Agent

Co-training: all 457 original LIBERO-Object demos, the steady single picks, and two-item chains only.

▸ run_chain_eval.sh 3k, 6k, 9k checkpoints

Input

3k, 6k, 9k checkpoints

Output

two items in order 1/80, 3/80, 1/80; own item 2/10 at 3k

No gain within the noise. More of the old data did not bring back the old skill, and it did not add grounding.

You

The pipeline sat idle for two hours. What happened?

Agent

My bug: the wait loop used pgrep -f to wait until no data-preparation process was running, and its own command line contained that name, so it waited for itself. I had made the same mistake once before with the training process. Restarted the steps directly without the wait.

You

What is the contribution this week?

Agent

Main story: the model memorized the table (1/50, 24 of 26 wrong lifts to the scene's own item), and breaking that shortcut did not make it read (wrong picks at chance). The two-item pipeline works, and the one success is real. The hidden pauses, the normalization check, and co-training go into a notes section at the end.

You

Write the post with DevRelay. Start from Week 2: fine-tuning worked there, so try a two-item task here.

Agent

Checked the Week 3 rules with get_challenges, searched DEV for related posts (a controlled LIBERO-Object dataset showing the same shortcut with pi0.5, and a teleop data-quality rubric that flags idle footage), staged an unpublished draft with create_article, and kept the Markdown source in the repo. This session is saved with submit_agent_session and embedded at the end of the post.

Note: I worked with the agent in Korean. This is a condensed English translation of the key moments, not a verbatim transcript.

Disclosure: I set the goal, ran every training and evaluation command, and reviewed every result. An AI coding agent (Claude Code) wrote the scripts with me, analyzed the results, made the charts, and drafted this post.

For the Price of a Coffee, My Picnic Robot Learned How I Talk

HYEONGSEOB JO — Sun, 18 Oct 2026 16:05:17 +0000

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 2: Side Hustle

What I Built

In Week 1, an open robot model (SmolVLA) packed my picnic only when I used its exact training sentence. This week I fine-tuned it on a few demos, each labeled with the ways people actually ask, and priced the fix.

For the price of a coffee, the robot learned how I talk.

	Before	After 1 demo per item
Phrasings I taught	31%	80%
Phrasings it never saw	29%	78%
Demos needed		4 in total
GPU cost		about $1.50

Who would pay: anyone whose robot meets people who did not write its training data.

Customer	What they say	What they get
Café with a kitchen arm	"grab me the oat milk"	A model that understands the staff
Care home	"could you get my cup?"	A robot that does not need exact commands
Robot vendor abroad	Their customers' dialect	A fine-tune per market, priced per job

Demo

Upper clip of each pair: the published model. Lower clip: after one demo per item, same scene.

Rows	Request
Top two	"pack the {item} for our picnic" (taught)
Bottom two	"the basket still needs the {item}" (never seen in training)

The results

Six fine-tuned models (1 to 40 demos per item) plus the published one, all tested on the same 420 episodes.

Demos per item	Taught	Never seen	Original sentence	6 untrained items
0 (published)	31%	29%	100%	87%
1	80%	78%	95%	60%
3	86%	74%	90%	70%
5	84%	74%	88%	70%
10	86%	77%	92%	72%
20	92%	79%	98%	68%
40	92%	78%	98%	68%

1. One demo per item does most of the work

	Phrasing	0 demos	1 demo
A ●	grab the {item} and put it in the basket	23/40	35/40
B ●	{item} in the basket	3/40	29/40
C ●	pack the {item} for our picnic	1/40	31/40
D ●	could you put the {item} in the picnic basket?	23/40	33/40
E ○	toss the {item} into the basket	8/40	33/40
F ○	the basket still needs the {item}	7/40	25/40
G ○	can you get the {item} into the basket for me?	18/40	32/40
H ○	move the {item} over to the basket	13/40	34/40

E to H were frozen before training, under rules that ban the verbs and sentence shapes of A to D.

2. The twist: more demos did not help with new words

	1 demo	40 demos
Taught phrasings	80%	92%
Never-seen phrasings	78%	78%
Orange juice, never-seen	36/40	23/40

Same scene, same request. Likely reason: every demo carried the same five sentences, so more demos taught more ways to move, not more ways to be asked. One training run per N and cream cheese went the other way, so this is a strong hint, not a proof (see also VLA or IL? by @fn8211).

Lesson for the side hustle: collect more phrasings, not more demos.

3. The fine print: it forgets what it was not taught

Original sentence	0 demos	1 to 40 demos
4 trained items	100%	88-98%
6 untrained items	87%	60-72%

The fine-tune only saw four items, so the model drifted toward them. Ketchup, the first item my robot ever packed in Week 1, still works in 6 to 8 of 10 scenes, but loses this one. Next fix to try: mix in a few demos of every item.

What It Costs

	Time	Money
One fine-tune (shared GPU, 3000 steps)	48 min	$1.50
One fine-tune (GPU alone, 1000 steps, estimated)	10 min	$0.30
Whole experiment (6 models, about 4,400 evaluation episodes)	6 h	$11
Buying the GPU instead		$10,000+

Validation stopped improving after step 1000 for 3+ demos per item (training chart below), so a real job can stop early.

How I Built It

Part	Choice
Model	`lerobot/smolvla_libero`, 450M parameters
Data	Public `lerobot/libero` demos, first N per item, 4 items
The trick	A wrapper around `lerobot-train` relabels each sample with one of 5 sentences at random; no new recordings
Training	Whole model, 3000 steps, batch 32, lr 2.5e-5
Checkpoint choice	Best of steps 1000/2000/3000 on held-out scenes, taught phrasings only
Test	420 episodes per model, LIBERO-Object in MuJoCo, same scenes and seeds for all
Hardware	1x A100 80 GB on RunPod, training and evaluation side by side in tmux

Gotcha: the pod showed 256 CPUs but had 16; evaluation threads starved training until I set OMP_NUM_THREADS=1 (setup guide).

Limits: one training run per N; simulation only; phrasing G turned out closer to D than my rules intended.

Code

johyeongseob / hacktoberfest-2026

Hacktoberfest 2026 projects and experiments focused on open-weight AI, VLA, and Physical AI.

Hacktoberfest 2026

A month-long exploration of open-weight AI, VLA, and Physical AI.

Goals

Explore open-weight AI models
Build VLA/Physical AI projects
Participate in DEV Challenges
Document experiments and learnings

Projects

hacktoberfest-2026/
├── dev-challenge/          # Hacktoberfest DEV Challenge entries
│   ├── weekend-challenge/  # Shared Workbench Reset Assistant (local open-weight VLM)
│   ├── week-1/             # Picnic Packing with SmolVLA (paraphrased instructions in LIBERO)
│   └── week-2/             # Teaching the Picnic Robot My Words (SmolVLA fine-tune on a rented GPU)
├── global-hack-week/       # Global Hack Week: Hacktoberfest challenge write-ups
│   ├── 01-registration.md  # Registration challenges
│   └── github/             # GitHub challenges (Codespaces, repo management, Copilot)
├── devrelay/               # DevRelay usage notes and reviewed agent session transcripts
└── models/                 # Shared local model weights; excluded from Git

Development Environment

Component	Configuration
Host	Windows
CPU	Intel Core Ultra X7 358H; 16 logical CPUs visible in WSL
GPU	Intel Arc B390 GPU; Windows driver 32.0.101.8356
RAM

…

View on GitHub

Where	What
`dev-challenge/week-2`	Relabeling wrapper, training and evaluation scripts, all tables, chart and video scripts
Hugging Face Hub	All six fine-tuned models with a model card

Why Does Open Innovation Matter?

Open piece	Why the business needs it
Open weights	You cannot fine-tune what you cannot download
Open data	I relabeled public demos instead of recording any
Open benchmark	I could measure what one demo buys and what it breaks
Hourly GPUs + small model	The customer pays for minutes, not a GPU or a per-call API
A model file	The customer runs it offline on their own robot, free of anyone's pricing

No paying user yet. The first person I would show it to: anyone who has watched a robot arm ignore a perfectly reasonable request.

My Agent Session

I built this with Claude Code over a week. The session below is a curated slice: the experiment design, the GPU run and its one bug, the results, and the charts and videos.

Teaching a picnic robot new words: fine-tuning SmolVLA on a rented GPU (curated, translated)

You

[Note: condensed English translation of a Claude Code session held in Korean, October 12-18, 2026. Messages are selected and shortened; wording is translated, not verbatim.]

Week 2 is Side Hustle. Can we go beyond Week 1's inference and actually fine-tune SmolVLA? I want Weeks 2 to 4 to continue the picnic robot story, and I'd rather pick something that shows real VLA work than something safe.

Agent

Proposal: teach lerobot/smolvla_libero the four Week 1 phrasings that failed, with N demos per item, and measure success against N. The curve is a price list for a "fit the robot to how your users talk" service.

You

Isn't smolvla_libero already fine-tuned on LIBERO-Object? Wouldn't training and testing be the same thing?

Agent

It has seen every Object demo, but only with one sentence per task. What is new is the wording. Split: train on the first N demos per item relabeled with the original sentence and A to D; choose checkpoints on initial states 10 to 14; test on states 0 to 9; and add four unseen phrasings E to H that are only ever used in the test.

You

Write E to H.

Agent

Wrote E to H under fixed rules (none of the verbs pick, place, grab, put, pack; a sentence structure different from A to D; the basket stays the target) and froze them in data/wordings.json before any training.

You

Train on the four items with N = 1, 3, 5, 10, 20, 40. With the same number of steps, won't the small N runs overfit?

Agent

Every N gets the same 3000 steps (lr 2.5e-5, warmup 200, batch 32), and checkpoints are scored on the validation states while training runs, so overfitting would show up as a falling validation curve. The best checkpoint per N goes to the test.

You

Can one command run everything?

Agent

run_all.sh starts one tmux session: the N = 0 test, the training chain, one watcher per run, and a final waiter. A dry run with stub scripts caught two bugs before any GPU time: parallel jobs appending to one results file corrupted it (now one file per job, merged at the end), and results depended on job grouping (now the RNG is reseeded per episode).

▸ bash dry run of run_all.sh with stub training and evaluation scripts

Input

$ dry run of run_all.sh with stub training and evaluation scripts

Output

interleaved lines found in results.jsonl; after the fix, a clean rerun

You

DigitalOcean, the sponsor, won't let my new account create a GPU Droplet. What now?

Agent

A new account needs a $250 prepay or a support ticket that takes one to two business days. RunPod rents an A100 SXM 80 GB by the hour with prepaid credit, so I rewrote setup.md for a RunPod pod. On the pod, the smoke test (butter, original sentence) succeeded, and a 200-step training probe used 28 GB of GPU memory at 0.46 s per step.

You

Training is stuck at step 0.

Agent

▸ top CPU use on the pod

Input

CPU use on the pod

Output

evaluation processes at 400-600% CPU each

nproc reports the host's 256 CPUs, but the pod has 16 vCPU, so every evaluation process spawned hundreds of threads and starved training. Fix: OMP_NUM_THREADS=1 (and MKL, OpenBLAS) in the environment, then restart. To fit the time budget, validate at steps 1000, 2000, and 3000 only.

You

I'm going to close everything and turn off my laptop. Will it keep running?

Agent

Yes, the jobs live in tmux on the pod. Before leaving, check that training passed step 1000, the first checkpoint exists, no process is far above 100% CPU, and there is disk and credit left.

Agent

All six runs finished in about six hours and $11. Taught phrasings: 31% at N = 0, 80% at N = 1, 92% at N = 20 and 40. Unseen: 29% at N = 0, then 74-79% for every N. Six untrained items: 87% to 60-72%.

You

So unseen phrasings don't scale with N?

Agent

No. Per item it is mixed: cream cheese rises, orange juice falls from 36/40 to 23/40, and they cancel out. A likely reason: every demo carries the same five sentences, so more demos add ways to move, not ways to be asked. With one seed per N, it is a hypothesis.

You

The original sentence was in the training mix. Why do the other items still fail?

Agent

The original sentence was only there for the four trained items; the other six never appeared, so the whole model drifted toward the four. The checkpoint choice never looked at them, and even the step-1000 checkpoints forgot as much, so it happens early. Mixing in a few demos of every item is the fix to try next.

You

What is the most important result?

Agent

One demo per item does most of the work: that is the price, and the Side Hustle story. Then the twist (more demos do not help new phrasings) and the cost (forgetting).

You

Make video grids: the four items, a taught and an unseen phrasing, before and after one demo.

Agent

▸ make_grid.py items

Input

items

Output

4x4 grid: N = 0 fails in all 8 tiles, N = 1 succeeds in all 8

Scenes chosen by a rule: N = 0 fails and every fine-tuned model succeeds. Also a 1x4 orange juice grid (N = 1, 10, 20, 40) and a butter-versus-ketchup grid for forgetting.

You

In the line chart, use filled dots for what was taught and hollow dots for what wasn't, so it reads like the heatmap.

Agent

Filled now means "in the fine-tuning data" (the original sentence and A to D); hollow means not (E to H, and the six other items). Both charts use the same rule and legend.

You

Publish the models.

Agent

Wrote a short model card with the charts and the results table, uploaded it, and made the Hugging Face repo public with all six checkpoints.

You

Let's write the DEV post. Use DevRelay actively, and lean on figures, tables, and videos rather than text.

Agent

Pulled the Week 2 rules and judging criteria with get_challenge_details, checked another submission for the template, searched DEV for related posts (one on whether fine-tuning turns a VLA into an imitation learner), and staged the post as an unpublished draft. Rewrote it around tables, charts, and videos. This session is saved with submit_agent_session and embedded in the post.

Note: I worked with the agent in Korean. This is a condensed English translation of the key moments, not a verbatim transcript.

Disclosure: I designed the experiment, ran the training and every test episode, and reviewed every result. An AI coding agent helped write the scripts, drafted phrasings E to H under rules I set (I reviewed and froze them before training), analyzed the results, made the charts and videos, and drafted this post.

Can I Ask a Robot to Pack My Picnic in My Own Words?

HYEONGSEOB JO — Sun, 11 Oct 2026 16:49:25 +0000

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 1: Touch Grass

What I Built

The plan was simple: before heading out for a picnic, you tell a robot what to pack, step away from the screen, and go. The robot does the rest.

I built that loop in simulation with an open vision-language-action (VLA) model, SmolVLA. It reads two camera images, the arm's state, and a sentence like "pick up the butter and place it in the basket", and outputs continuous arm actions. The picnic items are the ten groceries of the LIBERO-Object benchmark: alphabet soup, cream cheese, salad dressing, bbq sauce, ketchup, tomato sauce, butter, milk, chocolate pudding, and orange juice.

Then I tested the part that decides whether you actually get to leave: can you ask in your own words?

Demo

Each row is one item. Left to right: Original (the exact training sentence), then A, B, C, and D, four ways a person might say it instead (listed in the table under Step 2). Each tile shows the first successful episode out of ten, or episode 0 when all ten failed. For a closer look at the videos, see the full-resolution versions on GitHub.

Alphabet soup

Cream cheese

Butter

Orange juice

The training sentence worked every time. The paraphrases split in two: A and D, which keep "put ... in the basket", worked about half the time (22/40 and 26/40), while B and C almost never worked (2/40 each).

The results

Step 1: does it pack at all?

First, the baseline: all ten items with their original instructions, 10 episodes each from slightly different starting layouts.

89 of 100 episodes succeeded. BBQ sauce was the weakest (6/10); four items were perfect (10/10). That matches a public run of the same checkpoint in huggingface/lerobot#4614 (42/50, 84%), so the setup reproduces what others see.

Step 2: can a person ask in their own words?

I took the four items that scored 10/10, so any drop points at the wording and not the grasp, and gave each one four paraphrases:

	Instruction	Success
Original	pick up the {item} and place it in the basket	40/40
A	grab the {item} and put it in the basket	22/40
B	{item} in the basket	2/40
C	pack the {item} for our picnic	2/40
D	could you put the {item} in the picnic basket?	26/40

Across all four paraphrases: 52/160 (33%). Every failure ran to the 280-step limit without placing the item.

Three things stood out:

The two most natural picnic requests fail almost completely. "Butter in the basket" and "pack the butter for our picnic" are exactly what I would say on my way out the door. Across the four items, those two wordings worked 2 times out of 40 each.
Keeping "put ... in the basket" helps, but not reliably. A and D land around half.
The same sentence can work for one item and mostly fail for another. Butter was 10/10 on both A and D; cream cheese was 3/10 on both.

If only the exact template works, the person stays at the screen rephrasing instead of heading out. For a Touch Grass project, that is the result that matters.

Why do paraphrases fail?

These are hypotheses; the runs cannot separate them.

One fixed sentence per task. Every LIBERO-Object task uses one template with only the item name changed. The model may treat the sentence as a task ID rather than as meaning.
The language model was fine-tuned too. The checkpoint config has train_expert_only: False, so the SmolVLM2 language layers were trained on those same template sentences and may have drifted from general English.
Small model. The language part is about 252M parameters. A larger backbone might map "grab" or "pack" closer to "pick up". Running the same instructions on a bigger LIBERO checkpoint such as lerobot/pi05_libero_finetuned would test this.
Unseen words and shifted positions. B moves the item name to the front; C and D add words like "pack", "could you", and "picnic" that never appear in training.

How I Built It

Everything runs on one laptop CPU (Intel Core Ultra X7 358H, Windows 11, WSL Ubuntu 24.04). No GPU, no cloud.

Model: lerobot/smolvla_libero, SmolVLA fine-tuned on LIBERO.
Runner: LeRobot 0.6.1. The baseline uses lerobot-eval, wrapped to time every model call. The paraphrase runs use a small rollout loop that swaps in a new instruction while keeping the same initial states and seeds.
Simulation: LIBERO-Object on robosuite and MuJoCo.

Two flags matter (excerpt of the full command): one keeps the run from failing, the other keeps the success rate from dropping.

python baseline/benchmark_eval.py \
  --policy.path=lerobot/smolvla_libero \
  --policy.device=cpu \
  --policy.n_action_steps=10 \
  --env.type=libero \
  --env.task=libero_object \
  --rename_map='{"observation.images.image": "observation.images.camera1", "observation.images.image2": "observation.images.camera2"}'

--rename_map: the checkpoint expects cameras named camera1 and camera2, but LeRobot 0.6.1 does not apply the saved mapping automatically (huggingface/lerobot#4578).
--policy.n_action_steps=10: the checkpoint ships with 50, which replays a whole action chunk before looking again. In lerobot#4614 that cost 20 points on LIBERO-Object (84% to 64%).

The robot is fast in simulation and slow in real life

The simulator pauses while the model thinks. A real arm would not.

Each SmolVLA call plans 0.5 s of motion but takes 1.15 s on this CPU. On a real arm, the ketchup episode above would stop and go: 8.8 s of motion becomes about 29.5 s. Success rates are unaffected because the simulator waits, but real-time control would need a GPU or asynchronous inference. I did not test either.

What I did not do

No real robot and no outdoor test. I planned to pack a real picnic and dropped it. Everything here is simulation.
No training. The checkpoint is used as published.
One item per episode. Packing a whole basket in one go is outside the training distribution.

Code

johyeongseob / hacktoberfest-2026

Hacktoberfest 2026 projects and experiments focused on open-weight AI, VLA, and Physical AI.

Hacktoberfest 2026

A month-long exploration of open-weight AI, VLA, and Physical AI.

Goals

Explore open-weight AI models
Build VLA/Physical AI projects
Participate in DEV Challenges
Document experiments and learnings

Projects

dev-challenge/weekend-challenge/: Weekend Challenge project workspace
devrelay/: DevRelay usage notes and reviewed agent session transcripts
models/: Shared local model weights for all projects; excluded from Git

Development Environment

Component	Configuration
Host	Windows
CPU	Intel Core Ultra X7 358H; 16 logical CPUs visible in WSL
GPU	Intel Arc B390 GPU; Windows driver 32.0.101.8356
RAM	32 GB
Runtime	WSL with Ubuntu 24.04 LTS
Shell for project commands	Bash in Ubuntu
Python	3.12.3

View on GitHub

The project lives in dev-challenge/week-1, with run scripts and a results page for the baseline and for the paraphrase experiment, plus setup notes for WSL.

Why Does Open Innovation Matter?

This project would not exist with a closed model, because the interesting part was looking inside.

I could read the checkpoint. The train_expert_only: False line in the published config is what turned "it fails on paraphrases" into a testable hypothesis about why. A closed API would have given me a success rate and nothing else.
I could compare against strangers. Because the model, benchmark, and evaluation code are all public, I could line my 89/100 up against a public run on different hardware and know my setup was sound before trusting the paraphrase results.
It runs on my laptop for free. About 300 episodes on a CPU, offline after the first 3.3 GB download, with no API bill and no account.
The next step is open too. The fix most likely to help, fine-tuning on varied instructions or swapping in a larger open model, is something I can do myself rather than wait for a vendor.

The honest tradeoff: CPU inference is too slow to drive a real arm, and a small open model fine-tuned on one sentence per task does not understand casual English yet. But I can see exactly why, and that is what open gives me.

My Agent Session

I built this with Claude Code over four days. The session below is a curated slice: the decisions, the setup bug, the results, and the failure analysis.

Picnic-packing robot: testing SmolVLA with paraphrased instructions on a laptop CPU (curated, translated)

You

[Note: condensed English translation of a Claude Code session held in Korean, October 8-12, 2026. Messages are selected and shortened; wording is translated, not verbatim.]

Week 1 theme is Touch Grass: get people off the screen and outside. Let's do packing picnic items from the kitchen with a robot.

Agent

Good fit. A person tells the robot what to pack, then leaves. LIBERO-Object has ten grocery items (ketchup, butter, cream cheese, orange juice and more) and a basket, so it maps directly. The scene is a floor rather than a counter, so we frame it as a kitchen staging area and say so.

You

Rank SmolVLA, pi0, pi0.5, GR00T N1, and RLDX-1 by how hard they are to train and run. Is SmolVLA already trained on LIBERO-Object, or do we need to fine-tune?

Agent

SmolVLA is the lightest and runs on CPU. lerobot/smolvla_libero is already fine-tuned on LIBERO, including the ten LIBERO-Object tasks, so Week 1 can be inference only.

You

Week 1 uses lerobot/smolvla_libero as published. Give me step-by-step milestones.

Agent

Plan, smoke test (one ketchup episode), baseline (all ten items, 10 episodes each), paraphrased instructions, then the write-up. A dedicated venv and a separate LIBERO config folder keep other projects untouched.

Agent

The first smoke test stopped on an input name mismatch. The checkpoint expects cameras named camera1 and camera2, while the LIBERO environment emits image and image2.

▸ WebFetch lerobot/smolvla_libero train_config.json

Input

lerobot/smolvla_libero train_config.json

Output

rename_map: image to camera1, image2 to camera2. n_action_steps: 50

Fix: pass the rename_map flag explicitly, because LeRobot 0.6.1 does not apply the saved mapping (lerobot#4578). I also found lerobot#4614: the shipped n_action_steps of 50 lowers success, so we set n_action_steps to 10.

You

Should I open an issue upstream for this?

Agent

No need, both problems are already reported (lerobot#4517, #4578, #4614). We document the workarounds in setup.md.

You

From now on, always report inference time when we run experiments.

Agent

Added benchmark_eval.py, a thin wrapper around lerobot-eval that times every SmolVLA call.

▸ Bash benchmark_eval.py on libero_object task 4 (ketchup), 1 episode, CPU, n_action...

Input

$ benchmark_eval.py on libero_object task 4 (ketchup), 1 episode, CPU, n_action_steps 10

Output

ketchup: 1/1 success, 176 of 280 steps. 18 SmolVLA calls, 1.15 s mean per call

You

I watched the video: it puts the ketchup in the basket. What does 1.15 s per call mean for a real robot?

Agent

Each call plans 10 actions, which is 0.5 s of motion at 20 Hz, but takes 1.15 s on this CPU, about 2.3x slower than real time. The simulator pauses while the model thinks, so success rates are unaffected, but a real arm would stop and go.

You

Do we need a GPU for Week 1?

Agent

No. In simulation success rates do not depend on speed, so a laptop CPU is enough. The slowness goes into the write-up as a limitation.

You

Then no GPU in Week 1. Remove the GPU setup.

Agent

Baseline finished: 89/100 across the ten items (bbq sauce 6/10 weakest; alphabet soup, cream cheese, butter, orange juice 10/10). That is in line with the public 42/50 (84%) in lerobot#4614, so the setup reproduces.

You

What is the point of the paraphrase experiment? Checking whether SmolVLA still works when I use my own instruction instead of the trained one?

Agent

Exactly: can a person ask in their own words? Use items that scored 10/10 so a drop points at the wording, not the grasp. Per item: the original sentence as a control plus four paraphrases, 10 episodes each with the same seeds. I wrote picnic_rollout.py, a small rollout loop that swaps in the instruction.

You

Start with two items, and add one or two more if there is time.

You

My laptop battery is at 8% and I am outside. How far did it get? Is it OK to cancel now?

Agent

Each episode is written as it finishes, so stopping keeps completed runs. Restart later from the next run boundary with the same arguments and every cell still ends with exactly 10 episodes.

Agent

All four items done (alphabet soup, cream cheese, butter, orange juice). Original 40/40. Paraphrases 52/160: A grab/put 22/40, B '{item} in the basket' 2/40, C 'pack the {item} for our picnic' 2/40, D 'could you put the {item} in the picnic basket?' 26/40.

You

Why does SmolVLA do so badly on B and C? Because they are abstract or have no verb? Is it like CLIP having a caption template?

Agent

Similar idea. Every LIBERO-Object task uses one fixed sentence, so the model may treat it as a task ID. The checkpoint config also has train_expert_only False, so the language layers were fine-tuned on those same sentences.

You

Could the small model size also be a reason? With more parameters the transformer might understand better.

Agent

Plausible. The language part is about 252M parameters using 16 of SmolVLM2's 32 layers. Running the same instructions on a larger LIBERO checkpoint such as pi0.5 would test it. Four hypotheses total; the runs cannot separate them, so we label them as hypotheses.

You

For each item, make a 1x5 grid: original, A, B, C, D, first successful episode each. And drop the real-picnic milestone: I have no robot.

Agent

▸ Bash make_grid.py m3-row for task 6 (butter)

Input

$ make_grid.py m3-row for task 6 (butter)

Output

m3_row_task6.mp4 written

Done for all four items. The write-up will say clearly that everything is simulation and there was no outdoor test.

You

Review the whole README based on what we checked in the baseline and paraphrase results.

Agent

Found claims that no longer matched the results pages, an over-broad '20 points on LIBERO' that should say LIBERO-Object, and missing links to the run scripts. Fixed them and restructured the README around the paraphrase videos.

You

Let's move on to the DEV post. Use DevRelay actively.

Agent

Pulled the Week 1 rules and judging criteria with get_challenges, checked another submission for the template, and staged the post as an unpublished draft with create_article. This session is saved with submit_agent_session and embedded in the post.

Note: I worked with the agent in Korean. This is a condensed English translation of the key moments, not a verbatim transcript.

Disclosure: I designed the experiments, ran every episode, and reviewed every result. An AI coding agent helped write scripts, draft the paraphrased instructions (which I reviewed), and draft this post.

Where Does This Belong? An AI Workbench Reset Assistant for a Coworker

HYEONGSEOB JO — Sun, 04 Oct 2026 20:05:34 +0000

This is a submission for the Hacktoberfest Weekend Challenge: Build for a Friend

Figure: The demo interface before analysis. Left: the frame used as the reference layout. Right: the frame used as the current layout.

What I Built

I built Shared Workbench Reset Assistant for one coworker who has trouble remembering where objects originally belonged on our shared workbench.

The idea is simple:

Keep a reference photo of the intended layout,
Compare it with a photo taken after use, and
Prepare a short checklist for putting things back.

The reference photo provides a visual memory of the workspace.

The prototype uses an open-weight vision-language model to draft observations and a cleanup checklist. Both remain editable, and the user must review them against the photos before exporting a result.

Demo

The three demo screenshots show

the photo inputs,
reviewed observations, and
final reviewed checklist.

For a reproducible demonstration, I used public Tabletop Tidying Up (TTU) dataset images.

The sample shown below contains a red cup, a beverage can, and a remote control. I used the first frame as the reference layout and the last frame as the current layout. These roles were assigned for this prototype; they are not verified official tidy/untidy labels.

The workflow is:

Analyze the two photos independently. (Screenshot 1)
Correct and confirm the model's observations. (Screenshot 2)
Generate a checklist from those reviewed observations. (Screenshot 3)
Edit and confirm the final checklist, then download the result as JSON. (Screenshot 3)

Figure 1: Left: the frame used as the reference layout. Right: the frame used as the current layout.

Figure 2: Observations corrected with coding-assistant help and checked against the photos.

Figure 3: Final checklist corrected with coding-assistant help and confirmed against the reference photo.

Run the Local Demo

Follow the project README to prepare the Python environment, download the Q8_0 model and vision projector, and build llama-server.

Open two WSL terminals. Replace /path/to/hacktoberfest-2026 with the path to your local repository. The commands below use the virtual environment path specified in the README.

In terminal 1, start the model server:

cd /path/to/hacktoberfest-2026/dev-challenge/weekend-challenge
bash start_model_server.sh

After it reports that it is listening on port 8080, start the app in terminal 2:

source ~/.venvs/hacktoberfest-2026/bin/activate
cd /path/to/hacktoberfest-2026/dev-challenge/weekend-challenge
python -m streamlit run app.py \
  --server.address 127.0.0.1 \
  --server.port 8501 \
  --browser.gatherUsageStats false

Replace /path/to/hacktoberfest-2026 with the path to your local repository.

Open http://localhost:8501 in your Windows browser and select the sample. Review and correct both observations and the checklist against the photos before exporting. This address works on the machine running the app; it is not a public demo link. No hosted inference API key is needed.

How I Built It

Code

Project repository · Weekend project and setup

The main files are app.py for the Streamlit interface, workbench.py for local inference requests and image preparation, and start_model_server.sh for starting llama.cpp. Model weights are downloaded separately and excluded from Git.

Model and Runtime

Model: SmolVLM2 2.2B Instruct
Quantization: Q8_0 for both the model and vision projector
Inference server: Local llama.cpp server
Interface: Streamlit
Environment: WSL Ubuntu 24.04
Hardware: Intel Core Ultra X7 358H laptop with 32 GB of RAM
Execution: CPU only; no GPU or model fine-tuning

Why Does Open Innovation Matter?

Open weights and open-source inference tools let me run the model on my own laptop without sending photos to a hosted model API. They also let me compare models, inspect failures, and adapt the workflow without relying on a hosted model provider.

What’s Next?

This project is the starting point for my month-long exploration of VLA and Physical AI. It uses a VLM to draft text observations and instructions; it does not execute a VLA policy or control a robot.

AI Assistance Disclosure

I used Codex to help implement the app, debug experiments, revise the demo observations and cleanup instructions, and draft this post. I checked the final observations and checklist against the photos. The original local SmolVLM2 outputs are preserved separately from the Codex-assisted revisions.

Experiment results and the reviewed JSON export are stored locally and excluded from Git. The screenshots show the workflow after human review and correction. The prototype has not yet been tested by my coworker and needs broader evaluation.

Data Attribution

The sample images are third-party demo data from TTU, created by Hogun Kee, Wooseok Oh, and Songhwai Oh (2024). See the project README for the source, preserved upstream license notice, citation, and licensing caveat.

References

Hacktoberfest Weekend Challenge: Build for a Friend — challenge prompt and submission instructions.
Tabletop Tidying Up (TTU) Dataset — source of the tabletop demo images; see the project README for attribution and licensing details.
SmolVLM2 2.2B Instruct GGUF — model and vision projector used for local Q8 inference.
llama.cpp — local multimodal inference runtime and server.
Streamlit — framework used for the browser interface.
DevRelay — tool used to update the DEV draft and publish the submission; no agent-session transcript was uploaded for this project.
Intel Physical AI Studio — context for my separate open-source contributions and longer-term Physical AI goals; not integrated into this prototype.