This is a submission for the Hacktoberfest Open-Source AI Challenge Week 3: Slow and Steady
What I Built
In Week 2, fine-tuning worked: one demo per item, relabeled with the ways people actually ask, taught my open robot model (SmolVLA) new phrasings (31% to 80%) for the price of a coffee. It still packed one item at a time.
So this week I pushed the same idea toward the laziest picnic request of all: say one sentence, "grab the cream cheese and the salad dressing", and walk away while the robot puts both in the basket, in that order. If fine-tuning could teach it how I talk, could it teach it a two-item task?
I built the pieces around the model to make that a single request:
| Piece | What it does |
|---|---|
| Request splitter | "grab the cream cheese and the salad dressing" becomes two steps, in the order named |
| Step checker | The simulator reports when an item is in the basket, and the instruction moves on to the next item |
| SmolVLA | Does one step at a time from two camera images and the arm state |
| Scripted demo generator | Writes training demos from the simulator's object positions, no teleoperation |
It did not go to plan, and the reasons turned out to be the interesting part.
Demo
"Pick up the cream cheese ..." then, once it is in, "pick up the salad dressing ...". Neither item is the one this scene was trained on (bbq sauce). The cream cheese lands at step 95, the dressing at step 371 (20 steps per second).
The honest number: this is 1 success in 80 two-item episodes. The rest of this post is why.
Finding 1: the robot memorized the table, not the sentence
Every LIBERO-Object scene has six groceries, but the model only ever learned one task per scene: "pick up {that scene's item} and place it in the basket". It scores 96% on that benchmark. So I asked for each of the six items in each of the ten scenes.
| Published SmolVLA | Success |
|---|---|
| The scene's own item | 10/10 |
| Any other item on the same table | 1/50 |
When it failed on another item, it lifted the scene's own item instead in 24 of 26 episodes. It was not reading the item name; it was recognizing the table. I am not the first to see this: VLA or IL? by @fn8211 shows the same with π0.5 and summarizes LIBERO-PRO and LIBERO-Plus, which found that fine-tuned models keep the same trajectory when the instruction changes. This is the same failure in a 450M model, measured on every item of every scene.
A single sentence for two items ("put both the cream cheese and the butter in the basket") failed 0/3: the cream cheese, that scene's item, went in every time; the butter never did.
This also explains Week 2. There, every new phrasing still named the scene's own item, so the table alone gave the right answer. Two items in one scene is the first time the item name has to matter.
Finding 2: breaking the shortcut is not the same as reading
LIBERO-Object has no demos of "pick the other item", so I generated them. A short script reads the object positions from the simulator, moves the gripper over the asked item, grasps it, and places it in the basket. The training set had single picks of all six items in every scene (8 per item), chains of two to four items, and a few original demos; no teleoperation.
| Published | Fine-tuned on the scripted demos | |
|---|---|---|
| Scene's own item | 10/10 | 3 to 5 of 10 (3k, 6k, 9k steps) |
| Other items | 1/50 | 3 to 5 of 50 |
| Wrong lifts that were the scene's own item | 24 of 26 | 12 of 40 (3k), 16 of 41 (6k) |
The fine-tune broke the habit: wrong picks mostly stopped going to the scene's own item. But they did not go to the asked item either. I checked whether a wrong pick at least shared the asked item's kind (box, carton, can, bottle): 6 of 40 and 10 of 41, against 7.6 expected by chance. When it is wrong, the item name plays no role.
Removing a shortcut did not create the skill it was standing in for. The model stopped recognizing the table, and nothing took its place.
Two items, in order
| Model (same 80 episodes: 40 trained item orders, 2 table layouts) | Both items, in order | First item |
|---|---|---|
| Fine-tuned on the first scripted chains (6k) | 3/80 | |
| Fine-tuned on the scripted demos above (9k) | 1/80 | 15/80 |
With picking the right item at about 10%, two in a row is about 1%. The step checker and splitter work; the bottleneck is grounding one item at a time.
The lazy payoff
| Effort | |
|---|---|
| What I wanted | One sentence, then walk away |
| What works today | One sentence, then watch: the robot gets the order right about 1 time in 80 |
| What the pipeline saves already | No teleoperation: 657 training demos written by a script in under an hour on the rented machine's CPUs |
The data side is lazy in the good way. The robot side is not there yet, and I would rather show that than a cherry-picked clip.
How I Built It
| Part | Choice |
|---|---|
| Model |
lerobot/smolvla_libero, 450M parameters, LeRobot 0.6.1 |
| Simulation | LIBERO-Object on robosuite and MuJoCo |
| Demos | Scripted pick-and-place from simulator object poses, human pace, failed attempts discarded |
| Fine-tuning | Whole model (vision encoder frozen), batch 32, lr 2.5e-5, 9,000 steps, about 75 minutes |
| Evaluation | Every item in every scene (60 episodes), two-item chains (80 episodes), failure kinds from the lift height of every object |
| Hardware | 1x A100 80 GB on RunPod |
The request splitter is the smallest part:
def split_request(request: str) -> list[str]:
text = request.lower()
names = {item: item for item in ITEMS} | SHORT_NAMES
found = [] # (start, end, item)
# Longest names first, so "cream cheese" is taken before "cheese".
for name in sorted(names, key=len, reverse=True):
for match in re.finditer(rf"\b{re.escape(name)}\b", text):
if not any(match.start() < end and start < match.end() for start, end, _ in found):
found.append((match.start(), match.end(), names[name]))
items = []
for _, _, item in sorted(found):
if item not in items:
items.append(item)
return items
split_request("grab the cream cheese and the salad dressing") returns ["cream cheese", "salad dressing"]; each item becomes "pick up the {item} and place it in the basket", the sentence the model was trained on.
Limits: all numbers are from one training layout per scene (state 46) or two (46 and 47); simulation only; one training run per setting.
Code
johyeongseob
/
hacktoberfest-2026
Hacktoberfest 2026 projects and experiments focused on open-weight AI, VLA, and Physical AI.
Hacktoberfest 2026
A month-long exploration of open-weight AI, VLA, and Physical AI.
Goals
- Explore open-weight AI models
- Build VLA/Physical AI projects
- Participate in DEV Challenges
- Document experiments and learnings
Projects
hacktoberfest-2026/
├── dev-challenge/ # Hacktoberfest DEV Challenge entries
│ ├── weekend-challenge/ # Shared Workbench Reset Assistant (local open-weight VLM)
│ ├── week-1/ # Picnic Packing with SmolVLA (paraphrased instructions in LIBERO)
│ └── week-2/ # Teaching the Picnic Robot My Words (SmolVLA fine-tune on a rented GPU)
├── global-hack-week/ # Global Hack Week: Hacktoberfest challenge write-ups
│ ├── 01-registration.md # Registration challenges
│ └── github/ # GitHub challenges (Codespaces, repo management, Copilot)
├── devrelay/ # DevRelay usage notes and reviewed agent session transcripts
└── models/ # Shared local model weights; excluded from Git
DEV Challenge entries
Folder
Theme
Project
Post
weekend-challenge
Build for a Friend: solve a real problem for someone you know
Shared Workbench Reset Assistant: a local open-weight
| Where | What |
|---|---|
week-3/datagen |
Scripted demo generator and recorder |
week-3/eval |
Step-by-step rollout, request splitter, every-item-in-every-scene test |
week-3/train |
Training-set builder, normalization fix, training script |
Why Does Open Innovation Matter?
| Open piece | What it made possible |
|---|---|
| Open weights | I could fine-tune the model and inspect every checkpoint |
| Open simulator and benchmark | The simulator knows where every object is, so a script could write demos no one recorded |
| Open data | The original LIBERO demos are there to mix back in against forgetting |
| Open training code | Reading LeRobot's training loop let me check what fine-tuning changes, down to the normalization statistics |
| Open discussion | Others had already measured the same shortcut, so I knew what I was looking at |
A closed model would have given me a success rate. Open pieces let me find out why the robot fails.
What's Next
Grounding is the missing skill, and a detector already has it. Next I want to mark the asked item on the camera image (first with the simulator's own boxes, then with an open-vocabulary detector) and see whether SmolVLA learns to follow the mark.
Notes from the Bench
Things I tried along the way that did not make the main story.
1. My scripted demos had hidden pauses. The script moved the gripper in proportion to the remaining distance, and some targets were out of reach: the fingers stopped against a tall bottle or the basket floor, and the script kept nudging until its step limit. The average pace still matched the human demos (about 146 steps per item), so success rates and episode lengths looked fine.
A "steady" version ends a move once the gripper stops getting closer. After retraining on it, "picked up nothing" dropped from 34 to 18 of 60 at 3,000 steps (Fisher's exact test, p ≈ 0.006; the published model: 22). The steady set also had more single picks (about 450 vs 168), so this is the likely cause, not an isolated one. @ankk98 saw the same in teleop data: idle footage teaches a policy to linger. Slow and steady, it turns out, has to be steady.
My Agent Session
I built this with Claude Code over a week. The session below is a curated slice: the diagnosis, the scripted demos, the hidden pauses, the grounding check, and the one two-item success.
[Note: condensed English translation of a Claude Code session held in Korean, October 19-26, 2026. Messages are selected and shortened; wording is translated, not verbatim.]
Week 3 is Slow and Steady. The laziest picnic request: one sentence, two items, walk away. Can SmolVLA put two LIBERO-Object items in the basket, one after the other? I want the hardest version I can actually finish.
First, what the published model does today. Two tests in one LIBERO-Object scene: one sentence for both items (LIBERO-10 style), and the item sentences one at a time, switching when the simulator says the first item is in the basket.
Is it the butter, or is it the order?
Ask for every item in every scene, one at a time. If the model reads the item name, other items should work about as well as the scene's own.
And when it fails on another item, it lifts the scene's own item in 24 of 26 episodes. It recognizes the table, not the sentence. LIBERO-Object pairs each scene with exactly one item, so the table alone always gave the right answer.
There are no demos of picking the other item. Can we make them without teleoperation?
The simulator knows every object's pose, so a short script can move the gripper over the asked item, grasp it, and place it in the basket. Fixes along the way: aim at the object center (small boxes were missed from 1 cm above), hold the gripper orientation, special settings for tall bottles, and separate spots in the basket so a second item does not land on the first.
Record chains of two to four items and fine-tune. Plan the item orders so some are only ever used for testing.
The first fine-tune was worse than the published model. compare_demos.py showed why: the scripted demos took about 300 steps per item, twice the human demos' 146. A faster pace matched the human one, and after retraining: 3 of 80 two-item episodes in order. In the 10 episodes where the second item went in, the model had put the order's second item in first. The bottleneck is choosing the item, not chaining.
Then teach it to choose. Same table, every item.
Added single picks of all six items in every scene, so the table alone no longer gives the answer. Result: other items rose a little (1/50 -> 4-8/50), but the scene's own item fell from 10/10 to 0/10 at the first checkpoint.
Why would it get worse at what it already knew?
First suspect: lerobot-train replaces the checkpoint's normalization statistics with the new dataset's. The scripted data had an action mean of -0.34 on z instead of -0.09 and a 20x narrower orientation spread.
Why does the second video stop in the middle?
Measured frame-to-frame change per second: the arm sits nearly still for about five seconds over a tall item and again over the basket. The script moves in proportion to the remaining distance, and some targets are out of reach (fingers against a bottle or the basket floor), so it nudged until its 120-step limit. The average pace still matched the human demos, so success rates and lengths never showed it.
Retrain on the steady demos.
Same normalization, steady data. 'Picked up nothing' dropped from 34 to 18 of 60 at 3,000 steps (Fisher p = 0.006), back to the published model's level (22). Caveat: this run also had more single picks, so it is the likely cause, not an isolated one.
When it picks the wrong item, which item does it pick?
Every rollout now records how high every object on the table was lifted. Wrong lifts that were the scene's own item: 24 of 26 before fine-tuning, 12 of 40 after. The shortcut is broken. But does a wrong pick at least share the asked item's kind (box, carton, can, bottle)? 6 of 40 and 10 of 41, against 7.6 by chance. When it is wrong, the name plays no role.
Breaking the shortcut did not create grounding.
Today's minimum goal: one real two-item success.
Neither item is the scene's own (bbq sauce). One honest success, with the instruction switching exactly when the first item landed.
Would mixing the original LIBERO demos back in stop the forgetting?
Co-training: all 457 original LIBERO-Object demos, the steady single picks, and two-item chains only.
No gain within the noise. More of the old data did not bring back the old skill, and it did not add grounding.
The pipeline sat idle for two hours. What happened?
My bug: the wait loop used pgrep -f to wait until no data-preparation process was running, and its own command line contained that name, so it waited for itself. I had made the same mistake once before with the training process. Restarted the steps directly without the wait.
What is the contribution this week?
Main story: the model memorized the table (1/50, 24 of 26 wrong lifts to the scene's own item), and breaking that shortcut did not make it read (wrong picks at chance). The two-item pipeline works, and the one success is real. The hidden pauses, the normalization check, and co-training go into a notes section at the end.
Write the post with DevRelay. Start from Week 2: fine-tuning worked there, so try a two-item task here.
Checked the Week 3 rules with get_challenges, searched DEV for related posts (a controlled LIBERO-Object dataset showing the same shortcut with pi0.5, and a teleop data-quality rubric that flags idle footage), staged an unpublished draft with create_article, and kept the Markdown source in the repo. This session is saved with submit_agent_session and embedded at the end of the post.
Note: I worked with the agent in Korean. This is a condensed English translation of the key moments, not a verbatim transcript.
Disclosure: I set the goal, ran every training and evaluation command, and reviewed every result. An AI coding agent (Claude Code) wrote the scripts with me, analyzed the results, made the charts, and drafted this post.
Top comments (0)