Asking a language model for synthetic training data usually means one prompt: "Generate N rows as a JSON array", a schema, a few example rows. With a 3B model running on a laptop, that works at 20 rows and starts to fall apart at 40, in a very specific way: the model starts copying its own earlier rows and never stops. The numbers below come from running that kind of prompt many times on Qwen2.5-3B-Instruct (4-bit, through mlx-lm) on an 8 GB Apple M1, measuring when the loop starts, why it is so hard for the model to leave it, and which fixes hold up.
The prompt and the setup
The prompt is the one the dataset generator in the osFoundry desktop app builds (buildPrompt in its PromptEditor.tsx), for a text-classification dataset of online-shop messages with three allowed labels and three example rows:
SYSTEM = "You are a data generation assistant. Output only valid JSON arrays."
def build_prompt(n):
return f"""Generate {n} data samples as a JSON array.
Domain: Online shop customer messages
Language: English
Each object must have these fields:
"text" (text) — Input text
"label" (enum) — Classification label [allowed values: refund, shipping, spam]
Here are example rows to follow:
{json.dumps(FEW_SHOT, indent=2)}
Generate {n} more rows following the same style and patterns.
Output ONLY a valid JSON array, no other text."""
Generation used mlx_lm.stream_generate with temperature 0.8, top-p 0.95 and max_tokens=8192, each run with its own fixed seed. A row costs about 27 tokens, so 8,192 tokens is room for roughly 300 rows; running out of tokens is not the problem at 95.
What 95 rows in one request looks like
The first 95-row reply, recorded while making a video about this feature, is a good specimen. It contains 302 complete rows (the 303rd is cut off), only 24 of them different. From row 27 on, the model writes nothing new: it cycles through three sentences until the token limit, and the closing ] never arrives.
"my order is late, please refund my money"
"please refund my money as the product is defective"
"i need to cancel my order due to a change of mind"
"my order is late, please refund my money"
...
The Japanese version of the same prompt was worse: 272 complete rows, 6 different, and rows 7 to 272 are all the same sentence. A client that looks for a complete [...] and runs JSON.parse on it gets zero rows from either reply.
Where it breaks: a batch-size sweep
To see where this starts, the English prompt was run three times each at 10, 20, 40, 60 and 95 rows. To save time, a run was stopped as a loop when the last 12 rows contained 4 or fewer different texts.
| Rows requested | Runs that closed the array | Different rows in each run | Stopped as a loop |
|---|---|---|---|
| 10 | 3 of 3 | 10, 10, 10 of 10 | 0 |
| 20 | 3 of 3 | 17, 18, 20 of 20 | 0 |
| 40 | 2 of 3 | 40 of 44, 28 of 41 | 1 (at row 37) |
| 60 | 2 of 3 | 15 of 61, 41 of 51 | 1 (at row 22) |
| 95 | 1 of 3 | 15 of 81 | 1 (at row 39) |
The third 95-row run did not trip the 12-row rule because its cycle was longer than four sentences; it ran to the token limit (342 seconds) with 304 rows and 16 different ones.
Two things in this table are easy to miss. A closed array is not a good reply: the 60-row run that closed with 61 rows had only 15 different texts, and the 95-row run that closed had 81 rows and 15 texts. And the model does not count. Asked for 40, it returned 44 and 41; asked for 95 it stopped at 81. The number in the prompt is a hint, not a contract.
Japanese was less stable at every size. At 20 rows, two of three runs were stopped as loops (13 rows with 5 different, 15 rows with 6); at 95 rows all three were stopped before row 24.
Why it can't get out: the probabilities
Sampling at temperature 0.8 should, in principle, wander off a repeated sentence sooner or later. It doesn't, and the reason shows up when the recorded 95-row reply is fed back through the model (teacher forcing) and the probability the model gave each token it actually wrote is read off. Averaged over the tokens inside each "text" value:
| Rows in the 95-row reply | Rows | Mean token probability |
|---|---|---|
| First time a sentence appears | 24 | 0.555 |
| Repeats of an earlier sentence | 279 | 0.996 |
| Rows 41 to 303 | 263 | 1.000 (rounded) |
For comparison, a healthy 20-row reply with no repeats averages 0.493. Writing a new sentence is a sequence of fairly uncertain choices. Once a sentence is already in the context twice, the model predicts every token of the third copy almost with certainty, and a temperature of 0.8 changes almost nothing when the top token has 99.9% of the mass. Every copy makes the next one more likely, which is why the loop gets tighter instead of breaking up. A token budget of 8,192 doesn't help either: it only decides how long the loop runs. (The temperature glossary entry covers what temperature can and can't do here.)
Catching the loop while it streams
Since a looping reply is useless anyway, the cheapest fix is to stop generating as soon as it starts. Parsing the complete objects out of the partial reply is easy with json.JSONDecoder.raw_decode, which also salvages rows from a reply that was cut off:
def objects(text):
"""Complete {...} objects inside the first [, even if the array never closes."""
dec, i, rows = json.JSONDecoder(), text.find("["), []
if i < 0:
return rows
i += 1
while True:
while i < len(text) and text[i] in " \t\r\n,":
i += 1
if i >= len(text) or text[i] != "{":
return rows
try:
obj, i = dec.raw_decode(text, i)
except ValueError:
return rows # the last object is still being written
rows.append(obj)
A simpler stop rule than the one used in the sweep turned out to be more reliable: stop when the last 10 rows contained nothing new.
def stalled(rows, k=10):
seen, since_new = set(), 0
for r in rows:
t = str(r.get("text", "")).strip().lower()
since_new = since_new + 1 if t in seen else 0
seen.add(t)
return since_new >= k
text = ""
for chunk in stream_generate(model, tok, prompt, max_tokens=8192, sampler=sampler):
text += chunk.text
if "}" in chunk.text and stalled(objects(text)):
break
Replayed offline over the full recorded replies, this rule fires at row 36 on the 302-row English reply, at row 29 on the 304-row one, and at row 16 on the 272-row Japanese one, instead of letting each run to 8,192 tokens. Across all 47 replies collected for this post, it fired on every one in which fewer than a third of the rows were different (seven replies, including two 20-row Japanese batches), once on a reply with 30 different rows out of 51, and never on the other 39.
Repetition penalty: the window matters more than the strength
mlx-lm has a repetition penalty (make_logits_processors(repetition_penalty=..., repetition_context_size=...)), and the default window is 20 tokens. A row is about 27 tokens, so with the default window the penalty never sees the previous copy of a sentence. Two runs each of the 95-row prompt, penalty 1.1:
| Window | Rows returned | Different rows | Array closed |
|---|---|---|---|
| 20 tokens (default) | 49, 51 | 33, 30 | 2 of 2 |
| 200 tokens | 47, 53 | 47, 53 | 2 of 2 |
With a 200-token window, every row was different. The model still stopped at around half of the 95 rows it was asked for, so a penalty is not a way to get big batches in one call.
It also has a cost that shows up in JSON. The penalty applies to every token in the window, including ", : and the keys text and label, which repeat in every row by design. In one of five Japanese 20-row batches with the 200-token window, the model ended two strings with ...' instead of ..." and the reply stopped being valid JSON. objects() still recovered the 12 rows before the first broken one.
What worked: batches of 20, then clean up
Five batches of 20 per setup, each batch a separate call with the same prompt:
| Setup | Rows | Different rows | Labels |
|---|---|---|---|
| English, no penalty | 100 | 95 | 65 refund, 14 shipping, 21 spam |
| English, penalty 1.1, window 200 | 101 | 100 | 37 refund, 32 shipping, 32 spam |
| Japanese, no penalty | 99 | 64 | 67 shipping, 17 refund, 15 spam |
| Japanese, penalty 1.1, window 200 | 93 | 93 | 50 shipping, 14 refund, 29 spam |
Each setup took about two minutes for five batches on the M1 (113 to 126 seconds). Batches alone fix the "never finishes" problem; duplicates across and within batches remain, especially in Japanese, and the label balance can be lopsided either way. With one run per setup, the more even English labels under the penalty may be luck; the duplicate counts are consistent across all the runs above.
Whatever the generation settings, three checks after the batches are cheap and catch real problems:
def clean(rows, allowed):
out, seen = [], set()
for r in rows:
r = {k.strip(): v for k, v in r.items()} # "text " happens
t = str(r.get("text", "")).strip()
if not t or r.get("label") not in allowed: # labels outside the list
continue
key = " ".join(t.lower().split())
if key in seen: # exact duplicates
continue
seen.add(key)
out.append({"text": t, "label": r["label"]})
return out
print(collections.Counter(r["label"] for r in clean(rows, ALLOWED)))
None of the runs above returned a label outside the list, but the earlier recorded Japanese batches did: 15 of 99 rows came back labelled 配送料変更 ("shipping fee change"), which was never an allowed value. One of the recorded replies also used the key "text " with a trailing space. Both are the kind of thing that only shows up when the rows are checked by code instead of by eye.
Doing it in the app
The osFoundry desktop app's dataset generator (Models, Datasets, Generate) uses the prompt above, and its Generate More (append) button sends one batch per click, so keeping Samples at 20 and clicking until the table is big enough gives the batch behaviour measured here. Saved datasets can then be used for the training jobs described on the custom model training page.
Top comments (0)