Any fine-tuning project for invoice extraction starts with one question: how many of our invoices do you need before this works? For this task, the measured answer is about 100. Those 100 labelled invoices took the model to 98% on known layouts. The next 900 added 1.6 points, and none of them helped on layouts the model had never seen.
A guess is not an answer, so I measured the curve: the same model, the same settings, six training-set sizes, three seeds each.
This post follows "Prompting vs Fine-Tuning: Invoice Extraction at Scale" (link to add). That post trained one LoRA adapter on 240 invoices and found it beat prompting (86% vs 84%) at about a third of the token cost. It listed "no sweep over training-set size" as a limitation. This post closes that gap.
The setup in a few lines
Every run uses the same task, model and settings; only the number of training invoices (and the seed) changes.
- Task: pull the invoice number, date, vendor, line items and total out of a text invoice (German, UK English or French) and return JSON.
- Model: Qwen2.5-1.5B-Instruct, run locally on an RTX 5070 Ti (16 GB).
- Method: LoRA on the attention layers (rank 16, alpha 32), 3 epochs, learning rate 2e-4, batch size 4.
- Data: synthetic invoices generated from code, so the correct answer is exact by construction. Training invoices are drawn evenly across 8 layouts and 3 languages.
- Test sets, frozen and never trained on: 192 invoices in the 8 layouts the model trains on ("seen") and 45 invoices in 3 layouts it never trains on ("unseen").
- Sizes and seeds: 25, 50, 100, 250, 500 and 1,000 training invoices, each trained 3 times. A seed changes which invoices are picked and the adapter's starting values. For one seed, the smaller sets sit inside the larger ones.
- Baseline: no training, just the best prompt from the first post (instructions plus one worked example).
- Metric: invoice accuracy, the share of invoices with every field correct. One wrong field means a person has to check the invoice, so this is the number a business cares about.
The result: accuracy against training invoices
On known layouts, accuracy climbs from 81% at 25 invoices to 98% at 100 and 100% at 1,000; on unseen layouts it levels off at 77-81%, no better than the prompt alone.
| Training invoices | Seen layouts | Unseen layouts | Training time |
|---|---|---|---|
| 0 (prompt + 1 example) | 84.9% | 82.2% | none |
| 25 | 80.9% (77.1-85.4) | 45.2% (42.2-48.9) | ~7 s |
| 50 | 93.1% (92.7-93.8) | 74.8% (73.3-77.8) | ~9 s |
| 100 | 98.4% (97.9-99.0) | 80.7% (80.0-82.2) | ~20 s |
| 250 | 99.7% (99.0-100) | 80.7% (77.8-86.7) | ~1 min |
| 500 | 99.8% (99.5-100) | 77.0% (73.3-80.0) | ~2 min |
| 1,000 | 100% (100-100) | 80.0% (77.8-82.2) | ~3.2 min |
Invoice accuracy, mean of 3 seeds with the range in brackets. Every adapter is 16.7 MB, and peak GPU memory was about 12 GB in every run.
Where the curve flattens, and what it means for a budget
The curve flattens at about 100 invoices: going from 100 to 1,000 adds 1.6 points on known layouts (98.4% to 100%) for ten times the data.
The steps before that are large. From 25 to 50 invoices the gain is 12.2 points, and from 50 to 100 it is another 5.3. Training starts to beat the prompt at about 50 invoices on known layouts (93.1% vs 84.9%).
At 98.4%, roughly 3 of the 192 test invoices needed a person to look at them. That is the price of stopping at 100.
The GPU is not the cost. The longest run, 1,000 invoices, trained in 3.2 minutes. For a client, the real cost is collecting and labelling invoices, so the useful question is whether the extra 1.6 points are worth labelling 900 more.
The surprise: 25 invoices are worse than none
On average, a model trained on 25 invoices scores below the same model with a good prompt: 80.9% vs 84.9% on known layouts.
On known layouts the picture is mixed. Seed 1 reached 85.4%, one invoice above the baseline, while the other two seeds scored 77.1% and 80.2%. On unseen layouts it is clear: all three seeds land at 42-49%, against 82.2% for the prompt.
25 invoices is about one per layout-and-language group (there are 24). I did not test why that is too little; the sweep only shows that it is. If you have a handful of labelled invoices, put them in the prompt as examples instead of training on them.
Where it did not work: layouts the model never saw
More training data did not fix new layouts. From 100 to 1,000 invoices, unseen accuracy stays at 77-81% on average, and no size beats the prompt's 82.2%.
Single runs do reach it: two runs tie at 82.2%, and one seed at 250 invoices hit 86.7%. But the unseen set has only 45 invoices, so one invoice is about 2.2 points, and these differences are within noise. Meanwhile, accuracy on seen layouts rises to 100%. Training teaches the layouts it sees.
This matches the finding of the first post. If a client's invoices come in layouts that are not in the training data, a prompt with an example is at least as good, and fine-tuning should not be sold as the fix.
What to do with 20, 100 or 1,000 client invoices
The size of the labelled set decides the approach.
- About 20 invoices: use them as examples in the prompt, not for training. The sweep tested a prompt with one worked example; I did not test more.
- About 100 invoices: train. That was enough for 98.4% on known layouts. Cover every layout and language you expect, because this test drew the training invoices evenly across all of them.
- About 1,000 invoices: the extra 900 bought 1.6 points on the layouts already covered. I did not test whether more layouts in the training set would help unseen ones, and that is the obvious next experiment.
In every case, budget for the labelling, not the GPU.
Before you trust these numbers
These numbers describe this task and this model, and they come with limits.
- Synthetic data. The invoices are cleaner than real ones, so real documents may need more data than this sweep suggests.
- Scope of the claim. For this task and this model, the curve flattened around 100 invoices. That is a measurement, not a universal rule.
- A small unseen set. It has 45 invoices, so one invoice is about 2.2 points. The 250-invoice range (77.8-86.7%) shows how much that moves things.
- Three seeds. The ranges are the lowest and highest of three runs, not confidence intervals. The prompt-only baseline ran once.
- Batched evaluation. Test invoices are generated 16 at a time with greedy decoding. I re-ran one run (25 invoices, seed 0) one invoice at a time. Invoice accuracy was identical (77.1% seen, 42.2% unseen); field accuracy differed by 0.3 and 0.4 points. Evaluation took about 1 minute batched and 12.9 minutes one at a time.
-
Every number in this post matches a row in
results.csv, andplot.pyprints the same table.
How to rerun it
The whole experiment is one script, sweep.py, and it picks up where it stopped if interrupted: runs already in results.csv are skipped.
python sweep.py # baseline, then 6 sizes x 3 seeds (0, 1, 2)
python sweep.py --sizes 100 --seeds 0 1 2
python sweep.py --skip-baseline
python plot.py # prints the table, saves the chart
sweep.py expects an invoice_common folder in its parent directory, with the invoice data and the prompt, parsing and scoring helpers.
Three parts of the script matter for reading the results. First, how a training set is picked: invoices are drawn round-robin from the 24 layout-and-language groups, so every size is balanced and, for one seed, a smaller set is the start of a larger one.
def pick_subset(n: int, seed: int) -> list[str]:
"""n pool invoices, spread evenly over the 24 format x language groups."""
answers = json.loads((POOL_DIR / "answers.json").read_text(encoding="utf-8"))
assert n <= len(answers), f"pool only has {len(answers)} invoices"
groups = defaultdict(list)
for filename, info in sorted(answers.items()):
groups[(info["format"], info["language"])].append(filename)
rng = random.Random(seed)
keys = sorted(groups)
for key in keys:
rng.shuffle(groups[key])
rng.shuffle(keys) # with n < 24, which groups get the spare places varies by seed
picked = []
while len(picked) < n:
for key in keys:
if groups[key] and len(picked) < n:
picked.append(groups[key].pop())
return picked
Second, the training settings, which never change between runs. The seed is set before the adapter is created, so a seed repeats the adapter's starting values too.
EPOCHS = 3
RANK, ALPHA = 16, 32
set_seed(seed) # before get_peft_model, so the adapter's starting values repeat too
model = AutoModelForCausalLM.from_pretrained(MODEL_DIR, dtype=torch.bfloat16).to(
"cuda"
)
model = get_peft_model(
model,
LoraConfig(
r=RANK,
lora_alpha=ALPHA,
target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
lora_dropout=0.05,
task_type="CAUSAL_LM",
),
)
trainer = Trainer(
model=model,
args=TrainingArguments(
output_dir=str(out_dir),
num_train_epochs=EPOCHS,
per_device_train_batch_size=4,
learning_rate=2e-4,
logging_steps=10,
save_strategy="no",
report_to="none",
bf16=True,
seed=seed,
),
train_dataset=Dataset.from_list(examples),
data_collator=collate,
)
Third, evaluation: test invoices are generated 16 at a time, padded on the left so every prompt ends where its answer starts, with greedy decoding.
# Pad on the LEFT for generation, so every prompt ends right where its answer
# starts. (Training pads by hand in collate(), so this does not affect it.)
tokeniser.padding_side = "left"
# ... then, for each batch of 16 test invoices:
inputs = tokeniser(chat_texts, return_tensors="pt", padding=True).to("cuda")
n_in = inputs["input_ids"].shape[1] # same for all: padded to the longest
with torch.no_grad():
output = model.generate(**inputs, max_new_tokens=400, do_sample=False)
The full code
Save the files below in one folder. plot.py needs only matplotlib and results.csv; sweep.py needs a GPU and the invoice_common folder described above.
sweep.py
import argparse
import csv
import gc
import json
import random
import sys
import time
from collections import defaultdict
from pathlib import Path
import torch
from datasets import Dataset
from peft import LoraConfig, PeftModel, get_peft_model
from transformers import (
AutoModelForCausalLM,
AutoTokenizer,
Trainer,
TrainingArguments,
set_seed,
)
COMMON = Path(__file__).resolve().parent.parent / "invoice_common"
sys.path.insert(0, str(COMMON / "evaluations"))
from parsing import parse_output
from prompts import build_ft, build_p2
from score import score_one
POST_DIR = Path(__file__).resolve().parent
POOL_DIR = COMMON / "data" / "pool"
MODEL_DIR = COMMON / "qwen15b_base"
ADAPTERS_DIR = POST_DIR / "adapters"
RESULTS_CSV = POST_DIR / "results.csv"
SIZES = (25, 50, 100, 250, 500, 1000)
SEEDS = (0, 1, 2)
EVAL_BATCH = 16 # test invoices generated at once; 1 at a time leaves the GPU mostly idle
TEST_BUCKETS = {"test_seen": "seen", "eval_unseen": "unseen"}
FIELDS = ("invoice_number", "date", "vendor_name", "line_items", "total")
COLUMNS = [
"run", "n_train", "seed", "rank", "epochs",
"train_minutes", "peak_vram_gb", "adapter_mb",
"field_acc_seen", "invoice_acc_seen", "field_acc_unseen", "invoice_acc_unseen",
"eval_minutes",
]
# Fixed in every run - only the number of invoices and the seed change.
MAX_LENGTH = 768 # longest example measured in the first post: 538 tokens
EPOCHS = 3
RANK, ALPHA = 16, 32
tokeniser = AutoTokenizer.from_pretrained(MODEL_DIR)
if tokeniser.pad_token is None:
tokeniser.pad_token = tokeniser.eos_token
def pick_subset(n: int, seed: int) -> list[str]:
"""n pool invoices, spread evenly over the 24 format x language groups."""
answers = json.loads((POOL_DIR / "answers.json").read_text(encoding="utf-8"))
assert n <= len(answers), f"pool only has {len(answers)} invoices"
groups = defaultdict(list)
for filename, info in sorted(answers.items()):
groups[(info["format"], info["language"])].append(filename)
rng = random.Random(seed)
keys = sorted(groups)
for key in keys:
rng.shuffle(groups[key])
rng.shuffle(keys) # with n < 24, which groups get the spare places varies by seed
picked = []
while len(picked) < n:
for key in keys:
if groups[key] and len(picked) < n:
picked.append(groups[key].pop())
return picked
def build_example(prompt: str, answer: str) -> dict:
"""Tokenise one pair; labels of -100 mean the prompt is read but not graded."""
prompt_text = tokeniser.apply_chat_template(
[{"role": "user", "content": prompt}],
tokenize=False,
add_generation_prompt=True,
)
full_text = prompt_text + answer + tokeniser.eos_token
prompt_ids = tokeniser(prompt_text, add_special_tokens=False)["input_ids"]
full_ids = tokeniser(full_text, add_special_tokens=False)["input_ids"][:MAX_LENGTH]
labels = list(full_ids)
for i in range(min(len(prompt_ids), len(labels))):
labels[i] = -100
return {"input_ids": full_ids, "labels": labels}
def collate(batch: list[dict]) -> dict:
"""Pad each batch to its longest sequence; padding is neither learned nor attended."""
longest = max(len(item["input_ids"]) for item in batch)
input_ids, labels, attention = [], [], []
for item in batch:
pad = longest - len(item["input_ids"])
input_ids.append(item["input_ids"] + [tokeniser.pad_token_id] * pad)
labels.append(item["labels"] + [-100] * pad)
attention.append([1] * len(item["input_ids"]) + [0] * pad)
return {
"input_ids": torch.tensor(input_ids),
"labels": torch.tensor(labels),
"attention_mask": torch.tensor(attention),
}
def train_adapter(files: list[str], seed: int, out_dir: Path) -> dict:
"""Train one LoRA adapter on the given pool invoices; return its cost."""
answers = json.loads((POOL_DIR / "answers.json").read_text(encoding="utf-8"))
examples = [
build_example(
build_ft((POOL_DIR / f).read_text(encoding="utf-8")),
json.dumps(answers[f]["ground_truth"], ensure_ascii=False),
)
for f in files
]
set_seed(seed) # before get_peft_model, so the adapter's starting values repeat too
model = AutoModelForCausalLM.from_pretrained(MODEL_DIR, dtype=torch.bfloat16).to(
"cuda"
)
model = get_peft_model(
model,
LoraConfig(
r=RANK,
lora_alpha=ALPHA,
target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
lora_dropout=0.05,
task_type="CAUSAL_LM",
),
)
model.print_trainable_parameters()
trainer = Trainer(
model=model,
args=TrainingArguments(
output_dir=str(out_dir),
num_train_epochs=EPOCHS,
per_device_train_batch_size=4,
learning_rate=2e-4,
logging_steps=10,
save_strategy="no",
report_to="none",
bf16=True,
seed=seed,
),
train_dataset=Dataset.from_list(examples),
data_collator=collate,
)
torch.cuda.reset_peak_memory_stats()
start = time.perf_counter()
trainer.train()
minutes = (time.perf_counter() - start) / 60
peak_gb = torch.cuda.max_memory_allocated() / 1024**3
model.save_pretrained(out_dir)
adapter_mb = (out_dir / "adapter_model.safetensors").stat().st_size / 1024**2
# The Trainer keeps reference cycles to the model; without gc.collect() the
# training memory stays allocated and testing spills out of VRAM (very slow).
del trainer, model
gc.collect()
torch.cuda.empty_cache()
return {
"train_minutes": round(minutes, 2),
"peak_vram_gb": round(peak_gb, 2),
"adapter_mb": round(adapter_mb, 1),
}
def p2_prompt():
"""Baseline prompt from the first post: instructions + one fixed training example."""
train_dir = COMMON / "data" / "train"
answers = json.loads((train_dir / "answers.json").read_text(encoding="utf-8"))
example = (train_dir / "inv_001.txt").read_text(encoding="utf-8")
answer = json.dumps(answers["inv_001.txt"]["ground_truth"], ensure_ascii=False)
return lambda invoice_text: build_p2(invoice_text, example, answer)
def evaluate(adapter_dir: Path | None, out_file: Path) -> dict:
"""Run every test invoice; None means the untrained baseline with the P2 prompt."""
model = AutoModelForCausalLM.from_pretrained(MODEL_DIR, dtype=torch.bfloat16).to(
"cuda"
)
if adapter_dir is None:
build_prompt = p2_prompt()
else:
model = PeftModel.from_pretrained(model, adapter_dir)
build_prompt = build_ft
model.eval()
# Pad on the LEFT for generation, so every prompt ends right where its answer
# starts. (Training pads by hand in collate(), so this does not affect it.)
tokeniser.padding_side = "left"
result = {}
scored = 0
start = time.perf_counter()
with out_file.open("w", encoding="utf-8") as out:
for bucket, short in TEST_BUCKETS.items():
folder = COMMON / "data" / bucket
answers = json.loads((folder / "answers.json").read_text(encoding="utf-8"))
filenames = sorted(answers)
field_hits, exact = 0, 0
for i in range(0, len(filenames), EVAL_BATCH):
batch = filenames[i : i + EVAL_BATCH]
chat_texts = [
tokeniser.apply_chat_template(
[{"role": "user", "content": build_prompt(
(folder / filename).read_text(encoding="utf-8"))}],
tokenize=False,
add_generation_prompt=True,
)
for filename in batch
]
inputs = tokeniser(chat_texts, return_tensors="pt", padding=True).to("cuda")
n_in = inputs["input_ids"].shape[1] # same for all: padded to the longest
with torch.no_grad():
output = model.generate(**inputs, max_new_tokens=400, do_sample=False)
for filename, generated in zip(batch, output):
raw = tokeniser.decode(generated[n_in:], skip_special_tokens=True)
out.write(
json.dumps(
{"bucket": bucket, "invoice": filename, "raw_output": raw},
ensure_ascii=False,
)
+ "\n"
)
predicted, _ = parse_output(raw)
if isinstance(predicted, dict):
fields = score_one(predicted, answers[filename]["ground_truth"])
else:
fields = dict.fromkeys(FIELDS, False) # unparseable = every field wrong
field_hits += sum(fields.values())
exact += all(fields.values())
out.flush() # so progress is visible on disk while it runs
scored += len(batch)
elapsed = time.perf_counter() - start
print(f" {scored} invoices, {elapsed / scored:.2f} s each", flush=True)
n = len(answers)
result[f"field_acc_{short}"] = round(field_hits / (n * len(FIELDS)), 4)
result[f"invoice_acc_{short}"] = round(exact / n, 4)
print(f" {bucket:12} invoice accuracy {exact}/{n}")
result["eval_minutes"] = round((time.perf_counter() - start) / 60, 2)
del model
gc.collect()
torch.cuda.empty_cache()
return result
def finished_runs() -> set[str]:
"""Runs already in results.csv - lets an interrupted sweep pick up where it stopped."""
if not RESULTS_CSV.exists():
return set()
with RESULTS_CSV.open(newline="", encoding="utf-8") as f:
return {row["run"] for row in csv.DictReader(f)}
def append_row(row: dict) -> None:
is_new = not RESULTS_CSV.exists()
with RESULTS_CSV.open("a", newline="", encoding="utf-8") as f:
writer = csv.DictWriter(f, fieldnames=COLUMNS)
if is_new:
writer.writeheader()
writer.writerow(row)
if __name__ == "__main__":
parser = argparse.ArgumentParser(description="Post 1: accuracy vs training-set size")
parser.add_argument("--sizes", type=int, nargs="+", default=SIZES)
parser.add_argument("--seeds", type=int, nargs="+", default=SEEDS)
parser.add_argument("--skip-baseline", action="store_true")
args = parser.parse_args()
done = finished_runs()
if not args.skip_baseline and "baseline" not in done:
print("=== baseline: untrained model + P2 prompt ===")
out_dir = ADAPTERS_DIR / "baseline"
out_dir.mkdir(parents=True, exist_ok=True)
scores = evaluate(None, out_dir / "answers.jsonl")
append_row({"run": "baseline", "n_train": 0, "train_minutes": 0, **scores})
for n in args.sizes:
for seed in args.seeds:
run = f"n{n}_s{seed}"
if run in done:
print(f"=== {run}: already in results.csv, skipping ===")
continue
print(f"=== {run}: training on {n} invoices ===")
out_dir = ADAPTERS_DIR / run
cost = train_adapter(pick_subset(n, seed), seed, out_dir)
scores = evaluate(out_dir, out_dir / "answers.jsonl")
row = {"run": run, "n_train": n, "seed": seed, "rank": RANK, "epochs": EPOCHS,
**cost, **scores}
append_row(row)
print(row)
plot.py
"""
Reads results.csv (written by sweep.py), prints the summary table, and saves
accuracy_vs_training_size.png. Needs matplotlib only.
python plot.py
"""
import csv
from collections import defaultdict
from pathlib import Path
from statistics import mean
import matplotlib.pyplot as plt
HERE = Path(__file__).resolve().parent
RESULTS_CSV = HERE / "results.csv"
OUT_PNG = HERE / "accuracy_vs_training_size.png"
SEEN_COLOUR = "#1f6fb2"
UNSEEN_COLOUR = "#d2691e"
def load():
"""Return (baseline row, {n_train: [rows]}) from results.csv."""
baseline, by_size = None, defaultdict(list)
with RESULTS_CSV.open(newline="", encoding="utf-8") as f:
for row in csv.DictReader(f):
if row["run"] == "baseline":
baseline = row
else:
by_size[int(row["n_train"])].append(row)
if baseline is None:
raise SystemExit("results.csv has no baseline row")
return baseline, dict(sorted(by_size.items()))
def pct(rows, column):
"""Percent values of one column across the seeds of one training size."""
return [float(r[column]) * 100 for r in rows]
def main():
baseline, by_size = load()
sizes = list(by_size)
base_seen = float(baseline["invoice_acc_seen"]) * 100
base_unseen = float(baseline["invoice_acc_unseen"]) * 100
# Summary table: compare this against the table in the post.
print(f"{'train':>6} {'seen mean (min-max)':>24} {'unseen mean (min-max)':>26}")
print(f"{'prompt':>6} {base_seen:>24.1f} {base_unseen:>26.1f}")
stats = {}
for n, rows in by_size.items():
seen, unseen = pct(rows, "invoice_acc_seen"), pct(rows, "invoice_acc_unseen")
stats[n] = (seen, unseen)
print(
f"{n:>6} {mean(seen):>14.1f} ({min(seen):.1f}-{max(seen):.1f})"
f"{mean(unseen):>16.1f} ({min(unseen):.1f}-{max(unseen):.1f})"
)
fig, ax = plt.subplots(figsize=(9, 5.2), dpi=200)
for idx, (colour, marker, label, base) in enumerate(
[
(SEEN_COLOUR, "o", "Seen layouts (192 test invoices)", base_seen),
(UNSEEN_COLOUR, "s", "Unseen layouts (45 test invoices)", base_unseen),
]
):
means = [mean(stats[n][idx]) for n in sizes]
lows = [min(stats[n][idx]) for n in sizes]
highs = [max(stats[n][idx]) for n in sizes]
ax.fill_between(sizes, lows, highs, color=colour, alpha=0.15, linewidth=0)
for n in sizes: # the individual seeds
ax.scatter([n] * len(stats[n][idx]), stats[n][idx], s=14, color=colour, alpha=0.45, zorder=2)
ax.plot(sizes, means, color=colour, marker=marker, linewidth=2.2, markersize=7, label=label, zorder=3)
ax.hlines(base, 20, 1090, color=colour, linestyle="--", linewidth=1.3, alpha=0.8)
# Baseline labels sit in the right margin, clear of the data lines.
ax.text(1130, base_seen, f"prompt only {base_seen:.1f}%", color=SEEN_COLOUR,
ha="left", va="center", fontsize=8.5)
ax.text(1130, base_unseen, f"prompt only {base_unseen:.1f}%", color=UNSEEN_COLOUR,
ha="left", va="center", fontsize=8.5)
ax.annotate("100 invoices: 98.4%", xy=(100, mean(stats[100][0])), xytext=(40, 100.5),
fontsize=9, arrowprops=dict(arrowstyle="->", color="#444"), ha="center")
ax.annotate("25 invoices: worse than\nthe prompt alone", xy=(25, mean(stats[25][1])),
xytext=(34, 33), fontsize=9, arrowprops=dict(arrowstyle="->", color="#444"))
ax.set_xscale("log")
ax.set_xlim(20, 2100)
ax.set_xticks(sizes)
ax.set_xticklabels([f"{n:,}" for n in sizes])
ax.minorticks_off()
ax.set_ylim(30, 103)
ax.set_xlabel("Training invoices (log scale)")
ax.set_ylabel("Invoices with every field correct (%)")
ax.set_title("How many invoices does it take to teach a model?", fontsize=13, loc="left")
ax.grid(axis="y", alpha=0.25)
ax.spines[["top", "right"]].set_visible(False)
ax.legend(loc="lower right", frameon=False, fontsize=9)
fig.text(0.01, 0.01,
"Qwen2.5-1.5B-Instruct + LoRA (rank 16), 3 seeds per size. Line = mean, band = min-max, dots = single runs. "
"Synthetic invoices.", fontsize=7, color="#555")
fig.tight_layout(rect=(0, 0.03, 1, 1))
fig.savefig(OUT_PNG)
print(f"saved {OUT_PNG.name}")
if __name__ == "__main__":
main()
results.csv
run,n_train,seed,rank,epochs,train_minutes,peak_vram_gb,adapter_mb,field_acc_seen,invoice_acc_seen,field_acc_unseen,invoice_acc_unseen,eval_minutes
n25_s0,25,0,16,3,0.13,12.12,16.7,0.9302,0.7708,0.7911,0.4222,1.03
baseline,0,,,,0,,,0.9635,0.849,0.9467,0.8222,1.21
n25_s1,25,1,16,3,0.11,11.9,16.7,0.9583,0.8542,0.8311,0.4889,1.02
n25_s2,25,2,16,3,0.08,12.03,16.7,0.9448,0.8021,0.7822,0.4444,1.01
n50_s0,50,0,16,3,0.15,12.31,16.7,0.9781,0.9271,0.9422,0.7778,0.99
n50_s1,50,1,16,3,0.15,11.88,16.7,0.9792,0.9271,0.9378,0.7333,0.99
n50_s2,50,2,16,3,0.16,12.05,16.7,0.9854,0.9375,0.9378,0.7333,1.0
n100_s0,100,0,16,3,0.36,12.22,16.7,0.9969,0.9896,0.96,0.8,1.0
n100_s1,100,1,16,3,0.29,11.88,16.7,0.9958,0.9844,0.9511,0.8,0.99
n100_s2,100,2,16,3,0.34,12.08,16.7,0.9958,0.9792,0.9644,0.8222,0.99
n250_s0,250,0,16,3,0.87,12.21,16.7,0.9979,0.9896,0.9467,0.7778,0.99
n250_s1,250,1,16,3,1.3,12.11,16.7,1.0,1.0,0.9689,0.8667,1.03
n250_s2,250,2,16,3,0.7,12.08,16.7,1.0,1.0,0.9467,0.7778,0.99
n500_s0,500,0,16,3,2.46,12.27,16.7,1.0,1.0,0.9511,0.8,1.05
n500_s1,500,1,16,3,1.48,12.24,16.7,0.999,0.9948,0.9378,0.7778,1.0
n500_s2,500,2,16,3,1.86,12.3,16.7,1.0,1.0,0.9156,0.7333,1.0
n1000_s0,1000,0,16,3,3.14,12.27,16.7,1.0,1.0,0.9511,0.7778,1.0
n1000_s1,1000,1,16,3,3.11,12.26,16.7,1.0,1.0,0.96,0.8,1.0
n1000_s2,1000,2,16,3,3.45,12.3,16.7,1.0,1.0,0.9422,0.8222,1.0
Top comments (0)