Contributing to open-source LLM research is easier when you have a structured template to start from. I built a lightweight scaffold generator that takes a one-sentence research idea and produces a literature review, an experiment plan, and a runnable Python skeleton. Because it runs on Oxlo.ai, I do not have to worry about token costs ballooning when I paste long papers or multi-step reasoning traces into the context window.
What you'll need
- Python 3.10 or newer
- An Oxlo.ai API key from https://portal.oxlo.ai
- The OpenAI SDK:
pip install openai
Step 1: Set up the client and system prompt
Every request goes to Oxlo.ai through the OpenAI-compatible endpoint. I use Llama 3.3 70B as the general-purpose workhorse because it follows long instructions reliably. The system prompt forces the model to emit three distinct sections so we can parse them later.
from openai import OpenAI
client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")
SYSTEM_PROMPT = """You are a research assistant for open-source LLM development.
Given a topic, produce structured research scaffolding in three sections:
1. Literature context: 3 to 5 foundational concepts or related works.
2. Experiment design: a hypothesis, independent and dependent variables, and a dataset suggestion.
3. Code skeleton: a minimal Python file with TODO comments showing where to plug in the model, data loader, and evaluation metric.
Be concise. Use Markdown headers for each section."""
response = client.chat.completions.create(
model="llama-3.3-70b",
messages=[
{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content": "I want to study how chain-of-thought prompting affects hallucination rates in smaller LLMs."},
],
)
print(response.choices[0].message.content)
Step 2: Wrap the generator in a reusable function
Hardcoding the topic is fine for a smoke test, but a real tool needs a function. I also add a small helper that writes the full Markdown response to disk so I can review it later.
import os
from openai import OpenAI
client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")
SYSTEM_PROMPT = """You are a research assistant for open-source LLM development.
Given a topic, produce structured research scaffolding in three sections:
1. Literature context: 3 to 5 foundational concepts or related works.
2. Experiment design: a hypothesis, independent and dependent variables, and a dataset suggestion.
3. Code skeleton: a minimal Python file with TODO comments showing where to plug in the model, data loader, and evaluation metric.
Be concise. Use Markdown headers for each section."""
def generate_scaffold(topic: str) -> str:
response = client.chat.completions.create(
model="llama-3.3-70b",
messages=[
{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content": topic},
],
)
return response.choices[0].message.content
def save_scaffold(topic: str, content: str, out_dir: str = "contrib"):
os.makedirs(out_dir, exist_ok=True)
safe_name = topic.replace(" ", "_").replace("?", "").replace(",", "")[:40]
path = os.path.join(out_dir, f"{safe_name}.md")
with open(path, "w", encoding="utf-8") as f:
f.write(content)
print(f"Saved scaffold to {path}")
if __name__ == "__main__":
topic = "Does fine-tuning on synthetic data improve tool-use accuracy in 7B parameter models?"
scaffold = generate_scaffold(topic)
save_scaffold(topic, scaffold)
print(scaffold)
Step 3: Turn the skeleton into a runnable script
The scaffold contains a rough skeleton, but I need a standalone file I can execute. I extract the code block with a regex and send it back to Oxlo.ai for expansion. I use DeepSeek V3.2 here because it is strong on coding and reasoning, and Oxlo.ai offers it on the same flat per-request plan.
import os, re
from openai import OpenAI
client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")
CODE_PROMPT = """You are a research engineer. Expand the provided experiment skeleton into a complete, runnable Python script.
Use argparse for CLI arguments, include a main() function, and add docstrings.
Do not invent external datasets. If the user mentions a dataset, use a placeholder loader that generates synthetic data with the same schema.
Return only the Python code inside a fenced Markdown block."""
def extract_code_block(text: str) -> str:
match = re.search(r"
```python\n(.*?)```
", text, re.DOTALL)
return match.group(1) if match else text
def expand_to_script(skeleton_md: str, out_dir: str = "contrib") -> str:
skeleton = extract_code_block(skeleton_md)
response = client.chat.completions.create(
model="deepseek-v3.2",
messages=[
{"role": "system", "content": CODE_PROMPT},
{"role": "user", "content": f"Expand this skeleton into a full script:\n\n{skeleton}"},
],
)
code = response.choices[0].message.content
os.makedirs(out_dir, exist_ok=True)
with open(os.path.join(out_dir, "experiment.py"), "w", encoding="utf-8") as f:
f.write(extract_code_block(code))
return code
if __name__ == "__main__":
# In practice, skeleton_md comes from Step 2. Here we use a minimal stand-in.
stand_in = '''
```python\ndef run_experiment(model_path, dataset_path):\n # TODO: load model and data\n pass\n```
'''
print(expand_to_script(stand_in))
Step 4: Generate an evaluation checklist
A contribution is not complete without a way to judge success. I add a final agent call that reads the experiment script and produces a Markdown checklist of metrics, baselines, and significance tests.
import os
from openai import OpenAI
client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")
CHECKLIST_PROMPT = """You are a rigorous ML reviewer. Given an experiment script and its original topic, produce a Markdown checklist with:
- Recommended metrics (e.g., accuracy, F1, perplexity).
- Suggested baselines to compare against.
- Statistical tests to validate significance.
- A short paragraph on expected failure modes.
Be specific and actionable."""
def generate_checklist(topic: str, script: str, out_dir: str = "contrib") -> str:
response = client.chat.completions.create(
model="qwen-3-32b",
messages=[
{"role": "system", "content": CHECKLIST_PROMPT},
{"role": "user", "content": f"Topic: {topic}\n\nScript:\n{script}"},
],
)
checklist = response.choices[0].message.content
os.makedirs(out_dir, exist_ok=True)
with open(os.path.join(out_dir, "checklist.md"), "w", encoding="utf-8") as f:
f.write(checklist)
return checklist
if __name__ == "__main__":
topic = "Does fine-tuning on synthetic data improve tool-use accuracy in 7B parameter models?"
dummy_script = "def run_experiment(model_path, dataset_path):\n pass"
print(generate_checklist(topic, dummy_script))
Step 5: Assemble the final package
Now I wire everything together in a single function. One research topic enters, and a folder containing the scaffold, experiment.py, and checklist exits. All three calls hit Oxlo.ai, and because the platform charges per request rather than per token, the total cost is predictable even when the prompts are long.
import os, re
from openai import OpenAI
client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")
SCAFFOLD_PROMPT = """You are a research assistant for open-source LLM development.
Given a topic, produce structured research scaffolding in three sections:
1. Literature context: 3 to 5 foundational concepts or related works.
2. Experiment design: a hypothesis, independent and dependent variables, and a dataset suggestion.
3. Code skeleton: a minimal Python file with TODO comments showing where to plug in the model, data loader, and evaluation metric.
Be concise. Use Markdown headers for each section."""
CODE_PROMPT = """You are a research engineer. Expand the provided experiment skeleton into a complete, runnable Python script.
Use argparse for CLI arguments, include a main() function, and add docstrings.
Do not invent external datasets. If the user mentions a dataset, use a placeholder loader that generates synthetic data with the same schema.
Return only the Python code inside a fenced Markdown block."""
CHECKLIST_PROMPT = """You are a rigorous ML reviewer. Given an experiment script and its original topic, produce a Markdown checklist with:
- Recommended metrics (e.g., accuracy, F1, perplexity).
- Suggested baselines to compare against.
- Statistical tests to validate significance.
- A short paragraph on expected failure modes.
Be specific and actionable."""
def extract_code_block(text: str) -> str:
match = re.search(r"
```python\n(.*?)```
", text, re.DOTALL)
return match.group(1) if match else text
def build_contribution_package(topic: str, out_dir: str = "contrib"):
os.makedirs(out_dir, exist_ok=True)
safe = re.sub(r"[^\w]", "_", topic)[:40]
# 1. Scaffold
r1 = client.chat.completions.create(
model="llama-3.3-70b",
messages=[
{"role": "system", "content": SCAFFOLD_PROMPT},
{"role": "user", "content": topic},
],
)
scaffold = r1.choices[0].message.content
with open(os.path.join(out_dir, f"{safe}_scaffold.md"), "w", encoding="utf-8") as f:
f.write(scaffold)
# 2. Script
skeleton = extract_code_block(scaffold)
r2 = client.chat.completions.create(
model="deepseek-v3.2",
messages=[
{"role": "system", "content": CODE_PROMPT},
{"role": "user", "content": f"Expand this skeleton into a full script:\n\n{skeleton}"},
],
)
script = extract_code_block(r2.choices[0].message.content)
with open(os.path.join(out_dir, "experiment.py"), "w", encoding="utf-8") as f:
f.write(script)
# 3. Checklist
r3 = client.chat.completions.create(
model="qwen-3-32b",
messages=[
{"role": "system", "content": CHECKLIST_PROMPT},
{"role": "user", "content": f"Topic: {topic}\n\nScript:\n{script}"},
],
)
checklist = r3.choices[0].message.content
with open(os.path.join(out_dir, f"{safe}_checklist.md"), "w", encoding="utf-8") as f:
f.write(checklist)
print(f"Package ready in ./{out_dir}/")
if __name__ == "__main__":
idea = "Does fine-tuning on synthetic data improve tool-use accuracy in 7B parameter models?"
build_contribution_package(idea)
Run it
Save the final script as research_agent.py, set your API key, and run it.
export OXLO_API_KEY="sk-..."
python research_agent.pyExample output:
Package ready in ./contrib/
=== contrib/does_fine_tuning_on_synthetic_data_improve_tool_use_acc_scaffold.md ===
## Literature context
- Toolformer showed that LLMs can teach themselves to use external tools via API calls.
- Gorilla demonstrated fine-tuning for tool-use with curated API documentation.
- Synthetic data generation via self-instruction is a common data augmentation strategy for smaller models.
## Experiment design
- Hypothesis: Fine-tuning a 7B model on 10k synthetic tool-use dialogues improves API call accuracy by at least 5% over the base model.
- Independent variable: Presence of synthetic fine-tuning data.
- Dependent variable: Exact match accuracy of API calls on the ToolBench validation set.
- Dataset suggestion: Generate synthetic pairs with GPT-Oss 120B or use the public ToolBench split.
## Code skeleton
```python
# TODO: load model (Oxlo.ai supports Llama 3.3 70B, Qwen 3 32B, etc.)
# TODO: load synthetic dataset
# TODO: evaluate exact match
```
=== contrib/experiment.py ===
import argparse
def main():
parser = argparse.ArgumentParser(description="Synthetic tool-use experiment")
parser.add_argument("--model", default="llama-3.3-70b")
parser.add_argument("--data", required=True)
args = parser.parse_args()
# TODO: implement loading and evaluation
print(f"Running with {args.model} on {args.data}")
if __name__ == "__main__":
main()
=== contrib/does_fine_tuning_on_synthetic_data_improve_tool_use_acc_checklist.md ===
- Metrics: Exact match accuracy, BLEU, and API call validity score.
- Baselines: Base 7B model without fine-tuning, few-shot prompting with 3 examples.
- Statistical tests: Paired t-test across 5 random seeds.
- Failure modes: Synthetic data may leak test APIs, causing inflated exact match scores.
Next steps
Try swapping Llama 3.3 70B for Kimi K2.6 when you need advanced reasoning and vision capabilities, or use GLM 5 for long-horizon agentic planning. If you are ready to share your work, the flat per-request pricing on Oxlo.ai makes it cheap to run large-scale evaluations without token-count anxiety. Check out the pricing at https://oxlo.ai/pricing and scale up when you are ready.
Top comments (0)