Prompt Engineering Best Practices for Gemma

Getting quality output from Gemma isn't about clever tricks — it's about matching its exact chat template, structuring instructions clearly, and knowing where its function-calling and reasoning features need extra guardrails. This guide covers the practical prompting patterns that actually move the needle.

Understand Gemma's Chat Template First

Gemma models use <start_of_turn> and <end_of_turn> tokens as turn delimiters instead of the [INST]/[/INST] format used by Llama models — mixing formats from other model families is the single most common cause of degraded output quality.

<start_of_turn>user
{system instructions + user message}<end_of_turn>
<start_of_turn>model

A few non-negotiable rules:

• Gemma has no dedicated system role in its chat template — system-level instructions must be prepended directly into the first user turn rather than sent as a separate role
• The BOS token <bos> is added automatically by apply_chat_template — never add it manually or you'll get malformed prompts
• When using function calling, let apply_chat_template with a tools argument inject the schema automatically; hand-serializing tool JSON into the user message breaks the format the model was trained on
• Always use the tokenizer's chat template helper rather than hand-building the raw string — small formatting mistakes (extra spaces, missing tokens) measurably degrade output quality

Structure Every Prompt Clearly

Gemma responds best to prompts with explicit structure rather than long, unstructured paragraphs of instructions.

Task: Summarize the following transaction log into 3 bullet points.
Input: {log text}
Output:

Use delimiters like ## or custom tags (<context>, <instruction>) to visually separate distinct parts of a prompt — this significantly reduces the chance of the model misreading which text is an instruction versus which is data to process. Keep the actual instruction concise; verbose, repeated preambles tend to produce worse results than a short, direct ask.

Use Few-Shot Examples for Format Control

When you need a specific output shape — JSON with exact field names, a particular tone, or a fixed structure — showing 1-5 examples works far more reliably than describing the format in words alone.

Pattern	Description	Example
Zero-shot	Direct instruction, no examples	"Extract the invoice total as JSON: {amount: number}"
Few-shot	2-5 examples before the real task	Show 3 input/output pairs, then the actual input
Chain-of-thought	Ask the model to reason step by step	"Think step by step, then give the final answer"
RAG-style	Structured context + question	<context>{docs}</context>\nQuestion: {query}

One developer reported raising JSON-only output accuracy from roughly 65% to over 95% simply by pairing an explicit schema definition with a single well-formatted example — a small investment with a large payoff for backend integrations that parse model output programmatically.

Set Temperature Based on the Task

Temperature has an outsized effect on Gemma's reliability for structured tasks versus creative ones.

• Use 0.1-0.2 for deterministic tasks like entity extraction, translation, classification, or anything feeding directly into your backend logic
• Use 0.7-1.0 for general conversation or balanced reasoning tasks
• Use 1.2-2.0 only for creative generation like brainstorming or content drafting, where variability is desirable
• Leave other sampling parameters (top-p, top-k) at their defaults unless you have a specific reason to tune them — over-tuning multiple parameters at once makes debugging output quality much harder

Prompt Engineering for Function Calling and Agents

Gemma's tool-calling only works reliably when tool descriptions and system instructions are precise — vague descriptions are the top cause of agents either ignoring tools or calling them unnecessarily.

• Write detailed, specific tool descriptions — the model relies entirely on these descriptions to decide which tool to call and when
• Use required fields in your JSON schemas to prevent the model from omitting critical parameters
• Limit the number of available tools to 5-10 per call; too many options measurably confuses tool selection
• Include example values in parameter descriptions (e.g., "city name, e.g. 'Lagos', 'London'") to guide correct argument formatting
• If the model ignores tools and answers directly from its own knowledge, add an explicit instruction like "Never guess account balances or transaction data — always use the provided tools"
• If the model calls tools unnecessarily, tighten the description with a scoped condition like "Use ONLY when the user explicitly requests a balance check"
• Set a max_steps limit in your agent loop to prevent infinite tool-calling loops
• Handle tool errors gracefully by returning structured error info, so the model can retry or explain the failure to the user rather than silently failing

For a fintech agent handling account queries or bill payments, these guardrails matter more than general prompt wording — a well-described tool schema does more work than any amount of persuasive phrasing in the prompt itself.

Break Down Complex Multi-Step Logic

If a single prompt requires conditional multi-step reasoning ("do X first, if the result is A do M, otherwise do N, then do Y"), Gemma performs more reliably when you split this into separate calls chained together in code, rather than asking it to execute all the branching logic in one shot. Keep each individual call focused on one clear, bounded task.

Keep Output Short and Deliberate

Inference speed is heavily tied to output length, so trimming unnecessary verbosity has both a quality and a performance benefit.

• Ask explicitly for the shortest output that satisfies the task, then do lightweight post-processing in code rather than relying on the model to format everything perfectly
• For on-device or low-latency use cases (E2B/E4B), this matters even more since every extra generated token adds directly to response time
• Avoid asking for both a long explanation and a structured answer in the same call — request the structured answer only, and log reasoning separately if you need it for debugging

Python Example: Function Calling End-to-End

Here's a minimal, working pattern using the transformers library and apply_chat_template with a tools schema — the same approach that avoids the manual-serialization pitfall described above.

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch, json

model_id = "google/gemma-4-4b-it"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=torch.bfloat16, device_map="auto")

def check_balance(account_id: str) -> dict:
    """Look up the current balance for a given account ID."""
    return {"account_id": account_id, "balance": 42500.00, "currency": "NGN"}

tools = [
    {
        "type": "function",
        "function": {
            "name": "check_balance",
            "description": "Use ONLY when the user explicitly requests an account balance check. Never guess balances yourself.",
            "parameters": {
                "type": "object",
                "properties": {
                    "account_id": {
                        "type": "string",
                        "description": "The account identifier, e.g. 'ACC-10293'"
                    }
                },
                "required": ["account_id"]
            }
        }
    }
]

messages = [
    {"role": "user", "content": "Can you check the balance on account ACC-10293?"}
]

prompt = tokenizer.apply_chat_template(
    messages,
    tools=tools,
    add_generation_prompt=True,
    tokenize=False
)

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=256, temperature=0.1)
response = tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True)

try:
    call = json.loads(response)
    if call.get("name") == "check_balance":
        result = check_balance(**call["arguments"])
        print(result)
except json.JSONDecodeError:
    print("Model returned plain text:", response)

A few things this example demonstrates in practice:

• The tool description explicitly scopes when the model should call it, directly applying the "tighten the description" guidance above
• required: ["account_id"] prevents the model from calling the function without a valid argument
• Temperature is set to 0.1, matching the deterministic-task guidance for anything feeding into backend logic
• The response is parsed defensively with a try/except, since production agent loops should always handle the case where the model replies in plain text instead of a structured call

For a full agent loop, wrap this in a while loop with a max_steps counter, feeding the function's return value back into the message history as a tool role turn before calling generate again.

Common Prompting Mistakes to Avoid

• Mixing prompt formats from other model families ([INST], ChatML) instead of Gemma's native <start_of_turn> template
• Manually inserting the BOS token or hand-serializing tool schemas instead of using apply_chat_template
• Writing vague tool descriptions and expecting reliable function-calling decisions
• Using a single temperature setting across wildly different tasks (extraction vs. creative writing)
• Asking for multi-step conditional logic in one prompt instead of chaining focused calls
• Skipping few-shot examples when strict output formatting (like JSON) is required for backend parsing

Getting these fundamentals right — correct chat template, clear structure, task-appropriate temperature, and precise tool descriptions — resolves the majority of "Gemma isn't following instructions" complaints before you need any deeper fine-tuning.