DEV Community

Cover image for Stop Asking Gemma 4 to Just Summarize
Michael Neang
Michael Neang

Posted on

Stop Asking Gemma 4 to Just Summarize

Gemma 4 Challenge: Write about Gemma 4 Submission

This is a submission for the Gemma 4 Challenge: Write About Gemma 4

A small test of using open models to expose uncertainty, not hide it.

Most AI demos start with clean prompts.

Real work usually starts with messy notes.

A stakeholder says the number looks wrong. Someone else says a field changed. Another person says the source file was cleaned. The dashboard owner is out. A manager still needs an update before a leadership meeting.

That is the kind of situation I wanted to test with Gemma 4.

I work around business systems, reporting, data quality, and process handoffs, so this felt familiar. In real reporting work, the dangerous moment is not always when the dashboard breaks. It is when everyone wants an answer before anyone has checked the source.

So I tested one practical question:

Can Gemma 4 do something more useful than summarize messy notes?

More specifically:

Can it separate the mess into what is known, what is assumed, what is risky, and what still needs checking?

That is the difference I cared about.

A summary compresses messy information. A review packet separates it into facts, assumptions, risks, and next checks.

The test setup

I kept the test small on purpose.

I used Google AI Studio with Gemma 4 26B A4B IT. I used the same messy notes twice, with the same model settings: temperature 0.25, thinking level set to High, no tools, and no system instructions.

I left system instructions blank because I wanted the prompt itself to carry the behavior.

This was not a model comparison. It was a prompt-pattern comparison.

The only thing I changed was the prompt.

The scenario was synthetic, but realistic: a weekly operations report is due Monday morning, the totals look lower than expected, a field name changed last week, duplicate rows may have been removed, and the dashboard owner is out.

No private data. No company data. Just the kind of messy report situation that creates pressure before a meeting.

Here were the notes, shortened for the test:

A weekly operations report is due Monday morning.

The report usually refreshes Sunday night.

A field name changed last week.

A stakeholder says totals look lower than usual.

Duplicate rows may have been removed from the source file.

The dashboard owner is out.

Nobody knows yet whether the issue is source data, filter logic, or a definition change.

Prompt 1: ask for a normal summary

First, I asked:

Summarize these notes for the manager and explain what is going on.
Enter fullscreen mode Exit fullscreen mode

The result was not bad.

Baseline summary output in Google AI Studio using Gemma 4 26B A4B IT

Baseline run: same Gemma 4 model, same notes, no system instructions.

Gemma 4 gave me a manager-ready update and a plain-English explanation of why the team should avoid giving final numbers yet. One useful line was:

“Investigation in progress regarding a discrepancy in the weekly totals.”

I could imagine sending a version of that in Slack.

But the summary also showed the weakness of summarization. It used phrases like “red alert,” “flying blind,” and “the numbers look wrong.”

Those phrases made the situation easier to understand, but they also added more certainty and drama than the original notes supported.

The raw notes did not prove the report was wrong. They only said the totals looked lower than usual.

That matters.

A summary can make messy information easier to read while quietly making the situation feel more settled than it actually is.

Prompt 2: ask for a review packet

So I reran the same notes with one change: instead of asking for a summary, I asked for a review packet.

You are an analyst preparing a review packet.

Using only the notes below, return:
1. One-paragraph summary
2. Confirmed facts
3. Assumptions
4. Unverified claims
5. Risks
6. Questions for a human reviewer
7. Suggested next actions
8. What not to conclude yet
9. Final checklist

Rules:
- Do not invent facts.
- Separate facts from assumptions.
- Keep it calm, practical, and concise.
- Help a human decide what to verify next; do not make the final decision.
Enter fullscreen mode Exit fullscreen mode

The output was less polished, but more useful for the decision in front of the team.

It separated confirmed facts from assumptions and unverified claims. For example, it treated “totals look lower than usual” as an unverified claim because that came from a stakeholder report, not from a completed data check.

It also kept the biggest unknown visible:

“The root cause is currently unknown.”

That was the most important part of the output.

In a real reporting issue, people often want to jump straight to the cause: the field change broke it, the source file changed it, the filter logic is wrong.

But in this scenario, none of that had been verified yet.

The strongest section was “What not to conclude yet”:

  • Do not conclude that the data is incorrect.
  • Do not conclude that the field name change or duplicate removal caused the discrepancy.
  • Do not conclude whether the issue is source data, filter logic, or a definition change.

That section changed the output from a status update into a review tool.

Review packet output in Google AI Studio separating confirmed facts, assumptions, unverified claims, and risks

Review-packet run: the useful shift was separating facts, assumptions, risks, and what not to conclude yet.

What changed

Summary vs Review Packet comparison

The better output was not better because it sounded smarter.

It was better because it gave the team a safer shape for the next step.

The summary was better for status communication. The review packet was better for deciding what still needed verification.

The caution: this still does not verify anything

This is the boundary I would not cross.

Gemma 4 did not inspect the report. It did not open the source file. It did not check the dashboard logic. It did not know whether duplicate removal was correct or whether the field name change affected a calculation.

It organized the review path.

That is useful, but it is not the same as verification.

In a real report issue, I would still check the source rows, field mapping, report filters, refresh timing, and metric definition before saying anything final. The review packet does not remove that work. It makes sure I do not skip it.

For messy systems work, I would use this kind of output before the investigation, not after it. It is a way to prepare the human reviewer, not replace them.

The model can turn scattered notes into a checklist for the person doing the investigation.

It cannot do the investigation unless it has access to the actual systems, data, definitions, and business context.

The Review Packet Framework

If someone only copied one thing from this post, I would want it to be this prompt shape.

When the input is messy, uncertain, or time-sensitive, ask the model for:

  1. Summary
  2. Confirmed facts
  3. Assumptions
  4. Unverified claims
  5. Risks
  6. Questions for a human reviewer
  7. Suggested next actions
  8. What not to conclude yet
  9. Final checklist

The most useful parts are not always the summary or the checklist.

For me, the important sections are:

  • assumptions
  • unverified claims
  • risks
  • what not to conclude yet

Those sections force the model to keep uncertainty visible instead of hiding it inside a clean paragraph.

That is what made the review packet more useful than the summary.

Why I used Gemma 4 this way

I used Gemma 4 this way because messy notes are a real workload.

Reports, support handoffs, incident updates, and project notes rarely arrive in perfect form. The useful question is not only whether a model can write about them. It is whether it can preserve the uncertainty inside them.

That fits the part of Gemma 4 I care about most: not just text generation, but reasoning over messy context and producing something structured enough for a human to use.

For this test, I used the 26B A4B instruction model because I cared more about the reasoning structure than running the smallest possible model. If I were building a lightweight offline helper, I would test a smaller Gemma 4 model next.

Final takeaway

I would still ask Gemma 4 for summaries.

But after this test, I would be more careful about when a summary is enough.

If the situation is messy, time-sensitive, or full of unverified claims, I would rather ask for a review packet.

The useful output is not always the cleanest paragraph. Sometimes it is the output that says:

  • here is what we know
  • here is what we are assuming
  • here is what could go wrong
  • here is what not to conclude yet

That is the pattern I would reuse.

Gemma 4 did not remove the mess.

It made the mess reviewable.

Top comments (0)