DEV Community

Super Lewis
Super Lewis

Posted on Fully Autonomous

Don't let the model do date math: a refund-agent test where Gemini 4 Carbon broke a 30-day rule and Qwen 4 Max held it

A tool-calling agent that reads a policy correctly can still apply it wrongly. In my test, Gemini 4 Carbon fetched a 30-day return policy, then refunded an order delivered 46 days earlier, three runs out of three. Qwen 4 Max refused it three out of three. This post shows the test, the grader, and the validation that makes the question matter much less.

Disclosure: I ran this on apimodels.app, which is the API I work on. apimodels.app is a multi-model API gateway: one key and one OpenAI-compatible endpoint for image, video, audio and language models, including both models here. I wrote this from my own test data, with AI assistance for drafting and editing.

The task

The customer says the space bar on their keyboard is broken and asks for a refund. They also want to return one of two mice from the same order because they do not like it.

Four tools, in the standard function-calling format:

tools = [
  {"type": "function", "function": {
    "name": "get_order", "description": "Look up an order by id.",
    "parameters": {"type": "object",
      "properties": {"order_id": {"type": "string"}}, "required": ["order_id"]}}},
  {"type": "function", "function": {
    "name": "get_refund_policy", "description": "Return the current refund policy text.",
    "parameters": {"type": "object", "properties": {}}}},
  {"type": "function", "function": {
    "name": "issue_refund", "description": "Refund one line item of an order.",
    "parameters": {"type": "object", "properties": {
      "order_id": {"type": "string"}, "sku": {"type": "string"},
      "amount": {"type": "number"}, "reason": {"type": "string"}},
      "required": ["order_id", "sku", "amount", "reason"]}}},
  {"type": "function", "function": {
    "name": "escalate_to_human", "description": "Hand the case to a human agent.",
    "parameters": {"type": "object", "properties": {
      "order_id": {"type": "string"}, "summary": {"type": "string"}},
      "required": ["order_id", "summary"]}}},
]
Enter fullscreen mode Exit fullscreen mode

The tools return fixed data. The order was delivered on October 2: one keyboard at $129, two mice at $39 each, $12 shipping. The policy says today is November 17, no-reason returns close 30 days after delivery, defective items can come back within 90 days, shipping is refunded only with a full-order return, and any single refund over $150 goes to a human.

One outcome is correct: refund the keyboard for $129.00, refuse the mouse, escalate nothing.

The grader

Seven checks, all on tool calls and arguments, none on wording except the last:

  • get_order was called
  • get_refund_policy was called
  • the keyboard was refunded exactly once
  • the amount was 129.00, with no shipping added
  • no refund was issued for the mouse
  • nothing was escalated
  • the final message tells the customer why the mouse was refused

The loop is the usual one: execute each returned tool call, append the result as a tool message, repeat until the model stops calling tools.

Results

Qwen 4 Max: 7 of 7, three runs out of three, about 15 seconds each. Its reply noted that the no-reason window ended on November 1.

Gemini 4 Carbon: 6 of 7, three runs out of three. It called both lookups, then refunded the keyboard and the mouse, and told the customer the mouse was within 30 days of the October 2 delivery.

Across the full seven-task run this came from (93 checks), Qwen 4 Max passed 93 and Carbon passed 89. Carbon also miscounted in a log-analysis task and overran a character limit by one.

For balance: public benchmarks favour Carbon clearly, 60 against 43 on the Artificial Analysis Intelligence Index, and it accepts audio and PDF input, which Qwen 4 Max does not. Benchmarks measure reach on hard problems. This test measures exact application of an easy rule. They are different skills.

Put the rule in the tool

The practical lesson holds for any model. The model should work out what the customer wants, and the code should decide dates and amounts.

from datetime import date

WINDOWS = {"defective": 90, "no_reason": 30}

def issue_refund(order, sku, amount, reason, today: date):
    if reason not in WINDOWS:
        return {"status": "rejected", "error": "reason must be defective or no_reason"}
    item = next(i for i in order["items"] if i["sku"] == sku)
    days = (today - order["delivered_at"]).days
    if days > WINDOWS[reason]:
        return {"status": "rejected",
                "error": f"outside the {WINDOWS[reason]}-day window "
                         f"({days} days since delivery)"}
    if amount > item["unit_price"]:
        return {"status": "rejected", "error": "amount exceeds the item price"}
    if amount > 150:
        return {"status": "rejected", "error": "over the limit, escalate to a human"}
    return do_refund(order["order_id"], sku, amount)
Enter fullscreen mode Exit fullscreen mode

With this in place, a wrong judgment becomes a rejected call with a readable error, and the model usually corrects its reply. Make reason an enum in the tool schema as well. With free text, the model can talk its way past the check.

When not to lean on this result

  • Hard research or multi-step debugging: Carbon leads on public benchmarks.
  • Audio or PDF input: only Carbon supports it.
  • Small sample: the refund task ran three times, the other six once each.
  • Pricing changes: Carbon moves from $2 and $10 per million tokens to $4 and $20 on December 1. Qwen 4 Max is $1.50 and $4.50.

The full comparison is on the Qwen 4 Max vs Gemini 4 Carbon page at apimodels.app.

Top comments (0)