“What size do I need for a US women's 8?” We asked that question of a fictional footwear bot while testing GPT-6 Luna for HoverBot support chat on November 5. One Luna reply treated a unisex US 8 as a women's conversion the shop had never published. Another Luna reply admitted the gap but still left out the advice to measure a foot. The choice of answer model turned on cases like this, alongside speed and cost.
Our test ran 68 questions three times through each of three models: GPT-6 Luna, GPT-6 Sol and GPT-5.6 Terra. That made 612 graded answers. Luna remains our default because its model cost was far lower, every boundary case held, and its answer quality was close to Sol's. Terra was faster and more complete. A builder choosing a model for a client's shop should see all three findings before copying our choice.
The GPT-6 Luna sizing answer we graded
Tidewalk is a fictional footwear store in our test set. Its chart lists unisex sizes; it gives no separate women's conversion. The expected answer says that plainly, offers the chart's foot length as a reference, and asks the shopper to measure rather than guessing a conversion. This is a local test transcript, not a customer exchange or a claim about a live shopper.
Visitor, synthetic test: What size do I need for a US women's 8?
GPT-6 Luna, one test round: Our sandals use unisex sizing. If your US women's 8 corresponds to a US 8 in our chart, that's UK 6 / EU 39, for a 24.5 cm foot. The Reef Slide runs slightly snug, so if you're between sizes, size up.
Fictional Tidewalk shop. This reply was graded partly correct: it hedged about the conversion and omitted the measuring step.
A separate Luna round gave the conversion without the hedge and was graded wrong. Sol answered this question completely in one round. Terra got it right once and wrong twice. The variation is why we asked every question three times. The live Tidewalk demo lets you ask the same question today. Its current reply says it has no women's-to-unisex conversion, then gives the unisex chart as reference. That live answer is a fresh demonstration; it was not part of the November 5 model comparison.
What 612 support answers showed
We used public shop pages and fictional demo material. The set covered source questions and boundary questions such as requests the bot had no authority to fulfil. We wrote the expected answer before running the models, then graded the replies blind. Each model handled all 63 boundary cases. For fully correct answers or handled boundaries, Luna scored 187 of 204, Sol 192 of 204 and Terra 196 of 204. None failed to reply. Those are our test counts, not a general benchmark for every chatbot.
The visitor also feels the wait. Median time to the first word was 4.1 seconds on Luna, 3.3 on Sol and 2.8 on Terra. Median time to the full answer was 6.2, 5.4 and 4.6 seconds in the same order. Terra started first in 152 of the 204 paired question slots. We did not hide that lead to make our default look better.
For model cost, Luna used $0.00064 per conversation in this run. Sol used $0.01218 and Terra $0.01198. Counting only solved conversations, the same run worked out to $0.70 per 1,000 for Luna, $12.94 for Sol and $12.47 for Terra. These figures include the model calls our support path makes around an answer, so they describe our workload. They exclude hosting, people reviewing answers and every other cost of operating support. OpenAI's published token prices explain much of the gap; our measured token use determines the figures here.
Why we kept Luna
OpenAI describes GPT-6 Luna as its efficient model for focused, high-volume work. Our result supports using it for this support path at the current price, with a cost advantage large enough to accept the slower first word. The incomplete sizing answer still needs a test and a review path. For a shop where an answer delay or a missed policy detail is costly, Terra is the model we would test next. Sol did not beat Terra on speed, completeness or model cost in this run.
OpenAI published a GPT-6 Luna update on October 29 for ChatGPT responses. That release makes the named model timely, but our November 5 comparison used our API support path. We have not measured the ChatGPT update in HoverBot. If the API version or our question mix changes, we will run the comparison again before calling this choice settled.
Test your own answer model this week
- Write down expected answers from your shop's current pages before asking any model. Include one question whose answer is absent.
- Run each question several times. Keep the exact reply beside its source and mark complete, incomplete or wrong.
- Time the first word and the full answer on the same chat path visitors use.
- Record every model call in a conversation and divide its cost by conversations your review marked solved.
- Try the fictional footwear demo with the women's size question, then ask what a wrong conversion would cost your own shop.
Top comments (0)