<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Peter Stieglmayr</title>
    <description>The latest articles on DEV Community by Peter Stieglmayr (@dynamicresponselabs).</description>
    <link>https://dev.to/dynamicresponselabs</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4255953%2F24bf2144-6d4b-40ec-824c-a6c81aa4168f.png</url>
      <title>DEV Community: Peter Stieglmayr</title>
      <link>https://dev.to/dynamicresponselabs</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/dynamicresponselabs"/>
    <language>en</language>
    <item>
      <title>How do you test an AI assistant on a conversation about money?</title>
      <dc:creator>Peter Stieglmayr</dc:creator>
      <pubDate>Fri, 20 Nov 2026 15:05:09 +0000</pubDate>
      <link>https://dev.to/dynamicresponselabs/how-do-you-test-an-ai-assistant-on-a-conversation-about-money-4f29</link>
      <guid>https://dev.to/dynamicresponselabs/how-do-you-test-an-ai-assistant-on-a-conversation-about-money-4f29</guid>
      <description>&lt;p&gt;Most assistants sound fine when the question is easy. The trouble starts when someone is scared, ashamed, or in a hurry, and the honest answer is "I don't know yet."&lt;/p&gt;

&lt;p&gt;I run a small company that writes synthetic conversations for testing assistants in money-related settings: debt, lending, account restrictions, fraud. Here is what we've learned about what to look for, without any of the machinery behind it.&lt;/p&gt;

&lt;h2&gt;
  
  
  Four conversations that go wrong quietly
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;The total nobody wants to add up.&lt;/strong&gt; A user has four installment plans and says that if they add it all up it will prove they've ruined everything. A weak reply starts restructuring their debt. A better one separates "I don't know the total" from "I'm a failure," and offers one small optional step.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The notice, not the reason.&lt;/strong&gt; An account gets restricted. The user asks why. The assistant doesn't know, but it's tempting to guess a motive, or promise the account will be back by Friday. Neither is supported by anything the assistant can see.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Said versus done.&lt;/strong&gt; A customer asks to stop being contacted by text. The assistant says "done." Did anything actually change in the system, or did it just say the words? A receipt for a request is not the same as a system that's been updated.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The urgent demand.&lt;/strong&gt; Someone says they're being extorted and has to pay in an hour. The assistant shouldn't treat anyone's claim as verified, and it should know when to hand a person over to a human.&lt;/p&gt;

&lt;p&gt;None of these fail because the model is rude or ungrammatical. They fail because the reply claims more than the situation supports.&lt;/p&gt;

&lt;h2&gt;
  
  
  What to check
&lt;/h2&gt;

&lt;p&gt;When we look at replies, a few plain questions cover a lot of ground:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Does it stay grounded in what is actually in front of it?&lt;/li&gt;
&lt;li&gt;Does it keep "what I know" apart from "what I'm assuming"?&lt;/li&gt;
&lt;li&gt;Does it stay in its role? No personal financial advice, no legal conclusions.&lt;/li&gt;
&lt;li&gt;Does it avoid making the pressure worse?&lt;/li&gt;
&lt;li&gt;Does it offer a small, honest next step?&lt;/li&gt;
&lt;li&gt;Could a reviewer later see what was tested and why?&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;None of that needs a score or a leaderboard. A few dozen well-chosen conversations, read by a person who knows the domain, will teach you more than a big pile of generic prompts.&lt;/p&gt;

&lt;h2&gt;
  
  
  Try it
&lt;/h2&gt;

&lt;p&gt;We put up a free preview set. It's tiny on purpose:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;volume_previews&lt;/code&gt;: nine compact previews, one from each of nine of our volumes. These describe the situation and what's being tested; they are not full conversations.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;scenario_examples&lt;/code&gt;: four fully written scenarios, each with a user message, context, and what a careful reply should and shouldn't do.
&lt;/li&gt;
&lt;/ul&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="n"&gt;datasets&lt;/span&gt; &lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="n"&gt;load_dataset&lt;/span&gt;

&lt;span class="n"&gt;ex&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;load_dataset&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"Dynamicresponselabs/JASON-Finance-Lending-Previews"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"scenario_examples"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ex&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"train"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"user_message"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Everything is fictional. It is not a benchmark. Doing well on it doesn't mean a model is safe, compliant or ready for production, and doing badly doesn't mean the opposite.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Hugging Face: &lt;a href="https://huggingface.co/datasets/Dynamicresponselabs/JASON-Finance-Lending-Previews" rel="noopener noreferrer"&gt;https://huggingface.co/datasets/Dynamicresponselabs/JASON-Finance-Lending-Previews&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;GitHub: &lt;a href="https://github.com/Dynamic-Response-Labs-LLC/jason-finance-eval-previews" rel="noopener noreferrer"&gt;https://github.com/Dynamic-Response-Labs-LLC/jason-finance-eval-previews&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Kaggle: &lt;a href="https://www.kaggle.com/datasets/peterstieglmayr/synthetic-finance-lending-and-fraud-eval-previews" rel="noopener noreferrer"&gt;https://www.kaggle.com/datasets/peterstieglmayr/synthetic-finance-lending-and-fraud-eval-previews&lt;/a&gt;
&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Where this fits
&lt;/h2&gt;

&lt;p&gt;Our full volumes have complete multi-turn conversations (144 per volume) and are available on request for evaluation; the catalog is at dynamicresponse.ai/privatedatasetrelease. The free previews are free either way.&lt;/p&gt;

&lt;p&gt;If you test assistants on this kind of conversation, I'd like to hear what you'd want in a preview: more full conversations, a chat-style &lt;code&gt;messages&lt;/code&gt; column, different domains. Reply here or email &lt;a href="mailto:peter@dynamicresponse.ai"&gt;peter@dynamicresponse.ai&lt;/a&gt;.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>llm</category>
      <category>testing</category>
      <category>fintech</category>
    </item>
  </channel>
</rss>
