<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Shashank Agarwal</title>
    <description>The latest articles on DEV Community by Shashank Agarwal (@imshashank).</description>
    <link>https://dev.to/imshashank</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F1348998%2F681a6e08-bfe7-46f1-90ad-bb1a45963e91.jpeg</url>
      <title>DEV Community: Shashank Agarwal</title>
      <link>https://dev.to/imshashank</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/imshashank"/>
    <language>en</language>
    <item>
      <title>HTTP Success Is Not Agent Success: A Reproducible Python Acceptance Lab</title>
      <dc:creator>Shashank Agarwal</dc:creator>
      <pubDate>Thu, 29 Oct 2026 18:45:44 +0000</pubDate>
      <link>https://dev.to/imshashank/http-success-is-not-agent-success-a-reproducible-python-acceptance-lab-68j</link>
      <guid>https://dev.to/imshashank/http-success-is-not-agent-success-a-reproducible-python-acceptance-lab-68j</guid>
      <description>&lt;p&gt;&lt;em&gt;Disclosure: This article and its code are AI-generated. The results below were verified by executing the local synthetic test. No LLM, live customer system or paid API was called by the experiment.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;A booking agent returns a clean response: the request succeeded, the JSON contains a booking identifier, and the assistant tells the user that the appointment is confirmed. There is still a question neither the status code nor the JSON shape can answer: did the system book the appointment the user actually authorized?&lt;/p&gt;

&lt;p&gt;This is a practical evaluation problem for developers building agents around APIs. Infrastructure checks remain necessary, but a request can succeed at one boundary while the larger task fails at another. A booking for the wrong day is not made correct by a well-formed receipt.&lt;/p&gt;

&lt;p&gt;We can examine that difference without asking a language model to judge its own work. This article builds a small, deterministic Python lab with explicit input, simulated state and acceptance rules. You can run it locally, inspect every case and replace its fixtures with evidence from your own staging workflow.&lt;/p&gt;

&lt;h2&gt;
  
  
  What the experiment actually measures
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Quick answer:&lt;/strong&gt; In this hand-selected set of 12 synthetic fixtures, an HTTP-2xx-only rule accepted eight of nine known-invalid outcomes. A booking-shape-only rule accepted seven. There are also two valid outcomes and one outcome whose state cannot be observed. These counts demonstrate specific weaknesses in the acceptance rules; they do not estimate a production failure rate or the accuracy of any model or product.&lt;/p&gt;

&lt;p&gt;The distinction matters because the fixtures were deliberately chosen as counterexamples. There is no random traffic sample, customer cohort or benchmark leaderboard here. The chart describes these files and these rules. If you change the fixtures, the counts may change.&lt;/p&gt;

&lt;p&gt;The task-level rule is also deliberately simple. It is an oracle for a controlled environment where we supply authoritative simulated state. It is not an independently calibrated universal evaluator. Its matching labels are checked for fixture consistency, not offered as evidence that it can classify arbitrary production conversations correctly.&lt;/p&gt;

&lt;h2&gt;
  
  
  Start with a contract a reader can challenge
&lt;/h2&gt;

&lt;p&gt;The synthetic user wants an appointment on a particular day for a particular customer. The contract permits two successful outcomes:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;If a slot is available, exactly one matching booking exists, the required confirmation has been supplied, and the assistant accurately reports completion.&lt;/li&gt;
&lt;li&gt;If no slot is available, no booking is created and the assistant accurately reports that limitation.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The second branch prevents an evaluator from rewarding an agent simply for making a booking every time. The user asked for appropriate behavior, not an unconditional database write.&lt;/p&gt;

&lt;p&gt;The lab uses invented customer and booking identifiers and a future test date. They identify fixtures, not real people or events. The available-slot condition, confirmation flag and final state are supplied directly by the fixture. A real integration would need to establish those observations from appropriately authorized records.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Layer&lt;/th&gt;
&lt;th&gt;Question&lt;/th&gt;
&lt;th&gt;Evidence in this lab&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Request result&lt;/td&gt;
&lt;td&gt;Did the operation return a success status?&lt;/td&gt;
&lt;td&gt;Synthetic HTTP status&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Output shape&lt;/td&gt;
&lt;td&gt;Does the body look like a booking response?&lt;/td&gt;
&lt;td&gt;Required string fields and a status label&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Task outcome&lt;/td&gt;
&lt;td&gt;Does established state satisfy the request?&lt;/td&gt;
&lt;td&gt;Simulated booking records, confirmation and availability&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;User explanation&lt;/td&gt;
&lt;td&gt;Does the response describe the established result?&lt;/td&gt;
&lt;td&gt;A controlled reply classification&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Observability&lt;/td&gt;
&lt;td&gt;Is enough state available to decide?&lt;/td&gt;
&lt;td&gt;Explicit &lt;code&gt;None&lt;/code&gt; for unavailable evidence&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;These layers complement each other. A consumer still needs to parse the response and handle transport errors. The mistake is treating an earlier layer as sufficient evidence for every later one.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why a correct HTTP status can coexist with a failed task
&lt;/h2&gt;

&lt;p&gt;&lt;a href="https://www.rfc-editor.org/rfc/rfc9110.html#section-15.3.1" rel="noopener noreferrer"&gt;RFC 9110&lt;/a&gt; defines HTTP success in relation to the request and method semantics. That does not make a successful API operation proof that an agent selected the correct day, customer or action. If the client submits the wrong date and the server creates exactly that booking, the server may have fulfilled its interface contract while the agent failed the user's task.&lt;/p&gt;

&lt;p&gt;The distinction is even clearer for asynchronous work. A &lt;code&gt;202&lt;/code&gt; response concerns acceptance for processing, not proof that processing has finished. An assistant that announces completion immediately can overstate the available evidence. The lab includes this case with a queued response and no established booking.&lt;/p&gt;

&lt;p&gt;Output validation solves a different problem. The shape rule checks that &lt;code&gt;booking_id&lt;/code&gt;, &lt;code&gt;status&lt;/code&gt;, &lt;code&gt;date&lt;/code&gt; and &lt;code&gt;customer_id&lt;/code&gt; are strings and that the status belongs to a small allowed set. It does not compare the returned date with the requested date. It cannot establish that a receipt corresponds to a real state change. A schema is still useful; it simply answers a narrower question.&lt;/p&gt;

&lt;h2&gt;
  
  
  Keep the counterexamples inspectable
&lt;/h2&gt;

&lt;p&gt;The complete set includes a correct booking, an appropriate unavailable-slot explanation, a wrong day, missing confirmation, a duplicate, queued work described as complete, a receipt with no operation, a wrong customer, a cancellation described as complete, a friendly message without a useful result, a known dependency failure and an unobservable outcome.&lt;/p&gt;

&lt;p&gt;There is an intentional asymmetry here. The unavailable-slot explanation is a valid task outcome, but it does not look like a booking object. A shape checker designed only for bookings rejects that valid branch. In a real API, a tagged union or separate response type could describe both outcomes. It would still need task-specific checks to establish whether the right branch occurred.&lt;/p&gt;

&lt;p&gt;The unknown-state fixture is not silently labelled a failure. A timeout does not, by itself, tell us whether a state-changing operation happened. If the state record is unavailable, the task evaluator returns &lt;code&gt;UNKNOWN&lt;/code&gt;. An operational report can separately count the execution problem. Mixing those classifications loses the distinction between a demonstrated wrong result and missing evidence.&lt;/p&gt;

&lt;p&gt;This is also why retry behavior needs a contract. After an uncertain write, another attempt may create a duplicate. Use the provider's actual status lookup and replay guarantees where they exist; this local example does not establish them.&lt;/p&gt;

&lt;h2&gt;
  
  
  Run the complete experiment
&lt;/h2&gt;

&lt;p&gt;Save the following program as &lt;code&gt;acceptance_lab.py&lt;/code&gt;. The acceptance checks and CSV/JSON export use only Python's standard library. Plotting is optional and imports Matplotlib only when requested.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="sh"&gt;"""&lt;/span&gt;&lt;span class="s"&gt;Controlled acceptance-rule demonstration. No network, LLM or customer data.

Run: python3 acceptance_lab.py --out results
Optional figure: python3 acceptance_lab.py --out results --plot
&lt;/span&gt;&lt;span class="sh"&gt;"""&lt;/span&gt;
&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;argparse&lt;/span&gt;
&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;copy&lt;/span&gt;
&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;csv&lt;/span&gt;
&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;json&lt;/span&gt;
&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;platform&lt;/span&gt;
&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;datetime&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;datetime&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;timezone&lt;/span&gt;
&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;pathlib&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;Path&lt;/span&gt;

&lt;span class="n"&gt;REQUEST&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;date&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;2030-04-18&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;customer_id&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;synthetic-customer&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="n"&gt;BOOKING&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;booking_id&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;synthetic-booking&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;status&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;booked&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="o"&gt;**&lt;/span&gt;&lt;span class="n"&gt;REQUEST&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;


&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;fixture&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="o"&gt;**&lt;/span&gt;&lt;span class="n"&gt;changes&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;case&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;name&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;http&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;200&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;response&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;copy&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;deepcopy&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;BOOKING&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
            &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;bookings&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;copy&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;deepcopy&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;BOOKING&lt;/span&gt;&lt;span class="p"&gt;)],&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;reply&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;confirmed&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
            &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;confirmed&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="bp"&gt;True&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;available&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="bp"&gt;True&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;expected&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;PASS&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
    &lt;span class="k"&gt;case&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;update&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;changes&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="k"&gt;case&lt;/span&gt;


&lt;span class="n"&gt;CASES&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;
    &lt;span class="nf"&gt;fixture&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;correct_booking&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
    &lt;span class="nf"&gt;fixture&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;no_slot_explained&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;response&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;availability&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;none&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt;
            &lt;span class="n"&gt;bookings&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;[],&lt;/span&gt; &lt;span class="n"&gt;reply&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;unavailable&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;confirmed&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="bp"&gt;False&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;available&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="bp"&gt;False&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
    &lt;span class="nf"&gt;fixture&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;wrong_day&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;response&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="o"&gt;**&lt;/span&gt;&lt;span class="n"&gt;BOOKING&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;date&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;2030-04-19&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt;
            &lt;span class="n"&gt;bookings&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;[{&lt;/span&gt;&lt;span class="o"&gt;**&lt;/span&gt;&lt;span class="n"&gt;BOOKING&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;date&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;2030-04-19&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;}],&lt;/span&gt; &lt;span class="n"&gt;expected&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;FAIL&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
    &lt;span class="nf"&gt;fixture&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;missing_confirmation&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;confirmed&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="bp"&gt;False&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;expected&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;FAIL&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
    &lt;span class="nf"&gt;fixture&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;duplicate_booking&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;bookings&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;BOOKING&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="o"&gt;**&lt;/span&gt;&lt;span class="n"&gt;BOOKING&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;booking_id&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;second&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;}],&lt;/span&gt; &lt;span class="n"&gt;expected&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;FAIL&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
    &lt;span class="nf"&gt;fixture&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;queued_but_claimed_done&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;http&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;202&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;response&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="o"&gt;**&lt;/span&gt;&lt;span class="n"&gt;BOOKING&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;status&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;queued&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt;
            &lt;span class="n"&gt;bookings&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;[],&lt;/span&gt; &lt;span class="n"&gt;expected&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;FAIL&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
    &lt;span class="nf"&gt;fixture&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;receipt_without_operation&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;bookings&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;[],&lt;/span&gt; &lt;span class="n"&gt;expected&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;FAIL&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
    &lt;span class="nf"&gt;fixture&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;wrong_customer&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;response&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="o"&gt;**&lt;/span&gt;&lt;span class="n"&gt;BOOKING&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;customer_id&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;another-synthetic-customer&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt;
            &lt;span class="n"&gt;bookings&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;[{&lt;/span&gt;&lt;span class="o"&gt;**&lt;/span&gt;&lt;span class="n"&gt;BOOKING&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;customer_id&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;another-synthetic-customer&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;}],&lt;/span&gt; &lt;span class="n"&gt;expected&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;FAIL&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
    &lt;span class="nf"&gt;fixture&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;cancelled_but_claimed_done&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;response&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="o"&gt;**&lt;/span&gt;&lt;span class="n"&gt;BOOKING&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;status&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;cancelled&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt;
            &lt;span class="n"&gt;bookings&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;[],&lt;/span&gt; &lt;span class="n"&gt;expected&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;FAIL&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
    &lt;span class="nf"&gt;fixture&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;friendly_message_only&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;response&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;message&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Done!&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt; &lt;span class="n"&gt;bookings&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;[],&lt;/span&gt; &lt;span class="n"&gt;expected&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;FAIL&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
    &lt;span class="nf"&gt;fixture&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;known_dependency_failure&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;http&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;503&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;response&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;error&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;unavailable&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt;
            &lt;span class="n"&gt;bookings&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;[],&lt;/span&gt; &lt;span class="n"&gt;reply&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;error&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;expected&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;FAIL&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
    &lt;span class="nf"&gt;fixture&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;outcome_not_observable&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;http&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;504&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;response&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="bp"&gt;None&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;bookings&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="bp"&gt;None&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
            &lt;span class="n"&gt;available&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="bp"&gt;None&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;reply&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;unknown&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;expected&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;UNKNOWN&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
&lt;span class="p"&gt;]&lt;/span&gt;


&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;http_success&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;case&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="mi"&gt;200&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;=&lt;/span&gt; &lt;span class="k"&gt;case&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;http&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="mi"&gt;300&lt;/span&gt;


&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;booking_shape&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;case&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;body&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;case&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;response&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
    &lt;span class="n"&gt;fields&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;booking_id&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;status&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;date&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;customer_id&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="nf"&gt;return &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;isinstance&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;body&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;dict&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
            &lt;span class="ow"&gt;and&lt;/span&gt; &lt;span class="nf"&gt;all&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;isinstance&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;body&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;key&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;key&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;fields&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
            &lt;span class="ow"&gt;and&lt;/span&gt; &lt;span class="n"&gt;body&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;status&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;booked&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;queued&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;cancelled&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;})&lt;/span&gt;


&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;task_contract&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;case&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="sh"&gt;"""&lt;/span&gt;&lt;span class="s"&gt;Oracle for this controlled fixture, NOT a production truth service.&lt;/span&gt;&lt;span class="sh"&gt;"""&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="k"&gt;case&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;bookings&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="ow"&gt;is&lt;/span&gt; &lt;span class="bp"&gt;None&lt;/span&gt; &lt;span class="ow"&gt;or&lt;/span&gt; &lt;span class="k"&gt;case&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;available&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="ow"&gt;is&lt;/span&gt; &lt;span class="bp"&gt;None&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;UNKNOWN&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="k"&gt;case&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;available&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]:&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;PASS&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="k"&gt;case&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;bookings&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="ow"&gt;and&lt;/span&gt; &lt;span class="k"&gt;case&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;reply&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;unavailable&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;FAIL&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="k"&gt;case&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;confirmed&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="ow"&gt;or&lt;/span&gt; &lt;span class="nf"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;case&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;bookings&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt; &lt;span class="o"&gt;!=&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;FAIL&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
    &lt;span class="n"&gt;booking&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;case&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;bookings&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;][&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
    &lt;span class="n"&gt;matches&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;all&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;booking&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;key&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="n"&gt;value&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;key&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;value&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;REQUEST&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;items&lt;/span&gt;&lt;span class="p"&gt;())&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;PASS&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;matches&lt;/span&gt; &lt;span class="ow"&gt;and&lt;/span&gt; &lt;span class="n"&gt;booking&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;status&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;booked&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt; &lt;span class="ow"&gt;and&lt;/span&gt; &lt;span class="k"&gt;case&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;reply&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;confirmed&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;FAIL&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;


&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;run&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;out&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;plot&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="bp"&gt;False&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;out&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;mkdir&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;parents&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="bp"&gt;True&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;exist_ok&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="bp"&gt;True&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;rows&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[]&lt;/span&gt;
    &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="k"&gt;case&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;CASES&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;decision&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;task_contract&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;case&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="k"&gt;assert&lt;/span&gt; &lt;span class="n"&gt;decision&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="k"&gt;case&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;expected&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;case&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;name&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="n"&gt;decision&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="n"&gt;rows&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;append&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;case&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="k"&gt;case&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;name&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;expected&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="k"&gt;case&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;expected&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;
                     &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;http_2xx_accepts&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nf"&gt;http_success&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;case&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
                     &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;shape_accepts&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nf"&gt;booking_shape&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;case&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;task_decision&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;decision&lt;/span&gt;&lt;span class="p"&gt;})&lt;/span&gt;
    &lt;span class="n"&gt;invalid&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;row&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;row&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;rows&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;expected&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;FAIL&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
    &lt;span class="n"&gt;summary&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;fixtures&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nf"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;rows&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;valid&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nf"&gt;sum&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;expected&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;PASS&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;rows&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
               &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;invalid&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nf"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;invalid&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;unknown&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nf"&gt;sum&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;expected&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;UNKNOWN&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;rows&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
               &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;false_acceptances&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;HTTP 2xx only&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nf"&gt;sum&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;http_2xx_accepts&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;invalid&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
                                     &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Booking shape only&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nf"&gt;sum&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;shape_accepts&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;invalid&lt;/span&gt;&lt;span class="p"&gt;)},&lt;/span&gt;
               &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;scope&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Hand-selected synthetic counterexamples; not production frequency or model accuracy.&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
               &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;python&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;platform&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;python_version&lt;/span&gt;&lt;span class="p"&gt;(),&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;run_utc&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;datetime&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;now&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;timezone&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;utc&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;isoformat&lt;/span&gt;&lt;span class="p"&gt;()}&lt;/span&gt;
    &lt;span class="nf"&gt;with &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;out&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;cases.csv&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;open&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;w&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;newline&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;""&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;handle&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;writer&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;csv&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;DictWriter&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;handle&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;fieldnames&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nf"&gt;list&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;rows&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;]));&lt;/span&gt; &lt;span class="n"&gt;writer&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;writeheader&lt;/span&gt;&lt;span class="p"&gt;();&lt;/span&gt; &lt;span class="n"&gt;writer&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;writerows&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;rows&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;out&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;summary.json&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;write_text&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;dumps&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;summary&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;indent&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
    &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;out&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;fixtures.json&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;write_text&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;dumps&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;CASES&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;indent&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;plot&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;matplotlib&lt;/span&gt;
        &lt;span class="n"&gt;matplotlib&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;use&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Agg&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;matplotlib.pyplot&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;plt&lt;/span&gt;
        &lt;span class="n"&gt;fig&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;ax&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;plt&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;subplots&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;figsize&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;10&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mf"&gt;5.6&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="n"&gt;dpi&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;160&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="n"&gt;labels&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;values&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;zip&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;summary&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;false_acceptances&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;].&lt;/span&gt;&lt;span class="nf"&gt;items&lt;/span&gt;&lt;span class="p"&gt;())&lt;/span&gt;
        &lt;span class="n"&gt;bars&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;ax&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;barh&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;labels&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;values&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;color&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;#2563a6&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;#c16b22&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="n"&gt;height&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;bar&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;value&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="nf"&gt;zip&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;bars&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;values&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
            &lt;span class="n"&gt;ax&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;text&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;value&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="mi"&gt;12&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;bar&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get_y&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;bar&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get_height&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;value&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt; of &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="nf"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;invalid&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;va&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;center&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;fontsize&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;13&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="n"&gt;ax&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;set_xlim&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nf"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;invalid&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt; &lt;span class="n"&gt;ax&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;set_xticks&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;range&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;invalid&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
        &lt;span class="n"&gt;ax&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;set_xlabel&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Known-invalid fixtures accepted by the rule&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;fontsize&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;12&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="n"&gt;ax&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;set_title&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Success status and output shape missed task failures&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;loc&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;left&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;fontsize&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;15&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;pad&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;22&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="n"&gt;ax&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;spines&lt;/span&gt;&lt;span class="p"&gt;[[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;top&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;right&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;left&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]].&lt;/span&gt;&lt;span class="nf"&gt;set_visible&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="bp"&gt;False&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="n"&gt;ax&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;tick_params&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;axis&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;y&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;length&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;labelsize&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;12&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="n"&gt;ax&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;grid&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;axis&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;x&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;alpha&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="mi"&gt;18&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt; &lt;span class="n"&gt;ax&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;set_axisbelow&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="bp"&gt;True&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="n"&gt;fig&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;text&lt;/span&gt;&lt;span class="p"&gt;(.&lt;/span&gt;&lt;span class="mi"&gt;02&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="mi"&gt;02&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Controlled synthetic lab: 12 fixtures = 2 valid, 9 invalid, 1 unknown.&lt;/span&gt;&lt;span class="se"&gt;\n&lt;/span&gt;&lt;span class="s"&gt;Hand-selected counterexamples; no LLM calls, customer data or production-rate estimate.&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;fontsize&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;10&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;color&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;#374151&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="n"&gt;fig&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;tight_layout&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;rect&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="mi"&gt;13&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
        &lt;span class="n"&gt;fig&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;savefig&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;out&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;false-acceptances.png&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;facecolor&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;white&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="n"&gt;fig&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;savefig&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;out&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;false-acceptances.svg&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;facecolor&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;white&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="n"&gt;plt&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;close&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;fig&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;dumps&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;summary&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;indent&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;


&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;__name__&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;__main__&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;parser&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;argparse&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;ArgumentParser&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;description&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;__doc__&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;parser&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;add_argument&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;--out&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;type&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;Path&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;default&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nc"&gt;Path&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;results&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
    &lt;span class="n"&gt;parser&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;add_argument&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;--plot&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;action&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;store_true&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;args&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;parser&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;parse_args&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
    &lt;span class="nf"&gt;run&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;args&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;out&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;args&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;plot&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Run it in a clean directory:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;python3 acceptance_lab.py &lt;span class="nt"&gt;--out&lt;/span&gt; results
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The run writes three inspectable artifacts: &lt;code&gt;fixtures.json&lt;/code&gt; contains every input and expected label, &lt;code&gt;cases.csv&lt;/code&gt; contains each rule's decisions, and &lt;code&gt;summary.json&lt;/code&gt; records the counts, Python version and execution timestamp. The local run used for this article completed on October 29, 2026, with Python 3.14.2. There are no random seeds, model versions or remote services involved.&lt;/p&gt;

&lt;p&gt;To reproduce the figure in an isolated environment:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;python3 &lt;span class="nt"&gt;-m&lt;/span&gt; venv .venv
.venv/bin/python &lt;span class="nt"&gt;-m&lt;/span&gt; pip &lt;span class="nb"&gt;install &lt;/span&gt;matplotlib
.venv/bin/python acceptance_lab.py &lt;span class="nt"&gt;--out&lt;/span&gt; results &lt;span class="nt"&gt;--plot&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This also writes PNG and SVG versions of the figure. The recorded plotting environment used Matplotlib 3.11.2. The acceptance counts do not depend on that plotting library. A different plotting version can change rendering without changing the fixture decisions.&lt;/p&gt;

&lt;h2&gt;
  
  
  Read the decisions before reading the chart
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Synthetic case&lt;/th&gt;
&lt;th&gt;Expected task result&lt;/th&gt;
&lt;th&gt;HTTP 2xx accepts&lt;/th&gt;
&lt;th&gt;Booking shape accepts&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Correct booking&lt;/td&gt;
&lt;td&gt;PASS&lt;/td&gt;
&lt;td&gt;Yes&lt;/td&gt;
&lt;td&gt;Yes&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;No slot, accurately explained&lt;/td&gt;
&lt;td&gt;PASS&lt;/td&gt;
&lt;td&gt;Yes&lt;/td&gt;
&lt;td&gt;No&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Wrong day&lt;/td&gt;
&lt;td&gt;FAIL&lt;/td&gt;
&lt;td&gt;Yes&lt;/td&gt;
&lt;td&gt;Yes&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Required confirmation missing&lt;/td&gt;
&lt;td&gt;FAIL&lt;/td&gt;
&lt;td&gt;Yes&lt;/td&gt;
&lt;td&gt;Yes&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Duplicate booking&lt;/td&gt;
&lt;td&gt;FAIL&lt;/td&gt;
&lt;td&gt;Yes&lt;/td&gt;
&lt;td&gt;Yes&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Queued, but claimed complete&lt;/td&gt;
&lt;td&gt;FAIL&lt;/td&gt;
&lt;td&gt;Yes&lt;/td&gt;
&lt;td&gt;Yes&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Receipt without an operation&lt;/td&gt;
&lt;td&gt;FAIL&lt;/td&gt;
&lt;td&gt;Yes&lt;/td&gt;
&lt;td&gt;Yes&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Wrong customer&lt;/td&gt;
&lt;td&gt;FAIL&lt;/td&gt;
&lt;td&gt;Yes&lt;/td&gt;
&lt;td&gt;Yes&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Cancelled, but claimed complete&lt;/td&gt;
&lt;td&gt;FAIL&lt;/td&gt;
&lt;td&gt;Yes&lt;/td&gt;
&lt;td&gt;Yes&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Friendly message only&lt;/td&gt;
&lt;td&gt;FAIL&lt;/td&gt;
&lt;td&gt;Yes&lt;/td&gt;
&lt;td&gt;No&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Known dependency failure&lt;/td&gt;
&lt;td&gt;FAIL&lt;/td&gt;
&lt;td&gt;No&lt;/td&gt;
&lt;td&gt;No&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Outcome not observable&lt;/td&gt;
&lt;td&gt;UNKNOWN&lt;/td&gt;
&lt;td&gt;No&lt;/td&gt;
&lt;td&gt;No&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fx9a04y0mqvddjxjg4pwx.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fx9a04y0mqvddjxjg4pwx.png" alt="False acceptances in nine known-invalid synthetic fixtures: HTTP 2xx accepts eight; booking shape accepts seven." width="" height=""&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Figure: false acceptances among the nine known-invalid, hand-selected synthetic fixtures. HTTP 2xx accepts eight; the booking-shape rule accepts seven. This is a counterexample demonstration, not an estimate of customer traffic, production reliability or model accuracy. The one unknown outcome is excluded from this denominator.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;The wrong-day case illustrates an important boundary: the returned object passes the shape rule because every required field has an allowed type. Only a comparison with the request detects the date mismatch. Missing confirmation illustrates another boundary: a correct-looking final state cannot retroactively establish authorization. The duplicate fixture shows why checking that a matching record exists is weaker than checking that exactly one exists for this operation.&lt;/p&gt;

&lt;p&gt;The cancelled and queued fixtures deliberately pass a permissive shape rule. Tightening the schema to require &lt;code&gt;status == "booked"&lt;/code&gt; would catch those two cases. That is a useful improvement; it would not catch a wrong day, wrong customer, duplicate or fabricated receipt. You can change the rule and rerun the table to see which defects remain.&lt;/p&gt;

&lt;p&gt;The task rule's agreement with the expected column is unsurprising: both were written for this controlled fixture set. The assertion catches an accidental mismatch while editing the lab. It does not validate the oracle independently or establish generalization to unseen tasks. Keep that distinction visible when reporting evaluation results.&lt;/p&gt;

&lt;h2&gt;
  
  
  Replace simulated state with evidence from one staging workflow
&lt;/h2&gt;

&lt;p&gt;The first production adaptation should be narrow. Choose one operation whose desired result you can verify, such as creating a test appointment. Keep the user goal, permitted actions and observation deadline explicit. Avoid starting with a universal score for every conversation.&lt;/p&gt;

&lt;p&gt;Record the requested date and customer reference before execution, together with the authorization or confirmation required by that workflow. After the action, use a documented read path to establish the resulting state. An echoed response is not an independent observation: if the agent and evaluator both trust the same invented receipt, they can agree about a booking that never existed.&lt;/p&gt;

&lt;p&gt;For asynchronous work, use the provider's documented job state and terminal outcomes. Decide how long the test waits and what happens if observation remains incomplete. A pending operation should not become a completed task simply because a polling deadline elapsed. Preserve an unresolved category and reconcile later when the system exposes enough evidence.&lt;/p&gt;

&lt;p&gt;Adapt the duplicate check to the scope of the operation. A customer can legitimately have several appointments. A real test should look for unintended additional records caused by this action, using an operation identifier, a before/after observation or another supported reconciliation method. The fixture's single-record list represents that isolated operation; it is not a rule that each customer may own only one booking.&lt;/p&gt;

&lt;p&gt;The toy reply labels also need replacement. Here, &lt;code&gt;confirmed&lt;/code&gt;, &lt;code&gt;unavailable&lt;/code&gt;, &lt;code&gt;error&lt;/code&gt; and &lt;code&gt;unknown&lt;/code&gt; are controlled classifications. A real assistant writes natural language. For an initial integration, prefer explicit structured completion fields backed by state evidence. If a model grades the wording, calibrate it against independently reviewed examples, including ambiguous cases and plausible but false completion claims. Do not reuse a model's explanation as the sole evidence that its own action succeeded.&lt;/p&gt;

&lt;h2&gt;
  
  
  Preserve enough context to debug a disagreement
&lt;/h2&gt;

&lt;p&gt;When the status check passes and the task check fails, developers need a path from that decision back to the relevant operation. &lt;a href="https://opentelemetry.io/docs/concepts/signals/traces/" rel="noopener noreferrer"&gt;OpenTelemetry's trace model&lt;/a&gt; describes linked spans for units of work. In an instrumented workflow, the useful connection is between the evaluated run and the spans or operation identifiers that explain its actions.&lt;/p&gt;

&lt;p&gt;A compact evaluation record can keep these fields:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"run_id"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"synthetic-run-003"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"contract_version"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"booking-v1"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"fixture"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"wrong_day"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"transport_check"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"PASS"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"shape_check"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"PASS"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"task_check"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"FAIL"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"reason"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"requested_date_does_not_match_established_state"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"evidence_kind"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"synthetic_fixture"&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;In a real system, replace the fixture reference with appropriate evidence references and record when they were observed. Keep authorization evidence, state evidence and explanatory text distinguishable. They answer different questions. Apply the system's existing access and retention controls rather than copying full customer payloads into every evaluation record.&lt;/p&gt;

&lt;h2&gt;
  
  
  Use counterexamples as a release tool
&lt;/h2&gt;

&lt;p&gt;This lab is useful as a small regression suite, not as a production success dashboard. Start with the known counterexamples, then add failures found in staging or support investigations after removing private data. Each addition should explain which acceptance rule it challenges and what observation establishes the expected result.&lt;/p&gt;

&lt;p&gt;For a proposed evaluator change, compare decisions case by case. A new rule that rejects more outputs may reduce false acceptance while also rejecting valid unavailable-slot responses. Report both effects. Keep unknown outcomes separate so an apparent improvement cannot be manufactured by moving difficult cases out of the measured set without explanation.&lt;/p&gt;

&lt;p&gt;If you later measure production reliability, define the sampling frame, time window, unit of analysis and handling of retries before presenting a rate. The nine invalid examples here are deliberately selected; dividing their counts does not supply those missing ingredients.&lt;/p&gt;

&lt;p&gt;The immediate engineering change is concrete: add a task-specific acceptance check beside the HTTP and output checks for one workflow. Include a valid limitation response, a wrong target, a duplicate and an unobservable outcome. Run those cases before a release. A successful request is useful evidence, but completing the user's authorized task requires evidence at the task boundary too.&lt;/p&gt;




&lt;p&gt;&lt;strong&gt;About the author:&lt;/strong&gt; &lt;a href="https://www.linkedin.com/in/shashankagarwal2/" rel="noopener noreferrer"&gt;Shashank Agarwal&lt;/a&gt; is the founder and CEO of Noveum.ai and the creator of API.market. This article uses a standalone synthetic lab; it makes no claim about either company's customer results or revenue.&lt;/p&gt;

</description>
      <category>python</category>
    </item>
    <item>
      <title>Document API Changes Without Rewriting History</title>
      <dc:creator>Shashank Agarwal</dc:creator>
      <pubDate>Thu, 29 Oct 2026 16:05:40 +0000</pubDate>
      <link>https://dev.to/imshashank/document-api-changes-without-rewriting-history-3b17</link>
      <guid>https://dev.to/imshashank/document-api-changes-without-rewriting-history-3b17</guid>
      <description>&lt;p&gt;An API change note should help a developer decide whether their integration needs work. Replacing old documentation without a transition record can make that difficult.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Quick answer:&lt;/strong&gt; State the changed behavior, affected version or scope, effective timing and required consumer action. Preserve earlier contract context where useful and distinguish a correction in documentation from a change in server behavior. Keep examples and error guidance aligned with the supported version.&lt;/p&gt;

&lt;h2&gt;
  
  
  Name the observable change
&lt;/h2&gt;

&lt;p&gt;For an illustrative parser, an output field might move from a top-level string to an item-level result. The note should show the consumer impact rather than say only “response improvements.”&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Change-note field&lt;/th&gt;
&lt;th&gt;Developer question&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Scope&lt;/td&gt;
&lt;td&gt;Which operation and version are affected?&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Before&lt;/td&gt;
&lt;td&gt;What did the earlier contract establish?&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;After&lt;/td&gt;
&lt;td&gt;What does the current contract establish?&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Timing&lt;/td&gt;
&lt;td&gt;When does the change apply?&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Migration&lt;/td&gt;
&lt;td&gt;What must the consumer revise?&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Validation&lt;/td&gt;
&lt;td&gt;How can the integration be checked?&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;This is proposed documentation guidance, not an announcement of a real API.market change.&lt;/p&gt;

&lt;h2&gt;
  
  
  Keep correction and behavior separate
&lt;/h2&gt;

&lt;p&gt;If the server already returned a field but the example omitted it, describe the documentation correction. If the server contract changed, describe that change.&lt;/p&gt;

&lt;p&gt;A new page timestamp alone cannot tell a developer which event occurred.&lt;/p&gt;

&lt;h2&gt;
  
  
  Preserve meaningful history
&lt;/h2&gt;

&lt;p&gt;Link an appropriate earlier version or concise change record. Do not keep obsolete instructions in the current quickstart without a clear label.&lt;/p&gt;

&lt;p&gt;If a compatibility period or retirement date applies, use the actual supported policy rather than inventing one for the note.&lt;/p&gt;

&lt;h2&gt;
  
  
  Verify the consumer path
&lt;/h2&gt;

&lt;p&gt;Run the relevant controlled example where appropriate and inspect both expected and failure behavior. Ensure copied sample code uses the intended version.&lt;/p&gt;

&lt;p&gt;The &lt;a href="https://noveum.ai/en/blog/update-technical-writing-substantive-changes" rel="noopener noreferrer"&gt;technical-refresh guide&lt;/a&gt; and &lt;a href="https://noveum.ai/en/blog/capture-evidence-at-tool-boundary" rel="noopener noreferrer"&gt;tool-boundary guide&lt;/a&gt; help preserve the link between documentation and observed behavior.&lt;/p&gt;

&lt;p&gt;I'm Shashank Agarwal, creator of API.market and founder and CEO of Noveum AI. Follow my work on &lt;a href="https://www.linkedin.com/in/shashankagarwal2/" rel="noopener noreferrer"&gt;LinkedIn&lt;/a&gt;. Drafted with AI assistance.&lt;/p&gt;

</description>
      <category>api</category>
    </item>
    <item>
      <title>Make API Usage Reports Useful to the Person Paying</title>
      <dc:creator>Shashank Agarwal</dc:creator>
      <pubDate>Thu, 29 Oct 2026 16:02:49 +0000</pubDate>
      <link>https://dev.to/imshashank/make-api-usage-reports-useful-to-the-person-paying-nb0</link>
      <guid>https://dev.to/imshashank/make-api-usage-reports-useful-to-the-person-paying-nb0</guid>
      <description>&lt;p&gt;A usage report should help an account owner understand what consumed the allowance or balance. A large total without unit or scope leaves that decision unresolved.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Quick answer:&lt;/strong&gt; Show the billable unit, reporting window, product scope and relevant breakdowns, with a path to appropriately restricted request evidence. Distinguish estimated usage, provider-reported consumption and final billing. Explain missing or delayed data and preserve retries where they affect interpretation.&lt;/p&gt;

&lt;h2&gt;
  
  
  Start with the billing question
&lt;/h2&gt;

&lt;p&gt;For an illustrative extraction account, the owner may need to know whether a rise came from a larger batch, repeated attempts or a different operation.&lt;/p&gt;

&lt;p&gt;A dashboard showing only “requests” may not explain the actual billable unit.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Report item&lt;/th&gt;
&lt;th&gt;Question answered&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Unit and product&lt;/td&gt;
&lt;td&gt;What is being counted?&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Window and time zone&lt;/td&gt;
&lt;td&gt;Which activity is included?&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Breakdown&lt;/td&gt;
&lt;td&gt;Which workflow or operation contributed?&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Retry treatment&lt;/td&gt;
&lt;td&gt;Are repeated attempts distinguishable?&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Status&lt;/td&gt;
&lt;td&gt;Is the data preliminary or reconciled?&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Evidence route&lt;/td&gt;
&lt;td&gt;How can an authorized owner investigate?&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;This is a proposed report design, not an actual account statement.&lt;/p&gt;

&lt;h2&gt;
  
  
  Preserve privacy while supporting investigation
&lt;/h2&gt;

&lt;p&gt;Aggregate views can explain most trends. Detailed request evidence should follow the account's access and retention rules.&lt;/p&gt;

&lt;p&gt;Do not expose credentials or unnecessary payload content simply to make a billing chart more detailed.&lt;/p&gt;

&lt;h2&gt;
  
  
  Explain mismatches
&lt;/h2&gt;

&lt;p&gt;A usage event and an invoice line may use different processing times or aggregation rules. State the actual product policy and reporting status.&lt;/p&gt;

&lt;p&gt;Do not silently rewrite the original estimate to make it appear identical to a later charge.&lt;/p&gt;

&lt;h2&gt;
  
  
  Make the next action clear
&lt;/h2&gt;

&lt;p&gt;The owner may need to review a workflow, adjust planned usage or investigate failures. The report should support that decision without inventing a cause.&lt;/p&gt;

&lt;p&gt;See the &lt;a href="https://api.market/blog/pipfeed/parse/what-api-call-counter-measures" rel="noopener noreferrer"&gt;API call-counter guide&lt;/a&gt; and &lt;a href="https://api.market/blog/pipfeed/parse/retries-in-api-growth-reports" rel="noopener noreferrer"&gt;retry-reporting guide&lt;/a&gt; for distinctions that help explain a changing total.&lt;/p&gt;

&lt;p&gt;I'm Shashank Agarwal, creator of API.market and founder and CEO of Noveum AI. Follow my work on &lt;a href="https://www.linkedin.com/in/shashankagarwal2/" rel="noopener noreferrer"&gt;LinkedIn&lt;/a&gt;. Drafted with AI assistance.&lt;/p&gt;

</description>
      <category>api</category>
    </item>
    <item>
      <title>Review the Billing Contract Before Integrating an API</title>
      <dc:creator>Shashank Agarwal</dc:creator>
      <pubDate>Thu, 29 Oct 2026 16:02:34 +0000</pubDate>
      <link>https://dev.to/imshashank/review-the-billing-contract-before-integrating-an-api-16b2</link>
      <guid>https://dev.to/imshashank/review-the-billing-contract-before-integrating-an-api-16b2</guid>
      <description>&lt;p&gt;The integration team should be able to explain what a request may consume before the application depends on it. A price label alone cannot establish that contract.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Quick answer:&lt;/strong&gt; Identify the billable unit, request scope, included allowance, overage or balance behavior, and treatment of failures and retries. Work through the application's expected request pattern using the selected product's current plan. Keep estimates separate from observed usage and final billing.&lt;/p&gt;

&lt;h2&gt;
  
  
  Start with the actual request shape
&lt;/h2&gt;

&lt;p&gt;For an illustrative batch parser, a single request might contain several source URLs. A per-request and a per-item price would describe different costs.&lt;/p&gt;

&lt;p&gt;Do not infer which rule applies from the endpoint name.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Worksheet item&lt;/th&gt;
&lt;th&gt;What to establish&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Request shape&lt;/td&gt;
&lt;td&gt;Which operation and input quantity are planned?&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Billable event&lt;/td&gt;
&lt;td&gt;What does the provider count?&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Allowance&lt;/td&gt;
&lt;td&gt;Which unit, period and reset rule apply?&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Failure&lt;/td&gt;
&lt;td&gt;Which unsuccessful conditions are charged?&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Retry&lt;/td&gt;
&lt;td&gt;How are repeated attempts handled?&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Expansion&lt;/td&gt;
&lt;td&gt;What happens when the current plan is exhausted?&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;This is proposed integration planning, not a statement that every API follows one billing model.&lt;/p&gt;

&lt;h2&gt;
  
  
  Include uncertainty explicitly
&lt;/h2&gt;

&lt;p&gt;If the policy does not explain partial results, record the gap. An assumption of free retries should not silently enter the implementation.&lt;/p&gt;

&lt;p&gt;A bounded test may reveal an observed usage change, but one observation does not replace the current commercial policy.&lt;/p&gt;

&lt;h2&gt;
  
  
  Preserve the estimate's assumptions
&lt;/h2&gt;

&lt;p&gt;Input volume, retries and output options may change the cost. Record the quantity and configuration behind an estimate so the team can revise it appropriately.&lt;/p&gt;

&lt;p&gt;No numerical cost projection is provided here.&lt;/p&gt;

&lt;h2&gt;
  
  
  Make the first review useful
&lt;/h2&gt;

&lt;p&gt;Before a larger batch, ask a reviewer to describe one success, one failure and one repeated attempt under the selected plan.&lt;/p&gt;

&lt;p&gt;Use the &lt;a href="https://api.market/blog/pipfeed/parse/api-pricing-before-first-request" rel="noopener noreferrer"&gt;pricing-page guide&lt;/a&gt; and &lt;a href="https://api.market/blog/pipfeed/parse/document-quota-rate-concurrency-limits" rel="noopener noreferrer"&gt;quota/rate/concurrency guide&lt;/a&gt; to distinguish commercial units from operating limits.&lt;/p&gt;

&lt;p&gt;I'm Shashank Agarwal, creator of API.market and founder and CEO of Noveum AI. Follow my work on &lt;a href="https://www.linkedin.com/in/shashankagarwal2/" rel="noopener noreferrer"&gt;LinkedIn&lt;/a&gt;. Drafted with AI assistance.&lt;/p&gt;

</description>
      <category>api</category>
    </item>
    <item>
      <title>What to Show Beside an Example API Response</title>
      <dc:creator>Shashank Agarwal</dc:creator>
      <pubDate>Thu, 29 Oct 2026 16:01:13 +0000</pubDate>
      <link>https://dev.to/imshashank/what-to-show-beside-an-example-api-response-3j58</link>
      <guid>https://dev.to/imshashank/what-to-show-beside-an-example-api-response-3j58</guid>
      <description>&lt;p&gt;A JSON example can look self-explanatory while leaving the consumer unsure what was requested, which fields are optional and whether the result is complete.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Quick answer:&lt;/strong&gt; Show the relevant input, schema or version, field meanings, empty and partial-result behavior, and whether the example is synthetic or observed. Keep credentials and private records out of the example. A plausible sample demonstrates a contract only when its scope is clear.&lt;/p&gt;

&lt;h2&gt;
  
  
  Label the artifact
&lt;/h2&gt;

&lt;p&gt;The following is a synthetic response from a hypothetical batch-extraction service. It is not an API.market endpoint contract or an executed provider result.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"status"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"partial"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"items"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="nl"&gt;"source_id"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"sample-a"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"text"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"Synthetic article text"&lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="nl"&gt;"source_id"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"sample-b"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"text"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kc"&gt;null&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"reason"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"unavailable"&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The surrounding documentation would need to define what partial means, whether null is allowed and what unavailable establishes.&lt;/p&gt;

&lt;h2&gt;
  
  
  Add the consumer questions
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Context&lt;/th&gt;
&lt;th&gt;Question answered&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Request&lt;/td&gt;
&lt;td&gt;Which input produced this shape?&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Version&lt;/td&gt;
&lt;td&gt;Which contract does it illustrate?&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Field definition&lt;/td&gt;
&lt;td&gt;What can the consumer infer?&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Completeness&lt;/td&gt;
&lt;td&gt;Which items succeeded or remain unresolved?&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Provenance&lt;/td&gt;
&lt;td&gt;Synthetic or captured from an identified test?&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Next step&lt;/td&gt;
&lt;td&gt;How should a partial result be handled?&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Do not imply a billing or retry policy merely by showing a reason field.&lt;/p&gt;

&lt;h2&gt;
  
  
  Include a meaningful alternative
&lt;/h2&gt;

&lt;p&gt;Pair the happy path with a documented empty, partial or rejected result. Keep examples consistent with the actual selected product.&lt;/p&gt;

&lt;p&gt;If a captured response is transformed for readability, state what was changed and preserve an appropriate original reference for review.&lt;/p&gt;

&lt;h2&gt;
  
  
  Check the downstream assumption
&lt;/h2&gt;

&lt;p&gt;A consumer should not treat a parseable envelope as proof that every item succeeded. Inspect the field that determines usable output.&lt;/p&gt;

&lt;p&gt;The &lt;a href="https://noveum.ai/en/blog/agent-format-errors-task-failure" rel="noopener noreferrer"&gt;format-versus-task guide&lt;/a&gt; and &lt;a href="https://noveum.ai/en/blog/capture-evidence-at-tool-boundary" rel="noopener noreferrer"&gt;tool-boundary guide&lt;/a&gt; explain why those distinctions matter beyond documentation.&lt;/p&gt;

&lt;p&gt;I'm Shashank Agarwal, creator of API.market and founder and CEO of Noveum AI. Follow my work on &lt;a href="https://www.linkedin.com/in/shashankagarwal2/" rel="noopener noreferrer"&gt;LinkedIn&lt;/a&gt;. Drafted with AI assistance.&lt;/p&gt;

</description>
      <category>api</category>
    </item>
    <item>
      <title>Turn an API Support Question Into a Documentation Fix</title>
      <dc:creator>Shashank Agarwal</dc:creator>
      <pubDate>Thu, 29 Oct 2026 16:00:04 +0000</pubDate>
      <link>https://dev.to/imshashank/turn-an-api-support-question-into-a-documentation-fix-26ba</link>
      <guid>https://dev.to/imshashank/turn-an-api-support-question-into-a-documentation-fix-26ba</guid>
      <description>&lt;p&gt;A support request can reveal a specific missing contract. The documentation fix should answer that question rather than merely add another broad FAQ entry.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Quick answer:&lt;/strong&gt; Reconstruct the developer's task, identify what the public guidance failed to establish and write the explanation beside the relevant step. Use a controlled example to verify the new guidance. Preserve sensitive support evidence in the appropriate restricted place and label any public reconstruction as synthetic.&lt;/p&gt;

&lt;h2&gt;
  
  
  Work through an illustrative question
&lt;/h2&gt;

&lt;p&gt;Suppose a developer asks why a parser returned no article text even though the response parsed successfully. The missing explanation might concern an unavailable source, an empty extraction or the field containing the result.&lt;/p&gt;

&lt;p&gt;Those conditions need different handling. A generic “check your API key” answer may not address any of them.&lt;/p&gt;

&lt;h2&gt;
  
  
  Write a documentation change card
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Item&lt;/th&gt;
&lt;th&gt;What to record&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;User task&lt;/td&gt;
&lt;td&gt;The integration decision the developer faced&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Missing fact&lt;/td&gt;
&lt;td&gt;What the current page did not establish&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Evidence&lt;/td&gt;
&lt;td&gt;Relevant contract and minimized response&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Revision&lt;/td&gt;
&lt;td&gt;The exact explanation or example added&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Verification&lt;/td&gt;
&lt;td&gt;What a new reader should now understand&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;This is a proposed workflow using a synthetic support scenario, not an actual API.market ticket.&lt;/p&gt;

&lt;h2&gt;
  
  
  Put the answer where it is needed
&lt;/h2&gt;

&lt;p&gt;Explain empty-result semantics beside the response fields. Explain failure billing beside the plan. Use links when deeper detail is needed.&lt;/p&gt;

&lt;p&gt;A long troubleshooting page should not be the only place a required input format is documented.&lt;/p&gt;

&lt;h2&gt;
  
  
  Verify without private context
&lt;/h2&gt;

&lt;p&gt;Ask a reviewer to choose the next step using only the revised page and a controlled response. If they still need the support engineer's memory, the fix is incomplete.&lt;/p&gt;

&lt;p&gt;Do not claim a support-volume reduction without a suitable comparison.&lt;/p&gt;

&lt;p&gt;The &lt;a href="https://api.market/blog/pipfeed/parse/api-pricing-before-first-request" rel="noopener noreferrer"&gt;pricing-contract guide&lt;/a&gt; and &lt;a href="https://noveum.ai/en/blog/agent-format-errors-task-failure" rel="noopener noreferrer"&gt;format-versus-task guide&lt;/a&gt; show two contracts often hidden behind a successful request.&lt;/p&gt;

&lt;p&gt;I'm Shashank Agarwal, creator of API.market and founder and CEO of Noveum AI. Follow my work on &lt;a href="https://www.linkedin.com/in/shashankagarwal2/" rel="noopener noreferrer"&gt;LinkedIn&lt;/a&gt;. Drafted with AI assistance.&lt;/p&gt;

</description>
      <category>api</category>
    </item>
    <item>
      <title>An API Error Message Is Part of the Product</title>
      <dc:creator>Shashank Agarwal</dc:creator>
      <pubDate>Thu, 29 Oct 2026 15:59:52 +0000</pubDate>
      <link>https://dev.to/imshashank/an-api-error-message-is-part-of-the-product-3d0</link>
      <guid>https://dev.to/imshashank/an-api-error-message-is-part-of-the-product-3d0</guid>
      <description>&lt;p&gt;An error response is part of the API contract a developer has to integrate. A generic failure message can leave the customer guessing whether to fix input, wait or investigate an uncertain outcome.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Quick answer:&lt;/strong&gt; Explain the failure class, relevant invalid field or operating condition, request reference and supported next step. Keep machine-readable classification separate from prose that may change. Avoid exposing credentials or internal implementation details, and do not imply that every failure can be safely retried.&lt;/p&gt;

&lt;h2&gt;
  
  
  Design for the next decision
&lt;/h2&gt;

&lt;p&gt;For an illustrative conversion API, an unsupported input format needs a different next step from a temporary dependency failure. A timeout after submission may leave the outcome unknown.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Error information&lt;/th&gt;
&lt;th&gt;Developer decision&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Stable classification&lt;/td&gt;
&lt;td&gt;Choose the appropriate handling branch&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Relevant field or condition&lt;/td&gt;
&lt;td&gt;Correct a request when possible&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Request reference&lt;/td&gt;
&lt;td&gt;Support an authorized investigation&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Outcome status&lt;/td&gt;
&lt;td&gt;Distinguish rejection from uncertain completion&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Recovery guidance&lt;/td&gt;
&lt;td&gt;Decide whether another attempt is appropriate&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;This is a proposed design checklist, not an actual provider's response schema.&lt;/p&gt;

&lt;h2&gt;
  
  
  Use an appropriate representation
&lt;/h2&gt;

&lt;p&gt;&lt;a href="https://www.rfc-editor.org/rfc/rfc9457" rel="noopener noreferrer"&gt;RFC 9457&lt;/a&gt; defines a problem-details model for HTTP APIs. It can be useful when designing an interface, but linking the specification does not establish that a particular API implements it.&lt;/p&gt;

&lt;p&gt;Keep the selected product's documented contract authoritative for its actual error semantics.&lt;/p&gt;

&lt;h2&gt;
  
  
  Avoid a dangerous retry promise
&lt;/h2&gt;

&lt;p&gt;If an operation changes state, another request may duplicate work unless the contract establishes appropriate replay behavior.&lt;/p&gt;

&lt;p&gt;Do not turn “we did not receive a response” into “nothing happened.” Give the client an honest route to establish the outcome.&lt;/p&gt;

&lt;h2&gt;
  
  
  Verify the error experience
&lt;/h2&gt;

&lt;p&gt;Use a controlled invalid request and inspect whether a new developer can identify the issue and next step. Label any synthetic response clearly.&lt;/p&gt;

&lt;p&gt;The &lt;a href="https://api.market/blog/pipfeed/parse/explain-api-usage-limits" rel="noopener noreferrer"&gt;usage-limit guide&lt;/a&gt; and &lt;a href="https://noveum.ai/en/blog/agent-error-message-insufficient-evidence" rel="noopener noreferrer"&gt;uncertain-error guide&lt;/a&gt; provide useful boundaries around recovery.&lt;/p&gt;

&lt;p&gt;I'm Shashank Agarwal, creator of API.market and founder and CEO of Noveum AI. Follow my work on &lt;a href="https://www.linkedin.com/in/shashankagarwal2/" rel="noopener noreferrer"&gt;LinkedIn&lt;/a&gt;. Drafted with AI assistance.&lt;/p&gt;

</description>
      <category>api</category>
    </item>
    <item>
      <title>Connect a Marketplace to Agents Without Hiding Limits</title>
      <dc:creator>Shashank Agarwal</dc:creator>
      <pubDate>Thu, 29 Oct 2026 15:26:45 +0000</pubDate>
      <link>https://dev.to/imshashank/connect-a-marketplace-to-agents-without-hiding-limits-54h8</link>
      <guid>https://dev.to/imshashank/connect-a-marketplace-to-agents-without-hiding-limits-54h8</guid>
      <description>&lt;p&gt;A gateway can reduce connection work while still leaving important product limits. The integration should make those limits visible at the moment a developer or agent needs them.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Quick answer:&lt;/strong&gt; Keep access conditions, pricing, supported inputs, usage limits and result semantics close to discovery and execution. Present unknowns as unresolved questions. A convenient connection should help the user make product decisions, not imply that all listed operations share one contract.&lt;/p&gt;

&lt;h2&gt;
  
  
  Preserve the product boundary
&lt;/h2&gt;

&lt;p&gt;For an illustrative content workflow, extraction and image generation may have different input sizes, output formats and billing rules. A shared connection does not remove those differences.&lt;/p&gt;

&lt;p&gt;The developer needs the selected operation's current contract, especially before sending private material or starting a costly batch.&lt;/p&gt;

&lt;h2&gt;
  
  
  Show the limit beside the decision
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Decision point&lt;/th&gt;
&lt;th&gt;Useful information&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Candidate selection&lt;/td&gt;
&lt;td&gt;Supported task and input restrictions&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Plan selection&lt;/td&gt;
&lt;td&gt;Billable unit and allowance&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Execution&lt;/td&gt;
&lt;td&gt;Permission and current usage state&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Result handling&lt;/td&gt;
&lt;td&gt;Empty, partial and failed outcome semantics&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Expansion&lt;/td&gt;
&lt;td&gt;Batch cost and recovery behavior&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;This is a proposed product-design framework. It does not claim automatic enforcement of every boundary.&lt;/p&gt;

&lt;h2&gt;
  
  
  Make unknowns actionable
&lt;/h2&gt;

&lt;p&gt;If a plan does not explain whether failed requests count, identify the missing policy. Do not invent a refund or assume every retry is free.&lt;/p&gt;

&lt;p&gt;If a tool result is partial, tell the downstream consumer what is missing. A clean envelope should not hide a failed task.&lt;/p&gt;

&lt;h2&gt;
  
  
  Review the complete first experience
&lt;/h2&gt;

&lt;p&gt;Ask a developer to choose a product, describe the next call's conditions and inspect a bounded result. Where they need to guess, improve the contract or documentation.&lt;/p&gt;

&lt;p&gt;I'm Shashank Agarwal, creator of API.market and founder and CEO of Noveum AI. &lt;a href="https://api.market/mcp" rel="noopener noreferrer"&gt;API.market setup&lt;/a&gt; is the current gateway entry point.&lt;/p&gt;

&lt;p&gt;The &lt;a href="https://api.market/blog/pipfeed/parse/explain-api-usage-limits" rel="noopener noreferrer"&gt;usage-limit guide&lt;/a&gt; and &lt;a href="https://noveum.ai/en/blog/agent-format-errors-task-failure" rel="noopener noreferrer"&gt;format-versus-task guide&lt;/a&gt; explain two limits that deserve visibility after connection.&lt;/p&gt;

&lt;p&gt;Follow my work on &lt;a href="https://www.linkedin.com/in/shashankagarwal2/" rel="noopener noreferrer"&gt;LinkedIn&lt;/a&gt;. Drafted with AI assistance.&lt;/p&gt;

</description>
      <category>mcp</category>
    </item>
    <item>
      <title>A Practical Acceptance Test for a Tool-Using Agent</title>
      <dc:creator>Shashank Agarwal</dc:creator>
      <pubDate>Thu, 29 Oct 2026 15:25:24 +0000</pubDate>
      <link>https://dev.to/imshashank/a-practical-acceptance-test-for-a-tool-using-agent-35kj</link>
      <guid>https://dev.to/imshashank/a-practical-acceptance-test-for-a-tool-using-agent-35kj</guid>
      <description>&lt;p&gt;An agent acceptance test should establish whether the workflow performed the intended task. A tool invocation count is only part of that evidence.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Quick answer:&lt;/strong&gt; Define the user's goal, controlled tool responses, allowed operations and observable outcome. Check submitted arguments and the final explanation against that contract. Include a meaningful failure branch and a case where the agent should stop or clarify rather than invoke another tool.&lt;/p&gt;

&lt;h2&gt;
  
  
  Build a small fixture
&lt;/h2&gt;

&lt;p&gt;For an illustrative inventory assistant, the user requests an item at a named branch. The controlled result contains stock for that branch and a similarly named item elsewhere.&lt;/p&gt;

&lt;p&gt;The expected answer must use the requested item and branch. If reservation is outside the authorized task, the agent should not create one simply because a reservation tool is available.&lt;/p&gt;

&lt;h2&gt;
  
  
  Check the whole path
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Check&lt;/th&gt;
&lt;th&gt;Evidence&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Input interpretation&lt;/td&gt;
&lt;td&gt;Correct item and branch arguments&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Tool result use&lt;/td&gt;
&lt;td&gt;Answer follows the supplied inventory&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Action scope&lt;/td&gt;
&lt;td&gt;No unrequested reservation&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Failure handling&lt;/td&gt;
&lt;td&gt;Empty or unavailable results explained&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Final response&lt;/td&gt;
&lt;td&gt;Accurate scope and useful next step&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;These are proposed test conditions, not a report of a benchmark run.&lt;/p&gt;

&lt;h2&gt;
  
  
  Include a counterexample
&lt;/h2&gt;

&lt;p&gt;Return a valid empty result for the requested branch. The agent should not use another branch's stock as though it answered the original question.&lt;/p&gt;

&lt;p&gt;If broader search is allowed, record the expanded scope and describe it honestly. The acceptance condition should distinguish that helpful extension from a silent substitution.&lt;/p&gt;

&lt;h2&gt;
  
  
  Preserve the evidence
&lt;/h2&gt;

&lt;p&gt;Save the fixture version, actual arguments, relevant result and decision. If the test fails, another engineer should be able to inspect the mismatch without reconstructing it from a score.&lt;/p&gt;

&lt;p&gt;I'm Shashank Agarwal, creator of API.market and founder and CEO of Noveum AI. Start from &lt;a href="https://api.market/mcp" rel="noopener noreferrer"&gt;API.market setup&lt;/a&gt;, then use &lt;a href="https://noveum.ai/en/blog/write-agent-acceptance-criteria" rel="noopener noreferrer"&gt;acceptance criteria&lt;/a&gt; and the &lt;a href="https://noveum.ai/en/blog/turn-agent-failure-into-regression-test" rel="noopener noreferrer"&gt;regression-test guide&lt;/a&gt; to make the result repeatable.&lt;/p&gt;

&lt;p&gt;Follow my work on &lt;a href="https://www.linkedin.com/in/shashankagarwal2/" rel="noopener noreferrer"&gt;LinkedIn&lt;/a&gt;. Drafted with AI assistance.&lt;/p&gt;

</description>
      <category>mcp</category>
    </item>
    <item>
      <title>Document an MCP Gateway for a New Developer</title>
      <dc:creator>Shashank Agarwal</dc:creator>
      <pubDate>Thu, 29 Oct 2026 15:22:52 +0000</pubDate>
      <link>https://dev.to/imshashank/document-an-mcp-gateway-for-a-new-developer-42i2</link>
      <guid>https://dev.to/imshashank/document-an-mcp-gateway-for-a-new-developer-42i2</guid>
      <description>&lt;p&gt;A gateway setup page should carry a developer from connection details to a useful first result. A server URL alone does not explain the operating contract.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Quick answer:&lt;/strong&gt; Document the endpoint, supported transport, authentication choices, access requirements and a bounded first workflow. Explain where pricing applies, how to recognize a failed result and how access can be managed or revoked. Keep client-specific steps current and label the date or version they describe.&lt;/p&gt;

&lt;h2&gt;
  
  
  Separate setup from execution
&lt;/h2&gt;

&lt;p&gt;A developer needs to know whether they are configuring a client, granting access or invoking a product operation. Those stages can have different prerequisites.&lt;/p&gt;

&lt;p&gt;Avoid presenting a successful login as proof that every product operation is usable.&lt;/p&gt;

&lt;h2&gt;
  
  
  Structure the guide around decisions
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Section&lt;/th&gt;
&lt;th&gt;Developer question&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Connection&lt;/td&gt;
&lt;td&gt;Which endpoint and transport should I use?&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Authentication&lt;/td&gt;
&lt;td&gt;Which flow applies to my client?&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Product access&lt;/td&gt;
&lt;td&gt;What plan or allowance is needed?&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;First task&lt;/td&gt;
&lt;td&gt;What small operation should I inspect?&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Failure&lt;/td&gt;
&lt;td&gt;What should I do with an empty or rejected result?&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Access management&lt;/td&gt;
&lt;td&gt;Where can I review or revoke the connection?&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;This is a documentation template, not a promise that all gateways expose the same features.&lt;/p&gt;

&lt;h2&gt;
  
  
  Use current concrete details carefully
&lt;/h2&gt;

&lt;p&gt;As checked on October 29, 2026, &lt;a href="https://api.market/mcp" rel="noopener noreferrer"&gt;API.market's setup page&lt;/a&gt; shows the hosted endpoint &lt;code&gt;https://api.market/api/mcp/gateway&lt;/code&gt; and OAuth or API-key connection choices.&lt;/p&gt;

&lt;p&gt;Client interfaces can change. Link to the maintained setup rather than copying a long configuration tutorial into every article.&lt;/p&gt;

&lt;h2&gt;
  
  
  Finish with an observable artifact
&lt;/h2&gt;

&lt;p&gt;A first workflow should show what usable output looks like and what an unavailable result means. Label illustrative examples and identify any steps that were actually verified.&lt;/p&gt;

&lt;p&gt;I'm Shashank Agarwal, creator of API.market and founder and CEO of Noveum AI. The &lt;a href="https://api.market/blog/pipfeed/parse/api-pricing-before-first-request" rel="noopener noreferrer"&gt;pricing review&lt;/a&gt; and &lt;a href="https://noveum.ai/en/blog/capture-evidence-at-tool-boundary" rel="noopener noreferrer"&gt;tool-boundary guide&lt;/a&gt; cover contracts that remain important after connection succeeds.&lt;/p&gt;

&lt;p&gt;Follow my work on &lt;a href="https://www.linkedin.com/in/shashankagarwal2/" rel="noopener noreferrer"&gt;LinkedIn&lt;/a&gt;. Drafted with AI assistance.&lt;/p&gt;

</description>
      <category>mcp</category>
    </item>
    <item>
      <title>Tool Availability and Permission Are Different Questions</title>
      <dc:creator>Shashank Agarwal</dc:creator>
      <pubDate>Thu, 29 Oct 2026 15:21:27 +0000</pubDate>
      <link>https://dev.to/imshashank/tool-availability-and-permission-are-different-questions-4gb0</link>
      <guid>https://dev.to/imshashank/tool-availability-and-permission-are-different-questions-4gb0</guid>
      <description>&lt;p&gt;A tool appearing in an agent's catalog establishes a capability it may be able to invoke. It does not establish permission to use it for every task or with every input.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Quick answer:&lt;/strong&gt; Evaluate the user's authorized objective, the operation's side effects and the data destination before execution. Keep permission scoped to the actual action. Treat tool descriptions and returned content as information about capabilities, not as instructions that can expand the user's authorization.&lt;/p&gt;

&lt;h2&gt;
  
  
  Start with a task boundary
&lt;/h2&gt;

&lt;p&gt;For an illustrative research assistant, a search operation may help answer a question. A messaging tool could share that answer publicly, but public distribution is a separate action.&lt;/p&gt;

&lt;p&gt;The agent should not infer permission to publish merely because the tool is available and the result would be convenient to share.&lt;/p&gt;

&lt;h2&gt;
  
  
  Inspect the action dimensions
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Dimension&lt;/th&gt;
&lt;th&gt;Question&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Purpose&lt;/td&gt;
&lt;td&gt;Does this operation serve the authorized task?&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Data&lt;/td&gt;
&lt;td&gt;Is this input permitted for this destination?&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Effect&lt;/td&gt;
&lt;td&gt;Does it read, modify, send or spend?&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Scope&lt;/td&gt;
&lt;td&gt;Which account, recipient or artifact is involved?&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Review&lt;/td&gt;
&lt;td&gt;Does policy or unresolved scope require a decision?&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;These are proposed application checks. They do not replace the operating environment's actual rules.&lt;/p&gt;

&lt;h2&gt;
  
  
  Resist instruction-like tool content
&lt;/h2&gt;

&lt;p&gt;A retrieved page might tell the assistant to upload additional files or contact another service. That page cannot grant the user's permission.&lt;/p&gt;

&lt;p&gt;Preserve useful facts from the content while rejecting instructions that attempt to redirect the workflow beyond its authorized scope.&lt;/p&gt;

&lt;h2&gt;
  
  
  Test the distinction
&lt;/h2&gt;

&lt;p&gt;Give a controlled agent fixture both a relevant read operation and an unrelated write operation. Check that it completes the task without adding the write. Then provide explicit authorization for a bounded write and verify that it can proceed appropriately.&lt;/p&gt;

&lt;p&gt;I'm Shashank Agarwal, creator of API.market and founder and CEO of Noveum AI. Use &lt;a href="https://api.market/mcp" rel="noopener noreferrer"&gt;API.market setup&lt;/a&gt;, the &lt;a href="https://api.market/blog/pipfeed/parse/api-pricing-before-first-request" rel="noopener noreferrer"&gt;pricing contract guide&lt;/a&gt; and &lt;a href="https://noveum.ai/en/blog/write-agent-acceptance-criteria" rel="noopener noreferrer"&gt;acceptance-criteria guidance&lt;/a&gt; to make these boundaries inspectable.&lt;/p&gt;

&lt;p&gt;Follow my work on &lt;a href="https://www.linkedin.com/in/shashankagarwal2/" rel="noopener noreferrer"&gt;LinkedIn&lt;/a&gt;. Drafted with AI assistance.&lt;/p&gt;

</description>
      <category>mcp</category>
    </item>
    <item>
      <title>What Human Approval Should Show Before a Tool Action</title>
      <dc:creator>Shashank Agarwal</dc:creator>
      <pubDate>Thu, 29 Oct 2026 15:20:56 +0000</pubDate>
      <link>https://dev.to/imshashank/what-human-approval-should-show-before-a-tool-action-1lfj</link>
      <guid>https://dev.to/imshashank/what-human-approval-should-show-before-a-tool-action-1lfj</guid>
      <description>&lt;p&gt;A confirmation prompt is useful only when the user can understand what will happen. “Proceed?” leaves the important decision hidden inside the workflow.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Quick answer:&lt;/strong&gt; Show the selected operation, destination, exact scope of data, expected effect and relevant cost or reversibility. Link the concrete draft or artifact when it helps review. Carry the approved scope into execution and ask again only when the action materially changes or the operating policy requires it.&lt;/p&gt;

&lt;h2&gt;
  
  
  Make the proposed action concrete
&lt;/h2&gt;

&lt;p&gt;For an illustrative publishing assistant, the review packet should show the article, destination account and public audience. A request to “improve my presence” may explain the objective without resolving a newly discovered sensitive disclosure.&lt;/p&gt;

&lt;p&gt;The application's policy determines which actions require approval. This article proposes how to make that approval understandable.&lt;/p&gt;

&lt;h2&gt;
  
  
  Use a short decision card
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Field&lt;/th&gt;
&lt;th&gt;What the user reviews&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Action&lt;/td&gt;
&lt;td&gt;Create, modify, send or another defined operation&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Destination&lt;/td&gt;
&lt;td&gt;Service, account and intended audience&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Data&lt;/td&gt;
&lt;td&gt;Actual content or bounded input reference&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Effect&lt;/td&gt;
&lt;td&gt;What external state will change&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Cost&lt;/td&gt;
&lt;td&gt;Applicable charge basis and limit&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Recovery&lt;/td&gt;
&lt;td&gt;Available correction or undo path&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Keep unrelated internal implementation details out of the prompt.&lt;/p&gt;

&lt;h2&gt;
  
  
  Preserve scope after approval
&lt;/h2&gt;

&lt;p&gt;If a provider changes, a larger batch is proposed or extra data is required, evaluate whether the original approval still covers the action.&lt;/p&gt;

&lt;p&gt;A confirmation should not become a standing excuse to re-ask routine questions that were already settled. The goal is a concrete decision at the meaningful boundary.&lt;/p&gt;

&lt;h2&gt;
  
  
  Verify the outcome
&lt;/h2&gt;

&lt;p&gt;After execution, inspect the changed state or public result. An accepted request may still leave an unsaved draft or a failed operation.&lt;/p&gt;

&lt;p&gt;I'm Shashank Agarwal, creator of API.market and founder and CEO of Noveum AI. See &lt;a href="https://api.market/mcp" rel="noopener noreferrer"&gt;MCP setup&lt;/a&gt; for the current gateway entry point, the &lt;a href="https://api.market/blog/pipfeed/parse/api-pricing-before-first-request" rel="noopener noreferrer"&gt;pricing review&lt;/a&gt; for spending context and &lt;a href="https://noveum.ai/en/blog/capture-evidence-at-tool-boundary" rel="noopener noreferrer"&gt;tool-boundary evidence&lt;/a&gt; for the resulting record.&lt;/p&gt;

&lt;p&gt;Follow my work on &lt;a href="https://www.linkedin.com/in/shashankagarwal2/" rel="noopener noreferrer"&gt;LinkedIn&lt;/a&gt;. Drafted with AI assistance.&lt;/p&gt;

</description>
      <category>mcp</category>
    </item>
  </channel>
</rss>
