<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: JevStation</title>
    <description>The latest articles on DEV Community by JevStation (@jevstation).</description>
    <link>https://dev.to/jevstation</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4235025%2F820b941a-5fe4-4674-ab67-fe071008cbe6.png</url>
      <title>DEV Community: JevStation</title>
      <link>https://dev.to/jevstation</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/jevstation"/>
    <language>en</language>
    <item>
      <title>Scoring Code Changes in Cursor With Typed Judgments</title>
      <dc:creator>JevStation</dc:creator>
      <pubDate>Fri, 20 Nov 2026 13:31:12 +0000</pubDate>
      <link>https://dev.to/jevstation/scoring-code-changes-in-cursor-with-typed-judgments-28eo</link>
      <guid>https://dev.to/jevstation/scoring-code-changes-in-cursor-with-typed-judgments-28eo</guid>
      <description>&lt;p&gt;"This feels risky" is a sentence a reviewer can act on. It is not something a CI job can branch on. Getting a model to produce a number you can threshold usually means asking for JSON, then writing a parser, then writing a fallback for when the parser fails.&lt;/p&gt;

&lt;p&gt;This post sets up the other path: a diff goes in, typed answers come out. You get a score, the option the model picked, and a confidence value from 0 to 1. You will see the config for Cursor, the question set worth copying, and a script that turns the answers into an exit code.&lt;/p&gt;

&lt;h2&gt;
  
  
  What typed output changes
&lt;/h2&gt;

&lt;p&gt;A chat model asked to "rate this diff from 1 to 5" writes a sentence that contains a 4. A decision model answers the question as a distribution over the levels you defined, plus a confidence value that tells you whether the distribution was concentrated.&lt;/p&gt;

&lt;p&gt;The difference matters at the edges. A prose answer of "around a 3, leaning low" leaves you to decide what that means for a merge. A typed answer of &lt;code&gt;score: 2.4, confidence: 0.41&lt;/code&gt; says the model split its probability between two adjacent levels and is not sure. That is a signal to send the change to a person, and your code can check it in one comparison.&lt;/p&gt;

&lt;p&gt;None of this replaces review. Google's &lt;a href="https://google.github.io/eng-practices/review/reviewer/looking-for.html" rel="noopener noreferrer"&gt;reviewer guide&lt;/a&gt; puts design, functionality, complexity and tests ahead of style, and a single score does not cover any of them well. The score is a triage signal: it decides which changes get a second pair of eyes first, not whether a change is correct.&lt;/p&gt;

&lt;h2&gt;
  
  
  Add the server to Cursor
&lt;/h2&gt;

&lt;p&gt;Cursor reads MCP servers from &lt;code&gt;~/.cursor/mcp.json&lt;/code&gt; for every project, or from &lt;code&gt;.cursor/mcp.json&lt;/code&gt; in a repository for that project only. The &lt;a href="https://cursor.com/docs/mcp" rel="noopener noreferrer"&gt;Cursor MCP documentation&lt;/a&gt; lists both, along with the Customize page in the sidebar.&lt;/p&gt;

&lt;p&gt;Create a key under Settings › API Keys first. Then keep it in your environment, not in the file:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nl"&gt;"mcpServers"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="nl"&gt;"jevstation"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
      &lt;span class="nl"&gt;"url"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"https://jevstation.com/api/mcp"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="nl"&gt;"headers"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="nl"&gt;"Authorization"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"Bearer ${env:JEVSTATION_API_KEY}"&lt;/span&gt;
      &lt;span class="p"&gt;}&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Cursor resolves &lt;code&gt;${env:NAME}&lt;/code&gt; inside &lt;code&gt;url&lt;/code&gt; and &lt;code&gt;headers&lt;/code&gt;, so this file can be committed. Each developer sets &lt;code&gt;JEVSTATION_API_KEY&lt;/code&gt; in their shell profile, then restarts Cursor; &lt;a href="https://jevstation.com/integrations/jev-cursor" rel="noopener noreferrer"&gt;the Jev Cursor integration page&lt;/a&gt; notes that Cursor reads environment variables when it starts, and covers the same setup with the failure modes spelled out.&lt;/p&gt;

&lt;p&gt;There is no &lt;code&gt;command&lt;/code&gt;, &lt;code&gt;args&lt;/code&gt;, or &lt;code&gt;envFile&lt;/code&gt; here. The server is remote and speaks Streamable HTTP, one of the two transports defined in the &lt;a href="https://modelcontextprotocol.io/specification/2025-06-18/basic/transports" rel="noopener noreferrer"&gt;MCP specification&lt;/a&gt;, so nothing runs on your machine.&lt;/p&gt;

&lt;p&gt;After the restart, open Customize and confirm the toggle is on. The server exposes several tools, including &lt;code&gt;jev_decide&lt;/code&gt;, &lt;code&gt;jev_run_question_set&lt;/code&gt; and &lt;code&gt;jev_guard_tool_call&lt;/code&gt;. If the server shows an error, open Output, then MCP Logs. A 401 there usually means Cursor started before the variable existed, or the key is missing or has been deleted.&lt;/p&gt;

&lt;p&gt;One detail about the account behind the key: &lt;a href="https://jevstation.com/" rel="noopener noreferrer"&gt;JevStation&lt;/a&gt; is an independent workspace that runs Jev, TypeSafe's System One model, and bills per evaluation. New accounts include 50 credits. A call costs one credit when the state is at most 8,000 characters and the set has at most five questions, and three credits above either limit.&lt;/p&gt;

&lt;h2&gt;
  
  
  Write the rubric before the prompt
&lt;/h2&gt;

&lt;p&gt;The question set is the ceiling on the answer. "How risky is this change?" with levels &lt;code&gt;low, medium, high&lt;/code&gt; returns a number you cannot defend, because every reviewer means something different by medium.&lt;/p&gt;

&lt;p&gt;Concrete levels are the fix. This set asks three things about one diff:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;id&lt;/th&gt;
&lt;th&gt;type&lt;/th&gt;
&lt;th&gt;what it asks&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;risk&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;score&lt;/td&gt;
&lt;td&gt;how risky is this to ship without a second human review?&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;area&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;choice&lt;/td&gt;
&lt;td&gt;which part of the system does the diff change most?&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;tests&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;noul&lt;/td&gt;
&lt;td&gt;does the diff add or update a test that would fail if the behaviour changed?&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The score levels, written as descriptions rather than adjectives:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Mechanical change: rename, formatting, comment, or a dependency bump with no logic edit.&lt;/li&gt;
&lt;li&gt;Localized logic change covered by tests in the touched files.&lt;/li&gt;
&lt;li&gt;Logic change in a shared module, or the touched path has no test.&lt;/li&gt;
&lt;li&gt;Touches auth, payments, a data migration, or deletes data.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Level 4 is not "very risky". It names the conditions that make a change expensive to get wrong. A reviewer reading the level list can disagree with a specific level instead of arguing about a word.&lt;/p&gt;

&lt;p&gt;Two rules keep the set useful. Ask one judgement per question, and combine the answers in code. If you want a single number, weight the three answers yourself; a weighted sum is easier to change later than a rewritten prompt. And keep the count at five or fewer, because a sixth question moves the call from one credit to three.&lt;/p&gt;

&lt;p&gt;When you do get a score, read it with its distribution. A score of 2.4 across four levels usually means most of the probability sat on level 2 with some on level 3, not that the change is a little worse than level 2. The answer carries the probability of each level for exactly this reason.&lt;/p&gt;

&lt;h2&gt;
  
  
  Ask Cursor to score a diff
&lt;/h2&gt;

&lt;p&gt;With the server connected, the agent can call &lt;code&gt;jev_decide&lt;/code&gt; itself. Naming the tool and the questions removes most of the guesswork:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Run git diff --cached --stat and git diff --cached, then call jev_decide once.

state: a JSON object with keys "files" (the stat output), "diff" (the diff,
truncated to 8,000 characters if longer) and "message" (my last commit message).

questions:
  risk: score, "How risky is this change to ship without a second human review?"
    levels: [mechanical change / localized logic change with tests /
             shared module or missing test / auth, payments, migration, deletion]
  area: choice, "Which part of the system does the diff change most?"
    options: api, data, ui, build
  tests: noul, "The diff adds or updates a test that would fail if this behaviour changed."

Show me the score, the confidence, and the probability of each level.
Do not summarise the diff.
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The answer comes back in one response, with an object per question id:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nl"&gt;"answers"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="nl"&gt;"risk"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nl"&gt;"score"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mf"&gt;2.4&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nl"&gt;"confidence"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mf"&gt;0.41&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="nl"&gt;"area"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nl"&gt;"choice"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"api"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nl"&gt;"confidence"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mf"&gt;0.86&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="nl"&gt;"tests"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nl"&gt;"noul"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mf"&gt;0.12&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Two things are worth noticing. The &lt;code&gt;tests&lt;/code&gt; answer is a probability of 0.12, not a boolean, because a yes/no question returns a number you can threshold and noul answers carry no separate confidence value. And the &lt;code&gt;risk&lt;/code&gt; answer at 0.41 confidence is the model saying the diff sat between two levels. On a change that touches payments, that is a reason to stop, even though 2.4 looks unremarkable.&lt;/p&gt;

&lt;p&gt;Cursor asks for approval before each MCP call by default, and you can expand the call to read the exact arguments. That review step is where you catch a diff that includes a token or a connection string, because the state is sent to the model for evaluation.&lt;/p&gt;

&lt;h2&gt;
  
  
  Put the same judgement in CI
&lt;/h2&gt;

&lt;p&gt;The MCP route is for a person working in an editor. A pre-push hook or a PR job needs its own path, and the same questions work through the HTTP API. The thresholds below are placeholders; tune them on your own merged diffs. This script reads the staged diff and exits non-zero when the change needs a human:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="c1"&gt;// review-score.mjs — run: node review-score.mjs&lt;/span&gt;
&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;execSync&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="s2"&gt;"node:child_process"&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="no"&gt;MAX_STATE&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;8000&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;diff&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;execSync&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"git diff --cached --unified=3"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;encoding&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"utf8"&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;!&lt;/span&gt;&lt;span class="n"&gt;diff&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;trim&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nb"&gt;console&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;error&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"No staged changes. Stage a diff first."&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="nb"&gt;process&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;exit&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;state&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="n"&gt;files&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nf"&gt;execSync&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"git diff --cached --name-status"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;encoding&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"utf8"&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;trim&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;diff&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;diff&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;length&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="no"&gt;MAX_STATE&lt;/span&gt; &lt;span class="o"&gt;?&lt;/span&gt; &lt;span class="n"&gt;diff&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;slice&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="no"&gt;MAX_STATE&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="s2"&gt;"\n[diff truncated]"&lt;/span&gt; &lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;diff&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;res&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nb"&gt;fetch&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"https://jevstation.com/api/v1/systemone"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="n"&gt;method&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"POST"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;headers&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="nc"&gt;Authorization&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sb"&gt;`Bearer ${process.env.JEVSTATION_API_KEY}`&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="s2"&gt;"Content-Type"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"application/json"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;body&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;JSON&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;stringify&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;state&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;questions&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
      &lt;span class="n"&gt;risk&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="kd"&gt;type&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"score"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;instructions&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"How risky is this change to ship without a second human review?"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;criteria&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;
          &lt;span class="s2"&gt;"Mechanical change with no logic edit"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
          &lt;span class="s2"&gt;"Localized logic change covered by tests in the touched files"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
          &lt;span class="s2"&gt;"Logic change in a shared module, or no test for the touched path"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
          &lt;span class="s2"&gt;"Touches auth, payments, a data migration, or deletes data"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="n"&gt;tests&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="kd"&gt;type&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"noul"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;instructions&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
          &lt;span class="s2"&gt;"The diff adds or updates a test that would fail if this behaviour changed."&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;!&lt;/span&gt;&lt;span class="n"&gt;res&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;ok&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nb"&gt;console&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;error&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sb"&gt;`HTTP ${res.status}: ${await res.text()}`&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="nb"&gt;process&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;exit&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;body&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;res&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;json&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;data&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;body&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;data&lt;/span&gt; &lt;span class="o"&gt;??&lt;/span&gt; &lt;span class="n"&gt;body&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;            &lt;span class="c1"&gt;// the API wraps the result in "data"&lt;/span&gt;
&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;risk&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;tests&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;data&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;answers&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="nb"&gt;console&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;log&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
  &lt;span class="sb"&gt;`risk ${risk.score.toFixed(2)} (confidence ${risk.confidence.toFixed(2)}), `&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt;
  &lt;span class="sb"&gt;`tests ${tests.noul.toFixed(2)}`&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="nb"&gt;console&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;log&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sb"&gt;`credits: ${data.credits?.charged} charged, ${data.credits?.remaining} left`&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;needsHuman&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;risk&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;score&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;=&lt;/span&gt; &lt;span class="mf"&gt;2.5&lt;/span&gt; &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="n"&gt;risk&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;confidence&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="mf"&gt;0.6&lt;/span&gt; &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="n"&gt;tests&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;noul&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="mf"&gt;0.3&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;needsHuman&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nb"&gt;console&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;error&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"Route to a human reviewer before merge."&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="nb"&gt;process&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;exit&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Three choices in that script are deliberate. The threshold lives in one expression, so tuning it does not touch the questions. A truncated diff is marked with &lt;code&gt;[diff truncated]&lt;/code&gt; rather than silently clipped, because a state that quietly drops half the hunks produces a confident answer about the wrong change. And a missing key fails with the HTTP status instead of a stack trace, which is what you want in CI logs.&lt;/p&gt;

&lt;p&gt;Redact before you send. &lt;code&gt;state&lt;/code&gt; goes to the model for evaluation, so strip tokens, passwords and connection strings from the diff first, or the script becomes a way to leak them.&lt;/p&gt;

&lt;h2&gt;
  
  
  Three failure modes
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Confidence is not correctness.&lt;/strong&gt; A confidently wrong answer is still wrong. Start with thresholds that escalate more often than you think necessary, log the answers next to the outcome, then tighten. A &lt;a href="https://github.com/AbdelStark/jev-benchmarks" rel="noopener noreferrer"&gt;third-party pilot&lt;/a&gt; on 300 examples across three classification datasets found that calibration varied by dataset: Jev had a better Brier score on two and a worse one on the third. The author notes it is a pilot, not a leaderboard or a universal model ranking. That is another reason to treat confidence as a prior rather than a guarantee.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The score is not severity.&lt;/strong&gt; A weighted average hides which level carried the probability. A 2.4 with most mass on level 3 and some on level 1 can describe a change that deletes rows in a migration. When the score lands near a threshold, print the distribution and treat the scalar as only half the answer.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Aliases move under you.&lt;/strong&gt; &lt;code&gt;jev-latest&lt;/code&gt; points at the current release, so the answers your thresholds were tuned against can shift without a commit on your side. Log the versioned model id that answered, and pin it once the bands are stable.&lt;/p&gt;

&lt;p&gt;There is a fourth, quieter one: an MCP tool is a filter, not a security boundary. A guard tool that decides whether to run a command is useful in front of an approval prompt, and it does not replace the permissions underneath it or the allow-list in your harness.&lt;/p&gt;

&lt;h2&gt;
  
  
  Where this earns its keep
&lt;/h2&gt;

&lt;p&gt;The setup fits a specific job: deciding which changes deserve the careful reviewer first, when the alternative is a queue sorted by arrival time. It also fits routing, triage and release notes, where a branch in code is more useful than a paragraph.&lt;/p&gt;

&lt;p&gt;It does not fit style, formatting, or anything a linter already decides, and it cannot tell you a change is correct. The score is a second opinion with a number attached.&lt;/p&gt;

&lt;p&gt;Start with the three questions above on your last twenty merged diffs. Compare the scores with the incidents that followed, or with who ended up reviewing each change. If the bands hold, move the threshold into CI. If they do not, the questions need another level description, not a new model.&lt;/p&gt;

</description>
      <category>cursor</category>
      <category>mcp</category>
      <category>ai</category>
      <category>typescript</category>
    </item>
  </channel>
</rss>
