DEV Community

Cover image for Scoring A2A Agent Skills with System One (Jev) in .NET
ohalay
ohalay

Posted on

Scoring A2A Agent Skills with System One (Jev) in .NET

#tutorial #agents #dotnet #ai

Follow-up to

Multi-Agent Orchestration in .NET Using A2A
ohalay profile
ohalay

Multi-Agent Orchestration in .NET Using A2A

#ai #agents #tutorial #dotnet
1
Comments 3
3 min read

There, the orchestrator turned every discovered AgentCard into an AIFunction and let one LLM loop do the routing.

The problem

tools = agents.Select(ToTool) means the prompt grows linearly with the number of agents, and each tool description is a blob of the whole card:

Ask the SupplyChainAnalyst specialist. 
Manages warehouse logistics, stock levels, inbound shipment delivery statuses,
and DWH stock-velocity tracking.
Skills: GetStock, GetShipments.
Enter fullscreen mode Exit fullscreen mode

Worse, that blob is re-sent on every iteration of the tool loop, not once per request.

Two consequences:

  • prompt size scales with agents, not with the request;
  • the agent is described at card granularity while the request needs one skill.

System One (Jev)

System One is a classification primitive, not a chat model. You send a
state plus a map of typed questions, and get one typed answer per key. Three primitives:

  • noul - TypeSafe's name for the boolean primitive; returns the probability of "yes" rather than a yes/no token,
  • choice (pick one + distribution),
  • score (ordered rubric, probability-weighted).

Answers come back structured, so there is nothing to parse out of prose.

We use it as a pre-LLM gate: score each agent skill against the request, expose only the winners.

1. Categorization and scoring for tools

The unit of selection is the skill, not the agent. ToolCatalog flattens every card into scorable rubrics:

private static string BuildRubric(RemoteAgent agent, string name, string? description, IReadOnlyList<string>? tags)
{
    var rubric = string.IsNullOrWhiteSpace(description) ? name : $"{name} — {description}";
    rubric = $"{agent.Card?.Name}: {rubric}";
    if (tags is { Count: > 0 })
        rubric += $" (topics: {string.Join(", ", tags)})";
    return rubric; 
}
Enter fullscreen mode Exit fullscreen mode

One ScoreQuestion per skill, all in one HTTP call:

questions[skill.Key] = new ScoreQuestion
{
    Instructions = new
    {
        question = "How relevant is this skill to answering the user's latest request?",
        skill = skill.Rubric,
    },
    Criteria =
    [
        "Not needed; the request can be answered fully without this skill.",
        "Needed; the request (or part of it) requires this skill.",
    ],
};
Enter fullscreen mode Exit fullscreen mode

Using exactly two criteria makes the score a 0..1 relevance probability, directly comparable to RelevanceThreshold (default 0.6). Add a third criterion and the score rescales - 0.6 silently stops meaning what it did.

Request - POST /v1/systemone for "How much stock is left for the winter coat?". The state also carries the last few turns, so follow-ups like "and the shipments?" still score correctly:

{
  "model": "jev-latest",
  "state": {
    "latest_user_message": "How much stock is left for the winter coat?",
    "conversation": [
      { "role": "user", "content": "..." },
      { "role": "assistant", "content": "..." }
    ]
  },
  "questions": {
    "GetProduct": {
      "type": "score",
      "instructions": {
        "question": "How relevant is this skill to answering the user's latest request?",
        "skill": "AssortmentSpecialist: GetProduct — Look up a product's SKU, category, active status, and store coverage by name. (topics: catalog, assortment, product)"
      },
      "criteria": [
        "Not needed; the request can be answered fully without this skill.",
        "Needed; the request (or part of it) requires this skill."
      ]
    },
    "GetActiveCatalog": { "type": "score", "instructions": { "...": "..." }, "criteria": ["...", "..."] },
    "GetStock":         { "type": "score", "instructions": { "...": "..." }, "criteria": ["...", "..."] },
    "GetShipments":     { "type": "score", "instructions": { "...": "..." }, "criteria": ["...", "..."] }
  }
}
Enter fullscreen mode Exit fullscreen mode

Response - same keys, typed answers. The jev-latest alias resolves to a pinned version, so you can log exactly what scored:

{
  "model": "jev-1.13.0",
  "answers": {
    "GetProduct":       { "type": "score", "score": 0.21, "confidence": 0.88, "probabilities": { "0": 0.79, "1": 0.21 }, "legend": { "0": "Not needed...", "1": "Needed..." } },
    "GetActiveCatalog": { "type": "score", "score": 0.06, "confidence": 0.95, "probabilities": { "0": 0.94, "1": 0.06 } },
    "GetStock":         { "type": "score", "score": 0.96, "confidence": 0.93, "probabilities": { "0": 0.04, "1": 0.96 } },
    "GetShipments":     { "type": "score", "score": 0.44, "confidence": 0.61, "probabilities": { "0": 0.56, "1": 0.44 } }
  },
  "usage": { "input_tokens": 512, "output_tokens": 24 }
}
Enter fullscreen mode Exit fullscreen mode

GetStock clears 0.6, so only SupplyChainAnalyst becomes a tool. The assortment agent is never offered. Ask "which stores carry it, and how much stock is left?" and GetProduct also clears - both agents are exposed, and the parallel fan-out from part 1 still happens.

2. Prompt size and relevance

Fewer tools, and a narrower description per surviving tool - built only from the skills that scored:

if (selectedSkills.TryGetValue(agent.Card.Name!, out var kept) && kept.Count > 0)
    return $"Ask the {agent.Card.Name} specialist. Relevant capabilities: {skillText}";

// TypeSafe off / fallback → full card + all skills (part-1 behavior)
return $"Ask the {agent.Card.Name} specialist. {agent.Card.Description} Skills: {allSkills}.";
Enter fullscreen mode Exit fullscreen mode

3. Architecture change

One step inserted before the loop. OrchestrationService gains an IToolSelector:

var catalog   = ToolCatalog.FromAgents(await registry.GetAgents(ct));
var selection = await toolSelector.SelectAsync(catalog, userMessage, thread.Turns, ct);
var tools     = selection.Agents.Select(a => ToTool(a, selection.SelectedSkills)).Cast<AITool>().ToList();
Enter fullscreen mode Exit fullscreen mode 
graph LR
    User([User]) --> Cat[ToolCatalog<br/>cards → skill rubrics]
    Cat --> Sel{{IToolSelector}}
    Sel -. score skills .-> TS[(System One)]
    Sel -->|surviving tools only| LLM[LLM tool loop]
    LLM -- A2A --> A[Assortment]
    LLM -- A2A --> S[SupplyChain]

IToolSelector is an interface for a reason: without an API key the app registers a pass-through
AllToolsSelector and behaves exactly like part 1, so the gate is also its own off-switch.

Conclusion

  • Extra cost is one classifier call. It grows with the number of skills, but it replaces N tool descriptions re-sent on every loop iteration with N score questions sent once - linear growth moves off the expensive path onto the cheap one.
  • Extra latency is one blocking hop before the first token, and in an otherwise local-inference stack it is the only external dependency on the chat path.
  • What improved: the tool prompt now scales with the request, not with the fleet.
  • The full implementation is in the repo.

Top comments (0)