Cut Your AI Agent Prompt Size by 75% Using a Fast Decision Model (Jev / decision-1)

Vishal Shinde — Sat, 31 Oct 2026 10:21:03 +0000

How to split a bloated AI agent prompt into modular skills and use a fast Decision Model to route only the instructions and tools each user question actually needs.

When you build an AI assistant for a web application — say, an Online Travel & Flight Booking Assistant — it always starts simple. You write a short system prompt and attach one or two tools, like search_bookings and cancel_booking.

Fast-forward a few months. Your assistant now supports searching flights, switching UI tabs, applying multi-field filters, downloading boarding passes, checking refund eligibility, rebooking flights, and answering baggage policy questions. Suddenly, every single user message sends:

15,000+ characters of system prompt instructions covering every possible edge case
8 to 15 JSON Schema tool definitions
The live UI state and conversation history

Even when a user simply types "Switch to my Cancelled Trips tab", your application pays the token cost and latency of sending rules for baggage claims, refund penalties, and multi-city date filters to your primary LLM.

In this post, I’ll share how we used a lightweight, ultra-fast Decision Model (decision-1 / Jev) as a skill router in front of our primary LLM—cutting prompt size by 60% to 76% on every turn.

What Is a Decision Model (`decision-1` / Jev)?

Unlike a large generative model that writes free-form paragraphs, a Decision Model is a small, specialized model designed to answer a batch of binary (yes / no) classification questions against a given context in a single fast pass.

Instead of asking a heavyweight LLM to read 15,000 characters of instructions and 10 tool schemas on every turn, we use a two-stage pipeline:

Stage 1 — Skill Selection (~300–700ms): Send the user’s recent conversation and a list of skill descriptions to decision-1. It evaluates all skills in parallel and returns yes or no for each skill.
Stage 2 — Tailored LLM Call: Build the system prompt and tool list using only the skills where decision-1 answered yes, and send that compact payload to the primary LLM.

Step 1: Divide Your Monolithic Prompt into Skills

Instead of one giant system prompt, we group domain instructions and tools into self-contained Skills. Each skill has:

An ID (e.g., NAVIGATE_TAB, FILTER_BOOKINGS)
A concise 1-sentence description that tells the Decision Model when to select yes
The tool names associated with that skill
The detailed instructions that should be injected into the main LLM prompt only when that skill is selected

A key lesson we learned: don’t pass the entire multi-paragraph skill instructions to the Decision Model (it slows down evaluation), and don’t maintain separate custom yes and no rules per skill (it doubles maintenance). A single 1-sentence description per skill is fast, accurate, and easy to maintain:

public enum BookingSkill {

    SEARCH_BOOKINGS(
            "SEARCH_BOOKINGS",
            """
            Select 'yes' when the user asks to find, list, look up, or check the status of their flight or hotel bookings.
            """,
            Set.of("search_bookings"),
            """
            ### Searching Bookings (search_bookings)
            - Always call search_bookings before quoting PNR codes, flight numbers, or departure times.
            - Present results in a clean Markdown table with columns: PNR, Route, Date, and Status.
            """),

    NAVIGATE_TAB(
            "NAVIGATE_TAB",
            """
            Select 'yes' when the user asks to switch views or open the Upcoming, Past, or Cancelled bookings tab.
            """,
            Set.of("navigate_tab"),
            """
            ### Navigating Tabs (navigate_tab)
            - Valid tabs: UPCOMING, PAST, CANCELLED.
            - Check the current activeTab in the UI context first; if already on the target tab, do not call the tool.
            """),

    FILTER_BOOKINGS(
            "FILTER_BOOKINGS",
            """
            Select 'yes' when the user asks to filter bookings by airline, destination city, date range, or passenger name, or clear filters.
            """,
            Set.of("filter_bookings"),
            """
            ### Filtering Bookings (filter_bookings)
            - Map city names to 3-letter IATA airport codes when available.
            - Pass clearAll: true when the user asks to reset or remove active filters.
            """),

    REFUND_OR_REBOOK(
            "REFUND_OR_REBOOK",
            """
            Select 'yes' when the user asks to cancel a trip, request a refund, or change their flight date or time.
            """,
            Set.of("check_refund_eligibility", "request_refund"),
            """
            ### Refunds & Rebooking (check_refund_eligibility, request_refund)
            - Always call check_refund_eligibility first and show the breakdown to the user.
            - Never invoke request_refund without explicit user confirmation of the penalty amount.
            """);

    private final String id;
    private final String description;
    private final Set<String> toolNames;
    private final String instructions;

    BookingSkill(String id, String description, Set<String> toolNames, String instructions) {
        this.id = id;
        this.description = description;
        this.toolNames = Set.copyOf(toolNames);
        this.instructions = instructions;
    }
}

Step 2: How Skills Are Sent to the Decision Model

The Decision Model API (/v1/systemone) accepts a JSON request with three fields:

model: "decision-1"
context: The recent conversation turns plus the latest user question
questions: A map where each key is a Skill ID and each value defines the yes and no criteria

We map each skill’s description to "yes" and use a single shared constant for "no" across all skills:

static final String NO_CRITERION =
        "Select 'no' when this skill is not needed to answer the user's request.";

static Map<String, Map<String, String>> buildQuestions() {
    return Arrays.stream(BookingSkill.values())
            .collect(Collectors.toMap(
                    BookingSkill::getId,
                    skill -> Map.of(
                            "yes", skill.getDescription().strip(),
                            "no", NO_CRITERION
                    )
            ));
}

Sample Request to `decision-1`

Suppose the user asks: "Can you take me to my cancelled flights and filter by London?"

Here is the exact JSON payload sent to the Decision Model:

{
  "model": "decision-1",
  "context": "Recent conversation:\nUSER: Can you take me to my cancelled flights and filter by London?\n\nLatest user question:\nCan you take me to my cancelled flights and filter by London?",
  "questions": {
    "SEARCH_BOOKINGS": {
      "yes": "Select 'yes' when the user asks to find, list, look up, or check the status of their flight or hotel bookings.",
      "no": "Select 'no' when this skill is not needed to answer the user's request."
    },
    "NAVIGATE_TAB": {
      "yes": "Select 'yes' when the user asks to switch views or open the Upcoming, Past, or Cancelled bookings tab.",
      "no": "Select 'no' when this skill is not needed to answer the user's request."
    },
    "FILTER_BOOKINGS": {
      "yes": "Select 'yes' when the user asks to filter bookings by airline, destination city, date range, or passenger name, or clear filters.",
      "no": "Select 'no' when this skill is not needed to answer the user's request."
    },
    "REFUND_OR_REBOOK": {
      "yes": "Select 'yes' when the user asks to cancel a trip, request a refund, or change their flight date or time.",
      "no": "Select 'no' when this skill is not needed to answer the user's request."
    }
  }
}

Sample Response from `decision-1`

In a single call, decision-1 evaluates all four skills in parallel and returns its decisions:

{
  "model": "decision-1",
  "decisions": {
    "SEARCH_BOOKINGS": "no",
    "NAVIGATE_TAB": "yes",
    "FILTER_BOOKINGS": "yes",
    "REFUND_OR_REBOOK": "no"
  }
}

Notice that because each skill is evaluated independently, decision-1 naturally handles multi-intent queries—selecting both NAVIGATE_TAB and FILTER_BOOKINGS while excluding SEARCH_BOOKINGS and REFUND_OR_REBOOK.

Step 3: Filtering the Prompt and Tools

Once decision-1 returns the map of decisions, we filter both the system prompt sections and the tool definitions before calling the primary LLM:

// 1. Keep only skills where decision-1 returned "yes"
List<BookingSkill> selectedSkills = Arrays.stream(BookingSkill.values())
        .filter(skill -> "yes".equalsIgnoreCase(response.decisions().get(skill.getId())))
        .toList();

// 2. Assemble system prompt with only the selected skill instructions
String tailoredSystemPrompt = BASE_PROMPT + "\n\n" + selectedSkills.stream()
        .map(BookingSkill::getInstructions)
        .collect(Collectors.joining("\n\n"));

// 3. Expose only the tools belonging to the selected skills
Set<String> allowedToolNames = selectedSkills.stream()
        .flatMap(skill -> skill.getToolNames().stream())
        .collect(Collectors.toSet());

List<Tool> tailoredTools = allTools.stream()
        .filter(tool -> allowedToolNames.contains(tool.name()))
        .toList();

If the user’s message is conversational (e.g., "Thanks!") and all skills return "no", the primary LLM receives only the lightweight base persona prompt with zero tool schemas. And if the Decision Model ever times out or is unreachable, we simply fall back to the full prompt and tool set so the user experience never breaks.

Step 4: Real Impact in Production Logs

To measure the savings, we logged the character count and tool count of the primary LLM input without Decision-1 versus with Decision-1:

INFO BookingSkillRouter : Decision-1 selected skills for thread d34893b3-73d2-466c-949b-c2ddad5161aa in 775 ms: [NAVIGATE_TAB]
INFO BookingSkillRouter : Input size for thread d34893b3-73d2-466c-949b-c2ddad5161aa:
  without Decision-1 = 17567 chars (4 tools),
  with Decision-1    = 7052 chars (1 tools),
  saved              = 10515 chars (-60%)

INFO BookingSkillRouter : Decision-1 selected skills for thread d34893b3-73d2-466c-949b-c2ddad5161aa in 884 ms: [NAVIGATE_TAB]
INFO BookingSkillRouter : Input size for thread d34893b3-73d2-466c-949b-c2ddad5161aa:
  without Decision-1 = 13783 chars (4 tools),
  with Decision-1    = 3268 chars (1 tools),
  saved              = 10515 chars (-76%)

Key Takeaways

60% to 76% prompt size reduction: By stripping out irrelevant instructions and unused tool schemas, every turn sends a fraction of the tokens to your main LLM.
Better tool selection by the main LLM: When the primary LLM only sees 1 or 2 relevant tools instead of 10+, it makes fewer parameter mistakes and avoids calling the wrong tool.
Easy to scale: Adding a 10th or 15th skill to your agent no longer bloats every request — each new skill only adds a single 1-sentence question to decision-1, while the main LLM prompt stays lean.

DEV Community: Vishal Shinde