Rethinking LLM-Powered Apps: Ditching Tool Overload for Smarter Query Abstraction

#ai #rag #llm #sdl

I had one problem to solve, which is exposing system to LLM so that user queries can be addressed in a modern semantic way instead of just relying calling bunch of REST API calls. Preparing the system readiness for accessing it via any modality.

Initially I thought of writing bunch of tools and attaching it to LLM would solve the problem. It turned out that we cannot write tools for every functionality of the system also overloading the LLM with more number of tools will results in poor/improper tool selection leading to lot of problem.

Why Tools Don't Scale in LLM Routing from what I've seen in my projects:

Tool Selection Struggles: The LLM has to pick from a growing list, which leads to mistakes, delays, and wonky outputs. Once you cross 15-20 tools, it gets messy—the context overloads, and accuracy dips.
Multi-Step Messes: Complex stuff requires chaining tools (fetch → verify → process), each a separate call. That means more back-and-forth, higher latency, and more spots for errors.
Upkeep Nightmares: Documenting and securing all those tools is a pain. One wrong call, and you've got a security hole.

Basically, overloading the LLM with tools turns it into an inefficient mess. There has to be a better way.

Then I asked this question: Before LLM comes in, If I want to expose my system to outside world I will do one or both of the following:

Exposing an REST APIs
Exposing a Client SDK that abstracts away all the complexity of API, Permission, Etc..

I thought why not bring it back to LLM specifically the SDK part, And since LLMs are very good at coding, writing a code using SDK is going to be very much efficient compared to bunch of back and forth tool calls.

Abstracting System as a Query Language via SDK:

I wrote a complete OOP SDK by abstracting out all the functionality (Entity based) of my app and wrote simple developer documentation on how to use this SDK.

Why it works:

LLM Efficiency: No tool-picking headache—the LLM just writes a query based on intent. It's quicker and more precise.
Faster and More Reliable: One query beats multiple tool hops, cutting down on delays and failures.
Built-in Security: RBAC is in the engine. It uses user context (like ID and roles) to enforce access. The LLM generates the query but never sees unauthorized stuff—the engine filters it all. Scales Effortlessly: Extend the SDK for new features. No tool proliferation.

Query SDK for Project Management system:

Picture a project management app. A manager says: "Show me weekly stats for my project, tailored to my view."

Old way: LLM calls tools for fetch, permissions, analysis—clunky and error-prone.

New way: LLM crafts a JS query with the SDK. Engine executes, applies RBAC (e.g., manager sees budgets, others don't), returns data. LLM then customises the output—charts, tables, whatever fits.

Sample query the LLM might generate:

// SDK Query (LLM-Generated)
const app = require('project-sdk'); // Import the SDK

// Fetch project for the user (RBAC handled automatically)
const project = app.getProjectForUser("userId"); // userId from session/context

// Get weekly breakdown
const result = project.getWeeklyAnalysis({
  metrics: ['teamHours', 'taskCompletionRate', 'budgetVariance'],
  period: 'last7Days'
});

// Aggregate if it makes sense
const aggregated = result.aggregateByTeam();

// Return for LLM to format
return aggregated;

Engine side:

getProjectForUser only gives accessible projects.
Analysis checks roles—no access, no data.
Fails safely if needed.

LLM takes the results and tailors Dashboard viz for managers, basic summary for teams, or different formats like pies or CSVs. Nails the use case, keeps the LLM focused on query gen and output styling.