DEV Community: Quoc Bao An Nguyen

How I Built an AI Agent with LLM Function Calling (and Avoided Unnecessary Tool Calls)

Quoc Bao An Nguyen — Thu, 01 Oct 2026 00:40:17 +0000

1. Introduction

When I first started building my AI-powered course recommendation system, I thought integrating an LLM with backend APIs would be straightforward.

However, I quickly ran into a key problem:
The model was calling backend APIs almost every time—even when it wasn’t necessary.

This led to:

Increased latency
Unnecessary API calls
Inefficient system behavior

In this post, I’ll walk through how I redesigned my AI Agent to behave more intelligently using controlled tool-calling and multi-step reasoning.

2. The Problem

2.1 Initial Design (Naive Approach)

My initial architecture looked like this:

User Input → LLM → Function Call → Backend API → LLM → Response

The idea was simple:

Let the model decide which function to call
Always execute the function if suggested

2.2 What Went Wrong

In practice, this caused several issues:

The model triggered function calls even for simple messages like “Hello”
Redundant API calls increased backend load
No clear control over when tools should be executed
Poor user experience due to unnecessary delays

At this point, I realized:

Letting the LLM fully control execution without constraints leads to inefficient systems.

3. Key Insight

The turning point was understanding this:

An AI Agent should not just “call tools” — it should decide when NOT to call them.

This meant I needed:

A decision layer
Controlled execution logic
Better orchestration between LLM and backend

4. System Redesign

4.1 New Architecture

Instead of blindly executing tool calls, I redesigned the system:

User Input

→ LLM (intent + decision)
→ Orchestration Layer
→ (Conditional) Backend Tool Execution
→ LLM Final Response

4.2 Decision-Based Tool Calling

I introduced logic where:

Simple inputs → direct response (no tool call)
Informational queries → fetch data via API
Complex actions → multi-step reasoning before execution

Example:

Input	Behavior
“Hello”	No tool call
“What courses do you have?”	Call course API
“Enroll me in a backend course”	Gather info → then call tool

4.3 Deferred Execution (Important)

Instead of calling tools immediately, the agent:

Identifies missing information
Asks follow-up questions
Executes the function only when all parameters are available

This significantly reduced unnecessary calls.

5. Backend as Tools

On the backend (ASP.NET Core), I designed APIs as callable tools:

GetCourses()
ValidateUser()
EnrollCourse()

Each function was exposed with structured input/output schemas so the LLM could interact with them safely.

6. Prompt & Orchestration Strategy

To improve behavior, I refined:

System prompts (clear tool usage rules)
Function descriptions (explicit intent)
Context handling (multi-turn conversations)

Goal:

Improve intent recognition
Reduce incorrect tool selection
Maintain consistent responses

7. Evaluation & Testing

Since I didn’t have large-scale production traffic, I evaluated the system using:

Simulated user inputs (various intent scenarios)
Multi-turn conversation testing
Edge cases (incomplete or ambiguous queries)

Key observations:

Reduced unnecessary tool calls
Improved response consistency
Better handling of complex requests

8. Trade-offs

Every design has trade-offs:

Pros:

More efficient API usage
Better user experience
Clearer control over system behavior

Cons:

Increased system complexity
Requires careful prompt and flow design
Harder to debug than simple chatbot systems

9. What I Learned

This project changed how I think about AI systems:

LLMs should be controllers, not executors
Backend systems should be tools, not just APIs
Good AI systems require orchestration, not just prompts

10. Conclusion

Building an AI Agent is not just about calling an API.

It’s about designing a system where:

Decisions are controlled
Execution is efficient
Behavior is predictable

If you’re building AI-powered applications, focus less on “what the model can do”
and more on how your system controls it.

11. Future Improvements

Add conversation memory (persistent context)
Integrate vector search for better recommendations
Introduce performance metrics (latency, tool-call rate)
Optimize system for real-world scaling

12. Final Thoughts

This project pushed me beyond just using LLMs —
it helped me think like an engineer designing systems around them.

And that’s where real value comes from.

From MediaPipe to Adaptive AI: Building Mendly's Rehab CV Pipeline

Quoc Bao An Nguyen — Wed, 30 Sep 2026 21:42:46 +0000

How our Top 11 HopHacks project turned noisy MediaPipe pose data into reliable rehabilitation metrics for adaptive AI planning

What if a rehabilitation exercise could understand how a patient moved, instead of only knowing whether they finished?

That was one of the ideas behind Mendly, an AI-powered stroke rehabilitation platform my team built during HopHacks.

By the end of the hackathon, Mendly became a Top 11 finalist out of 87 projects and won Best Use of DigitalOcean.

I worked mainly on the computer vision side of the project: pose tracking, joint-angle and range-of-motion measurement, valid-repetition detection, tracking reliability, and the pipeline that turns movement into structured performance data for adaptive AI planning.

Our core flow looked roughly like this:

Camera
  ↓
MediaPipe Pose
  ↓
Joint Angles / Range of Motion
  ↓
Repetition Validation
  ↓
Structured Performance Data
  ↓
Future AI Planning

The interesting part was not simply getting MediaPipe to detect a person.

It was figuring out when we should actually trust what the camera was telling us.

Missing tracking should stay missing

One bug changed how I thought about computer vision.

Suppose the tracker sees:

Frame 1: 40°
Frame 2: 55°
Frame 3: tracking fails

What should Frame 3 become?

At first, two obvious choices are:

use 0°
reuse the previous 55°

Both are wrong.

If tracking failed, we do not know where the patient's arm actually was.

Using zero invents movement. Reusing the old value assumes the patient stopped moving.

So we changed failed inference into an explicit invalid observation.

Pose succeeds
→ process measurement

Pose fails
→ ignore the observation
→ do not update movement state
→ do not count a repetition

The rule became:

Unknown is not zero. Unknown stays unknown.

That sounds simple, but it prevented missing camera evidence from becoming fake movement data.

Counting reps is more than crossing a threshold

Another challenge was repetition detection.

Real movement data does not look perfectly clean:

155°
148°
139°
121°
97°
76°
81°
95°
119°
142°
151°

If we count a repetition every time an angle crosses a threshold, noise or partial motion can create false reps.

Instead, we treated a repetition as a full movement cycle:

Start
  ↓
Movement begins
  ↓
Required range reached
  ↓
Optional hold
  ↓
Return
  ↓
Valid repetition

This also helped us separate attempts from valid repetitions.

At one point, an exercise could finish after enough failed attempts even if the patient had not reached the required number of valid reps.

We changed that so automatic completion depends only on:

validRepCount >= targetRepCount

If the target is five valid reps, the patient needs five valid reps.

Failed attempts can still be recorded, but they do not magically complete the exercise.

Recorded data is not always trusted data

We ran into the same distinction with range-of-motion measurements.

A patient might complete an attempt while tracking confidence is poor.

That attempt can still be useful as history, but its movement measurement should not automatically become trusted ROM data.

So we separated:

Did an attempt happen?

from:

Do we trust the measurement?

Low-confidence repetitions could remain in the result history while being excluded from trusted movement statistics.

That ended up being one of the most important ideas in the project:

Recorded does not necessarily mean reliable.

The LLM never sees the raw video

One architectural decision I liked about Mendly was keeping computer vision separate from AI reasoning.

Gemini does not inspect the patient's camera feed or count repetitions.

The CV layer converts movement into structured performance information first.

Conceptually:

{
  "exercise": "arm_flexion",
  "attempts": 9,
  "valid_repetitions": 7,
  "range_of_motion": 83,
  "tracking_quality": "good"
}

The computer vision layer answers:

What happened?

The AI planning layer answers:

How should that performance influence a future exercise set?

The patient's current session still follows a practitioner-approved plan.

When a future set is drafted, Mendly uses Backboard and Gemini with practitioner-defined constraints and accumulated performance data.

The proposal then goes through deterministic guardrails and practitioner review before it reaches the patient.

So the model assists with planning, but it does not autonomously decide treatment in real time.

What I took away from the project

Before Mendly, it was easy for me to think about computer vision as:

input → model → prediction

After debugging this system, I started thinking more about:

input
  ↓
prediction
  ↓
confidence
  ↓
validation
  ↓
structured data
  ↓
application behavior

The model is only one part of the system.

The engineering around it determines whether its output is actually useful.

For me, the biggest lesson was:

Good AI is not only about better models. It is also about giving those models reliable information to reason over.

That was the most interesting part of building Mendly - and probably the part I learned the most from.