Automate Your Health: Building an AI Medical Agent with GPT-4 Function Calling and Google Calendar

We’ve all been there: you’re feeling under the weather, but the thought of navigating a clunky hospital portal or staying on hold for 20 minutes to book a doctor's appointment feels like another symptom. What if you could just say, "I have a sharp pain in my lower back, find me an orthopedic specialist for next Tuesday afternoon," and have a smart AI agent handle the rest?

In this tutorial, we are building a Healthcare Agent Controller. By leveraging OpenAI Function Calling, AI Agents, and the Google Calendar API, we will create a system that understands natural language, queries doctor availability, cross-references your personal schedule, and books the appointment—all via a clean FastAPI backend. This is the future of Healthcare Automation and personalized AI assistants.

The Architecture: How the Agent Thinks

To build a reliable agent, we need a clear separation between the "Brain" (LLM) and the "Hands" (APIs). Here is the data flow:

sequenceDiagram
    participant User
    participant FastAPI as API Gateway
    participant OpenAI as GPT-4o (The Brain)
    participant GCal as Google Calendar API (The Hands)

    User->>FastAPI: "Book a dentist for next Monday morning"
    FastAPI->>OpenAI: Prompt + Tools Definition
    OpenAI->>OpenAI: Analyze Intent & Extract Date/Type
    OpenAI-->>FastAPI: Call Function: search_doctors(specialty, date)
    FastAPI->>GCal: GET /events (Check slots)
    GCal-->>FastAPI: Returns Available Slots
    FastAPI->>OpenAI: "Found 10:00 AM and 11:30 AM"
    OpenAI-->>FastAPI: Call Function: create_appointment(time)
    FastAPI->>GCal: POST /events (Book it!)
    GCal-->>FastAPI: Confirmation
    FastAPI-->>User: "Success! You're booked for 10:00 AM."

Prerequisites

Before we dive into the code, ensure you have the following in your tech_stack:

• OpenAI API Key: For GPT-4o access.
• Google Cloud Console: Enable Google Calendar API and download your credentials.json.
• FastAPI & Uvicorn: For the web server.
• Python-dotenv: To manage our secrets.

Step 1: Define the Tools (Function Calling)

Function calling allows GPT-4 to describe the arguments for functions you've defined in your code. We define a tool to fetch availability and another to create the event.

# tools.py
tools = [
    {
        "type": "function",
        "function": {
            "name": "get_calendar_availability",
            "description": "Check if a specific time slot is free on the doctor's calendar",
            "parameters": {
                "type": "object",
                "properties": {
                    "start_time": {"type": "string", "description": "ISO format string"},
                    "end_time": {"type": "string", "description": "ISO format string"},
                },
                "required": ["start_time", "end_time"],
            },
        },
    },
    {
        "type": "function",
        "function": {
            "name": "book_appointment",
            "description": "Create a new calendar event for the medical appointment",
            "parameters": {
                "type": "object",
                "properties": {
                    "summary": {"type": "string", "description": "Reason for visit"},
                    "start_time": {"type": "string"},
                    "end_time": {"type": "string"},
                },
                "required": ["summary", "start_time", "end_time"],
            },
        },
    }
]

Step 2: The Google Calendar Logic

We need a helper to interact with Google. For brevity, we'll assume you've handled the OAuth2 flow.

from googleapiclient.discovery import build

def create_appointment(summary, start_time, end_time):
    # service is a pre-authenticated Google Calendar service object
    event = {
        'summary': summary,
        'start': {'dateTime': start_time, 'timeZone': 'UTC'},
        'end': {'dateTime': end_time, 'timeZone': 'UTC'},
    }
    return service.events().insert(calendarId='primary', body=event).execute()

Step 3: The FastAPI Agent Controller

This is where the magic happens. We receive a user request, send it to OpenAI with our tools, and execute the functions the model requests.

from fastapi import FastAPI
from openai import OpenAI
import os

app = FastAPI()
client = OpenAI(api_key=os.getenv("OPENAI_API_KEY"))

@app.post("/schedule")
async def handle_medical_request(user_input: str):
    # 1. Send request to OpenAI
    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": user_input}],
        tools=tools,
        tool_choice="auto"
    )

    message = response.choices[0].message

    # 2. Check if the model wants to call a function
    if message.tool_calls:
        for tool_call in message.tool_calls:
            if tool_call.function.name == "book_appointment":
                # Extract arguments and call the real GCal function
                args = json.loads(tool_call.function.arguments)
                result = create_appointment(args['summary'], args['start_time'], args['end_time'])
                return {"status": "success", "data": result}

    return {"status": "clarification_needed", "message": message.content}

Elevating Your Agent Architecture

Building a basic "happy path" is easy, but medical data is sensitive. In a production environment, you need robust HIPAA-compliant data handling, advanced RAG (Retrieval-Augmented Generation) for medical knowledge, and complex state management for multi-turn conversations.

For a deeper dive into production-ready agent patterns and securing LLM-based healthcare workflows, check out the specialized guides over at WellAlly Blog. They cover advanced topics like prompt injection prevention and multi-agent orchestration that are essential for taking this from a hobby project to a real-world tool.

Conclusion

By combining the reasoning power of GPT-4 with the structural reliability of Function Calling, we've transformed a tedious administrative task into a seamless conversation.

Next Steps:

1. Add a "Personal Calendar" check to ensure the doctor's appointment doesn't clash with your existing meetings.
2. Implement a Twilio integration to send SMS confirmations.
3. Add a "Specialist Lookup" tool using a local database or a medical directory API.

Are you building AI agents for healthcare or productivity? Drop a comment below or share your repo—I'd love to see how you're implementing these patterns!