Revolutionizing Human-Computer Interaction with LLM Inference

#learnai #oxlo #ai

We are building a conversational system agent that turns plain English into structured desktop commands while remembering context across turns. This gives users a natural language layer on top of traditional GUIs and CLIs. Because the conversation history grows with every turn, Oxlo.ai's flat per-request pricing means long sessions do not trigger unexpected token costs. Details are at https://oxlo.ai/pricing.

What you'll need

An Oxlo.ai API key from https://portal.oxlo.ai
Python 3.10 or newer
The OpenAI SDK: pip install openai

Step 1: Define the system prompt and schema

The model needs to know exactly what format to return. I use a system prompt that forces a JSON schema with action, target, and parameters fields. This keeps the agent predictable and easy to parse.

SYSTEM_PROMPT = """You are a precise system command interpreter. The user controls their computer through natural language. Convert each user request into a JSON object with exactly these keys:
- action: one of [open, close, list, search, notify, unknown]
- target: the file, app, or item name
- parameters: a dict of extra flags or options
- reply: a short friendly confirmation message to show the user

Rules:
- Return ONLY the JSON object, no markdown fences.
- If the user refers to a previous turn, resolve pronouns using the conversation history.
- If the intent is unclear, set action to unknown and reply to ask for clarification.

Example:
User: Open my downloads folder
Output: {"action":"open","target":"downloads","parameters":{},"reply":"Opening your downloads folder."}
"""

Step 2: Initialize the Oxlo.ai client

We point the OpenAI SDK at Oxlo.ai's endpoint. I am using Llama 3.3 70B here because it follows structured instructions reliably, but you can swap in Qwen 3 32B or Kimi K2.6 without changing any other code.

from openai import OpenAI
import json

client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")

def call_agent(messages):
    response = client.chat.completions.create(
        model="llama-3.3-70b",
        messages=messages,
        temperature=0.2,
    )
    return response.choices[0].message.content

Step 3: Manage conversation history

Human-computer interaction falls apart without memory. We keep a message list that accumulates system, user, and assistant turns. Before each API call we truncate only if we exceed a generous limit, but because Oxlo.ai charges per request rather than per token, growing history does not increase cost.

MAX_HISTORY = 20

def run_turn(history, user_input):
    history.append({"role": "user", "content": user_input})
    raw = call_agent(history)
    history.append({"role": "assistant", "content": raw})
    # Trim history while keeping system prompt at index 0
    while len(history) > MAX_HISTORY + 1:
        history.pop(1)
    return raw

Step 4: Parse and execute mock actions

We parse the JSON and simulate execution. In a real deployment you would wire these to PyAutoGUI, AppleScript, or a window manager API. For this tutorial we print the structured intent so you can verify accuracy before hooking up real side effects.

def execute_intent(raw_json):
    try:
        data = json.loads(raw_json)
        action = data.get("action", "unknown")
        target = data.get("target", "")
        params = data.get("parameters", {})
        reply = data.get("reply", "Done.")

        print(f"[INTENT] action={action}, target={target}, parameters={params}")
        # Simulation only. Replace with real OS automation here.
        if action == "open":
            print(f"Simulating: launch {target}")
        elif action == "close":
            print(f"Simulating: terminate {target}")
        elif action == "list":
            print(f"Simulating: list contents of {target}")
        elif action == "search":
            print(f"Simulating: search for {target}")
        else:
            print("No executable action mapped.")

        return reply
    except json.JSONDecodeError:
        return "Sorry, I could not parse that response into a command."

Step 5: Wire the interactive loop

This ties the client, memory, and executor into a simple REPL. You can drive it from a terminal or wrap it in a web socket later.

def main():
    history = [{"role": "system", "content": SYSTEM_PROMPT}]
    print("System agent ready. Type 'exit' to quit.")
    while True:
        user_input = input("> ").strip()
        if user_input.lower() in {"exit", "quit"}:
            break
        if not user_input:
            continue

        raw = run_turn(history, user_input)
        reply = execute_intent(raw)
        print(reply)
        print()

if __name__ == "__main__":
    main()

Run it

Save the script as hci_agent.py, export your key, and run it. Here is a sample interaction showing multi-turn resolution.

$ export OXLO_API_KEY="sk-..."
$ python hci_agent.py
System agent ready. Type 'exit' to quit.
> Open the budget spreadsheet
[INTENT] action=open, target=budget spreadsheet, parameters={}
Simulating: launch budget spreadsheet
Opening your budget spreadsheet.

> Close it
[INTENT] action=close, target=budget spreadsheet, parameters={}
Simulating: terminate budget spreadsheet
Closing budget spreadsheet.

> Search for Q4 reports
[INTENT] action=search, target=Q4 reports, parameters={}
Simulating: search for Q4 reports
Searching for Q4 reports.

Wrap up

This agent is fully functional as a natural language control layer. Two concrete next steps: add real tool definitions via Oxlo.ai's function calling support so the model emits native JSON schemas instead of parsing free text, or integrate vision by switching to kimi-k2.6 and passing screenshots of the desktop so the agent can reason about what is actually on screen.