DEV Community

Cover image for The AI Has Hands
Hamed Hajiloo
Hamed Hajiloo

Posted on

The AI Has Hands

#dotnet #csharp #blazor #ai

Ask Crysta — the AI agent on CrystaCode.ai — to switch the site to dark mode, and the site actually turns dark. Ask it to show the login popup, and the modal actually opens. The model doesn't just answer anymore; it operates the UI.

But here's the thing: the LLM lives on the server, and the UI lives in the browser. A model can't click buttons. So how do you give a remote brain hands?

The answer is a pattern we ended up calling the Client Driver Skill: function calling, with SignalR as the hand.

The Problem

The first version of our chat was a one-way street. The model could say "sure, I'll take you to the plans page" — and then nothing happened. The answer was text; the UI was deaf.

You have two classic options:

  • The client polls the server for commands (ugly, wasteful, feels like 2010)
  • The server pushes commands to the client (real-time, instant, exactly what SignalR is for)

We went with the push. The flow became: the model calls a function → the function runs on the server → the server pushes a typed command over SignalR → the client executes it.

How It Works (The Full Loop) 

[Browser]                    [Server]
   |                             |
   | 1. "switch to dark mode"    |
   | ---- InvokeAsync ---------> |
   |                             | 2. Brain runs, model sees
   |                             |    the UpdateSiteTheme tool
   |                             | 3. Model calls the function
   |                             |    (function calling)
   |                             | 4. Push to the exact tab:
   | <--- ChangeSiteTheme ------ |    Clients.Client(connId)
   | 5. ThemeService flips it    |
   | 6. "Site theme changed..."  | 5b. return value re-enters
   |                             |     the model's context
   | <--- chat answer ---------- |
Enter fullscreen mode Exit fullscreen mode
  1. User types "switch to dark mode" → the Blazor client calls the hub
  2. The server session runs the brain; the model sees a tool called UpdateSiteTheme
  3. The model decides the user wants dark mode and calls the function
  4. The server pushes ChangeSiteTheme(DarkMode) to the exact browser connection
  5. The client applies the theme and re-renders
  6. The function's return value goes back into the model's context, so the AI knows the theme changed and confirms it in the chat

1) The Client Registers Its Remote Controls

On the client, our chat component subscribes to a set of hub events. Each one is a "remote control" the server can press:

HubConnection.On<SiteTheme>("ChangeSiteTheme", OnChangeTheme);
HubConnection.On("ShowLoginPopup", ShowLoginPopup);
HubConnection.On<CrystaPages>("NavigateToPage", HubOnNavigateToPage);
HubConnection.On("ShowInvitationModal", HanleOnShowInvitationModal);
Enter fullscreen mode Exit fullscreen mode

That's it. The UI doesn't poll, doesn't guess — it just waits for the server to press a button.

2) The Model's Tools Are Functions With Descriptions

The trick that makes this work: we turn our skill methods into LLM tools with AIFunctionFactory.Create. The [Description] attribute is the user manual the model reads — write it like an instruction, not a docstring:

[Description("When user ask to change site ui theme, call this method
  to change user ui, site theme can be dark or light mode")]
public async Task<string> UpdateSiteTheme(
    [Description("Site theme based on user request")] SiteTheme siteTheme)
{
    await OnUpdateSiteTheme(siteTheme);

    return $"Site theme changed by crysta to {siteTheme}.";
}
Enter fullscreen mode Exit fullscreen mode

Then the skill exposes its tool set:

public List<AIFunction> GetTools()
{
    return
    [
        AIFunctionFactory.Create(UpdateSiteTheme),
        AIFunctionFactory.Create(NavigateToPage),
        AIFunctionFactory.Create(ShowLoginPopup),
        AIFunctionFactory.Create(ShowMarkdown),
        AIFunctionFactory.Create(StartInitialFreeTrialSubscription),
        // ... every UI action the model is allowed to trigger
    ];
}
Enter fullscreen mode Exit fullscreen mode

The brain collects these from all registered skills and attaches them to the Microsoft.Extensions.AI chat pipeline. The model doesn't know (or care) that the effect happens in a browser somewhere — to it, these are just tools it can call.

3) The Server Pushes to the Exact Tab

This is the part that matters most. The function body runs on the server, and the effect must land in one specific browser tab:

protected override async Task OnUpdateSiteTheme(SiteTheme siteTheme)
{
    var lastConnectionId = await interactionCoordinator.GetLastConnectionId(UserSessionId);

    if (!string.IsNullOrWhiteSpace(lastConnectionId))
    {
        await HubContext.Clients.Client(lastConnectionId).ChangeSiteTheme(siteTheme);
    }
}
Enter fullscreen mode Exit fullscreen mode

Two details here are easy to miss, and both cost us debugging time:

Target the connection, not the user. Clients.User(userId) broadcasts to every tab and device a user has open. The command must land in the tab where the conversation is happening — so we track the session's last active connection id and push to Clients.Client(connectionId).

Type your hub. We use IHubContext<CrystaHub, ICrystaHubClient> — the client contract is a real C# interface:

public interface ICrystaHubClient
{
    Task ChangeSiteTheme(SiteTheme siteTheme);
    Task NavigateToPage(CrystaPages page);
    Task ShowLoginPopup();
    Task ShowMarkDown(string markdown);
    Task ShowInitialTrialSubscriptionStartedModal();
    // ...
}
Enter fullscreen mode Exit fullscreen mode

No magic strings, no typos killing a production demo at 2am. If the client and server disagree, it doesn't compile.

4) The Client Applies It

Back in the browser, the registered handler fires:

private async Task OnChangeTheme(SiteTheme theme)
{
    await Task.Delay(500);
    await themeService.ChangeSiteTheme(theme);
}
Enter fullscreen mode Exit fullscreen mode

One small detail: half a second of patience before the flip. We found that giving the UI a moment to settle after the chat bubble renders makes the theme transition feel intentional instead of jarring.

ThemeService does the actual work through the component library's theme manager, then notifies every component that cares via pub/sub:

public async Task ChangeSiteTheme(SiteTheme siteTheme)
{
    var current = await bitThemeManager.GetCurrentThemeAsync();
    var shouldBeDark = siteTheme == SiteTheme.DarkMode;

    if ((shouldBeDark && current != "dark") || (!shouldBeDark && current == "dark"))
    {
        await bitThemeManager.ToggleDarkLightAsync();
    }

    await bitDeviceCoordinator.ApplyTheme(shouldBeDark);
    pubSubService.Publish(ClientPubSubMessages.THEME_CHANGED, theme);
}
Enter fullscreen mode Exit fullscreen mode

5) Modals Are the Same Trick

The theme was the easy one. The fun part: opening a modal is just another hub event. The model decides the user needs to log in, calls ShowLoginPopup, and the server pushes the command:

private async Task ShowLoginPopup()
{
    _ = signInModalService.SignIn();
}
Enter fullscreen mode Exit fullscreen mode

Even richer: when the model wants to show details about a team member, it doesn't just navigate — it passes the data with the command:

HubConnection.On<TeamMemberDto>("OnNavigateToTeamMemberPage", dto =>
{
    AddComponentItem(typeof(CrystaMember), new() { { "TeamMemberDto", dto } },
        new PageInfo() { FullComponent = typeof(TeamMemberModal) });
});
Enter fullscreen mode Exit fullscreen mode

The model already had the data from a GetTeamInfo call — so the command carries a typed DTO and the client opens the exact modal with the data pre-filled. The AI describes the person in chat and a modal with their full profile appears at the same time.

6) The Reverse Channel: The UI Talks Back

The loop doesn't stop at server → client. Sometimes the UI needs to tell the model what happened — because the model's next sentence depends on it.

Example: the invitation-code modal closes. The client pushes an instruction straight into the model's context:

// user closed the modal without a code
var instructionText = """
    The user could not enter an invitation code.
    Tell them that if they don't have one, they can get it from their friends.
    """;

await HubConnection.InvokeAsync("InstructCrysta", instructionText);
Enter fullscreen mode Exit fullscreen mode

The server injects that instruction into the running session, the model responds to the actual UI event, and the answer streams back through the same chat channel. Browser and brain end up in a genuine two-way conversation — the UI has a voice too.

The Pattern, Summarized

  • LLM tools don't have to return data. They can trigger side effects. The model's "toolbox" is also its remote control.
  • SignalR is the hand. Server-side brain, browser-side UI, one typed channel between them.
  • Target the session's last connection, not the user. Multi-tab users will make you debug this for a day.
  • Type the hub. IHubContext<Hub, IClientContract> turns runtime typos into compile errors.
  • Return a status string. The model reads the function result — "Site theme changed to DarkMode" — so it can confirm what actually happened instead of guessing.
  • Build the reverse channel. Let the UI push instructions back (InstructCrysta), and the model's answers become consistent with what's really on screen.

That's the difference between "AI that answers" and "AI that acts" — and it's exactly what we built at CrystaCode.ai.

Top comments (0)