Building a production-ready AI backend in Go that combines streaming responses with tool-calling poses a distinct challenge. Many LLM SDKs are either Python-first, overly abstracted, or require complex manual orchestration to handle token-by-token streaming and dynamic tool invocation simultaneously. In a Go web application, you often need to manage channels, SSE endpoints, and lifecycle events for both text generation and tool calls — a setup that quickly becomes brittle without a dedicated SDK. This is where the Grafana AI SDK for Go steps in. It provides a unified interface for streaming LLM responses, defining custom tools via a simple schema, and automatically routing tool invocations back to your handler functions. With it, you can focus on application logic rather than low-level API wrangling. In this tutorial, you will build a full-stack AI chat feature: a Go backend that streams LLM tokens and tool call results over Server-Sent Events, paired with a React frontend that renders the stream in real time. By the end, you’ll have a reusable pattern for adding conversational AI with tool support to any Go‑based web project.
Prerequisites and Project Setup
Before diving into the code, ensure your environment has the necessary tools. You'll need Go 1.21 or later — check your version with go version. If you don't have Go installed, download it from the official website. This version is required for the Grafana AI SDK's generics and error-handling features.
Next, initialize a Go module for your backend project:
go mod init my-ai-backend
Install the Grafana AI SDK package:
go get github.com/grafana/ai-sdk
This SDK provides high-level abstractions for LLM streaming and tool-calling, reducing boilerplate.
For the frontend, create a new React application using Vite (recommended for its speed). Run:
npm create vite@latest my-ai-frontend -- --template react
cd my-ai-frontend
npm install
This gives you a minimal React setup ready to connect to your Go backend. Keep both projects open: you'll build the backend in the my-ai-backend directory and later integrate the frontend from my-ai-frontend.
With these foundations in place, you're ready to write the core AI backend logic using the Grafana SDK.
Initializing the Go AI Backend with Grafana SDK
With the project scaffolded from the previous step, create the core backend file main.go inside the backend/ directory. Start by declaring the main package and importing the necessary Grafana AI SDK packages along with standard library packages for configuration and logging.
package main
import (
"context"
"log"
"os"
"github.com/grafana/ai-sdk/pkg/client"
"github.com/grafana/ai-sdk/pkg/llm/openai"
)
Next, configure the LLM provider using environment variables for the API key and model. The SDK supports OpenAI and compatible providers; here we use OpenAI as an example. Set the OPENAI_API_KEY environment variable before running the backend.
func main() {
apiKey := os.Getenv("OPENAI_API_KEY")
if apiKey == "" {
log.Fatal("OPENAI_API_KEY is not set")
}
model := os.Getenv("OPENAI_MODEL")
if model == "" {
model = "gpt-4o-mini" // a fast, cost-effective default
}
Now create an LLM client using the OpenAI provider. Enable streaming by setting Streaming to true in the provider options. The client handles token‑by‑token delivery and tool call orchestration automatically.
provider, err := openai.NewProvider(openai.ProviderOptions{
APIKey: apiKey,
Model: model,
Streaming: true,
})
if err != nil {
log.Fatalf("failed to create provider: %v", err)
}
c := client.New(client.Options{
Provider: provider,
})
ctx := context.Background()
_ = ctx // will be used in streaming calls later
Finally, confirm the setup by logging a simple message. This completes the initialization of the Go AI backend with streaming enabled, ready for the next steps of implementing streaming responses and tool calling.
log.Println("Grafana AI SDK client initialized with streaming")
}
Running go run main.go (with the API key set) should print the confirmation without errors. The project is now wired to communicate with the LLM and can be extended to handle user queries.
Implementing Streaming Responses
With the LLM client configured in the previous section, you can now request streaming responses. The Grafana AI SDK provides the ChatStream() method that returns a channel of response fragments, allowing you to process tokens as they arrive.
Start by building a user message and calling ChatStream():
messages := []llm.Message{
{Role: llm.RoleUser, Content: "Explain the Go scheduler in one sentence."},
}
stream, err := client.ChatStream(ctx, messages, nil)
if err != nil {
log.Fatalf("ChatStream error: %v", err)
}
defer stream.Close()
The returned stream object contains a channel that yields llm.StreamResult values. Iterate over the channel using a for range loop to receive tokens incrementally:
var fullResponse strings.Builder
for result := range stream.Stream() {
if result.Error != nil {
log.Printf("Stream error: %v", result.Error)
break
}
token := result.Content
fmt.Print(token) // print token as it arrives
fullResponse.WriteString(token)
}
fmt.Println()
log.Printf("Accumulated response: %s", fullResponse.String())
Each StreamResult includes Content (the token text) and an optional Error field. By checking for errors inside the loop and breaking on failure, you ensure graceful handling of network interruptions or API limits. The fullResponse buffer collects all tokens for later use — for example, to display in a chat UI or pass to subsequent tool calls.
This channel-based pattern is idiomatic in Go and gives you full control over the streaming lifecycle: you can update a UI, log progress, or cancel the stream via context cancellation. In the next section, you will extend this setup by adding tool definitions so the model can invoke external functions during the conversation.
Adding Tool-Calling Capability
Tool calling enables your LLM to request execution of external functions, such as fetching live data or querying a database. The Grafana AI SDK provides a clean abstraction to define tools using JSON schema and handle them within your streaming loop.
First, define a tool with a descriptive name and a JSON schema for its parameters. For example, a weather lookup tool:
weatherTool := sdk.Tool{
Name: "get_weather",
Description: "Get the current weather for a given location",
Parameters: map[string]interface{}{
"type": "object",
"properties": map[string]interface{}{
"location": map[string]interface{}{
"type": "string",
"description": "City and state, e.g., San Francisco, CA",
},
},
"required": []string{"location"},
},
}
Register the tool with your LLM client by passing it in the configuration:
client, err := sdk.NewClient(
openai.WithAPIKey(os.Getenv("OPENAI_API_KEY")),
sdk.WithModel("gpt-4"),
sdk.WithTools(weatherTool),
)
Now, inside the streaming loop you built in Section 4, the model may respond with a tool call instead of a text token. The SDK’s ChatStream() returns events of type EventTypeToolCall. When you receive one, execute the corresponding function and submit the result back to the stream using SubmitToolResult. Here's how:
for event := range stream {
switch event.Type {
case sdk.EventTypeToken:
fmt.Print(event.Token)
accumulated += event.Token
case sdk.EventTypeToolCall:
result := executeTool(event.ToolCall)
stream.SubmitToolResult(event.ToolCall.ID, result)
case sdk.EventTypeDone:
// streaming complete
}
}
The executeTool function switches on the tool name and returns a string. The SDK automatically sends the result back to the model, which may invoke additional tools or produce a final answer. This loop continues until the model returns a text response, maintaining real-time output while enabling the LLM to leverage external data sources — a critical capability for building a production-ready tool-calling AI backend in Go.
Exposing a REST API for the Backend
Now that we have streaming responses with tool-calling working in isolation, the next step is to expose this functionality as a REST API. We'll create an HTTP server using Go's standard net/http package and serve the AI backend through a Server-Sent Events (SSE) endpoint. SSE is ideal for streaming because it keeps a single long-lived HTTP connection and allows the server to push events to the client.
Setting up the HTTP server
Create a new file server.go and add a simple HTTP server that listens on port 8080. We'll define a single POST /chat endpoint that accepts a JSON body containing the user's message.
package main
import (
"encoding/json"
"fmt"
"log"
"net/http"
)
type ChatRequest struct {
Message string `json:"message"`
}
func main() {
http.HandleFunc("/chat", chatHandler)
log.Println("Server listening on :8080")
log.Fatal(http.ListenAndServe(":8080", nil))
}
Implementing the SSE handler
The chatHandler function reads the request body, validates the message, and then starts streaming using our existing LLM client from Section 3. We set the appropriate SSE headers and then write events as we receive tokens or tool calls from the ChatStream channel.
func chatHandler(w http.ResponseWriter, r *http.Request) {
// Only accept POST
if r.Method != http.MethodPost {
http.Error(w, "Method not allowed", http.StatusMethodNotAllowed)
return
}
var req ChatRequest
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
http.Error(w, "Bad request", http.StatusBadRequest)
return
}
if req.Message == "" {
http.Error(w, "Message is required", http.StatusBadRequest)
return
}
// Set SSE headers
w.Header().Set("Content-Type", "text/event-stream")
w.Header().Set("Cache-Control", "no-cache")
w.Header().Set("Connection", "keep-alive")
flusher, ok := w.(http.Flusher)
if !ok {
http.Error(w, "Streaming unsupported", http.StatusInternalServerError)
return
}
// Use the Grafana AI SDK's ChatStream (assumes llmClient configured globally)
stream := llmClient.ChatStream(r.Context(), req.Message)
for resp := range stream {
if resp.Error != nil {
// Send error event to client
fmt.Fprintf(w, "event: error\ndata: %s\n\n", resp.Error.Error())
flusher.Flush()
break
}
if resp.ToolCall != nil {
// Send tool call event with name and arguments
args, _ := json.Marshal(resp.ToolCall.Arguments)
fmt.Fprintf(w, "event: tool-call\ndata: %s\n\n", string(args))
flusher.Flush()
// If the tool result is available synchronously, send it
if resp.ToolResult != nil {
result, _ := json.Marshal(resp.ToolResult)
fmt.Fprintf(w, "event: tool-result\ndata: %s\n\n", string(result))
flusher.Flush()
}
}
if resp.Content != "" {
// Send token event
fmt.Fprintf(w, "event: token\ndata: %s\n\n", resp.Content)
flusher.Flush()
}
}
// Signal end of stream
fmt.Fprintf(w, "event: done\ndata: [DONE]\n\n")
flusher.Flush()
}
Explanation of events
-
token: Each piece of text generated by the LLM. The client appends these to the displayed message. -
tool-call: Indicates the LLM invoked a tool. The data contains the tool's name and arguments (e.g.,{"name":"get_weather","arguments":{"location":"Berlin"}}). The frontend can display this as an intermediate step. -
tool-result: The result returned after executing the tool. This is streamed back so the LLM can continue the conversation with the tool output. -
error: Any error during streaming, such as a network timeout or rate limit. -
done: Signals the end of the conversation turn.
By structuring the SSE events clearly, the React frontend (built in Section 7) can handle each event type separately and update the UI accordingly. The tool-call events are particularly useful for showing the user that the AI is performing an action, making the interaction transparent and engaging.
Building the React Frontend
Now that the backend streams AI responses with tool-calling results via Server-Sent Events (SSE), we need a React frontend that consumes this stream and provides a real-time chat interface. Since our endpoint is a POST /chat (which sends a user message in the request body), we cannot use the native EventSource API — it only supports GET requests. Instead, we’ll use the Fetch API with ReadableStream to manually parse the SSE data.
Create a Chat component inside your React app (e.g., src/Chat.jsx). Start by managing state for the user input, the accumulated AI response, and any tool call results:
import { useState, useRef } from 'react';
function Chat() {
const [input, setInput] = useState('');
const [response, setResponse] = useState('');
const [toolResults, setToolResults] = useState([]);
const abortRef = useRef(null);
const sendMessage = async (message) => {
setResponse('');
setToolResults([]);
try {
const res = await fetch('http://localhost:8080/chat', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ message }),
});
const reader = res.body.getReader();
const decoder = new TextDecoder();
let buffer = '';
while (true) {
const { done, value } = await reader.read();
if (done) break;
buffer += decoder.decode(value, { stream: true });
const lines = buffer.split('\n');
buffer = lines.pop(); // keep incomplete line
for (const line of lines) {
if (line.startsWith('data: ')) {
const data = line.slice(6);
try {
const parsed = JSON.parse(data);
if (parsed.type === 'token') {
setResponse((prev) => prev + parsed.content);
} else if (parsed.type === 'tool_call') {
// Optionally display the tool being called
setToolResults((prev) => [...prev, { tool: parsed.function, status: 'calling...' }]);
} else if (parsed.type === 'tool_result') {
setToolResults((prev) => {
const updated = [...prev];
const last = updated[updated.length - 1];
if (last && last.status === 'calling...') {
updated[updated.length - 1] = { tool: last.tool, result: parsed.result };
}
return updated;
});
}
} catch (e) {
// ignore malformed JSON
}
}
}
}
} catch (err) {
console.error('Stream error:', err);
}
};
return (
<div className="chat-container">
<div className="messages">
<p><strong>AI:</strong> {response}</p>
{toolResults.length > 0 && (
<div className="tool-cards">
{toolResults.map((t, i) => (
<div key={i} className="tool-card">
<strong>Tool: {t.tool}</strong>
{t.result ? <pre>{JSON.stringify(t.result, null, 2)}</pre> : <em>calling...</em>}
</div>
))}
</div>
)}
</div>
<input
value={input}
onChange={(e) => setInput(e.target.value)}
onKeyDown={(e) => e.key === 'Enter' && sendMessage(input)}
placeholder="Type a message..."
/>
</div>
);
}
export default Chat;
The component updates response state as each token arrives, giving users a live streaming effect. Tool call results appear in separate cards below the message, showing the function name and its returned data (e.g., weather information). This structure mirrors the backend’s SSE event types (token, tool_call, tool_result) and keeps the UI clean and informative. You can style the .tool-card with borders and background colors to distinguish it from regular text.
Remember to handle the case where the user sends multiple messages — the state should reset each time to avoid mixing conversations. With this React frontend, you now have a complete full-stack AI chat that streams responses and displays tool outputs in real time.
Handling Errors and Edge Cases
Even with a well-designed streaming and tool-calling backend, errors and edge cases are inevitable. This section covers practical strategies to make your Go LLM streaming application robust.
Detecting Stream Errors and Informing the User
The Grafana AI SDK returns errors through the channel-based streaming API. In your ChatStream loop, check the error sentinel or use a select statement with a context timeout. When an error occurs, send an SSE event with type error containing a user-friendly message. For example, if the LLM returns a 429 rate-limit error, you might emit:
case err := <-stream.Err():
if err != nil {
sendSSEEvent(w, "error", "The AI service is temporarily unavailable. Please try again.")
return
}
Exponential Backoff for Retries
When the SDK returns a transient error (e.g., network timeout, rate limit), implement exponential backoff before retrying the request. Use a helper function that sleeps for increasing durations (e.g., 1s, 2s, 4s) up to a maximum of 5 retries. Be careful not to retry non-idempotent operations; instead, re‑send the entire conversation history:
func retryWithBackoff(ctx context.Context, fn func() error) error {
for i := 0; i < 5; i++ {
if err := fn(); err != nil {
if !isRetryable(err) {
return err
}
time.Sleep(time.Duration(math.Pow(2, float64(i))) * time.Second)
continue
}
return nil
}
return fmt.Errorf("max retries exceeded")
}
Tool Call Timeouts
External tool functions can hang or take too long. Use context.WithTimeout when invoking a tool. If the tool exceeds the deadline, cancel the context and emit a dedicated SSE event so the frontend can display a warning:
toolCtx, cancel := context.WithTimeout(ctx, 5*time.Second)
defer cancel()
result, err := callWeatherTool(toolCtx, args)
if err != nil {
sendSSEEvent(w, "tool_error", "Weather lookup timed out. Please try again later.")
continue
}
Graceful Shutdown of SSE Connection
Clients may disconnect at any moment. Check the request context to detect cancelled connections. In your HTTP handler, use a select that listens on ctx.Done() and stops streaming cleanly. Also, ensure your server handles OS signals (SIGINT, SIGTERM) to allow in‑flight streams to finish:
func chatHandler(w http.ResponseWriter, r *http.Request) {
ctx := r.Context()
flusher, ok := w.(http.Flusher)
if !ok { /* error */ }
for {
select {
case <-ctx.Done():
log.Println("Client disconnected, stopping stream")
return
case token, ok := <-streamChan:
if !ok { return }
fmt.Fprintf(w, "data: %s\n\n", token)
flusher.Flush()
}
}
}
For a complete production‑ready implementation that includes these patterns and more, refer to the examples at https://paradane.com. By handling errors gracefully, you ensure a reliable experience even when the underlying AI service or network is unpredictable.
Taking Your AI Integration to Production
Now that you have a working streaming AI backend with tool-calling, it's time to harden it for real users. Start by adding authentication. For a REST API, implement JWT middleware that validates tokens on the /chat endpoint. This prevents unauthorized access and protects your API keys.
Next, consider scaling. Your Go backend is inherently concurrent, but under high load you may need to horizontally scale instances. Use a shared state store like Redis for conversation history and tool-call context so that any instance can resume a session. Also, set rate limits per user to avoid abuse.
Monitoring is crucial. Export metrics (request latency, tokens per second, tool-call success rates) using the OpenTelemetry SDK or Prometheus client library. Visualize them in Grafana to detect bottlenecks. Log streaming errors with context to debug issues quickly.
Finally, apply this tutorial's architecture to a real project—perhaps a customer support chatbot or an internal knowledge assistant. The combination of streaming, tool-calling, and a responsive frontend can dramatically improve user experience.
For further implementation support, including authentication templates and scaling patterns, Paradane provides detailed guides at https://paradane.com. Use these as a blueprint to take your AI integration from prototype to production.
Top comments (0)