These days, many applications rely on cloud-based AI providers to add "intelligence" features to their capabilities, and while this is perfectly acceptable in many cases; in some scenarios keeping the AI and inference work local to the machine may be the desired option. For example, when network accessibility is unreliable (or non-existent), or when there are privacy concerns about sending data externally; or even just to remove the dependency on third-party services outside of your application control. Whatever the case may be, running models locally directly from a Go application on Windows, without relying on Python or NodeJS tooling or standing up a separate model server, is an area with comparatively few simple, reliable options. This post describes a solution, go-ov, to address this issue and provide a streamlined option for adding text-generation features to applications written in Go that target the Windows operating system without adding significant complexity or overhead.
Here is a a grossly over-simplified diagram of what a stack for running AI models locally entails:
There exist multiple runtime libraries and engines capable of running AI models locally, and one of them is OpenVINO. OpenVINO is an open-source AI inference engine and toolkit created by Intel. With OpenVINO you can load an AI model and run inferences against it to generate text, transcribe speech, analyze images and many more tasks. The solution presented here focuses on integrating with OpenVINO for local model execution.
Why OpenVINO? Since it is created by Intel, and a very large number of laptops and desktops in corporate environments are running Windows on Intel architecture, OpenVINO provides an engine that is highly optimized for Intel hardware, while still supporting other platforms. In addition, when focusing only on inference workloads and not training, OpenVINO runs very well on client systems that may have limited resources for dedicated AI work.
How To Call OpenVINO From Go
To integrate OpenVINO into an application you can use the C/C++ code directly or you can use one of the provided clients, however these are only provided for Python and NodeJS. This means that if you want to integrate with an application written in a different language, let's say Go, you don't have an "easy" path.
So, what do you do when you want to add local AI model inference to an existing application written in Go? You could use something like OVMS (OpenVINO model server) but that requires running an entirely separate server application that needs to be present or carried along with your software. The official Python and NodeJS libraries would require separate client processes, but anything developed on either language negates all the advantages of developing in Go in the first place (typed-language, compiles to single binary, low footprint, etc.). That leaves us with trying to come up with some way to use OpenVINO's C/C++ APIs directly from our Go code.
The "runtime" components of OpenVINO are distributed in the form of a set of DLLs containing all the APIs necessary to interact with the toolkit. The DLLs can be easily obtained from the OpenVINO distribution. Furthermore, the Go language provides plenty of support to load DLLs and bind to the functions and types provided, including the CGO subsystem which allow Go packages to call C code natively. In theory, it should be relatively simple to just load the necessary DLLs and then rely on CGO to call whatever functions are required. But like with most things, the devil is in the details.
The first roadblock comes from the title of this post, the purpose of this integration is to use local LLMs on Windows applications. On Windows, CGO uses the MinGW/GCC compiler, but the OpenVINO DLLs are built with the Microsoft's Visual C++ compiler (MSVC), which makes linking between your Go program and the DLLs basically impossible. A secondary problem is that to use generative AI functions, you would want to use OpenVINO GenAI APIs, which is a C++ API and CGO is a way to call and use C functions and types, not C++.
After a lot of failed attempts (and a lot of wasted GitHub credits) trying to make this approach work, it was clear that a direct in-process integration would not be practical or maybe even possible. Even when at a superficial level it is tempting to think that a pure/direct/in-process integration is the ideal solution, a more detailed examination leads to the realization that maybe the in-process solution is riskier than other alternatives. So, rather than fighting ABI and compiler incompatibilities, a better approach is to adopt the sidecar pattern.
The sidecar or "worker process" is basically a small companion process to the main application on which certain tasks are delegated. In this case, a small optimized C++ helper process, compiled using MSVC that is responsible for making the calls to the OpenVINO DLLs natively. Thus, we gain absolute memory isolation and process safety. Given that local model compilation and inference execution can be resource-intensive, isolating these processes is critical. If the AI engine runs out of system memory or crashes for any reason, the primary Go application survives and is free to handle the crash gracefully.
How Does Text Generation Work in OpenVINO
The sidecar or worker process then is the one that is responsible for interacting with the OpenVINO APIs and call the generation functions. Let's take a look under the hood and see how the magic is done.
The first thing to know is that the component that does the actual inference is called a "Pipeline". The pipeline is the part of the AI machine that instantiates and executes all the workflows, layers and neural networks defined within an LLM in order to make it "do" something, like for example generating a text completion or identifying an image. So, before we do anything with the LLM, we need to ensure we have an instance of a pipeline.
llm = ov::genai::LLMPipeline(model, device, config);
model is the path to where LLM files are located
device is a reference to where do we want the processing to happen (e.g. CPU, GPU, NPU, mix and match?)
config is a map of engine-level configuration settings. Here you can control things like threading, CPU pinning, among other things.
At this point, when the pipeline is created, the model is compiled and loaded into memory and becomes ready to be used. Depending on the available system resources and model size, this operation may take some time to complete.
Once we have the engine instance, we can now tell it to generate some text.
llm->generate(history, config, streamer);
history: This is an array of objects representing the input(s) and message history. Here you can find things like system prompt, previous interactions, and the user prompt
config: Configuration settings for text generation (context length, temperature, etc.)
streamer: Callback function to handle decoded tokens as they are generated
When this function is called, OpenVINO will use the loaded model to generate the text based on the given input (history) and configuration settings.
Of course, on a real application, there is plenty of supporting code needed to generate the required variables in the right types and formats, but the gist of the magic is just those 2 steps. Create a pipeline, then use it to execute an inference.
Enter The Go-ov
Besides the actual pipeline initialization and execution, you also need to define the protocol that dictates how the main application will communicate with the sidecar/worker process. How will prompts be encoded and sent, how the response will be received, how will streaming work, etc. There is enough overhead and scaffolding needed to justify the use of a dedicated component to abstract the internals and present a clean interface to the host application. This is where go-ov comes in.
Go-ov is a light-weight Golang library to handle loading AI models and doing text-generation work using OpenVINO on Windows. The library bundles a small executable that acts as a bridge between your Go code and the OpenVINO runtime DLLs.
Obligatory "hello world" example:
package main
import (
"fmt"
goov "github.com/intel/go-ov"
)
func main() {
opts := goov.DefaultEngineOpts()
opts.ModelPath = `C:\models\OpenVINO\Phi-3.5-mini-instruct-int4-ov`
opts.LibraryPath = `C:\openvino\bin`
engine, _ := goov.New(opts)
defer engine.Close()
resp, _ := engine.Generate("What are the 3 main tourist attractions " +
"in Paris?", goov.ChatOpts{
SystemPrompt: "You are a helpful assistant.",
MaxTokens: 30,
})
fmt.Println(resp)
}
// Sample output:
// The 3 main tourist attractions in Paris are the Eiffel Tower,
// the Louvre Museum, and the Notre-Dame Cathedral.
When the host application creates a new instance of the go-ov engine (using New()), the library will automatically place a bundled copy of the worker process exe (the "sidecar") into a local per-user cache directory (AppData) and copy the OpenVINO runtime DLLs to the same directory. This serves two purposes, first it frees application developers from having to deal with the worker process directly, and second by placing the DLLs in the same directory it minimizes the risk of things like DLL injection.
All the communication between the Go application and the worker process is entirely abstracted by the go-ov API. Internally, go-ov uses standard I/O (stdin/stdout) to pass and receive messages using a very simple text-based custom protocol.
Runtime Requirements
In terms of system requirements, those are driven primarily by the AI model and the hardware device you want to use for the inference. However, for the library to work, two required runtime components are needed.
- OpenVINO GenAI runtime DLLs
- AI Model (LLM) in a format compatible with OpenVINO
Both of these need to be provided by the caller application, and must exist on the filesystem wherever the application using go-ov is running.
The runtime DLLs include the core OpenVINO components along with a set of "plugins" that handle the different devices (CPU/GPU/NPU) where the inference can run. They can be obtained directly from an OpenVINO distribution, or, more easily, by using the helper script provided in Go-ov. In total, with all the device plugins, they represent about 200mb (or 70mb zipped). This can be streamlined even more if you drop any devices your application does not need to support. For example, dropping NPU support removes about a 100mb worth of DLLs.
Note: The OpenVINO runtime DLLs are not embedded in the library itself, so the host application needs to be responsible for its provisioning, packaging and distribution.
OpenVINO uses its own optimized native model format called Intermediate Representation (IR), consisting mainly of an xml file describing the network topology, and a bin file with the model weights. You can find plenty of models in this format by searching an open-source model repository like Hugging Face. Depending on the model, the overall size might be anywhere from a few hundred megabytes to a few gigabytes. A directory containing the model files needs to exist on the filesystem when creating a new go-ov instance.
The Go-ov API
The go-ov API is intentionally very streamlined, since its main goal is to provide an "easy" way to leverage text-generation features in go-based applications. For example, applications that are not primarily AI-focused but want to add natural-language explanations of stats and metrics.
These are the main elements of the API:
New() creates a new instance of the LLM engine. Here you provide the main configuration, like where the model files are located and which hardware device should handle the inference work (CPU vs GPU vs NPU).
Once you have an engine instance, you can use it to directly generate a text completion by calling engine.Generate() or alternatively, you can start a new LLMSession. When calling Generate, Go-ov will instantiate the worker, load the model, execute the inference and unload from memory; which is great for one-off interactions, but if the application needs to make multiple calls, then using an LLMSession is better. On a session, the model is initialized and loaded when the session is created, and then remains active until you close the session. With a session object, you can make multiple session.Generate() calls, without incurring the startup cost each time.
Something very critical to keep in mind is that even though it is not immediately implied by looking at the go-ov API, is that there is a noticeable performance cost to loading an AI model and getting it ready for inference jobs. When the engine is instructed to load a model, the entire model and weights must be loaded into the memory space available to the inference device (i.e. GPU, CPU, NPU), and then the network topology needs to be compiled and be primed for execution. In practice, this means that there is a significant latency cost (in terms of time) for the first Generate() call. When using a session, all follow-up generation function calls will no longer need to pay this penalty and will result in much faster turnaround time.
Additionally, Go-ov supports system prompts, chat memory, token streaming and batch generation. The go-ov README provides additional details on how each method is called. See the examples folder for different uses of the library APIs.
Conclusion
Running AI locally in Go doesn't have to be complicated. go-ov gives Windows applications a simple, self-contained path to OpenVINO-powered text generation; keeping your data private, free of cloud and server dependencies, and optimized for Intel hardware. Go-ov is still in its beginning stages, but you can find it at github.com/intel/go-ov and see if it works for your use case.





Top comments (0)