Some teams do not want prompts or models to leave hardware they control. This post walks through running Kronk, an open-source Go model server built on llama.cpp, on a GPU instance behind Amazon SageMaker Studio's Code Editor, with an install that survives restarts and no public endpoint.
What we built
- A SageMaker Studio domain in VPC-only mode, with subnets that reach the internet through a NAT gateway.
- A Code Editor space on
ml.g5.2xlarge(one NVIDIA A10G, 24 GB) with a 100 GB EBS volume. - A lifecycle configuration (LCC) that installs Kronk once, repairs CUDA library paths, and starts the server on every app start.
- Access to the API and the admin UI through an SSM tunnel. Nothing is exposed on the network.
Kronk serves OpenAI-compatible chat, embeddings and reranking endpoints, plus an Anthropic-compatible Messages API and a browser admin UI. By default it listens on 127.0.0.1:11435.
Why Code Editor
Code Editor is a VS Code-based IDE, so writing and running non-Python code in it (here a Go server and a Go client) feels like working on a local machine, with a terminal and an EBS-backed home directory. We did not try a JupyterLab space, which also has a terminal and exposes ports through /jupyterlab/default/proxy/<port>/. The trade-off for Code Editor is in the networking section below.
The lifecycle configuration
An LCC for a Studio app must finish within five minutes, and a download plus library install can take longer. The pattern that works is an LCC that does almost nothing: it writes a launcher script to EBS and runs it with nohup in the background. The launcher is idempotent, so it installs only on the first start.
The version below is the one that ended up working:
#!/bin/bash
# CodeEditor LCC: persist the Kronk model server. Returns fast; heavy work runs under nohup.
set -u
KDIR=/home/sagemaker-user/kronk-bin
mkdir -p "$KDIR"
cat > "$KDIR/start-kronk.sh" <<'EOS'
#!/bin/bash
# Idempotent and instance-agnostic: installs Kronk once (on EBS), makes sure the CUDA backend's
# shared libraries resolve, then (re)starts the server detached on loopback.
set -u
KDIR=/home/sagemaker-user/kronk-bin
KVER=1.33.2
LOG="$KDIR/start.log"
EXTRA="$KDIR/cuda-libs" # pip-installed CUDA libs live on EBS (installed once)
exec >> "$LOG" 2>&1
echo "=== $(date -u) start-kronk begin ==="
unset HTTP_PROXY HTTPS_PROXY http_proxy https_proxy
export NO_PROXY=localhost,127.0.0.1 no_proxy=localhost,127.0.0.1
export KRONK_BASE_PATH=/home/sagemaker-user/.kronk
export KRONK_WEB_API_HOST=127.0.0.1:11435 # loopback only; reach it via the CE proxy
export PATH="$KDIR:$PATH"
HAS_GPU=0
if nvidia-smi -L; then HAS_GPU=1; else echo "no NVIDIA GPU detected -> cpu"; fi
curl -sS -m 15 -o /dev/null -w "github reachability http=%{http_code}\n" https://github.com || echo "WARN: github unreachable"
if [ ! -x "$KDIR/kronk" ]; then
echo "Installing kronk v$KVER"
curl -fsSL --retry 3 --connect-timeout 20 -o "$KDIR/kronk.tgz" \
"https://github.com/ardanlabs/kronk/releases/download/v${KVER}/kronk_${KVER}_linux_amd64.tar.gz" || { echo "download failed"; exit 1; }
tar -xzf "$KDIR/kronk.tgz" -C "$KDIR"
f=$(find "$KDIR" -type f -name kronk -perm -u+x | head -1)
[ -n "$f" ] && [ "$f" != "$KDIR/kronk" ] && mv "$f" "$KDIR/kronk"
chmod +x "$KDIR/kronk"; rm -f "$KDIR/kronk.tgz"
fi
"$KDIR/kronk" --version || true
if [ "$HAS_GPU" = 1 ]; then export KRONK_PROCESSOR=cuda; else export KRONK_PROCESSOR=cpu; fi
"$KDIR/kronk" libs --local || echo "WARN: libs install failed"
# --- Make the CUDA backend's dependencies resolve (generic across images/instances) ---
if [ "$HAS_GPU" = 1 ]; then
CUDA_SO=$(find "$KRONK_BASE_PATH/libraries" -name 'libggml-cuda.so' 2>/dev/null | head -1)
if [ -n "$CUDA_SO" ]; then
missing() { LD_LIBRARY_PATH="${LD_LIBRARY_PATH:-}" ldd "$CUDA_SO" 2>/dev/null | awk '/not found/{print $1}'; }
add_libdirs() { # add every dir that holds libcublas* to LD_LIBRARY_PATH
for d in $(find /usr/local/cuda* /opt/conda "$EXTRA" -name 'libcublas*.so*' 2>/dev/null | xargs -r -n1 dirname | sort -u); do
case ":${LD_LIBRARY_PATH:-}:" in *":$d:"*) ;; *) export LD_LIBRARY_PATH="$d:${LD_LIBRARY_PATH:-}";; esac
done
}
add_libdirs
M=$(missing)
if [ -n "$M" ]; then
echo "CUDA backend still missing: $M -> installing via pip into $EXTRA"
MAJOR=$(echo "$M" | grep -o 'so\.[0-9]*' | head -1 | cut -d. -f2)
case "$MAJOR" in 12) PKGS="nvidia-cublas-cu12";; *) PKGS="nvidia-cublas nvidia-cublas-cu${MAJOR:-13}";; esac
mkdir -p "$EXTRA"
for p in $PKGS; do
python3 -m pip install --quiet --target "$EXTRA" "$p" && { echo "pip installed $p"; add_libdirs; [ -z "$(missing)" ] && break; } || echo "pip $p failed"
done
M=$(missing)
fi
if [ -z "$M" ]; then echo "CUDA deps OK; LD_LIBRARY_PATH=${LD_LIBRARY_PATH:-}"; else echo "WARN: unresolved CUDA deps: $M (Kronk will fall back to CPU)"; fi
else
echo "WARN: libggml-cuda.so not found under $KRONK_BASE_PATH/libraries"
fi
fi
# Env file so interactive shells (kronk CLI, tests) match the server
cat > "$KDIR/env.sh" <<EOF
export PATH="$KDIR:\$PATH"
export KRONK_BASE_PATH=$KRONK_BASE_PATH
export KRONK_PROCESSOR=$KRONK_PROCESSOR
export LD_LIBRARY_PATH="${LD_LIBRARY_PATH:-}"
EOF
"$KDIR/kronk" server stop || true
"$KDIR/kronk" server start --detach
sleep 8
curl -s -o /dev/null -w "health http=%{http_code}\n" http://127.0.0.1:11435/v1/models || true
grep -o '"gpu-count":[0-9]*' "$KRONK_BASE_PATH/kronk.log" | tail -1
echo "=== $(date -u) start-kronk done ==="
EOS
chmod +x "$KDIR/start-kronk.sh"
# Fire and forget so the LCC finishes well under the 5-minute limit
nohup "$KDIR/start-kronk.sh" > "$KDIR/nohup.out" 2>&1 &
disown
echo "kronk launcher started in background"
exit 0
Things to note:
- Kronk and its models live under
/home/sagemaker-user, on EBS, so the 640 MB model and the 217 MB CUDA library bundle are downloaded once. - The version is pinned (
1.33.2). The project releases often. - The server binds to loopback. The UI and API are reached through the tunnel described below.
Problems we hit
1. A dead proxy. An older LCC in the domain set HTTP_PROXY to a proxy host that no longer existed, so the first install failed after a 137-second timeout. Checking the route tables showed the subnets already had 0.0.0.0/0 through a NAT gateway, so the proxy was not needed. Lesson: do not copy proxy settings from old LCCs without checking that the proxy exists.
2. The default Code Editor image was not allowed. The domain's default Code Editor image was a custom one that the user profile could not launch (not present in list of custom images). Setting the first-party sagemaker-distribution-gpu image on the space and the app fixed it.
3. The LCC must be allowed on the user profile. CreateApp rejects an LCC that is not in the user profile's (or domain's) LifecycleConfigArns. Updating a profile replaces the whole list, so read the existing settings first.
4. It ran on the CPU. The first chat worked, but nvidia-smi showed 0 MiB used and the reply came back at 18.9 tokens per second. The Kronk log said "gpu-count":0 and memory-topology: cpu-only. ldd on libggml-cuda.so showed libcublas.so.13 => not found: the driver was fine, but the linker could not find the cuBLAS library the CUDA backend needs. The libraries were present under /opt/conda, just not on the search path. The launcher now reads ldd, adds the directories that contain libcublas*, falls back to pip install --target if they are missing, and logs what it decided. After that the log showed "gpu-count":1 and the same request ran at about 352 tokens per second (346 in the admin chat).
To confirm the fix, open System > Running in the admin UI after a request. With the model on the GPU, the resource budget shows CUDA0 with the model's 4.44 GiB reserved there and only about 158 MiB in system RAM. In the CPU run, the same plan was cpu-only, with the memory charged to system RAM.
One caveat: the admin Info page still prints a warning that "ggml enumerated no GPU device". We believe that is the page's own probe running without the library path that the launcher sets for the server, but we did not confirm it. The speed (about 350 tok/s against 19), the Running page and the server log all agree on the GPU.
5. Code Editor has no /proxy/<port>/ path. In JupyterLab you can open a local server at /jupyterlab/default/proxy/<port>/. Code Editor in the new Studio does not offer that, and we got Unsupported URL path. The answer was remote access.
Reaching the UI without exposing anything
Enable RemoteAccess on the space (this requires recreating the app), make sure the execution role can start SSM sessions, and add an SSH entry on your laptop that uses the AWS-provided sagemaker_connect helper. Then forward the port:
ssh -L 11435:127.0.0.1:11435 <your-space-host-alias>
and open http://localhost:11435.
If you prefer a GUI, desktop VS Code with the AWS Toolkit can connect to the same space over SSM. In the AWS Explorer, open SageMaker AI, then Studio, find the running space, and click the remote-access icon next to it.
The Ports panel then forwards 11435 for you, and you can browse the Kronk files on EBS (kronk-bin, .kronk) next to a terminal on the GPU instance.
You also need the Session Manager plugin and credentials that allow sagemaker:StartSession. One caution: if your connect helper script runs with set -x, it prints the SSM stream URL and session token to the terminal. Remove the -x.
Calling it from Go
The server speaks the OpenAI chat-completions format, so a client needs only the standard library. This one does the same thing as the curl command and prints the speed numbers that Kronk returns in usage:
// Minimal client for a Kronk model server (OpenAI-compatible chat completions).
// Does the same thing as:
//
// curl http://127.0.0.1:11435/v1/chat/completions \
// -H 'Content-Type: application/json' \
// -d '{"model":"unsloth/Qwen3-0.6B-Q8_0","messages":[{"role":"user","content":"Say hi"}]}'
//
// Run on the instance (VS Code remote window) or on your laptop through the
// SSM tunnel (ssh -L 11435:127.0.0.1:11435 <host>); both reach http://127.0.0.1:11435.
package main
import (
"bytes"
"encoding/json"
"flag"
"fmt"
"io"
"net/http"
"os"
"strings"
"time"
)
type message struct {
Role string `json:"role"`
Content string `json:"content"`
}
type request struct {
Model string `json:"model"`
Messages []message `json:"messages"`
MaxTokens int `json:"max_tokens,omitempty"`
}
type response struct {
Choices []struct {
Message struct {
Content string `json:"content"`
ReasoningContent string `json:"reasoning_content"`
} `json:"message"`
FinishReason string `json:"finish_reason"`
} `json:"choices"`
Usage struct {
PromptTokens int `json:"prompt_tokens"`
CompletionTokens int `json:"completion_tokens"`
TotalTokens int `json:"total_tokens"`
TokensPerSecond float64 `json:"tokens_per_second"`
TTFTMillis float64 `json:"time_to_first_token_ms"`
} `json:"usage"`
Error any `json:"error"`
}
func main() {
baseURL := flag.String("url", envOr("KRONK_URL", "http://127.0.0.1:11435"), "Kronk base URL")
model := flag.String("model", "unsloth/Qwen3-0.6B-Q8_0", "model ID exactly as listed by /v1/models")
prompt := flag.String("prompt", "Say hi", "user prompt")
maxTokens := flag.Int("max-tokens", 0, "max completion tokens (0 = server default)")
showReasoning := flag.Bool("reasoning", false, "also print reasoning_content")
flag.Parse()
body, err := json.Marshal(request{
Model: *model,
Messages: []message{{Role: "user", Content: *prompt}},
MaxTokens: *maxTokens,
})
if err != nil {
fatal(err)
}
req, err := http.NewRequest(http.MethodPost, strings.TrimRight(*baseURL, "/")+"/v1/chat/completions", bytes.NewReader(body))
if err != nil {
fatal(err)
}
req.Header.Set("Content-Type", "application/json")
// Only needed if you later enable authorization on the server.
if key := os.Getenv("KRONK_API_KEY"); key != "" {
req.Header.Set("Authorization", "Bearer "+key)
}
client := &http.Client{Timeout: 5 * time.Minute}
start := time.Now()
resp, err := client.Do(req)
if err != nil {
fatal(err)
}
defer resp.Body.Close()
raw, err := io.ReadAll(resp.Body)
if err != nil {
fatal(err)
}
if resp.StatusCode != http.StatusOK {
fatal(fmt.Errorf("HTTP %d: %s", resp.StatusCode, strings.TrimSpace(string(raw))))
}
var out response
if err := json.Unmarshal(raw, &out); err != nil {
fatal(fmt.Errorf("decode response: %w\n%s", err, raw))
}
if len(out.Choices) == 0 {
fatal(fmt.Errorf("no choices in response: %s", raw))
}
c := out.Choices[0]
if *showReasoning && c.Message.ReasoningContent != "" {
fmt.Printf("[reasoning]\n%s\n\n", strings.TrimSpace(c.Message.ReasoningContent))
}
fmt.Println(strings.TrimSpace(c.Message.Content))
fmt.Printf("\n-- finish=%s prompt=%d completion=%d tok/s=%.1f ttft=%.0fms wall=%s\n",
c.FinishReason, out.Usage.PromptTokens, out.Usage.CompletionTokens,
out.Usage.TokensPerSecond, out.Usage.TTFTMillis, time.Since(start).Round(time.Millisecond))
}
func envOr(k, def string) string {
if v := os.Getenv(k); v != "" {
return v
}
return def
}
func fatal(err error) {
fmt.Fprintln(os.Stderr, "error:", err)
os.Exit(1)
}
Run it from your laptop through the tunnel, or on the instance itself. The SageMaker Distribution image does not include Go, so install it under your home directory, which lives on the EBS volume and survives restarts:
cd ~
curl -fsSLO https://go.dev/dl/go1.23.5.linux-amd64.tar.gz
mkdir -p ~/go-sdk && tar -xzf go1.23.5.linux-amd64.tar.gz -C ~/go-sdk --strip-components=1
rm go1.23.5.linux-amd64.tar.gz
echo 'export PATH=$HOME/go-sdk/bin:$PATH' >> ~/.bashrc && source ~/.bashrc
go version
Then:
go run main.go -prompt "Say hi" -reasoning
Use the model ID exactly as /v1/models lists it. Kronk rejects bare names.
What the UI shows
The admin UI covers model management (including pulling from Hugging Face), a VRAM calculator, a chat playground, running models and slots, and the library bundles for the llama.cpp, whisper.cpp and stable-diffusion.cpp runtimes.
Security notes
- The admin page shows a banner: "running unsecured on all ports without an admin password." The server was started with open authorization, and only the loopback interface and the SSM tunnel stand between it and anyone else. For anything beyond a single-user test, set an admin password and enable authorization, as described in Kronk's security documentation, before binding it to any other interface.
- Keep the instance in private subnets. The setup above never needs an inbound rule.
- Model downloads from Hugging Face need no token for public repos such as the Qwen3 GGUF build we used. Gated repos do.
Results
| CPU fallback | GPU (A10G) | |
|---|---|---|
| Model | Qwen3-0.6B Q8_0 | Qwen3-0.6B Q8_0 |
| Decode speed | 18.9 tok/s | about 346-360 tok/s (curl, admin chat, Go client) |
| Time to first token | n/a | about 65 ms cold, about 7 ms warm |
A 0.6B model is a smoke test, not a benchmark. With roughly 20 GB of usable VRAM, a 14B model at Q8 or a 30B-class mixture-of-experts model at Q4 is the realistic next step. Context length and slot count drive the KV cache size, so tune context-window and nseq-max in model_config.yaml.
Trade-offs
llama.cpp is built for low to moderate concurrency. For many simultaneous users, a server such as vLLM usually has higher throughput on the same GPU. For private, single-team inference with a simple operational story, Kronk was easy to run.
Cleanup
The GPU instance bills while the app runs. Stop the app when you are done (an idle timeout helps), then delete the app, the space and the LCC.









Top comments (0)