<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Sibonelo N.</title>
    <description>The latest articles on DEV Community by Sibonelo N. (@neloh).</description>
    <link>https://dev.to/neloh</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F758002%2F96440ec4-036d-4c61-8eeb-465b82ccbc99.jpeg</url>
      <title>DEV Community: Sibonelo N.</title>
      <link>https://dev.to/neloh</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/neloh"/>
    <language>en</language>
    <item>
      <title>Self-hosting an LLM server on a GPU SageMaker Code Editor space with Kronk</title>
      <dc:creator>Sibonelo N.</dc:creator>
      <pubDate>Wed, 18 Nov 2026 15:56:27 +0000</pubDate>
      <link>https://dev.to/neloh/self-hosting-an-llm-server-on-a-gpu-sagemaker-code-editor-space-with-kronk-3h0m</link>
      <guid>https://dev.to/neloh/self-hosting-an-llm-server-on-a-gpu-sagemaker-code-editor-space-with-kronk-3h0m</guid>
      <description>&lt;p&gt;Some teams do not want prompts or models to leave hardware they control. This post walks through running &lt;a href="https://www.kronkai.com/" rel="noopener noreferrer"&gt;Kronk&lt;/a&gt;, an open-source Go model server built on llama.cpp, on a GPU instance behind Amazon SageMaker Studio's Code Editor, with an install that survives restarts and no public endpoint.&lt;/p&gt;

&lt;h2&gt;
  
  
  What we built
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;A SageMaker Studio domain in &lt;strong&gt;VPC-only&lt;/strong&gt; mode, with subnets that reach the internet through a NAT gateway.&lt;/li&gt;
&lt;li&gt;A &lt;strong&gt;Code Editor&lt;/strong&gt; space on &lt;code&gt;ml.g5.2xlarge&lt;/code&gt; (one NVIDIA A10G, 24 GB) with a 100 GB EBS volume.&lt;/li&gt;
&lt;li&gt;A &lt;strong&gt;lifecycle configuration (LCC)&lt;/strong&gt; that installs Kronk once, repairs CUDA library paths, and starts the server on every app start.&lt;/li&gt;
&lt;li&gt;Access to the API and the admin UI through an &lt;strong&gt;SSM tunnel&lt;/strong&gt;. Nothing is exposed on the network.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Kronk serves OpenAI-compatible chat, embeddings and reranking endpoints, plus an Anthropic-compatible Messages API and a browser admin UI. By default it listens on &lt;code&gt;127.0.0.1:11435&lt;/code&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why Code Editor
&lt;/h2&gt;

&lt;p&gt;Code Editor is a VS Code-based IDE, so writing and running non-Python code in it (here a Go server and a Go client) feels like working on a local machine, with a terminal and an EBS-backed home directory. We did not try a JupyterLab space, which also has a terminal and exposes ports through &lt;code&gt;/jupyterlab/default/proxy/&amp;lt;port&amp;gt;/&lt;/code&gt;. The trade-off for Code Editor is in the networking section below.&lt;/p&gt;

&lt;h2&gt;
  
  
  The lifecycle configuration
&lt;/h2&gt;

&lt;p&gt;An LCC for a Studio app must finish within five minutes, and a download plus library install can take longer. The pattern that works is an LCC that does almost nothing: it writes a launcher script to EBS and runs it with &lt;code&gt;nohup&lt;/code&gt; in the background. The launcher is idempotent, so it installs only on the first start.&lt;/p&gt;

&lt;p&gt;The version below is the one that ended up working:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;#!/bin/bash&lt;/span&gt;
&lt;span class="c"&gt;# CodeEditor LCC: persist the Kronk model server. Returns fast; heavy work runs under nohup.&lt;/span&gt;
&lt;span class="nb"&gt;set&lt;/span&gt; -u
&lt;span class="nv"&gt;KDIR&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;/home/sagemaker-user/kronk-bin
mkdir -p &lt;span class="s2"&gt;"$KDIR"&lt;/span&gt;

cat &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="s2"&gt;"$KDIR/start-kronk.sh"&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&amp;lt;&lt;/span&gt;&lt;span class="s1"&gt;'EOS'&lt;/span&gt;
&lt;span class="c"&gt;#!/bin/bash&lt;/span&gt;
&lt;span class="c"&gt;# Idempotent and instance-agnostic: installs Kronk once (on EBS), makes sure the CUDA backend's&lt;/span&gt;
&lt;span class="c"&gt;# shared libraries resolve, then (re)starts the server detached on loopback.&lt;/span&gt;
&lt;span class="nb"&gt;set&lt;/span&gt; -u
&lt;span class="nv"&gt;KDIR&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;/home/sagemaker-user/kronk-bin
&lt;span class="nv"&gt;KVER&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;1.33.2
&lt;span class="nv"&gt;LOG&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"$KDIR/start.log"&lt;/span&gt;
&lt;span class="nv"&gt;EXTRA&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"$KDIR/cuda-libs"&lt;/span&gt;          &lt;span class="c"&gt;# pip-installed CUDA libs live on EBS (installed once)&lt;/span&gt;
&lt;span class="nb"&gt;exec&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&amp;gt;&lt;/span&gt; &lt;span class="s2"&gt;"$LOG"&lt;/span&gt; 2&lt;span class="o"&gt;&amp;gt;&amp;amp;&lt;/span&gt;1
&lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"=== $(date -u) start-kronk begin ==="&lt;/span&gt;

&lt;span class="kd"&gt;unset&lt;/span&gt; HTTP_PROXY HTTPS_PROXY http_proxy https_proxy
&lt;span class="kd"&gt;export&lt;/span&gt; &lt;span class="nv"&gt;NO_PROXY&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;localhost,127.0.0.1 &lt;span class="nv"&gt;no_proxy&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;localhost,127.0.0.1
&lt;span class="kd"&gt;export&lt;/span&gt; &lt;span class="nv"&gt;KRONK_BASE_PATH&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;/home/sagemaker-user/.kronk
&lt;span class="kd"&gt;export&lt;/span&gt; &lt;span class="nv"&gt;KRONK_WEB_API_HOST&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;127.0.0.1:11435   &lt;span class="c"&gt;# loopback only; reach it via the CE proxy&lt;/span&gt;
&lt;span class="kd"&gt;export&lt;/span&gt; &lt;span class="nv"&gt;PATH&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"$KDIR:$PATH"&lt;/span&gt;

&lt;span class="nv"&gt;HAS_GPU&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;0
&lt;span class="k"&gt;if&lt;/span&gt; nvidia-smi -L&lt;span class="o"&gt;;&lt;/span&gt; &lt;span class="k"&gt;then&lt;/span&gt; &lt;span class="nv"&gt;HAS_GPU&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;1&lt;span class="o"&gt;;&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"no NVIDIA GPU detected -&amp;gt; cpu"&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt; &lt;span class="k"&gt;fi&lt;/span&gt;

curl -sS -m 15 -o /dev/null -w &lt;span class="s2"&gt;"github reachability http=%{http_code}\n"&lt;/span&gt; https://github.com &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"WARN: github unreachable"&lt;/span&gt;

&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;[&lt;/span&gt; ! -x &lt;span class="s2"&gt;"$KDIR/kronk"&lt;/span&gt; &lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt; &lt;span class="k"&gt;then&lt;/span&gt;
  &lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"Installing kronk v$KVER"&lt;/span&gt;
  curl -fsSL --retry 3 --connect-timeout 20 -o &lt;span class="s2"&gt;"$KDIR/kronk.tgz"&lt;/span&gt; \
    &lt;span class="s2"&gt;"https://github.com/ardanlabs/kronk/releases/download/v${KVER}/kronk_${KVER}_linux_amd64.tar.gz"&lt;/span&gt; &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt; &lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"download failed"&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt; &lt;span class="nb"&gt;exit&lt;/span&gt; 1&lt;span class="o"&gt;;&lt;/span&gt; &lt;span class="o"&gt;}&lt;/span&gt;
  tar -xzf &lt;span class="s2"&gt;"$KDIR/kronk.tgz"&lt;/span&gt; -C &lt;span class="s2"&gt;"$KDIR"&lt;/span&gt;
  &lt;span class="nv"&gt;f&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nv"&gt;$(&lt;/span&gt;find &lt;span class="s2"&gt;"$KDIR"&lt;/span&gt; -type f -name kronk -perm -u+x &lt;span class="o"&gt;|&lt;/span&gt; head -1&lt;span class="o"&gt;)&lt;/span&gt;
  &lt;span class="o"&gt;[&lt;/span&gt; -n &lt;span class="s2"&gt;"$f"&lt;/span&gt; &lt;span class="o"&gt;]&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; &lt;span class="o"&gt;[&lt;/span&gt; &lt;span class="s2"&gt;"$f"&lt;/span&gt; != &lt;span class="s2"&gt;"$KDIR/kronk"&lt;/span&gt; &lt;span class="o"&gt;]&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; mv &lt;span class="s2"&gt;"$f"&lt;/span&gt; &lt;span class="s2"&gt;"$KDIR/kronk"&lt;/span&gt;
  chmod +x &lt;span class="s2"&gt;"$KDIR/kronk"&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt; rm -f &lt;span class="s2"&gt;"$KDIR/kronk.tgz"&lt;/span&gt;
&lt;span class="k"&gt;fi&lt;/span&gt;
&lt;span class="s2"&gt;"$KDIR/kronk"&lt;/span&gt; --version &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;

&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;[&lt;/span&gt; &lt;span class="s2"&gt;"$HAS_GPU"&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; 1 &lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt; &lt;span class="k"&gt;then&lt;/span&gt; &lt;span class="kd"&gt;export&lt;/span&gt; &lt;span class="nv"&gt;KRONK_PROCESSOR&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;cuda&lt;span class="o"&gt;;&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="kd"&gt;export&lt;/span&gt; &lt;span class="nv"&gt;KRONK_PROCESSOR&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;cpu&lt;span class="o"&gt;;&lt;/span&gt; &lt;span class="k"&gt;fi&lt;/span&gt;
&lt;span class="s2"&gt;"$KDIR/kronk"&lt;/span&gt; libs --local &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"WARN: libs install failed"&lt;/span&gt;

&lt;span class="c"&gt;# --- Make the CUDA backend's dependencies resolve (generic across images/instances) ---&lt;/span&gt;
&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;[&lt;/span&gt; &lt;span class="s2"&gt;"$HAS_GPU"&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; 1 &lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt; &lt;span class="k"&gt;then&lt;/span&gt;
  &lt;span class="nv"&gt;CUDA_SO&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nv"&gt;$(&lt;/span&gt;find &lt;span class="s2"&gt;"$KRONK_BASE_PATH/libraries"&lt;/span&gt; -name &lt;span class="s1"&gt;'libggml-cuda.so'&lt;/span&gt; 2&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;/dev/null &lt;span class="o"&gt;|&lt;/span&gt; head -1&lt;span class="o"&gt;)&lt;/span&gt;
  &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;[&lt;/span&gt; -n &lt;span class="s2"&gt;"$CUDA_SO"&lt;/span&gt; &lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt; &lt;span class="k"&gt;then&lt;/span&gt;
    missing&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt; &lt;span class="nv"&gt;LD_LIBRARY_PATH&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"${LD_LIBRARY_PATH:-}"&lt;/span&gt; ldd &lt;span class="s2"&gt;"$CUDA_SO"&lt;/span&gt; 2&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;/dev/null &lt;span class="o"&gt;|&lt;/span&gt; awk &lt;span class="s1"&gt;'/not found/{print $1}'&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt; &lt;span class="o"&gt;}&lt;/span&gt;
    add_libdirs&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;  &lt;span class="c"&gt;# add every dir that holds libcublas* to LD_LIBRARY_PATH&lt;/span&gt;
      &lt;span class="k"&gt;for&lt;/span&gt; d &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="nv"&gt;$(&lt;/span&gt;find /usr/local/cuda* /opt/conda &lt;span class="s2"&gt;"$EXTRA"&lt;/span&gt; -name &lt;span class="s1"&gt;'libcublas*.so*'&lt;/span&gt; 2&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;/dev/null &lt;span class="o"&gt;|&lt;/span&gt; xargs -r -n1 dirname &lt;span class="o"&gt;|&lt;/span&gt; sort -u&lt;span class="o"&gt;)&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt; &lt;span class="k"&gt;do&lt;/span&gt;
        &lt;span class="k"&gt;case&lt;/span&gt; &lt;span class="s2"&gt;":${LD_LIBRARY_PATH:-}:"&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; *&lt;span class="s2"&gt;":$d:"&lt;/span&gt;*&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="o"&gt;;&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt; *&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="kd"&gt;export&lt;/span&gt; &lt;span class="nv"&gt;LD_LIBRARY_PATH&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"$d:${LD_LIBRARY_PATH:-}"&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt; &lt;span class="k"&gt;esac&lt;/span&gt;
      &lt;span class="k"&gt;done&lt;/span&gt;
    &lt;span class="o"&gt;}&lt;/span&gt;
    add_libdirs
    &lt;span class="nv"&gt;M&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nv"&gt;$(&lt;/span&gt;missing&lt;span class="o"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;[&lt;/span&gt; -n &lt;span class="s2"&gt;"$M"&lt;/span&gt; &lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt; &lt;span class="k"&gt;then&lt;/span&gt;
      &lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"CUDA backend still missing: $M -&amp;gt; installing via pip into $EXTRA"&lt;/span&gt;
      &lt;span class="nv"&gt;MAJOR&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nv"&gt;$(&lt;/span&gt;&lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"$M"&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt; grep -o &lt;span class="s1"&gt;'so\.[0-9]*'&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt; head -1 &lt;span class="o"&gt;|&lt;/span&gt; cut -d. -f2&lt;span class="o"&gt;)&lt;/span&gt;
      &lt;span class="k"&gt;case&lt;/span&gt; &lt;span class="s2"&gt;"$MAJOR"&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; 12&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="nv"&gt;PKGS&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"nvidia-cublas-cu12"&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt; *&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="nv"&gt;PKGS&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"nvidia-cublas nvidia-cublas-cu${MAJOR:-13}"&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt; &lt;span class="k"&gt;esac&lt;/span&gt;
      mkdir -p &lt;span class="s2"&gt;"$EXTRA"&lt;/span&gt;
      &lt;span class="k"&gt;for&lt;/span&gt; p &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="nv"&gt;$PKGS&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt; &lt;span class="k"&gt;do&lt;/span&gt;
        python3 -m pip install --quiet --target &lt;span class="s2"&gt;"$EXTRA"&lt;/span&gt; &lt;span class="s2"&gt;"$p"&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt; &lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"pip installed $p"&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt; add_libdirs&lt;span class="o"&gt;;&lt;/span&gt; &lt;span class="o"&gt;[&lt;/span&gt; -z &lt;span class="s2"&gt;"$(missing)"&lt;/span&gt; &lt;span class="o"&gt;]&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; &lt;span class="nb"&gt;break&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt; &lt;span class="o"&gt;}&lt;/span&gt; &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"pip $p failed"&lt;/span&gt;
      &lt;span class="k"&gt;done&lt;/span&gt;
      &lt;span class="nv"&gt;M&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nv"&gt;$(&lt;/span&gt;missing&lt;span class="o"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;fi&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;[&lt;/span&gt; -z &lt;span class="s2"&gt;"$M"&lt;/span&gt; &lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt; &lt;span class="k"&gt;then&lt;/span&gt; &lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"CUDA deps OK; LD_LIBRARY_PATH=${LD_LIBRARY_PATH:-}"&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"WARN: unresolved CUDA deps: $M (Kronk will fall back to CPU)"&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt; &lt;span class="k"&gt;fi&lt;/span&gt;
  &lt;span class="k"&gt;else&lt;/span&gt;
    &lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"WARN: libggml-cuda.so not found under $KRONK_BASE_PATH/libraries"&lt;/span&gt;
  &lt;span class="k"&gt;fi&lt;/span&gt;
&lt;span class="k"&gt;fi&lt;/span&gt;

&lt;span class="c"&gt;# Env file so interactive shells (kronk CLI, tests) match the server&lt;/span&gt;
cat &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="s2"&gt;"$KDIR/env.sh"&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&amp;lt;&lt;/span&gt;EOF
&lt;span class="kd"&gt;export&lt;/span&gt; &lt;span class="nv"&gt;PATH&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"$KDIR:\$PATH"&lt;/span&gt;
&lt;span class="kd"&gt;export&lt;/span&gt; &lt;span class="nv"&gt;KRONK_BASE_PATH&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nv"&gt;$KRONK_BASE_PATH&lt;/span&gt;
&lt;span class="kd"&gt;export&lt;/span&gt; &lt;span class="nv"&gt;KRONK_PROCESSOR&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nv"&gt;$KRONK_PROCESSOR&lt;/span&gt;
&lt;span class="kd"&gt;export&lt;/span&gt; &lt;span class="nv"&gt;LD_LIBRARY_PATH&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"${LD_LIBRARY_PATH:-}"&lt;/span&gt;
EOF

&lt;span class="s2"&gt;"$KDIR/kronk"&lt;/span&gt; server stop &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;
&lt;span class="s2"&gt;"$KDIR/kronk"&lt;/span&gt; server start --detach
sleep 8
curl -s -o /dev/null -w &lt;span class="s2"&gt;"health http=%{http_code}\n"&lt;/span&gt; http://127.0.0.1:11435/v1/models &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;
grep -o &lt;span class="s1"&gt;'"gpu-count":[0-9]*'&lt;/span&gt; &lt;span class="s2"&gt;"$KRONK_BASE_PATH/kronk.log"&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt; tail -1
&lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"=== $(date -u) start-kronk done ==="&lt;/span&gt;
EOS
chmod +x &lt;span class="s2"&gt;"$KDIR/start-kronk.sh"&lt;/span&gt;

&lt;span class="c"&gt;# Fire and forget so the LCC finishes well under the 5-minute limit&lt;/span&gt;
nohup &lt;span class="s2"&gt;"$KDIR/start-kronk.sh"&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="s2"&gt;"$KDIR/nohup.out"&lt;/span&gt; 2&lt;span class="o"&gt;&amp;gt;&amp;amp;&lt;/span&gt;1 &lt;span class="o"&gt;&amp;amp;&lt;/span&gt;
&lt;span class="nb"&gt;disown&lt;/span&gt;
&lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"kronk launcher started in background"&lt;/span&gt;
&lt;span class="nb"&gt;exit&lt;/span&gt; 0
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Things to note:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Kronk and its models live under &lt;code&gt;/home/sagemaker-user&lt;/code&gt;, on EBS, so the 640 MB model and the 217 MB CUDA library bundle are downloaded once.&lt;/li&gt;
&lt;li&gt;The version is pinned (&lt;code&gt;1.33.2&lt;/code&gt;). The project releases often.&lt;/li&gt;
&lt;li&gt;The server binds to loopback. The UI and API are reached through the tunnel described below.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Problems we hit
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;1. A dead proxy.&lt;/strong&gt; An older LCC in the domain set &lt;code&gt;HTTP_PROXY&lt;/code&gt; to a proxy host that no longer existed, so the first install failed after a 137-second timeout. Checking the route tables showed the subnets already had &lt;code&gt;0.0.0.0/0&lt;/code&gt; through a NAT gateway, so the proxy was not needed. Lesson: do not copy proxy settings from old LCCs without checking that the proxy exists.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;2. The default Code Editor image was not allowed.&lt;/strong&gt; The domain's default Code Editor image was a custom one that the user profile could not launch (&lt;code&gt;not present in list of custom images&lt;/code&gt;). Setting the first-party &lt;code&gt;sagemaker-distribution-gpu&lt;/code&gt; image on the space and the app fixed it.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;3. The LCC must be allowed on the user profile.&lt;/strong&gt; &lt;code&gt;CreateApp&lt;/code&gt; rejects an LCC that is not in the user profile's (or domain's) &lt;code&gt;LifecycleConfigArns&lt;/code&gt;. Updating a profile replaces the whole list, so read the existing settings first.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;4. It ran on the CPU.&lt;/strong&gt; The first chat worked, but &lt;code&gt;nvidia-smi&lt;/code&gt; showed 0 MiB used and the reply came back at 18.9 tokens per second. The Kronk log said &lt;code&gt;"gpu-count":0&lt;/code&gt; and &lt;code&gt;memory-topology: cpu-only&lt;/code&gt;. &lt;code&gt;ldd&lt;/code&gt; on &lt;code&gt;libggml-cuda.so&lt;/code&gt; showed &lt;code&gt;libcublas.so.13 =&amp;gt; not found&lt;/code&gt;: the driver was fine, but the linker could not find the cuBLAS library the CUDA backend needs. The libraries were present under &lt;code&gt;/opt/conda&lt;/code&gt;, just not on the search path. The launcher now reads &lt;code&gt;ldd&lt;/code&gt;, adds the directories that contain &lt;code&gt;libcublas*&lt;/code&gt;, falls back to &lt;code&gt;pip install --target&lt;/code&gt; if they are missing, and logs what it decided. After that the log showed &lt;code&gt;"gpu-count":1&lt;/code&gt; and the same request ran at about 352 tokens per second (346 in the admin chat).&lt;/p&gt;

&lt;p&gt;To confirm the fix, open &lt;strong&gt;System &amp;gt; Running&lt;/strong&gt; in the admin UI after a request. With the model on the GPU, the resource budget shows &lt;code&gt;CUDA0&lt;/code&gt; with the model's 4.44 GiB reserved there and only about 158 MiB in system RAM. In the CPU run, the same plan was &lt;code&gt;cpu-only&lt;/code&gt;, with the memory charged to system RAM.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F7tksap9cuv3vtkefj3wr.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F7tksap9cuv3vtkefj3wr.png" alt="Running models: the model's VRAM is reserved on CUDA0" width="800" height="380"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;One caveat: the admin &lt;strong&gt;Info&lt;/strong&gt; page still prints a warning that "ggml enumerated no GPU device". We believe that is the page's own probe running without the library path that the launcher sets for the server, but we did not confirm it. The speed (about 350 tok/s against 19), the Running page and the server log all agree on the GPU.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;5. Code Editor has no &lt;code&gt;/proxy/&amp;lt;port&amp;gt;/&lt;/code&gt; path.&lt;/strong&gt; In JupyterLab you can open a local server at &lt;code&gt;/jupyterlab/default/proxy/&amp;lt;port&amp;gt;/&lt;/code&gt;. Code Editor in the new Studio does not offer that, and we got &lt;code&gt;Unsupported URL path&lt;/code&gt;. The answer was &lt;strong&gt;remote access&lt;/strong&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Reaching the UI without exposing anything
&lt;/h2&gt;

&lt;p&gt;Enable &lt;code&gt;RemoteAccess&lt;/code&gt; on the space (this requires recreating the app), make sure the execution role can start SSM sessions, and add an SSH entry on your laptop that uses the AWS-provided &lt;code&gt;sagemaker_connect&lt;/code&gt; helper. Then forward the port:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;ssh -L 11435:127.0.0.1:11435 &lt;span class="o"&gt;&amp;lt;&lt;/span&gt;your-space-host-alias&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;and open &lt;code&gt;http://localhost:11435&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;If you prefer a GUI, desktop VS Code with the AWS Toolkit can connect to the same space over SSM. In the AWS Explorer, open SageMaker AI, then Studio, find the running space, and click the remote-access icon next to it.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fyh8hxdyor1n0g8tejo2y.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fyh8hxdyor1n0g8tejo2y.png" alt="AWS Toolkit in VS Code: the remote-access icon next to the running space" width="799" height="53"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;The Ports panel then forwards 11435 for you, and you can browse the Kronk files on EBS (&lt;code&gt;kronk-bin&lt;/code&gt;, &lt;code&gt;.kronk&lt;/code&gt;) next to a terminal on the GPU instance.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fjz66j5ea6z2myg48en8n.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fjz66j5ea6z2myg48en8n.png" alt="VS Code connected to the space over SSH, showing the kronk-bin folder and the kronk process in top" width="800" height="410"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;You also need the Session Manager plugin and credentials that allow &lt;code&gt;sagemaker:StartSession&lt;/code&gt;. One caution: if your connect helper script runs with &lt;code&gt;set -x&lt;/code&gt;, it prints the SSM stream URL and session token to the terminal. Remove the &lt;code&gt;-x&lt;/code&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Calling it from Go
&lt;/h2&gt;

&lt;p&gt;The server speaks the OpenAI chat-completions format, so a client needs only the standard library. This one does the same thing as the &lt;code&gt;curl&lt;/code&gt; command and prints the speed numbers that Kronk returns in &lt;code&gt;usage&lt;/code&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight go"&gt;&lt;code&gt;&lt;span class="c1"&gt;// Minimal client for a Kronk model server (OpenAI-compatible chat completions).&lt;/span&gt;
&lt;span class="c1"&gt;// Does the same thing as:&lt;/span&gt;
&lt;span class="c1"&gt;//&lt;/span&gt;
&lt;span class="c1"&gt;//  curl http://127.0.0.1:11435/v1/chat/completions \&lt;/span&gt;
&lt;span class="c1"&gt;//    -H 'Content-Type: application/json' \&lt;/span&gt;
&lt;span class="c1"&gt;//    -d '{"model":"unsloth/Qwen3-0.6B-Q8_0","messages":[{"role":"user","content":"Say hi"}]}'&lt;/span&gt;
&lt;span class="c1"&gt;//&lt;/span&gt;
&lt;span class="c1"&gt;// Run on the instance (VS Code remote window) or on your laptop through the&lt;/span&gt;
&lt;span class="c1"&gt;// SSM tunnel (ssh -L 11435:127.0.0.1:11435 &amp;lt;host&amp;gt;); both reach http://127.0.0.1:11435.&lt;/span&gt;
&lt;span class="k"&gt;package&lt;/span&gt; &lt;span class="n"&gt;main&lt;/span&gt;

&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="s2"&gt;"bytes"&lt;/span&gt;
    &lt;span class="s2"&gt;"encoding/json"&lt;/span&gt;
    &lt;span class="s2"&gt;"flag"&lt;/span&gt;
    &lt;span class="s2"&gt;"fmt"&lt;/span&gt;
    &lt;span class="s2"&gt;"io"&lt;/span&gt;
    &lt;span class="s2"&gt;"net/http"&lt;/span&gt;
    &lt;span class="s2"&gt;"os"&lt;/span&gt;
    &lt;span class="s2"&gt;"strings"&lt;/span&gt;
    &lt;span class="s2"&gt;"time"&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;type&lt;/span&gt; &lt;span class="n"&gt;message&lt;/span&gt; &lt;span class="k"&gt;struct&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="nc"&gt;Role&lt;/span&gt;    &lt;span class="kt"&gt;string&lt;/span&gt; &lt;span class="sb"&gt;`json:"role"`&lt;/span&gt;
    &lt;span class="nc"&gt;Content&lt;/span&gt; &lt;span class="kt"&gt;string&lt;/span&gt; &lt;span class="sb"&gt;`json:"content"`&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="k"&gt;type&lt;/span&gt; &lt;span class="n"&gt;request&lt;/span&gt; &lt;span class="k"&gt;struct&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="nc"&gt;Model&lt;/span&gt;     &lt;span class="kt"&gt;string&lt;/span&gt;    &lt;span class="sb"&gt;`json:"model"`&lt;/span&gt;
    &lt;span class="nc"&gt;Messages&lt;/span&gt;  &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="n"&gt;message&lt;/span&gt; &lt;span class="sb"&gt;`json:"messages"`&lt;/span&gt;
    &lt;span class="nc"&gt;MaxTokens&lt;/span&gt; &lt;span class="kt"&gt;int&lt;/span&gt;       &lt;span class="sb"&gt;`json:"max_tokens,omitempty"`&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="k"&gt;type&lt;/span&gt; &lt;span class="n"&gt;response&lt;/span&gt; &lt;span class="k"&gt;struct&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="nc"&gt;Choices&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="k"&gt;struct&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="nc"&gt;Message&lt;/span&gt; &lt;span class="k"&gt;struct&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
            &lt;span class="nc"&gt;Content&lt;/span&gt;          &lt;span class="kt"&gt;string&lt;/span&gt; &lt;span class="sb"&gt;`json:"content"`&lt;/span&gt;
            &lt;span class="nc"&gt;ReasoningContent&lt;/span&gt; &lt;span class="kt"&gt;string&lt;/span&gt; &lt;span class="sb"&gt;`json:"reasoning_content"`&lt;/span&gt;
        &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="sb"&gt;`json:"message"`&lt;/span&gt;
        &lt;span class="nc"&gt;FinishReason&lt;/span&gt; &lt;span class="kt"&gt;string&lt;/span&gt; &lt;span class="sb"&gt;`json:"finish_reason"`&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="sb"&gt;`json:"choices"`&lt;/span&gt;
    &lt;span class="nc"&gt;Usage&lt;/span&gt; &lt;span class="k"&gt;struct&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="nc"&gt;PromptTokens&lt;/span&gt;     &lt;span class="kt"&gt;int&lt;/span&gt;     &lt;span class="sb"&gt;`json:"prompt_tokens"`&lt;/span&gt;
        &lt;span class="nc"&gt;CompletionTokens&lt;/span&gt; &lt;span class="kt"&gt;int&lt;/span&gt;     &lt;span class="sb"&gt;`json:"completion_tokens"`&lt;/span&gt;
        &lt;span class="nc"&gt;TotalTokens&lt;/span&gt;      &lt;span class="kt"&gt;int&lt;/span&gt;     &lt;span class="sb"&gt;`json:"total_tokens"`&lt;/span&gt;
        &lt;span class="nc"&gt;TokensPerSecond&lt;/span&gt;  &lt;span class="kt"&gt;float64&lt;/span&gt; &lt;span class="sb"&gt;`json:"tokens_per_second"`&lt;/span&gt;
        &lt;span class="nc"&gt;TTFTMillis&lt;/span&gt;       &lt;span class="kt"&gt;float64&lt;/span&gt; &lt;span class="sb"&gt;`json:"time_to_first_token_ms"`&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="sb"&gt;`json:"usage"`&lt;/span&gt;
    &lt;span class="nc"&gt;Error&lt;/span&gt; &lt;span class="kt"&gt;any&lt;/span&gt; &lt;span class="sb"&gt;`json:"error"`&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="k"&gt;func&lt;/span&gt; &lt;span class="nf"&gt;main&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;baseURL&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;flag&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;String&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"url"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nf"&gt;envOr&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"KRONK_URL"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"http://127.0.0.1:11435"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"Kronk base URL"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;model&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;flag&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;String&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"model"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"unsloth/Qwen3-0.6B-Q8_0"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"model ID exactly as listed by /v1/models"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;prompt&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;flag&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;String&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"prompt"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"Say hi"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"user prompt"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;maxTokens&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;flag&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Int&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"max-tokens"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"max completion tokens (0 = server default)"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;showReasoning&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;flag&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Bool&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"reasoning"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="kc"&gt;false&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"also print reasoning_content"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;flag&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Parse&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

    &lt;span class="n"&gt;body&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Marshal&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;request&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="nc"&gt;Model&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;     &lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="nc"&gt;Messages&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;  &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="n"&gt;message&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="nc"&gt;Role&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"user"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nc"&gt;Content&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;prompt&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="nc"&gt;MaxTokens&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;maxTokens&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt; &lt;span class="o"&gt;!=&lt;/span&gt; &lt;span class="kc"&gt;nil&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="nf"&gt;fatal&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;err&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;

    &lt;span class="n"&gt;req&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;http&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;NewRequest&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;http&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;MethodPost&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;strings&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;TrimRight&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;baseURL&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"/"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="s2"&gt;"/v1/chat/completions"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;bytes&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;NewReader&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;body&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt; &lt;span class="o"&gt;!=&lt;/span&gt; &lt;span class="kc"&gt;nil&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="nf"&gt;fatal&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;err&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;
    &lt;span class="n"&gt;req&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Header&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Set&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"Content-Type"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"application/json"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="c1"&gt;// Only needed if you later enable authorization on the server.&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;key&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;os&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Getenv&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"KRONK_API_KEY"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="n"&gt;key&lt;/span&gt; &lt;span class="o"&gt;!=&lt;/span&gt; &lt;span class="s2"&gt;""&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="n"&gt;req&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Header&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Set&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"Authorization"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"Bearer "&lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="n"&gt;key&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;

    &lt;span class="n"&gt;client&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&lt;/span&gt;&lt;span class="n"&gt;http&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Client&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="nc"&gt;Timeout&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;5&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Minute&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
    &lt;span class="n"&gt;start&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Now&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;resp&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Do&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;req&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt; &lt;span class="o"&gt;!=&lt;/span&gt; &lt;span class="kc"&gt;nil&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="nf"&gt;fatal&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;err&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;
    &lt;span class="k"&gt;defer&lt;/span&gt; &lt;span class="n"&gt;resp&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Body&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Close&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

    &lt;span class="n"&gt;raw&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;io&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;ReadAll&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;resp&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Body&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt; &lt;span class="o"&gt;!=&lt;/span&gt; &lt;span class="kc"&gt;nil&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="nf"&gt;fatal&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;err&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;resp&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;StatusCode&lt;/span&gt; &lt;span class="o"&gt;!=&lt;/span&gt; &lt;span class="n"&gt;http&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;StatusOK&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="nf"&gt;fatal&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;fmt&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Errorf&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"HTTP %d: %s"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;resp&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;StatusCode&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;strings&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;TrimSpace&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kt"&gt;string&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;raw&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;

    &lt;span class="kd"&gt;var&lt;/span&gt; &lt;span class="n"&gt;out&lt;/span&gt; &lt;span class="n"&gt;response&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Unmarshal&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;raw&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&lt;/span&gt;&lt;span class="n"&gt;out&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt; &lt;span class="o"&gt;!=&lt;/span&gt; &lt;span class="kc"&gt;nil&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="nf"&gt;fatal&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;fmt&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Errorf&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"decode response: %w\n%s"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;raw&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;out&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Choices&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="nf"&gt;fatal&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;fmt&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Errorf&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"no choices in response: %s"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;raw&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;

    &lt;span class="n"&gt;c&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;out&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Choices&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;showReasoning&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; &lt;span class="n"&gt;c&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Message&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;ReasoningContent&lt;/span&gt; &lt;span class="o"&gt;!=&lt;/span&gt; &lt;span class="s2"&gt;""&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="n"&gt;fmt&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Printf&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"[reasoning]\n%s\n\n"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;strings&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;TrimSpace&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;c&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Message&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;ReasoningContent&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;
    &lt;span class="n"&gt;fmt&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Println&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;strings&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;TrimSpace&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;c&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Message&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Content&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;fmt&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Printf&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"\n-- finish=%s prompt=%d completion=%d tok/s=%.1f ttft=%.0fms wall=%s\n"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;c&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;FinishReason&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;out&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Usage&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;PromptTokens&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;out&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Usage&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;CompletionTokens&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;out&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Usage&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;TokensPerSecond&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;out&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Usage&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;TTFTMillis&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Since&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;start&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Round&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Millisecond&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="k"&gt;func&lt;/span&gt; &lt;span class="nf"&gt;envOr&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;k&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;string&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="kt"&gt;string&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;v&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;os&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Getenv&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;k&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="n"&gt;v&lt;/span&gt; &lt;span class="o"&gt;!=&lt;/span&gt; &lt;span class="s2"&gt;""&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;v&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;def&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="k"&gt;func&lt;/span&gt; &lt;span class="nf"&gt;fatal&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;err&lt;/span&gt; &lt;span class="kt"&gt;error&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;fmt&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Fprintln&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;os&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Stderr&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"error:"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;os&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Exit&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Run it from your laptop through the tunnel, or on the instance itself. The SageMaker Distribution image does not include Go, so install it under your home directory, which lives on the EBS volume and survives restarts:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nb"&gt;cd&lt;/span&gt; ~
curl -fsSLO https://go.dev/dl/go1.23.5.linux-amd64.tar.gz
mkdir -p ~/go-sdk &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; tar -xzf go1.23.5.linux-amd64.tar.gz -C ~/go-sdk --strip-components=1
rm go1.23.5.linux-amd64.tar.gz
&lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s1"&gt;'export PATH=$HOME/go-sdk/bin:$PATH'&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&amp;gt;&lt;/span&gt; ~/.bashrc &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; &lt;span class="nb"&gt;source&lt;/span&gt; ~/.bashrc
go version
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Then:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;go run main.go -prompt &lt;span class="s2"&gt;"Say hi"&lt;/span&gt; -reasoning
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Use the model ID exactly as &lt;code&gt;/v1/models&lt;/code&gt; lists it. Kronk rejects bare names.&lt;/p&gt;

&lt;h2&gt;
  
  
  What the UI shows
&lt;/h2&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fqhi1kgcj4tfrv1xmmf8d.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fqhi1kgcj4tfrv1xmmf8d.png" alt="Kronk admin home" width="800" height="415"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fs905qo3zg82w37jpj0rm.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fs905qo3zg82w37jpj0rm.png" alt="GGUF models list" width="800" height="401"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fsbjh8hdvgsllxpmr03jk.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fsbjh8hdvgsllxpmr03jk.png" alt="Model catalog" width="800" height="402"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F7z3xk2ydekkbh2trjjeu.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F7z3xk2ydekkbh2trjjeu.png" alt="Running models and resource budget (CUDA0)" width="800" height="395"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fvbgfecd3s706zcidrvsz.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fvbgfecd3s706zcidrvsz.png" alt="Model card" width="800" height="401"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Flmxa9ci11v2y36v9boxu.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Flmxa9ci11v2y36v9boxu.png" alt="Chat playground with decode TPS" width="800" height="402"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;The admin UI covers model management (including pulling from Hugging Face), a VRAM calculator, a chat playground, running models and slots, and the library bundles for the llama.cpp, whisper.cpp and stable-diffusion.cpp runtimes.&lt;/p&gt;

&lt;h2&gt;
  
  
  Security notes
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;The admin page shows a banner: &lt;em&gt;"running unsecured on all ports without an admin password."&lt;/em&gt; The server was started with open authorization, and only the loopback interface and the SSM tunnel stand between it and anyone else. For anything beyond a single-user test, set an admin password and enable authorization, as described in Kronk's security documentation, before binding it to any other interface.&lt;/li&gt;
&lt;li&gt;Keep the instance in private subnets. The setup above never needs an inbound rule.&lt;/li&gt;
&lt;li&gt;Model downloads from Hugging Face need no token for public repos such as the Qwen3 GGUF build we used. Gated repos do.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Results
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;/th&gt;
&lt;th&gt;CPU fallback&lt;/th&gt;
&lt;th&gt;GPU (A10G)&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Model&lt;/td&gt;
&lt;td&gt;Qwen3-0.6B Q8_0&lt;/td&gt;
&lt;td&gt;Qwen3-0.6B Q8_0&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Decode speed&lt;/td&gt;
&lt;td&gt;18.9 tok/s&lt;/td&gt;
&lt;td&gt;about 346-360 tok/s (curl, admin chat, Go client)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Time to first token&lt;/td&gt;
&lt;td&gt;n/a&lt;/td&gt;
&lt;td&gt;about 65 ms cold, about 7 ms warm&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;A 0.6B model is a smoke test, not a benchmark. With roughly 20 GB of usable VRAM, a 14B model at Q8 or a 30B-class mixture-of-experts model at Q4 is the realistic next step. Context length and slot count drive the KV cache size, so tune &lt;code&gt;context-window&lt;/code&gt; and &lt;code&gt;nseq-max&lt;/code&gt; in &lt;code&gt;model_config.yaml&lt;/code&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Trade-offs
&lt;/h2&gt;

&lt;p&gt;llama.cpp is built for low to moderate concurrency. For many simultaneous users, a server such as vLLM usually has higher throughput on the same GPU. For private, single-team inference with a simple operational story, Kronk was easy to run.&lt;/p&gt;

&lt;h2&gt;
  
  
  Cleanup
&lt;/h2&gt;

&lt;p&gt;The GPU instance bills while the app runs. Stop the app when you are done (an idle timeout helps), then delete the app, the space and the LCC.&lt;/p&gt;

</description>
      <category>aws</category>
      <category>ai</category>
      <category>llm</category>
      <category>go</category>
    </item>
    <item>
      <title>Shipping a Strands agent on Amazon Bedrock AgentCore: A2A, MCP, Guardrails, CI/CD and rollback</title>
      <dc:creator>Sibonelo N.</dc:creator>
      <pubDate>Fri, 06 Nov 2026 04:05:11 +0000</pubDate>
      <link>https://dev.to/neloh/shipping-a-strands-agent-on-amazon-bedrock-agentcore-a2a-mcp-guardrails-cicd-and-rollback-jmc</link>
      <guid>https://dev.to/neloh/shipping-a-strands-agent-on-amazon-bedrock-agentcore-a2a-mcp-guardrails-cicd-and-rollback-jmc</guid>
      <description>&lt;p&gt;Most agent demos stop at "it answers in a notebook". A customer asked me for the other half: how do you host an agent, give it tools, keep it safe, ship changes through a pipeline, and undo a bad release? I built a small reference to answer that on Amazon Bedrock AgentCore, using Strands Agents. This post walks through the design and the problems I hit.&lt;/p&gt;

&lt;h2&gt;
  
  
  What it does
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;A &lt;strong&gt;Strands agent&lt;/strong&gt; served over &lt;strong&gt;A2A&lt;/strong&gt; on AgentCore Runtime&lt;/li&gt;
&lt;li&gt;A separate &lt;strong&gt;MCP tool server&lt;/strong&gt;, also on AgentCore Runtime, called with SigV4&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Amazon Bedrock Guardrails&lt;/strong&gt;, plus a tool-policy layer and tool-side validation&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;ADOT&lt;/strong&gt; tracing, CloudWatch alarms and a dashboard&lt;/li&gt;
&lt;li&gt;A &lt;strong&gt;CodePipeline&lt;/strong&gt; with ARM64 CodeBuild, an eval gate, a manual approval and a bake step&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Rollback&lt;/strong&gt; by moving an endpoint back to the previous runtime version&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Code: &lt;a href="https://github.com/Neloh/agentcore-enterprise-poc" rel="noopener noreferrer"&gt;https://github.com/Neloh/agentcore-enterprise-poc&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Pinned versions: strands-agents 1.63.0, bedrock-agentcore 1.25.2, mcp 2.3.0, a2a-sdk 0.3.26, aws-opentelemetry-distro 0.22.0.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;caller --A2A--&amp;gt; AgentCore Runtime: agent (staging | live)
                  |-- Strands Agent + BedrockModel + Guardrail
                  |-- ToolPolicyHook (allowlist, call budget, refund limit)
                  `-- MCP client --SigV4--&amp;gt; AgentCore Runtime: MCP tools (staging | live)
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  Hosting: two runtimes, two protocols
&lt;/h2&gt;

&lt;p&gt;AgentCore Runtime has a server protocol per runtime, and each has a fixed contract.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;/th&gt;
&lt;th&gt;Port&lt;/th&gt;
&lt;th&gt;Path&lt;/th&gt;
&lt;th&gt;Notes&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;HTTP&lt;/td&gt;
&lt;td&gt;8080&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;/invocations&lt;/code&gt;, &lt;code&gt;/ping&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;&lt;code&gt;BedrockAgentCoreApp&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;A2A&lt;/td&gt;
&lt;td&gt;9000&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;/&lt;/code&gt; (JSON-RPC), agent card, &lt;code&gt;/ping&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;JSON-RPC 2.0&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;MCP&lt;/td&gt;
&lt;td&gt;8000&lt;/td&gt;
&lt;td&gt;&lt;code&gt;/mcp&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;stateless streamable HTTP&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The SDK builds the A2A app for you, so the agent is mostly wiring:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="n"&gt;bedrock_agentcore&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;runtime&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;a2a&lt;/span&gt; &lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="n"&gt;build_a2a_app&lt;/span&gt;
&lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="n"&gt;strands&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;multiagent&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;a2a&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;executor&lt;/span&gt; &lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="nc"&gt;StrandsA2AExecutor&lt;/span&gt;

&lt;span class="n"&gt;executor&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;StrandsA2AExecutor&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;agent_factory&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nf"&gt;make_agent_factory&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;settings&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;registry&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;app&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;build_a2a_app&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;executor&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="no"&gt;AGENT_CARD&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;runtime_url&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;settings&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;public_url&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;ping_handler&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;ping&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;I used &lt;code&gt;agent_factory&lt;/code&gt; rather than a single shared &lt;code&gt;Agent&lt;/code&gt;, so each A2A context gets its own agent and conversation state.&lt;/p&gt;

&lt;p&gt;The MCP server must be stateless:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;app&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;mcp&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;streamable_http_app&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;streamable_http_path&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"/mcp"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;stateless_http&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="kc"&gt;True&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;json_response&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="kc"&gt;True&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;One thing that cost me time: &lt;strong&gt;mcp 2.x renamed &lt;code&gt;FastMCP&lt;/code&gt; to &lt;code&gt;MCPServer&lt;/code&gt;&lt;/strong&gt; (&lt;code&gt;from mcp.server.mcpserver import MCPServer&lt;/code&gt;). Most samples online still use the old import and fail with a &lt;code&gt;ModuleNotFoundError&lt;/code&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Guardrails in three layers
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Bedrock Guardrails&lt;/strong&gt; on the model: content filters, prompt-attack detection, a denied topic (investment advice), and PII handling.
&lt;/li&gt;
&lt;/ol&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;   &lt;span class="nc"&gt;BedrockModel&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;model_id&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;guardrail_id&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;guardrail_version&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
                &lt;span class="n"&gt;guardrail_redact_output&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="kc"&gt;True&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;guardrail_redact_output_message&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"Sorry, I can't help with that request."&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Tool policy&lt;/strong&gt; as a Strands hook. &lt;code&gt;BeforeToolCallEvent.cancel_tool&lt;/code&gt; blocks a call and returns the reason to the model. Mine enforces an allowlist, a per-request call budget, and "refunds above 200 need a human".&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Tool-side validation&lt;/strong&gt; in the MCP server. The tool never trusts the agent.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;I tested the deployed guardrail directly with &lt;code&gt;apply-guardrail&lt;/code&gt;: a prompt-injection string and an investment question were blocked, a normal order question passed.&lt;/p&gt;

&lt;p&gt;For production scale, Gateway plus Policy in AgentCore (Cedar) is the managed way to do layer 2. I called the MCP runtime directly to keep the moving parts visible.&lt;/p&gt;

&lt;h2&gt;
  
  
  Agent to MCP: SigV4
&lt;/h2&gt;

&lt;p&gt;The MCP runtime uses the default IAM inbound authorizer, so every request has to be signed for the &lt;code&gt;bedrock-agentcore&lt;/code&gt; service. Strands accepts an &lt;code&gt;httpx.Auth&lt;/code&gt;, so a small signer is enough:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;class&lt;/span&gt; &lt;span class="nc"&gt;SigV4HttpxAuth&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;httpx&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Auth&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;requires_request_body&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kc"&gt;True&lt;/span&gt;
    &lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;auth_flow&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;self&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;request&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;creds&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;_session&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get_credentials&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get_frozen_credentials&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="n"&gt;aws&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;AWSRequest&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;method&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;request&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;method&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;url&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="kt"&gt;str&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;request&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;url&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;data&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;request&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;content&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
                         &lt;span class="n"&gt;headers&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="n"&gt;k&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;v&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;k&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;v&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;request&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;headers&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;items&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;k&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;lower&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"content-type"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"accept"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="nc"&gt;SigV4Auth&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;creds&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"bedrock-agentcore"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;_region&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;add_auth&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;aws&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="n"&gt;request&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;headers&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;update&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kt"&gt;dict&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;aws&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;headers&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;items&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="k"&gt;yield&lt;/span&gt; &lt;span class="n"&gt;request&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The agent's execution role can only invoke the MCP runtime ARN.&lt;/p&gt;

&lt;h2&gt;
  
  
  Observability
&lt;/h2&gt;

&lt;p&gt;Both images start with &lt;code&gt;opentelemetry-instrument&lt;/code&gt;. Strands emits GenAI spans for the agent loop, model calls and tool calls. Three things matter:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;CloudWatch Transaction Search must be enabled&lt;/strong&gt; (once per account and Region) or the GenAI Observability dashboard stays empty.&lt;/li&gt;
&lt;li&gt;AgentCore publishes service metrics in &lt;code&gt;AWS/Bedrock-AgentCore&lt;/code&gt; (&lt;code&gt;Invocations&lt;/code&gt;, &lt;code&gt;SystemErrors&lt;/code&gt;, &lt;code&gt;Latency&lt;/code&gt;, &lt;code&gt;Throttles&lt;/code&gt;).&lt;/li&gt;
&lt;li&gt;For guardrail and policy events I emit &lt;strong&gt;EMF&lt;/strong&gt; lines to stdout, which CloudWatch Logs turns into custom metrics:
&lt;/li&gt;
&lt;/ul&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;  &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;"_aws"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;"CloudWatchMetrics"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;"Namespace"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"AgentCorePoc"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"Dimensions"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"Service"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"Tool"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
            &lt;span class="s2"&gt;"Metrics"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;"Name"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"ToolCallsDenied"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"Unit"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"Count"&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"Service"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"poc_agent"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"Tool"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"calculate_refund"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"ToolCallsDenied"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The alarm dimensions I used are &lt;code&gt;Resource=&amp;lt;runtime ARN&amp;gt;&lt;/code&gt;, &lt;code&gt;Operation=InvokeAgentRuntime&lt;/code&gt; and &lt;code&gt;Name=&amp;lt;runtime&amp;gt;::&amp;lt;endpoint&amp;gt;&lt;/code&gt;. The monitoring stack deployed, but I did not confirm the alarms receive data. Check them in your own account.&lt;/p&gt;

&lt;h2&gt;
  
  
  CI/CD
&lt;/h2&gt;

&lt;p&gt;CodePipeline V2 with native ARM64 CodeBuild (AgentCore requires &lt;code&gt;linux/arm64&lt;/code&gt; images):&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Build and test:&lt;/strong&gt; unit tests, cfn-lint, image build, an offline integration test of the exact images (using a stub model), push to ECR with immutable tags.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Staging:&lt;/strong&gt; deploy new runtime versions, point the &lt;code&gt;staging&lt;/code&gt; endpoints at them, deploy alarms and the dashboard, then run the &lt;strong&gt;eval gate&lt;/strong&gt; against the real model and guardrail.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Approval:&lt;/strong&gt; manual.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Production:&lt;/strong&gt; promote &lt;code&gt;live&lt;/code&gt;, run a smoke eval, bake on a composite alarm, roll back on failure.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The eval gate is a list of golden prompts with expected or forbidden text, covering tool use, policy denial and prompt-attack blocking. It is deterministic and cheap. In my staging run it passed 6/6 on Claude Haiku 4.5 with the guardrail attached. I'd add AgentCore Evaluations for quality scoring as a second step.&lt;/p&gt;

&lt;p&gt;The offline stub model deserves a mention. It speaks Strands' streaming event protocol, so the real event loop, hooks, MCP calls and A2A server all run in CI without any model access.&lt;/p&gt;

&lt;h2&gt;
  
  
  Rollback
&lt;/h2&gt;

&lt;p&gt;Every &lt;code&gt;UpdateAgentRuntime&lt;/code&gt; creates an immutable version. Named endpoints point at a version. Production callers use &lt;code&gt;qualifier=live&lt;/code&gt;, and &lt;strong&gt;never &lt;code&gt;DEFAULT&lt;/code&gt;&lt;/strong&gt;, which always tracks the newest version.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;ctl&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;update_agent_runtime_endpoint&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;agentRuntimeId&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;rid&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;endpointName&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"live"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;agentRuntimeVersion&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;previous&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;No rebuild, no redeploy. Promotion stores the previous version in SSM, so &lt;code&gt;deploy/agentcore.py rollback --name poc_agent --endpoint live&lt;/code&gt; works at any time. The bake step watches a composite alarm (agent 5xx, agent p90 latency, MCP 5xx, model call errors) and rolls both endpoints back if it fires. The MCP side must stay backward compatible, because the previous agent version keeps calling &lt;code&gt;MCP::live&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;What is verified and what is not:&lt;/strong&gt; the promote, rollback and bake logic is covered by unit tests against a fake control plane. I ran the pipeline through Build and Staging in a real account, and the agent and MCP runtimes were serving the &lt;code&gt;staging&lt;/code&gt; endpoints. I had not yet approved the production stage when I wrote this, so the live promotion and a live rollback are not demonstrated here.&lt;/p&gt;

&lt;h2&gt;
  
  
  Gotchas
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;ARM64 only.&lt;/strong&gt; An x86 image fails with &lt;code&gt;exec format error&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;MMDSv2.&lt;/strong&gt; AgentCore has required &lt;code&gt;requireMMDSV2: true&lt;/code&gt; on runtimes since 30 June 2026. &lt;code&gt;CreateAgentRuntime&lt;/code&gt; does not accept it, only &lt;code&gt;UpdateAgentRuntime&lt;/code&gt;, so my deploy script always calls update after create.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;403 Forbidden&lt;/code&gt; from &lt;code&gt;public.ecr.aws&lt;/code&gt;&lt;/strong&gt; during &lt;code&gt;docker build&lt;/code&gt;: an expired cached login. &lt;code&gt;docker logout public.ecr.aws&lt;/code&gt; fixes it.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;A2A is fixed to port 9000.&lt;/strong&gt; The SDK warns if you bind another port, and invocations fail with a 424.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;CloudFormation rejects YAML anchors&lt;/strong&gt; unless you run &lt;code&gt;package&lt;/code&gt;. I expanded them inline.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;CodeConnections&lt;/strong&gt; authorizations can be revoked on the GitHub side. The connection goes to &lt;code&gt;ERROR&lt;/code&gt;, and the pipeline fails at Source until you re-authorize it in the browser. Also make sure the GitHub app installation includes the repo, or you get "No Branch [main] found".&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Regions.&lt;/strong&gt; Check the AgentCore Regions page. Not every feature, such as Memory, is in every Region.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Cost and cleanup
&lt;/h2&gt;

&lt;p&gt;Nothing here runs when idle: runtimes scale to zero. Standing costs are ECR storage, CloudWatch alarms and a dashboard, and the artifact bucket. The runtimes are created by the pipeline rather than CloudFormation, so delete them explicitly (&lt;code&gt;delete-agent-runtime&lt;/code&gt;) before deleting the stacks.&lt;/p&gt;

&lt;h2&gt;
  
  
  Takeaways
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Treat the &lt;strong&gt;endpoint&lt;/strong&gt;, not the image, as the release unit. That is what makes rollback instant.&lt;/li&gt;
&lt;li&gt;Layer your guardrails. Content filtering alone does not stop a bad tool call.&lt;/li&gt;
&lt;li&gt;Make the pipeline testable offline, then gate on the real model in staging.&lt;/li&gt;
&lt;li&gt;Turn on Transaction Search first. Without it, you debug blind.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The code is a sample for reference, not an official AWS solution.&lt;/p&gt;

</description>
      <category>aws</category>
      <category>ai</category>
      <category>python</category>
      <category>devops</category>
    </item>
    <item>
      <title>What Is AgentCore Runtime, Really? Breaking Down the Thing I Already Benchmarked</title>
      <dc:creator>Sibonelo N.</dc:creator>
      <pubDate>Sun, 16 Aug 2026 10:41:46 +0000</pubDate>
      <link>https://dev.to/neloh/what-is-agentcore-runtime-really-breaking-down-the-thing-i-already-benchmarked-5bbd</link>
      <guid>https://dev.to/neloh/what-is-agentcore-runtime-really-breaking-down-the-thing-i-already-benchmarked-5bbd</guid>
      <description>&lt;p&gt;In the &lt;a href="https://dev.to/neloh/bedrock-agentcore-runtime-latency-what-actually-matters-multi-language-benchmarking-3dko"&gt;first article&lt;/a&gt; I measured AgentCore Runtime six different ways: four languages, three frameworks, PUBLIC vs VPC-mode, in-VPC vs external callers. Lots of numbers. What I skipped over is the more basic question: what is Runtime actually &lt;em&gt;doing&lt;/em&gt; to produce those numbers?&lt;/p&gt;

&lt;p&gt;This article fills that gap, using the same &lt;a href="https://github.com/Neloh/agentcore-latency-benchmarks" rel="noopener noreferrer"&gt;benchmark repo&lt;/a&gt; as the worked example instead of starting from a blank slate. It also covers something the first article couldn't have: two days after that benchmark ran, AWS shipped a second compute type for Runtime entirely. More on that below.&lt;/p&gt;

&lt;h2&gt;
  
  
  What AgentCore Runtime Is
&lt;/h2&gt;

&lt;p&gt;AWS describes it plainly: Runtime is "a secure, serverless runtime environment purpose-built for deploying and scaling dynamic AI agents and tools," offering fast cold starts, session isolation, built-in identity, and support for multi-modal, multi-agent workloads. It doesn't care which framework you brought (CrewAI, LangGraph, LlamaIndex, Google ADK, OpenAI Agents SDK, Strands, or nothing at all) or which model you call (Bedrock, OpenAI, Gemini, anything). It speaks two agent-to-agent/tool protocols out of the box: MCP and A2A.&lt;/p&gt;

&lt;p&gt;That framework-agnostic, model-agnostic design is exactly why Finding 1 and Finding 5 in the benchmark article landed the way they did — container language and framework choice added zero measurable overhead, because Runtime doesn't touch either. It just runs your container and enforces a contract.&lt;/p&gt;

&lt;h2&gt;
  
  
  What's Actually Inside a microVM
&lt;/h2&gt;

&lt;p&gt;I used the word "microVM" in the first article without stopping to explain it, so here it is properly: a microVM is a minimal virtual machine, built on the open-source &lt;a href="https://firecracker-microvm.github.io/" rel="noopener noreferrer"&gt;Firecracker&lt;/a&gt; virtualization technology that also powers AWS Lambda. It gives you real VM-level isolation — its own kernel, its own memory space, no shared host kernel with any other tenant — but strips away everything a general-purpose VM boots that an agent session doesn't need, so it can start in tens of milliseconds instead of the seconds a traditional EC2 instance takes.&lt;/p&gt;

&lt;p&gt;That's the specific trade AgentCore Runtime is making by default: stronger isolation than a container (which shares the host kernel), at close to container-speed startup, rather than the isolation of a full EC2 instance at EC2 boot times. Every session gets one, and it's why the isolation guarantee reads the way it does in the docs: "no shared state, no shared filesystem" between sessions, even concurrent ones for the same runtime.&lt;/p&gt;

&lt;p&gt;Here's what that actually looks like on a warm call — the case that produced the 80-160ms in-VPC numbers from article 1:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fpws39udct8cdlj3bthri.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fpws39udct8cdlj3bthri.png" alt="Warm invocation sequence" width="800" height="420"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;And here's the cold path — a brand-new &lt;code&gt;sessionId&lt;/code&gt; with no existing microVM to route to:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fpjh20zvi62lmzd94keqe.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fpjh20zvi62lmzd94keqe.png" alt="Cold start sequence" width="799" height="443"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;The only structural difference between the two is one extra step: provisioning a microVM and waiting for &lt;code&gt;/ping&lt;/code&gt; to report &lt;code&gt;Healthy&lt;/code&gt; before the first &lt;code&gt;/invocations&lt;/code&gt; call can go through. Everything downstream of that is identical. That single step is also the entire explanation for why cold start is seconds and warm is milliseconds — it's not a slower code path, it's a whole extra phase that warm calls skip.&lt;/p&gt;

&lt;p&gt;Here's a warm call for real, run through the console's own test panel against one of the benchmark repo's echo runtimes:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fk8vzzwztlbyvghkwx24k.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fk8vzzwztlbyvghkwx24k.png" alt="Console test/invoke panel showing a live warm invocation" width="799" height="396"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;code&gt;request_processed_ms: 37&lt;/code&gt;. That's not from a load-testing script, it's a single live click. The rest of the response — &lt;code&gt;container_id&lt;/code&gt;, &lt;code&gt;uptime_seconds&lt;/code&gt;, &lt;code&gt;language&lt;/code&gt; — comes straight from the runtime's own code, which is worth looking at directly next.&lt;/p&gt;

&lt;h2&gt;
  
  
  The Four Building Blocks
&lt;/h2&gt;

&lt;p&gt;Peel back "Runtime" and there are four concrete objects, each with its own lifecycle:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Runtime&lt;/strong&gt; — the top-level resource. Has a unique identity and is versioned. This is the thing you &lt;code&gt;CreateAgentRuntime&lt;/code&gt; once and then update repeatedly.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Version&lt;/strong&gt; — an immutable snapshot. Every time you change the container image, protocol settings, or network settings, a new version is cut. Version 1 is created automatically the first time you create the Runtime.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Endpoint&lt;/strong&gt; — an addressable ARN pointing at a specific version. A &lt;code&gt;DEFAULT&lt;/code&gt; endpoint is created automatically and always tracks the latest version; you can also create named endpoints (dev/test/prod) that pin to a specific version deliberately, so you're not forced onto whatever was last deployed. Endpoints move through &lt;code&gt;CREATING → READY&lt;/code&gt; (or &lt;code&gt;CREATE_FAILED&lt;/code&gt;), and &lt;code&gt;UPDATING → READY&lt;/code&gt; (or &lt;code&gt;UPDATE_FAILED&lt;/code&gt;) without downtime to callers.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F58ekeaym7h38ku3e8wg3.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F58ekeaym7h38ku3e8wg3.png" alt="Runtime, Version, and Endpoint relationship" width="715" height="222"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;That's the theory. Here's what it looks like on an actual runtime — &lt;code&gt;goHttpLatencyTest&lt;/code&gt;, compute type &lt;code&gt;microVMs&lt;/code&gt;, one version, two endpoints:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F4w1nvt7m0gefx56bp4f0.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F4w1nvt7m0gefx56bp4f0.png" alt="Runtime detail page showing compute type, versions, and endpoints" width="799" height="391"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;And drilling into one of those endpoints:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Foyyfz5kqb1ltjiyr35tm.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Foyyfz5kqb1ltjiyr35tm.png" alt="Endpoint detail page" width="800" height="365"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Session&lt;/strong&gt; — the actual unit of execution. Identified by a &lt;code&gt;runtimeSessionId&lt;/code&gt; (yours, or generated for you on first call), a session runs in a dedicated microVM with fully isolated CPU, memory, and filesystem. It persists for up to 8 hours, moves between &lt;code&gt;Active&lt;/code&gt; and &lt;code&gt;Idle&lt;/code&gt;, and gets &lt;code&gt;Terminated&lt;/code&gt; after 15 minutes of inactivity, at the 8-hour cap, or if deemed unhealthy. Termination destroys the entire microVM and sanitizes memory — reusing the same &lt;code&gt;runtimeSessionId&lt;/code&gt; afterward just gets you a fresh environment, not the old one back.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F4qlxoq7tow1r9reo47sp.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F4qlxoq7tow1r9reo47sp.png" alt="Session lifecycle state diagram" width="412" height="484"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;This is the mechanism behind the repo's session-affinity test: embedding a random container ID at startup and checking whether it stays constant across calls is really just checking "am I still inside the same microVM, or did my session get torn down and rebuilt."&lt;/p&gt;

&lt;h2&gt;
  
  
  Three Protocols, One Contract Each
&lt;/h2&gt;

&lt;p&gt;Whatever's inside the container, Runtime expects one of three fixed contracts, chosen when you create the Runtime:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Protocol&lt;/th&gt;
&lt;th&gt;Port&lt;/th&gt;
&lt;th&gt;Main path&lt;/th&gt;
&lt;th&gt;Health check&lt;/th&gt;
&lt;th&gt;Shape&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;HTTP&lt;/td&gt;
&lt;td&gt;8080&lt;/td&gt;
&lt;td&gt;&lt;code&gt;POST /invocations&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;GET /ping&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;REST, optional SSE/WebSocket&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;MCP&lt;/td&gt;
&lt;td&gt;8000&lt;/td&gt;
&lt;td&gt;&lt;code&gt;POST /mcp&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;GET /ping&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;JSON-RPC 2.0, Streamable-HTTP&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;A2A&lt;/td&gt;
&lt;td&gt;9000&lt;/td&gt;
&lt;td&gt;&lt;code&gt;POST /&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;GET /.well-known/agent-card.json&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;JSON-RPC 2.0&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Every echo runtime in the benchmark repo — Go, Node.js, Java, Python — implements exactly the HTTP contract: bind &lt;code&gt;0.0.0.0&lt;/code&gt;, answer &lt;code&gt;/ping&lt;/code&gt; with a health status, answer &lt;code&gt;/invocations&lt;/code&gt; with JSON. Nothing about the contract is language-specific, which is the whole reason a 22 MB Go binary and a 494 MB Java/Corretto image land within 10ms of each other once warm: they're both just satisfying the same thin HTTP contract, and the platform's own routing floor dwarfs whatever the container itself does with the request.&lt;/p&gt;

&lt;h2&gt;
  
  
  What This Looks Like in Code
&lt;/h2&gt;

&lt;p&gt;It's worth seeing how little code actually satisfies that contract. Here's the Go runtime's entire structure — no subdirectories, no dependencies, one source file:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;source/agentcore-go-runtime/
├── Dockerfile
├── deploy.sh
├── go.mod
└── main.go
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The Dockerfile is a standard two-stage build: compile a static binary, then copy just the binary into a bare &lt;code&gt;alpine&lt;/code&gt; image. Nothing else goes in:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight docker"&gt;&lt;code&gt;&lt;span class="c"&gt;# Multi-stage build for minimal container size&lt;/span&gt;
&lt;span class="k"&gt;FROM&lt;/span&gt; golang:1.22-alpine AS builder
&lt;span class="k"&gt;WORKDIR&lt;/span&gt; /app

&lt;span class="c"&gt;# Copy module file (no external deps)&lt;/span&gt;
&lt;span class="k"&gt;COPY&lt;/span&gt; go.mod ./
&lt;span class="k"&gt;COPY&lt;/span&gt; main.go .
&lt;span class="k"&gt;RUN&lt;/span&gt; &lt;span class="nv"&gt;CGO_ENABLED&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;0 &lt;span class="nv"&gt;GOOS&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;linux &lt;span class="nv"&gt;GOARCH&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;arm64 go build -ldflags=&lt;span class="s2"&gt;"-s -w"&lt;/span&gt; -o /runtime main.go

&lt;span class="c"&gt;# Final image: minimal alpine&lt;/span&gt;
&lt;span class="k"&gt;FROM&lt;/span&gt; alpine:3.20
&lt;span class="k"&gt;RUN&lt;/span&gt; apk --no-cache add ca-certificates
&lt;span class="k"&gt;COPY&lt;/span&gt; --from=builder /runtime /runtime

&lt;span class="c"&gt;# AgentCore HTTP protocol requires port 8080&lt;/span&gt;
&lt;span class="k"&gt;EXPOSE&lt;/span&gt; 8080

&lt;span class="k"&gt;RUN&lt;/span&gt; adduser -D -u 1001 agentcore
&lt;span class="k"&gt;USER&lt;/span&gt; agentcore

&lt;span class="k"&gt;ENTRYPOINT&lt;/span&gt; &lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"/runtime"&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;GOARCH=arm64&lt;/code&gt; isn't a style choice — AgentCore's Firecracker microVMs are ARM64, so anything built for the wrong architecture fails at deploy time, not at runtime. And &lt;code&gt;main.go&lt;/code&gt; itself is just two HTTP handlers wired to the two paths from the contract table:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight go"&gt;&lt;code&gt;&lt;span class="k"&gt;func&lt;/span&gt; &lt;span class="nf"&gt;ping&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;w&lt;/span&gt; &lt;span class="n"&gt;http&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;ResponseWriter&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;http&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Request&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;w&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Header&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Set&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"Content-Type"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"application/json"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;NewEncoder&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;w&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Encode&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nc"&gt;PingResponse&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="nc"&gt;Status&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;           &lt;span class="s2"&gt;"Healthy"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="nc"&gt;TimeOfLastUpdate&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Now&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Unix&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="k"&gt;func&lt;/span&gt; &lt;span class="nf"&gt;invocations&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;w&lt;/span&gt; &lt;span class="n"&gt;http&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;ResponseWriter&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;http&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Request&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="c1"&gt;// ...decode the request, run whatever "action" it asked for...&lt;/span&gt;
    &lt;span class="n"&gt;result&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"_meta"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;Meta&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="nc"&gt;ContainerID&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;        &lt;span class="n"&gt;containerID&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="nc"&gt;UptimeSeconds&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;      &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Since&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;startupTime&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Seconds&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="nc"&gt;RequestProcessedMs&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;requestProcessedMs&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="nc"&gt;Language&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;           &lt;span class="s2"&gt;"go"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;
    &lt;span class="n"&gt;w&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Header&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Set&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"Content-Type"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"application/json"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;NewEncoder&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;w&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Encode&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;result&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="k"&gt;func&lt;/span&gt; &lt;span class="nf"&gt;main&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;http&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;HandleFunc&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"/ping"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;ping&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;http&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;HandleFunc&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"/invocations"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;invocations&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;log&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Fatal&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;http&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;ListenAndServe&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;":8080"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="kc"&gt;nil&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Those exact field names — &lt;code&gt;container_id&lt;/code&gt;, &lt;code&gt;uptime_seconds&lt;/code&gt;, &lt;code&gt;request_processed_ms&lt;/code&gt;, &lt;code&gt;language&lt;/code&gt; — are what showed up in the test-panel response a few sections back. There's no gap between the code and the observed behavior; the console screenshot is this &lt;code&gt;main.go&lt;/code&gt; running.&lt;/p&gt;

&lt;p&gt;The console will even generate a starter invocation snippet for any runtime you create, in Python, TypeScript, or JavaScript:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F6zryxqylag04msrfwhws.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F6zryxqylag04msrfwhws.png" alt="Console-generated invocation code sample" width="800" height="400"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Why PUBLIC vs VPC-Mode Didn't Move the Numbers
&lt;/h2&gt;

&lt;p&gt;Network mode changes &lt;em&gt;where the container's network interface lives&lt;/em&gt; — platform-managed infrastructure for PUBLIC, an ENI in your own subnet for VPC-mode — not how sessions, versions, or the request path work. Once a session's microVM is warm, the request still goes through the same routing and lands in the same kind of isolated execution environment either way. That's why Finding 3 showed identical warm latency between the two: the difference is cold-start ENI attachment time, not steady-state architecture.&lt;/p&gt;

&lt;h2&gt;
  
  
  The New Alternative: Runtime Instances and Capacity Providers
&lt;/h2&gt;

&lt;p&gt;Everything above describes the microVM compute type — the only one that existed when I ran the first article's benchmarks. On 06 Aug 2026, two days after that article published, AWS announced a second compute type for Runtime: &lt;strong&gt;Instances&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fg0dp1w7ritj6d3c0xqz6.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fg0dp1w7ritj6d3c0xqz6.png" alt="AgentCore console showing the new Instances banner and Capacity providers tab" width="799" height="381"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Where microVMs are fully AWS-managed, ephemeral, and capped at 8 hours per session, Instances runs your agent on Amazon EC2 managed instances &lt;em&gt;inside your own AWS account&lt;/em&gt;. AgentCore still provisions, patches, scales, and tears them down for you — you don't do standard EC2 lifecycle operations on them yourself — but the compute lives in your account, so your existing Savings Plans, Reserved Instances, and On-Demand Capacity Reservations apply, and the instances (and any attached EBS volumes) show up in your own EC2 console.&lt;/p&gt;

&lt;p&gt;The infrastructure itself is defined by a new resource called a &lt;strong&gt;capacity provider&lt;/strong&gt;: a reusable template specifying the OS, allowed instance types, VPC/subnets, storage volumes, and IAM roles. You attach a capacity provider to a Runtime at creation time via &lt;code&gt;capacityProviderConfiguration&lt;/code&gt;, and you can't change compute type after that — it's a decision made once, per Runtime.&lt;/p&gt;

&lt;p&gt;What Instances buys you that microVMs structurally can't:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Characteristic&lt;/th&gt;
&lt;th&gt;microVMs&lt;/th&gt;
&lt;th&gt;Instances&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Best suited for&lt;/td&gt;
&lt;td&gt;Lightweight, fast-scaling API agents&lt;/td&gt;
&lt;td&gt;Long-running, stateful, or collaborative workloads&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Maximum session duration&lt;/td&gt;
&lt;td&gt;Up to 8 hours&lt;/td&gt;
&lt;td&gt;Up to 14 days&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Operating systems&lt;/td&gt;
&lt;td&gt;Linux containers (arm64 only)&lt;/td&gt;
&lt;td&gt;Linux (x86_64 and arm64)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Networking&lt;/td&gt;
&lt;td&gt;PUBLIC or VPC&lt;/td&gt;
&lt;td&gt;VPC only&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Agents per session&lt;/td&gt;
&lt;td&gt;One runtime hosts one agent (1:1)&lt;/td&gt;
&lt;td&gt;One session can host multiple agents (1:N)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;GPU access&lt;/td&gt;
&lt;td&gt;Not supported&lt;/td&gt;
&lt;td&gt;Supported (g4dn/g5/g6/g6e/gr6/g6f/gr6f/g7e, inf2)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Pricing&lt;/td&gt;
&lt;td&gt;Consumption-based, billed by AgentCore&lt;/td&gt;
&lt;td&gt;Runs in your account; your EC2 pricing agreements apply&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Two consequences worth calling out for anyone coming from the first article's mental model:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Sessions can now survive stop/resume, not just stay idle.&lt;/strong&gt; On microVMs, termination is final — a reused &lt;code&gt;runtimeSessionId&lt;/code&gt; after termination gets a brand-new environment. On Instances, when a session hits its lifetime or is stopped, AgentCore tears down the EC2 instance but &lt;em&gt;keeps the attached EBS volume&lt;/em&gt;. Invoking the same &lt;code&gt;runtimeSessionId&lt;/code&gt; again provisions a fresh instance, re-attaches that volume, and your agent's filesystem picks up where it left off — potentially on a freshly patched machine image.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fbikh552wowbglrxc6qc7.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fbikh552wowbglrxc6qc7.png" alt="Instances stop/resume sequence with persistent EBS volume" width="800" height="488"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Multiple agents can now share one execution environment.&lt;/strong&gt; The microVM model is strictly 1:1 — one Runtime, one agent, one microVM per session. On Instances, if two separate Runtimes are attached to the &lt;em&gt;same&lt;/em&gt; capacity provider and invoked with the &lt;em&gt;same&lt;/em&gt; &lt;code&gt;runtimeSessionId&lt;/code&gt;, both agents land on the same EC2 instance and share its filesystem. That's a materially different collaboration primitive than anything the microVM model offers.&lt;/p&gt;

&lt;p&gt;None of the numbers in the first article measured Instances — that compute type didn't exist yet, and every benchmark there was run against microVMs specifically. That's a legitimate follow-up benchmark on its own: does a GPU-backed Instances session behave the same way under the "does language matter" and "does network mode matter" questions, or does the EC2-backed model change the shape of the answer? Worth its own repo, not a retrofit onto this one.&lt;/p&gt;

&lt;h2&gt;
  
  
  Seeing It for Real
&lt;/h2&gt;

&lt;p&gt;Everything above is easier to believe with an actual &lt;code&gt;DescribeAgentRuntime&lt;/code&gt; response in front of you instead of a docs paraphrase. Here's the real output for one of the benchmark repo's own runtimes, the Go echo container in PUBLIC mode:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="nl"&gt;"agentRuntimeArn"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"arn:aws:bedrock-agentcore:us-east-1:&amp;lt;account-id&amp;gt;:runtime/goHttpLatencyTest-5CKthu5tD0"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="nl"&gt;"agentRuntimeName"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"goHttpLatencyTest"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="nl"&gt;"agentRuntimeVersion"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"1"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="nl"&gt;"networkConfiguration"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="nl"&gt;"networkMode"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"PUBLIC"&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="nl"&gt;"status"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"READY"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="nl"&gt;"lifecycleConfiguration"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="nl"&gt;"idleRuntimeSessionTimeout"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;900&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="nl"&gt;"maxLifetime"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;28800&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="nl"&gt;"agentRuntimeArtifact"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="nl"&gt;"containerConfiguration"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
            &lt;span class="nl"&gt;"containerUri"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"&amp;lt;account-id&amp;gt;.dkr.ecr.us-east-1.amazonaws.com/agentcore-go-runtime:http-v1"&lt;/span&gt;
        &lt;span class="p"&gt;}&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="nl"&gt;"protocolConfiguration"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="nl"&gt;"serverProtocol"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"HTTP"&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Two numbers to notice: &lt;code&gt;idleRuntimeSessionTimeout: 900&lt;/code&gt; and &lt;code&gt;maxLifetime: 28800&lt;/code&gt; are exactly the 15-minute idle timeout and 8-hour max lifetime from the session lifecycle diagram above, in seconds. That's not a coincidence or a rounded approximation — it's the literal same value the docs describe, straight from the API.&lt;/p&gt;

&lt;p&gt;For the VPC-mode Python runtime, the only structural difference is the &lt;code&gt;networkConfiguration&lt;/code&gt; block growing to include &lt;code&gt;securityGroups&lt;/code&gt; and &lt;code&gt;subnets&lt;/code&gt; — everything else (protocol, lifecycle, artifact shape) is identical. That's the API-level version of Finding 3: PUBLIC and VPC-mode differ in exactly one config block and nowhere else.&lt;/p&gt;

&lt;h2&gt;
  
  
  References
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;&lt;a href="https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/what-is-bedrock-agentcore.html" rel="noopener noreferrer"&gt;Amazon Bedrock AgentCore overview&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/runtime-how-it-works.html" rel="noopener noreferrer"&gt;AgentCore Runtime: microVMs, versions, endpoints, sessions&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/runtime-instances-how-it-works.html" rel="noopener noreferrer"&gt;AgentCore Runtime: Instances and capacity providers&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://aws.amazon.com/blogs/aws/runtime-instances-persistent-compute-for-production-ai-agents-on-amazon-bedrock-agentcore/" rel="noopener noreferrer"&gt;AWS News Blog: Runtime instances — persistent compute for production AI agents&lt;/a&gt; (06 Aug 2026)&lt;/li&gt;
&lt;li&gt;&lt;a href="https://firecracker-microvm.github.io/" rel="noopener noreferrer"&gt;Firecracker microVM project&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/Neloh/agentcore-latency-benchmarks" rel="noopener noreferrer"&gt;Benchmark repo: agentcore-latency-benchmarks&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/Neloh/agentcore-latency-benchmarks/tree/main/source/agentcore-go-runtime" rel="noopener noreferrer"&gt;Go runtime source: main.go, Dockerfile&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/neloh/bedrock-agentcore-runtime-latency-what-actually-matters-multi-language-benchmarking-3dko"&gt;Article 1: Bedrock AgentCore Runtime Latency&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;

</description>
      <category>aws</category>
      <category>bedrock</category>
      <category>serverless</category>
      <category>architecture</category>
    </item>
    <item>
      <title>Bedrock AgentCore Runtime Latency: What Actually Matters (Multi-Language Benchmarking)</title>
      <dc:creator>Sibonelo N.</dc:creator>
      <pubDate>Tue, 04 Aug 2026 20:24:28 +0000</pubDate>
      <link>https://dev.to/neloh/bedrock-agentcore-runtime-latency-what-actually-matters-multi-language-benchmarking-3dko</link>
      <guid>https://dev.to/neloh/bedrock-agentcore-runtime-latency-what-actually-matters-multi-language-benchmarking-3dko</guid>
      <description>&lt;p&gt;This article presents comprehensive latency benchmarking results for AWS Bedrock AgentCore Runtime across four programming languages (Go, Node.js, Java, Python), three framework options (raw boto3, Strands SDK, LangChain), multiple network configurations, and caller locations. The goal is to provide data-driven guidance on what design choices affect latency and what does not.&lt;/p&gt;

&lt;h2&gt;
  
  
  Background
&lt;/h2&gt;

&lt;p&gt;AgentCore Runtime is the compute layer for hosting agents and tools in Bedrock AgentCore. Customers frequently ask:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Which language should I write my runtime container in?&lt;/li&gt;
&lt;li&gt;Does PUBLIC vs VPC network mode affect performance?&lt;/li&gt;
&lt;li&gt;Should I use Strands SDK, LangChain, or raw SDK calls?&lt;/li&gt;
&lt;li&gt;Does the client SDK language matter?&lt;/li&gt;
&lt;li&gt;Where should my caller run for best latency?&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;This article answers each question with measured data from a controlled test environment.&lt;/p&gt;

&lt;h2&gt;
  
  
  Test Environment
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Region: us-east-1&lt;/li&gt;
&lt;li&gt;Protocol: HTTP (port 8080)&lt;/li&gt;
&lt;li&gt;Model (for real agent tests): Claude Haiku 4.5 via cross-region inference profile&lt;/li&gt;
&lt;li&gt;VPC: Private subnets with &lt;code&gt;com.amazonaws.us-east-1.bedrock-agentcore&lt;/code&gt; interface endpoint&lt;/li&gt;
&lt;li&gt;Caller locations tested: In-VPC Lambda (private subnet) and external (boto3 with connection reuse)&lt;/li&gt;
&lt;li&gt;All container images built for linux/arm64 (AgentCore Firecracker VMs are ARM64)&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Finding 1: Container language does not affect warm latency
&lt;/h2&gt;

&lt;p&gt;Four echo runtimes were deployed, each implementing the same trivial HTTP server (receive JSON, return it with metadata). All use the HTTP protocol on port 8080.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fm1ip0og8rqapo12qsq27.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fm1ip0og8rqapo12qsq27.png" alt="4-Language Latency Comparison" width="800" height="573"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Language&lt;/th&gt;
&lt;th&gt;Image Size&lt;/th&gt;
&lt;th&gt;In-VPC Median&lt;/th&gt;
&lt;th&gt;External Mean&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Go&lt;/td&gt;
&lt;td&gt;22 MB&lt;/td&gt;
&lt;td&gt;85ms&lt;/td&gt;
&lt;td&gt;968ms&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Node.js&lt;/td&gt;
&lt;td&gt;229 MB&lt;/td&gt;
&lt;td&gt;80ms&lt;/td&gt;
&lt;td&gt;950ms&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Java (Corretto 21)&lt;/td&gt;
&lt;td&gt;494 MB&lt;/td&gt;
&lt;td&gt;77ms&lt;/td&gt;
&lt;td&gt;945ms&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Python (3.12 + gunicorn)&lt;/td&gt;
&lt;td&gt;52 MB&lt;/td&gt;
&lt;td&gt;84ms&lt;/td&gt;
&lt;td&gt;937ms&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;All four are within 10ms of each other on warm in-VPC calls. The platform routing floor dominates. Language choice should be driven by team expertise and ecosystem, not latency.&lt;/p&gt;

&lt;h2&gt;
  
  
  Finding 2: Caller location is the single biggest factor
&lt;/h2&gt;

&lt;p&gt;The same runtime invoked from three different locations:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fkwe68ayg45o87vp6n08z.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fkwe68ayg45o87vp6n08z.png" alt="VPC vs External" width="800" height="298"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Caller Location&lt;/th&gt;
&lt;th&gt;Mean Latency&lt;/th&gt;
&lt;th&gt;What Adds Cost&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;In-VPC Lambda (VPC endpoint)&lt;/td&gt;
&lt;td&gt;80-90ms&lt;/td&gt;
&lt;td&gt;Platform routing only&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;External (boto3, connection reuse)&lt;/td&gt;
&lt;td&gt;937-968ms&lt;/td&gt;
&lt;td&gt;Network RTT to region&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;External (AWS CLI, no reuse)&lt;/td&gt;
&lt;td&gt;1,750-1,830ms&lt;/td&gt;
&lt;td&gt;Network RTT + TLS + SDK init&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Placing callers in the same VPC with a &lt;code&gt;bedrock-agentcore&lt;/code&gt; interface endpoint provides 5-10x latency improvement. This is the single most impactful optimization.&lt;/p&gt;

&lt;h2&gt;
  
  
  Finding 3: PUBLIC vs VPC-MODE runtime has no latency impact
&lt;/h2&gt;

&lt;p&gt;VPC-MODE places the container's ENI in your private subnet. PUBLIC mode runs on platform-managed infrastructure.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F1ii21elgnougpwrmeasw.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F1ii21elgnougpwrmeasw.png" alt="PUBLIC vs VPC-MODE" width="800" height="265"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Language&lt;/th&gt;
&lt;th&gt;PUBLIC Median&lt;/th&gt;
&lt;th&gt;VPC-MODE Median&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Go&lt;/td&gt;
&lt;td&gt;87ms&lt;/td&gt;
&lt;td&gt;85ms&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Node.js&lt;/td&gt;
&lt;td&gt;80ms&lt;/td&gt;
&lt;td&gt;84ms&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Java&lt;/td&gt;
&lt;td&gt;77ms&lt;/td&gt;
&lt;td&gt;85ms&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Python&lt;/td&gt;
&lt;td&gt;84ms&lt;/td&gt;
&lt;td&gt;87ms&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Identical once warm. VPC-MODE has higher cold-start spikes (ENI attachment takes 500-2500ms on first invocation) and takes longer to provision (~2.5 minutes vs 5 seconds for PUBLIC). Use VPC-MODE for security and compliance (network isolation, access to private resources), not for performance.&lt;/p&gt;

&lt;p&gt;The 3-way comparison (External vs In-VPC PUBLIC vs In-VPC VPC-MODE) shows the full picture:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F4zfr4r9n5exvwuuugsf5.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F4zfr4r9n5exvwuuugsf5.png" alt="3-Way Comparison" width="800" height="307"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Finding 4: Client SDK language does not affect warm latency
&lt;/h2&gt;

&lt;p&gt;Three Lambda functions were deployed, each written in its native language, calling its corresponding VPC-mode runtime:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Lambda Language&lt;/th&gt;
&lt;th&gt;SDK&lt;/th&gt;
&lt;th&gt;Warm Median&lt;/th&gt;
&lt;th&gt;Cold Start&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Go&lt;/td&gt;
&lt;td&gt;aws-sdk-go-v2&lt;/td&gt;
&lt;td&gt;98ms&lt;/td&gt;
&lt;td&gt;904ms&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Node.js&lt;/td&gt;
&lt;td&gt;@aws-sdk/client-bedrock-agentcore&lt;/td&gt;
&lt;td&gt;115ms&lt;/td&gt;
&lt;td&gt;873ms&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Python&lt;/td&gt;
&lt;td&gt;boto3&lt;/td&gt;
&lt;td&gt;96ms&lt;/td&gt;
&lt;td&gt;2,631ms&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Warm latency is identical. Cold start differs: Go and Node.js initialize in under 1 second, Python (boto3) takes 2.6 seconds due to interpreter and credential chain resolution overhead.&lt;/p&gt;

&lt;h2&gt;
  
  
  Finding 5: Framework choice adds zero measurable overhead
&lt;/h2&gt;

&lt;p&gt;Three variants of the same SageMaker SME agent were deployed, all using Claude Haiku 4.5 with the same system prompt and question:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F0k4r4uwssy3i0whu07oh.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F0k4r4uwssy3i0whu07oh.png" alt="Framework Comparison" width="799" height="308"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Framework&lt;/th&gt;
&lt;th&gt;Framework Overhead&lt;/th&gt;
&lt;th&gt;Image Size&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Raw boto3 (invoke_model)&lt;/td&gt;
&lt;td&gt;0.2ms&lt;/td&gt;
&lt;td&gt;180 MB&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Strands Agents SDK&lt;/td&gt;
&lt;td&gt;0.2ms&lt;/td&gt;
&lt;td&gt;347 MB&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;LangChain (ChatBedrock)&lt;/td&gt;
&lt;td&gt;0.1ms&lt;/td&gt;
&lt;td&gt;438 MB&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Framework processing overhead is under 0.3ms for all three. The model inference call (5-26 seconds depending on output length) accounts for over 95% of end-to-end latency. Choose frameworks based on developer productivity:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Raw boto3&lt;/strong&gt;: Lightest image, full control, manual orchestration&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Strands SDK&lt;/strong&gt;: Built-in tool registration, memory management, conversation handling&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;LangChain&lt;/strong&gt;: Rich ecosystem of chains, RAG integrations, prompt templates&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Finding 6: Connection pool size matters only at high concurrency
&lt;/h2&gt;

&lt;p&gt;Connection pool stress tests with boto3 (urllib3 backend) at varying pool sizes:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fdmmt70hf84lg9r74c5n3.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fdmmt70hf84lg9r74c5n3.png" alt="Connection Pool Stress" width="799" height="264"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Concurrency&lt;/th&gt;
&lt;th&gt;Pool=2 Mean&lt;/th&gt;
&lt;th&gt;Pool=10 Mean (default)&lt;/th&gt;
&lt;th&gt;Pool=50 Mean&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;1 thread&lt;/td&gt;
&lt;td&gt;84ms&lt;/td&gt;
&lt;td&gt;81ms&lt;/td&gt;
&lt;td&gt;81ms&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;10 threads&lt;/td&gt;
&lt;td&gt;153ms&lt;/td&gt;
&lt;td&gt;139ms&lt;/td&gt;
&lt;td&gt;132ms&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;50 threads&lt;/td&gt;
&lt;td&gt;204ms&lt;/td&gt;
&lt;td&gt;155ms&lt;/td&gt;
&lt;td&gt;113ms&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;100 threads&lt;/td&gt;
&lt;td&gt;254ms&lt;/td&gt;
&lt;td&gt;138ms&lt;/td&gt;
&lt;td&gt;142ms&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;No errors at any concurrency level. The default pool size (10) is adequate for most workloads.&lt;/p&gt;

&lt;h2&gt;
  
  
  Finding 7: Model inference dominates real agent latency
&lt;/h2&gt;

&lt;p&gt;For a real SageMaker SME agent answering technical questions:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F814spdrrpy5oex6cx46i.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F814spdrrpy5oex6cx46i.png" alt="Echo vs Real Agent" width="800" height="563"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Component&lt;/th&gt;
&lt;th&gt;Latency&lt;/th&gt;
&lt;th&gt;Percentage of E2E&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Container processing (framework + parsing)&lt;/td&gt;
&lt;td&gt;less than 1ms&lt;/td&gt;
&lt;td&gt;less than 0.01%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Platform routing (SigV4, DDB, Firecracker)&lt;/td&gt;
&lt;td&gt;95-375ms&lt;/td&gt;
&lt;td&gt;3-5%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Model inference (Claude Haiku 4.5)&lt;/td&gt;
&lt;td&gt;5,700-26,000ms&lt;/td&gt;
&lt;td&gt;95%+&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The model call scales with output token count (approximately 5ms per output token for Haiku 4.5).&lt;/p&gt;

&lt;h2&gt;
  
  
  What actually matters (ranked)
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Rank&lt;/th&gt;
&lt;th&gt;Factor&lt;/th&gt;
&lt;th&gt;Impact&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;td&gt;Caller location (in-VPC vs external)&lt;/td&gt;
&lt;td&gt;5-10x&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;2&lt;/td&gt;
&lt;td&gt;Model choice (for real agents)&lt;/td&gt;
&lt;td&gt;Determines total time&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;3&lt;/td&gt;
&lt;td&gt;SDK cold start (Go/Node vs Python)&lt;/td&gt;
&lt;td&gt;3x on first call only&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;4&lt;/td&gt;
&lt;td&gt;Connection pool size&lt;/td&gt;
&lt;td&gt;Mild at high concurrency&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;5&lt;/td&gt;
&lt;td&gt;Container language&lt;/td&gt;
&lt;td&gt;Zero impact&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;6&lt;/td&gt;
&lt;td&gt;Runtime network mode (PUBLIC vs VPC)&lt;/td&gt;
&lt;td&gt;Zero impact&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;7&lt;/td&gt;
&lt;td&gt;Client SDK language (warm)&lt;/td&gt;
&lt;td&gt;Zero impact&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;h2&gt;
  
  
  Optimization Recommendations
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Place callers in-VPC&lt;/strong&gt; with a &lt;code&gt;bedrock-agentcore&lt;/code&gt; interface endpoint for 5-10x latency reduction.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Reuse SDK clients&lt;/strong&gt; across invocations. Never create a new client per call. In Lambda, initialize at module scope outside the handler.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Do not optimize container language for latency.&lt;/strong&gt; Pick what your team knows. The platform routing floor makes it invisible.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Use VPC-MODE for security&lt;/strong&gt;, not performance. Identical once warm.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Consolidate multi-tool agents&lt;/strong&gt; into fewer runtimes. Each sequential InvokeAgentRuntime call adds platform routing overhead.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Use streaming&lt;/strong&gt; for better perceived performance. Model inference is the bottleneck.&lt;/p&gt;&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  Code Configuration Support
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Language&lt;/th&gt;
&lt;th&gt;Code Config (zip to S3)&lt;/th&gt;
&lt;th&gt;Container Config (Docker)&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Python&lt;/td&gt;
&lt;td&gt;Yes (3.10 through 3.14)&lt;/td&gt;
&lt;td&gt;Yes&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Node.js&lt;/td&gt;
&lt;td&gt;Yes (NODE_22)&lt;/td&gt;
&lt;td&gt;Yes&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Go&lt;/td&gt;
&lt;td&gt;No&lt;/td&gt;
&lt;td&gt;Yes&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Java&lt;/td&gt;
&lt;td&gt;No&lt;/td&gt;
&lt;td&gt;Yes&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;h2&gt;
  
  
  Architecture
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Caller (Lambda / ECS / App)
  -&amp;gt; [VPC Endpoint or Public Endpoint]
    -&amp;gt; AgentCore Platform (SigV4 validation, session lookup, routing)
      -&amp;gt; Container (PUBLIC: platform VM | VPC: ENI in your subnet)
        -&amp;gt; [Optional] Bedrock InvokeModel (model inference)
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;

&lt;h2&gt;
  
  
  Source Code
&lt;/h2&gt;

&lt;p&gt;All source code, raw data, Lambda functions, and charts are available here:&lt;/p&gt;


&lt;div class="ltag-github-readme-tag"&gt;
  &lt;div class="readme-overview"&gt;
    &lt;h2&gt;
      &lt;img src="https://assets.dev.to/assets/github-logo-5a155e1f9a670af7944dd5e12375bc76ed542ea80224905ecaf878b9157cdefc.svg" alt="GitHub logo"&gt;
      &lt;a href="https://github.com/Neloh" rel="noopener noreferrer"&gt;
        Neloh
      &lt;/a&gt; / &lt;a href="https://github.com/Neloh/agentcore-latency-benchmarks" rel="noopener noreferrer"&gt;
        agentcore-latency-benchmarks
      &lt;/a&gt;
    &lt;/h2&gt;
    &lt;h3&gt;
      AgentCore Runtime latency benchmarking: Go, Node.js, Java, Python across PUBLIC/VPC modes
    &lt;/h3&gt;
  &lt;/div&gt;
  &lt;div class="ltag-github-body"&gt;
    
&lt;div id="readme" class="md"&gt;&lt;div class="markdown-heading"&gt;
&lt;h1 class="heading-element"&gt;AgentCore Runtime Latency Benchmarks&lt;/h1&gt;
&lt;/div&gt;

&lt;p&gt;Comprehensive latency benchmarking of AWS Bedrock AgentCore Runtime across 4 programming languages (Go, Node.js, Java, Python), multiple network configurations, and caller locations.&lt;/p&gt;

&lt;div class="markdown-heading"&gt;
&lt;h2 class="heading-element"&gt;Key Findings&lt;/h2&gt;
&lt;/div&gt;

&lt;p&gt;&lt;/p&gt;&lt;div class="table-wrapper-paragraph"&gt;&lt;br&gt;&lt;br&gt;&lt;br&gt;&lt;br&gt;&lt;br&gt;&lt;br&gt;&lt;br&gt;&lt;br&gt;&lt;br&gt;&lt;br&gt;&lt;br&gt;&lt;br&gt;&lt;br&gt;&lt;br&gt;&lt;br&gt;&lt;br&gt;&lt;br&gt;&lt;br&gt;&lt;br&gt;&lt;br&gt;&lt;br&gt;&lt;br&gt;&lt;br&gt;&lt;br&gt;&lt;br&gt;&lt;br&gt;&lt;br&gt;&lt;br&gt;&lt;br&gt;&lt;br&gt;&lt;br&gt;&lt;br&gt;&lt;br&gt;&lt;br&gt;&lt;br&gt;&lt;br&gt;&lt;br&gt;&lt;table&gt;

&lt;thead&gt;

&lt;tr&gt;

&lt;th&gt;Factor&lt;/th&gt;

&lt;th&gt;Impact on Warm Latency&lt;/th&gt;

&lt;/tr&gt;

&lt;/thead&gt;

&lt;tbody&gt;

&lt;tr&gt;

&lt;td&gt;
&lt;br&gt;
&lt;strong&gt;Caller location&lt;/strong&gt; (in-VPC vs external)&lt;/td&gt;

&lt;td&gt;
&lt;br&gt;
&lt;strong&gt;5-10x&lt;/strong&gt; - the #1 factor&lt;/td&gt;

&lt;/tr&gt;

&lt;tr&gt;

&lt;td&gt;Model choice (for real agents)&lt;/td&gt;

&lt;td&gt;Determines total time (95%+)&lt;/td&gt;

&lt;/tr&gt;

&lt;tr&gt;

&lt;td&gt;SDK cold start (Go/Node vs Python)&lt;/td&gt;

&lt;td&gt;3x on first call only&lt;/td&gt;

&lt;/tr&gt;

&lt;tr&gt;

&lt;td&gt;Connection pool size&lt;/td&gt;

&lt;td&gt;Mild at high concurrency&lt;/td&gt;

&lt;/tr&gt;

&lt;tr&gt;

&lt;td&gt;Container language&lt;/td&gt;

&lt;td&gt;Zero impact&lt;/td&gt;

&lt;/tr&gt;

&lt;tr&gt;

&lt;td&gt;Runtime network mode (PUBLIC vs VPC)&lt;/td&gt;

&lt;td&gt;Zero impact&lt;/td&gt;

&lt;/tr&gt;

&lt;tr&gt;

&lt;td&gt;Client SDK language (warm)&lt;/td&gt;

&lt;td&gt;Zero impact&lt;/td&gt;

&lt;/tr&gt;

&lt;/tbody&gt;

&lt;/table&gt;&lt;/div&gt;&lt;br&gt;
&lt;p&gt;&lt;/p&gt;


&lt;div class="markdown-heading"&gt;
&lt;h2 class="heading-element"&gt;Test Results&lt;/h2&gt;
&lt;/div&gt;

&lt;div class="markdown-heading"&gt;
&lt;h3 class="heading-element"&gt;1. Four-Language Echo Comparison (In-VPC)&lt;/h3&gt;

&lt;/div&gt;

&lt;p&gt;All languages perform identically once warm. The platform routing floor (~80ms in-VPC) dominates.&lt;/p&gt;

&lt;p&gt;&lt;a rel="noopener noreferrer" href="https://github.com/Neloh/agentcore-latency-benchmarks/charts/agentcore-latency-comparison.png"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fraw.githubusercontent.com%2FNeloh%2Fagentcore-latency-benchmarks%2FHEAD%2Fcharts%2Fagentcore-latency-comparison.png" alt="4-Language Latency Comparison"&gt;&lt;/a&gt;&lt;/p&gt;




&lt;div class="markdown-heading"&gt;
&lt;h3 class="heading-element"&gt;2. In-VPC vs External Caller&lt;/h3&gt;

&lt;/div&gt;

&lt;p&gt;Calling from within the VPC (via &lt;code&gt;bedrock-agentcore&lt;/code&gt; interface endpoint) is 5-10x faster than calling over the public internet.&lt;/p&gt;

&lt;p&gt;&lt;a rel="noopener noreferrer" href="https://github.com/Neloh/agentcore-latency-benchmarks/charts/agentcore-latency-vpc-vs-external.png"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fraw.githubusercontent.com%2FNeloh%2Fagentcore-latency-benchmarks%2FHEAD%2Fcharts%2Fagentcore-latency-vpc-vs-external.png" alt="VPC vs External"&gt;&lt;/a&gt;&lt;/p&gt;




&lt;div class="markdown-heading"&gt;
&lt;h3 class="heading-element"&gt;3. Full Comparison: Echo vs Real Agent (SME + Claude Haiku 4.5)&lt;/h3&gt;

&lt;/div&gt;

&lt;p&gt;Model inference (Bedrock) accounts for 95%+ of…&lt;/p&gt;&lt;/div&gt;


&lt;/div&gt;
&lt;br&gt;
  &lt;div class="gh-btn-container"&gt;&lt;a class="gh-btn" href="https://github.com/Neloh/agentcore-latency-benchmarks" rel="noopener noreferrer"&gt;View on GitHub&lt;/a&gt;&lt;/div&gt;
&lt;br&gt;
&lt;/div&gt;
&lt;br&gt;


&lt;h2&gt;
  
  
  References
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;&lt;a href="https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/" rel="noopener noreferrer"&gt;Bedrock AgentCore Runtime documentation&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/strands-agents/sdk-python" rel="noopener noreferrer"&gt;Strands Agents SDK&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://python.langchain.com/docs/integrations/chat/bedrock/" rel="noopener noreferrer"&gt;LangChain AWS integration&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;

</description>
    </item>
    <item>
      <title>Chainlink on Docker: Part 1</title>
      <dc:creator>Sibonelo N.</dc:creator>
      <pubDate>Thu, 29 Jun 2023 06:34:50 +0000</pubDate>
      <link>https://dev.to/neloh/chainlink-on-docker-part-1-4nig</link>
      <guid>https://dev.to/neloh/chainlink-on-docker-part-1-4nig</guid>
      <description>&lt;p&gt;Adding off-chain data to a Blockchain network is not a straightforward process. For a Blockchain network to achieve Byzantine Fault tolerance, all the nodes or peer nodes in the network must unanimously agree on the state of the blockchain, also known as the world state. This means that the ledger, which contains the complete history of transactions, must always be consistent and verifiable by all nodes. No node can add a transaction to the ledger without the agreement of other peer nodes. To ensure resilience, every peer node maintains a copy of the ledger. Therefore, even if a particular node becomes unavailable due to network issues or malicious attacks, other nodes can continue to operate within the blockchain network and handle requests. If a compromised node tries to rejoin the network, it may need to be in-line with the blockchain rules (or policies) or be removed from the network through voting, especially if it continues to introduce conflicting transactions that deviate from the overall world state. In Blockchain, the principle of quorum rules applies, where the majority consensus is crucial, and mechanisms are in place to prevent a central entity from validating every transaction of each node.&lt;/p&gt;

&lt;p&gt;In my own opinion, &lt;a href="https://docs.chain.link/" rel="noopener noreferrer"&gt;Chainlink&lt;/a&gt; seems to be the leading oracle that allows integration of data that is considered off-chain to the Blockchain network. With the consistent trend of Web3 and Generative AI it is important to know what Chainlink is. What is the role of Chainlink in various use cases such as  Decentralized Finance, Non-Fungible Tokens (NFTs), Music and Gaming. In a overly simplified way, basically, an oracle or mainly a "decentralised oracle network" (DON) allows &lt;code&gt;off-chain&lt;/code&gt; data to be integrated to a Blockchain network such as Ethereum Mainnet or the Testnets such as Sepolia. &lt;/p&gt;

&lt;p&gt;The aim of this article is to cover the steps that one would  follow when configuring a Chainlink node locally via Docker. Please use the &lt;a href="https://docs.chain.link/chainlink-nodes/resources/requirements" rel="noopener noreferrer"&gt;link&lt;/a&gt; for the hardware requirements if you aim to set this up within your own machine. The steps  are as follows for this article:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;PostgreSQL database setup&lt;/li&gt;
&lt;li&gt;Details on setting up an Alchemy or Infura free account for Sepolia, getting the API KEY.&lt;/li&gt;
&lt;li&gt;Configure an Testnet environment via docker for Sepolia as well.&lt;/li&gt;
&lt;li&gt;Attempt to fulfil requests with the Chainlink node through a Smart contract written in Solidity.&lt;/li&gt;
&lt;li&gt;Validation of transactions via &lt;a href="https://sepolia.etherscan.io/contractsVerified" rel="noopener noreferrer"&gt;Etherscan&lt;/a&gt;.&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  Setting up PostgreSQL
&lt;/h3&gt;

&lt;p&gt;You can pull an existing docker for PostgreSQL and setup the following commands to create a database user, password and the database where Chainlink will create the relational databases of what store Chainlink events, transactions and events.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;docker run --name cl-postgres -v $HOME/.chainlink-sepolia/db:/var/lib/postgresql/data -e POSTGRES_PASSWORD=myPostgresPW -d -p 5432:5432 postgres:11.12
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Once the above has been ran, you need to create the USER and database.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;sudo docker exec -it cl-postgres psql -U postgres -c "CREATE USER $USERNAME WITH PASSWORD '$PASSWORD';"

sudo docker exec -it cl-postgres psql -U postgres -c "CREATE DATABASE "chainlink_sepoliadb";"

sudo docker exec -it cl-postgres psql -U postgres -c "GRANT ALL PRIVILEGES ON DATABASE "chainlink_sepoliadb" TO $USERNAME;"
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The above is crucial as these PostgreSQL will be the database where the tables for Chainlink node are stored.&lt;/p&gt;

&lt;p&gt;You can use the following ways to test if the PostgreSQL database is up and running&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;if you have &lt;code&gt;psql&lt;/code&gt; CLI within your local machine, you can run the command &lt;code&gt;psql postgresql://$USER:$PASSWORD@localhost:5432/chainlink_sepoliadb?sslmode=disable&lt;/code&gt; to see if you get logged in to the PostgreSQL shell where the database is the &lt;code&gt;chainlink_sepolia&lt;/code&gt; as mentioned within the command. For an example, see this command below:
&lt;/li&gt;
&lt;/ul&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;~/.chainlink-sepolia$ psql postgresql://$USER:$PASSWORD@localhost:5432/chainlink_sepoliadb?sslmode=disable
psql (14.8 (Ubuntu 14.8-0ubuntu0.22.04.1))
Type "help" for help.

chainlink_sepoliadb=&amp;gt; 
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;ul&gt;
&lt;li&gt;Or you can use &lt;a href="https://hub.docker.com/r/dpage/pgadmin4/" rel="noopener noreferrer"&gt;pgAdmin&lt;/a&gt; in order to connect to your &lt;code&gt;localhost&lt;/code&gt; on port &lt;code&gt;5432&lt;/code&gt; as it was exposed via Docker commands. You can do administrator related tasks on the database via this tool.&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  Signup for Alchemy Sepolia Testnet.
&lt;/h3&gt;

&lt;p&gt;You need to get the Websocket URL of your Alchemy application. The &lt;code&gt;.env&lt;/code&gt; file for the Chainlink node will need the Websocket URL one can always use HTTPS URL to interact with blockchain network however for the Chainlink setup, you need Websocket URL.&lt;/p&gt;

&lt;h3&gt;
  
  
  Setting up the Testnet client.
&lt;/h3&gt;

&lt;p&gt;This is up to you. However, in this tutorial we are using the Sepolia Geth client. Using the free faucets to fund this type of testnet for fake currency seems to be currently supported as Goerli is deprecated. You can do the configuration for Goerli but the tutorial on &lt;a href="https://docs.chain.link/chainlink-nodes/v1/running-a-chainlink-node" rel="noopener noreferrer"&gt;official documentation&lt;/a&gt; places Sepolia network forward and regards Goerli deprecated and might have potential issues when setting up. One suggestion, is to run the commands that produce constant logs via the Unix &lt;code&gt;screen&lt;/code&gt; command so that they execute in the background of each terminal. &lt;/p&gt;

&lt;p&gt;As we did for PostgreSQL, you can configure the &lt;a href="https://docs.chain.link/chainlink-nodes/resources/run-an-ethereum-client" rel="noopener noreferrer"&gt;Geth client&lt;/a&gt; for Sepolia using the following steps:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;&lt;code&gt;docker pull ethereum/client-go:latest&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;code&gt;mkdir ~/.geth-sepolia&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;code&gt;docker run --name eth -p 8546:8546 -v ~/.geth-sepolia:/geth -it \
ethereum/client-go --sepolia --ws --ipcdisable \
--ws.addr 0.0.0.0 --ws.origins="*" --datadir /geth&lt;/code&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;If the container was stopped for any reason, you can restart it with the following command&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;sudo docker start -i eth
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;You can detach from the running client by using the commands &lt;code&gt;Ctrl + P, Ctrl + Q&lt;/code&gt; as we do not want to stop it. Chainlink requires this container on running status in order to connect to it.&lt;/p&gt;

&lt;h3&gt;
  
  
  Setting up the Chainlink node
&lt;/h3&gt;

&lt;p&gt;Once you have the docker up and running for PostgreSQL, Sepolia testnet and you have setup the hidden files &lt;code&gt;.api&lt;/code&gt; file API KEY for Infura/Alchemy, .password file for your Chainlink UI and a &lt;code&gt;.env&lt;/code&gt; file within a hidden directory &lt;code&gt;.chainlink-sepolia&lt;/code&gt;. The .env will contain all your underlying network configurations such as the ChainID for the Testnet and the LINK address to use for your chainlink node. It might look like the following:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;LOG_LEVEL=debug
ROOT=/chainlink
ETH_CHAIN_ID=11155111
CHAINLINK_TLS_PORT=0
SECURE_COOKIES=false
CHAINLINK_DEV=true
LINK_CONTRACT_ADDRESS=0x779877A7B0D9E8603169DdbD7836e478b4624789
ALLOW_ORIGINS=*
MIN_INCOMING_CONFIRMATIONS=1
ETH_URL=wss://eth-sepolia.g.alchemy.com/v2/RTDG...
DATABASE_URL=postgresql://$USER:$PASSWORD@localhost:5432/chainlink_sepoliadb?sslmode=disable

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;You need to make sure that you do not commit the above file to Github or any version control service. You start the chainlink node by running the following command.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;sudo docker run --name chainlink_sepolia \ 
     --network host -p 6688:6688 \
     -v $HOME/.chainlink-sepolia:/chainlink \
     -it --env-file=$HOME/.chainlink-sepolia/.env \ 
      smartcontract/chainlink:1.1.0 local n -p \
      /chainlink/.password -a /chainlink/.api
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This generates logs and push to a local folder in the same hidden directoty called, &lt;code&gt;log.jsonl&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;Once the docker containers are up and running, you can visit the URL &lt;code&gt;http://localhost:6688&lt;/code&gt; and you should be able to see the Chainlink UI tool. This is where you should locate the Chainlink key, along with the Job UI console where you would define your HTTP request syntax using TOML.&lt;/p&gt;

&lt;h3&gt;
  
  
  Setup a Smart contract within Solidity for a basic HTTP request (will be included in &lt;strong&gt;Part 2&lt;/strong&gt;):
&lt;/h3&gt;

&lt;p&gt;This section will be in &lt;strong&gt;part 2&lt;/strong&gt; of this article. I will discuss the underlying smart contract that is responsible for reaching out an world-wide web service (via HTTPs) to fetch &lt;code&gt;off-chain&lt;/code&gt;. Further I aim to discuss how Chainlink works with the &lt;code&gt;on-chain&lt;/code&gt; smart contracts to make sure the blockchain network remains deterministic. This part would also involve setting up a frontend for the project and cover security best practices.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Please note that the suggestions made here are purely from my understanding of the Chainlink software and can be corrected where misleading or false. The purpose is to learn and grow in the space of Web3, so comments are welcome for improvements.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>chainlink</category>
      <category>docker</category>
      <category>web3</category>
      <category>beginners</category>
    </item>
  </channel>
</rss>
