DEV Community

Cover image for Run Amazon Bedrock locally, with real completions from Ollama
Nahuel Nucera
Nahuel Nucera

Posted on

Run Amazon Bedrock locally, with real completions from Ollama

#aws #bedrock #llm #opensource

MiniStack 1.4.0 ships four new services that emulate Amazon Bedrock end to end, and one env var that turns the mock into a real LLM backend.

The default: deterministic, shape-perfect mocks

Out of the box, Converse and InvokeModel return deterministic mock responses whose wire shape matches the model family you asked for. Anthropic, Titan, Nova, Llama, Mistral, Cohere and AI21 each have distinct response schemas, and MiniStack selects the right one by model ID prefix. Streaming operations (ConverseStream, InvokeModelWithResponseStream) use the real eventstream wire format, so your SDK's streaming parser runs exactly as against AWS.

That means your test suite exercises the real code paths: request building, response parsing, stream handling. No network call, no API bill.

One honest caveat: token counts are a chars/4 heuristic. Shape-correct, but don't assert exact values against a mock.

The upgrade: real completions from your local model 

docker run -p 4566:4566 \
  -e MINISTACK_BEDROCK_PROXY_URL=http://host.docker.internal:11434/v1 \
  ministackorg/ministack
Enter fullscreen mode Exit fullscreen mode

With the proxy set to any OpenAI-compatible /chat/completions endpoint (Ollama, llama.cpp, vLLM), MiniStack translates your Bedrock prompt to OpenAI shape, forwards it, and translates the response back into Bedrock shape:

bedrock = boto3.client("bedrock-runtime", endpoint_url="http://localhost:4566",
                       region_name="us-east-1", aws_access_key_id="test",
                       aws_secret_access_key="test")
resp = bedrock.converse(
    modelId="anthropic.claude-3-5-sonnet-20240620-v1:0",
    messages=[{"role": "user", "content": [{"text": "Hello from MiniStack!"}]}],
)
print(resp["output"]["message"]["content"][0]["text"])  # a real completion
Enter fullscreen mode Exit fullscreen mode

Unchanged AWS SDK. Real model. Zero cloud.

If the proxy endpoint is unreachable, MiniStack falls back to the mock silently. Your CI doesn't care whether Ollama is running.

The rest of the surface

  • bedrock (control plane): 66 operations verified against botocore. Foundation model catalog with real model IDs, inference profiles, guardrails with versioning, custom and imported models, provisioned throughput, and the customization, import, copy and batch-invocation job families.
  • bedrock-agent: 72 operations. Agents with versions, aliases and action groups, knowledge bases with data sources and ingestion jobs, flows, prompts.
  • bedrock-agent-runtime: 31 operations. InvokeAgent, Retrieve and RetrieveAndGenerate, reranking, the full session API, flow executions.

All four services are account and region scoped (1.4.0 also shipped multi-region support, separate post).

The Bedrock work was led by dcabib.

Try it 

docker run -p 4566:4566 ministackorg/ministack
Enter fullscreen mode Exit fullscreen mode

MIT, free forever. Changelog: https://ministack.org/blog/changelog-v1-4-0.html

Top comments (0)