DEV Community: Roger Viñas Alcon

Strands Agents + Langfuse Evaluations

Roger Viñas Alcon — Thu, 02 Jul 2026 13:18:29 +0000

Before diving in, check out AI was supposed to take my job — instead it gave me a new one: Evaluations, a presentation that walks through this PoC.

See source code at Github

In this project we will build a Python banking assistant agent using Strands Agents and make it observable and continuously evaluated using Langfuse — step by step.

Strands Agents is a lightweight Python SDK for building LLM-powered agents with tool use and session memory, open-sourced by AWS in May 2025. It is Python-native — which pairs well with the Langfuse Python SDK — and new enough to be worth exploring. Any other Python agent framework would work just as well for this PoC.

With classic applications, quality is enforced through unit tests, integration tests, and static analysis — every function has a defined contract and a deterministic output you can assert on. In production, metrics (error rates, latency, memory) surface failures reliably.

AI applications break both of these. The same input may yield different outputs on each run — wording changes, tools get called in a different order, edge cases surface unpredictably. And in production, a request can return 200 OK in 300ms with a confident, completely wrong answer — classic metrics won't catch it. You need something more!

That's where traces and evaluations come in — supported by a growing number of platforms: Langfuse, Arize Phoenix, MLflow, LangSmith, W&B Weave, Datadog, AWS AgentCore, Azure AI Foundry, Google Vertex AI

This PoC uses Langfuse because it is open-source and is self-hostable with a single docker compose up, providing these features:

Tracing — recording a structured tree of every LLM call, tool call, and sub-agent step: inputs, outputs, latency, cost.
Evaluations — running scored assessments of agent outputs:
- Offline — run against a fixed, curated dataset before or after a change; re-runnable as a CI quality gate.
- Online — async, triggered by live traces; catch issues that didn't appear in your fixed dataset.
External Evaluations — attaching scores to live traces programmatically from your own code.
Annotation Queues — routing traces to human reviewers via explicit programmatic calls.
Prompt Management — versioning prompt templates and pulling them at runtime via SDK.

The app under evaluation is the banking sentinel — a customer support agent for ROGERVINAS bank built with Strands Agents: 3 mock accounts with 5 transactions each, and tools to freeze/unfreeze cards, look up transactions, and open or track disputes.

Ready? Get the agent and Langfuse running (Configuration → Run), open the chat UI and play with it, then go through the Implementation steps to see how each piece works.

Configuration
Run
Implementation
- Step 1: The Banking Agent
- Step 2: Tracing
- Step 3: Strands Native Evaluations
- Step 4: Experiments
- Step 5: Online Evaluations
- Step 6: External Evaluations
- Step 7: Annotation Queues
- Step 8: Prompt Management
CI/CD
Documentation

Configuration

Prerequisites

Python (version pinned in .python-version)
uv
Docker + Docker Compose
A model provider (see below)

Install dependencies

uv sync --extra dev --extra evals

dev — dependencies for unit tests
evals — dependencies for evaluations

Environment

cp .env.example .env

Edit .env to set your model provider and Langfuse credentials.

Model providers

Set MODEL_PROVIDER in .env:

Provider	`MODEL_PROVIDER`	Requirements
Ollama (default)	`ollama`	`ollama serve` + `ollama pull llama3.1:8b`
AWS Bedrock	`bedrock`	AWS credentials configured
Google Gemini	`gemini`	`GOOGLE_API_KEY` in `.env`

Langfuse

We use the self-hosted with Docker Compose deployment. This PoC's docker-compose-langfuse.yml is a copy of the official docker-compose.

To start:

docker compose -f docker-compose-langfuse.yml up -d

Open the Langfuse UI at http://localhost:3000 → project banking-sentinel, with these pre-provisioned credentials:

Email:      admin@local.dev
Password:   password

To stop:

docker compose -f docker-compose-langfuse.yml down

To stop and delete all data:

docker compose -f docker-compose-langfuse.yml down -v

Run

uv run uvicorn banking_sentinel.api:app --reload

Open http://localhost:8000 and try it out:

Use the textboxes to simulate a different User ID, Account ID, Tier, and Session ID.
Chat with the agent — for example, say "Help, I do not have Netflix!".
Each answer may include suggested actions you can click directly, plus 👍 / 👎 feedback buttons.

Agent traces are sent to Langfuse automatically via OpenTelemetry, as the LANGFUSE_* and OTEL_* env vars are set in .env.

Implementation

With the agent and Langfuse running, let's walk through how each piece works, step by step.

Step 1: The Banking Agent

The domain: a banking customer support agent — the Sentinel — for ROGERVINAS bank. Three mock accounts (ACC-1001, ACC-1002, ACC-1003), five transactions each, and seven tools:

freeze_card — Freeze a card
unfreeze_card — Unfreeze a card
is_card_frozen — Check freeze status
get_transactions — List transactions between two dates
open_dispute — Open a dispute on a transaction
get_dispute_status — Check dispute status
list_disputes — List all disputes for an account

The implementation is intentionally minimal — a single process, file-based session storage, no RAG (the knowledge base is short enough to inline directly in the system prompt), no MCP, no external service calls. The goal here is to keep the agent simple so the focus stays on observability and evaluations. The full implementation lives under src/banking_sentinel; if you are new to Strands, follow the Strands Agents tutorial first.

In production, the agent would run behind a load balancer with multiple replicas, connect to real external services via MCP or direct API calls, and swap the local file-based session storage for a persistent store — see the Strands session management docs.

Step 2: Tracing

Traces are not standardised — what you get depends entirely on the framework and instrumentation. The GenAI OTel semantic conventions are still experimental, so SDKs and platforms implement them inconsistently — always verify your traces in the UI before relying on them.

In this PoC, for example, the Strands [otel] extra emits spans but does not set trace-level input/output on the root span — and we need those in later steps. So api.py wraps the entire /chat request in langfuse.start_as_current_observation(), which sets input/output on the root span while capturing the Strands OTel spans as children:

with langfuse.start_as_current_observation(name="banking-sentinel-chat", as_type="generation") as span:
    # ... run the agent ...
    span.update(input=request.message, output=response.answer, prompt=prompt_obj)

This produces the following span hierarchy in Langfuse:

banking-sentinel-chat  ← Langfuse-native root span (input/output/user_id)
  └── Strands OTel spans  ← captured automatically

Open http://localhost:3000 → Traces to see it:

Step 3: Strands Native Evaluations

The Strands Evals SDK provides Case, Experiment, and OutputEvaluator. No Langfuse required — runs fully offline and produces a local report. The full code for this step lives in evals/strands/run_evaluations.py.

Each Case bundles an input and expected output:

CASES = [
    Case(
        name="unauthorized-netflix-charge",
        input={
            "userId": "user-1001", "accountId": "ACC-1001", "accountTier": "Standard",
            "message": "I don't have Netflix but I see a charge on my account",
        },
        expected_output={
            "suggestedActions": ["FREEZE_CARD"],
            "claim": "The AI agent found a Netflix charge of 9.99 and offered the user to open a dispute",
        },
    ),
    # ...
]

Two evaluators score each result — one deterministic, one LLM-as-judge:

class CorrectnessEvaluator(OutputEvaluator):
    pass

class ClaimEvaluator(OutputEvaluator):
    pass

correctness_evaluator = CorrectnessEvaluator(
    model=_model,
    rubric="Score 1.0 if the actual output's suggested_actions contains all actions listed in expected_output's suggestedActions. Score 0.0 if any expected action is missing.",
)

claim_evaluator = ClaimEvaluator(
    model=_model,
    rubric="Score 1.0 if the actual output's answer matches the claim in expected_output. Score 0.0 if the answer does not match the claim.",
)

Note: Strands Evals can evaluate any Python callable — there is no coupling to the Strands agent itself. If you prefer a different evaluation framework, alternatives include DeepEval, Ragas (RAG-focused), Braintrust autoevals, or plain pytest with custom assertions.

There are two ways to run the task:

Embedded — the agent is instantiated in-process; external services are mocked, and you can inspect internal state (white-box). Fast, no server needed, ideal for CI. Use this when you want fast, isolated, reproducible runs.
API — the task hits a running server with real external services (black-box). Use this to validate against a live deployment or when mocking is not practical.

Run embedded — no server needed:

uv run python -m evals.strands.run_evaluations embedded

Run against a running server:

uv run python -m evals.strands.run_evaluations api --url http://localhost:8000

Report should look like:

============================================================
Evaluator: CorrectnessEvaluator
Overall score: 1.00
  ✅ unauthorized-netflix-charge: score=1.00 — suggested_actions contains the expected 'FREEZE_CARD'.
  ✅ expired-dispute-window: score=1.00 — suggested_actions contains the expected 'FREEZE_CARD'.

============================================================
Evaluator: ClaimEvaluator
Overall score: 1.00
  ✅ unauthorized-netflix-charge: score=1.00 — identifies the Netflix charge and offers to open a dispute.
  ✅ expired-dispute-window: score=1.00 — explains the 14-day dispute window has expired.

✅ Evaluation PASSED

Step 4: Experiments

A Langfuse experiment is an offline evaluation run: your agent is executed against a curated dataset, each output is scored automatically, and the results are stored so you can compare across code versions, prompt changes, and model upgrades over time.

A dataset is a versioned collection of test cases — each item has an input, an expected_output, and optional metadata. Each experiment execution is named and stored against that dataset, with evaluator scores recorded per item.

Where Step 3: Strands Native Evaluations produces a local pass/fail report with no history, Langfuse Experiments add:

Comparison across runs — see how scores change between code versions, prompt changes, or model upgrades side by side in the dashboard
Persistent results — every run is stored; you can go back and audit any historical experiment

Both Step 3: Strands Native Evaluations and Step 4: Experiments act as a CI quality gate — the script exits non-zero if scores drop below threshold.

This step has three parts:

Create the dataset
Implement the experiment
Run the experiment

Create the dataset

Each item has an input, an expected_output, and optional metadata. Datasets can be created in two ways:

Via UI: go to http://localhost:3000 → Datasets → New dataset, then add items manually
Programmatically (idempotent — safe to run repeatedly), see evals/langfuse/create_dataset.py:

uv run python -m evals.langfuse.create_dataset

Open http://localhost:3000 → Datasets → Items to see the dataset items:

Implement the experiment

Evaluators are plain Python callables — there are no built-in evaluators in the SDK and no base class to inherit from. Any function that matches this signature works:

from langfuse.experiment import Evaluation

def my_evaluator(
    *,
    input: Any,           # the dataset item's input
    output: Any,          # what the task returned
    expected_output: Any, # the dataset item's expected_output
    metadata: Optional[Dict[str, Any]],
    **kwargs: Any,
) -> Evaluation:          # or List[Evaluation] for multiple metrics at once
    ...
    return Evaluation(
        name="my-metric",   # metric name shown in the dashboard
        value=1.0,          # int | float | str | bool
        comment="optional", # shown alongside the score
    )

We implement the same two evaluators as in Step 3: Strands Native Evaluations — one deterministic (correctness), one LLM-as-judge (claim). The LLM-as-judge runs locally using whatever MODEL_PROVIDER you have configured (Ollama, Bedrock, Gemini) — it is not Langfuse's evaluation infrastructure, just your own model called from Python (see evals/langfuse/run_experiment.py):

def correctness_evaluator(*, output, expected_output, **kwargs) -> Evaluation:
    """Deterministic: checks if all expected suggested actions are present."""
    expected = set(expected_output.get("suggestedActions", []))
    actual = set(output.get("suggested_actions", []))
    score = len(expected & actual) / len(expected) if expected else 1.0
    return Evaluation(name="correctness", value=score, comment=f"Expected {expected}, got {actual}")

def claim_evaluator(*, output, expected_output, **kwargs) -> Evaluation:
    """LLM-as-judge: runs locally with your configured MODEL_PROVIDER."""
    judge = Agent(model=_model, callback_handler=lambda **_: None)
    result = judge(
        f"Does the following answer match the claim? Reply with YES or NO only.\n\n"
        f"Answer: {output['answer']}\n\nClaim: {expected_output['claim']}"
    )
    return Evaluation(name="claim_match", value=1.0 if "YES" in str(result).upper() else 0.0)

result = langfuse.run_experiment(
    name="banking-sentinel",
    data=dataset.items,
    task=embedded_task,
    evaluators=[correctness_evaluator, claim_evaluator],
    max_concurrency=1,
)

Run the experiment

Run embedded — no server needed:

uv run python -m evals.langfuse.run_experiment embedded

Run against a running server:

uv run python -m evals.langfuse.run_experiment api --url http://localhost:8000

Open http://localhost:3000 → Datasets → Experiments to see the experiment runs:

Step 5: Online Evaluations

Langfuse can automatically score live traces as they arrive — no code changes needed. In this PoC all chat traces are tagged banking-sentinel, and the root span of each is named banking-sentinel-chat, making them easy to target.

Setup:

1 — Add LLM Connection:
Go to http://localhost:3000 → Settings → LLM Connections → add your model provider API key.

2 — Set default evaluation model:
Go to http://localhost:3000 → Evaluators → set the Default Evaluation Model to the connection you just added.

3 — Create the evaluator:
Go to http://localhost:3000 → Evaluators → Create Evaluator. Two options:

Create from scratch (custom prompt) — your own domain-aware prompt. This is what we use for the PoC.
Use existing (built-in templates, e.g. Helpfulness, Hallucination) — often too generic for a specific domain.

Fill in the custom evaluator:

Name — e.g. banking-sentinel-helpfulness
Model — leave Use default evaluation model checked (the model you set in step 2)
Evaluation prompt — reference the trace content with {{input}} and {{output}} (see below)
Score type — numeric, boolean, or categorical
Score reasoning prompt and Score range prompt — leave at their defaults or customize at will

Evaluation prompt:

You are evaluating a banking customer-support assistant for ROGERVINAS bank.
Customers often report problems informally, without using precise banking
terms. Evaluate how helpful the assistant's reply is: does it identify the
relevant transaction, explain the applicable policy (e.g. the dispute window),
and offer concrete next steps (freeze card, open dispute)?

Customer message:
{{input}}

Assistant reply:
{{output}}

4 — Configure the rule:

Set target to Observations, filter by Type = GENERATION
Ensure Run on live incoming observations is checked (it is the default)
Add filter: Tags → any of → banking-sentinel
Add filter: Name → any of → banking-sentinel-chat — targets only the root span; avoids double-scoring the inner Strands generation (both carry the tag but have different names)
Set Sampling (100% is fine for PoC — reduce in production to control costs)
Map the evaluator's prompt variables to the trace fields accordingly
Click Execute — scores existing matching observations immediately and new ones going forward

Open http://localhost:3000 → Traces and click a banking-sentinel-chat trace. The banking-sentinel-helpfulness score shows as a small badge on the root span, with the full detail in the Scores tab:

Note: The online evaluations themselves produce traces — each LLM-as-judge call is a generation Langfuse records.

Note: The Langfuse API to create evaluators and rules programmatically is unstable and only available on Langfuse Cloud — not in self-hosted deployments.

Step 6: External Evaluations

Langfuse lets you attach scores to any trace programmatically from your own code using langfuse.create_score(). Common use cases include:

User feedback — 👍/👎 ratings from end users
Guardrail results — PII checks, content policy, format validation
Agent self-scoring — quality signal computed inline and submitted back
Custom pipelines — any score your application logic can produce

In this PoC we implement user feedback as our example. Automated evaluators catch a lot, but human judgment is still essential — collecting 👍/👎 feedback from real users gives you a ground-truth signal that no automated evaluator can fully replace.

How it works:

The /chat endpoint returns a trace_id in every response (read from span.trace_id)
The chat UI attaches 👍/👎 buttons to each assistant message, keyed to that trace_id
On click, the UI posts to /feedback with value: 1.0 (👍) or value: 0.0 (👎)
The backend (api.py) calls langfuse.create_score() — the score appears on the trace immediately

@app.post("/feedback")
def feedback_endpoint(request: FeedbackRequest):
    langfuse.create_score(
        trace_id=request.trace_id,
        name="user-feedback",
        value=request.value,   # 1.0 = 👍, 0.0 = 👎
        comment=request.comment,
    )

In the chat UI, send a message and click 👍 or 👎 on the reply. Then open http://localhost:3000 → Traces and click that trace. The user-feedback score shows as a small badge on the root span, with the full detail in the Scores tab:

Step 7: Annotation Queues

Annotation queues are a human review workflow — domain experts manually score traces to build ground truth, validate LLM-as-judge results, or investigate failures.

Key concept: Langfuse provides the queue infrastructure and a programmatic API to add items — but there are no automatic routing rules or triggers built in. Items only enter a queue through an explicit call, either from the UI (ad-hoc) or from your code. Your code owns the routing logic.

Common triggers you can implement:

User gives 👎 — route negative feedback traces for human investigation
Experiment score below threshold — enqueue failing traces to build better ground truth
Online evaluator scores low — poll scores and enqueue traces below a quality bar
Specific intent detected — route traces matching certain patterns (complaints, edge cases)
Random sampling — periodically enqueue a % of production traces for ongoing quality checks

1 — Create the queue (once, idempotent):

uv run python -m evals.langfuse.create_annotation_queue

This creates a quality score config (numeric, 0–1) and the banking-sentinel-review queue that uses it, so reviewers score each trace on quality — you could attach more score configs for other dimensions (see evals/langfuse/create_annotation_queue.py).

Set ANNOTATION_QUEUE_ID in .env to the returned queue ID.

2 — Enqueue traces (example: on 👎):
In this PoC, the /feedback endpoint in api.py adds the trace to the queue whenever the user gives a thumbs down:

if request.value == 0.0 and _annotation_queue_id:
    langfuse.api.annotation_queues.create_queue_item(
        _annotation_queue_id,
        object_id=request.trace_id,
        object_type=AnnotationQueueObjectType.TRACE,
    )

3 — Review the queue:

Go to http://localhost:3000 → Human Annotation
Open banking-sentinel-review
Review the conversation
Assign a quality score (and optionally add a comment)
Correct the output if necessary
Click Mark Completed
Scores and corrected output appear on the trace and contribute to your evaluation dashboard

Annotating an item — assign the quality score and an optional comment:

The quality score then shows as a small badge on the root banking-sentinel-chat span:

Step 8: Prompt Management

Langfuse can store and version system prompts independently of your code — iterate on the prompt without redeploying the app.

1 — Create the prompt programmatically:

uv run python -m evals.langfuse.create_prompt

Each run creates a new version (see evals/langfuse/create_prompt.py). The production label is set automatically, so it is served at runtime.

2 — Or create the prompt via the UI:

Go to http://localhost:3000 → Prompts → New prompt, then:

Name it banking-sentinel-system, type Text
Paste the template using {{variable}} syntax (Mustache)
Add the production label
Save

Either way, the prompt appears in Prompts, versioned and labeled production:

3 — Use the prompt from code:

Set USE_LANGFUSE_PROMPT=true in .env and create_agent() in agent.py fetches the prompt from Langfuse; otherwise it uses the hardcoded template:

def create_agent(langfuse, model, tools, user_tier, account_id, reference_date, session_manager=None) -> tuple:
    if langfuse is not None and os.getenv("USE_LANGFUSE_PROMPT", "false").lower() == "true":
        ...  # fetch the versioned prompt via langfuse.get_prompt(...)
    else:
        ...  # use the hardcoded template
    ...

It returns prompt_obj, which api.py will pass back to Langfuse to link the prompt version to the trace:

span.update(input=request.message, output=response.answer, prompt=prompt_obj)

The trace then shows a Prompt: banking-sentinel-system - v1 tag linking to the exact version used:

Benefits: version history, compare prompt versions across experiments, iterate without redeploying, A/B test prompts.

CI/CD

For this PoC, CI runs three sequential jobs that gate on each other — each stage must pass before the next starts:

Build — installs dependencies, builds the package, runs unit tests
Standalone Evals — runs Strands native evaluations in embedded mode (no Langfuse). Fails if any score drops below a threshold.
Langfuse Evals — runs Langfuse experiments and reports results to the dashboard. Fails if any score drops below a threshold. For this PoC we start Langfuse with Docker just for the CI run, but a real setup would point at a shared, always-on instance.

This means a code or prompt change that degrades agent quality will fail CI before it can reach production.

Langfuse acts as an active quality gate:

Traces, scores, and experiment history accumulate across every PR and deploy — production traces surface regressions and datasets grow from real failures.
A deployment job runs only after all eval jobs pass, so CI blocks deploys that would degrade quality — with score thresholds tuned per metric as you build up baseline data.

This is the continuous AI Engineering Loop as documented by Langfuse:

Documentation

Happy GenAI coding! 💙

Spring Boot AI Evaluation Testing

Roger Viñas Alcon — Tue, 22 Apr 2025 07:26:28 +0000

See source code at Github

In this example, inspired by Building Agents with AWS: Complete Tutorial, we will build a simple AI agent application using Spring AI, highlighting key features like:

Chat Client API and its Advisors
Model Context Protocol (MCP)
Retrieval Augmented Generation (RAG)
Testing with AI Model Evaluation 🤩

The original example uses AWS Bedrock, but one of the great things about Spring AI is that with just a few config tweaks, the same code works with any other supported model. This project supports three profiles: Ollama (local), Google Gemini and AWS Bedrock.

The application features an AI agent that helps users book accommodations in tourist destinations.

Through MCP, the agent can use the following tools:

Weather Tool: Retrieves weather information for a specific city and date.
Booking Tool: Books accommodations in a city for a specific date.

The Weather tool will be implemented locally using MCP, while the Booking tool will be provided by a remote MCP server. The current date is provided via the system prompt. Additional information about cities will be retrieved from a vector store using RAG.

Implementation
- MCP Server
- Chat Server
  - Components
    - Weather Tool
    - Booking Tool
    - Chat Client
    - Chat Service
    - Chat Controller
    - Vector Store Initializer
Configuration
- Ollama profile
- Gemini profile
- Bedrock profile
Test
- Test MCP Server
- Test Chat Server
Run
- Running the vector database
- Running Ollama locally
- Running the application
How to use other AI models
Documentation

Implementation

MCP Server

As is often the case with Spring Boot, implementing the MCP Server is pretty straightforward. Following the MCP Server Boot Starter guide, you just need to:

1) Add the spring-ai-starter-mcp-server-webflux or spring-ai-starter-mcp-server-webmvc dependency

2) Create an instance and annotate it with @Tool and @ToolParam:

@Service // or @Bean / @Component
class BookingTool(private val bookingService: BookingService) {
  @Tool(
    description = "make a reservation for accommodation for a given city and date",
  )
  fun book(
    @ToolParam(description = "the city to make the reservation for")
    city: String,
    @ToolParam(description = "the check-in date, when the reservation begins")
    checkinDate: LocalDate,
    @ToolParam(description = "the check-out date, when the reservation ends")
    checkoutDate: LocalDate
  ): String = bookingService.book(city, checkinDate, checkoutDate) // Delegate to a service
}

3) Register it as a MethodToolCallbackProvider:

@Configuration
class BookingToolConfiguration {
  @Bean
  fun bookingToolCallbackProvider(bookingTool: BookingTool) =
    MethodToolCallbackProvider.builder()
      .toolObjects(bookingTool)
      .build()
}

Chat Server

The Chat Server is a Spring Boot application built with the following dependencies:

spring-boot-starter-web or -webflux - to expose a REST API for the chat interface
spring-ai-starter-mcp-client - to use MCP
spring-ai-starter-vector-store-pgvector and spring-ai-advisors-vector-store - to enable RAG with PGVector
spring-ai-starter-model-ollama - to use Ollama models
spring-ai-starter-model-google-genai and spring-ai-starter-model-google-genai-embedding - to use Google Gemini models
spring-ai-starter-model-bedrock and spring-ai-starter-model-bedrock-converse - to use AWS Bedrock models

Components

MCP Tools
- Weather Tool - a local MCP tool that queries a WeatherService for the weather in a given city on a given date
- Booking Tool - a remote MCP tool that connects to the Booking MCP Server to reserve accommodations
Chat
- Chat Client - a Spring AI ChatClient configured with:
  - A system prompt to define the AI agent’s role and the current date
  - The AI model autoconfigured by Spring Boot via the active profile
  - The above MCP tools as part of the AI agent’s toolset
- Chat Service - wraps the Chat Client and adds three advisors:
  - QuestionAnswerAdvisor - fetches context from a vector store and augments the user input (RAG)
  - MessageChatMemoryAdvisor - adds conversation history to the user input (chat memory)
  - SimpleLoggerAdvisor - logs the chat history to the console (for debugging)
- Chat Controller - exposes a simple REST POST endpoint that takes user input, calls the Chat Service, and returns the AI agent’s response
Vector Store Initializer - loads some sample data into the vector store at startup

Let's implement this step by step ...

Weather Tool

Here's how the Weather Tool is implemented:

1) Create an instance and annotate it with @Tool and @ToolParam:

@Service // or @Bean / @Component
class WeatherTool(private val weatherService: WeatherService) {
  @Tool(description = "get the weather for a given city and date")
  fun getWeather(
    @ToolParam(description = "the city to get the weather for")
    city: String,
    @ToolParam(description = "the date to get the weather for")
    date: LocalDate
  ): String = weatherService.getWeather(city, date) // Delegate to a service
}

2) Register it as a MethodToolCallbackProvider:

@Configuration
class WeatherToolConfiguration {
  @Bean
  fun weatherToolCallbackProvider(weatherTool: WeatherTool) =
    MethodToolCallbackProvider.builder()
      .toolObjects(weatherTool)
      .build()
}

Booking Tool

To set up the Booking Tool as a remote MCP tool, we just need to configure the MCP client SSE connection in application.properties:

spring.ai.mcp.client.toolcallback.enabled=true
spring.ai.mcp.client.sse.connections.booking-tool.url=http://localhost:8081

You can find all the alternative configurations in MCP Client Boot Starter documentation.

Chat Client

We create the Chat Client using Spring AI's ChatClient.Builder, which is already autoconfigured via spring.ai configuration properties (we'll talk at that later in Configuration), and initialize it with a custom system prompt and the available MCP tools:

@Configuration
class ChatClientConfiguration {
  @Bean
  fun chatClient(
    builder: ChatClient.Builder,
    toolCallbackProviders: List<ToolCallbackProvider>
  ): ChatClient {
    return chatClientBuilder(builder, toolCallbackProviders).build()
  }

  private fun chatClientBuilder(
    builder: ChatClient.Builder,
    toolCallbackProviders: List<ToolCallbackProvider>
  ): ChatClient.Builder {
    val system = """
      You are an AI powered assistant to help people book accommodation in touristic cities around the world.
      If there is no information, then return a polite response suggesting you don't know.
      If the response involves a timestamp, be sure to convert it to something human-readable.
      Do not include any indication of what you're thinking.
      Use the tools available to you to answer the questions.
      Just give the answer.
      When booking accommodation for a weekend, assume check-in on Saturday and check-out on Monday.
      Current date: {currentDate}
      """.trimIndent()
    return builder
      .defaultSystem(system)
      .defaultToolCallbacks(*toolCallbackProviders.toTypedArray())
  }
}

Chat Service

The Chat Service exposes a single chat method that takes a chat ID and a user question. It calls the Chat Client with the user question along with a set of advisors to enrich the interaction:

QuestionAnswerAdvisor - retrieves relevant context from a vector store and injects it to the context (RAG)
MessageChatMemoryAdvisor - adds conversation history to the user input (chat memory) for the given chat ID
SimpleLoggerAdvisor - logs internal advisor traces to the console (if logging.level.org.springframework.ai.chat.client.advisor is set to DEBUG)

Additionally, the question and answer are logged to the console.

Here’s the implementation:

@Service // or @Bean / @Component
class ChatService(
  vectorStore: VectorStore,
  private val clock: Clock,
  private val chatClient: ChatClient
) {
  private val logger = LoggerFactory.getLogger(ChatService::class.java)
  private val questionAnswerAdvisor = QuestionAnswerAdvisor.builder(vectorStore).build()
  private val simpleLoggerAdvisor = SimpleLoggerAdvisor()
  private val chatMemoryAdvisor = MessageChatMemoryAdvisor.builder(
    MessageWindowChatMemory.builder()
      .chatMemoryRepository(InMemoryChatMemoryRepository())
      .build()
  ).build()

  fun chat(chatId: String, question: String): String {
    return chatClient
      .prompt()
      .system { it.param("currentDate", LocalDate.now(clock)) }
      .user(question)
      .advisors(questionAnswerAdvisor, chatMemoryAdvisor, simpleLoggerAdvisor)
      .advisors { a -> a.param(ChatMemory.CONVERSATION_ID, chatId) }
      .call()
      .content().apply {
        logger.info("Chat #$chatId question: $question")
        logger.info("Chat #$chatId answer: $this")
      }!!
  }
}

Chat Controller

The Chat Controller exposes a simple REST POST endpoint that takes user input, calls the Chat Service, and returns the AI agent’s response:

@RestController
class ChatController(private val chatService: ChatService) {
  @PostMapping("/{chatId}/chat")
  fun chat(
    @PathVariable chatId: String,
    @RequestParam question: String
  ): String? {
    return chatService.chat(chatId, question)
  }
}

Vector Store Initializer

It's as simple as using Spring AI’s autoconfigured VectorStore and adding documents to it. This automatically invokes the embedding model to generate embeddings and store them in the vector store:

@Bean
fun vectorStoreInitializer(vectorStore: VectorStore) =
  ApplicationRunner {
    // TODO check if the vector store is empty ...
    // TODO load cities from a JSON file or any other source ...
    cities.forEach { city ->
      val document = Document(
        "name: ${city.name} " +
        "country: ${city.country} " +
        "description: ${city.description}"
      )
      vectorStore.add(listOf(document))
    }
  }

You can find the full version of vectorStoreInitializer in ChatServerApplication.kt.

Configuration

We use application.properties instead of application.yml because YAML cannot have both spring.ai.model.embedding (scalar, used by Bedrock and Ollama) and spring.ai.model.embedding.text (nested, used by Gemini) at the same time. Maybe some day this will be solved, but anyway this is a PoC that supports multiple models — in a production application you'd likely use only one, so this wouldn't be an issue.

In this file, we define global configuration values:

Disable all model auto-configurations by default.
Configure the datasource to connect to a PostgreSQL database with PGVector support.
Set the server port to 8080.
Configure the URL of the remote Booking Tool MCP server.
Set the logging level for chat advisor debug traces.

spring.application.name=chat-server

spring.datasource.url=jdbc:postgresql://localhost:5432/postgres
spring.datasource.username=postgres
spring.datasource.password=password
spring.datasource.driver-class-name=org.postgresql.Driver

spring.ai.model.chat=none
spring.ai.model.embedding=none
spring.ai.model.embedding.text=none

spring.ai.mcp.client.toolcallback.enabled=true
spring.ai.mcp.client.sse.connections.booking-tool.url=http://localhost:8081

server.port=8080

logging.level.org.springframework.ai.chat.client.advisor=INFO

The AI model is configured via Spring profiles. Each profile sets the chat model, embedding model, and vector store dimensions. The active profile must be specified at runtime using SPRING_PROFILES_ACTIVE.

Ollama profile

In application-ollama.yml, we configure Spring AI to use Ollama models:

Set the base URL for the Ollama server to http://localhost:11434.
Set the chat model to llama3.1:8b (must be a tools-enabled model).
Set the embedding model to nomic-embed-text.
Use pull-model-strategy: when_missing to only pull models if they are not available locally.
Configure PGVector as the vector store with 768 dimensions (matching the embedding model size).

spring:
  ai:
    model:
      embedding: "ollama"
      chat: "ollama"
    ollama:
      base-url: "http://localhost:11434"
      init:
        pull-model-strategy: "when_missing"
      chat:
        options:
          model: "llama3.1:8b"
      embedding:
        options:
          model: "nomic-embed-text"
    vectorstore:
      pgvector:
        table-name: "vector_store_ollama"
        dimensions: 768
        initialize-schema: true

Gemini profile

In application-gemini.yml, we configure Spring AI to use Google Gemini models:

Set the Google API key from the GOOGLE_API_KEY environment variable.
Set the chat model to gemini-2.5-flash.
Set the embedding model to gemini-embedding-001 with 768 dimensions.
Configure PGVector as the vector store with 768 dimensions.

spring:
  ai:
    model:
      chat: "google-genai"
      embedding:
        text: "google-genai"
    google:
      genai:
        api-key: "${GOOGLE_API_KEY}"
        chat:
          options:
            model: "gemini-2.5-flash"
            temperature: 0.7
        embedding:
          api-key: "${GOOGLE_API_KEY}"
          text:
            options:
              model: "gemini-embedding-001"
              dimensions: 768
    vectorstore:
      pgvector:
        table-name: "vector_store_gemini"
        dimensions: 768
        initialize-schema: true

Bedrock profile

In application-bedrock.yml, we configure Spring AI to use AWS Bedrock models:

Set the AWS credentials and region from environment variables.
Set the chat model using Bedrock Converse API.
Set the embedding model using Bedrock Cohere.
Configure PGVector as the vector store with 1024 dimensions (matching the Cohere embedding model size).

spring:
  ai:
    model:
      embedding: "bedrock-cohere"
      chat: "bedrock-converse"
    bedrock:
      aws:
        access-key: "${AWS_ACCESS_KEY_ID}"
        secret-key: "${AWS_SECRET_ACCESS_KEY}"
        region: "${AWS_REGION:eu-central-1}"
      converse:
        chat:
          options:
            model: "${AWS_BEDROCK_CHAT_MODEL}"
            max-tokens: 2048
      cohere:
        embedding:
          model: "${AWS_BEDROCK_EMBEDDING_MODEL}"
    vectorstore:
      pgvector:
        table-name: "vector_store_bedrock"
        dimensions: 1024
        initialize-schema: true

Test

Test MCP Server

To test the MCP Server, we will use a McpClient to call the book method of the Booking Tool, mocking the downstream BookingService:

See the simplified test implementation below. For the complete implementation, including a test that verifies the list of available tools, refer to McpServerApplicationTest.kt.

@SpringBootTest(webEnvironment = RANDOM_PORT)
class McpServerApplicationTest {

  // 1. Inject the server port (it is random)
  @LocalServerPort
  val port: Int = 0

  // 2. Mock the BookingService instance
  @MockitoBean
  lateinit var bookingService: BookingService

  @Test
  fun `should book`() {
    // 3. Create a McpClient connected to the server
    val client = McpClient.sync(
      HttpClientSseClientTransport("http://localhost:$port")
    ).build()
    client.initialize()
    client.ping()

    // 4. Mock the bookingService using argument captors
    val bookResult = "Booking is done!"
    val cityCaptor = argumentCaptor<String>()
    val checkinDateCaptor = argumentCaptor<LocalDate>()
    val checkoutDateCaptor = argumentCaptor<LocalDate>()
    doReturn(bookResult)
      .whenever(bookingService)
      .book(
        cityCaptor.capture(),
        checkinDateCaptor.capture(),
        checkoutDateCaptor.capture()
      )

    // 5. Call the tool
    val city = "Barcelona"
    val checkinDate = LocalDate.parse("2025-04-15")
    val checkoutDate = LocalDate.parse("2025-04-18")
    val result = client.callTool(CallToolRequest(
      "book",
      mapOf(
        "city" to city,
        "checkinDate" to checkinDate.toEpochDay(),
        "checkoutDate" to checkoutDate.toEpochDay()
      )
    ))

    // 6. Verify the result
    assertThat(result.isError).isFalse()
    assertThat(result.content).singleElement()
      .isInstanceOfSatisfying(TextContent::class.java) {
        // TODO why is text double quoted?
        assertThat(it.text).isEqualTo("\"$bookResult\"")
      }

    // 7. Verify that the bookingService was called with
    // the correct parameters
    assertThat(cityCaptor.allValues).singleElement()
      .isEqualTo(city)
    assertThat(checkinDateCaptor.allValues).singleElement()
      .isEqualTo(checkinDate)
    assertThat(checkoutDateCaptor.allValues).singleElement()
      .isEqualTo(checkoutDate)

    // 8. Close the client
    client.close()
  }
}

Test Chat Server

To test the Chat Server, we will:

Override the default profile to gemini and disable the MCP client in application-test.yml (overridable via SPRING_PROFILES_ACTIVE).
Replace the remote Booking Tool by a local Booking Test Tool with the same signature.
- Create the local Book Testing Tool in BookingTestToolConfiguration.kt.
Mock the downstream services Weather Service and Booking Service with MockitoBean.
Create a fixed Clock to control the current date in ClockTestToolConfiguration.kt
- Declare it as @Primary @Bean to override the default Clock bean.
Start Docker Compose with PGVector using Testcontainers

You might’ve noticed that the test doesn’t actually check the MCP client’s SSE connection as it is disabled. I tried spinning up an McpServer using McpServerAutoConfiguration, and it almost worked. The problem? The client tries to connect before the server is up, which causes the whole application to fail on startup. Maybe is just an ordering issue, and hopefully something that can be fixed in the future 🤞

Now for the interesting part, how do we test the AI agent’s response? This is where Evaluation Testing comes in:

One method to evaluate the response is to use the AI model itself for evaluation. Select the best AI model for the evaluation, which may not be the same model used to generate the response.

This aligns with the evaluation techniques described in Martin Fowler’s Evals GenAI pattern:

Self-evaluation: The LLM evaluates its own response, but this can reinforce its own mistakes or biases.
LLM as a judge: Another model scores the output, reducing bias by introducing a second opinion.
Human evaluation: People manually review responses to ensure the tone and intent feel right.

To keep things simple, we’ll go with self-evaluation 🤓

Each test will follow this structure:

@Test
fun `should do something`() {
  // 1. Mock downstream service(s)
  // Optionally use argument captors depending on how you plan
  // to verify parameters in step 5
  // Example for BookingService:
  doReturn("Your booking is done!")
    .whenever(bookingTestService).book(any(), any(), any())

  // 2. Call the chat service
  val chatId = UUID.randomUUID().toString()
  val chatResponse = chatService.chat(
    chatId,
    "Can you book accommodation for Barcelona from 2025-04-15 to 2025-04-18?"
  )

  // 3. Evaluate the response using the AI model
  val evaluationResult = TestEvaluator(chatClientBuilder) { evaluationRequest, userSpec ->
    userSpec.text(
      """
      Your task is to evaluate if the answer given by an AI agent to a human user matches the claim.
      Return YES if the answer matches the claim and NO if it does not.
      After returning YES or NO, explain why.
      Assume that today is ${LocalDate.now(clock)}.
      Answer: {answer}
      Claim: {claim}
      """.trimIndent()
    )
      .param("answer", evaluationRequest.responseContent)
      .param("claim", evaluationRequest.userText)
  }.evaluate(EvaluationRequest(
    "Accommodation has been booked Barcelona from 2025-04-15 to 2025-04-18",
    chatResponse
  ))

  // 4. Assert the evaluation result is successful, show feedback if not
  assertThat(evaluationResult.isPass).isTrue
    .withFailMessage { evaluationResult.feedback }

  // 5. If applicable, verify the parameters passed to the service
  // You can verify with argument captors or use the `verify` method as in the example below:
  verify(bookingTestService).book(
    eq("Barcelona"),
    eq(LocalDate.parse("2025-04-15")),
    eq(LocalDate.parse("2025-04-18"))
  )
}

See the full test implementation in ChatServerApplicationTest.kt.

Each evaluation uses a custom prompt tailored to the specific response being tested, and as you experiment, you'll notice some surprisingly quirky behavior. That’s why I ended up creating a custom TestEvaluator, based on Spring AI’s RelevancyEvaluator and FactCheckingEvaluator, which may not yet offer the level of customization you might want.

I had to adjust each prompt after running into odd results. For example, the evaluation model assuming it was still 2023 and refusing to believe the AI agent could predict weather for 2025. Or it mistaking "you" in the answer as referring to itself instead of the user. The weirdest? One evaluation just answered “NO” but the explanation said, “well, maybe I should have said YES” 🤣

For a production system, you'd definitely need a lot of prompt tuning and testing to get things right, for both the system and the evaluator, I suppose that’s part of the "fun" when working with GenAI.

By default, tests use the Gemini profile both locally and in CI. You can also test with Ollama (results may be flaky without adequate GPU hardware) or Bedrock.

Run

You can configure the application using environment variables or a system.properties file in the root directory. This file is ignored by Git and is loaded by both ./gradlew bootRun and tests.

Example system.properties:

# AWS Bedrock
AWS_ACCESS_KEY_ID=...
AWS_SECRET_ACCESS_KEY=...
AWS_REGION=eu-central-1
AWS_BEDROCK_CHAT_MODEL=...
AWS_BEDROCK_EMBEDDING_MODEL=...

# Google Gemini
GOOGLE_API_KEY=...

Running the vector database

All profiles require PGVector for the RAG vector database:

cd chat-server
docker compose -f docker-compose-vectordb.yml up -d

To stop it:

cd chat-server
docker compose -f docker-compose-vectordb.yml down

To stop it and remove all volumes (removes all vector database data):

cd chat-server
docker compose -f docker-compose-vectordb.yml down -v

Running Ollama locally

If you want to use local LLMs, you can run Ollama either via Docker Compose or as a native application (more info at ollama.com).

cd chat-server
docker compose -f docker-compose-ollama.yml up -d

Running the application

1) Start MCP server

cd mcp-server
./gradlew bootRun

2) Start Chat Server with one of the following profiles:

Ollama (requires Ollama and vector database):

cd chat-server
SPRING_PROFILES_ACTIVE=ollama ./gradlew bootRun

Gemini (requires vector database and GOOGLE_API_KEY):

cd chat-server
SPRING_PROFILES_ACTIVE=gemini ./gradlew bootRun

Bedrock (requires vector database and AWS credentials):

cd chat-server
SPRING_PROFILES_ACTIVE=bedrock ./gradlew bootRun

3) Start chatting

A simple chat UI is available at http://localhost:8080/chat.html (disclaimer: this UI was entirely AI-generated as frontend is not the goal of this PoC).

You can also explore the API endpoints at http://localhost:8080/swagger-ui.html.

How to use other AI models

To use any of the other AI models supported by Spring AI, follow these steps:

Add the required dependencies
Configure the model in its own application-<model>.yml file
Activate the profile using SPRING_PROFILES_ACTIVE=<model>

Documentation

Spring AI documentation
Martin Fowler's GenAI patterns
Inspired by sample project spring-ai-java-bedrock-mcp-rag
Awesome Spring AI

Happy GenAI coding! 💙

Tests Everywhere - Swift

Roger Viñas Alcon — Mon, 18 Dec 2023 10:32:24 +0000

rogervinas / tests-everywhere

🤠 Tests, Tests Everywhere!

Swift testing this simple Hello World with XCTest and Cuckoo

Show me the code
- Implementation
- Test
Run this project using 🐳 docker
Run this project locally

Show me the code

Implementation

Sorry but reading that constant and variable names can contain almost any character, including Unicode characters I could not resist 😅 ...

1) Create HelloMessage class in HelloMessage.swift:

class HelloMessage {
  var 👋: String {
    get {
      return "Hello World!"
    }
  }
}

2) Same way create HelloConsole class in HelloConsole.swift:

class HelloConsole {
  func print(👋: String) {
    Swift.print(👋)
  }
}

3) Create HelloApp class in HelloApp.swift:

class HelloApp {

  let message: HelloMessage
  let console: HelloConsole

  init(message: HelloMessage, console: HelloConsole) {
    self.message = message
    self.console = console
  }

  func printHello() {
    console.print(👋: message.👋)
  }
}

4) Create main class in Main.swift that wraps it all together:

@main
public struct Main {
  public static func main() {
    let message = HelloMessage()
    let console = HelloConsole()
    let app = HelloApp(message: message, console: console)
    app.printHello()
  }
}

Test

Following XCTest and Cuckoo guides ...

1) Mock generation

Mocks have to be generated statically, you can follow these instructions or alternatively:

Build generator (only once, XCode command line tools required):

git clone git@github.com:Brightify/Cuckoo.git /tmp/Cuckoo
/tmp/Cuckoo/build_generator

Generate mocks (as many times as needed):

/tmp/Cuckoo/Generator/bin/cuckoo_generator generate \
  --testable Hello \
  --output Tests/Mocks.swift \
  Sources/HelloMessage.swift \
  Sources/HelloConsole.swift

You can also integrate mocks generation in the build, before compile phase.

2) Test HelloMessage in HelloMessageTest.swift:

final class HelloMessageTest: XCTestCase {
  func testHelloMessageShouldReturnHelloWorld() throws {
    let message = HelloMessage()
    XCTAssertEqual(message.👋, "Hello World!")
  }
}

3) Test HelloApp in HelloAppTest.swift:

final class HelloAppTest: XCTestCase {
  func testHelloAppShouldDisplayHelloMessage() throws {
    let messageText = "Hello Test!"
    // 2.1 Create a mock of HelloMessage
    let message = MockHelloMessage()
    // 2.2 Return "Hello Test!" whenever 👋 getter is called
    stub(message) { stub in
      when(stub.👋.get).thenReturn(messageText)
    }

    // 2.2 Create a mock of HelloConsole
    let console = MockHelloConsole()
    // 2.3 Do nothing whenever print is called
    stub(console) { stub in
      when(stub.print(👋: anyString())).thenDoNothing()
    }

    // 2.4 Create a HelloApp, the one we want to test, passing the mocks
    let app = HelloApp(message: message, console: console)
    // 2.5 Execute the method we want to test
    app.printHello()

    // 2.6 Verify HelloConsole mock print() method
    // has been called once with "Hello Test!"
    verify(console).print(👋: messageText)
  }
}

4) Test output should look like:

Test Suite 'All tests' started at 2023-12-12 16:48:21.225
Test Suite 'debug.xctest' started at 2023-12-12 16:48:21.227
Test Suite 'HelloAppTest' started at 2023-12-12 16:48:21.227
Test Case 'HelloAppTest.testHelloAppShouldDisplayHelloMessage' started at 2023-12-12 16:48:21.227
Test Case 'HelloAppTest.testHelloAppShouldDisplayHelloMessage' passed (0.003 seconds)
Test Suite 'HelloAppTest' passed at 2023-12-12 16:48:21.230
    Executed 1 test, with 0 failures (0 unexpected) in 0.003 (0.003) seconds
Test Suite 'HelloMessageTest' started at 2023-12-12 16:48:21.230
Test Case 'HelloMessageTest.testHelloMessageShouldReturnHelloWorld' started at 2023-12-12 16:48:21.231
Test Case 'HelloMessageTest.testHelloMessageShouldReturnHelloWorld' passed (0.001 seconds)
Test Suite 'HelloMessageTest' passed at 2023-12-12 16:48:21.232
    Executed 1 test, with 0 failures (0 unexpected) in 0.001 (0.001) seconds
Test Suite 'debug.xctest' passed at 2023-12-12 16:48:21.232
    Executed 2 tests, with 0 failures (0 unexpected) in 0.004 (0.004) seconds
Test Suite 'All tests' passed at 2023-12-12 16:48:21.232
    Executed 2 tests, with 0 failures (0 unexpected) in 0.004 (0.004) seconds

Happy Testing! 💙

Tests Everywhere - COBOL

Roger Viñas Alcon — Mon, 11 Dec 2023 09:45:30 +0000

rogervinas / tests-everywhere

🤠 Tests, Tests Everywhere!

COBOL testing this simple Hello World with cobol-check

Show me the code
- Implementation
- Test
Run this project using 🐳 docker
Run this project locally

Show me the code

Implementation

1) Create a program HELLO-CONSOLE in HELLO-CONSOLE.CBL:

Declare an alpha-numeric parameter MSG
Program will just display MSG

IDENTIFICATION DIVISION.
PROGRAM-ID. HELLO-CONSOLE.

DATA DIVISION.
LINKAGE SECTION.
01 MSG PIC A(20).

PROCEDURE DIVISION USING MSG.
    DISPLAY MSG.

2) Create a program HELLO in HELLO.CBL:

Declare an alpha-numeric variable MSG
Create HELLO-MESSAGE paragraph that sets a value to MSG variable
Create HELLO paragraph calling:
- HELLO-MESSAGE paragraph
- HELLO-CONSOLE program
Create MAIN paragraph just calling the HELLO paragraph

💡 We use a paragraph for HELLO-MESSAGE and a program for HELLO-CONSOLE just to show two different options and how to mock them in the tests.

IDENTIFICATION DIVISION.
PROGRAM-ID. HELLO.

DATA DIVISION.
WORKING-STORAGE SECTION.
01 MSG PIC A(20).

PROCEDURE DIVISION.

MAIN.
    PERFORM HELLO.
    STOP RUN.

HELLO.
    PERFORM HELLO-MESSAGE.
    CALL 'HELLO-CONSOLE' USING MSG.

HELLO-MESSAGE.
    MOVE 'HELLO WORLD!' TO MSG.

Test

Following A Brief Example, Mock a paragraph and Mock a CALL statement guides ...

In HELLO.cut file:

1) Test HELLO-MESSAGE paragraph:

TestSuite "HELLO-MESSAGE"

TestCase "SHOULD RETURN HELLO WORLD"
  MOVE "" TO MSG
  PERFORM HELLO-MESSAGE
  Expect MSG = "HELLO WORLD!"

2) Test HELLO paragraph mocking HELLO-MESSAGE and HELLO-CONSOLE:

TestSuite "HELLO"

TestCase "SHOULD DISPLAY HELLO MESSAGE"
  * 2.1 Mock HELLO-MESSAGE
  MOCK PARAGRAPH HELLO-MESSAGE
      MOVE "HELLO TEST!" TO MSG
  END-MOCK

  * 2.2 Mock HELLO-CONSOLE
  MOCK CALL 'HELLO-CONSOLE' USING MSG
      CONTINUE
  END-MOCK

  * 2.3 Execute the paragraph we want to test
  PERFORM HELLO

  * 2.4 Verify MSG has the expected value
  Expect MSG = "HELLO TEST!"
  * 2.5 Verify HELLO-MESSAGE has been called once
  VERIFY PARAGRAPH HELLO-MESSAGE HAPPENED ONCE
  * 2.6 Verify HELLO-CONSOLE has been called once
  VERIFY CALL 'HELLO-CONSOLE' USING MSG HAPPENED ONCE

3) Test output should look like:

TESTSUITE:
HELLO-MESSAGE
     PASS:   1. SHOULD RETURN HELLO WORLD
TESTSUITE:
HELLO
     PASS:   2. SHOULD DISPLAY HELLO MESSAGE
     PASS:   3. VERIFY EXACT 1 ACCESS TO PARAGRAPH HELLO-MESSAGE
     PASS:   4. VERIFY EXACT 1 ACCESS TO CALL 'HELLO-CONSOLE'

  4 TEST CASES WERE EXECUTED
  4 PASSED
  0 FAILED
  0 CALLS NOT MOCKED
================================================

I have no idea how to unit test HELLO-CONSOLE program as it seems cobol-check does not support PROCEDURE DIVISION USING X sections so I created this issue 🤞

Happy Testing! 💙

Tests Everywhere - Scala

Roger Viñas Alcon — Thu, 23 Nov 2023 00:25:51 +0000

rogervinas / tests-everywhere

🤠 Tests, Tests Everywhere!

Scala testing this simple Hello World with ScalaTest and Mockito

Show me the code
- Implementation
- Test
Run this project using 🐳 docker
Run this project locally

Show me the code

Implementation

Create HelloMessage class in HelloMessage.scala:

class HelloMessage:
  def text: String = "Hello World!"

We can create it as a class as Mockito is able to mock final classes.

Same way create HelloConsole class in HelloConsole.scala:

class HelloConsole:
  def print(text: String): Unit =
    println(text)

Create HelloApp class in HelloApp.scala:

class HelloApp(message: HelloMessage, console: HelloConsole):
  def printHello(): Unit =
    console.print(message.text)

Create main class in Main.scala that wraps it all together:

@main def main(): Unit =
  val message = HelloMessage()
  val console = HelloConsole()
  val app = HelloApp(message, console)
  app.printHello()

Test

Following ScalaTest > Writing your first test and ScalaTest + Mockito guides ...

Test HelloMessage in HelloMessageSpec.scala:

class HelloMessageSpec extends AnyFlatSpec {

  "A HelloMessage" should "return hello world" in {
    val message = HelloMessage()
    assertResult(message.text) {
      "Hello World!"
    }
  }
}

Test HelloApp in HelloAppSpec.scala:

class HelloAppSpec extends AnyFlatSpec {

  "A HelloApp" should "print hello message" in {
    val messageText = "Hello Test!"

    // 2.1 Create a mock of HelloMessage
    val message = mock[HelloMessage]
    // 2.2 Return "Hello Test!" whenever text is called
    when(message.text).thenReturn(messageText)

    // 2.3 Create a mock of HelloConsole
    val console = mock[HelloConsole]

    // 2.4 Create a HelloApp, the one we want to test, passing the mocks
    val app = HelloApp(message, console)
    // 2.5 Execute the method we want to test
    app.printHello()

    // 2.6 Verify HelloConsole mock print() method
    // has been called once with "Hello Test!"
    verify(console).print(messageText)
  }
}

Test output should look like:

[info] HelloMessageSpec:
[info] A HelloMessage
[info] - should return hello world
[info] HelloAppSpec:
[info] A HelloApp
[info] - should print hello message
[info] Run completed in 515 milliseconds.
[info] Total number of tests run: 2
[info] Suites: completed 2, aborted 0
[info] Tests: succeeded 2, failed 0, canceled 0, ignored 0, pending 0
[info] All tests passed.
[success] Total time: 6 s

Happy Testing! 💙

Tests Everywhere - Rust

Roger Viñas Alcon — Thu, 23 Nov 2023 00:23:01 +0000

rogervinas / tests-everywhere

🤠 Tests, Tests Everywhere!

Rust testing this simple Hello World with mockall

Show me the code
- Implementation
- Test
Run this project using 🐳 docker
Run this project locally

Show me the code

Implementation

Create HelloMessage trait and HelloWorldMessage implementation in hello_message.rs:

pub trait HelloMessage {
  fn text(&self) -> String;
}

pub struct HelloWorldMessage {}

impl HelloMessage for HelloWorldMessage {
  fn text(&self) -> String {
    return String::from("Hello World!");
  }
}

impl HelloWorldMessage {
  pub fn new() -> Self {
    Self {}
  }
}

Creating it as a trait will allow us to mock it for testing.

Note that we create a HelloWorldMessage::new() method just as a convention to create new instances, although we could just use HelloWorldMessage {}.

Same way create HelloConsole trait and HelloSystemConsole implementation in hello_console.rs:

pub trait HelloConsole {
  fn print(&self, text: String);
}

pub struct HelloSystemConsole {}

impl HelloConsole for HelloSystemConsole {
  fn print(&self, text: String) {
    println!("{}", text);
  }
}

impl HelloSystemConsole {
  pub fn new() -> Self {
    Self {}
  }
}

Create HelloApp class in hello_app.rs:

pub struct HelloApp {
  message: Box<dyn HelloMessage>,
  console: Box<dyn HelloConsole>
}

impl HelloApp {
  pub fn new(message: Box<dyn HelloMessage>, console: Box<dyn HelloConsole>) -> Self {
    Self { message, console }
  }

  pub fn print_hello(&self) {
    self.console.print(self.message.text());
  }
}

Note that we pass constructor arguments as Box<dyn T> to avoid doesn't have a size known at compile-time compilation error.

Create main function in main.rs that wraps it all together:

fn main() {
  let message = HelloWorldMessage::new();
  let console = HelloSystemConsole::new();
  let app = HelloApp::new(Box::new(message), Box::new(console));
  app.print_hello();
}

Test

Following Rust By Example > Unit testing and mockall > Getting Started guides ...

Test HelloMessage in hello_message.rs:

#[cfg(test)]
mod tests {
  use super::*;

  #[test]
  fn should_return_hello_world() {
    let message = HelloWorldMessage::new();
    assert_eq!(message.text(), String::from("Hello World!"))
  }
}

Setup HelloMessage mocking needed by HelloApp test in hello_message.rs:

// 2.1 Enable automock
#[cfg(test)]
use mockall::automock;

// 2.2 Enable automock for HelloMessage
#[cfg_attr(test, automock)]
pub trait HelloMessage {
  fn text(&self) -> String;
}

Setup HelloConsole mocking needed by HelloApp test in hello_console.rs:

// 3.1 Enable automock
#[cfg(test)]
use mockall::automock;

// 3.2 Enable automock for HelloConsole
#[cfg_attr(test, automock)]
pub trait HelloConsole {
  fn print(&self, text: String);
}

Test HelloApp in hello_app.rs:

#[cfg(test)]
mod tests {
  use super::*;
  use mockall::predicate::*;
  // 4.1 Use MockHelloMessage, enabled by 2.2
  use crate::hello_message::MockHelloMessage;
  // 4.2 Use MockHelloConsole, enabled by 3.2
  use crate::hello_console::MockHelloConsole;

  #[test]
  fn should_print_hello_message() {
    let message_text = "Hello Test!";

    // 4.3 Create a mock of HelloMessage
    let mut message = MockHelloMessage::new();
    // 4.4 Return "Hello Test!" whenever text is called
    message.expect_text().return_const(String::from(message_text));

    // 4.5 Create a mock of HelloConsole
    let mut console = MockHelloConsole::new();
    // 4.6 Expect print to be called once with "Hello Test!"
    console.expect_print()
      .with(eq(String::from(message_text)))
      .times(1)
      .return_const(());

    // 4.7 Create a HelloApp, the one we want to test, passing the mocks
    let app = HelloApp::new(Box::new(message), Box::new(console));
    // 4.8 Execute the method we want to test
    app.print_hello();

    // Expectation 4.6 will be checked once test ends
  }
}

Test output should look like:

running 2 tests
test hello_app::tests::should_print_hello_message ... ok
test hello_message::tests::should_return_hello_world ... ok

test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

Happy Testing! 💙

Tests Everywhere - Ruby

Roger Viñas Alcon — Thu, 23 Nov 2023 00:19:46 +0000

rogervinas / tests-everywhere

🤠 Tests, Tests Everywhere!

Ruby testing this simple Hello World with RSpec

Show me the code
- Implementation
- Test
Run this project using 🐳 docker
Run this project locally

Show me the code

Implementation

Create HelloMessage class in HelloMessage.rb:

class HelloMessage
  def text
    @text = "Hello World!"
  end
end

Create HelloConsole class in HelloConsole.rb:

class HelloConsole
  def print(text)
    puts text
  end
end

Create HelloApp class in HelloApp.rb:

class HelloApp
  def initialize(message, console)
    @message = message
    @console = console
  end

  def printHello
    @console.print(@message.text)
  end
end

Create main function in Main.rb that wraps it all together:

message = HelloMessage.new
console = HelloConsole.new
app = HelloApp.new message, console
app.printHello

Test

Following RSpec Core and RSpec Mocks guides ...

Test HelloMessage in HelloMessage_spec.rb:

RSpec.describe HelloMessage do
  it "should return hello world" do
    message = HelloMessage.new
    expect(message.text).to eq("Hello World!")
  end
end

Test HelloApp in HelloApp_spec.rb:

RSpec.describe HelloApp do
  it "should print hello message" do
    messageText = "Hello Test!"

    # 2.1 Create a mock of HelloMessage
    # that will return "Hello Test!" whenever text is called
    message = instance_double("HelloMessage", :text => messageText)

    # 2.2 Create a mock of HelloConsole
    console = instance_double("HelloConsole")
    # 2.3 Expect print to be called once with "Hello Test!"
    expect(console).to receive(:print).once.with(messageText)

    # 2.4 Create a HelloApp, the one we want to test, passing the mocks
    app = HelloApp.new message, console
    # 2.5 Execute the method we want to test
    app.printHello

    # Expectation 2.3 will be checked once test ends
  end
end

Test output should look like:

HelloApp
  should print hello message

HelloMessage
  should return hello world

Finished in 0.00457 seconds (files took 0.05704 seconds to load)
2 examples, 0 failures

Happy Testing! 💙

Tests Everywhere - Python

Roger Viñas Alcon — Thu, 23 Nov 2023 00:15:04 +0000

rogervinas / tests-everywhere

🤠 Tests, Tests Everywhere!

Python testing this simple Hello World with unittest

Show me the code
- Implementation
- Test
Run this project using 🐳 docker
Run this project locally

Show me the code

Implementation

Create HelloMessage class in hello_message.py:

class HelloMessage:
  def __init__(self):
    self.text = "Hello World!"

Create HelloConsole class in hello_console.py:

from builtins import print as __print__

class HelloConsole:
  def print(self, text):
    __print__(text)

Note that we have to import print system function as __print__ to avoid conflict with HelloConsole own print method.

Create HelloApp class in hello_app.py:

class HelloApp:
  def __init__(self, message, console):
    self.message = message
    self.console = console

  def print_hello(self):
    self.console.print(self.message.text)

Create main function in __main__.py that wraps it all together:

message = HelloMessage()
console = HelloConsole()
app = HelloApp(message, console)
app.print_hello()

Test

Following unittest > Basic example and unittest.mock > Quick Guide guides ...

Test HelloMessage in test_hello_message.py:

class HelloMessageTest(unittest.TestCase):
  def test_should_return_hello_world(self):
    message = HelloMessage()
    self.assertEqual(message.text, "Hello World!")

Test HelloApp in test_hello_app.py:

class HelloAppTest(unittest.TestCase):
  def test_should_print_hello_message(self):
    message_text = "Hello Test!"

    # 2.1 Create a mock of HelloMessage
    message = MagicMock()
    # 2.2 Return "Hello Test!" whenever text is called
    message.text = message_text

    # 2.3 Create a mock of HelloConsole
    console = MagicMock()
    # 2.4 Mock print method
    console.print = MagicMock()

    # 2.5 Create a HelloApp, the one we want to test, passing the mocks
    app = HelloApp(message, console)
    # 2.6 Execute the method we want to test
    app.print_hello()

    # 2.7 Verify HelloConsole mock print() method
    # has been called once with "Hello Test!"
    console.print.assert_called_once_with(message_text)

Test output should look like:

test_should_print_hello_message (test.test_hello_app.HelloAppTest.test_should_print_hello_message) ... ok
test_should_return_hello_world (test.test_hello_message.HelloMessageTest.test_should_return_hello_world) ... ok

----------------------------------------------------------------------
Ran 2 tests in 0.001s

OK

Happy Testing! 💙

Tests Everywhere - Kotlin

Roger Viñas Alcon — Thu, 23 Nov 2023 00:12:28 +0000

rogervinas / tests-everywhere

🤠 Tests, Tests Everywhere!

Kotlin testing this simple Hello World with Kotest and MockK

Show me the code
- Implementation
- Test
Run this project using 🐳 docker
Run this project locally

Show me the code

Implementation

Create HelloMessage class in HelloMessage.kt:

class HelloMessage {
  val text: String = "Hello World!"
}

We can create it as a class as MockK is able to mock final classes.

Same way create HelloConsole class in HelloConsole.kt:

class HelloConsole {
  fun print(text: String) {
    println(text)
  }
}

Create HelloApp class in HelloApp.kt:

class HelloApp(private val message: HelloMessage, private val console: HelloConsole) {
  fun printHello() {
    console.print(message.text)
  }
}

Create main class in Main.kt that wraps it all together:

fun main() {
  val message = HelloMessage()
  val console = HelloConsole()
  val app = HelloApp(message, console)
  app.printHello()
}

Test

Following Kotest > Writing Tests and MockK > Class mock guides ...

Test HelloMessage in HelloMessageTest.kt:

class HelloMessageTest : DescribeSpec({

  describe("HelloMessage") {
    it("should return hello world") {
      val message = HelloMessage()
      message.text shouldBe "Hello World!"
    }
  }
})

Test HelloApp in HelloAppTest.kt:

class HelloAppTest : DescribeSpec({

  describe("HelloApp") {
    it("should print hello message") {
      val messageText = "Hello Test!"

      // 2.1 Create a mock of HelloMessage
      val message = mockk<HelloMessage>()
      // 2.2 Return "Hello Test!" whenever text is called
      every { message.text } returns (messageText)

      // 2.3 Create a mock of HelloConsole
      val console = mockk<HelloConsole>()
      // 2.4 Do nothing whenever print is called
      every { console.print(any()) } just Runs

      // 2.5 Create a HelloApp, the one we want to test, passing the mocks
      val app = HelloApp(message, console)
      // 2.6 Execute the method we want to test
      app.printHello()

      // 2.7 Verify HelloConsole mock print() method
      // has been called once with "Hello Test!"
      verify { console.print(messageText) }
    }
  }
})

Test output should look like:

> Task :test

org.hello.HelloAppTest > HelloApp > should print hello message PASSED

org.hello.HelloMessageTest > HelloMessage > should return hello world PASSED

BUILD SUCCESSFUL in 3s

Happy Testing! 💙

Tests Everywhere - JavaScript

Roger Viñas Alcon — Thu, 23 Nov 2023 00:09:06 +0000

rogervinas / tests-everywhere

🤠 Tests, Tests Everywhere!

JavaScript testing this simple Hello World with Jest

Show me the code
- Implementation
- Test
Run this project using 🐳 docker
Run this project locally

Show me the code

Implementation

Create HelloMessage class in HelloMessage.js:

class HelloMessage {
  text = "Hello World!";
}

Create HelloConsole class in HelloConsole.js:

class HelloConsole {
  print(text) {
    console.log(text);
  }
}

Create HelloApp class in HelloApp.js:

class HelloApp {
  constructor(helloMessage, helloConsole) {
    this.helloMessage = helloMessage;
    this.helloConsole = helloConsole;
  }

  printHello() {
    this.helloConsole.print(this.helloMessage.text);
  }
}

Create main function in Main.js that wraps it all together:

const HelloMessage = require("./HelloMessage");
const HelloConsole = require("./HelloConsole");
const HelloApp = require("./HelloApp");

const message = new HelloMessage();
const console = new HelloConsole();
const app = new HelloApp(message, console);
app.printHello();

Test

Following Jest > Getting Started guide ...

Test HelloMessage in HelloMessage.test.js:

describe("HelloMessage", () => {
  it("should return hello world", () => {
    const message = new HelloMessage();
    expect(message.text).toBe("Hello World!");
  });
});

Test HelloApp in HelloApp.test.js:

// 2.1 Mock ./HelloMessage module
jest.mock("./HelloMessage");
// 2.2 Mock ./HelloConsole module
jest.mock("./HelloConsole");

describe("HelloApp", () => {
  it("should print hello message", () => {
    const messageText = "Hello Test!";

    // 2.3 Set mock implementation to be returned
    // anytime new HelloMessage() is called
    HelloMessage.mockImplementation(() => {
      return {
        text: messageText,
      };
    });
    // 2.4 Create a HelloMessage
    // that will be a mock thanks to 2.1 and 2.2
    const message = new HelloMessage();

    // 2.5 Create a HelloConsole
    // that will be a mock thanks to 2.3
    const console = new HelloConsole();

    // 2.6 Create a HelloApp, the one we want to test, passing the mocks
    const app = new HelloApp(message, console);
    // 2.7 Execute the method we want to test
    app.printHello();

    // 2.8 Verify HelloConsole mock print() method
    // has been called once with "Hello Test!"
    expect(console.print).toHaveBeenCalledTimes(1);
    expect(console.print).toHaveBeenCalledWith(messageText);
  });
});

Test output should look like:

> jest

 PASS  ./HelloMessage.test.js
 PASS  ./HelloApp.test.js

Test Suites: 2 passed, 2 total
Tests:       2 passed, 2 total
Snapshots:   0 total
Time:        1.229 s
Ran all test suites.

Happy Testing! 💙

Tests Everywhere - Java

Roger Viñas Alcon — Thu, 23 Nov 2023 00:00:13 +0000

rogervinas / tests-everywhere

🤠 Tests, Tests Everywhere!

Java testing this simple Hello World with JUnit5, Mockito and AssertJ

Show me the code
- Implementation
- Test
Run this project using 🐳 docker
Run this project locally

Show me the code

Implementation

Create HelloMessage class in HelloMessage.java:

public class HelloMessage {
  public String getText() {
    return "Hello World!";
  }
}

We can create it as a class as Mockito is able to mock final classes.

Same way create HelloConsole class in HelloConsole.java:

public class HelloConsole {
  public void print(String text) {
    System.out.println(text);
  }
}

Create HelloApp class in HelloApp.java:

public class HelloApp {

  private final HelloMessage message;
  private final HelloConsole console;

  public HelloApp(HelloMessage message, HelloConsole console) {
    this.message = message;
    this.console = console;
  }

  public void printHello() {
    console.print(message.getText());
  }
}

Create main class in Main.java that wraps it all together:

public class Main {

  public static void main(String[] args) {
    var message = new HelloMessage();
    var console = new HelloConsole();
    var app = new HelloApp(message, console);
    app.printHello();
  }
}

Test

Following JUnit5 > Writing Tests, Mockito > Stubbing and AssertJ > Core assertions guide guides ...

Test HelloMessage in HelloMessageTest.java:

@Test
void shouldReturnHelloWorld() {
  var message = new HelloMessage();
  assertThat(message.getText()).isEqualTo("Hello World!");
}

Test HelloApp in HelloAppTest.java:

@Test
void shouldPrintHelloMessage() {

  var messageText = "Hello Test!";

  // 2.1 Create a mock of HelloMessage
  var message = mock(HelloMessage.class);
  // 2.2 Return "Hello Test!" whenever getText() is called
  when(message.getText()).thenReturn(messageText);

  // 2.2 Create a mock of HelloConsole
  var console = mock(HelloConsole.class);

  // 2.3 Create a HelloApp, the one we want to test, passing the mocks
  var app = new HelloApp(message, console);
  // 2.4 Execute the method we want to test
  app.printHello();

  // 2.5 Verify HelloConsole mock print() method
  // has been called once with "Hello Test!"
  verify(console).print(messageText);
}

Test output should look like:

> Task :test

HelloAppTest > shouldPrintHelloMessage() PASSED

HelloMessageTest > shouldReturnHelloWorld() PASSED

BUILD SUCCESSFUL in 2s

Happy Testing! 💙

Tests Everywhere - Go

Roger Viñas Alcon — Wed, 22 Nov 2023 23:56:58 +0000

rogervinas / tests-everywhere

🤠 Tests, Tests Everywhere!

Go testing this simple Hello World with Standard library

Show me the code
- Implementation
- Test
Run this project using 🐳 docker
Run this project locally

Show me the code

Implementation

Create HelloMessage interface and HelloWorldMessage implementation in HelloMessage.go:

type HelloMessage interface {
  Text() string
}

type HelloWorldMessage struct{}

func (message *HelloWorldMessage) Text() string {
  return "Hello World!"
}

Creating it as an interface will allow us to mock it for testing.

Same way create HelloConsole interface and HelloSystemConsole implementation in HelloConsole.go:

type HelloConsole interface {
  Print(text string)
}

type HelloSystemConsole struct{}

func (console *HelloSystemConsole) Print(text string) {
  fmt.Println(text)
}

Create HelloApp class in HelloApp.go:

type HelloApp struct {
  message HelloMessage
  console HelloConsole
}

func (app *HelloApp) PrintHello() {
  app.console.Print(app.message.Text())
}

Create main function in Main.go that wraps it all together:

func main() {
  message := HelloWorldMessage{}
  console := HelloSystemConsole{}
  app := HelloApp{&message, &console}
  app.PrintHello()
}

Test

Following Standard library > testing guide ...

Test HelloMessage in HelloMessage_test.go:

func TestShouldReturnHelloWorld(t *testing.T) {
  messageText := "Hello World!"
  message := HelloWorldMessage{}
  if message.Text() != messageText {
    t.Fatalf("Expected %s but got %s", messageText, message.Text())
  }
}

Test HelloApp in HelloApp_test.go:

// 2.1 Define a HelloMessageMock struct that ...
type HelloMessageMock struct {
  text string
}

// ... fullfils HelloMessage interface
func (message *HelloMessageMock) Text() string {
  return message.text
}

// 2.2 Define a HelloConsoleMock that ...
type HelloConsoleMock struct {
  Calls int
  Text  string
}

// ... fullfills HelloConsole interface
func (console *HelloConsoleMock) Print(text string) {
  console.Calls++
  console.Text = text
}

func TestShouldReturnPrintHelloMessage(t *testing.T) {

  messageText := "Hello Test!"

  // 2.3 Create a HelloMessageMock
  // that will return "Hello Test!"
  message := HelloMessageMock{messageText}

  // 2.4 Create a HelloConsoleMock
  // that will capture its calls
  console := HelloConsoleMock{}

  // 2.5 Create a HelloApp, the one we want to test, passing the mocks
  app := HelloApp{&message, &console}
  // 2.6 Execute the method we want to test
  app.PrintHello()

  // 2.7 Assert HelloConsoleMock has been called once
  if console.Calls != 1 {
    t.Fatalf("HelloConsole expected calls 1 but got %d", console.Calls)
  }

  // 2.8 Assert HelloConsoleMock has been called with "Hello Test!"
  if console.Text != messageText {
    t.Fatalf("HelloConsole expected text %s but got %s", messageText, console.Text)
  }
}

Test output should look like:

=== RUN   TestShouldReturnPrintHelloMessage
--- PASS: TestShouldReturnPrintHelloMessage (0.00s)
=== RUN   TestShouldReturnHelloWorld
--- PASS: TestShouldReturnHelloWorld (0.00s)
PASS
ok      org.hello/main  0.412s

Happy Testing! 💙