DEV Community: Daniel Marques

Using AutoGen to automate wiki content review

Daniel Marques — Mon, 26 Jan 2026 00:38:49 +0000

Using AI to review documentation wikis, identify inconsistencies, and suggest structural improvements. This post explains how to perform this analysis locally without needing API keys. We’ll use AutoGen and Ollama to analyze a documentation wiki, examining both its content and hierarchy, and then ask AI agents to propose improvements.

What Are AutoGen and Ollama?

AutoGen is an open-source, multi-agent framework developed by Microsoft designed to simplify the creation and orchestration of applications powered by Large Language Models (LLMs). It enables developers to create AI agent systems where multiple, specialized agents communicate with each other, use tools, and incorporate human feedback to solve complex tasks.

Ollama is an open-source tool designed to simplify running and managing Large Language Models (LLMs) directly on your local machine (computer or server). It acts as a bridge between powerful open-source models (such as Llama, Mistral, and Gemma) and your hardware, making it easy to use AI without needing deep technical expertise.

Requirements

To follow this tutorial, you will need:

Ollama installed on your local machine. You can download it from Ollama's official website.
A sample documentation repository (or use your own). In my case, I used the Kubernetes official documentation.
Python 3.13 (note: Python 3.14 may not yet be fully supported by all dependencies)

Once you have these prerequisites, proceed to set up your Python environment:

pip install autogen ag2[openai]

The ag2[openai] dependency is only needed because if not installed, autogen raises runtime errors

Starting Ollama

First, download and install a model in Ollama. For this tutorial, we'll use the gemma3:4B model:

ollama pull gemma3:4b

Next, start the Ollama server. This step is essential—the Python script will connect to this server at http://localhost:11434/v1:

ollama serve

Important: Ensure the Ollama server is running before executing your Python script. You should see output confirming the server is listening.

Setting Up the AutoGen Agents

Now, let's create a Python script to set up the AutoGen agents that will analyze the documentation.

Step 1: Configure the LLM

First, configure the LLM settings:

OLLAMA_MODEL = "gemma3:4b"
OLLAMA_BASE_URL = "http://localhost:11434/v1"

llm_config = {
    "model": OLLAMA_MODEL,
    "base_url": OLLAMA_BASE_URL,
    "api_key": "ollama",
    "temperature": 0,  # Set to 0 for deterministic output
}

Setting temperature to 0 ensures deterministic, consistent responses from the model.

Step 2: Create the Content Evaluation Agent

Next, create an agent to evaluate the quality of individual documentation files:

from autogen import AssistantAgent

DOC_TYPE = "setup guide"
DOC_LANGUAGE = "English"

content_agent = AssistantAgent(
    name="ContentAgent",
    llm_config=llm_config,
    system_message=f"""
You evaluate individual markdown files as follows:
- document type is {DOC_TYPE}
- language is {DOC_LANGUAGE}
- the evaluation should return a score between 0 and 1, where 1 is best
- this is an evaluation task; do not suggest rewrites
"""
)

The system prompt makes it clear that this agent should evaluate content, not rewrite it.

Step 3: Execute the Evaluation Prompt

Now, execute the evaluation prompt for each file. The prompt explicitly requires JSON output, which makes it easy to parse results programmatically.

The following code shows how to do that.

...
for path in files:
    with open(path, "r", encoding="utf-8") as f:
        content = f.read()

    content_prompt = f"""
You are a documentation-quality evaluator. Evaluate this markdown file and return ONLY valid JSON (either a raw JSON object or a fenced ```
{% endraw %}
json block). Do NOT include any extra text, commentary, or explanations.

Output requirements (MANDATORY):
- Reply with exactly one JSON object with these top-level keys and types:
  - path (string): must equal the provided path.
  - score (number): 0.00 to 1.00 (float). Holistic quality combining clarity, correctness, and completeness. Round to two decimal places.
  - status (string): one of "OK", "WARN", or "FAIL" determined by score as follows:
      - score >= 0.70 -> "OK"
      - 0.50 <= score < 0.70 -> "WARN"
      - score < 0.50 -> "FAIL"
  - notes (string, optional): up to 300 characters with concise diagnostic observations (do NOT include rewritten text or long examples).

Validation rules:
- The 'path' value must exactly match the provided path.
- Numeric fields must be within [0.00, 1.00] and formatted with two decimal places.
- Do not include any additional top-level keys beyond path, score, status, notes.

Example valid response:
{{"path":"{path}","score":0.78,"status":"WARN","notes":"Clear structure but missing prerequisites section."}}

Input (do not modify):
- path: {path}
- content: {content}
"""

    reply = content_agent.generate_reply(
        messages=[{"role": "user", "content": content_prompt}]
    )

    ...
{% raw %}

Note on large files: If you're evaluating large documentation files, consider truncating the content to avoid exceeding token limits. Add this before sending the prompt:


python
MAX_CONTENT_LENGTH = 4000
if len(content) > MAX_CONTENT_LENGTH:
    content = content[:MAX_CONTENT_LENGTH] + "\n... [content truncated] ..."
    # Note this in your prompt so the evaluator knows

Analyzing the Results

The full code that processes results and generates a markdown report is available in my GitHub repository: documentation-advises.

You can find the complete implementation in doc_review_agents.py.

The script generates a markdown report with:

Folder & File Moves: Structural improvements recommended by the AI
Document Quality Scores: Individual file assessments with status (OK/WARN/FAIL)

Example report output:

Problems Encountered and Solutions

During my tests, I faced several challenges:

1. Prompt Debugging Difficulty

Problem: There's no easy way to debug prompts sent to the LLM. If output is unexpected, testing becomes tedious.

Solution:

Use Ollama's desktop app to test prompts interactively before integrating them
Log all prompts and responses to a file for analysis
Start with simple, single-purpose prompts before adding complexity

2. Unreliable JSON Output

Problem: The LLM sometimes returns invalid JSON or mixes JSON with explanatory text (not following the prompt request).

Solution:

Implement validation: check for required fields before processing
Set temperature: 0 for deterministic output

3. Conditional Instructions Fail

Problem: Conditional instructions like "if content is truncated, do X, else do Y" are often ignored by LLMs.

Solution:

Avoid conditionals; use explicit, imperative instructions instead
Pre-process data before sending (truncate files yourself rather than asking the model to)
Keep prompts focused on a single task

Future Improvements

Here are some enhancements I'm considering for this approach:

1. Vector Database Integration

Store document embeddings in a local vector database (e.g. ChromaDB) to enable semantic comparison across files (without using the LLM). This would help detect duplicate content or similar documentation that could be consolidated.

2. Purpose-Aware Evaluation

Create evaluation prompts that understand document purpose:

index.md files should provide an overview of the folder's documentation
Setup guides should explain installation and initial configuration
Tutorial pages should include step-by-step instructions with expected outputs

This would improve accuracy of quality assessments.

Conclusion

Using AutoGen and Ollama provides a practical way to automate documentation quality checks and structural analysis. While LLMs have limitations (non-deterministic output, occasional errors), these can be mitigated with careful prompt design, validation, and error handling.

The approach is particularly valuable for teams maintaining large documentation repositories where manual review is impractical. Start small, validate results, and gradually expand the scope of automation.

Resources:

Manual version bumps using semantic release with Azure DevOps

Daniel Marques — Sun, 28 Dec 2025 21:43:54 +0000

The semantic-release package (https://github.com/semantic-release/semantic-release) automates the version management and release process of a project. It determines the next version number based on the commit messages that adhere to the Conventional Commits specification, generates release notes, and publishes the release automatically.

However, some developers prefer more control over when to increment major and minor versions. Companies like JetBrains use the year as a major version and an auto-incrementing integer as a minor version as illustrated in the image below.

Since semantic-release offers various built-in tools, such as a release notes generator, it is interesting to keep using it and just change the bumping logic. That is what I will show in this post.

Azure DevOps pipeline for Manual bumping

My approach involves creating an Azure DevOps pipeline that runs the semantic-release with the following steps:

The pipeline prompts the user to specify the bump type (major, minor, or patch) and the desired version number.
Once the user confirms the choice, if the user opted for a major or minor version, the pipeline transitions to an approval state.
Upon approval, the pipeline increments the version according to the chosen bump type.
The pipeline verifies that the bump aligns with the desired version number.

The code snipet below shows the Azure Devops pipeline YAML with steps above. The parameters is used to request user bump type (default bump type set to patch) and wished version. Its values is set as environment variables that are used at semantic release configuration.

parameters:
  - name: bumpType
    type: string
    default: "patch"
    values:
      - major
      - minor
      - patch
  - name: bumpNumber
    type: string
    default: "0"

pool:
  vmImage: ubuntu-latest

jobs:
- job: approval
  pool: server
  steps:
    - task: ManualValidation@1
      ...
      condition: ne('${{ parameters.bumpType }}', 'patch')
- job: create_tag
  dependsOn: approval
  steps:

   ...

    - script: |
        npx semantic-release
      displayName: Run semantic-release setting ${{ parameters.bumpType }} version to ${{ parameters.bumpNumber }}
      env:
        SEMANTIC_RELEASE_BUMP_TYPE: ${{ parameters.bumpType }}
        SEMANTIC_RELEASE_BUMP_NUMBER: ${{ parameters.bumpNumber }

That’s it! With this setup, you can manually bump the versions of your project without having to worry about commit messages.

The following code snipet, show the semantic release configuration plugin configuration at release.config.cjs. For the plugin @semantic-release/commit-analyzer (which bumps version according to commit messages) is set to always increase the version according to bump type options choosen by the user.

// file release.config.cjs
module.exports = {
  ...
  plugins: [
    ...
    ['@semantic-release/commit-analyzer', { releaseRules: [
        { release: process.env.SEMANTIC_RELEASE_BUMP_TYPE }
    ] }],
    './verify-release.js'
  ]
};

The verify-release.js plugin verifies if the new version is incremented as expected. This ensures that if the pipeline is executed with the same input for the second time, it will fail because the bump will go to an undesired value (taking the JetBrains example, setting the major version to the next year). You can see the verify-release.js code in the next snippet.

// file verify-release.js
module.exports = {
      verifyRelease: async (pluginConfig, context) => {
        const { lastRelease = {}, nextRelease = {}, logger = console } = context;
        const bumpType = process.env.SEMANTIC_RELEASE_BUMP_TYPE;
        const bumpNumber = process.env.SEMANTIC_RELEASE_BUMP_NUMBER;
        logger.log('Verifying expected release.');
        if (!bumpType) {
          logger.log('SEMANTIC_RELEASE_BUMP_TYPE not set — skipping version verification.');
          return;
        }
        if (bumpType == 'patch') {
          logger.log('Bump type set to patch. Nothing to verify.');
          return;
        }
        const actual = nextRelease && nextRelease.version;
        const match = actual.match(/^(\d+)\.(\d+)\.\d+$/);

        if (!match) {
          throw new Error(`Invalid tag format: ${lastRelease}`);
        }

        const actualMajor = Number(match[1]);
        const actualMinor = Number(match[2]);
        if (bumpType == 'major' && actualMajor != bumpNumber) {
          logger.error(`Major version mismatch: expected ${bumpNumber} but will publish ${actualMajor}`);
          throw new Error(`Version verification failed: expected major version ${bumpNumber}, got ${actualMajor}`);
        }
        if (bumpType == 'minor' && actualMinor != bumpNumber) {
          logger.error(`Minor version mismatch: expected ${bumpNumber} but will publish ${actualMinor}`);
          throw new Error(`Version verification failed: expected minor version ${bumpNumber}, got ${actualMinor}`);
        }
        logger.log(`Version verification passed: ${actual}`);
      }
}

The picture below shows one example of pipeline execution that bumped major version.

Building new version

Once the release is ready, you can begin building your application. For this example, I’ve created a simple hello world CLI in Go.

The pipeline’s steps are as follows:

Checkout the code with tags
Gets a tag-based description of the current commit. If the commit lacks a tag, it creates a descriptive version based on the latest tag.
Sets Azure DevOps build number to the description from previous step
Generates cli executable and publishes it as build artifact.

Here’s a code snippet that shows the above pipeline:


variables:
  appName: hello-world
  buildDir: build

steps:
  - checkout: self
    fetchTags: true
    fetchDepth: 0

  - script: |
      export VERSION=$(git describe --tags)
      echo "##vso[build.updatebuildnumber]${VERSION}"
    displayName: "Set build number"

  - task: GoTool@0
    inputs:
      version: "1.25"

  - script: |
      mkdir -p $(buildDir)
      go build -o $(buildDir)/$(appName) ./cmd
    displayName: "Build Go binary"

  - script: |
      cd $(buildDir)
      zip $(appName)-$(Build.BuildNumber).zip $(appName)
    displayName: "Create ZIP with version"

  - task: PublishBuildArtifacts@1
    inputs:
      PathtoPublish: "$(buildDir)"
      ArtifactName: "release"
      publishLocation: "Container"

The following image illustrates one successful build.

The code used in this post is available in the github repository https://github.com/dmo2000/semantic-release-manual

Real-Time Data with gRPC Streaming: .NET & React with Connect RPC

Daniel Marques — Sat, 05 Jul 2025 20:53:48 +0000

gRPC streaming is a powerful technology for building real-time, high-performance applications. Unlike traditional request/response APIs, gRPC streaming enables continuous data exchange between client and server. This makes it ideal for scenarios such as live dashboards or IoT telemetry, where clients need to receive updates as soon as they are available, reducing latency and greatly improving user experience.

To get started, you first define your service in a protobuf file. In this example, I created a service called PatientMonitor that retrieves a patient's heartbeat, temperature, and SpO2 levels. The server returns a stream, allowing the client to receive multiple responses over time. Here’s the protobuf definition:

message VitalRequest {
  string patientId = 1;
}

message VitalResponse {
  string timestamp = 1;
  double heartRate = 2;
  double spo2 = 3;
  double temperature = 4;
}

service PatientMonitorService {
  rpc StreamVitals (VitalRequest) returns (stream VitalResponse);
}

With this definition, you can generate the code needed to implement both the server and client.

Server-side Implementation

On the server side, I used C# to implement the gRPC streaming endpoint. The .NET gRPC library makes it straightforward to define streaming services and handle asynchronous data flows. By adding Grpc.Tools to your .csproj file, the necessary client and server code is generated automatically via the <Protobuf /> elements. Here’s an example project file snippet:

<Project Sdk="Microsoft.NET.Sdk.Web">

  ...

  <ItemGroup>
    <Protobuf Include="../protos/patient_monitor.proto" GrpcServices="Server" />
  </ItemGroup>

  <ItemGroup>
    ...
    <PackageReference Include="Grpc.Tools" Version="2.54.0">
      <PrivateAssets>All</PrivateAssets>
    </PackageReference>
  </ItemGroup>

</Project>

Once the base implementation is generated, you can override the methods to provide your custom logic. Below is an example implementation for the StreamVitals endpoint. Here, a loop sends updates of the patient's vitals every 5 seconds, and the stream ends only when the client cancels the request.

using Grpc.Core;
using PatientMonitoring;

public class PatientMonitorServiceImpl : PatientMonitorService.PatientMonitorServiceBase
{
    private readonly Random _rand = new();

    public override async Task StreamVitals(VitalRequest request,
        IServerStreamWriter<VitalResponse> responseStream,
        ServerCallContext context)
    {
        var HEALTHY_HEARTBEAT = new List<int>([
            72, 73, 71, 74, 72, 70, 71, 73, 74, 72, 73, 72
        ]);
        int count = 0;
        while (!context.CancellationToken.IsCancellationRequested)
        {
            var vitals = new VitalResponse
            {
                Timestamp = DateTimeOffset.UtcNow.ToString(),
                HeartRate = HEALTHY_HEARTBEAT[count % HEALTHY_HEARTBEAT.Count],
                Spo2 = Math.Round(95 + _rand.NextDouble() * 5, 1),
                Temperature = Math.Round(36 + _rand.NextDouble() * 1.5, 1)
            };

            await responseStream.WriteAsync(vitals);
            await Task.Delay(5000);
            count++;
        }
    }
}

Frontend Visualization

For the frontend, I chose JavaScript and React to build a dynamic, real-time UI.

Instead of using the standard gRPC JavaScript library (grpc-web), which hasn’t been updated in almost two years, I opted for the Connect RPC implementation. Connect RPC offers a modern, robust, and developer-friendly experience for gRPC in the browser, making it easy to consume streaming endpoints and integrate them seamlessly into React components.

First, install the following npm packages. With them is possible to generate both client and server code with buf command, which is similar to gRPC protoc.

npm install -D @bufbuild/buf @bufbuild/protobuf @bufbuild/protoc-gen-es
npm install -D @connectrpc/connect @connectrpc/connect-web

Next, create a buf.gen.yaml file to configure code generation (for example, generating TypeScript for the frontend).

version: v2
plugins:
- local: ./node_modules/.bin/protoc-gen-es
  out: src/gen
  opt: target=ts
inputs:
  - proto_file: '../protos/patient_monitor.proto'

Then execute buf generate at the buf.gen.yaml folder. This command will create typescript client implementation at src/gen local folder.

You can set up the gRPC client using the createClient function providing generated metadata for the server and the transport type. Connect RPC supports both standard gRPC and its own protocol. It is worth to mention that Connect RPC does not provide a server implementation for .NET of its own protocol.

import { createClient } from "@connectrpc/connect";
import { createGrpcWebTransport } from "@connectrpc/connect-web";
import { PatientMonitorService } from "./gen/patient_monitor_pb";

...

const transport = createGrpcWebTransport({
    baseUrl: '',
});
return createClient(
    PatientMonitorService,
    transport,
);

Once the client is configured, you can call the streaming endpoint. This returns an async generator that yields values as they arrive. You can iterate over this stream using for await. Here’s an example integrated with React state:

export default function App() {
  const [vitals, setVitals] = useState<VitalResponse[]>([]);
  const [latestVital, setLatestVital] = useState<VitalResponse | null>(null);

  ...

  useEffect(() => {
    const streamVitals = async () => {
      const req = { patientId: "12345" } as VitalRequest;
      const stream = client.streamVitals(req);
      for await (const item of stream) {
        setVitals(prev => {
          if (prev.length >= config.heartTrendLength) {
            return [...prev.slice(1), item];
          }
          return [...prev, item];
        });
        setLatestVital(item);
      }
    };

    streamVitals().catch(console.error);
  }, []);

With this setup, you can visualize the patient's vital signals in real time.

For more details, check out the code at https://github.com/dmo2000/grpc-streaming-dot-net.

References

Review Qodana static code analysis and SCA/SBOM license audit

Daniel Marques — Thu, 12 Jun 2025 00:51:05 +0000

I was on the hunt for a tool that could give me a clear picture of my system’s SBOM (software bill of materials). I wanted to check license info and see which parts are used in all my microservices. That’s when I stumbled upon Qodana, which has a feature called SCA (Software Component Analysis). In this post, I’ll share my thoughts on this tool.

I requested a trial account on https://qodana.cloud/ so I could test out all the features. Then I looked over public repositories for the technologies I usually work with. This resulted in the following table.

Language	Repository	# lines(1)
GO	https://github.com/jackc/pgx	91K
Javascript/Typescript	https://github.com/ngxs/store	100K
Java	https://github.com/liquibase/liquibase-hibernate	7K
C#	https://github.com/FluentValidation/FluentValidation	32K

(1) - To get number of lines, I used the command inside clone directory git ls-files | xargs wc -l

Getting started

To begin, you’ll need to create a project. Once you’ve done that, you’ll need to run qodana. For simplicity, I chose qodana CLI for simplicity. Here’s a visual guide that shows all the steps involved in executing the analysis.

Under the hoods, qodana executes the analysis in Docker containers. On my personal MacBook, I had to install Docker Desktop because it doesn’t work with Rancher Desktop. I also noticed that Docker images are quite large (starting from 4GB), as shown below.

Analysis overview

After analyzing the data, you can check the problems in the first tab. One cool feature is that you can mark problems you won’t solve in the short term and move them to the baseline.

The gadget that gives an overview of the problems is visually appealing, but it’s not very user-friendly because:

You have to click in several dropdowns, which makes it hard to drill down on the problems because you have to keep clicking. On the other hand, the dropdowns allow you to make multiple selections.
The text orientation is set to around the gadget circle, which makes it hard to read.

You can also configure which code inspection rules will be enabled or not.

The final tab shows the license audit results, which is the main reason I wanted to test this tool. You can easily navigate through the dependency tree.

You can also download the SBOM license list in different formats, including CSV. This can be useful so you do not have create a SBOM gathering for every language in your projects. Although, it lacks to provide dependency type or package manager (like npm or Nuget) because some packages have the same name but come from different repositories (for instance, azure sdks for python and java).

Tested languages

Out of the languages I tested, the only major issue I encountered was that is not possible to provide license audit for the NGXS repository (a yarn-based project). Even though the official documentation states that yarn is supported, this was the only sticking point so far.

How to create custom Azure DevOps Pipelines that autoscale with Virtual Machine Scale Sets (VMSS)

Daniel Marques — Thu, 29 May 2025 00:07:51 +0000

Microsoft-hosted Azure DevOps pipelines have some limitations, such as not being able to access Azure resources in private networks or having a disk size limit of 10GB. Fortunately, you can work around these by using custom pipelines. One effective approach is to use Virtual Machine Scale Sets (VMSS), which I’ll explain in detail in this post. The source code is available on my GitHub. You can also read the official comparison between VMSS and Microsoft-hosted agents here.

A VMSS is an Azure compute resource that lets you deploy and manage a group of identical virtual machines at scale. In Azure Pipelines, VMSS can host custom agents and automatically scale the number of build and deployment agents based on workload. This provides efficient resource usage, access to private networks, and more control over your build environment compared to Microsoft-hosted agents.

For this post, we’ll consider a scenario where you need to access an Azure Key Vault and it are only visible for specific networks.

The picture below illustrates that.

We’ll achieve this by following these steps:

Install Packer and Azure CLI
Create a VM image that will be used in the VMSS
Create the VMSS
Create a service principal
Configure the VMSS in Azure DevOps
Allow VMSS to access Azure Key Vault

1. Install Packer and Azure CLI

First, install Packer and the Azure CLI using the official guides:

Next, install the Packer Azure plugin:

packer plugins install github.com/hashicorp/azure

If you are running in Windows, you can execute packer in WSL.

2. Create a VM image that will be used in the VMSS

Azure provides several base VM images, but they may not include all the tools you need (for example, Ubuntu images do not come with the Azure CLI pre-installed). To create a custom image, we will use Packer to automate the process.

You’ll need a Packer configuration file and two scripts: one to install your required software (such as the Azure CLI) and another to deprovision the VM, removing machine-specific data. Packer will output a reusable VM image in your chosen region and resource group.

After preparing your files, build the image (replace my-image as needed):

packer build -var 'output_image_name=my-image' modified-ubuntu-image.pkr.hcl

Key points about these files:

The azure-arm / source_image section defines the temporary VM details used by Packer.
The output_image_name variable makes the image name configurable.
The use_azure_cli_auth option uses authentication from the az login command.
Choose the architecture (ARM or Intel) based on your workload; ARM VMs are often cheaper if you don't need x64 compatibility (like running x64 docker images).
The install-requirements.sh script installs required tools, such as the Azure CLI, without using sudo commands.
The deprovision.sh script removes machine-specific data and credentials, ensuring each VM created from the image starts in a clean, secure state.
The execute_command tells Packer to run scripts as root (sudo -E sh), so you don't need to add sudo commands inside your install script.

File: modified-ubuntu-image.pkr.hcl

variable "output_image_name" {
  type    = string
  default = ""
}

source "azure-arm" "source_image" {
  azure_tags = {
    dept = "Engineering"
    task = "Image deployment"
  }
  use_azure_cli_auth                = true
  image_offer                       = "ubuntu-24_04-lts"
  image_publisher                   = "Canonical"
  image_sku                         = "server"
  location                          = "South Central US"
  managed_image_name                = var.output_image_name
  managed_image_resource_group_name = "my-resource-group"
  os_type                           = "Linux"
  vm_size                           = "Standard_D2s_v6"
}

build {
  sources = ["source.azure-arm.source_image"]

  provisioner "shell" {
    execute_command = "chmod +x {{ .Path }}; {{ .Vars }} sudo -E sh '{{ .Path }}'"
    scripts          = ["./install-requirements.sh", "./deprovision.sh"] 
  }
}

File: deprovision.sh

/usr/sbin/waagent -force -deprovision+user && export HISTSIZE=0 && sync

You can also automate this process using an Azure DevOps pipeline with the file create-vmss-image.yaml. The picture below shows the pipeline execution result followed by the resource created in Azure Portal.

3. Create the VMSS

Create the VMSS using the Azure CLI command below. Update the variables as needed for your environment:

vmss_name=my-azure-devops-pool
vmss_image_name=my-image
resource_group=my-resource-group
vm_size=Standard_DC1s_v3
vm_storage=Standard_LRS

az vmss create \
    --name ${vmss_name} \
    --resource-group ${resource_group} \
    --image ${vmss_image_name} \
    --vm-sku ${vm_size} \
    --storage-sku ${vm_storage} \
    --authentication-type SSH \
    --generate-ssh-keys \
    --instance-count 0 \
    --disable-overprovision \
    --upgrade-policy-mode manual \
    --single-placement-group false \
    --platform-fault-domain-count 1 \
    --load-balancer "" \
    --orchestration-mode Uniform

Tips:

Set the initial scaling to 0 (--instance-count 0); Azure DevOps will scale the VMSS based on pipeline demand.
Choosing the right VM size is important for cost and performance. You may want to select any available size initially and adjust it later in the Azure Portal. The image below shows VM sizes ordered by cost — make sure that you select a size with the same architecture as your custom image.

You can also automate this process using an Azure DevOps pipeline with the file create-vmss.yaml. The picture below shows the pipeline execution result followed by the VMSS and its private network created in Azure Portal.

4. Create a service principal

To enable Azure DevOps to manage your VMSS, you need a technical user known as a service principal. In many organizations, this is created by the corporate IT administrator team. If that's your case, you can skip to the next section.

If you need to create it yourself, follow the official guide to register an app. For this setup, register with the option Accounts in this organizational directory only.

After registration, create credentials for the service principal by generating a client secret as described here.

5. Configure the VMSS in Azure DevOps

With your service principal ready, assign it the Contributor role on your VMSS. This allows Azure DevOps to scale the VMSS up and down as needed for your pipelines.

Next, in Azure DevOps, create a service connection using Azure Resource Manager and enter the service principal details. Follow the steps in the official documentation.

Once the service connection is set up, create an agent pool in Azure DevOps. This pool name will be referenced in your pipeline YAML files.

To verify your setup, create a simple pipeline that runs a command on the VMSS agent. For example, use az --version to confirm the Azure CLI is installed. Make sure the job uses the VMSS agent pool you created. The screenshot below shows a successful test run.

6. Allow VMSS to access Azure Key Vault

First, configure the VMSS to use a system-assigned identity.

Then assign the Key Vault Administrator role to your user and the system-assigned identity on your Key Vault.

Next, make sure to disable public access and attach the Key Vault to the same network as the VMSS, as illustrated below.

Finally, create a pipeline using the VMSS agent pool and paste the following command line code:

az login --identity

az keyvault key create \
  --vault-name my-private-key-vault \
  --name my-pipeline-key \
  --protection software \
  --kty RSA \
  --size 2048

Then execute the pipeline and check that the key was created.

References

Building VM images with Packer - https://learn.microsoft.com/en-us/azure/virtual-machines/linux/build-image-with-packer
Installing Azure Cli - https://learn.microsoft.com/en-us/cli/azure/install-azure-cli-linux
Description of Azure VM size notations - https://learn.microsoft.com/en-us/azure/virtual-machines/sizes/overview
Registering an App - https://learn.microsoft.com/en-us/entra/identity-platform/quickstart-register-app
Create secrets for an App - https://learn.microsoft.com/en-us/entra/identity-platform/how-to-add-credentials?tabs=client-secret
Creating a service connection with Service Principal - https://learn.microsoft.com/en-us/azure/devops/pipelines/library/connect-to-azure?view=azure-devops#create-a-service-connection-for-an-existing-user-assigned-managed-identity
Create Scale Set Agents - https://learn.microsoft.com/en-us/azure/devops/pipelines/agents/scale-set-agents

Shading Removal of Illustrated Documents

Daniel Marques — Sat, 17 May 2025 16:17:10 +0000

During my PhD, I developed and published an algorithm — together with my advisor and a colleague — to enhance photos of documents captured with digital cameras. These images often suffered from uneven lighting across the page. Our algorithm corrects this by normalizing the illumination, similar to what CamScanner does. The implementation is written in C++ and is available on my GitHub: https://github.com/dmo2000/shading-removal-illustrated-docs.

The method works in four steps:

Detection of low-variation regions: The algorithm first identifies areas in the image with low color variation, which are likely to correspond to blank regions of the paper (i.e., without text or illustrations).
Background clustering: From these low-variation regions, it extracts the cluster most likely to represent the paper's true background.
Shading estimation: Once the paper background is identified, the algorithm estimates the shading pattern for the rest of the document, including non-blank areas.

To perform this step, I used a technique known as Natural Neighbor Interpolation (also called Voronoi interpolation). This method is especially useful when data points (in this case, known shading values) are unevenly distributed — a situation similar to how weather data is interpolated from scattered measurement stations.

For the interpolation and related geometric computations, I used the CGAL library. I also relied on data structures from the Boost C++ libraries to support the implementation.
Final enhancement: Finally, the image is enhanced removing the shading obtained in previous step.

The images below shows one example of processing:

Original Image

Cluster with detected background (steps 1 and 2)

Background estimation (step 3)

Final enhancement (step 4)

The complete source code is publicly available on GitHub:
👉 https://github.com/dmo2000/shading-removal-illustrated-docs

The full paper is available at: https://www.researchgate.net/publication/300028589_Shading_Removal_of_Illustrated_Documents

How to Call gRPC Methods Dynamically in Go

Daniel Marques — Sat, 10 May 2025 20:49:23 +0000

gRPC (Google Remote Procedure Call) is a high-performance, open-source framework for making remote procedure calls. Unlike traditional REST APIs that use HTTP/1.1 and JSON, gRPC leverages HTTP/2 and Protocol Buffers for more efficient communication between distributed systems. This modern approach offers significant advantages in terms of performance, type safety, and code generation capabilities.

When working with gRPC, you need to understand the fundamental components and workflow that differentiate it from other API communication methods. The process begins with defining your service interface using Protocol Buffers (protobuf) in a proto file, which serves as a contract between client and server. This definition is then compiled into language-specific code that handles all the underlying communication complexities.

You can make gRPC calls dynamically in two ways: using server reflection or providing uncompiled proto files.

Dynamic Calls Using gRPC Server Reflection

Typically, services are registered with a gRPC server immediately after creation, as shown in the helloworld example provided by grpc team:

import (
    "google.golang.org/grpc"
)

type server struct {
    helloworld.UnimplementedGreeterServer
}

func main() {
    // ...
    s := grpc.NewServer()
    helloworld.RegisterGreeterServer(s, &server{})
    // ...
}

To enable server reflection, register the reflection service by calling the Register function from the reflection package as follows.

import (
    "google.golang.org/grpc"
    "google.golang.org/grpc/reflection"
)

type server struct {
    helloworld.UnimplementedGreeterServer
}

func main() {
    // ...
    s := grpc.NewServer()
    helloworld.RegisterGreeterServer(s, &server{})
    reflection.Register(s)
    // ...
}

After starting the server, you can check the new service by using Kreya's (https://kreya.app/) server reflection importing. The picture below shows this result with server reflection service:

Calling gRPC dynamically with reflection involves two steps: first, a request to the ServerReflectionInfo stream retrieves proto files (FileDescriptors). The sequence diagram illustrates this process. Note that for every service call, one call must be made to the reflection service to fetch the proto file.

The following code implements the process shown in the diagram above. The function grpcurl.DescriptorSourceFromServer retrieves the file descriptor from the reflection service. The call to the target method is done by grpcurl.InvokeRPC function.

package main

import (
    "bytes"
    "context"
    "log"
    "strings"

    "github.com/fullstorydev/grpcurl"
    "github.com/jhump/protoreflect/grpcreflect"

    "google.golang.org/grpc"
    "google.golang.org/grpc/credentials/insecure"
)

func main() {
    // Inputs
    serverAddr := "localhost:50051"
    methodFullName := "helloworld.Greeter/SayHello"
    jsonRequest := `{ "name": "goodbye, hello goodbye, you say stop and I say go go..." }`

    // Output buffer
    var output bytes.Buffer

    // Create gRPC channel
    grpcChannel, err := grpc.Dial(serverAddr, grpc.WithTransportCredentials(insecure.NewCredentials()))
    if err != nil {
        log.Fatalf("Failed to dial server: %v", err)
    }
    defer grpcChannel.Close()

    // Create reflection client
    reflectionClient := grpcreflect.NewClient(context.Background(), grpcreflect.NewClientV1(grpcChannel))
    defer reflectionClient.Reset()

    // Use grpcurl to get the method descriptor
    descriptorSource := grpcurl.DescriptorSourceFromServer(context.Background(), reflectionClient)

    // Prepare formatter for the response
    options := grpcurl.FormatOptions{EmitJSONDefaultFields: true}
    jsonRequestReader := strings.NewReader(jsonRequest)
    rf, formatter, err := grpcurl.RequestParserAndFormatter(grpcurl.Format("json"), descriptorSource, jsonRequestReader, options)
    if err != nil {
        log.Fatalf("Failed to construct request parser and formatter: %v", err)
    }
    eventHandler := &grpcurl.DefaultEventHandler{
        Out:            &output,
        Formatter:      formatter,
        VerbosityLevel: 0,
    }

    headers := []string{}

    err = grpcurl.InvokeRPC(context.Background(), descriptorSource, grpcChannel, methodFullName, headers, eventHandler, rf.Next)
    if err != nil {
        log.Fatalf("RPC call failed: %v", err)
    }
    log.Println("Received output:")
    log.Print(output.String())
}

The images below show the console outputs from the client and server runs, respectively. You can seem from the log output one call for the stream and other for the target service.

Dynamic Calls Using Proto Files

If you have access to the proto file, you can call the target service directly by changing the function grpcurl.DescriptorSourceFromServer to grpcurl.DescriptorSourceFromProtoFiles . In this way, the steps needed to call the service are simplified as shown in the sequence diagram below.

The results from the server and client can be seen below, where the stream is not called anymore.

Conclusion

There are two effective approaches for making dynamic gRPC calls. Server reflection offers several advantages: it eliminates the need to maintain proto files on the client side, enables dynamic discovery of services, and simplifies integration testing and debugging tools. With reflection, clients can automatically discover and interact with services without prior knowledge of their interfaces.

However, server reflection does have drawbacks. Each service call requires an additional reflection call, which adds network overhead. While you can mitigate this by caching file descriptors after the first reflection call. Reflection may also expose additional server details that could potentially reduce security.

Alternatively, using proto files directly provides a more efficient approach with fewer network calls, but requires keeping the client's proto files synchronized with the server.

The complete code examples from this article are available at https://github.com/dmo2000/grpc-dynamic-calls.

Solving Bugs

Daniel Marques — Fri, 18 Apr 2025 22:16:19 +0000

In the world of software development, bugs are inevitable. No matter how experienced the team or how mature the process, every software system will eventually encounter issues that cause unexpected behaviors.

What is a Software Bug?

A software bug is a flaw or error in a program that produces incorrect results or unintended behaviors. Bugs range from:

Minor visual glitches
To serious system crashes
Or even security vulnerabilities

Where Do Bugs Come From?

Bugs emerge from various sources:

Mistakes in code logic
Misunderstandings in requirements
Integration problems
Differences between development and production environments

Some bugs are easy to spot and fix, while others remain deeply hidden and difficult to reproduce.

Why a Strategic Approach Matters

Having a systematic approach to bug fixing—especially when handling multiple bugs—is crucial for maintaining stable, reliable, and user-friendly software.software.

The Bug-Fixing Process

The bug-fixing process follows three clear steps:

Organize bugs
Choose bugs strategically
Fixing the Bugs

Step 1: Organize the Bugs

Before diving in, align with your team to organize bugs using these key criteria:

Criterion	Key Question
Criticality	How severely does this bug impact users or system functionality?
Priority	What's the business importance of fixing this issue?
Expected resolution date	When does this need to be fixed?
Software component	Which part of the system is affected?
Ease of resolution	How complex will the fix likely be?

This structured organization makes it easier to make informed decisions about what to tackle first.

Step 2: Choose Bugs Strategically

Once organized, select bugs to fix using these effective strategies:

For Unfamiliar Components

🔍 Start small: Choose an easy bug in a component you're not familiar with

This builds knowledge without overwhelming you
Provides foundation for tackling more complex issues later

For Related Issues

🔗 Batch similar bugs: When several related bugs exist in the same component and they're easy to fix

Reduces backlog more quickly
Makes the remaining backlog more manageable

When Blocked

⏳ Work in parallel: If one bug investigation is time-consuming

Switch to an easier bug while waiting
Maintains productivity despite blockers

Step 3: Fixing the Bugs

When addressing each bug, first assess the solution path:

Solution Assessment

✅ Straightforward solution: Proceed directly to implementation
🧩 Complex problem: Apply the methodology from my post on Solving software system problems

Time Management Warning Signs

Most bugs should be resolved within a few days. If a fix takes longer, it might indicate:

⚠️ Potential Issues:

Environment problems: Slow development cycle for making changes and verifying fixes
Architectural issues: Underlying design problems making fixes difficult
Misclassification: What seems like a bug might actually be:
- A feature request
- Documentation issue
- User misunderstanding

When facing these issues, document the real problem in your backlog and schedule it for proper resolution later.

After Fixing a Bug

Testing & Verification

Once you've implemented a fix, determine what testing is necessary for verification. Consider which manual tests need to be performed and identify any unit tests that require updates to properly validate your solution. After establishing your test plan, execute all identified tests thoroughly and document the results to maintain a clear record of verification.

Code Integration

After testing, submit your code for review by opening a Pull Request (PR). Address any feedback received from reviewers promptly to ensure the solution meets quality standards. Once your PR is merged, wait for the test environment to update with your changes, then perform a final verification to confirm the bug is truly resolved in the integrated environment.

Continue the Cycle

With one bug successfully resolved, select your next target by returning to the "Choose Bugs Strategically" guidelines outlined earlier. Begin the process again with your next bug, maintaining the same systematic approach. This methodical cycle ensures consistent quality, maintains momentum in your development process, and systematically reduces your bug backlog over time.

This methodical approach ensures quality, maintains momentum, and systematically reduces your bug backlog.

Solving software system problems

Daniel Marques — Sun, 16 Feb 2025 01:08:14 +0000

Some software system problems are difficult to solve due to various reasons, such as conflicting requirements or complex code structures. To ensure a solution achieves the expected outcome, I follow some steps that are inspired by the Six Thinking Hats and Parkinson’s Law.

Edward de Bono developed the Six Thinking Hats as a problem-solving and decision-making method that encourages diverse perspectives. Each “hat” represents a different way of thinking: the White Hat focuses on facts and data, the Red Hat on emotions and intuition, the Black Hat on risks and caution, the Yellow Hat on benefits and optimism, the Green Hat on creativity and new ideas, and the Blue Hat on organization and control of the thinking process. By systematically using each hat, individuals and teams can explore issues more comprehensively, leading to well-rounded and effective decisions.

Cyril Northcote Parkinson formulated Parkinson’s Law, which states that “work expands to fill the time available for its completion.” This means that if a task has a longer deadline than necessary, it will likely take up all that time, even if it could be completed sooner. The law highlights inefficiencies in time management and productivity, often seen in workplaces and bureaucracies. To counteract this, setting shorter, more focused deadlines helps improve efficiency and prevent unnecessary procrastination.

To verify whether a solution achieves the expected outcome, I follow these steps:

Set a short deadline (1–2 days).
Narrow the scope based on the deadline.
Create a Proof of Concept (POC).
If the POC meets the expected outcome, list the software requirements based on it.
Analyze implementation options.
Plan the necessary changes.

For steps 1 and 2, Parkinson’s Law helps me focus on identifying small, high-impact changes within a constrained timeframe.

For step 3, I use the Yellow, Red, and Green Hats to solve the problem in a small scale, even if it compromises quality. However, I keep an eye on the deadline to maintain focus on the goal.

For step 4, I proceed only if the POC meets the target goal. At this stage, I switch to the White, Black, and Blue Hats to list the functional and nonfunctional requirements based on the POC implementation.

For step 5, with the requirements defined, I consider at least two implementation options with their pros and cons. Then I share these with other stakeholders to gather feedback. The items below show some examples of questions that I do this phase:

Does the change fulfill its objective?
How can negative side effects be mitigated?
How can the benefits be measured?
How can the change remain backward compatible?
Are the changes importants for the user?

Finally, at step 6, I decide on an implementation option with the team and document the requirements in a ticket in the release tracking system of the software.

Using this approach, I have solved several software problems throughout my career. I hope this methodology proves helpful to others as well.