Gating a Merge on an Eval Score in Azure Pipelines

#ai #machinelearning #tutorial #llm

If your Azure Pipelines eval gate runs on pushes to main but never on a pull request, the YAML is not the problem. Microsoft’s documentation is explicit: for an Azure Repos Git repository you cannot configure a PR trigger in the YAML file, and the functionality is implemented by a branch policy instead.

Why your pr trigger does nothing

The pr: key exists in the Azure Pipelines YAML schema, and it works — for GitHub and Bitbucket Cloud repositories. For Azure Repos Git it is inert. The Azure Repos Git documentation states that pull request triggers are implemented using branch policies, and that to enable PR validation you configure the Build validation policy on the target branch. A pr: block in the file is not an error and produces no warning; it simply never causes a run.

Two related things surprise people once the policy exists. Draft pull requests do not trigger a pipeline even with a branch policy configured, so a gate that seems not to run may be running against a draft. And you must be a project administrator of the project to configure validation builds at all, which is why this is often the step that a developer cannot complete themselves.

This is a product behaviour rather than a version detail, but it is the kind of thing that changes. Check the Azure Repos Git page in Microsoft’s Azure Pipelines documentation before assuming it still holds.

The pipeline

A single-stage pipeline is enough. The CI trigger below covers pushes; the pull request path comes from the policy in the next section, and no pr: key appears at all because on Azure Repos it would only be misleading to a reader.

trigger:
  branches:
    include:
      - main
  paths:
    exclude:
      - docs/*

pool:
  vmImage: ubuntu-latest

variables:
  - group: llm-eval-keys
  - name: EVAL_MODEL
    value: gpt-4.1-mini-2025-04-14

steps:
  - task: UsePythonVersion@0
    inputs:
      versionSpec: '3.12'

  - script: pip install -r evals/requirements.txt
    displayName: Install eval dependencies

  - script: |
      python -m evals.run \
        --cases evals/cases.jsonl \
        --thresholds evals/thresholds.json \
        --junit-out $(Build.ArtifactStagingDirectory)/eval.xml \
        --json-out $(Build.ArtifactStagingDirectory)/eval.json
    displayName: Score the golden set
    timeoutInMinutes: 15
    env:
      EVAL_API_KEY: $(EVAL_API_KEY)
      EVAL_MODEL: $(EVAL_MODEL)

  - task: PublishTestResults@2
    condition: always()
    inputs:
      testResultsFormat: JUnit
      testResultsFiles: $(Build.ArtifactStagingDirectory)/eval.xml
      testRunTitle: Eval gate
      failTaskOnFailedTests: true

condition: always() on the publish task is the difference between seeing which cases failed and seeing only a red script step. By default a task does not run once a previous one failed, and the eval step failing is precisely when you want the report.

failTaskOnFailedTests: true is belt and braces rather than the gate itself: the script step has already failed the job by exiting non-zero. It matters only in the case where the runner is misconfigured to exit 0 while writing failing JUnit cases, and it costs nothing to have both. timeoutInMinutes on the step is the more important line — a job-level timeout also exists and, if the job timeout elapses first, the running job including your step is terminated regardless of the longer step value, so set the step bound below whatever the job allows rather than above it.

The branch policy people miss

Open the repository settings, choose Branches, and open the branch policies for the target branch.
Under Build validation, add a policy and select the eval pipeline as the build pipeline.
Set Trigger to Automatic (whenever the source branch is updated) rather than Manual. Manual means the check exists and sits unqueued until someone remembers to run it.
Set Policy requirement to Required. This is the setting that is missed. Microsoft’s documentation describes Optional as providing a notification of the build failure while still allowing pull requests to complete — the check goes red and the merge proceeds.
Set a build expiration. The useful middle option expires a passing build after a number of hours if the protected branch has been updated, so a gate that passed against a three-week-old baseline is re-run rather than trusted.
Optionally add a path filter so the policy does not apply to documentation-only changes. Doing it here rather than in the YAML is what keeps the reported status consistent.

Secret variables are not in your environment

This is the second thing that costs an hour. Ordinary pipeline variables are injected into every task’s environment automatically. Secret variables — whether from a variable group backed by a key vault or marked secret in the UI — are not. They are available for macro substitution as $(EVAL_API_KEY) but absent from the process environment unless you map them explicitly, and that is exactly what the env: block in the script step above is for.

Omit the mapping and the symptom is an authentication error from the provider on a pipeline that has a perfectly good key configured, which reads like a credential problem and is a plumbing one. A quick check: print the length of the variable rather than the variable.

Variable groups add a second layer to this. A group has to be authorised for the pipeline before its variables resolve, and the first run after adding a group can wait on a permission prompt rather than failing outright — a pipeline that appears to hang on its first eval run is sometimes waiting for that approval rather than for a model. If the group is backed by Azure Key Vault, the service connection’s identity needs get and list on the secrets, and a missing list permission is the one that produces a confusing empty result instead of a clear access error.

Publishing results so the score is findable

Azure’s test tab is genuinely good at the thing an eval needs — showing which named cases failed and which are new failures rather than an aggregate. That only works if your runner emits per-case JUnit entries with stable test names, one per eval case, rather than a single test called “eval” that passes or fails. Stable names also let the tab mark a case as newly failing, which is the question a reviewer actually has.

Derive each test name from the case identifier in your case file rather than from its position, or an insertion into the middle of the file renames everything after it and the whole suite reads as newly failing. Keep the suite name constant across runs for the same reason: the history attaches to a case, not to a run.

Publish the JSON as a build artifact alongside it. The XML is for the UI; the JSON is what a later run reads to compare against a baseline, and what a reviewer downloads when the number moved and nobody knows which cases moved it. If you want the numbers in front of the reviewer without a download, see posting eval results as a pull request comment, and for keeping a run-over-run series, storing eval history for a CI trend.