DEV Community

Odd_Background_328
Odd_Background_328

Posted on

Copilot's Organization-Level Custom Agents Need a Diff Review Before Rollout

#devops #agents #github #architecture

Visual Studio 2026's July update added organization-level custom agents for GitHub Copilot. An organization owner can now define a custom agent that every repository in the org automatically detects and offers in the agent selector. This is powerful and introduces a rollout risk: a single agent definition affects every developer in the organization simultaneously.

The rollout problem

When an org-level agent is published, every developer working in any repo in that org sees it in their agent selector. If the agent definition contains an incorrect instruction, a broken tool reference, or an overly permissive scope, it affects all developers at once. There is no canary by default.

The agent definition review checklist

Before publishing an org-level custom agent, review its definition against this checklist.

1. Instruction review 

# agent-definition.yml (example structure)
name: org-code-reviewer
description: Reviews PRs for security and style
instructions: |
  Review each file change. Flag:
  - SQL injection in string concatenation
  - Missing input validation on public endpoints
  - Hardcoded credentials
  Do not modify files. Only comment.
tools:
  - read_file
  - search_code
  - post_comment
scope: organization
Enter fullscreen mode Exit fullscreen mode

Check:

  • [ ] Instructions are specific enough to be testable ("flag SQL injection in string concatenation" not "review for security")
  • [ ] Instructions do not conflict with existing repository-level AGENTS.md files
  • [ ] The agent does not claim capabilities it does not have (e.g., "run tests" when no tools entry supports it)

2. Tool surface audit 

# Extract all tool references from the agent definition
rg -n 'tools:' agent-definition.yml -A 20 | grep '^\s*-'
Enter fullscreen mode Exit fullscreen mode

For each tool:

  • What resource does it access? (filesystem, network, repository API)
  • Is it read-only or read-write?
  • Does it require elevated permissions beyond the developer's own role?

A post_comment tool can create noise across every PR. A write_file tool can modify code in every repo. Audit accordingly.

3. Canary test

Before publishing to the organization:

  1. Create a test repository with a known set of PRs
  2. Run the agent definition against those PRs locally or in a sandbox
  3. Record the agent's comments and verify they match expected behavior
  4. Check for false positives (flagging safe code) and false negatives (missing real issues) 
# canary_test.py
"""
Runs an agent definition against a set of test PRs
and records the results for review before org-wide rollout.
"""
import json

TEST_PRS = [
    {
        "pr_id": "test-001",
        "files": [
            {"path": "app.py", "diff": '+query = f"SELECT * FROM users WHERE id = {user_input}"'},
        ],
        "expected_flags": ["SQL injection"],
    },
    {
        "pr_id": "test-002",
        "files": [
            {"path": "utils.py", "diff": '+def format_name(name): return name.strip()'},
        ],
        "expected_flags": [],
    },
]

def run_canary(agent_output_file):
    with open(agent_output_file) as f:
        results = json.load(f)

    pass_count = 0
    fail_count = 0

    for test in TEST_PRS:
        pr_result = next(
            (r for r in results if r["pr_id"] == test["pr_id"]),
            None
        )
        if not pr_result:
            print(f"FAIL: {test['pr_id']} - no output")
            fail_count += 1
            continue

        actual_flags = set(pr_result.get("flags", []))
        expected_flags = set(test["expected_flags"])

        if actual_flags == expected_flags:
            print(f"PASS: {test['pr_id']}")
            pass_count += 1
        else:
            missing = expected_flags - actual_flags
            extra = actual_flags - expected_flags
            print(f"FAIL: {test['pr_id']}")
            if missing:
                print(f"  Missing: {missing}")
            if extra:
                print(f"  Extra (false positive): {extra}")
            fail_count += 1

    print(f"\nCanary result: {pass_count} passed, {fail_count} failed")
    return fail_count == 0

if __name__ == "__main__":
    import sys
    success = run_canary(sys.argv[1] if len(sys.argv) > 1 else "agent_output.json")
    exit(0 if success else 1)
Enter fullscreen mode Exit fullscreen mode

4. Rollback plan

Before publishing:

  • [ ] Save the current agent definition (if replacing one)
  • [ ] Document how to unpublish the org-level agent
  • [ ] Set a review date (e.g., 7 days after rollout)
  • [ ] Define the criteria for rollback (e.g., false-positive rate above 10%)

What to check

If you publish an org-level custom agent today, can you name the one instruction most likely to produce a false positive across your repos? If not, run the canary test first.

Top comments (0)