DEV Community

Nilofer 🚀
Nilofer 🚀

Posted on

Agentsync: Version, Merge, and Audit AI Agent Configurations Like Code

#agents #machinelearning #node #cli

Most AI engineering teams now run a stack of agent configs across many repos - model choices, tool allowlists, prompt templates, eval thresholds, safety rules. These configs drift the moment two engineers touch them. One repo gets a new policy, another keeps the old one, and nobody notices until an agent makes a decision in production that no one signed off on. Merging configs by hand is error-prone, and there is rarely an audit trail of what changed, when, or why.

Agentsync is a Node.js CLI tool that makes agent configuration something you can version, merge, and audit like code. Load JSON, YAML, or INI configs from any repo, three-way merge with conflict detection, run a 52-point compliance rubric on every change, and keep a full merge history you can revert. The point is that "which config is the source of truth for the agent in production?" should always have a clear, auditable answer.

Features

  • 7 Core Commands - init, push, pull, diff, audit, status, revert
  • Smart Merging - three-way merge algorithm with automatic conflict detection, manual resolution support, and conflict tracking
  • Compliance Auditing - 52-point security and compliance rubric covering security, compliance, structure, performance, and documentation
  • Git Integration - seamless push and pull with git-based version control
  • Merge History - full audit trail with revert capability
  • Format Support - JSON, YAML, and INI configs

Architecture 

┌─────────────────────────────────────────────────────────────────┐
│                         agentsync CLI                           │
├──────────────┬──────────────┬────────────┬───────────┬──────────┤
│ init         │ push         │ pull       │ diff      │ audit    │
│ Initialize   │ Push changes │ Merge      │ Compare   │ Validate │
│ repository   │ to remote    │ remote     │ configs   │ configs  │
└──────────────┴──────────────┴────────────┴───────────┴──────────┘
        │           │                 │
        └───────────┴─────────────────┘
                    │
        ┌───────────┴───────────┐
        │                       │
   ┌────▼─────┐        ┌─────────▼──┐
   │   Git    │        │   Config   │
   │ Manager  │        │   Loader   │
   └────┬─────┘        └─────┬──────┘
        │                    │
        │   ┌────────────────┘
        │   │
   ┌────▼───▼─────────┐
   │  Merge Engine    │
   │  - 3-way merge   │
   │  - Conflict Mgmt │
   └────┬─────────────┘
        │
   ┌────▼──────────────────┐
   │  Audit Engine         │
   │  - Security scoring   │
   │  - Compliance audit   │
   │  - 52-point rubric    │
   └───────────────────────┘
Enter fullscreen mode Exit fullscreen mode

How It Works

The workflow follows a clear sequence. You initialize agentsync in your repository, which sets up local storage at ~/.agentsync/ and connects to a central git remote. From there:

Push - local config changes are committed and pushed to the remote with a message.

Pull - remote configs are fetched and merged into the local state using the three-way merge algorithm. Changes that only one side made are merged automatically. Conflicts - where both sides changed the same key - are surfaced for resolution. Manual resolution mode (--manual) enables interactive conflict handling.

Diff - shows configuration differences between any two refs, letting you see what changed between versions before committing to a merge.

Audit - runs the 52-point compliance rubric against any config directory. The rubric checks security (hardcoded credentials, encryption, secrets), compliance (audit logs, access control, data retention), structure (proper hierarchy, no duplicates, versioning), performance (object sizes, caching, connection pooling), and documentation (comments, examples, change logs). Every config gets a score from 0 to 100.

Revert - restores configuration from any point in the merge history. Every merge is stored as a timestamped JSON file in ~/.agentsync/history/.

Installation 

npm install
Enter fullscreen mode Exit fullscreen mode

Requires Node.js 16+. Git integration expects a repository with a remote named origin and a default branch of main with write access.

Usage

Initialize

agentsync init -r https://github.com/org/configs
Enter fullscreen mode Exit fullscreen mode

Push Changes

agentsync push -m "Update API configs"
agentsync push --directory ./configs
Enter fullscreen mode Exit fullscreen mode

Pull and Merge

agentsync pull
agentsync pull --manual  # Interactive conflict resolution
Enter fullscreen mode Exit fullscreen mode

View Differences

agentsync diff --from HEAD~1 --to HEAD
Enter fullscreen mode Exit fullscreen mode

Run Audit

agentsync audit --directory ./configs
agentsync audit --directory ./configs --report  # Generate report
Enter fullscreen mode Exit fullscreen mode

Check Status

agentsync status
Enter fullscreen mode Exit fullscreen mode

Restore from History

agentsync revert                    # List recent merges
agentsync revert 2026-05-13T12:30   # Revert to specific merge
Enter fullscreen mode Exit fullscreen mode

Results and Output

Status Output

=== Git Status ===
Branch: main
Modified files: 2
Untracked files: 0

=== Agentsync Config ===
Initialized: true
Version: 1.0.0
Repository: https://github.com/dakshjain-1616/agentsync-configs

=== Merge History ===
- 2026-05-13T12:30:45.123Z: Update configurations
- 2026-05-13T12:25:30.456Z: Sync team configs
- 2026-05-13T12:20:15.789Z: Initial merge
Enter fullscreen mode Exit fullscreen mode

Audit Report Output

=== AUDIT RESULTS ===

config.json: 95/100
  - Missing version specification
  - Potential hardcoded credentials detected

api-config.yaml: 88/100
  - Config not properly documented
  - Missing compliance metadata
Enter fullscreen mode Exit fullscreen mode

Merge Report Example

# Merge Report

**Date:** 2026-05-13T12:30:45Z
**Message:** Update API configurations

## Merged Configurations

- api-keys.json
- database.yaml
- cache-config.json (⚠️ CONFLICT)
Enter fullscreen mode Exit fullscreen mode

Compliance Scoring

Configs are scored from 0 to 100:
100 - perfect configuration
75–99 - minor issues
50–74 - moderate concerns
< 50 - serious compliance issues

Common violations that trigger score deductions:

  • Hardcoded API keys or passwords
  • Missing version specification
  • Improper config structure

Key Capabilities

3-Way Merge - Intelligent conflict detection. Changes on only one side merge automatically.
52-Point Audit - Catches security issues: hardcoded credentials, missing encryption, compliance gaps.
Format Support - Works with JSON, YAML, and INI configs seamlessly.
Full History - Complete audit trail - who changed what and when.
Revert Support - Roll back to any previous state instantly.

Comparison

When to Use agentsync

Perfect for:

  • Distributed AI engineering teams
  • Multi-stage deployment pipelines
  • Compliance-heavy organizations
  • Configuration-driven microservices

Not ideal for:

  • Single-person projects (use git directly)
  • Non-text binary configs
  • Real-time streaming configs

Configuration Formats

JSON:

{
  "apiKey": "...",
  "version": "1.0.0"
}
Enter fullscreen mode Exit fullscreen mode

YAML:

apiKey: "..."
version: "1.0.0"
Enter fullscreen mode Exit fullscreen mode

INI:

[database]
host=localhost
port=5432
Enter fullscreen mode Exit fullscreen mode

Data Storage

Local data stored in ~/.agentsync/:

~/.agentsync/
├── config/              # Saved configurations
│   └── agentsync.json
└── history/             # Merge audit trail
    └── {timestamp}.json
Enter fullscreen mode Exit fullscreen mode

Performance

Config parsing - O(n) where n = file size
3-way merge - O(k) where k = number of keys
Audit scoring - O(m) where m = config size
Typical operation - under 100ms

Project Structure 

src/
├── index.js                    # CLI entry point
├── modules/
│   ├── errors.js              # Custom error classes
│   ├── logger.js              # Logging utility
│   ├── config-loader.js       # Load configs (JSON, YAML, INI)
│   ├── config-writer.js       # Write configs with backup
│   ├── git-manager.js         # Git operations
│   ├── local-storage.js       # ~/.agentsync persistence
│   ├── merge-engine.js        # 3-way merge algorithm
│   ├── merge-history.js       # Merge audit trail
│   ├── audit-engine.js        # Compliance scoring
│   └── report-generator.js    # Report generation
└── commands/
    ├── init.js
    ├── push.js
    ├── pull.js
    ├── diff.js
    ├── audit.js
    ├── status.js
    └── revert.js
Enter fullscreen mode Exit fullscreen mode

Error Handling

Custom error types handle every failure mode cleanly:
AgentsyncError - base error class
ConfigError - config file issues
GitError - git operation failures
MergeError - merge conflicts or invalid operations

Limitations

  • Single branch syncing (main only)
  • No binary file support (text configs only)
  • Conflict resolution is text-based only

Testing 

npm test
Enter fullscreen mode Exit fullscreen mode

30 tests covering all core modules:

Error handling - 4 tests
Logging - 3 tests
Config loading and writing - 12 tests
Merge engine - 6 tests
Audit engine - 5 tests

All code is test-driven - write test first, implement to pass, refactor for clarity.

Contributing

All code is test-driven:

  • Write test first
  • Implement to pass test
  • Refactor for clarity

How I Built This Using NEO

This project was built using NEO. NEO is a fully autonomous AI engineering agent that can write code and build solutions for AI/ML tasks including AI model evals, prompt optimization and end to end AI pipeline development.

The requirement was a CLI tool for synchronizing AI team configurations across repositories - with three-way merge, a 52-point compliance audit, git integration, merge history, and revert capability, all supporting JSON, YAML, and INI formats. NEO built the full implementation: the CLI entry point, all seven command modules, the merge engine with three-way merge and conflict management, the audit engine with the 52-point rubric, the config loader and writer, the git manager via simple-git, the local storage layer at ~/.agentsync/, the merge history tracker, the report generator, and the 30-test test suite covering all core modules.

How You Can Use and Extend This With NEO

Use it to enforce compliance before configs reach production.
Run agentsync audit --directory ./configs --report as part of your deployment pipeline. Any config scoring below your threshold fails the pipeline before it can introduce hardcoded credentials or compliance gaps into production.

Use the merge history as a compliance audit trail.
Every merge is stored as a timestamped JSON file in ~/.agentsync/history/. For teams with compliance requirements, this gives you a complete record of what changed, when, and under what commit message - queryable and revertable at any point.

Use revert to recover from bad merges instantly.
When a config change causes unexpected agent behavior, agentsync revert 2026-05-13T12:30 restores the full config state to any point in history. No manual git archaeology needed.

Extend it with additional compliance checks.
The audit engine in audit-engine.js implements the 52-point rubric. New compliance checks for domain-specific requirements follow the same scoring pattern and surface automatically in audit reports and scores.

Final Notes

Agent configuration drift is a silent production risk. agentsync makes it manageable by treating configs the way engineers already treat code - versioned, merged with conflict detection, audited for compliance, and fully revertable. The 52-point rubric catches what manual review misses. The merge history means there is always a clear answer to "what is the source of truth?"

The code is at https://github.com/dakshjain-1616/agentsync
You can also build with NEO in your IDE using the VS Code extension or Cursor.
You can use NEO MCP with Claude Code: https://heyneo.com/claude-code

Top comments (0)