DEV Community

Михаил
Михаил

Posted on • Originally published at agentlabjournal.online

Skills for a Coding Agent: Measuring Their Value on a Real Task

#ai #agents #automation

Skills for a Coding Agent: Measuring Their Value on a Real Task | Agent Lab Journal

  Agent Lab Journal

    Guides
    Glossary
Enter fullscreen mode Exit fullscreen mode

Practice · Coding agents · Quality measurement

Skills for a Coding Agent: Measuring Their Value on a Real Task 

      Level: intermediate
      Reading and practice time: 75 minutes
      Outcome: an installed skill set and a comparative report covering review completeness, test quality, execution time, and context usage



      If a general-purpose coding agent catches a critical defect in one run, writes superficial tests in the next, and forgets the documentation in the third, the model may not be the main problem.
      The agent may lack a stable working method: explicit review criteria, a repeatable test-design sequence, and a rule for synchronizing documentation with code. In this laboratory, we turn those rules into three reusable agent skills, install them, and compare the skilled and unskilled modes on exactly the same task. The outcome is an evidence-backed report rather than an impression based on polished prose.
Enter fullscreen mode Exit fullscreen mode

Contents

  • What you will produce

  • What skills are meant to fix

  • The concrete case

  • Experiment design

  • Quality rubric

  • The three-skill set

  • Installation and discovery

  • Mode A: general-purpose agent

  • Mode B: skilled agent

  • Time and context measurement

  • Scoring the review

  • Scoring the tests

  • Scoring the documentation

  • Comparative report

  • Verification

  • Failure cases

  • Limitations

  • Adopting the skills

What you will produce 

      You will compare two modes of the same coding agent while keeping the model, repository, task, and available tools unchanged:
Enter fullscreen mode Exit fullscreen mode

  • Mode A — general-purpose. The agent receives the task and the repository’s normal instructions, but not the experimental skills.

  • Mode B — skill-equipped. The agent receives the identical task and can apply three local guides: evidence-based review, behavior-oriented test design, and contract documentation synchronization.

The experiment should leave these artifacts:

  • three installed skill files;

  • an immutable copy of the task prompt;

  • separate working directories for modes A and B;

  • agent responses, patches, command output, and run logs;

  • a manual review-completeness score;

  • test execution and negative-control results;

  • a documentation score;

  • measured wall-clock time;

  • input and output token usage when the runtime exposes it;

  • a comparison table with no predetermined winner.

Testable hypothesis 

        Explicit skills should make the work more complete and repeatable. They also add instructions to the agent’s context and may increase execution time. The skills are useful only if the measured quality gain justifies that cost for the task class you tested.
Enter fullscreen mode Exit fullscreen mode

What skills are meant to fix 

      A universal agent can read code, execute commands, edit files, and explain a patch. However, “do a thorough job” does not define which checks are mandatory. One run may begin with tests, another with implementation, and a third may stop after describing a possible defect.



      In this laboratory, a skill is a small versioned document containing activation conditions, a sequence of actions, completion criteria, and prohibitions. It is neither a separate model nor a hidden knowledge base. It organizes capabilities that the agent already has.





          Inconsistent behavior
          Likely cause
          What the skill makes explicit




          The review discusses style but misses a behavioral defect
          No definition of a useful finding
          Failure scenario, evidence, severity, and code location


          The new test repeats only the happy path
          No boundary or failure analysis
          A behavior matrix and minimum negative cases


          A test passes alone but fails in the complete suite
          No verification order
          Focused test, module suite, full suite, and stability check


          The README disagrees with the public interface
          Documentation is treated as optional
          A changed-contract map and executable example checks


          The agent declares completion without evidence
          No required final-report format
          Commands, results, limitations, and changed files






      A skill is also different from a long prompt. The prompt states the current objective; the skill describes a reusable method. If a method applies across many tasks, store and version it separately, then activate it only when its conditions match.
Enter fullscreen mode Exit fullscreen mode

The concrete case: reserving inventory 

      A useful comparison task must involve code, tests, and public documentation. Use a small training service or isolate a similarly sized module in your own repository. The example function reserves every line in an order:
Enter fullscreen mode Exit fullscreen mode 
def reserve_order(inventory, order):
    reserved = []

    for item in order.items:
        inventory.reserve(item.sku, item.quantity)
        reserved.append(item.sku)

    return {
        "order_id": order.id,
        "reserved": reserved,
    }
Enter fullscreen mode Exit fullscreen mode

Give both agent modes this exact task:

      Review the inventory reservation function, fix the problems you confirm, add the necessary tests, and update the documentation of its public behavior. Do not change the successful response format unless necessary. Run appropriate checks and report what was verified.
Enter fullscreen mode Exit fullscreen mode

Before either run, a human evaluator records the known requirements:

  • Quantity must be a positive integer.

  • The same SKU may occur more than once in one order.

  • If any reservation fails, earlier reservations must be compensated.

  • An error must not produce a partially successful response.

  • An empty order must follow an explicitly selected contract.

  • Documentation must explain operation atomicity and expected errors.

      These requirements form the evaluator’s reference sheet. Do not give the sheet to the agent. Otherwise, the experiment measures the ability to reproduce a supplied checklist instead of the value of the skills. For your own task, write an equivalent reference sheet before the first run.

Why this case works 

      The change is small enough to run twice, but it requires several kinds of reasoning. A superficial response may add only a successful-order test. A more complete review may identify the partial side effect, repeated items, quantity validation, and the need to document the failure contract.



      Use local test fixtures and a fake inventory adapter. Do not include production credentials, real customer data, or uncontrolled network dependencies.
Enter fullscreen mode Exit fullscreen mode

Experiment design 

      Treat the comparison as a small benchmark. The availability of the skills should be the only intentional difference between modes.
Enter fullscreen mode Exit fullscreen mode

Freeze the controlled conditions

  • the same model and model version;

  • the same generation parameters, when configurable;

  • the same starting commit;

  • the same task text, byte for byte;

  • the same command and tool-calling permissions;

  • the same network policy;

  • the same time and step limits;

  • equally cold or equally warmed dependency caches;

  • separate worktrees with no artifacts from the other mode.

Create independent worktrees 

      Do not run the two modes sequentially in the same working directory. The second run must not see the first run’s code, tests, notes, or logs.
Enter fullscreen mode Exit fullscreen mode 
mkdir -p experiment
git rev-parse HEAD > experiment/base-sha.txt

git worktree add ../skills-exp-baseline "$(cat experiment/base-sha.txt)"
git worktree add ../skills-exp-skilled  "$(cat experiment/base-sha.txt)"
Enter fullscreen mode Exit fullscreen mode 
      If project-level skills must reside inside the repository, either store them outside both worktrees or create a common commit containing them and explicitly disable their loading in mode A. Record the exact disabling mechanism in the report.
Enter fullscreen mode Exit fullscreen mode

Control the run order 

      For one pair, run A then B and record that order. For a larger set, alternate it: A–B for the first task, B–A for the second. This reduces bias from warm caches, provider load, and the evaluator remembering the preceding response.
Enter fullscreen mode Exit fullscreen mode

Create the quality rubric first 

      Define all scoring rules before opening either result. Otherwise, a polished answer can unconsciously change what the evaluator considers important.
Enter fullscreen mode Exit fullscreen mode

Review completeness

Create experiment/review-rubric.csv:

id,requirement,weight,evidence_rule
R1,"Compensate earlier reservations after a later failure",3,"Names the partial-reservation scenario and points to the responsible code"
R2,"Validate a positive integer quantity",2,"Identifies at least one concrete invalid input"
R3,"Define behavior for a repeated SKU",2,"Considers aggregation or sequential handling"
R4,"Define the empty-order contract",1,"States and verifies the selected behavior"
R5,"Prevent a partially successful response",2,"Connects the exception path to the public contract"
R6,"Synchronize public documentation",1,"Covers atomicity and documented errors"
Enter fullscreen mode Exit fullscreen mode 
      Weight 3 represents a data-integrity risk, weight 2 represents significant behavior, and weight 1 represents contract completeness. Do not adjust weights after seeing the answers.
Enter fullscreen mode Exit fullscreen mode

Test quality 

          Behavior check
          Points
          Required evidence




          Happy path with multiple items
          1
          Asserts both the response and inventory-adapter calls


          Invalid quantity
          2
          Covers zero, negative, or non-integer values as required by the contract


          Failure on the second item
          3
          Verifies compensation of the first reservation


          Repeated SKU
          2
          Asserts the selected behavior, not merely the absence of an exception


          Empty order
          1
          The expectation agrees with the documentation


          Tests expose the original defect
          3
          At least one new test fails on the original implementation and passes after the fix
Enter fullscreen mode Exit fullscreen mode

Documentation quality 

      Award one point for each accurately documented item: public entry point, valid input, successful result, partial-failure behavior, empty-order behavior, and the test command. The maximum documentation score is 6.
Enter fullscreen mode Exit fullscreen mode

The three-skill set 

      Skill locations and manifests vary by agent. The portable structure below uses one directory per skill and a SKILL.md file containing metadata and instructions. If your environment uses another manifest, preserve the activation conditions and procedure.
Enter fullscreen mode Exit fullscreen mode 
skills-src/
├── evidence-code-review/
│   └── SKILL.md
├── behavior-test-design/
│   └── SKILL.md
└── contract-doc-sync/
    └── SKILL.md
Enter fullscreen mode Exit fullscreen mode

Skill 1: evidence-based code review

Create skills-src/evidence-code-review/SKILL.md:

---
name: evidence-code-review
description: "Apply when reviewing a code change or investigating defects before editing."
---

# Evidence-based code review

1. Read the task, the diff, and applicable repository instructions first.
2. List changed public contracts, side effects, and data boundaries.
3. For every risk, state a failure scenario:
   input → state → action → observable incorrect result.
4. Verify each hypothesis against code, tests, and configuration.
5. Do not report a finding without a concrete location and evidence.
6. Separate:
   - confirmed defects;
   - risks requiring more information;
   - style improvements.
7. Before editing, write a short repair plan.
8. After editing, recheck every confirmed scenario.

Required risk classes:
- validation errors and boundary values;
- partial side effects and compensation;
- repeated and concurrent operations;
- exception handling;
- changes to external contracts;
- security and trust boundaries.

Finding format:
- severity;
- file and line;
- failure scenario;
- evidence;
- minimal verification or repair.

Do not expand the review into unrelated old code unless it is
needed to verify changed behavior.
Enter fullscreen mode Exit fullscreen mode

Skill 2: behavior-oriented test design

Create skills-src/behavior-test-design/SKILL.md:

---
name: behavior-test-design
description: Apply when adding, repairing, or evaluating automated tests.
---

# Behavior-oriented test design

1. Before writing tests, create a matrix:
   behavior, input, precondition, observation, expected result.
2. Cover at least:
   - a representative successful path;
   - input boundaries;
   - failure of an external dependency;
   - a partially completed sequence;
   - empty or missing input when applicable;
   - a repeated operation when side effects exist.
3. Assert public behavior and meaningful side effects.
4. Avoid coupling tests to internal implementation without a reason.
5. Do not weaken existing expectations to obtain a green run.
6. Run each new defect test against the original implementation first.
   It must fail for the expected reason.
7. Apply the smallest repair, then run:
   - the new focused test;
   - the module test suite;
   - the complete available suite.
8. If the complete suite cannot run, report the exact reason.

Final report:
- scenarios added;
- commands executed;
- observed pass and failure counts;
- anything left unverified.
Enter fullscreen mode Exit fullscreen mode

Skill 3: contract documentation synchronization

Create skills-src/contract-doc-sync/SKILL.md:

---
name: contract-doc-sync
description: Apply when code changes public behavior, errors, configuration, or usage commands.
---

# Contract documentation sync

1. Identify the audience and public entry point.
2. Compare the old and new contracts:
   inputs, outputs, errors, side effects, and limitations.
3. Find documentation, examples, and comments describing the contract.
4. Document only behavior confirmed by code and tests.
5. Do not promise guarantees that the implementation does not provide.
6. For each usage example, verify:
   - function and parameter names;
   - result format;
   - expected error;
   - execution command.
7. Run executable examples or at least validate their syntax.
8. List updated documents and known limitations in the final report.

Documentation is complete when a reader can determine:
- what the operation does;
- which inputs are valid;
- what success means;
- what happens on failure;
- how to verify the described behavior.
Enter fullscreen mode Exit fullscreen mode 
      The files are intentionally short. A skill that controls architecture, security, Git, testing, and documentation at once becomes another vague global prompt.
Enter fullscreen mode Exit fullscreen mode

Installation and discovery 

      Find the documented user-level or project-level skill directory for your agent. Do not guess: some agents read skills directly from a repository, others use a user directory, and some require registration in configuration.
Enter fullscreen mode Exit fullscreen mode

If your agent supports .agent/skills, installation can look like this:

install -d .agent/skills

cp -R skills-src/evidence-code-review .agent/skills/
cp -R skills-src/behavior-test-design .agent/skills/
cp -R skills-src/contract-doc-sync .agent/skills/

find .agent/skills -maxdepth 2 -name SKILL.md -print
Enter fullscreen mode Exit fullscreen mode

The command should print exactly three paths:

.agent/skills/evidence-code-review/SKILL.md
.agent/skills/behavior-test-design/SKILL.md
.agent/skills/contract-doc-sync/SKILL.md
Enter fullscreen mode Exit fullscreen mode 
      Next, use the agent’s supported skill-listing command or interface. Record the actual command and its output. Do not substitute an example command from a different product.
Enter fullscreen mode Exit fullscreen mode 
Discovery checklist:
[ ] evidence-code-review is listed
[ ] behavior-test-design is listed
[ ] contract-doc-sync is listed
[ ] each description is visible to the agent
[ ] all three are disabled in mode A
[ ] all three are available in mode B
Enter fullscreen mode Exit fullscreen mode

Verify activation, not only file presence 

      A correctly placed file may still be ignored because of invalid metadata. Give the agent a read-only diagnostic request: “Which available skills apply to a task involving review, tests, and documentation? Name them without changing files.”



      Mode B should identify all three installed names. Mode A must not expose them. Save both diagnostic responses. If the modes see the same skills, stop: the intended treatment was not applied.
Enter fullscreen mode Exit fullscreen mode

Mode A: run the general-purpose agent 

      Treat the unskilled run as the baseline. Enter its worktree, verify the starting commit, and confirm that the existing suite is green:
Enter fullscreen mode Exit fullscreen mode 
cd ../skills-exp-baseline
git status --short
git rev-parse HEAD
python -m pytest
Enter fullscreen mode Exit fullscreen mode 
      If the starting suite already fails, record the known failures or select another commit. Do not attribute a pre-existing failure to either agent mode.
Enter fullscreen mode Exit fullscreen mode

Store the task in one immutable file:

sha256sum experiment/task.txt
date -u +"%Y-%m-%dT%H:%M:%SZ" > experiment/baseline-started-at.txt
Enter fullscreen mode Exit fullscreen mode 
      Send the exact contents of experiment/task.txt without extra hints. Save the complete response, patch, command log, model identifier, runtime limits, and usage information exposed by the environment.
Enter fullscreen mode Exit fullscreen mode 
git diff --binary > experiment/baseline.patch
git diff --stat > experiment/baseline-stat.txt
git status --short > experiment/baseline-status.txt
date -u +"%Y-%m-%dT%H:%M:%SZ" > experiment/baseline-finished-at.txt
Enter fullscreen mode Exit fullscreen mode

Mode B: run the skill-equipped agent 

      Repeat the same preparation in the skilled worktree. Confirm that its initial SHA matches the recorded base and that the three skills are discoverable.
Enter fullscreen mode Exit fullscreen mode 
cd ../skills-exp-skilled
git status --short
test "$(git rev-parse HEAD)" = "$(cat ../agentlabjournal/experiment/base-sha.txt)"
python -m pytest
Enter fullscreen mode Exit fullscreen mode 
      Send the same task file without adding “use every skill” or naming expected defects. Skill activation is part of the behavior being tested.
Enter fullscreen mode Exit fullscreen mode 
sha256sum experiment/task.txt
date -u +"%Y-%m-%dT%H:%M:%SZ" > experiment/skilled-started-at.txt

# Run the agent through the normal interface.

git diff --binary > experiment/skilled.patch
git diff --stat > experiment/skilled-stat.txt
git status --short > experiment/skilled-status.txt
date -u +"%Y-%m-%dT%H:%M:%SZ" > experiment/skilled-finished-at.txt
Enter fullscreen mode Exit fullscreen mode 
      Preserve any trace showing which skills were loaded. A final response claiming that a skill was used is weaker evidence than a runtime event or discovery record.
Enter fullscreen mode Exit fullscreen mode

Measure time and context usage 

      Prefer timestamps emitted by the agent runtime. If none are available, wrap the complete invocation with a monotonic timer or GNU time:
Enter fullscreen mode Exit fullscreen mode 
/usr/bin/time -f \
'elapsed_seconds=%e
user_seconds=%U
system_seconds=%S
max_rss_kb=%M' \
-o experiment/timing.txt \
your-agent-command < experiment/task.txt
Enter fullscreen mode Exit fullscreen mode 
      Measure the entire agent run, but report dependency installation separately if only one mode had to perform it.
Enter fullscreen mode Exit fullscreen mode

What “context usage” means

Record distinct values rather than merging them:

  • uncached input tokens;

  • cached input tokens, if reported;

  • output tokens;

  • total model calls;

  • peak context size, if available;

  • number and type of tool calls;

  • bytes or lines read from repository files as a secondary proxy.

{
  "mode": "A-or-B",
  "model": "copy-from-runtime",
  "started_at": "copy-from-log",
  "finished_at": "copy-from-log",
  "elapsed_seconds": null,
  "input_tokens_uncached": null,
  "input_tokens_cached": null,
  "output_tokens": null,
  "peak_context_tokens": null,
  "model_calls": null,
  "tool_calls": null,
  "usage_source": "runtime-log-or-unavailable"
}
Enter fullscreen mode Exit fullscreen mode 
      Leave unavailable values as null or label them “not available.” Do not estimate token counts from character length and present them as provider-reported usage.
Enter fullscreen mode Exit fullscreen mode

Score review completeness 

      Blind the variants before evaluation. Copy the responses and patches to neutral names such as variant-x and variant-y, then score each rubric row.
Enter fullscreen mode Exit fullscreen mode 
id,weight,variant_x_found,variant_y_found,x_evidence,y_evidence
R1,3,0,0,"",""
R2,2,0,0,"",""
R3,2,0,0,"",""
R4,1,0,0,"",""
R5,2,0,0,"",""
R6,1,0,0,"",""
Enter fullscreen mode Exit fullscreen mode

Calculate weighted completeness:

review_completeness =
  sum(weight for confirmed applicable findings)
  / sum(weight for all applicable rubric findings)
Enter fullscreen mode Exit fullscreen mode 
      A finding counts only if it describes an applicable failure scenario and cites enough code evidence to verify it. Vague advice such as “improve error handling” does not satisfy R1 or R5.
Enter fullscreen mode Exit fullscreen mode

Track false findings separately 

      A longer review can appear more complete simply because it makes more claims. For every reported issue, classify it as confirmed, unconfirmed because information is missing, or contradicted by the repository.
Enter fullscreen mode Exit fullscreen mode 
finding_precision =
  confirmed_findings / all_checkable_findings
Enter fullscreen mode Exit fullscreen mode 
      Do not force an opinion-based suggestion into the defect category. Style recommendations may be useful, but they do not prove review accuracy.
Enter fullscreen mode Exit fullscreen mode

Score test quality 

      A green test file is not enough. Evaluate executability, defect detection, observed behavior, and stability.
Enter fullscreen mode Exit fullscreen mode

1. Executability 

python -m pytest -q path/to/new_test_file.py
python -m pytest -q path/to/module_tests
python -m pytest -q
Enter fullscreen mode Exit fullscreen mode

Save raw output and exit status for each command.

2. Negative control 

      A useful regression test should expose the behavior it claims to protect. Apply the new tests to a clean copy of the original implementation:
Enter fullscreen mode Exit fullscreen mode 
git worktree add ../skills-exp-control "$(cat experiment/base-sha.txt)"
cd ../skills-exp-control

# Copy only the new or modified tests from the evaluated variant.
python -m pytest -q path/to/new_test_file.py
Enter fullscreen mode Exit fullscreen mode 
      At least one defect-specific test should fail for the expected reason. If every new test passes on the original implementation, the tests may not exercise the repaired behavior.
Enter fullscreen mode Exit fullscreen mode

3. Observable effects 

      For the second-item failure, verify the compensation call rather than only checking that an exception was raised. For a repeated SKU, assert the selected quantity behavior rather than merely checking that the function returned.
Enter fullscreen mode Exit fullscreen mode

4. Stability 

for run in 1 2 3 4 5; do
  python -m pytest -q path/to/new_test_file.py || exit 1
done
Enter fullscreen mode Exit fullscreen mode 
      Five local repetitions do not prove the absence of flakiness, but they can reveal state leakage, order dependence, random inputs, and uncontrolled time.
Enter fullscreen mode Exit fullscreen mode

Score documentation quality 

      Compare documentation with the final code and tests. Natural-sounding prose is not evidence that the contract is correct.





          Documented item
          Verification rule




          Public entry point
          The documented function or endpoint exists


          Valid quantity
          The documented boundary matches validation and tests


          Success result
          The example matches the returned structure


          Partial failure
          The description matches implemented compensation


          Empty order
          The description and test assert the same contract


          Verification command
          The command runs from the stated directory






      If documentation says “the operation is always atomic” while compensation can itself fail, the statement is too strong. Document the actual guarantee and any known limit of the compensation mechanism.
Enter fullscreen mode Exit fullscreen mode

Build the comparative report 

      Do not complete the table from memory. Every value should point to a response, patch, command output, trace, or scoring sheet.





          Metric
          Mode A: no skills
          Mode B: skills enabled
          Evidence




          Weighted review completeness
          To measure
          To measure
          review-score.csv


          Confirmed finding precision
          To measure
          To measure
          Manual verification of every finding


          Test quality score
          To measure
          To measure
          test-score.csv


          New tests fail on original code
          To verify
          To verify
          Negative-control output


          Complete suite passes
          To verify
          To verify
          Saved pytest output


          Documentation score out of 6
          To measure
          To measure
          docs-score.csv


          Elapsed seconds
          To measure
          To measure
          timing.json


          Uncached input tokens
          To measure
          To measure
          Runtime usage record


          Cached input tokens
          To measure
          To measure
          Runtime usage record


          Output tokens
          To measure
          To measure
          Runtime usage record


          Peak context
          To measure or unavailable
          To measure or unavailable
          Model trace


          Tool calls
          To measure
          To measure
          Event log
Enter fullscreen mode Exit fullscreen mode

Write a defensible conclusion

Use four parts:

  • Observation: state which metrics actually differed.

  • Evidence: identify the saved artifacts supporting the values.

  • Interpretation: explain which skill may have contributed to the difference.

  • Decision: state where the set should be used and where its cost is not justified.

      On this task, mode B confirmed [number] of [number] applicable risks, while mode A confirmed [number]. The additional findings were verified by tests [identifiers]. Mode B used [measured value] uncached input tokens and [measured value] seconds, compared with [values] for mode A. We therefore accept the skill set for [tested task class], but not yet for [untested task class].

  If no meaningful difference appears, report that result. Possible explanations include an already detailed project prompt, a task that was too simple, failed skill activation, or a model that already followed the same method consistently.

Verify the experiment 

      The saved artifacts provide basic observability: they let another reader reconstruct what the agent saw, changed, and executed. Before accepting the conclusion, verify:
Enter fullscreen mode Exit fullscreen mode

  • Both worktrees started from the same SHA.

  • The task files have identical hashes.

  • The model and configurable parameters were unchanged.

  • Mode A could not read the experimental skills.

  • Mode B discovered the intended skills.

  • The reference requirements were written before either run.

  • The rubric and weights were not changed after reading the outputs.

  • Every reported number comes from a saved record.

  • New tests were exercised against the original implementation.

  • The full available suite ran for both variants.

  • Documentation was compared with behavior, not only proofread.

  • Unavailable measurements are explicitly marked unavailable.

Repeatability check 

      One paired run demonstrates one case but says little about consistency. If the budget allows, repeat both modes three times from clean states. Do not select the best response from each set. Report median time and token usage, the quality range, and the fraction of runs that confirmed each critical risk.
Enter fullscreen mode Exit fullscreen mode 
finding_consistency =
  runs_where_finding_was_confirmed / total_runs
Enter fullscreen mode Exit fullscreen mode 
      This consistency value directly addresses the original problem: the agent’s review, testing, and documentation behavior changes between runs.
Enter fullscreen mode Exit fullscreen mode

Failure cases

The files are installed, but the skills never activate 

      Common causes are an incorrect directory, invalid metadata, or an activation description that is too narrow. Test discovery and a read-only diagnostic request before running the experiment.
Enter fullscreen mode Exit fullscreen mode

The agent loads every skill for every task 

      This consumes context and dilutes priorities. Narrow the descriptions: the testing skill applies when tests are created or assessed, while the documentation skill applies when a public contract changes.
Enter fullscreen mode Exit fullscreen mode

A skill leaks the answer to the case 

      A skill that explicitly says “check repeated SKUs and compensate inventory reservations” contains the evaluator’s reference answer. Skills should describe general risk classes such as boundaries, repetitions, partial side effects, and dependency failures.
Enter fullscreen mode Exit fullscreen mode

Mode B gets a more detailed task 

      One extra instruction such as “make sure the operation is atomic” can explain the entire difference. Store the task once and supply it without manual edits.
Enter fullscreen mode Exit fullscreen mode

Quality is measured by lines changed 

      A large patch is not necessarily a good repair. The agent may rewrite the module, add brittle mocks, and introduce new risks. Score observable behavior and patch minimality separately.
Enter fullscreen mode Exit fullscreen mode

Green tests are accepted without a negative control 

      A new test may already pass before the repair. Running it against the original implementation distinguishes a useful defect test from a decorative one.
Enter fullscreen mode Exit fullscreen mode

The final response is treated as execution evidence 

      “All tests pass” does not replace saved command output. A run may have reached a limit, hidden part of the output, or executed only one test file.
Enter fullscreen mode Exit fullscreen mode

Timing is compared across different environments 

      The first run may install dependencies or warm caches. Prepare both environments in advance or account for setup time in exactly the same way.
Enter fullscreen mode Exit fullscreen mode

The agent edits the evaluation rubric 

      Keep experiment definitions outside the writable task worktree or make them read-only. Otherwise, the agent may accidentally alter the criteria used to judge its own result.
Enter fullscreen mode Exit fullscreen mode

Limitations 

      A single case does not prove universal value. The skills may work well on a small Python service and add friction in a monorepo, mobile application, generated-code project, or infrastructure migration.



      Manual review scoring contains judgment. Reduce subjectivity by defining the rubric in advance, hiding variant identities, and involving a second evaluator for disputed findings.



      Token consumption depends on more than skill text. Agent implementation, tool schemas, repeated file reads, caching, history compression, and hidden system instructions all affect usage. The comparison is valid only within the frozen environment.



      A compensation test against a fake adapter does not prove transactional behavior in a distributed inventory system. A real integration may require idempotent commands, durable queues, reconciliation, or a separate compensation process.



      Skills become outdated as the repository changes. A new test framework, documentation format, architecture, or review policy requires a skill update and another comparison.



      Agent runs are nondeterministic. Even repeated pairs cannot guarantee future behavior; they provide an empirical range for a defined task class and environment.
Enter fullscreen mode Exit fullscreen mode

Adopting the skills in a real project 

      If the comparison supports adoption, turn the set into a versioned workflow:
Enter fullscreen mode Exit fullscreen mode

  • Store skill sources beside the code or in a controlled shared repository.

  • Assign an owner to every skill.

  • Record the skill-set version or SHA in run logs.

  • Activate only skills relevant to the current task.

  • Validate skill metadata and discovery in continuous integration.

  • Maintain a small evaluation set of anonymized real tasks.

  • Repeat the controlled comparison after changing a skill.

  • Reject updates that improve completeness by generating too many false findings.

  • Monitor time and uncached input-token usage.

  • Remove instructions that do not change observable behavior.

Practical acceptance rule 

      Accept the set for the tested task class only when it repeatedly improves at least one important quality metric, does not reduce critical completeness, remains within the chosen false-finding threshold, and fits the project’s time and context budgets.



  If the full set improves complex repairs but makes trivial edits unnecessarily expensive, route by risk. Small local changes may not need all three skills. Changes involving side effects, public interfaces, or error handling should activate the relevant methods.






    Enter fullscreen mode
    


    Exit fullscreen mode
Enter fullscreen mode Exit fullscreen mode




Conclusion


        Skills are valuable not because they make an agent abstractly “smarter,” but because they turn team expectations into a repeatable procedure: evidence-based review, behavior-focused tests, synchronized documentation, and a verifiable completion report. Their value must be established on identical tasks with saved patches, real test runs, measured time, and honest context accounting.
Enter fullscreen mode Exit fullscreen mode




Replication checklist

  • Select a task involving code, tests, and documentation.

  • Record known risks before running the agent.

  • Freeze the model, parameters, SHA, limits, and environment.

  • Create independent worktrees for modes A and B.

  • Install the three skills and verify discovery.

  • Confirm that mode A cannot access them.

  • Send both modes the identical task.

  • Save responses, patches, commands, usage, and raw logs.

  • Run new tests against the original implementation.

  • Score review, tests, and documentation using the predefined rubric.

  • Use only recorded values for time and context usage.

  • Repeat the paired runs to estimate consistency.

  • Accept, restrict, revise, or reject the set based on measured evidence.

    ← All guides
·
Laboratory glossary →

    Agent Lab Journal
    Real experiments. Verifiable conclusions.

Original article: https://agentlabjournal.online/en/cc-arsenal-skills-comparison.html?utm_source=devto&utm_medium=referral&utm_campaign=agentlabjournal-en-global-all&utm_content=article&utm_term=cc-arsenal-skills-comparison

Top comments (0)