A pull request gets merged after a maintainer repairs it. What should an automation learn from that outcome: that its original approach worked, or that the maintainer rescued it? GitHub’s October 28 update to Copilot Automations makes that question practical. The useful feedback is the path from the original output to the accepted result, including the work someone had to do in between.
Disclosure: This article was generated and fact-checked with AI using the public sources linked here. It was published through an automated workflow and has not been reviewed by Keith.
GitHub says automations can use outcomes such as changed issue labels, accepted review comments, and merged optimization PRs to propose improvements. Each proposed change is reviewed before it applies. The announcement describes changes to prompts, tools, or triggers, rather than an unchecked system rewriting itself in production. GitHub’s automation documentation currently limits availability to eligible private and internal repositories; public repositories are excluded.
That review step matters. It creates a place to ask whether the feedback actually supports the proposed change. A positive outcome is evidence worth examining. It does not explain, on its own, why the work succeeded.
Preserve the work that made acceptance possible
TMC Price Intelligence’s public case study describes a delivered 2023–2025 engagement that turned supplier emails and attachments into structured pricing information, then connected that information to reports, quotes, and invoice workflows. Product type, size, price, region, and supplier needed to become comparable records. Review was part of the workflow before an extracted value became a business decision.
Consider a fictional extension of that kind of system. A supplier attachment lists a price per carton. An extraction proposes the same number as a price per item. A reviewer corrects the unit, accepts the record, and uses it in a quote.
A feedback record containing only “accepted and used” would reward the wrong thing. It would hide the unit error and the correction that made the record usable. This example is synthetic; the case study does not claim that this mistake occurred or that TMC used Copilot Automations.
Keep the original proposal and the reviewed result distinct. Record which fields changed and, where practical, the reason. A changed price could represent a model error, a supplier revision, a commercial adjustment, or an unrelated typo. Those events should not all teach the same lesson.
The same principle applies to a coding automation. A merge is a valuable destination, but the changes between the agent’s output and the merged revision explain much more about what to improve.
Let “unknown” remain unknown
An untouched output may be correct. It may also be waiting for review, unused, or overlooked. Design feedback so the system can represent those possibilities without treating silence as approval.
For the hypothetical pricing workflow, separate whether the record was reviewed, whether it was corrected, and whether it was used. A reviewer could confirm that an extraction matches the source while deciding not to use that supplier’s offer. That is useful evidence about extraction quality even though no quote followed.
Conversely, a record could be used after extensive repair. That is evidence of business use and human effort, with a separate story about the original extraction.
Adding detailed feedback has a cost. Asking operators to write an essay after every record could erase some of the benefit of automation. Start with a few meaningful correction reasons and capture differences the system already has. Allow a short explanation when the predefined choices miss the issue. Improve the feedback process when repeated ambiguities become visible.
Review the evidence behind the proposed improvement
When an automation proposes a prompt change, inspect the examples it used. Were they accepted unchanged? Did someone repair them? Were the rejected outputs wrong, or merely outside that reviewer’s preferred scope?
Suppose a proposed change would stop flagging uncertain units because flagged records are accepted less often. That pattern has at least two possible explanations: the warnings are noisy, or the warnings are correctly identifying difficult records. Removing them without checking the source would turn a useful signal into a misleading reward.
Before approving that change, try the old and proposed instructions against a small synthetic set with known expectations. Include an unambiguous unit, an ambiguous unit, and a case requiring clarification. Check whether the revision improves the behavior it claims to improve without losing the necessary uncertainty. These are proposed evaluation steps, not results from the TMC engagement.
Keep the previous configuration and the reason for adopting the new one. If later feedback reveals a mistake, the team needs to know which behavior changed and how to return to the earlier version.
Make the learning loop answerable
For organizations adopting this capability, the first design task is defining which outcomes mean what. A merged PR, accepted extraction, or dismissed comment can tell a useful story when paired with its context. A single success flag leaves too much of that story out.
GitHub’s new capability makes proposing improvements easier. The team still decides what counts as improvement. Preserve corrections, distinguish unused work from rejected work, and ask what the automation would learn from the record you are giving it. Otherwise, it could learn to depend on the very cleanup you wanted to reduce.
Top comments (0)