The frozen test printed three passes. The pull request list printed zero and zero. Both were true at the same time, and that is the whole episode.
tests/test_issue_670.py::test_run_all_no_jobs_no_sleep PASSED [ 33%]
tests/test_issue_670.py::test_run_all_single_job_no_trailing_sleep PASSED [ 66%]
tests/test_issue_670.py::test_run_all_delay_only_between_jobs PASSED [100%]
============================== 3 passed in 0.15s ===============================
Open (0). Closed (0). No branch, no commit, no pull request. Copilot wrote the fix, tested it, and then could not finish the last part of the job — the part where the work leaves the chat and becomes a pull request. I spent the evening watching it stall, then copied the fix out of the chat's own Changes view and verified it myself.
The bug
dbader/schedule is a small Python job scheduler — one module, 945 lines, one test file. Issue #670, open since 17 October, filed by Gambit-Checkmate: Scheduler.run_all(delay_seconds=2) calls time.sleep(2) after the last job, including when there is only one job. The docstring says the delay goes between jobs. The code disagrees with its own documentation.
I checked it before the run. The loop was three lines:
for job in self.jobs[:]:
self._run_job(job)
time.sleep(delay_seconds)
Sleep after every job, last one included. With one job registered and the sleep mocked, the event order came out ['job0', 'sleep(2)']. It should have been ['job0']. With three jobs, a trailing sleep after the third. The issue is real, the reproduction is exact, and the fix is small — which is why I picked it. This episode is not about whether an AI can solve a hard problem. It is about whether it can do a small real job end to end: fix, test, pull request.
One honest detail about the repo's own tests: they cannot see this bug. Every existing run_all test calls it with the default delay of 0, so the trailing sleep never fires. The suite passes 40 tests and skips 41 on the broken code, happily. So I wrote my own test before the run — zero, one and three jobs, the same three cases the issue names — froze it with its hash recorded (8f3dcd75…93b9c606 — the full hash is in the data pack), and kept it away from Copilot. On the unfixed code it failed 2 of 3, exactly where the bug is.
The setup
A fork of the repo, a mirror of the issue in the fork, one fresh Copilot chat in agent mode with both attached. One instruction, the whole brief: "Fix the issue attached to this chat. Open a pull request with the fix when you're done." Nothing else. No hints, no line numbers, no mention of my frozen test. Nothing upstream was touched — no code to the real repo, no comment on the real issue.
One more thing to say plainly before the results: the issue itself hands Copilot the shape of the answer. Its proposed fix reads "sleep before each job except the first, retaining the existing snapshot." So this run tests whether Copilot implements a given fix correctly and finishes the job — not whether it can invent the fix. Keep that in mind when you read the verdict.
What Copilot did — the fast part
Copilot was quick and, as it turned out, right. It read the issue and the code, and within the first turn it had the fix written: enumerate the snapshot, sleep before each job except the first, drop the trailing sleep.
for index, job in enumerate(self.jobs[:]):
if index > 0:
time.sleep(delay_seconds)
self._run_job(job)
Plus a regression test of its own, test_run_all_delays_between_jobs, covering the three-job case with the sleep mocked. Two files, +20 −2. It ran its own tests too — pytest was not installed in its workspace, so it fell back to the standard-library runner and got "Ran 2 tests in 0.001s — OK". It also ran the full suite and a type check, but their outputs never rendered — the page showed only loading placeholders — so I cannot tell you what those said, and I will not guess.
Then it stopped. Not finished — stopped.
What Copilot did not do — the long part
After its last tool call, turn 1 sat in "Thinking…" for about 27 minutes. I stopped it at 37. The page showed a banner: "Copilot finished without creating a pull request. Check the chat for details."
The "Create pull request" button does not create a pull request, by the way. It sends "Create pull request" as a message and starts a new turn. Turn 2 said it would finalize the branch, commit the two-file fix, push it, then use the pull-request tool. Then it sat in "Thinking…" for 16 more minutes and did none of those things. The page also lost its connection to the chat four times — "We couldn't reconnect" — and after each reconnect the newest entries vanished from the timeline for a while before coming back.
I went back to the same chat a few minutes later and tried once more. Turn 2 was still stuck, at 17 minutes 26 seconds. I stopped it and sent one short nudge: commit, push, open the pull request. The first send bounced the text back into the composer. The second started a turn that died within two minutes with a pink banner: "Something went wrong. Please try again." Retry failed. Send again failed. Four attempts, all failed at the send step, and the message was still sitting unsent in the composer when I left it. The chat would not take another message.
Final check in the fork: pull requests Open (0), Closed (0). Branches: master and release-1.2.2 only. The branch Copilot had been working on, fix/run-all-final-delay, was never pushed. No commit exists anywhere. The fix lived only as uncommitted changes in the chat's workspace — and even that view was giving up, reporting "Changed file information needs refreshing."
The recovery, and the verdict
So I copied the complete diff out of the chat's Changes view — both files, verbatim — applied it to a fresh copy of the repo at the same release, and ran the tests myself.
| Check | Before the fix | After Copilot's fix |
|---|---|---|
| Frozen test (0/1/3 jobs) | 1 passed, 2 failed | 3 passed |
| Repo's own suite | 40 passed, 41 skipped | 41 passed, 41 skipped |
| 1-job event order | ['job0', 'sleep(2)'] |
['job0'] |
| 3-job event order | trailing sleep(2)
|
['job0', 'sleep(2)', 'job1', 'sleep(2)', 'job2'] |
| Pull request | — | none. 0 open, 0 closed |
The suite went from 40 to 41 passed because Copilot's own new test passes too — all 40 original tests still pass, the same 41 skipped, zero regressions. The diff touches the bug, not the tests: no existing test was changed. By the scoring I wrote before the run — frozen test 3/3, suite green, fix in the code — the verdict is FIXED.
But the verdict has a second half, and it is the more interesting one. The job I gave Copilot had two parts: fix the issue, open the pull request. It did the first part well and the second part not at all. The fix was right. The workflow failed.
What the numbers don't show
Copilot's own test covers only the three-job case. The issue names zero, one and three. In one of its shown thinking blocks it even said it wanted to check whether the tests cover zero or one jobs — and then it wrote the three-job test anyway. My frozen test covered all three, and all three pass, so the fix is genuinely complete. But the gap between what it considered and what it wrote is worth noticing.
The repo's own suite passing on broken code is worth noticing too. Forty tests green, and none of them could see a sleep in the wrong place, because none of them ever passed a delay. A green suite is not the same as a tested behaviour — this episode is a small, concrete case of that.
And the stall itself: I do not know why the chat stalled. Twice, after its tool calls had finished, with nothing left to do but commit and push. I am not going to invent a cause. What I can say is what happened, with times: 27 minutes, then 16, then a dead chat. If you have seen the same thing, I would like to hear about it — one evening is one data point.
The limits, stated plainly
- One attempt, one issue. A demonstration, not a statistic. It says what happened here, on this repo, with this issue — nothing wider.
- The issue handed it the fix's shape. "Sleep before each job except the first" is in the issue text. The test was implementation and follow-through, not invention.
- The fix was recovered, not pulled from a branch. There is no branch and no commit, so I verified the diff copied from the chat's Changes view. The applied diff matches that record exactly (+20 −2, same lines), and the verification outputs are in the pack — but the honest chain is: recovered diff, verified locally. Not a pull request reviewed.
- Copilot's own full-suite result is unknown. Its full-suite and type-check outputs never rendered. My local run of the full suite is what the verdict rests on.
- Small job, one evening. A three-line fix in a 945-line module, one Saturday evening in November. No repeat, no second attempt — the rules said one attempt, and one attempt is what the episode reports.
Run it yourself
The pack has the recovered diff, the frozen test with its hash record, the baseline outputs and the verification outputs. Apply the diff to dbader/schedule at Release 1.2.2 and run the frozen test — it should come back 3/3, and the repo's suite should come back 41 passed, 41 skipped. If your numbers disagree with mine, that is data too. Tell me.
The full Episode 3 files — the method and scoring sheets, the frozen test with its hash record, the baseline and verification outputs, Copilot's replies verbatim, and the recovered diff — are in the members' data pack on Patreon, live now, under "LGTM S3 — The Fix Was Right: Complete Files". The articles stay free; the receipts are there if you want to audit mine.
Next episode, Thursday 19:00: I go looking for verify=False in public code — the TLS switch people turn off "temporarily", and the comments that admit it was never going back on.
So the question I'll leave open until then: if the coding is the part the AI does well and the finishing is the part where it falls over, who is actually doing the job — the tool, or the person watching it?
P.S. Everything I write stays free — that part doesn't change. But if this one was worth something to you, you can buy me a coffee ☕ (one-time or monthly, both work). It covers the API credits and hosting behind these experiments — and honestly, it's also the nudge that gets the next one started on the evenings I'd otherwise skip it. — Salam


Top comments (0)