Midterm evaluations ran from the 10th to the 14th. Deep linking and roster sync both
shipped, two more pull requests merged, and then I sat down to write an honest account of
where the project stands — which is how I found out that the plan I have been working
from for a month builds a machine nobody can switch on.
Deep linking, end to end
Two pull requests, two days apart.
#7772 handles the request
side: a launch arrives with LtiDeepLinkingRequest as its message type instead of
LtiResourceLinkRequest, and everything downstream has to route differently. The
platform sends along deep_linking_settings — where to return the response, whether
multiple items are allowed, whether the platform wants a title back — and that has to be
stashed for the response, which happens on a completely separate request.
#7777 is what the instructor
actually sees: a picker listing their circuits and assignments, with the selection posted
back to the LMS as a signed JWT.
The satisfying part is that neither pull request needed anything new from the protocol
layer. The JWT signing is the key manager from week 4. The validation on the way in is
the validator from week 9. Deep linking is a different conversation with the platform,
but it is the same handshake underneath, and by this point the handshake is just there.
The roster
#7779 reads a course roster
over the Names and Roles service — pagination via Link headers, roles parsed out of the
IMS role URIs. #7781 turns
that into group members.
The interesting decision in the second one is how users get matched. Not on email — that
was week 5's lesson, and the reason for it holds here too: a platform's privacy settings
may mean no email arrives at all, and even when one does, it might belong to a different
CircuitVerse account than the person clicking. The match is on sub plus issuer, the
same identity pair the launch signs the user in with. A roster import that mapped on
email could silently add a stranger to a course.
Members created this way get marked as LTI-synced, which matters for the drop-handling
pull request that follows: when someone disappears from the LMS roster, only synced
members get deactivated. Somebody who joined the CircuitVerse group directly should not
be removed because they were never in Canvas to begin with.
Two merges
#7715, the JWT validator,
merged on the 11th. #7763,
the resource link model, merged on the 12th. Both had been sitting approved for a while —
the validator since the end of July.
Five merged now: the 1.1 passback fix, the deployment model, the JWKS and tool
configuration endpoints, the validator, and the resource link. Eight more open and
waiting.
The evaluation
What the evaluation actually asked me to do was describe the state of the project, which
sounds trivial and was not. Writing "here is what works" forces a different question than
"here is what I built this week," and the two answers were not the same shape.
So I went back to the three issues this project exists to close and read them line by
line against what is in the repository. Not the proposal — the issues, as filed.
A pipeline nobody could feed
The autograding phase of my plan had six pull requests: a test case model, a headless
runner wrapper, a grading result, a job that runs on submission, a results page, and a
push to the LMS gradebook. Student submits, autograder runs, grade lands in Canvas.
Every one of those is about executing test cases. Not one of them is about creating
test cases.
The issue says it plainly — "a test case editor in the assignment settings UI where
instructors can add, edit and delete test cases" — and I had somehow read that as
satisfied by a database table. It is not. A model with no UI means the only way to define
a test suite is the Rails console, which is not a feature an instructor can use. I had
planned six pull requests that would ship a grading pipeline with no way to put anything
into it.
Reading further with that lens, four more gaps fell out. There was no visible/hidden flag
on test cases, which both issues ask for. There was nothing for the autograde
configuration — partial credit, allowed attempts, whether students see the cases before
submitting. "Versioned with the assignment" had no plan behind it at all, and my model
actively worked against it: one row per assignment, edited in place, so changing a suite
would quietly invalidate every grade already produced from it.
And the least visible one, which I only found by reading the Grade model rather than
remembering it: nothing in the plan mapped a score onto an assignment's grading scale.
The autograder produces a fraction. Grade validates against letter, percent, or
custom, and rejects no_scale outright. Autograding an assignment with no grading
scale would have failed at the last step, after the run, with the student watching.
Twenty-four becomes twenty-seven
Week 7's post was called "Twenty-Four Pull Requests." That plan is now twenty-seven, all
three additions in the autograding phase: the test case editor, the autograde settings,
and the grading-scale mapping — plus reworking the grading result to carry an attempt
number and a snapshot, and folding hidden-case redaction into the results page.
The snapshot is how "versioned with the assignment" gets solved, and it is the change I
am happiest with. Rather than a version table with all the machinery that implies, each
grading result stores a copy of the suite it ran against. Edit the suite afterwards and
old results stay reproducible, because the thing that produced them is still sitting
right there next to them.
Three of the nine are written. The model and the runner are pushed; the editor is done
and waiting locally, because it needs the model's table to exist and I would rather open
one clean pull request after the first merges than one showing two commits.
Reusing a format instead of inventing one
The nicest thing I wrote this week is four words long.
A test case needs a name, input pin values, expected output pin values, and a hidden
flag. The simulator's testbench format already has groups — a named collection of cases
with input and output signals. My first instinct was to add fields: a name array, a
hidden array, indices lining up across them.
Then: a test case is just a group with n: 1.
One case, one name, one hidden boolean. It validates against the model I had already
written without changing a line of it, it invents no schema, and when the suite reaches
the simulator it is a format the simulator already knows how to run. The editor
serialises its table into exactly that on submit.
I have written the other version of this before — the one with parallel arrays and a
comment explaining how the indices correspond. It works right up until someone deletes a
row from one array and not the others.
Not reimplementing the simulator
The runner wrapper was the piece I was most nervous about, and the design question was
where the simulation actually happens.
The tempting answer was Ruby: run the circuit server-side, in the language the rest of
the grading code is written in. I talked myself out of it in about ten minutes. It means
reimplementing thousands of lines of simulation logic, and then maintaining two engines
that will drift — and the failure mode of that drift is a student's circuit passing in
the browser and failing at grading. For a grading tool there is no worse bug.
Driving a real browser was the next candidate, and capybara-playwright-driver is
already in the Gemfile, so it looked nearly free. A browser process per submission is not
nearly free.
Shelling out to Node was the closest call, because there is precedent: the Yosys
integration in this repository does exactly that. I went with an HTTP call to a sidecar
service instead, mirroring the other half of that same integration — a URL from the
environment, the same timeout-and-post shape the simulator controller already uses. It
keeps the engine on the frontend repository's release cadence, where that code and the
people who know it live.
Every failure collapses into one error class: bad circuit data, a non-2xx, a timeout, a
non-JSON body, an empty result. That is deliberate. The only caller is a background job,
and there is exactly one right answer to all of them — mark the run failed, leave the
student's grade alone.
Where the project stands
Five merged, eight open, three written and waiting. The protocol layer is done and
merged. Grade passback, deep linking, and roster sync are all in the queue. Autograding is
started.
The plan grew by three pull requests this week, which on a burndown chart is a bad week.
It is the most useful thing I did. I have spent eleven weeks getting good at making
mechanisms work, and the evaluation caught me having built most of a system that no
instructor could have used, because I kept reading a feature request as a description of
a machine instead of a description of a person trying to get something done.
Next week: the editor lands, and then the settings and the grading result behind it.
Top comments (0)