DEV Community: Akanksha Trehun

Circuitverse Assignment Enhancement Suite | GSoC 2026 | Final Report

Akanksha Trehun — Thu, 29 Oct 2026 16:17:42 +0000

The time was 11.30pm and the day was 30th of April 2026. I was sitting at my desk with a compiler design assignment open in front of me, pretending to be deeply engrossed in college work. In reality, I hadn't made an ounce of progress since my focus was completely hijacked by my GSoC application tab open on screen. I kept telling myself I’d just check in the morning, but my fingers had a mind of their own on the refresh button. When the page finally updated in my favour, all pretense of studying vanished.

Flash forward to today: I'm writing my final Google Summer of Code 2026 report. Looking back across these past months, the journey has been an intense, incredible blur from tearing apart and rebuilding LTI 1.3 specs from scratch, to navigating PR feedback loops, untangling complex Rails authorization policies, and building an end-to-end autograding engine for digital circuits.

Before diving into the code, here is a bit of context on how I got here.

About Me

I’m Akanksha Trehun (magic-peach), a Computer Science & Engineering student at St. Joseph Engineering College, Mangalore in India.

My relationship with CircuitVerse actually started back in my 3rd semester during a Digital Principles of Design lab. Our professor projected CircuitVerse onto the screen to demonstrate how logic gates, adders, and flip-flops worked in practice rather than just static textbook diagrams. That 20-minute demo stuck with me.

Fast forward to my 6th semester: when the GSoC 2026 accepted organizations list came out and I saw CircuitVerse, it clicked immediately. I knew this was the right time to apply. Seventh and eighth semesters quickly get swallowed by final-year capstone projects and placement prep, so waiting for a "more convenient" moment meant it would never happen.

I’ll be completely transparent—while I was familiar with git workflows and open-source contributions, the CircuitVerse backend runs on Ruby on Rails. Coming into February, I had zero prior experience with Ruby or Rails. It wasn’t part of my college coursework, and my initial glance at the repository felt overwhelming.

So, I took it one step at a time:

Watched beginner Rails tutorials and read through the CircuitVerse developer docs.

Started small—fixing broken documentation, updating outdated specs, and tackling minor issue scopes.

Balancing proposal writing at 5:45 AM before morning lectures wasn't easy, but seeing everything come together made every early alarm worth it.

The Project

This summer, I worked on the Assignment Suite & Classroom Workflow Enhancement project under the mentorship of Anas Khan, Aman Asrani, and Aboobacker M K as org admin.

The main goal was to solve key operational bottlenecks for educators using CircuitVerse at scale:

LTI 1.3 & Canvas Integration: Upgrading authentication from legacy OAuth 1.0a / LTI 1.1 to LTI 1.3 (OAuth 2.0 / OIDC) so institutions can seamlessly connect Canvas/Moodle with secure grade passback and roster sync.

Subgroup & Team Management: Adding nested classroom structures to let instructors divide large course groups into smaller project teams with dedicated submission tracking.

Autograding & Test Suite Editor: Building a background execution workflow where instructors can define input/output test cases to automatically grade student circuit submissions.

Starter Circuit Templates: Creating a template library so teachers can distribute pre-wired base circuits attached directly to assignments.

How It Unfolded

My early contributions started with foundational bug fixes—like scoping AssignmentsController queries to policy scopes and refining invitation rules—which helped me understand CircuitVerse's internal data model inside out.

By March, I drafted a comprehensive proposal complete with an end-to-end proof-of-concept branch demonstrating how LTI 1.3, subgroups, and autograding tied together. After clearing two technical interview rounds including one scheduled right before my compiler design lab exam. I was thrilled to join the 2026 cohort

This post is about what that means for you, whether you teach, study or run a CircuitVerse install, and what you should expect to see and when.

Who this is for

If you are:	What changes for you
An instructor	Assignments launch from your LMS, grades go back to your gradebook, class lists stay in sync, you can split a class into teams, hand every student the same starting circuit and have a circuit graded by tests you write
A student	You open the assignment from your course without a second login, you start from the circuit your instructor prepared, you see which tests passed, and you can work on one shared circuit with your team
An LMS admin	You register CircuitVerse as an LTI 1.3 tool using a standard configuration, instead of the older shared-secret setup
A CircuitVerse admin or maintainer	Every new feature is behind a flag or a setting, so you can merge code without switching anything on

What you can use today, and what is coming

Feature	Who it helps	Status
Grade saved in CircuitVerse before it is sent to the LMS (LTI 1.1)	Instructors using LMS grading today	Merged (#7647)
Foundation for LTI 1.3: platform registration, tool configuration, token checking, resource links, service tokens	LMS admins	Merged, switched off by default
LTI 1.3 sign-in from the LMS	Everyone	In review (#7746)
Deep linking, roster sync, grade passback services	Instructors	In review
Autograding with test suites	Instructors, students	In review
Subgroups (teams)	Instructors, students	In review
Circuit templates	Instructors, students	In review

If you are an instructor

Open an assignment from your LMS without a second login

LTI is the standard that lets CircuitVerse sit inside Canvas or Moodle. The old version, LTI 1.1, proves a request is genuine. Canvas has deprecated that for new installs. LTI 1.3 uses signed tokens instead, so your LMS admin can register CircuitVerse the normal modern way.

What this means for you: your students click the assignment in the course and land in CircuitVerse already signed in. What it means for safety: a launch is only trusted if three checks pass (a state value, a one-time nonce, and a match to the platform that started the login), and the signature on the token is verified.

We identify people by the LMS's stable sub identifier, scoped to the platform, and never by email. An email in a launch is whatever the LMS has on file, and users can often edit it themselves, so matching on it would let someone sign in as somebody else.

User.find_or_create_by!(provider: "lti", uid: "#{deployment.id}:#{payload['sub']}")

The token validator is built to refuse the known tricks:

Lti::JwtValidator.validate!(token, deployment:, nonce:)
encoded_token.verify_signature!(algorithm: "RS256", key: key)

Grades reach the gradebook, and never disagree with CircuitVerse

This one is live today for LTI 1.1. Before, CircuitVerse pushed the score to LMS first and saved it second. If the local save failed, the LMS showed a grade that CircuitVerse had never recorded. Now the grade is saved first (#7647). The review also found that a forged request could wipe a teacher's grading session before it was rejected, so stale context is now cleared only after a launch is verified.

app/controllers/grades_controller.rb:

unless @grade.save
  render json: { error: "Grade is invalid" }, status: :bad_request
  return
end
return unless Flipper.enabled?(:lms_integration, current_user) && session[:is_lti]

submit_grade_to_lms(project, assignment, grade)

For 1.3, the score push (#8029) sends autograded scores through the LMS's line item. It logs a failure instead of hiding it, and it uses a separate identity per LMS platform, so a student who uses two LMSs never has a score sent to the wrong one.

Autograder::AgsPush:

sub = link && project.author&.lti_sub_for(link.lti_deployment)

Lti::Score.publish(link.lineitem_url, token, user_id: sub, score_given: given, score_maximum: 100)
true
rescue Lti::AccessToken::Error, Lti::Score::Error => e
  Rails.logger.error("Autograder::AgsPush failed for project #{project.id}: #{e.message}")
  false

Pick CircuitVerse content from your course page

Deep linking (#7772, #7777) lets you choose one of your circuits or assignments from inside the LMS, and the choice is posted back as a signed message.

Keep the class list in sync

Today a roster enters CircuitVerse once, at launch. A student who drops the course stays in the group. The new Sync Roster button (#8043) fetches the current roster, adds who joined, removes who left and records when it last worked. Only people who came from the LMS are removed. A student you added directly stays. I chose a button over an automatic sync so a flaky LMS shows up as a visible last synced time, not a silent failure.

GroupsController#sync_roster:

token = Lti::AccessToken.fetch(link.lti_deployment, [Lti::Membership::SCOPE], signing_key: Lti::KeyManager.private_key)
members = Lti::Membership.fetch(link.context_memberships_url, token)
ActiveRecord::Base.transaction do
  Lti::RosterImport.call(@group, members, link.lti_deployment)
  Lti::RosterDrop.call(@group, members, link.lti_deployment)
  @group.update!(last_synced_at: Time.current)
end

Write tests and let CircuitVerse grade submissions

You write a test suite in an editor on the assignment page (#7907). Each test case is a named set of input values and expected outputs, and you can mark it hidden. When a student submits, a background job runs the circuit and stores a result (#7942). Three settings give you control (#7926):

Partial credit: give a fraction of the score, or all or nothing
Maximum attempts: how many graded runs a student gets
Reveal test cases before submission: whether students see the cases up front

AutogradeJob:

def perform(project_id)
  project = Project.find_by(id: project_id)
  assignment = project&.assignment
  testbench = assignment&.testbench
  return unless testbench

  max = assignment.max_attempts
  return if max && GradingResult.where(project: project).count >= max

  begin
    result = Autograder::Runner.call(project: project, testbench: testbench)
    breakdown = result.groups
    score = (result.score * 100).round(2)
  rescue Autograder::Runner::RunnerError
    breakdown = []
    score = nil
  end

  GradingResult.create!(project: project, suite_snapshot: testbench.data,
                        breakdown: breakdown, score: score)
end

The score is turned into the scale your assignment uses (percent or letter), and an LMS-linked assignment is always a percentage, because that is what the gradebook expects (#7944).

GradeScaleMapper:

LETTER_THRESHOLDS = [[90, "A"], [80, "B"], [70, "C"], [60, "D"], [50, "E"]].freeze

def self.call(assignment, fraction)
  return unless mappable?(assignment)

  percent = (fraction * 100).round.clamp(0, 100)
  case scale_for(assignment)
  when "letter" then LETTER_THRESHOLDS.find { |min, _| percent >= min }&.last || "F"
  else percent.to_s
  end
end

def self.scale_for(assignment)
  assignment.lti_integrated? ? "percent" : assignment.grading_scale
end

Every result stores a copy of the suite it ran against. If you edit your tests later, grades already given still show the tests they were earned against, so you never see a grade next to a test it was not run on.

Split a class into teams

Instructors asked for teams, so a group page gets a Subgroups section (#8053). You create a team, then choose its members from the parent group's list. Each team gets a progress line such as 2 assignments, 5 submitted, 3 graded (#8058), and you can designate one team project that everyone in the team collaborates on (#8064). A REST API covers the same actions (#8056).

A subgroup is just a group with a parent, so everything that already works on groups (assignments, grades, mentors) works on a team. Creating a team does not send the "new group" email, so sorting a class into ten teams does not send ten emails. Teams do not show up in "my groups" next to your real classes.

Group:

scope :top_level, -> { where(parent_group_id: nil) }
validates :name, uniqueness: { scope: :parent_group_id, case_sensitive: false }, if: :subgroup?

Sharing the team project reuses Collaboration:

belongs_to :team_project, class_name: "Project", optional: true
validate :team_project_author_must_be_member, if: :team_project_id?

def share_team_project
  group_members.where.not(user_id: team_project.author_id).pluck(:user_id).each do |user_id|
    Collaboration.find_or_create_by!(project_id: team_project_id, user_id: user_id)
  end
end

Give every student the same starting circuit

Templates are circuits you can save, share and use as the base of an assignment (in review, #8065 to #8107). The flow:

Save as template from one of your projects. An assignment project cannot be a template, because it is a student's working copy.
Choose who can find it: private (only you), group (people who share a group with you, plus those groups' mentors) or public (everyone).
Browse the gallery, filtered by tag.
When you create an assignment, pick a base template. Each student's project starts from a copy when they open the assignment.
Fork a template to customise it, and see how many assignments use it.

A template can also carry its own test suite, and an assignment that uses it gets a copy of the tests, unless you have already written your own (#8092).

Visibility is enforced in one place, so what the gallery lists, what the picker offers and what the server accepts are always the same list. A template you cannot see returns a 404 if you guess its URL:

scope :templates_for, lambda { |user|
  return where(is_template: true).template_public if user.nil?

  groups = Group.where(id: GroupMember.where(user_id: user.id).select(:group_id))
                .or(Group.where(primary_mentor_id: user.id)).select(:id)
  peers = GroupMember.where(group_id: groups).select(:user_id)
  mentors = Group.where(id: groups).select(:primary_mentor_id)
  shared = template_group.where(author_id: peers).or(template_group.where(author_id: mentors))
  where(is_template: true).and(template_public.or(by(user.id)).or(shared))
}

If you are a student

You open the assignment from your course and you are signed in. No second account.
You start from your instructor's circuit. The copy is made when you open the assignment, so you get the template as it stands then. If its author made it private in the meantime, the copy is checked again, so you are never handed a circuit you should not have.
You see which tests passed. A results page shows your score and one row per test case. Hidden tests show only pass or fail: the inputs and expected outputs are not printed, so you cannot write a circuit that satisfies a list instead of solving the problem. If your instructor allows revealing, you see everything.
Only people who should see your result can. The page uses the same access rule as the project itself, which for an assignment means you, your instructor and your collaborators.
Teams share one circuit. If your group is split into teams, everyone in your team collaborates on the team project.

This is how the results page decides what to print. It reads only the saved copy of the suite, so rows always line up with their verdicts:

def visible_test_groups(result, reveal_hidden)
  snapshot = result.suite_snapshot["groups"] || []
  result.breakdown.each_with_index.flat_map do |group, i|
    hidden = snapshot.dig(i, "hidden") && !reveal_hidden
    Array(group["cases"]).each_with_index.map do |c, j|
      { "label"   => case_label(group["label"], j),
        "passed"  => c["passed"], "hidden" => hidden,
        "inputs"  => hidden ? nil : pin_values(snapshot.dig(i, "inputs"), j),
        "outputs" => hidden ? nil : pin_values(snapshot.dig(i, "outputs"), j) }
    end
  end
end

And this is the check that runs when a template is copied into your project:

def template_circuit_data
  return unless circuit_template
  return if circuit_template.project_access_type == "Private" && !template_author_mentors_group?

  circuit_template.project_datum&.data
end

If you run an LMS or a CircuitVerse install

Nothing turns on by itself. LTI 1.3 endpoints return 404 until an operator enables the lti_advantage flag. The launch creates user accounts, so I did not want that live the moment code merged. The autograde settings are also gated by a flag.
LTI 1.1 keeps working. The 1.3 code is built on ruby-jwt next to it, and the ims-lti dependency stays for the old path.
You register CircuitVerse the standard way. There is a platform registration model (LtiDeployment), a JWKS endpoint and a tool configuration document, all merged. The configuration includes Canvas placements, so instructors can find the tool.
Autograding needs a simulator service. The app calls a separate service to run a circuit, because the simulator lives in the browser and Rails cannot run it. You set SIMULATOR_RUNNER_URL and SIMULATOR_RUNNER_TOKEN. There are no defaults on purpose: a missing URL fails loudly instead of sending student circuits to localhost. That service is not in this repository yet, so autograding will not work until it exists.

SIMULATOR_RUNNER_URL=
SIMULATOR_RUNNER_TOKEN=

HTTP.timeout(**TIMEOUTS)
    .auth("Bearer #{ENV.fetch('SIMULATOR_RUNNER_TOKEN')}")
    .post(endpoint, json: { circuit: circuit, testbench: testbench.data })

What to expect

Timing.
- Expect features to appear gradually, not all at once.
- Nothing from LTI 1.3 will be visible to instructors until the sign-in pull request (#7746) merges and an operator enables the flag.
Things that behave as designed that are new.
- A template's visibility is separate from the project's access. A project can be public without being offered as a template.
- Collaborators cannot save someone else's circuit as a template. Only the author, an admin or a mentor of the author's group can.
- If you delete a template that an assignment uses, the assignment keeps working and simply loses the link.

How it was built, in short

Four habits shaped every feature, and they are the reason I trust the parts that are finished.

Small pull requests. I built everything in the form of small pull requests in order to make them simple to review and easier to understand at a later date by future contributors
Check access where the data moves. The biggest review findings were all the same question: someone is about to be given something, so was anyone checking whether they should have it at that moment? That is why a template is checked again at copy time, and why hidden test cases are redacted on the results page and not in the database.
Reuse before inventing. A test case is a group with n: 1, the simulator's own format. A team project reuses Collaboration. A subgroup is a Group with a parent.

Pull requests

Merged:

PR	What it does
#7647	Saves a grade locally before pushing it to the LMS
#7654	`LtiDeployment` model for platform registrations
#7659	JWKS and tool configuration endpoints
#7715	`Lti::JwtValidator` for `id_token` verification
#7763	`LtiResourceLink` for launch service endpoints
#7764	`Lti::AccessToken` for platform service calls

In review:

Area	PRs
LTI 1.3 sign-in and services	#7746, #7765, #7767, #7772, #7777, #7779, #7781
Autograding	#7836, #7906, #7907, #7926, #7940, #7942, #7944, #8003, #8029
Subgroups and roster	#8038, #8043, #8046, #8048, #8053, #8055, #8056, #8058, #8064
Templates	#8065, #8068, #8069, #8070, #8084, #8092, #8100, #8102, #8107

What comes next

Get the stack reviewed and merged in order, starting with LTI 1.3 sign-in.
Fix the known gaps above, beginning with removing a student from their teams when they leave the parent group.
Land the simulator service, so autograding can run for real.
A screen for authoring a test suite on a template.
A short guide for instructors and LMS admins moving from LTI 1.1 to 1.3.

If you teach with CircuitVerse and want to try any of this, or hit one of the gaps above, open an issue or comment on the pull request. Real classroom feedback is the most useful thing the stack can get now.

Acknowledgements

I would like to express my gratitude to my Org Admin Aboobacker M K and my mentors, Anas Khan and Aman Asrani, for the reviews that shaped this project.

Thank you to the CircuitVerse community and contributors for this wonderful platform.

Thank you to Google Summer of Code for the opportunity to code, learn and become a better developer.

Week 21: Access Has to Be Checked Where the Data Moves

Akanksha Trehun — Thu, 29 Oct 2026 15:49:21 +0000

Week 21: Access Has to Be Checked Where the Data Moves

Eleven pull requests opened this week: five for subgroups on the 19th and 20th, four for circuit templates on the 20th, and two more
for templates on the 23rd and the 25th. The subject changed halfway through. The review findings did not. Almost every comment, on both features,
was the same question in a different place: someone is about to be given something, and was anyone checking whether they should have
it at that moment?

The subgroup screen, with no new JavaScript

#8053 is the part an instructor actually touches. A top level group's page gets a subgroups section: a list, an Add Subgroup button,
and for each subgroup a Manage Members dialog with checkboxes of the parent's members and a Delete confirmation. It is 441 added
lines, and it adds no JavaScript, no routes and no controllers, because the actions already existed from the previous pull requests.

The two alternatives were a Stimulus controller with one shared dialog and a separate page for managing members. The shared dialog is
less markup and needs script to rewrite the form's target for each subgroup, which is more code to test. A separate page avoids dialogs
and adds a route, an action and a view for a short checkbox list. Rendering one dialog per subgroup, with ids like
manage-subgroup-<id>, uses Bootstrap's data-bs-toggle and data-bs-target and the styles the app already has.

The section is gated by manage_subgroups? in the view as well as the server, so a student never sees a control the server would
refuse. That is the habit I would defend: a button the server rejects is a UI bug, and the check in the view and the check in the
controller come from one policy method so they cannot disagree.

AccessLint found the one real defect in the markup: a <ul> whose direct children were <div>s, which breaks the list semantics a
screen reader relies on.

The user who left the class and kept the team

The security finding of the week came on that pull request and again on the progress one:

This filter checks IDs submitted to update_members, but it does not handle an existing subgroup member removed from the parent
group. That user retains the subgroup membership and can still pass GroupPolicy#show_access? for the subgroup.

A subgroup member is a GroupMember row on the subgroup. Removing someone from the parent group deletes the parent's row and nothing
else. The model has no callback that follows:

class GroupMember < ApplicationRecord
  belongs_to :group, counter_cache: true
  belongs_to :user
  after_commit :send_welcome_email, on: :create, unless: -> { group.subgroup? }
end

So a student dropped from the class, whether by a mentor or by the roster sync from last week, is still a member of a team inside it and
can still open its page and its shared project. I checked that against the branch rather than trusting the comment, and it is true. It
is not in this pull request and I am not going to pretend it is. The fix is its own change, one that removes the user's subgroup
memberships when the parent membership goes, and it has to cover the API destroy path too. Two ways to remove a member, and only
the one I thought of while writing the code was safe.

A missing parameter that deletes a roster

Subgroups got a JSON API on the 19th (#8056): list and create under a group, and show, delete and update members by the subgroup's
own id. Reads go through the group's existing show_access?. Writes authorize the parent with manage_subgroups?, and the members
endpoint calls the same Group#sync_subgroup_members as the web controller, so the two cannot drift. Index and create are nested
because they only make sense with a parent. Show, destroy and members use the subgroup id alone so a client that already holds one does
not have to know the parent.

The review found what happens when the request is sloppy:

params[:user_ids] is passed unfiltered to sync_subgroup_members. A request without the key removes every non-mentor member of the
subgroup. A request with a hash or nested value raises on to_i, because Hash#to_i does not exist and the request returns 500. Use
params.expect(user_ids: []) so the API returns 400 for a missing or malformed key.

"Set the members to this list" with a missing list is not a no-op. It is "set the members to nothing." An API call that forgets a field
had the same effect as one that meant to empty the team. The web controller does the same thing in one line,
Array(params[:user_ids]).map(&:to_i), so the bug is in both. Making the parameter required changes a silent destructive default into
a 400, which is the right trade for an endpoint whose whole job is replacing a list.

A smaller one on the web side, from the shared team project pull request: manage_subgroups? lets a parent group's co-mentor act, but
show_access? does not let them view a subgroup they are not a member of. A co-mentor could update the members and then be redirected
to a page they are not allowed to see. The redirect should stay on the parent page. Two policy methods that answer slightly different
questions will always eventually be used in the same sentence.

Progress and the team project

The progress rollup (#8058) puts a line next to each subgroup, such as "2 assignments, 5 submitted, 3 graded", so an instructor can see
how teams are doing without opening each one. It is a model method with three counts:

def progress
  submitted = Project.where(assignment_id: assignments.select(:id), project_submission: true)
  { assignments: assignments.count, submissions: submitted.count,
    graded: Grade.where(project_id: submitted.select(:id)).count }
end

I did not add cached counters. They can drift when a project is un-submitted or a grade is deleted, and they need callbacks and a
migration for numbers read on exactly one page. A bug on that page was a fair one: "1 assignments". The fix on the 20th is a plural
form in the locale file, not a conditional in the view.

The team project (#8064) lets a subgroup designate one project and have everyone in the subgroup collaborate on it. Rather than a new
table or a group level collaborator type, it reuses Collaboration:

belongs_to :team_project, class_name: "Project", optional: true
validate :team_project_author_must_be_member, if: :team_project_id?

def share_team_project
  group_members.where.not(user_id: team_project.author_id).pluck(:user_id).each do |user_id|
    Collaboration.find_or_create_by!(project_id: team_project_id, user_id: user_id)
  end
end

Reusing Collaboration means every existing project access rule applies unchanged, and the alternative, teaching the project policy about
groups, would have touched rules that matter far more than this feature. The validation says the project's author must be a member
of the subgroup. Without it, a mentor could attach someone else's private circuit to a team and share it with everyone in it, so the
rule is an access check disguised as data validation. find_or_create_by! makes sharing idempotent, so running it twice, as happens
when members are updated, does not duplicate collaborations. Deleting the project clears the link rather than failing or deleting the
subgroup, because the foreign key is on_delete: :nullify.

Templates: three places a circuit can be handed to someone

On the 20th the template stack went up, four small pull requests in a chain: a flag and a visibility enum on Project, a
save-as-template action, a gallery, and a link from an assignment to a template. The fifth, on the 23rd, is the one where a student
actually receives a copy. Every one of them is a different answer to "who may receive this circuit".

The rules start small. An assignment project cannot be a template, because it is a student's working copy and publishing it would
expose submitted work. Saving as a template is allowed for the author, an admin, and a mentor of a group the author belongs to, and
refused for collaborators, because a collaborator can edit a circuit without owning it. is_template and template_visibility are
separate from project_access_type: access decides who can view a project and visibility decides who can find and copy it, and a
project can be public without being offered.

That separation is exactly what the first review attacked, on the gallery:

If a private project is saved with template_visibility: "public", the gallery selects it, but check_view_access denies its detail
page to anonymous visitors. The save action permits this combination. Apply one access rule across template eligibility, the
gallery, and the detail page.

Two independent switches create a combination nobody designed. The template could be listed, with a card, and refuse to open. The
direction of the fix is a single rule that every path uses, rather than a patch for each page.

The copy that has to ask again

#8070 lets an instructor pick a base template for an assignment. It stores a reference, not a copy, and a controller check limits
the choice to templates the instructor could use, with the same Project.templates_for scope that fills the dropdown, so what the
form offers and what the server accepts are always the same list. CodeQL flagged the first version of the check as a SQL query built
from user input, which was fair, and the check became exists?(id: template_id), which lets ActiveRecord bind the value.

#8084 makes the link do something: when a student opens an assignment, their project starts from the template's circuit. It copies
only the circuit data rather than calling Project#fork, because a fork also notifies the template's author and attaches a preview
image, and a student opening an assignment is not a fork and a template with no preview would raise. It copies when the student
opens it, not when the instructor saves, so each student gets the template as it stands when they start and the instructor can still
fix it before anyone begins.

That choice created the review's best finding:

template_visibility: private alone does not make the underlying project private. The privacy issue is reachable when the author
changes the linked template's project_access_type to "Private". The assignment keeps its existing template link, and both
AssignmentsController#start and LtiController#create_project_if_student_present copy the linked template's current project data
without checking its current access.

Because the assignment stores a reference and the copy happens later, access has to be decided at the later time, and I had only
decided it at the earlier one. The instructor chose a legitimate template on Monday, its author made the project private on Tuesday,
and on Wednesday a student received its data. The copy now asks:

def template_circuit_data
  return unless circuit_template
  return if circuit_template.project_access_type == "Private" && !template_author_mentors_group?

  circuit_template.project_datum&.data
end

A private template is still copied when its author is a mentor of the assignment's group, since in that case the student is receiving
it from their own teacher, and refused otherwise. Both the web start action and the LTI launch call the same method, so there is one
place that answers.

A template that carries its own tests

The last pull request of the week, on the 25th, lets a template carry a test suite and hands it to the assignments that use it. It adds
a template_testbench column to projects and validates it by building a Testbench around it, so a suite on a template has to pass
the same shape, label and pin checks as any other suite instead of getting a second, weaker validator:

validate :template_testbench_is_runnable, if: :template_testbench?

def template_testbench_is_runnable
  suite = Testbench.new(assignment: Assignment.new, data: template_testbench)
  suite.validate
  errors.add(:template_testbench, suite.errors[:data].to_sentence) if suite.errors.key?(:data)
end

When an assignment's template changes, the suite is copied into a testbench, but only if the assignment does not already have one:

after_save :copy_template_testbench, if: :saved_change_to_circuit_template_id?

def copy_template_testbench
  suite = circuit_template&.template_testbench
  create_testbench!(data: suite) if suite.present? && testbench.nil?
end

An instructor who has already written their own tests keeps them. Nothing in the interface sets template_testbench yet, so this is
the plumbing for a later change, and I would rather say that than imply a feature.

Where the project stands

Subgroups can be created, filled, shown, progress-tracked, exposed through an API and given a shared project. One of the largest gaps,
a user removed from the parent keeping their team, is real and open, and so is the destructive empty parameter on the web path. Templates
can be saved, browsed, linked to an assignment and copied to a student, with access checked at the copy, and they can carry a
test suite that follows the link.

The lesson I would write on the wall is that a reference is a promise about later, and every promise about later needs a check at later.

Next week: template visibility stops being a label on a column and becomes a rule, and a template gets a fork button.

Week 20: A Pull Request That Cannot See Its Prerequisites

Akanksha Trehun — Thu, 29 Oct 2026 14:12:05 +0000

Week 20: A Pull Request That Cannot See Its Prerequisites

Eight pull requests moved this week, five of them new. The grading pipeline got its LMS leg, group rosters got a manual sync, and
subgroups started with a data layer, a listing scope and a policy. The thing that connects all of them is not a feature. It is one
sentence that showed up on #8003, #8043 and #8048, in slightly different words each time: the thing this code calls is not on
this branch. This post is about the two ways I answered it and which one I would not do again.

The sentence

CodeRabbit on the roster sync:

sync_roster reaches Lti::Membership::SCOPE. The reviewed head defines Lti::AccessToken, but it defines neither Lti::Membership
nor Lti::RosterImport or Lti::RosterDrop. The request can therefore raise NameError and return 500 without syncing.

On the subgroup controller:

On the current PR branch, Group has no subgroups association, and the schema has no parent_group_id. A create request therefore
fails at @group.subgroups.

And on the results page from last week, a human reviewer: the model and table it reads were absent, so an authorized request fails at
the lookup.

The reviewers are right every time, and none of them is wrong to be annoyed. Each branch is cut from master and only contains its own
commits, so a reviewer who checks out the branch sees code that calls something that does not exist. In my head these are links in a
chain. On GitHub each one is a separate pull request that has to make sense by itself.

Answer one: put the prerequisites in the branch

On the 12th I handled the two oldest ones by merging the branches they depend on into them. The editor got the testbench model merged
in, and the results page got the settings and the result model. A single commit message captures it: Merge branch 'feat/grading-result' into feat/grading-results-ui. The branches now run from a clean checkout.

That is the correct answer to a stack that is going to merge in order, and the price is the diff. A pull request that adds a results
page now shows 21 changed files, and most of them belong to other pull requests. I know which lines are mine. A reviewer has to
work it out. I would only call this a fix in the sense that the CI is green.

While I was in the editor I fixed a different finding that had been waiting, from a reviewer who read the JavaScript:

beforeSubmit() has already converted the deadline to iso when this blocks submission. fixing the pin mismatch and retrying
produces an invalid deadline; validate before converting it.

The submit handler rewrote the deadline field into an ISO string and then discovered a problem and refused the submit. The form came
back with the field already rewritten, so the second attempt posted a deadline in the wrong format. The rule is old and I broke it
anyway: validate first, mutate second, so a refusal leaves the form the way the user left it.

The same day, AutogradeJob stopped giving fractional credit when partial_credit is false, which is last week's finding finally
closed, and the results page started drawing one row per test case instead of one row per group. Neither needed a new idea. They needed
the review comment to be read slowly.

Answer two: make the code survive the absence

The roster sync and the subgroup controller got a different treatment, and I am less happy with it.

For the roster action I added a guard in front of the call:

return redirect_to(group_path(@group), alert: t(".not_available")) unless roster_services_loaded?

def roster_services_loaded?
  defined?(Lti::Membership) && defined?(Lti::RosterImport) && defined?(Lti::RosterDrop)
end

For subgroups, the equivalent on the 18th:

before_action :check_subgroups_available

def check_subgroups_available
  return if @group.respond_to?(:subgroups)

  redirect_to group_path(@group), alert: t("subgroups.not_available")
end

Both turn a 500 into a redirect with a message, and both make the branch pass a review that was about the branch. Neither is something
I would write if the prerequisite were on master. After the stack merges, each one is a permanent branch that can never be taken, a
test for a state production will never be in, and a translation key nobody will see. I wrote a feature detection for a feature that is
going to exist, because it was cheaper than answering a reviewer's question by changing the shape of the stack.

The honest accounting is that answer one costs the reviewer's attention and answer two costs the codebase's. A reviewer's attention is
spent once and the dead branch is paid for by everyone who reads that controller later. My rule for next time is that code which only
exists to survive its own pull request being early gets deleted in the pull request that makes it unnecessary, and that I write that
deletion down in the description so someone knows to look for it.

The LMS leg

#8029 is the last link in the grading pipeline: once a submission is graded and the assignment came from an LMS, post the score
back to the instructor's gradebook through the AGS line item. It went up on the 14th, and CodeRabbit found two real problems in it
that I had listed last week as unchecked assumptions.

The first was the rescue that returns nil:

If Lti::Score.publish fails, this rescue returns nil without reporting the failure. The caller cannot distinguish a failed
publish from a skipped push, so a transient LMS failure can leave the gradebook without the score.

I had written nil for both "nothing to push" and "the push failed", so no caller could tell a silent skip from a lost grade. It
returns a boolean now and logs the failure:

Lti::Score.publish(link.lineitem_url, token, user_id: sub, score_given: given, score_maximum: 100)
true
rescue Lti::AccessToken::Error, Lti::Score::Error => e
  Rails.logger.error("Autograder::AgsPush failed for project #{project.id}: #{e.message}")
  false

The second was the single users.lti_sub column:

LTI defines sub as unique within an issuer, not across issuers. If one user launches from two LMS issuers, users.lti_sub can
retain only one subject. Autograder::AgsPush can then send that subject to the other issuer's line item.

This one scared me more than the first, because it is a wrong grade going to the wrong place with no error. A subject identifier is
meaningful only together with the platform that issued it. The column became a separate identity per deployment, and the lookup takes
the deployment from the assignment's resource link:

sub = link && project.author&.lti_sub_for(link.lti_deployment)

A third finding, that a stored line item URL should be rejected unless it is HTTPS before the bearer token is sent to it, is the same
cleartext concern CodeRabbit raised on the runner in September. It is still open on the score service.

Rosters, and a migration that would have locked a table

#8043 adds a Sync Roster button on a group page. It fetches the current course roster from the LMS, adds the members who joined,
drops the ones who left and records when it last succeeded. Until now a roster entered CircuitVerse once, at launch, so a student who
dropped the course stayed in the group. I chose a button over syncing on every launch because a visible button with a visible "last
synced" time is easier to reason about when the LMS is flaky, and an automatic sync would hide failures and slow every launch.

The migration needed work. strong_migrations refused the foreign key on groups as one step, so the commit on the 17th split it into
a two step version that does not hold a long lock on the table. It is the shape I now reach for by default, so I will not belabour it.

Subgroups begin

The subgroup work started on the 16th and it begins with a choice I am glad I made early: a subgroup is just a Group with a
parent_group_id, not a new table. Everything already built for groups, such as members, assignments and the mentor checks, works on a
subgroup for free. The model keeps the rules in one place: one level of nesting, the subgroup takes the parent's mentor, names are
unique within a parent, and creating a subgroup does not send the "new group" email, because sorting a class into teams should not
trigger one email per team.

scope :top_level, -> { where(parent_group_id: nil) }
validates :name, uniqueness: { scope: :parent_group_id, case_sensitive: false }, if: :subgroup?

The consequence of that choice showed up the next day in #8046. A subgroup is a row in groups, so every "list my groups" query
returns it next to the real classes, on the dashboard and in the API. I filter in SQL with top_level at each listing, rather than
in the serializer, so pagination counts stay right and later listings get the behaviour from the scope instead of remembering to
filter. A reviewer found the hole in the inheritance rule:

The callback runs only on creation. If Api::V1::GroupsController#update changes a subgroup's primary_mentor_id, the subgroup no
longer has its parent's primary mentor.

Inheritance at creation is not an invariant. A rule that is supposed to always hold has to be enforced on update too, including when a
parent's mentor changes and its subgroups are left pointing at the old one. The commit on the 17th makes it hold past creation.

Where the project stands

The grading pipeline is complete as code, from test editor to gradebook. Subgroups have a data layer, a listing scope and a policy with
a controller. Two of the answers to "the prerequisite is missing" are in the code, and I would use only one of them again.

Next week: the subgroup UI, API and team project go up, and every review is about who is allowed to see what.

Week 19: What the Student Is Allowed to See

Akanksha Trehun — Thu, 29 Oct 2026 14:05:16 +0000

Week 19: What the Student Is Allowed to See

This was a one pull request week, and I want to say so plainly rather than pad it. #8003 opened on the 10th, a results page for a graded
submission, and the rest of the week went on the next piece of the pipeline, which I will get to at the end. The results page looks like
the simplest thing in the autograding project, a score and a table, and it contained the one design decision in the whole pipeline
where a mistake leaks something the instructor explicitly chose to hide.

The problem is not showing the result

A graded submission already has a stored GradingResult. Until this week it existed only in the database, so a student who was graded
had no way to find out. The new route renders the overall score and a table with one row per test case.

The harder half is that some test cases are hidden on purpose. An instructor marks a case hidden so students cannot work out the exact
inputs and outputs the grader expects and write a circuit that satisfies the list instead of the problem. If the page simply printed
every stored value, it would give that away. What a page may print and what the database knows are different sets, and the page is the
only thing standing between them.

Why the stored result is not enough on its own

The breakdown column on a result holds pass or fail for each case, by position, and nothing else. A page built only from it would say
"case 1: passed, case 2: failed" and a student would have no idea what either case tested. Showing the raw JSON had the same problem and
a worse one, since it is unreadable. To show something useful the page needs the input and output pin values, and those live in the
suite, not the result.

This is where last week's design earns its keep. A result stores a snapshot of the suite as it was when the run happened, so the two
halves come from the same moment and their positions line up. If the page read the live suite, an instructor who edited the suite after
grading would shift every row against its result, and a student would see one test's inputs next to another test's verdict. The helper
that joins them depends on that guarantee and says so by only ever reading the snapshot:

def visible_test_groups(result, reveal_hidden)
  snapshot = result.suite_snapshot["groups"] || []
  result.breakdown.each_with_index.flat_map do |group, i|
    hidden = snapshot.dig(i, "hidden") && !reveal_hidden
    Array(group["cases"]).each_with_index.map do |c, j|
      { "label"   => case_label(group["label"], j),
        "passed"  => c["passed"], "hidden" => hidden,
        "inputs"  => hidden ? nil : pin_values(snapshot.dig(i, "inputs"), j),
        "outputs" => hidden ? nil : pin_values(snapshot.dig(i, "outputs"), j) }
    end
  end
end

For a hidden case the pins come back nil, so the view has nothing to print. I put the hiding in the helper rather than in the model on
purpose. GradingResult records what happened and has no opinion about who is looking, and an instructor who opens the same page with
the reveal setting on gets the full table from the same code.

Who can open the page

The action is a member of the project resource, guarded by the same check_view_access that protects the project itself:

before_action :check_view_access, only: %i[show create_fork change_stars grading_result]

I did not add a new permission, and it is worth being careful about why that is safe. The grading page reuses the rule for viewing the
project, so whoever can open the submission can open its result. For an assignment project that is a short list, because a project that
belongs to an assignment is not allowed to be public, which the model validation enforces. The audience is the author, the assignment's
mentors and any collaborators. A new rule would have been a second place for the access logic to drift. The cost of reusing the rule is
that the page's privacy is only as good as the project's. The case that matters most is a hidden test case with the reveal setting off,
and the controller spec checks it in both directions: the pin values are absent from the page when reveal is off and present when it is
on.

The next piece, started

The other thing I did this week was begin the last link in the chain. Once a submission is graded, the instructor's gradebook should
hear about it when the assignment came from an LMS. Autograder::AgsPush takes a project and a score, finds the assignment's LTI
resource link and the author's LTI subject, fetches a token and posts the score to the line item:

def self.call(project:, score:)
  link = project.assignment&.lti_resource_link
  sub = project.author&.lti_sub
  return unless link&.lineitem_url.present? && sub.present?

  token = Lti::AccessToken.fetch(link.lti_deployment, [Lti::Score::SCOPE], signing_key: Lti::KeyManager.private_key)
  Lti::Score.publish(link.lineitem_url, token, user_id: sub, score_given: score.round.clamp(0, 100), score_maximum: 100)
rescue Lti::AccessToken::Error, Lti::Score::Error
  nil
end

It mirrors the existing token and score split so grading code only deals with a project and a score. It also assumes a few things I
had not checked. One of them is stored in a single column, and one of them is a rescue that returns nil. Both of them get questioned next week.

Where the project stands

The student side of the loop exists: a graded submission has a page, the page shows each case, and what an instructor hid stays hidden.
It also depends on two other pull requests, one for the result model and one for the reveal setting, and I opened it before deciding
how to deal with that. The instructor side, pushing the grade out to an LMS, is written and has not met a reviewer.

Next week: the page gets its first review, the LMS side gets its first review, and the same prerequisite comment turns up on three more
pull requests.

Week 18: Four Small Pieces and the Seams Between Them

Akanksha Trehun — Thu, 29 Oct 2026 14:04:15 +0000

Week 18: Four Small Pieces and the Seams Between Them

Four pull requests opened this week, on the 28th, 29th, 29th and 30th, and none of them is more than about a hundred lines. Autograde settings, a
GradingResult model, AutogradeJob, and GradeScaleMapper. Each one is easy to review on its own, and almost every review comment
was about the gap between two of them rather than about the code inside one. A small piece is easy to get right. The seam between two
small pieces belongs to nobody until someone writes it down.

Settings that nothing reads

#7926 adds three per assignment settings to the form: allow partial credit, a maximum number of attempts, and whether to reveal test
cases before submission. Three columns, a form block, 82 added lines. The reviewer's comment was the one I should have written myself:

these three settings get saved but nothing reads them yet (not on master, not in #7836/#7906/#7907), so an instructor can set max
attempts to 1 and students still get unlimited runs. i'd add each column in the pr that enforces it, or hide this block until then.

A control that saves and does nothing is worse than a missing control. An instructor sets attempts to one, sees "saved", and believes
something is being enforced. I split the settings out to keep the job pull request small, and the split quietly shipped a promise before
the thing that keeps it.

I did not want to fold the columns back into the job's pull request, because that undoes the point of splitting. The commit that followed
gates the whole block behind a Flipper flag, so the form does not show it unless autograding is switched on for the
deployment. That keeps the pieces separate and keeps the promise honest: until the enforcing pull requests merge, nobody can make it.
The alternative of adding each column in the pull request that reads it was the reviewer's first suggestion and is the better long term
rule. It would have meant re-cutting the stack, and I would rather say that than pretend the flag is the same thing.

A default that defeats its own constraint

GradingResult stores the outcome of one grading attempt: a copy of the suite it ran against, the per test breakdown, and a score
validated to sit between 0 and 100 when present. The reason for the copy is that an instructor can edit a suite after students have been
graded, and a result that only points at the live suite would show a grade next to a test it was never run against.

CodeRabbit found a flaw in the migration that I would not have found by testing through the model:

if a database insert omits suite_snapshot, this default stores {} instead of rejecting the result. Rails insert_all skips model
validations, so such a result can be persisted without the suite context needed to interpret it after the live suite changes.

I had written the column as null: false, default: {}, and the default is exactly what lets a bad row in.
Validations live in the model and insert_all does not call the model, so the only guard that applies to every write path is the one in
the database, and default: {} switches that guard off. Dropping the default means an insert that forgets the snapshot fails loudly at
the database, which is the whole reason the column exists.

The job, and the four questions CodeRabbit asked it

AutogradeJob is the piece that finally connects submission to grade. It is triggered after the deadline job forks a student's project
into a submission, and only when the assignment has a testbench. It runs the runner and stores a GradingResult:

def perform(project_id)
  project = Project.find_by(id: project_id)
  assignment = project&.assignment
  testbench = assignment&.testbench
  return unless testbench

  max = assignment.max_attempts
  return if max && GradingResult.where(project: project).count >= max

  begin
    result = Autograder::Runner.call(project: project, testbench: testbench)
    breakdown = result.groups
    score = (result.score * 100).round(2)
  rescue Autograder::Runner::RunnerError
    breakdown = []
    score = nil
  end

  GradingResult.create!(project: project, suite_snapshot: testbench.data,
                        breakdown: breakdown, score: score)
end

I chose a background job over running the grade in the request because the runner call is a network request with real latency and
timeouts, and a submission should not wait on it. The review found four different ways the seams in this short method fail.

It enqueues before the data exists. The deadline job calls this from inside assignment.with_lock. With default settings Rails
enqueues immediately, so a Sidekiq worker can start before the transaction commits, Project.find_by returns nothing, and the first
line quietly returns. The submission is never graded and nothing says why. The fix is to enqueue after commit, either through
enqueue_after_transaction_commit on the job or an after commit callback.

An outage uses up an attempt. If the runner is down, the rescue stores a result with a nil score. With max_attempts at one,
that row is counted by the next run's limit check, so a student who submitted once and hit an outage can never be graded. A failed run
is not an attempt the student made.

It can run twice. Sidekiq can execute the same job more than once. Two executions can both pass the count check before either has
written a row, so one submission ends up with two results and exceeds its attempt limit. The count and the write are not one atomic
step.

It ignores a setting I had just added. The reviewer noticed that with partial_credit: false from #7926, one passing case out of
two still stores 50 percent. The job was written before the setting existed, and neither pull request mentions the other. The fix
landed in a follow up commit on the 12th, a few lines that turn a partial score into all or nothing.

Two of those four are problems with the job. The other two, the outage and the partial credit rule, are problems with what the job
assumes the neighbouring pieces mean. That ratio is the point of this week.

Making the raw number mean something

AutogradeJob stores a fraction between 0 and 1, and a stored 0.75 does not say whether the assignment is graded on a percentage, a
letter scale or not at all. GradeScaleMapper converts it:

LETTER_THRESHOLDS = [[90, "A"], [80, "B"], [70, "C"], [60, "D"], [50, "E"]].freeze

def self.call(assignment, fraction)
  return unless mappable?(assignment)

  percent = (fraction * 100).round.clamp(0, 100)
  case scale_for(assignment)
  when "letter" then LETTER_THRESHOLDS.find { |min, _| percent >= min }&.last || "F"
  else percent.to_s
  end
end

def self.scale_for(assignment)
  assignment.lti_integrated? ? "percent" : assignment.grading_scale
end

An LTI integrated assignment is always treated as a percentage, whatever its grading_scale says, because that is the constraint the
LMS side already imposes. A grade that would have to be a letter in one system and a number in the other is a bug waiting for a
gradebook. no_scale returns nil, and the job checks GradeScaleMapper.mappable? before it runs the runner at all, since there is
no point spending a simulator call on an assignment that will not be graded.

I considered storing the mapped grade on the result row instead of computing it. I kept the raw fraction because the scale is an
assignment setting that an instructor can change, and the stored data should not go stale when they do.

The runner's fixes

The two configuration findings from last week landed on the 29th. The default URL is gone, so a missing SIMULATOR_RUNNER_URL raises
KeyError instead of posting to localhost, and the request now carries a bearer token from SIMULATOR_RUNNER_TOKEN:

HTTP.timeout(**TIMEOUTS)
    .auth("Bearer #{ENV.fetch('SIMULATOR_RUNNER_TOKEN')}")
    .post(endpoint, json: { circuit: circuit, testbench: testbench.data })

The response validation findings, the null body and the string "false", are still open.

Where the project stands

The pipeline now exists end to end as code: a suite is authored, a submission triggers a job, the job calls the runner, the result is
stored with the suite it ran against, and the score can be expressed on the scale the assignment uses. It does not yet exist as
something I would trust, because the seams I listed are mostly still open: enqueue after commit, outage versus attempt, idempotency.
Those are not hard fixes. They are the fixes you only find by reading the code next to its neighbours.

Next week: a student sees the result, and the first question is how much of the suite they are allowed to see.

Week 17: A Client That Fails Loud

Akanksha Trehun — Thu, 29 Oct 2026 14:03:14 +0000

Week 17: A Client That Fails Loud

Two pull requests opened this week, the runner on the 23rd and the test case
editor on the 24th, and between them they taught the same lesson from opposite
directions. The runner taught me that a client for a service nobody has built
yet has to be stricter than a client for a service that exists, because
nothing downstream will catch what it lets through. The editor taught me that
a pull request which leans on an unmerged one fails in CI before a reviewer
reads a single line of it.

What the runner is, and what it refuses to be

Autograder::Runner is 159 lines across three files and it does one thing:
it posts a student's circuit and a test suite to an external simulator
service and turns the answer into a pass count. It does not touch the
database. It does not decide what a score means. It is the messenger between
Rails and the thing that can actually simulate a circuit, because the
simulator lives entirely in the browser and Rails has no way to run a circuit
by itself.

I considered a headless browser per submission and dropped it, since booting
a full frontend to grade one circuit is too heavy to do on every run. A
background queue was the other option I set aside. The job that calls the
runner is its own pull request, so the runner stays a plain object that
either returns a result or raises.

class RunnerError < StandardError; end
TIMEOUTS = { connect: 5, write: 10, read: 60 }.freeze

def run
  response = HTTP.timeout(**TIMEOUTS)
                 .post(endpoint, json: { circuit: circuit, testbench: testbench.data })
  raise RunnerError, "Runner returned #{response.status}" unless response.status.success?

  JSON.parse(response.body.to_s)
rescue HTTP::Error => e
  raise RunnerError, "Runner unreachable: #{e.message}"
rescue JSON::ParserError
  raise RunnerError, "Runner returned invalid JSON"
end

Every failure path ends in one exception class, so a caller has one thing to
rescue. That was the design. The reviews were about the places where it
quietly was not true.

A response that is valid JSON is not a valid response

The sharpest finding came from CodeRabbit, on how the answer gets scored:

a successful response containing null raises NoMethodError instead of
RunnerError. a response containing {"groups":[{}]} returns a zero
score, while a case with "passed":"false" counts as passed.

I had checked that the body parsed and that groups was a non empty array,
and stopped there. The scoring line counts a case as passed when
kase["passed"] is truthy, and in Ruby the string "false" is truthy. A
runner that serialised its booleans as strings would have turned every failing
case into a passing one and handed a student full marks. The finding's last
sentence is the rule I am taking from it: malformed responses must not become
grades.

The same review pointed at the timeouts. connect, write and read are
per operation limits. A service that keeps trickling bytes just inside the
sixty second read window can hold a Sidekiq worker for as long as it likes,
because nothing bounds the whole request. Per operation timeouts and a total
deadline are different protections and I had only written the first kind.

The fallback that would have posted student work to localhost

A human reviewer found the one I am least comfortable with. The endpoint was
built like this:

"#{ENV.fetch('SIMULATOR_RUNNER_URL', DEFAULT_URL)}/run"

with DEFAULT_URL set to the loopback address the development runner uses.
The comment on it:

with this fallback a prod box missing SIMULATOR_RUNNER_URL silently posts
student circuits to its own localhost and every run just shows as
"unreachable".

That is a configuration mistake that produces no error you would look at. The
symptom is an outage that looks like the simulator being down, and the cost is
student circuits going to a port on the same machine. A default is helpful in
development and dangerous in production, and .env.example already covers
development. The fix is to drop the default so a missing variable raises
KeyError on the first call instead of on the first complaint.

The same reviewer asked the next question before I did: the request carries no
authentication, so anything that can reach the runner can make it simulate
arbitrary circuits, and is a shared secret planned for when the service
lands? It was not planned. It is now, as a bearer token read from the
environment in the same strict way. CodeRabbit followed up with the logical
conclusion: a bearer token sent over plain http:// to a non loopback host is
sent in cleartext along with the circuit and the suite, so non loopback URLs
should have to be HTTPS.

The contract only one side has seen

The most useful comment was not about any line:

the response shape (groups[].cases[].passed) is only defined by this
client and the stubbed spec, there's no runner service in the repo or an
open pr yet. i'd land this with or after the service so the contract comes
from the side that actually simulates.

I wrote the specs against a stub that I also wrote, which means the test and
the code agree with each other and with nothing else. If the real service
names a field differently, every spec still passes and the first real run
fails. I cannot fix that inside this pull request. The honest options are to land
the runner together with the service, or to say plainly in the description
that the response shape is a proposal, and the second is cheaper to do this
week. It also means the strict response checks above matter more, because
they are the only thing standing between a wrong shape and a wrong grade.

One more review item belongs to the specs: they stubbed only the localhost
URL, so setting SIMULATOR_RUNNER_URL in a test environment would make the
request miss WebMock and fail. The endpoint in the spec should be derived from
the same configuration as the service.

The pull request that failed before anyone read it

The editor went up on the 24th as #7907, 523 lines over 13 files, and CI
failed with Missing model class Testbench. The cause was plain
once I looked. The editor edits a Testbench, and the model that defines it
is a separate pull request that was still open, so on a branch cut from master
the class does not exist. A reviewer's comment said the same thing and added
the second half: both pull requests declare the same has_one :testbench
line, so they would conflict the moment either merged.

The fix is the one the stack already implied, which is to build the editor on
top of the model's branch and keep a single declaration:

has_one :testbench, dependent: :destroy, autosave: true

I had split the work into small pull requests to keep each one reviewable,
and I had not paid for the split. A small pull request that cannot run alone
is not small for the reviewer, because they have to reconstruct what it
depends on. Stacking is honest about that and cutting from master hides it.
That cost is going to come back.

Where the project stands

The runner's configuration and authentication findings are settled in
principle: no silent default, a shared secret, one exception class. The
response validation findings are the open half, and they are the ones that
decide whether a grade can be trusted. The editor is stacked on the model it
depends on.

Next week: the runner's fixes land, and the three pieces that turn its answer
into a stored, scaled, rate limited grade start going up.

Week 16: The Editor That Learned When to Say No

Akanksha Trehun — Thu, 01 Oct 2026 12:29:03 +0000

Week 16: The Editor That Learned When to Say No

With the testbench model and the runner both fixed and moving, the test case
editor finally got its own review round this week. It found the same class
of bug the model found last week, in the client side JSON this time instead
of the server side validation, and fixing it properly meant redesigning a
piece of the editor rather than patching around the symptom. A second,
unrelated bug turned up in a completely different pull request, caught only
because a setting this phase just added made an old code path reachable for
the first time.

The overwrite that would have erased a title on every save

First finding, on Assignment#testbench_data=:

(testbench || build_testbench).data = parsed

This replaces the entire data hash with whatever the editor's JSON posts.
The editor only ever authors type and groups. If an existing suite had
any other top level key, a title, say, something the simulator itself reads
at testbench.js:872, it survives right up until the next time an instructor
edits an unrelated setting and saves the form. Then it is gone, not because
anyone touched it, but because the write path never preserved what it was not
explicitly given.

tb = testbench || build_testbench
tb.data = (tb.data || {}).merge(parsed)

Merge instead of replace. I wrote a regression test that seeds a suite with a
title, saves an unrelated change through the editor, and asserts the title
survives, then deleted the merge to confirm the test catches the loss. It
does.

The same binding bug, one layer up

The second finding was the model's groups[0] binding problem again, except
this time the mismatched pins would come from the editor itself, from an
instructor adding a second test case row with different inputs than the
first:

each row becomes its own group, but the simulator binds pins from
groups[0] only. a row with a different pin set than the first makes
group.outputs.find(...).results.push return undefined and throw.

Same root cause as last week, different code path producing the malformed
suite. The fix is a shape comparison in the JavaScript controller, mirroring
what the Ruby model already does server side:

sameShape(groups) {
    const shape = (group) => JSON.stringify(['inputs', 'outputs'].map((side) => group[side].map((p) => [p.label, p.bitWidth])));
    return groups.every((group) => shape(group) === shape(groups[0]));
}

Submission blocks client side now if any row's pins do not match the first
row's, with the same pattern used for incomplete rows: preventDefault,
stopPropagation because rails-ujs still disables the submit button on a
merely prevented submit as it bubbles to its document level listener, show
the error, stop.

A third finding, a values regex that accepted any non-empty string instead
of exact-width binary digits, was the exact bug from last week's model
review, just not yet enforced in the form that produces the JSON in the
first place. Tightened the same way: [01]+, checked against the declared
bit width.

Why I did not try to merge the unsupported case in

The harder finding was about suites the editor cannot fully represent:
multi vector test cases, where one row of the JSON actually encodes several
values per pin instead of one. My first fix tried to preserve these by
tracking each group's original array position, splitting edited groups from
untouched ones, and re-merging them back into the original order on save.
It worked, but CodeRabbit found two more bugs in that merge logic within
a day: the merge could still reorder an unsupported group relative to an
edited one, and it silently rewrote the suite's type from seq to comb
on every save because the serialization always hardcoded comb.

Two fixes for a problem that a different design does not have in the first
place made me stop and reconsider the approach rather than the code. The
editor cannot safely round trip a multi vector suite. Trying to merge it
back together precisely is solving a harder problem than the feature needs:
tracking position, preserving type, reconciling two sources of truth. The
simpler rule is to never let the editor touch what it cannot fully
represent. If any group in a suite has more than one value per pin, the
whole editor goes read only for that assignment, with a note saying why,
instead of rendering an editable interface that would corrupt the suite the
moment someone saved through it.

def test_case_editable?(assignment)
  test_case_groups(assignment).all? do |group|
    (group["n"] || 1) == 1 &&
      (group["inputs"] || []).all? { |pin| Array(pin["values"]).size <= 1 } &&
      (group["outputs"] || []).all? { |pin| Array(pin["values"]).size <= 1 }
  end
end

This deleted the position tracking, the merge-back logic, and every test
written against it, and replaced all of it with one predicate and an
if/else in the view. The line count dropped by about sixty lines for a
bigger guarantee: nothing this editor cannot represent can be silently
damaged by it, because it is never editable in the first place.

Getting back under budget without cutting a fix

This phase holds pull requests to roughly two hundred hand written lines.
Chasing every one of these findings, plus the accessibility fixes an
automated check flagged separately, pushed this diff past four hundred at
its peak. Cutting it back down meant finding the difference between a line
that is load bearing and a line that is just verbose.

Consolidating five near identical validation specs into one table driven
test recovered real lines without losing a single case. Dropping a helper
spec file for a follow up PR, per this phase's own stated rule about
splitting spec edge cases rather than thinning validation, recovered more.
The read only redesign recovered the most of all, because it deleted code
that existed to solve a problem a different design does not have. What did
not move: the testbench model's shape and label validation, the values
regex, the malformed pin token check. Those stayed exactly as strict as the
reviews asked for, because a line count is a budget for how a feature is
built, not a ceiling on whether a reported bug gets fixed.

The bug that was always possible and just became reachable

A different pull request from this same run, the one adding autograde
settings like max attempts, got a review comment that had nothing to do with
any of the new columns:

when a mentor submits max_attempts: 0, the new validation rejects
@assignment.update. the action renders :edit, but
after_action :check_reopening_status still runs. if the assignment has a
forked submission, Assignment#check_reopening_status can move its
assignment association and destroy the fork despite the failed update.

after_action callbacks run regardless of whether the action they follow
succeeded. This one always had that property. It just never mattered before,
because almost nothing about updating an assignment used to fail validation.
Adding a max_attempts column with an actual numericality check gave
mentors, for the first time, an easy way to submit a rejected update, which
is the one condition this pre-existing callback was never guarded against.

def check_reopening_status
  return if @assignment.errors.present?

  @assignment.check_reopening_status
end

Two lines. I reverted them and reran the new regression test to confirm a
rejected update really does destroy a student's forked submission without
the guard, which it does, then restored the fix and confirmed it survives.
The bug was not introduced this phase. The conditions for triggering it
were.

Where the project stands

Three pull requests carry review fixes now instead of open findings: the
testbench model, the runner, and the test case editor, the last one smaller
and safer than when it went up. The autograde settings pull request gained
an unrelated but real fix along the way. Underneath all of it, the actual
grading pipeline, a result model that snapshots the suite it ran against
rather than pointing at the live one, a job that calls the runner and
respects max attempts, a mapper from raw score to whatever scale the
assignment actually uses, and a results page that redacts hidden test cases
unless an instructor has chosen to reveal them, went up this week as well.

Next week: watching whether any of these five pull requests actually merge,
and what the first one to land changes for everything still stacked behind
it.

Week 15: What the Simulator Actually Binds

Akanksha Trehun — Thu, 01 Oct 2026 12:26:33 +0000

Week 15: What the Simulator Actually Binds

#7746 merged clean, rebase and all, closing out last week's story. That
freed the week for the thing I actually promised: picking the test case
editor back up. Except the editor cannot be evaluated on its own terms until
its two dependencies, the testbench model (#7836) and the runner (#7906),
are actually mergeable, and both came back from review this week with
findings that needed fixing before either could move.

A label that is not a string

CodeRabbit's first flag on the testbench model was on the signal validator:

def validate_signal(signal, cases)
  return errors.add(:data, "every signal needs a label") unless signal.is_a?(Hash) && signal["label"].present?
  ...

present? rejects nil, empty strings, and whitespace-only strings. It does
not reject 1. A JSON body with "label": 1 sails through this check, gets
stored, and later reaches the simulator, which calls .trim() on every
signal's label while binding pins to the circuit. Numbers do not have a
.trim() method in JavaScript. The failure is not "this test case behaves
oddly," it is a TypeError that kills the run.

The fix is one clause:

return errors.add(:data, "every signal needs a label") unless signal.is_a?(Hash) && signal["label"].is_a?(String) && signal["label"].present?

Small, but it is the kind of small that only shows up if you trace the value
forward into the file that actually consumes it, rather than stopping at "the
model validates its own shape."

The binding a human reviewer caught that a linter would not

The second finding was not from CodeRabbit. A reviewer left this on the same
file:

nothing checks that later groups use the same input/output labels as
groups[0]. the simulator binds pins from groups[0] only
(testbench.js:818-823) and then does inputs[label].state and
group.outputs.find(...).results.push per group, so a suite that passes
this validator can throw a TypeError mid-run.

I went and read testbench.js before touching anything, because the claim is
specific enough to verify rather than trust. It is exactly right:

function bindIO(data, scope) {
    const inputs = {};
    data.groups[0].inputs.forEach((dataInput) => {
        inputs[dataInput.label.trim()] = scope.Input.find(...);
    });
    ...
}

Only data.groups[0] ever gets read here. Every other group is bound
against whatever bindIO built from the first one. If group two declares an
input labeled c that group one never had, inputs['c'] is undefined, and
setInputValues dies trying to set .state on it. The model's own
validation, one group at a time, had no way to catch this, because the bug is
not in any single group, it is in the relationship between groups.

The fix compares each group after the first against the first's shape:

def validate_group(group, reference)
  ...
  next unless reference

  shape = ->(list) { list.map { |s| [s["label"], s["bitWidth"]] } }
  next if shape.call(signals) == shape.call(reference[side])

  errors.add(:data, "#{side} must match the first group's signals")
end

Values that are the right length and the wrong alphabet

Same reviewer, same file, a second finding: values were only length checked
against the case count, never checked for actually being binary digits. The
simulator feeds inputs through parseInt(v, 2) and compares outputs against
dec2bin(value, bitWidth) as strings. [0, 1] as integers, "2" on a one
bit pin, all of it passes the old check and then fails every single case at
run time, silently, which reads to a student as "I got every case wrong"
rather than "the test data itself is malformed."

def valid_values?(values, cases, width)
  values.is_a?(Array) && values.size == cases &&
    values.all? { |v| v.is_a?(String) && width.is_a?(Integer) && v.match?(/\A[01]{#{width}}\z/) }
end

Exact width, exact alphabet, exact type. I added a regression case for a
numeric label, one for wrong-width values, and one for a second group with a
mismatched pin set, then reverted each fix in turn to confirm its matching
test actually fails without it. All three did. A test that cannot fail is
not a test, it is a comment that happens to run.

The runner's two findings were a different flavor entirely

#7906, the runner, got two findings that had nothing to do with the shape
of any data structure:

with this fallback a prod box missing SIMULATOR_RUNNER_URL silently posts
student circuits to its own localhost and every run just shows as
"unreachable".

def endpoint
  "#{ENV.fetch('SIMULATOR_RUNNER_URL', DEFAULT_URL)}/run"
end

The default was meant for local development convenience and quietly doubled
as a production misconfiguration that fails silent instead of loud. Dropping
the fallback entirely, ENV.fetch('SIMULATOR_RUNNER_URL') with no second
argument, means a missing variable raises immediately instead of routing
every grading attempt to a service that was never there.

The second finding was that the request carries no authentication at all,
so anything that can reach the runner can make it simulate arbitrary
circuits. I added a bearer token, ENV.fetch('SIMULATOR_RUNNER_TOKEN'), sent
on every request. The runner service on the other end still needs to
actually check it, which is a different pull request's job, but the caller
no longer has an excuse not to send one.

Two different ways of being right

The testbench findings came from reading JavaScript nobody asked the model
to know about. The runner findings came from reading the word "fallback" and
asking what happens when the thing it falls back to is wrong. Neither is
something a type checker or a schema validator would have caught on its
own; both needed someone to actually trace a value from where it is written
to where it is read. That is becoming the throughline of this whole review
cycle: static analysis catches shapes, a reader who follows the data catches
contracts.

Where the project stands

Both the testbench model and the runner have their review fixes pushed, all
cases verified failing without their fix and passing with it. The test case
editor is still sitting locally, unstacked, waiting for these two to actually
merge before its own compare page can show one clean commit instead of three
borrowed ones.

Next week: the editor's own review round, which turns out to have found the
same class of bug in the client side JSON that this week found in the model,
plus one unrelated bug that only became reachable because of a setting this
phase never had before.

Week 14: The Approved Pull Request That Stopped Passing

Akanksha Trehun — Tue, 08 Sep 2026 17:38:43 +0000

#7746, the OIDC login
initiation, has been approved and waiting since week 10. This week I went back
to actually merge it and found its test suite failing on an error that had
nothing to do with anything I remembered writing.

A NameError in code I had already shipped

CI was red across the whole oidc_login context, ten examples, all the same
failure:

NameError:
  uninitialized constant LtiController::LTI_STATE_PURPOSE

That should have been impossible. oidc_login and lti_state_verifier both
reference LTI_STATE_PURPOSE, and the request spec references
LtiController::LTI_STATE_PURPOSE directly to verify the state it gets back.
If that constant had never existed, this whole context should have failed the
day it was written, back in week 10. It did not. Something had removed a
constant that used to be there.

Tracing it

git log -p on the controller file gave a clean timeline. The constants first
appear on 5 August, in the commit that adds oidc_login itself, with a comment
explaining why the state is signed rather than kept in the session. They survive
the review fixes from week 10. They even survive a rough merge from master on 9
August that briefly left the file unparseable, a splice that dropped the closing
end of oidc_login and spliced two of master's filter lines into the method
body. The very next commit repairs that, and its diff still shows the constants
sitting untouched at the top of the class, comment and all.

The actual disappearance happened somewhere I was not looking. On 21 August, a
different contributor merged a repository wide change to master, adding
after_action :verify_authorized safety nets across dozens of controllers. On
this controller, on master's copy of the file, that meant inserting
skip_after_action :verify_authorized as the very first line inside the class
body. Master's copy at that point had no LTI_STATE_PURPOSE constants, because
none of this branch's work had merged into master yet.

On 5 September I merged master into my branch again, to catch up before opening
the merge. Both sides had inserted a new line in exactly the same spot, right
after class LtiController < ApplicationController: master's
skip_after_action, and my branch's constants with their comment. Diffing the
file immediately before and after that merge shows exactly what happened:

 class LtiController < ApplicationController
-  # The state is signed rather than kept in the session: the launch returns as a
-  # cross-site POST that a SameSite cookie would not survive, and the signature
-  # is what proves the launch answers an initiation we made.
-  LTI_STATE_PURPOSE = "lti.launch.state"
-  LTI_STATE_TTL = 5.minutes
+  skip_after_action :verify_authorized

No conflict markers survive in the final commit, which is the part that makes
this easy to miss. Two branches touching the same handful of lines usually forces
a manual conflict resolution, and somewhere in resolving that one I kept
master's line and let mine go, without noticing the branch had put something in
that exact spot too. A merge that finishes clean does not look like something
that needs a second look, which is exactly why this one went unnoticed for two
weeks.

The fix

Both constants, reinstated:

LTI_STATE_PURPOSE = "lti_oidc_state"
LTI_STATE_TTL = 5.minutes

The purpose string itself does not need to match the original literal, nothing
outside this file ever reads it, only the two internal call sites and the spec
need to agree on the same constant, which they now do by definition. The TTL
does need to stay at five minutes, since the spec travels six minutes forward
and checks the state is gone by then. I confirmed nothing else in the codebase
referenced either constant, ran the project's RuboCop config against the change
to be sure the addition was clean, and pushed. CI went from ten failures back to
green across every job, not just the one that had been failing.

What this says about "approved and waiting"

An approved pull request sitting unmerged is not a pull request that is done.
It is a pull request whose last green check run is quietly getting further from
the truth every time master moves underneath it, and nothing about the GitHub UI
makes that visible. The checkmark from three weeks ago looks exactly like a
checkmark from this morning.

This is the same lesson as week 10's development database drift, just at a
different layer. State that nobody actively re-verifies accumulates damage that
stays invisible until something finally runs against it again. There, it was a
migration nobody had rolled back. Here, it was a merge nobody had rerun. Both
times the fix was cheap once found and the finding was the entire cost.

The practical rule I am taking from this: before merging anything that has sat
approved for more than a couple of weeks, rebase it against current master and
rerun the suite first, rather than trusting a check that ran against a branch
state that no longer exists.

Where the project stands

#7746 is rebased, fixed, and green again, waiting on nothing but the merge
button. #7764, the access token exchange, is still open from last week. The
testbench model and runner from two weeks ago are both under review. The test
case editor is still finished and waiting locally.

Next week: actually watching the login initiation merge, and picking the editor
back up now that it has an access token to eventually authenticate against.

Week 13: A Kid Only the Platform Can Read

Akanksha Trehun — Tue, 08 Sep 2026 17:29:36 +0000

The access token work that jumped the queue last week went up as
#7764. It took a
security minded review to catch a bug that would have made every single token
request fail, and a second reviewer's confusion, arriving weeks later, to remind
me that a review thread can point at code that no longer exists.

What the tool needs before it can call anything

Every LTI Advantage service call, posting a grade, reading a roster, is
authenticated the same way. The tool signs a JWT assertion with its own private
key, POSTs it to the platform's token endpoint under the OAuth 2.0 client
credentials grant, and gets back a bearer token scoped to whatever it asked for.
None of that is CircuitVerse specific. It is 1EdTech's Security Framework
section 5.4.1 built on top of RFC 7523, the JWT profile for client assertions,
which itself sits on RFC 6749's client credentials grant.

Tokens are cached per deployment and per scope set, refreshed thirty seconds
before they expire rather than exactly at expiry, so a request landing in that
window does not race a token that is technically still valid when read and
expired by the time it reaches the platform. A blank or non string
access_token in the response is refused instead of cached, and a response with
no expires_in at all gets no token stored, rather than a guessed default
lifetime. Both of those came from reading the spec's failure modes rather than
just its happy path.

The bug a reviewer caught before merge

The first version signed the assertion like this:

JWT.encode(
  { iss: deployment.client_id, sub: deployment.client_id, ... },
  signing_key, "RS256", { kid: JWT::JWK.new(signing_key).kid }
)

A reviewer flagged the kid within a day. Lti::KeyManager, the class that
publishes CircuitVerse's public key at /lti/jwks, derives its key id from a
JSON::JWK thumbprint. JWT::JWK.new(signing_key).kid computes a kid using a
different digest entirely. The two never agree, which means the assertion would
have named a key that does not match anything the platform can find at our own
published JWKS endpoint. Every token request would fail, not occasionally, every
time, because the mismatch is structural rather than a bug that only shows up
under some inputs.

The fix reuses the exact computation KeyManager already does:

def kid_for(signing_key)
  JSON::JWK.new(signing_key.public_key).thumbprint
end

Same input, the public key, same digest, JSON::JWK's thumbprint. I checked
this by rereading KeyManager#public_jwk line by line rather than trusting that
two thumbprint calls with similar names must produce the same output, and they
do:

def public_jwk
  jwk = JSON::JWK.new(private_key.public_key)
  jwk[:kid] ||= jwk.thumbprint
  jwk.merge(use: "sig", alg: "RS256")
end

A comment that arrived three weeks late, pointed at code that no longer existed

The fix shipped within a day of the review comment. Three weeks later, a second
reviewer reread the thread from the top and asked why I was "still" using
JWT::JWK.new(signing_key), pointing straight at the original code block.

GitHub pins a review comment to the diff position it was written against. Once
that hunk changes, the thread gets marked "Outdated," but the stale code stays
visible inline right where the comment is anchored, and the label is easy to
miss if you are reading a long thread quickly rather than checking the Files
Changed tab. The question was reasonable. The premise, that the flagged line
was still in the branch, was not.

The reply pointed at the fix commit by hash, quoted the current kid_for
method, and reread KeyManager's source alongside it to show the two
computations are the same, rather than just asserting "it's fixed, trust me."
That habit, rederiving a claim instead of restating it, is the same one from
week 9's JWT validator work: a test that passes for the wrong reason is worse
than no test, and an explanation that only restates the conclusion is worse than
one that shows the work.

A question with a better answer than "I just know"

The same reviewer asked, on a different thread, how I knew which fields
belonged in the token request body at all, and whether it was written down
anywhere or just something I had picked up.

It is written down, in three places that stack on each other. grant_type and
scope come from RFC 6749 section 4.4, the client credentials grant itself.
client_assertion and client_assertion_type come from RFC 7523 section 2.2,
the general mechanism for authenticating with a signed JWT instead of a shared
secret. The specific URI value platforms expect for
client_assertion_type, and the requirement that the JWT's iss and sub both
equal the tool's own client id, come from the IMS Security Framework, which is
the layer that actually ties LTI Advantage to that OAuth mechanism.

I answered with the three documents and the section numbers rather than a
summary of what they say, because a summary is something to remember and a
citation is something to reopen. Anyone auditing this code later, including me
in six months, needs the second thing.

Where the project stands

The access token pull request is open with both review threads addressed, the
kid bug fixed and explained, and the spec provenance question answered with
citations rather than recollection. Roster sync and grade passback can now, once
this merges, actually authenticate the calls they have been assuming all along.

Next week: going back to merge an older, already approved pull request, and
finding out it had stopped passing while nobody was looking at it.

Week 12: Two Small Pull Requests

Akanksha Trehun — Tue, 08 Sep 2026 17:26:16 +0000

Both pieces of the autograding groundwork from last week's "pipeline nobody could
feed" post finally went up for review. Getting them out the door taught me more
about the packaging rule I set for myself than the underlying feature did, and
along the way I found a gap in my own merged code that rhymes uncomfortably with
the one I found last week.

The rule I made myself follow

Somewhere back in this phase I decided to open small pull requests one at a time
and hold dependent branches locally until the parent merges, instead of stacking
them or folding a dependency into one oversized diff. I have rejected my own
compare pages twice for showing more than one commit, which is a strange thing to
do to yourself but a useful one.

The test case editor is the clearest reason why. Its branch holds a cherry pick
of the testbench model's commit plus the editor's own commit, because the editor
needs the model's table to exist to run its tests at all. Opening it now would
show two commits on the compare page, and a reviewer would have to work out which
one is actually new. So it stays local until the model merges, at which point I
rebase, the borrowed commit drops out, and what is left is one clean diff over
the editor's own files.

Which one first, and why it wasn't obvious

That left a smaller decision: open the testbench model or the runner first.

The runner's spec is deliberately built so it does not need the model to exist.
It signs its double with Struct.new(:data) instead of pulling in the real
Testbench factory, so the runner is green against master on its own. Nothing
about it forces the model to land first.

I opened the model first anyway, for three reasons that all point the same way.
The test case editor is stacked on the model, not the runner, so unblocking the
editor means merging the model first regardless. The model is also the smaller,
easier diff: one migration, one class, one spec, no external HTTP contract to
explain to a reviewer. And both branches touch db/schema.rb, which is exactly
the kind of file you do not want two open pull requests editing at once, since
whichever merges second has to rebase past a version bump it did not cause.

What actually lives in the Testbench model

One row per assignment, JSON in a jsonb column, validated in three layers that
mirror the shape of the data itself: suite, then group, then signal.

def data_is_runnable
  return errors.add(:data, "must be a testbench object") unless data.is_a?(Hash)

  errors.add(:data, "type must be comb or seq") unless TYPES.include?(data["type"])
  return errors.add(:data, "must define a group") if groups.empty?

  groups.each { |group| validate_group(group) }
end

Each group needs a case count and at least one input and one output. Each signal
needs a label to bind to a circuit pin, a positive bit width, and exactly as many
values as its group's case count. That last check is why validation lives in
Ruby instead of a database constraint. A jsonb column schema check can enforce
shape, but it cannot easily say "this array's length must equal that other
field's value," and that cross-field relationship is most of what makes a
testbench actually runnable rather than merely well formed.

assignment_id is unique at both levels, a database index and a model
validation, because a race on create could otherwise slip a second row past a
Ruby only check. TYPES is closed to comb and seq, the two suite types the
simulator can actually execute, so a suite claiming verilog fails at save time
instead of failing later, mid grade, in front of a student.

The runner, and the decision that already happened

Most of the runner's design was decided last week, before a line of it existed.
This week was writing it down.

def self.call(project:, testbench:)
  new(project, testbench).call
end

One entrypoint. It posts the project's circuit JSON and the testbench's suite
JSON to a sidecar service over HTTP, with timeouts split into connect, write,
and read rather than one blanket number, because a hung connection should fail
fast while an actual simulation run legitimately needs more time than the
handshake. Every way this can go wrong, a bad status, an unreachable host, a
body that is not JSON, a response with no groups, collapses into one
Autograder::Runner::RunnerError, so the grading job that calls this later only
ever needs to rescue one thing.

Result = Struct.new(:passed, :total, :groups) do
  def score
    total.zero? ? 0.0 : passed.fdiv(total)
  end
end

The raw groups array rides along inside the result even though only score
matters right now, because the results page a few pull requests from now needs
to show which specific case failed, not just the fraction. The endpoint's host
comes from SIMULATOR_RUNNER_URL, following the same undocumented environment
variable pattern the codebase already uses for YOSYS_PATH, so anyone who has
configured one external tool location here already knows how to configure this
one.

What "next" turned out to mean

Last week's post ended by promising the editor would land next. It has not, and
writing the runner's pull request description is what changed my mind.

I went back to reread the roster sync pull request from two weeks ago, the one
that calls the platform's Names and Roles service, to check whether it needed
anything from this new access token machinery. It does. So does grade passback.
Both of those already assume an Authorization: Bearer header carrying a token
scoped to the right service, and until this week nothing in the codebase could
produce one. I had built two pull requests that read from and write to platform
services that require an OAuth exchange neither of them performs.

This is the same shape of gap as last week's missing test case editor: a feature
description satisfied on paper by code that quietly assumes a piece nobody has
built yet. The difference is I caught this one myself, by rereading my own
merged work with the new piece in mind, rather than having a mentor's evaluation
catch it for me.

Where the project stands

The testbench model and the runner are both open for review, in that order.
Roster sync and grade passback are merged but cannot actually authenticate
against a platform yet. The test case editor is finished and waiting locally for
the model to merge.

Next week: the access token exchange that both of those merged pull requests
have been silently depending on.

Week 11: Welcome to my Midterm evaluation

Akanksha Trehun — Sun, 16 Aug 2026 06:06:06 +0000

Midterm evaluations ran from the 10th to the 14th. Deep linking and roster sync both
shipped, two more pull requests merged, and then I sat down to write an honest account of
where the project stands — which is how I found out that the plan I have been working
from for a month builds a machine nobody can switch on.

Deep linking, end to end

Two pull requests, two days apart.

#7772 handles the request
side: a launch arrives with LtiDeepLinkingRequest as its message type instead of
LtiResourceLinkRequest, and everything downstream has to route differently. The
platform sends along deep_linking_settings — where to return the response, whether
multiple items are allowed, whether the platform wants a title back — and that has to be
stashed for the response, which happens on a completely separate request.

#7777 is what the instructor
actually sees: a picker listing their circuits and assignments, with the selection posted
back to the LMS as a signed JWT.

The satisfying part is that neither pull request needed anything new from the protocol
layer. The JWT signing is the key manager from week 4. The validation on the way in is
the validator from week 9. Deep linking is a different conversation with the platform,
but it is the same handshake underneath, and by this point the handshake is just there.

The roster

#7779 reads a course roster
over the Names and Roles service — pagination via Link headers, roles parsed out of the
IMS role URIs. #7781 turns
that into group members.

The interesting decision in the second one is how users get matched. Not on email — that
was week 5's lesson, and the reason for it holds here too: a platform's privacy settings
may mean no email arrives at all, and even when one does, it might belong to a different
CircuitVerse account than the person clicking. The match is on sub plus issuer, the
same identity pair the launch signs the user in with. A roster import that mapped on
email could silently add a stranger to a course.

Members created this way get marked as LTI-synced, which matters for the drop-handling
pull request that follows: when someone disappears from the LMS roster, only synced
members get deactivated. Somebody who joined the CircuitVerse group directly should not
be removed because they were never in Canvas to begin with.

Two merges

#7715, the JWT validator,
merged on the 11th. #7763,
the resource link model, merged on the 12th. Both had been sitting approved for a while —
the validator since the end of July.

Five merged now: the 1.1 passback fix, the deployment model, the JWKS and tool
configuration endpoints, the validator, and the resource link. Eight more open and
waiting.

The evaluation

What the evaluation actually asked me to do was describe the state of the project, which
sounds trivial and was not. Writing "here is what works" forces a different question than
"here is what I built this week," and the two answers were not the same shape.

So I went back to the three issues this project exists to close and read them line by
line against what is in the repository. Not the proposal — the issues, as filed.

A pipeline nobody could feed

The autograding phase of my plan had six pull requests: a test case model, a headless
runner wrapper, a grading result, a job that runs on submission, a results page, and a
push to the LMS gradebook. Student submits, autograder runs, grade lands in Canvas.

Every one of those is about executing test cases. Not one of them is about creating
test cases.

The issue says it plainly — "a test case editor in the assignment settings UI where
instructors can add, edit and delete test cases" — and I had somehow read that as
satisfied by a database table. It is not. A model with no UI means the only way to define
a test suite is the Rails console, which is not a feature an instructor can use. I had
planned six pull requests that would ship a grading pipeline with no way to put anything
into it.

Reading further with that lens, four more gaps fell out. There was no visible/hidden flag
on test cases, which both issues ask for. There was nothing for the autograde
configuration — partial credit, allowed attempts, whether students see the cases before
submitting. "Versioned with the assignment" had no plan behind it at all, and my model
actively worked against it: one row per assignment, edited in place, so changing a suite
would quietly invalidate every grade already produced from it.

And the least visible one, which I only found by reading the Grade model rather than
remembering it: nothing in the plan mapped a score onto an assignment's grading scale.
The autograder produces a fraction. Grade validates against letter, percent, or
custom, and rejects no_scale outright. Autograding an assignment with no grading
scale would have failed at the last step, after the run, with the student watching.

Twenty-four becomes twenty-seven

Week 7's post was called "Twenty-Four Pull Requests." That plan is now twenty-seven, all
three additions in the autograding phase: the test case editor, the autograde settings,
and the grading-scale mapping — plus reworking the grading result to carry an attempt
number and a snapshot, and folding hidden-case redaction into the results page.

The snapshot is how "versioned with the assignment" gets solved, and it is the change I
am happiest with. Rather than a version table with all the machinery that implies, each
grading result stores a copy of the suite it ran against. Edit the suite afterwards and
old results stay reproducible, because the thing that produced them is still sitting
right there next to them.

Three of the nine are written. The model and the runner are pushed; the editor is done
and waiting locally, because it needs the model's table to exist and I would rather open
one clean pull request after the first merges than one showing two commits.

Reusing a format instead of inventing one

The nicest thing I wrote this week is four words long.

A test case needs a name, input pin values, expected output pin values, and a hidden
flag. The simulator's testbench format already has groups — a named collection of cases
with input and output signals. My first instinct was to add fields: a name array, a
hidden array, indices lining up across them.

Then: a test case is just a group with n: 1.

One case, one name, one hidden boolean. It validates against the model I had already
written without changing a line of it, it invents no schema, and when the suite reaches
the simulator it is a format the simulator already knows how to run. The editor
serialises its table into exactly that on submit.

I have written the other version of this before — the one with parallel arrays and a
comment explaining how the indices correspond. It works right up until someone deletes a
row from one array and not the others.

Not reimplementing the simulator

The runner wrapper was the piece I was most nervous about, and the design question was
where the simulation actually happens.

The tempting answer was Ruby: run the circuit server-side, in the language the rest of
the grading code is written in. I talked myself out of it in about ten minutes. It means
reimplementing thousands of lines of simulation logic, and then maintaining two engines
that will drift — and the failure mode of that drift is a student's circuit passing in
the browser and failing at grading. For a grading tool there is no worse bug.

Driving a real browser was the next candidate, and capybara-playwright-driver is
already in the Gemfile, so it looked nearly free. A browser process per submission is not
nearly free.

Shelling out to Node was the closest call, because there is precedent: the Yosys
integration in this repository does exactly that. I went with an HTTP call to a sidecar
service instead, mirroring the other half of that same integration — a URL from the
environment, the same timeout-and-post shape the simulator controller already uses. It
keeps the engine on the frontend repository's release cadence, where that code and the
people who know it live.

Every failure collapses into one error class: bad circuit data, a non-2xx, a timeout, a
non-JSON body, an empty result. That is deliberate. The only caller is a background job,
and there is exactly one right answer to all of them — mark the run failed, leave the
student's grade alone.

Where the project stands

Five merged, eight open, three written and waiting. The protocol layer is done and
merged. Grade passback, deep linking, and roster sync are all in the queue. Autograding is
started.

The plan grew by three pull requests this week, which on a burndown chart is a bad week.
It is the most useful thing I did. I have spent eleven weeks getting good at making
mechanisms work, and the evaluation caught me having built most of a system that no
instructor could have used, because I kept reading a feature request as a description of
a machine instead of a description of a person trying to get something done.

Next week: the editor lands, and then the settings and the grading result behind it.