LTI 1.3, Autograding, Subgroups and Templates for CircuitVerse | GSoC 2026 | Final Report
TL;DR: I set out to move CircuitVerse's LMS integration from LTI 1.1 to LTI 1.3 and to add four things on top of it: grade passback, deep linking, roster sync and autograding. Over 22 weeks I opened 44 pull requests for the project. Six are merged, including the whole base of the LTI 1.3 handshake. The rest are small, stacked pull requests waiting for review. The most useful thing I learned is not about LTI. It is that a change nobody can review does not exist, however well it works.
About me and the project
I am magic-peach on GitHub. My project is called LTI 1.3 upgrade, grade passback and autograding for CircuitVerse, an online platform where students and teachers build and simulate digital circuits.
LTI is the standard that lets a tool like CircuitVerse live inside a learning system like Canvas or Moodle. A student clicks an assignment in their course, lands inside CircuitVerse already signed in, and their grade flows back to the gradebook. CircuitVerse spoke LTI 1.1, which signs a launch with a shared secret (OAuth 1.0). Canvas has deprecated that for new installations. LTI 1.3 replaces it with OpenID Connect and signed JWTs.
On 25 May I broke the proposal into four issues so that the work could be discussed where it happens:
- #7405: upgrade the LMS integration from LTI 1.1 to LTI 1.3
- #7406: roster sync, so a class list flows from the LMS into a CircuitVerse group
- #7407: grade passback through Assignment and Grade Services
- #7408: deep linking, so an instructor can embed CircuitVerse content in a course page
Along the way the project grew two more areas that came from conversations during the work: subgroups (teams inside a group) and circuit templates. I explain why below.
How the 22 weeks went
The plan I wrote in the proposal did not survive contact with code review. The project had four phases, and they were not the four I planned.
| Phase | Weeks | What happened |
|---|---|---|
| Community bonding | 0 | About 40 small pull requests before the start, then four feature issues |
| Learning to make small changes | 1–8 | A 2,000 line proof of concept was closed, I rebuilt LTI 1.3 from the spec, then sent one tiny pull request that merged |
| The protocol and the services | 9–15 | JWT validation, OIDC login, access tokens, deep linking and roster sync, cut into small pull requests |
| Autograding, subgroups and templates | 16–22 | A test suite editor, a grading pipeline, an LMS score push, team groups and a template library |
The shape of the project is easier to see through three turning points.
Turning point 1: the pull request nobody could review (weeks 2–3)
At the start of week 2 I opened #7488, a proof of concept for the whole of LTI 1.3: migrations, a JWT validator, an OIDC login endpoint, a rewritten launch, key management, a JWKS endpoint and tests. It worked. I had a real Canvas container talking to a local CircuitVerse and a real user landing signed in.
It was closed the next day.
My first reaction was that I had explained it badly. Then I tried to review my own diff as a maintainer. The change signed people in based on a token from an external server and created accounts automatically. To approve it, a reviewer would have to hold all of that in their head and be sure nothing was wrong, because the failure is somebody signing in as somebody else. I could not have reviewed it myself, and I had written it. The size was the problem, and no description fixes that.
Week 3 had no commits at all. It was the week I changed how I work.
Turning point 2: twenty-four small pull requests (weeks 4–11)
The rewrite started from the spec instead of from example code. I tried the lti-advantage 0.1.0 gem first (#7510), found it gave me wrappers around things ruby-jwt already does and none of the hard parts, and dropped it. The plan became 24 small pull requests that each do one thing and can be reviewed in about ten minutes. Week 7's review of the first one found a security bug I had looked straight past, which I describe below.
Turning point 3: the midterm showed I had built a machine nobody could switch on (week 11)
For the midterm evaluation I went back to the issues as they were filed and read them against the repository. My autograding plan had six pull requests, all about running test cases. Not one was about creating test cases. A model with no editor meant the only way to define a suite was the Rails console. I also found that nothing mapped a score onto an assignment's grading scale, so grading an assignment with no scale would have failed at the last step with the student watching. The plan grew from 24 to 27 pull requests, all three additions in the autograding phase. It is a bad week on a burndown chart and the most useful thing I did all summer.
Part 1: LTI 1.3 base
This is the part that is merged.
Signing the launch correctly
A 1.3 launch is only trustworthy if three separate checks pass. In the proof of concept all of it was one function. Splitting them and naming them is what made each one testable.
- State is a value the tool creates when login starts and checks when the launch comes back. It is the CSRF defence of the handshake.
- Nonce ties the returned token to this login attempt, so a captured token cannot be replayed.
- Deployment binding ties the token to the platform that started the login. A token can have a valid signature and still come from a different institution's Canvas.
Deployment binding is the one I had missed completely. It is the kind of hole you do not find by testing, because every test you write by hand uses a single platform.
Identity is not an email address
The proof of concept signed users in with User.find_by(email: payload["email"]). That works in every test you would write and is wrong. The token's signature proves the platform sent it. It does not prove the platform verified the email inside it, and a user can often edit their own email in the LMS. Anyone who can get an LMS to emit a launch with maintainer@circuitverse.org would be signed in as that maintainer.
The fix keys identity on the sub claim, scoped to the deployment it came from:
User.find_or_create_by!(provider: "lti", uid: "#{deployment.id}:#{payload['sub']}")
The scope matters as much as the claim. Two institutions can both have a user with sub of 12345.
Signing the state instead of weakening the cookie
When the platform posts the id_token back, it is a cross-site POST. Lax cookies are not sent on those, so the session was empty and the launch failed. My first fix was SameSite=None on the session cookie. It worked, and it weakened CSRF protection for every page in the app to serve one endpoint.
The better fix came from asking what the requirement actually was. The state does not need to be stored. It needs to be recognisable when it comes back, so I made it a signed token that carries its own contents:
state = lti_state_verifier.generate(
{ "nonce" => nonce, "deployment_id" => deployment.id },
purpose: LTI_STATE_PURPOSE,
expires_in: LTI_STATE_TTL
)
No session, no table, no cleanup job, and the cookie policy stays untouched. The purpose argument scopes the signature to this one use.
The JWT validator
#7715 added Lti::JwtValidator. It has one public method and no knowledge of controllers or sessions:
Lti::JwtValidator.validate!(token, deployment:, nonce:)
Most of its tests are attacks: a token signed by the wrong key, alg: none, an expired token, the wrong issuer or audience, a replayed nonce, and algorithm confusion. The last one is the subtle one. If a validator reads the algorithm from the token header, an attacker can sign a token with HS256 using the platform's public key as the HMAC secret, and the validator accepts it. The defence is never to read the algorithm from the token:
encoded_token.verify_signature!(algorithm: "RS256", key: key)
The validator also fetches the platform's key by kid from its JWKS endpoint. A missing kid triggers a refetch, because the platform may have rotated its keys.
OIDC login and an argument with CodeQL
#7746 adds /lti/login, the first leg of the handshake. CodeQL flagged a skip_before_action :verify_authenticity_token. My first reaction was that it was a false positive, since an LMS cannot hold a token from our forms. The better reading was that the scanner was pointing at something real: the skip switches the check off for every request shape. I overrode the predicate instead, so only a well-formed LTI initiation counts as verified:
def verified_request?
super || (action_name == "oidc_login" && params[:iss].present? && params[:login_hint].present?)
end
I removed the override to make sure the cross-site spec failed with InvalidAuthenticityToken without it, because a test that passes for the wrong reason is worse than no test.
Keeping it switched off
The 1.3 launch creates user accounts. A default-off flag means every 1.3 endpoint returns 404 until an operator turns it on, so the code can be merged and reviewed without running in production.
Calling the platform back
#7764 adds Lti::AccessToken, which signs a client assertion with the tool's private key and exchanges it for a bearer token. Tokens are cached per deployment and per scope set and refreshed 30 seconds early. A reviewer caught a bug that would have made every token request fail, before it merged.
Services on top
Deep linking (#7772, #7777), the course roster (#7779, #7781), line items (#7765) and score publishing (#7767) are each their own small pull request. None of them needed anything new from the protocol layer. A deep linking request is a different conversation with the platform, but it is the same handshake underneath.
The roster import matches users on sub plus issuer, never on email, for the same reason as above. Members created this way are marked as LTI-synced, so when someone disappears from the LMS roster only synced members are removed. Someone who joined the group directly should not be removed for never having been in Canvas.
The first merge
The first thing of mine to reach master was #7647, a small fix to the existing LTI 1.1 code. It makes sure a grade is saved locally before it is pushed to the LMS, so the gradebook and the database cannot disagree when a local save fails. I picked it because it was small and helped the integration that already has real users.
A reviewer found a bug in it that I had looked straight past. I had made the grading context get written after the launch signature was verified. But the clearing of stale context ran in a before_action, before verification. Anyone could post garbage to /lti/launch, get a correct 401, and still wipe a teacher's grading session on the way out. The rule I took from it: when you fix ordering around a security check, audit every operation on that path, not just the one you came for. The invariant is not "the write happens after verification". It is "nothing that changes state happens before verification".
Part 2: Autograding
The goal was: a student submits, the circuit is run against the instructor's tests, and the grade lands in the gradebook.
One suite format, no new schema
A test case needs a name, input values, expected outputs and a hidden flag. The simulator already has a testbench format with groups, each a named set of cases with input and output signals. My first idea was parallel arrays. Then I realised a test case is just a group with n: 1. That validates against the model I had already written, and it is a format the simulator already knows how to run.
#7836 adds the Testbench model that stores and validates it, and #7907 adds the editor.
Not rebuilding the simulator
The tempting answer was to run the circuit in Ruby. That means maintaining two simulation engines that will drift, and for a grading tool the worst possible bug is a circuit that passes in the browser and fails at grading. I also ruled out a browser per submission. Autograder::Runner (#7906) is a plain object that posts the circuit and suite to a simulator service and turns the answer into a pass count. Every failure ends in one exception class:
class RunnerError < StandardError; end
TIMEOUTS = { connect: 5, write: 10, read: 60 }.freeze
def run
response = HTTP.timeout(**TIMEOUTS)
.post(endpoint, json: { circuit: circuit, testbench: testbench.data })
raise RunnerError, "Runner returned #{response.status}" unless response.status.success?
JSON.parse(response.body.to_s)
rescue HTTP::Error => e
raise RunnerError, "Runner unreachable: #{e.message}"
rescue JSON::ParserError
raise RunnerError, "Runner returned invalid JSON"
end
Reviews made it stricter. A default URL of localhost would have posted student circuits to the same machine if an environment variable were missing, so the default is gone and the call raises KeyError. The request now carries a bearer token.
The pieces
| Piece | PR | What it does |
|---|---|---|
| Settings | #7926 | Partial credit, max attempts, reveal cases before submission |
GradingResult |
#7940 | Stores the score, the breakdown and a copy of the suite it ran against |
AutogradeJob |
#7942 | Runs after a submission is forked, stores a result |
GradeScaleMapper |
#7944 | Turns a 0 to 1 fraction into the assignment's scale |
| Student view | #8003 | A page showing the score and each case |
| LMS push | #8029 | Sends the score to the gradebook |
Two ideas ran through all of them.
Store a snapshot of the suite. An instructor can edit a suite after students are graded. A result that only points at the live suite would show a grade next to a test it was never run against. Each GradingResult stores the suite as it was, so old results stay reproducible, and the results page reads only the snapshot so rows line up with their verdicts.
Reviews are about the seams. Each piece is under about a hundred lines. Almost every review comment was about the gap between two pieces. A column default of {} defeated its own NOT NULL constraint, because insert_all skips model validations and the default let a row without a snapshot in. The job could be enqueued before the transaction committed, find nothing, and quietly return. A runner outage counted as one of a student's limited attempts. The settings saved values nothing read yet, so an instructor could set attempts to one and students still got unlimited runs. That last one I answered by gating the whole block behind a Flipper flag.
What a student is allowed to see
Some test cases are hidden on purpose, so students cannot write a circuit that satisfies the list instead of the problem. The only thing between the database and the page is the code that decides what to print. I put it in a helper, not the model, so GradingResult records what happened and has no opinion about who is looking:
def visible_test_groups(result, reveal_hidden)
snapshot = result.suite_snapshot["groups"] || []
result.breakdown.each_with_index.flat_map do |group, i|
hidden = snapshot.dig(i, "hidden") && !reveal_hidden
Array(group["cases"]).each_with_index.map do |c, j|
{ "label" => case_label(group["label"], j),
"passed" => c["passed"], "hidden" => hidden,
"inputs" => hidden ? nil : pin_values(snapshot.dig(i, "inputs"), j),
"outputs" => hidden ? nil : pin_values(snapshot.dig(i, "outputs"), j) }
end
end
end
The page reuses check_view_access, so whoever can open the submission can open its result, and a controller spec checks the hidden pins are absent when reveal is off and present when it is on.
Pushing the grade to the LMS
#8029 posts a graded score back through the line item. Two review findings mattered most. First, my rescue returned nil both for "nothing to push" and "the push failed", so a lost grade looked like a skip. It now returns a boolean and logs the failure. Second, I stored one lti_sub per user, but LTI defines sub as unique only within an issuer. A user in two LMSs could have a score sent to the wrong one. The lookup now takes the deployment from the assignment's resource link:
sub = link && project.author&.lti_sub_for(link.lti_deployment)
Part 3: Subgroups
A class is often split into teams. The design decision I would defend is that a subgroup is a Group with a parent_group_id, not a new model. Assignments, membership, grades, notifications and policies are all written against Group, and a new model would have forced a copy of every one.
I first built this in week 5 and opened it as #7632. It was finished: data layer, UI, controller, API and tests. I closed it myself in week 6. It had the same problem as my proof of concept, a large change to a core model arriving at once while the maintainers' attention was on LTI. The framing that helped was treating review capacity as a budget you spend. In October I rebuilt it as small pull requests:
| Part | PR |
|---|---|
| Data layer | #8038 |
| Hide subgroups from dashboards and the groups API | #8046 |
| Policy and controller | #8048 |
| Member management and group page UI | #8053 |
| REST API v1 | #8056 |
| Progress rollup | #8058 |
| Shared team project | #8064 |
The model keeps the rules in one place: one level of nesting, the subgroup takes the parent's primary mentor, names are unique within a parent, and creating one does not send the "new group" email, since sorting a class into teams should not send an email per team.
scope :top_level, -> { where(parent_group_id: nil) }
validates :name, uniqueness: { scope: :parent_group_id, case_sensitive: false }, if: :subgroup?
Because a subgroup is a row in groups, every "my groups" query would return it beside the real classes. I filter with top_level in SQL rather than in the serializer, so pagination counts stay right.
The UI (#8053) adds no JavaScript, routes or controllers. Each subgroup gets its own Bootstrap modal for managing members. The section is gated by manage_subgroups? in the view as well as the server, so a student never sees a button the server would refuse.
The team project (#8064) lets a subgroup designate one project that all its members collaborate on. It reuses Collaboration, so every existing project access rule applies unchanged:
belongs_to :team_project, class_name: "Project", optional: true
validate :team_project_author_must_be_member, if: :team_project_id?
def share_team_project
group_members.where.not(user_id: team_project.author_id).pluck(:user_id).each do |user_id|
Collaboration.find_or_create_by!(project_id: team_project_id, user_id: user_id)
end
end
The validation is an access check disguised as data validation. Without it a mentor could attach someone else's private circuit to a team and share it with everyone in it.
What the reviews found
Almost every review finding on this feature was the same question in a different place: someone is about to be given something, and was anyone checking whether they should have it at that moment?
- A user removed from the parent group keeps their subgroup membership, because removing the parent row deletes nothing else. That is a real gap and it is still open.
- An API call without
user_idsmeant "set the members to nothing" and removed everyone. The API now rejects a missing key withparams.expect(user_ids: []). The web controller has the same shape and is still open. - Inheriting the parent's mentor only on creation is not an invariant. It has to hold on update too.
Part 4: Circuit templates
Instructors kept asking for a starting circuit that every student gets. I built this as a stack of small pull requests, each a different answer to "who may receive this circuit".
| Part | PR |
|---|---|
is_template flag and template_visibility enum |
#8065 |
| Save as template and template page | #8068 |
| Gallery with tags | #8069 |
| Link a template to an assignment | #8070 |
| Copy the circuit into each student project | #8084 |
| A template carries a test suite | #8092 |
| Enforce private, group and public visibility | #8100 |
| Fork button | #8102 |
| Usage count | #8107 |
is_template and template_visibility are separate from project_access_type. Access decides who can view a project. Visibility decides who can find and copy it. An assignment project cannot be a template, because it is a student's working copy. Only the author, an admin or a mentor of one of the author's groups can save a template, and collaborators cannot.
The copy that has to ask again
An assignment stores a reference to its template, and the copy into a student's project happens when the student opens the assignment. That created the best review finding of the project: an instructor picks a legitimate template on Monday, its author makes the project private on Tuesday, and on Wednesday a student still receives its data. Access was decided at the earlier time and never at the later one. The copy now asks again:
def template_circuit_data
return unless circuit_template
return if circuit_template.project_access_type == "Private" && !template_author_mentors_group?
circuit_template.project_datum&.data
end
Both the web start action and the LTI launch call the same method, so one place answers. I copy only the circuit data and do not call Project#fork, because a fork also notifies the author and needs a preview image, and a student opening an assignment is not a fork.
CodeQL also flagged the first version of the template check in #8070 as a SQL query built from user input. It became exists?(id: template_id), which lets ActiveRecord bind the value.
One rule for visibility
The week 22 pull request (#8100) turns visibility from a label into a rule. Project.templates_for(user) is now the single place that decides what a person may see, and both the gallery and the template page use it. A template you cannot see is not listed and returns 404 if you open its URL:
scope :templates_for, lambda { |user|
return where(is_template: true).template_public if user.nil?
groups = Group.where(id: GroupMember.where(user_id: user.id).select(:group_id))
.or(Group.where(primary_mentor_id: user.id)).select(:id)
peers = GroupMember.where(group_id: groups).select(:user_id)
mentors = Group.where(id: groups).select(:primary_mentor_id)
shared = template_group.where(author_id: peers).or(template_group.where(author_id: mentors))
where(is_template: true).and(template_public.or(by(user.id)).or(shared))
}
A public template is visible to everyone, a private one to its author, and a group one to people who share a group with the author, plus those groups' primary mentors. The template picker on the assignment form uses the same scope, so what the form offers and what the server accepts are the same list. A bug the earlier gallery review found, a template that is listed and then refuses to open, cannot happen when every path asks one question.
Fork and usage
PR #8102 adds a Fork button to the template page so an instructor can copy a template and customise it. It posts to the existing create_fork_project_path, so there is no new route or controller code. The copy gets is_template: false and private visibility.
PR #8107 shows "Used in N assignments" on the template page and on each gallery card.
Pull request summary
| PR | What it does |
|---|---|
| #7647 | Saves a grade locally before pushing it to the LMS |
| #7654 |
LtiDeployment model for platform registrations |
| #7659 | JWKS and tool configuration endpoints |
| #7715 |
Lti::JwtValidator for id_token verification |
| #7763 |
LtiResourceLink for launch service endpoints |
| #7764 |
Lti::AccessToken for platform service calls |
The four closed ones are #7488 (the proof of concept), #7502 (a NameError fix in the 1.1 code), #7510 (the gem) and #7632 (the first subgroups attempt). The 34 open pull requests are the ones listed in the parts above. Most are stacked on each other, so they have to merge in order.
I also sent about 40 small pull requests before the coding period: typo fixes, and a few real bug fixes such as scoping an assignments index to the current user's groups, removing a duplicate update! call and adding includes(:user) to notification callbacks that fired N+1 queries.
What I learnt
Small is a property of the review, not of the code. A 2,000 line branch was unreviewable even though every line of it worked. The fix was to cut the work into pull requests that can be understood alone. That also means a pull request that needs an unmerged one is not small. The cost of stacking comes back in review as "the thing this code calls is not on this branch".
Two ways to survive an early pull request, and I would use only one. Merging the prerequisite into the branch makes the diff larger but keeps the code honest. Writing guards like respond_to?(:subgroups) turns a 500 into a message, but leaves a branch that can never be taken once the stack merges. Next time, code that only exists to survive its own pull request being early gets deleted in the pull request that makes it unnecessary, and I write that deletion in the description.
Check access where the data moves. Almost every security finding of the second half was the same shape. A reference is a promise about later, and every promise about later needs a check at later: the template copy, the subgroup membership after a parent is left, the API with a missing list.
Validate first, mutate second. The form that converted a deadline to ISO before refusing the submit, the stale-context clear that ran before verification, and the skip_before_action that turned a check off for every request are the same mistake from different angles.
Do not trust a convenient library or a convenient default. The gem that did not do the hard parts, the localhost default URL, the {} column default and the nil that meant two things all made the code easier to write and harder to trust.
Read the issue as a person, not as a machine. At the midterm I had built most of a system no instructor could use, because I read a feature request as a description of a mechanism and not of a person trying to get something done.
Scanners and reviewers are usually pointing at something real, even when their conclusion does not fit. CodeQL on CSRF, CodeRabbit on "passed":"false" and on insert_all all found something I would not have found by testing through the model.
What is still open
I want to be plain about this, because a final report that only lists what went well is not useful to the next person.
- Most of the work is not merged. 34 pull requests are open, mostly stacked. Merging them in order is the main job after the coding period.
- Parent removal leaves subgroup membership. A student removed from a class, by a mentor or by the roster sync, still belongs to the team inside it. This needs its own fix that also covers the API destroy path.
-
A missing
user_idson the web path still empties a subgroup. The API path is fixed. -
Runner response validation. The strict checks on a runner reply are still open: a body of
null, or a case with"passed":"false", should never become a grade. The runner's response shape is also defined only by the client and its spec, because the simulator service is not in the repository yet. - HTTPS on stored line item URLs. A bearer token should not be sent to a non-HTTPS line item URL.
-
Nothing sets
template_testbenchin the interface yet. #8092 is the plumbing for a later change. - Moving existing installations. The 1.3 code is behind a default-off flag. Moving real installations over is a rollout task, not a coding one.
Future work
- Merge the stack in order and delete the temporary guards (
roster_services_loaded?,check_subgroups_available) in the pull requests that make them unnecessary. - Land the simulator service so the runner's contract comes from the side that simulates.
- A UI for authoring a test suite on a template.
- Recovery paths for partial roster failures.
- A migration guide for instructors moving from LTI 1.1 to 1.3.
Weekly blogs
- Pre GSoC: Finding My Way to CircuitVerse
- Getting Selected: The Night I Couldn't Sleep
- Community Bonding: Before the First Line of Project Code
- Week 1: What Happens When a Student Clicks "Open Assignment"?
- Week 2: Pull Requests, Rejected Code, and the Art of Not Breaking Things
- Week 3: The Week Nothing Merged
- Week 4: Rebuilding From the Spec (and Dropping the Gem)
- Week 5: Identity Is Not an Email Address
- Week 6: Signing the State, and Closing My Own Pull Requests
- Week 7: Creating phases for the tasks
- Week 8: The First Merge, and Learning to Wait
- Week 9: Verifying the Token
- Week 10: The Handshake Begins
- Week 11: Welcome to my Midterm evaluation
- Week 12: Two Small Pull Requests
- Week 13: A Kid Only the Platform Can Read
- Week 14: The Approved Pull Request That Stopped Passing
- Week 15: What the Simulator Actually Binds
- Week 16: The Editor That Learned When to Say No
- Week 17: A Client That Fails Loud
- Week 18: Four Small Pieces and the Seams Between Them
- Week 19: What the Student Is Allowed to See
- Week 20: A Pull Request That Cannot See Its Prerequisites
- Week 21: Access Has to Be Checked Where the Data Moves
Acknowledgements
Thank you to my mentors, Anas Khan and Aman Asrani, for the reviews that shaped this project, in particular the week 7 review that found the bug I had looked straight past. Thank you to the CircuitVerse maintainers and contributors for reading a long stack of small pull requests, and to CodeRabbit, CodeQL and AccessLint, which found things I would not have found by testing. Thank you to Google Summer of Code for the opportunity.
I am not going anywhere. The stack is open and I will keep working through it until it is merged.
Top comments (0)