Sometimes a performance bug is one missing line. Sometimes the fix for it has a second bug hiding inside, one only a careful reviewer would spot. This merge request had both.
This is the story of gitlab-org/gitlab!270160, which fixed an N+1 query in GitLab's GraphQL group labels. I'll cover what the issue was, the plan I proposed, the first fix, the review comment that changed it, and what I'd tell anyone starting out with GitLab contributions.
The issue: two extra queries for every subgroup
The issue was gitlab-org/gitlab#635535, opened by Terri Chu. It was one of the best-written bug reports I've picked up. In short:
-
Group.labelswithincludeDescendantGroups: trueran 2 queries per subgroup that owns a returned label. - The queries came from
GroupPolicyconditions that read each label group's root ancestor. -
EE::Preloaders::LabelsPreloaderalready preloaded root ancestors for project labels' groups, but not for group labels' groups. - The
search_labelsMCP tool always setsincludeDescendantGroups: true, so it hit this on every group call.
The issue even named the two queries and where they come from:
| Query | Condition |
|---|---|
root ancestor lookup (find_namespaces_by_id) |
user_banned_from_namespace → @subject.root_ancestor
|
saml_providers (cached) |
needs_new_sso_session → SsoEnforcer.access_restricted? → root_saml_provider → root_ancestor.saml_provider
|
There was also an existing spec that documented the problem instead of hiding it. The MCP search_labels N+1 example carried a threshold: 4 allowance with a TODO pointing at the issue:
# TODO: https://gitlab.com/gitlab-org/gitlab/-/work_items/635535
it_behaves_like 'an MCP tool that avoids N+1 queries', threshold: 4 do
That's a nice touch for a contributor. When the fix works, you can delete the allowance and the spec becomes the guard.
The plan, before writing any code
GitLab's community workflow asks you to comment on the issue with a plan and get assigned before you start. Here's what I wrote:
My plan is to preload root ancestors for group labels' groups in
EE::Preloaders::LabelsPreloader(this is already done for project labels), add a QueryRecorder example onGroup.labels(include_descendant_groups: true), and tighten the MCPsearch_labelsthreshold.
Terri confirmed the plan and made the threshold part concrete: drop threshold: 4 completely, "so the example fails if the per-subgroup queries come back."
So the plan had three parts:
- Production change: run the existing root-ancestor preload over group labels' groups too.
-
A new regression spec: a
QueryRecorderexample on the group labels GraphQL query. -
Remove the allowance: delete
threshold: 4so the MCP spec guards the fix.
Version 1: one combined list
The preloader code was short. Before the fix, only project labels' groups got their root ancestors preloaded:
# preloading the root ancestors for the project labels for the authorizations checks
project_groups = project_labels.filter_map(&:project).filter_map(&:group)
return if project_groups.empty?
::Namespaces::Preloaders::GroupRootAncestorPreloader.new(project_groups).execute
My first commit (ee096b74) followed the fix suggested in the issue: build one list of groups from both kinds of labels and preload it once.
# preloading the root ancestors for the authorizations checks
groups = group_labels.filter_map(&:group) + project_labels.filter_map(&:project).filter_map(&:group)
return if groups.empty?
::Namespaces::Preloaders::GroupRootAncestorPreloader.new(groups).execute
The saml_provider preload that follows then ran over groups.map(&:root_ancestor).
For the regression spec, I added an example to ee/spec/requests/api/graphql/group/labels_query_spec.rb. It records a control run, adds one more private subgroup with its own label, and checks the query count doesn't grow:
it 'does not add queries for additional subgroup labels' do
run_query
expect(graphql_errors).to be_blank
control = ActiveRecord::QueryRecorder.new(skip_cached: false) { run_query }
extra_group = create(:group, :private, parent: group)
create(:group_label, group: extra_group, title: 'fff')
expect { run_query }.not_to exceed_all_query_limit(control)
expect(graphql_errors).to be_blank
expect(graphql_data.dig('group', 'labels', 'nodes').pluck('title')).to include('fff')
end
Two details matter here:
-
skip_cached: falsewithexceed_all_query_limitcounts cached queries too. One of the two queries in the issue (saml_providers) was cached, so a normal count could miss it. - The last line checks that the new label really comes back. A query-count spec that passes because the data vanished doesn't prove anything.
I removed threshold: 4 from the MCP spec, opened the MR, and asked Terri for review. The pipeline passed.
The review: "the preloader drops the second copy"
Terri asked Alex Filatov for an initial review. Alex ran both validation steps locally and confirmed they passed. Alex also confirmed that with the production change reverted, both new examples fail, and then left this:
suggestion: A subgroup that owns both a group label and a project label still costs one extra query, because the root ancestor preloader drops the second copy of that group.
The explanation was precise. group_labels.filter_map(&:group) and project_labels...filter_map(&:group) come from separate Preloader calls. So when one subgroup owns both a group label and a project with a label, that subgroup exists as two different Ruby objects.
Here's the start of the shared preloader:
def initialize(namespaces, root_ancestor_preloads = [])
@namespaces = namespaces.uniq.compact
@root_ancestor_preloads = root_ancestor_preloads
end
uniq uses ActiveRecord equality, and ActiveRecord compares records by id. So the two objects count as duplicates, and only the first one keeps its place in @namespaces. Later, the preloader sets namespace.root_ancestor = root_ancestor only on the objects in that list. The project-side copy never gets one, so the first time policy code asks for its root ancestor, it runs a query of its own.
In other words, my version 1 fixed the group-label side and quietly broke something that used to work. Before my change, the project-side groups were preloaded, because they were the only list.
Alex also brought proof: a small spec that fails on version 1 (Expected a maximum of 21 queries, got 23, where the extra query is namespaces ... LIMIT 1) and passes with the proposed fix.
Version 2: preload once per list
The fix is to run the preloader once for each list, so every Ruby object that policy code will touch gets its own root_ancestor. This is commit 26d2a3d3:
# Preload the root ancestors for the authorization checks. The two lists hold different
# instances of a shared group, and the preloader drops duplicates by id, so run it per list.
group_lists = [group_labels.filter_map(&:group), project_labels.filter_map(&:project).filter_map(&:group)]
.reject(&:empty?)
return if group_lists.empty?
group_lists.each { |groups| ::Namespaces::Preloaders::GroupRootAncestorPreloader.new(groups).execute }
groups = group_lists.flatten
This costs one extra query that doesn't grow with the number of groups. A constant query beats a per-group query.
Before applying it, I opened NamespaceRootAncestorPreloader and confirmed the namespaces.uniq behavior myself. A reviewer's suggestion is a strong signal, but I still want to understand why before I push it under my name. I also added Alex's example as a new file, ee/spec/models/ee/preloaders/labels_preloader_spec.rb, so this exact case stays covered:
it 'does not query root ancestors per group that owns both group and project labels' do
labels = create_group_with_both_labels
control = preload(labels)
labels += Array.new(2) { create_group_with_both_labels }.flatten
expect { preload(labels) }.not_to exceed_query_limit(control)
end
The commit message explains the root cause and credits the reviewer:
Preload root ancestors per label list in LabelsPreloader
Group labels and project labels are loaded by separate preloaders, so a
group that owns both kinds of labels is two Ruby objects.
GroupRootAncestorPreloader drops duplicates by id, which left the
project-side copy without a root ancestor and cost one query per such
group. Run the root ancestor preloader once per list instead.
Suggested by @alexfilatov.gitlab in review.
Alex re-checked the new version: 25 examples passing across the four related spec files, the new preloader spec failing as expected against the old code, and RuboCop clean. Alex then asked a maintainer to merge.
Merge
Fred de Gier did the maintainer review. Beyond the specs, Fred checked that permissions behaved the same after the change:
I checked locally that the new examples fail without your change and pass with it, and that users with different access to the subgroups still see exactly the same labels as before, so this looks good to me.
That's a check I want to remember. A preloading change in an authorization path should never change what a user can see, only how many queries it takes to decide. The MR merged on the same day it got its first review. All four of its pipelines passed.
What I took away
1. ActiveRecord equality is not object identity. Two Group objects with the same id are ==, so uniq treats them as duplicates. But a value cached on one object (like root_ancestor) does not appear on the other. When a preloader dedupes by id and writes to objects, every copy you'll use later has to be in the list it writes to.
2. Count cached queries in N+1 specs. QueryRecorder.new(skip_cached: false) with exceed_all_query_limit caught a query that a normal count could miss.
3. Delete the allowance; don't just loosen it. Removing threshold: 4 turned a spec that documented a bug into one that guards against it.
4. Credit reviewers in the commit. "Suggested by … in review" in the commit body keeps the history honest and makes the next reader's job easier.
5. Stay assigned until the issue is closed. After the merge I unassigned myself, because the MR had merged. Terri reassigned me and explained why: the issue should move with the MR (In dev, In review, then Verification after the merge) and only close once the change is verified in production. I didn't know that, and it's useful for any GitLab contributor to hear.
Links
- Issue: gitlab-org/gitlab#635535
- Merge request: gitlab-org/gitlab!270160
- The spec that introduced
threshold: 4: gitlab-org/gitlab!264415
Thanks to Terri Chu for a bug report that made the work easy to start, Alex Filatov for a review that made the fix correct, and Fred de Gier for the careful final check.
If you've been thinking about contributing to GitLab, the community issues are a great place to start: contributors.gitlab.com.


Top comments (0)