Designing Rolling-Deploy-Safe Async REST APIs: Returning Release Evidence IDs in GitLab

An asynchronous API endpoint should give clients a way to identify and track the resource being created. Returning 202 Accepted with an empty response body makes that difficult: the server has accepted the work, but the client has no resource ID to use for subsequent requests.

That was the problem with GitLab's release evidence collection endpoint:

POST /projects/:id/releases/:tag_name/evidence

In GitLab MR !268983, addressing issue #214819, I worked on returning the newly created release evidence ID immediately while keeping the asynchronous workflow compatible with existing Sidekiq jobs and multi-node deployments.

The interesting part was not simply returning an ID. It was making the lifecycle of an asynchronously created database record safe across retries, deployments, and existing query paths.

The Problem: Accepted Work Without a Resource Handle

An endpoint that schedules background processing commonly returns HTTP 202 Accepted. This communicates that the request has been accepted, but processing is not necessarily complete.

For GitLab's release evidence endpoint, the response did not provide a handle to the evidence record being collected.

A more useful flow is:

1. The API creates a placeholder evidence record.
2. The API schedules background collection.
3. The API returns the evidence resource, including its ID.
4. The background worker fills in the collected evidence data.

This gives the caller an identifier immediately, without making the API request wait for the potentially expensive collection process.

However, introducing a placeholder record creates lifecycle and compatibility problems that need to be addressed explicitly.

The Architecture Challenges

Creating a Releases::Evidence record synchronously and passing its ID to a background worker introduces several production concerns.

1. In-flight Sidekiq jobs during rolling deployments

Before this change, existing jobs could have been enqueued with two arguments:

(release_id, pipeline_id)

The updated worker also needed to accept an evidence ID:

(release_id, pipeline_id, evidence_id)

During a rolling deployment, jobs enqueued by an older application version can still execute after newer code has been deployed. Changing the worker's argument contract without preserving compatibility could break those jobs.

2. Incomplete records becoming visible

The placeholder evidence record exists before collection finishes.

At that point, fields such as summary and summary_sha may not represent completed evidence. If existing GraphQL queries expose the record immediately, users can see an incomplete resource.

3. Orphaned records after permanent failure

Sidekiq retries failed jobs. But retries can eventually be exhausted.

If collection never succeeds, the placeholder record should not remain indefinitely as an uncollected database record.

The implementation needed to address all three concerns without making the API synchronous.

1. Create the Evidence Record Before Scheduling the Worker

The first part of the change was in:

ee/lib/ee/api/releases.rb

Instead of waiting for the background worker to create the evidence record, the endpoint creates a stub immediately:

evidence = release.evidences.create!(summary: {})

The evidence ID is then passed to the worker, and the endpoint returns the serialized resource:

present evidence, with: ::API::Entities::Releases::Evidence

The API entity was updated to expose id alongside the existing evidence attributes:

• sha
• filepath
• collected_at

This changes the API contract in an important way: the caller receives an identifiable resource even though its evidence collection is still pending.

The API does not need to wait for the background job to finish.

2. Preserve Compatibility with Existing Sidekiq Jobs

The worker change had to account for jobs already present in the queue.

In:

app/workers/releases/create_evidence_worker.rb

The new evidence_id argument defaults to nil:

def perform(release_id, pipeline_id = nil, evidence_id = nil)
  release = Release.find_by_id(release_id)
  return unless release

  pipeline = Ci::Pipeline.find_by_id(pipeline_id)

  evidence = release.evidences.find_by_id(evidence_id) if evidence_id
  return if evidence_id && evidence.nil?

  kwargs = { pipeline: pipeline }
  kwargs[:evidence] = evidence if evidence

  ::Releases::CreateEvidenceService.new(release, **kwargs).execute
end

This preserves the older two-argument invocation:

perform(release_id, pipeline_id)

When evidence_id is absent, the worker continues through the existing service flow without requiring a pre-created evidence record.

When an ID is provided, the worker looks up the corresponding evidence record and passes it to CreateEvidenceService.

The additional guard:

return if evidence_id && evidence.nil?

prevents the worker from continuing with an explicitly supplied ID that no longer resolves to an evidence record.

This is an important rolling-deployment principle: when changing background job arguments, new code should account for jobs that were serialized by older code and are still waiting to execute.

3. Clean Up Placeholder Records When Retries Are Exhausted

Backward compatibility protects existing jobs, but it does not solve permanent collection failures.

For that, the worker uses Sidekiq's sidekiq_retries_exhausted callback:

sidekiq_retries_exhausted do |job, _exception|
  evidence_id = job['args'][2]

  if evidence_id
    evidence = Releases::Evidence.find_by_id(evidence_id)
    evidence.destroy if evidence && evidence.summary_sha.nil?
  end
end

The callback retrieves the third job argument, which is the evidence ID introduced by the change.

It then checks whether the record still exists and whether summary_sha is still nil.

Only an uncollected record is removed.

That condition matters because the callback should clean up an incomplete placeholder, not delete evidence that has already been successfully collected.

The result is a lifecycle with two possible outcomes:

• Successful collection: the evidence record is populated and retained.
• Permanently failed collection: the uncollected placeholder is cleaned up after retries are exhausted.

4. Keep Incomplete Evidence Out of Existing Queries

Creating a placeholder immediately means database records can exist before they are ready for normal consumption.

To keep incomplete records out of existing query paths, the change introduced a collected scope in:

app/models/releases/evidence.rb

scope :collected, -> { where.not(summary_sha: nil) }

It also added a collected? method to represent whether evidence collection has completed.

The GraphQL ReleaseType#evidences resolver was then filtered so that uncollected placeholders are not exposed as normal evidence records.

This separates two concepts that should not be conflated:

• Existence: the evidence record has been created and has an ID.
• Collection completion: the evidence data has been successfully collected.

The API can return the resource ID immediately, while existing GraphQL consumers continue to see only collected evidence.

That distinction is useful in asynchronous systems: a resource can exist before it is ready to be presented as completed data.

5. Roll Out the Change Safely

The new behavior was guarded behind the gitlab_com_derisk feature flag.

The corresponding configuration was added at:

config/feature_flags/gitlab_com_derisk/return_release_evidence_id.yml

The API documentation was also updated in:

doc/api/openapi/openapi_v3.yaml

The OpenAPI schema now reflects the 202 Accepted response using the API::Entities::Releases::Evidence response entity.

This keeps the implementation, feature rollout, and API contract aligned.

Feature flags are particularly useful for changes that affect API behavior and background processing. They provide a controlled way to introduce the new behavior rather than requiring every environment to adopt it at once.

The Complete Lifecycle

The resulting workflow can be understood as a sequence:

1. A client sends a request to collect release evidence.
2. The API creates an empty evidence record.
3. The API enqueues CreateEvidenceWorker with the evidence ID.
4. The API returns the serialized evidence resource, including its ID.
5. The worker collects the evidence and updates the record.
6. GraphQL exposes the record only after it has been collected.
7. If the job permanently fails, the retry-exhaustion callback removes the uncollected placeholder.

Each part addresses a different failure mode: API usability, old queued jobs, incomplete data visibility, and permanent background-job failure.

Engineering Takeaways

This change illustrates several principles for building reliable asynchronous APIs.

Return a resource handle as early as possible. A 202 Accepted response becomes more useful when clients can identify the resource associated with the accepted operation.

Treat background job arguments as a compatibility contract. Sidekiq jobs can outlive the application version that enqueued them. Optional arguments and safe defaults help preserve compatibility during rolling deployments.

Separate record creation from readiness. A database record existing does not mean its asynchronous processing is complete. Explicit scopes and completion checks help prevent incomplete data from leaking into existing interfaces.

Design cleanup for permanent failure. Retry mechanisms improve resilience, but they do not eliminate the need to handle exhausted retries and orphaned intermediate state.

Keep API documentation synchronized with behavior. When a response changes from an empty accepted response to a serialized resource, the OpenAPI contract should reflect that change.

Conclusion

Returning a release evidence ID from an asynchronous endpoint sounds like a small API improvement. In a distributed application, however, it affects database lifecycle management, background job compatibility, query visibility, failure cleanup, feature rollout, and API documentation.

GitLab MR !268983 addressed these concerns together: creating the evidence record immediately, passing its ID to a backward-compatible Sidekiq worker, hiding incomplete records from GraphQL, and cleaning up placeholders after permanent failure.

The broader lesson is that asynchronous API design is not just about scheduling work. It is about making the resource lifecycle predictable before, during, and after that work executes.

References

• Issue: gitlab-org/gitlab#214819
• Merge Request: gitlab-org/gitlab!268983

Author: Manish Kumar

GitLab: @manishpatel00