Performance Comparison of OneDev and GitLab

TLDR;

• Git Push: OneDev is 40% faster than GitLab
• Git Clone: OneDev is 20% slower than GitLab
• Web UI: OneDev is 10x~30x faster than GitLab
• Memory: OneDev uses 70% less memory than GitLab

For every feature built into OneDev, I take performance seriously. It is very fast with moderate resource usage. In this article I compare it with GitLab to show some numbers.

Environment Setup

OneDev and GitLab runs on same machine. When testing OneDev, GitLab is stopped, and vice versa.

• Hardware: AWS EC2 c5a.xlarge (4 core, 8G mem, GP2 SSD EBS storage)
• OS: Ubuntu 18.04
• OneDev: 7.3.2, installs using the zip file, running on openjdk 8
• GitLab: 14.10.2-ee, installs from official repository with apt-get
• Test repository: Kubernetes (100K+ commits)

Since OneDev by default uses 50% of available memory, and this is too deluxe for a 8G machine, I modified its conf/wrapper.conf to change max memory to be 20%.

Repository Push/Clone

Command used for a fresh push (repository is empty initially):

$ time git push <repository url> master:master

And command used for clone:

$ time git clone <repository url>

Tests done from another c5a.xlarge linux machine in same subnet. Result is as following:

Conclusion:

• Git Push: OneDev is 40% faster than GitLab
• Git Clone: OneDev is 20% slower than GitLab

This is measured for full push/clone of a large repository. For daily increment push/pull , this is almost negligible.

I did not measure concurrent push/clone, as both rely on native git to do the heavy lift, and the numbers should be on par.

Web UI

Web UI test is performed with JMeter. The selenium web drive plugin is installed running chrome in headless mode to do end-to-end test. The idea is to visit a url, then wait for certain elements to appear. Taking GitLab pull request changes page for instance, I am using below groovy script for web drive sampler:

import org.openqa.selenium.*
import org.openqa.selenium.support.ui.*
import java.time.Duration
def wait = new WebDriverWait(WDS.browser, 60)
WDS.sampleResult.sampleStart()
WDS.browser.get('http://testserver/root/kubernetes/-/merge_requests/1/diffs')
wait.until(ExpectedConditions.presenceOfElementLocated(By.cssSelector('.diff-content')))
WDS.sampleResult.sampleEnd()

I started four c5a.2xlarge EC2 machine (8 cores, 16G mem) in same subnet as test server, each machine running 8 browsers via JMeter, and start another c5a.xlarge machine to aggregate results. This effectively simulates 32 concurrent users, and can drive server CPU to nearly 100%. Note that running 32 browsers on a 32 core VM can not drive OneDev to use 100% CPU, due to browser resource contention on same machine.

Six similar pages in OneDev and GitLab are selected for test, displaying same set of data. For each round of test, the page is warmed up for 1 minute, then run 10 minutes to get the performance metrics. Gravatar of both products are disabled to avoid out-of-server resource loading. GitLab auto devops is also disabled to save cpu cycles.

Repository Root Page

Both products show root of the repository including the readme content:

Test result:

Note that for OneDev test in this case, I have to launch five c5a.2xlarge machine to simulate 40 concurrent users to drive it to use nearly 100% cpu.

Recent Commits Page

Both products show recent commits of the repository. OneDev shows 50 commits initially, while GitLab shows 40 commits. OneDev shows commits as verified as it trust GitHub GPG signing key by default.

Test result:

Commit Detail Page

Both products show detail of same commit, including diff of all changed files

Test result:

File Content Page

Both products show content of same repository file.

Test result:

Pull Request Commits Page

Both products show commit list of same pull request. OneDev shows earliest commit first, while GitLab shows latest commit first

Test result:

Pull Request Changes Page

Both products show diffs of changed files of same pull request.

Test result:

For all tests above, OneDev consumes 2.2G mem, while GitLab consumes 4G~4.5G mem.

Conclusion

OneDev web UI performs orders of magnitude faster than GitLab, often 10x~30x faster, with much less memory usage. The UI performance difference is very obvious when navigating through different pages of both products: OneDev loads page instantly after warm up, while GitLab has noticeable delays loading pages all the time.

Thanks for watching. If you would like to run the tests yourself, check the setup detail here.