This article explains how to see how much code in one repository is duplicated in other projects. After reading it, you can create a code traceability task, choose the repository to check and the repositories to compare, set the similarity thresholds, and go from the duplicate rate and similarity bands down to the repeated files and functions.
Two names run through the whole article. Trace Target is the code you want to check. Here it is a repository. A traced repository is a repository you compare it with. You can select at most 50.
Code traceability here means a similarity and duplication check. It does not tell you which requirement the code was written for.
Code traceability is available on Team and Enterprise. Community has no Traceability menu.
The finished task to follow is fixture: source vs dup (repo). The trace target is the repository xlliu/test-trace-source-example. The traced repository is xlliu/test-trace-dup-example. Three other tasks on the trace list use the same sample data: one with the file InvoicePriceCalculator as the target, one with the function calculateInvoiceTotal as the target, and one compared with xlliu/test-trace-unique-example.
1. Create the task
- Open Code Traceability In the sidebar, open Advanced → Traceability. The page title is Code Traceability. Click Create Trace at the top right. A four-step wizard opens.
- Step 1: Basic Information Trace Name is required. Say what this check is, for example “source vs dup duplicate check”. Trace Description can be left empty.
-
Step 2: Select Target
Trace Target has four types: Repository, File, Code Block, and Function. To check a whole repository, choose Repository. This example uses
test-trace-source-example. The branch can be left unset. The Deduplicate Files switch below identifies and excludes duplicate files. Leave it at the default. - Step 3: Choose languages and similarity thresholds The title of this step is Configure Parameters. Leave Check Languages unselected. The note says: “Leave empty to check all languages automatically”. There are two similarity thresholds. File Similarity Threshold defaults to 80%. Code Block Similarity Threshold defaults to 90%. Use the defaults for the first run.
-
Step 4: Select traced repositories
The title of this step is Select Repositories. Find
test-trace-dup-examplein the search box and click Add. The top right shows how many are selected, for example1 / 50. Do not pick a same-named repository whose name contains-Fork. Configuration Summary below gathers every choice so far. Read it once before you continue. - Create and start Click Create & Start. The task is created and starts at once. If you click only Create, the task stays Pending. Later, click Start Trace in the list to run it.
2. How to choose the parameters
For the parameters in step 3 and in Advanced Configuration, the defaults are enough in most cases. The table lists each default, and when you would change it:
| Parameter | Default | When to change it |
|---|---|---|
| Check Languages | Unselected. Every language is checked | Usually leave it. Select a language only when you want that one language |
| File Similarity Threshold | 80% | Lower it to see more files that are somewhat alike. Raise it to keep only files that are almost the same |
| Code Block Similarity Threshold | 90% | Same idea, applied to code snippets |
| File Encoding | UTF-8 (Default) | Change it to GBK when an old project’s source is GBK |
| Ignore Identifier Differences | Off | Turn it on when you suspect someone copied code and renamed variables or functions |
| Ignore Literal Differences | Off | Turn it on when you want code whose strings and numbers differ and whose logic is the same |
| Enable Repository Cache | Enabled | Leave it on when you will check the same repositories again |
Two of them need an extra sentence. File Encoding has to match the encoding of the source. If GBK source is still read as UTF-8, Chinese comments become garbled and the comparison changes. When Enable Repository Cache is on, the next check of the same repositories does not clone the code again. After you turn it off, the code cloned for this run is deleted when the task ends.
The threshold slider moves in steps of 5%. Run once with the defaults, look at the result, then decide whether to raise or lower them. If you set the thresholds very low at the start, the result list is huge and the important rows are harder to find.
Directories whose names start with ., plus node_modules and target, are left out of the comparison. There is no option to exclude other directories.
3. Wait for the task to finish
In the task list, status moves through Pending, Running, and Completed. An error during the run becomes Failed. A task you stop by hand shows Cancelled. A running task has Stop Trace.
How long it takes depends on the size of the repositories and how many traced repositories you selected. Larger repositories and more selected repositories take longer. You can leave the page while it runs and come back later.
4. Read the result: from the whole repository down to a function
After the task is Completed, click View Details. The detail page has two tabs, Trace Results and Execution Logs. Read Trace Results from the whole picture down to the specific match.
Start with the totals. Across the top are Traced Repositories, Traced Files, Code Lines, Duplicate Rate, and Unique Rate. The duplicate rate is only an overall hint. Projects on the same team often share a framework and templates, so some similar code is expected. A duplicate rate above zero is normal.
Then read the bands. Each match is placed in one of five bands by similarity: Identical (95–100%), High Similarity (80–95%), Moderate Similarity (60–80%), Low Similarity (40–60%), and Unique Code (0–40%). The bands worth reading first are Identical and High Similarity.
Then read the matches. The result list can Sort by Similarity, so the closest matches come first. Type a directory or file name in the “Search repository or file...” box to keep only the related matches. Each match shows the files on both sides and a Similarity Score. Click Compare Code to open both sides. There are two views:
- Side by Side: Source File on the left, Trace File on the right. Use it to see the overall shape of the two pieces.
- Unified: stacked. Use it to see small differences.
In source vs dup, the result has no Identical match. Both matches are High Similarity. Open Compare Code on InvoicePriceCalculator.java. In Side by Side, both sides are public class InvoicePriceCalculator, and the Similarity Score is 86.86%.
Then judge each row. After you see duplicated code, find out why it is duplicated, then decide what to do. Three cases are common:
| What you see | Why it is duplicated | What to do |
|---|---|---|
| Framework-generated code, or a config template | A scaffold or a shared template generated it | Normal duplication. Leave it |
| A utility function copied into several projects | An old copy and paste | Consider pulling it into a shared module, and refactor through a pull request |
| Business code that should not be here | A copy across projects by mistake, or reuse that breaks a rule | Ask the owner where it came from, and whether it should be removed |
5. Export and run again
Export the report. On the task, click Export Report. Choose Export Format and Export Scope (all results, or the results loaded so far). The formats are PDF (HTML), Excel, JSON, and Markdown. Click Preview Report if you want to see the shape first. Two points: PDF (HTML) exports an HTML file. Open it in a browser, then print it to PDF. Excel currently exports CSV. Excel may warn that the format does not match the extension, so when you need a spreadsheet, export Markdown or JSON first and convert it.
Run again. When the code has a new commit and you want to check again, click Retry. Only a Pending task can have its parameters changed. For a task that has already run, create a new task if you want different parameters. That keeps the two runs easy to tell apart.
6. What to check when you get no useful result
When the task fails, or the result looks wrong, use this table:
| What you see | Cause | What to do |
|---|---|---|
| The sidebar has no Traceability menu | Community does not include code traceability, or this account has no code traceability permission | Confirm the edition, or ask an admin to grant the permission (see after the table) |
| Step 4 cannot continue, and the page says “Please select at least one traced repository” | No traced repository was added | Add at least one |
| Status becomes Failed | The run hit an error, for example fetching one repository’s code failed | Open Execution Logs and read the cause |
| Completed, and there are almost no matches | The thresholds are too high, or the two sides really are unlike | Lower each threshold by 10%, create a new task, and run it |
| You selected a language and the result is empty | The language filter does not match the files in the repository | Leave Check Languages unselected so every language is checked |
| Matches near Chinese comments look strange | The source is not UTF-8 | Set File Encoding to GBK and create the task again |
If it is a permission problem, ask an admin to turn on Code Traceability in the member permissions. Its note says: “Allowed to use code traceability for duplicate detection”.
Common questions
Can a push start a duplicate check by itself?
Each code traceability run is created and started by hand. A push and a pull request do not start it.
I want to find which repositories contain one piece of code. How?
In step 2, set Trace Target to Code Block or Function. Fill in the name and paste the code. A code block should be at least 10 lines. A function needs the full definition and implementation. In step 4, select the traced repositories you want to search.
Which branch of a traced repository is compared?
Each traced repository is compared on its default branch. The branch of the target repository is chosen in step 2.
Summary
To check a repository for duplicate code, open Advanced → Traceability, click Create Trace, and build a trace task in four steps: fill in the name; choose the target repository and branch; leave languages unselected and start with the default similarity thresholds of 80% and 90%; add 1 to 50 traced repositories. Click Create & Start and wait until the task finishes. In View Details, read Duplicate Rate first, then the identical and high-similarity bands, and click Compare Code on the matches at the top to decide which ones need work. To keep a copy, use Export Report. PDF (HTML) and Markdown are the simplest. Code traceability is available on Team and Enterprise. Community has no Traceability menu.
Conclusion: Under Advanced → Traceability → Create Trace, choose the target repository and 1 to 50 traced repositories (start with the default similarity thresholds of 80% and 90%). After the task finishes, sort the detail page by similarity and click Compare Code to check the duplicated snippets.
Top comments (0)