DEV Community

Sheng-Lin Yang
Sheng-Lin Yang

Posted on

Understanding a new feature from Repomix

I needed to read the code from the Repomix website and choose one feature to implement in my own CLI tool.
I chose the feature, --token-count-tree [threshold].

This feature displays a file tree with token counts, with an optional threshold to show only files with N or more tokens (for example: --token-count-tree 30).
It takes a file path and calculates how many tokens are in each file, ignoring tokens that start with an underscore (_) or tokens that represent object or array types.

At first, I checked the code on GitHub, but I realized it wasn’t easy to find and understand all the dependencies there. So I used the command git grep "buildTokenCountTree" to locate the related code blocks.
I found that it probably involves five main files:

1. `cliRun.ts` - set up optional features.
2. `configSchema.ts` - groups related functions and calls them when needed.
3. `defaultAction.ts` – builds the CLI configuration and assigns options based on the user’s input.
4. `cliReport.ts` – called by `defaultAction` to gather the required information.
5. `buildTokenCountStructure.ts` - called by `cliReport` to handle the logic.
6. The function `buildTokenCountTree` calculates token usage for each file and calls `calculateTokenSums` to produce the result.
Enter fullscreen mode Exit fullscreen mode

I learned that larger projects are usually organized more cleanly with interfaces, folder structures, and topic-based grouping.

So far, I haven’t found a single best method for exploring large codebases. Based on my experience, I first check the GitHub repository, then use VSCode with git grep to find more detailed code references, such as:

yang@Mac repomix % git grep "buildTokenCountTree"
src/cli/reporters/tokenCountTreeReporter.ts:  buildTokenCountTree,
src/cli/reporters/tokenCountTreeReporter.ts:  const tree = buildTokenCountTree(filesWithTokens);
src/core/tokenCount/buildTokenCountStructure.ts:export const buildTokenCountTree = (filesWithTokens: FileWithTokens[]): TreeNode => {
tests/core/tokenCount/buildTokenCountStructure.test.ts:import { buildTokenCountTree, type FileWithTokens } from '../../../src/core/tokenCount/buildTokenCountStructure.js';
tests/core/tokenCount/buildTokenCountStructure.test.ts:  const root = buildTokenCountTree(filesWithTokens);
yang@Mac repomix % 
yang@Mac repomix % 
yang@Mac repomix % git grep "reportTokenCountTree"
src/cli/cliReport.ts:import { reportTokenCountTree } from './reporters/tokenCountTreeReporter.js';
src/cli/cliReport.ts:    reportTokenCountTree(packResult.processedFiles, packResult.fileTokenCounts, config);
src/cli/reporters/tokenCountTreeReporter.ts:export const reportTokenCountTree = (
tests/cli/reporters/tokenCountTreeReporter.test.ts:import { reportTokenCountTree } from '../../../src/cli/reporters/tokenCountTreeReporter.js';
tests/cli/reporters/tokenCountTreeReporter.test.ts:describe('reportTokenCountTree', () => {
tests/cli/reporters/tokenCountTreeReporter.test.ts:    reportTokenCountTree(processedFiles, fileTokenCounts, config);
tests/cli/reporters/tokenCountTreeReporter.test.ts:    reportTokenCountTree(processedFiles, fileTokenCounts, config);
tests/cli/reporters/tokenCountTreeReporter.test.ts:    reportTokenCountTree(processedFiles, fileTokenCounts, config);
tests/cli/reporters/tokenCountTreeReporter.test.ts:    reportTokenCountTree(processedFiles, fileTokenCounts, config);
yang@Mac repomix %                                
yang@Mac repomix % 
yang@Mac repomix % git grep "reportResults"       
src/cli/actions/defaultAction.ts:import { reportResults } from '../cliReport.js';
src/cli/actions/defaultAction.ts:    reportResults(cwd, result.packResult, result.config);
src/cli/cliReport.ts:export const reportResults = (cwd: string, packResult: PackResult, config: RepomixConfigMerged): void => {
tests/cli/actions/defaultAction.tokenCountTree.test.ts:  const mockReportResults = cliReport.reportResults as Mock;
yang@Mac repomix % 
Enter fullscreen mode Exit fullscreen mode

Following this approach helped me trace how the --token-count-tree feature works across different modules. I believe combining git grep with IDE search tools like those in VSCode is the best strategy for reading and understanding complex project workflows.

Top comments (0)