In this new feature, if users specify the option with --token-count-tree [threshold], then (N tokens) will be shown next to each file.
This feature helps users get an idea of whether they want to use an LLM for developing the file or project, as it shows how many tokens the file costs.
I built this functionality through the following steps:
1. Set up the optional feature in the `cli.ts` file.
2. Add this feature in `tree-structure.ts` with an interface and several small functions.
3. Display the token count `(N tokens)` in `output-builder.ts`, utilizing parameters and interfaces from `tree-structure.ts`.
Once I finished implementing this feature, I could easily see how many tokens each file costs. This helps me develop my project more efficiently with an LLM, as I can better manage token limits per session.
Although the idea is inspired by Repomix, the project structure, code, and display methods are different from Repomix.
However, I believe this feature is not fully complete yet, because I have not used any libraries to calculate the token count precisely. My current implementation estimates tokens based on spaces between words.
Next time, I plan to import a library to calculate token counts more accurately. With this improvement, I will enhance the Repo-snapshot CLI tool to provide precise token counts, enabling users to work more effectively with LLMs by having reliable token usage information.
Top comments (0)