This document explains how TonRadar Team developed a solution for the ML Competition 2023 organized by Telegram.
The goal of the competition is to create a library that can detect the programming language of a given source code snippet.
The complete source code and documentation are available on our GitHub repository: github.com/tonradar/contest-tglang.
Task Breakdown
We divided our work into three teams to win this contest. Each team had a specific role and responsibility:
- Crawler Team: This team was responsible for crawling source codes from GitHub and preparing an appropriate dataset. They ensured that the dataset had at least 1000 samples for each language.
- Model Team: This team was responsible for training a model using the prepared dataset and optimizing it to reach the maximum accuracy. They experimented with different architectures and hyperparameters to find the best model.
-
LibTgLang Team: This team was responsible for integrating the trained model as a
.solibrary that can be referenced in C. They also made sure that the library could predict the language in less than 10ms.
Source Code
You can find the resulting source code for each team:
- crawler: Crawling code files from GitHub (C#)
- model: Training the model (Python, AzureML)
- libtglang: Building the final .so file (C)
Language Selection
We analyzed the list of languages provided by the competition and decided to exclude some of them for the following reasons:
- Some languages are not widely used or not even real languages (e.g., 1S_ENTERPRISE, BISON).
- Some languages are very similar to other languages and can confuse the model (e.g., Delphi~Pascal and VISUAL_BASIC~VBSCRIPT).
- Some languages have very few or no source code files available on GitHub (e.g., HACK).
Here is the list of excluded languages:
- 1S_ENTERPRISE
- BISON
- HACK
- DELPHI (will map to PASCAL)
- VBSCRIPT (will map to VISUAL_BASIC)
We also chose to use .aspx files instead of .asp files for the ASP language, as .aspx is the newer and more popular framework.
Data Collection
We used GitHub as our main source of data for training our model. We developed a project called TgLang.CodeCrawler that can crawl GitHub and download 1000 source code files for each language. The project is written in C# and consists of two parts:
- TgLang.CodeCrawler: A console application that crawls GitHub and saves the files in a folder named after the language.
- TgLang.CodeCrawler.Test: A set of unit tests and integration tests to ensure the correctness and reliability of the crawler.
Model Training
We used Python and AzureML to train our model using the collected dataset. We chose a deep neural network architecture that can learn from the syntactic and semantic features of the source code. We saved our model in ONNX format so that we could use it in a C program to create the final library.
We trained the model using AzureML and here is the result:
During training, we observed that some languages caused confusion using the Confusion Matrix:
As you can see in this confusion matrix, some blue columns show a high possibility of confusing most languages with the language of that column.
These confusing languages are:
- JSON
- REGEX
- CSV
A common pattern in these languages is that they can be used within other languages. For example, a regex pattern or a JSON value can be used in any language.
Final Library
The final library is written in C and conforms to the specifications of the competition. It loads the ONNX model and uses it to predict the programming language of an input string. It returns the related enum of the predicted language as output.
Accuracy and Performance
We achieved an accuracy of 87.14% (Weighted Accuracy of 99.29%) and our function is answering in an average of 9ms, which is amazing!
Top comments (1)
Great work