Detecting LLM-Generated Text with Classical Machine Learning: Bridging the Gap Between Old and New

Originally published on tamiz.pro.

The Urgent Need for AI Text Detection

Modern large language models (LLMs) produce text indistinguishable from human writing in many cases. As these systems proliferate, detecting synthetic content has become critical for content moderation, academic integrity, and information security. While deep learning dominates current detection research, classical machine learning methods remain valuable for their interpretability, low computational cost, and effectiveness in constrained environments.

Why Classical ML Still Matters

Classical machine learning approaches offer three key advantages:

1. Explainability: Logistic regression coefficients provide clear insight into detection patterns
2. Efficiency: Models like Naive Bayes require minimal compute resources
3. Interoperability: Easier integration with legacy systems and real-time pipelines

These properties make classical approaches ideal for edge deployments or as complementary systems to deep learning detectors.

Feature Engineering for LLM Detection

Successful classical detection relies on extracting discriminative linguistic features. Key categories include:

1. N-gram Analysis

# Example: Extracting 3-gram frequencies
from collections import Counter

def extract_ngrams(text, n=3):
    tokens = text.lower().split()
    return [' '.join(tokens[i:i+n]) for i in range(len(tokens)-n+1)]

# Human text might show different n-gram distributions
human_ngrams = extract_ngrams("The quick brown fox jumps over the lazy dog")
ai_ngrams = extract_ngrams("The rapid brown fox leaps above the dormant canine")

LLMs often generate semantically coherent but statistically anomalous n-gram patterns compared to natural writing.

2. Syntax Metrics

Feature	Human Text	AI Text
Sentence Complexity	Higher variance	More uniform
Passive Voice Rate	15-20%	5-8%
Discourse Markers	3-5 per 100 words	0.5-1 per 100 words

These metrics can be calculated using libraries like syntok or nltk.

3. Statistical Anomalies

LLM outputs frequently display:

• More consistent sentence length (lower standard deviation)
• Reduced lexical diversity (lower Type-Token Ratio)
• Unnatural repetition patterns

Model Selection and Performance

Baseline Approach

1. Feature Selection: Combine 2000+ handcrafted features from linguistic patterns
2. Model Training: Logistic regression with L2 regularization
3. Evaluation: F1-score typically reaches 78-85% on standard datasets

Advanced Classical Methods

• SVM with TF-IDF weighted n-grams (82-88% F1)
• Random Forest with syntax features (76-84% F1)
• Gradient Boosting on combined features (85-90% F1)

These results approach but don't yet surpass deep learning models (93-97% F1), but maintain advantages in edge cases.

Challenges and Solutions

1. Distribution Shift

LLMs evolve rapidly while classical models require retraining. Solution: Use adaptive training pipelines that ingest new human/LLM samples weekly.

2. Feature Engineering Complexity

Manual feature creation is labor-intensive. Consider automated feature selection:

from sklearn.feature_selection import SelectKBest
selector = SelectKBest(score_func=chi2, k=500) # Select top 500 features
X_selected = selector.fit_transform(X, y)

3. Model Interpretability

Use SHAP values to visualize feature importance:

explainer = shap.LinearExplainer(model)
shap_values = explainer.shap_values(X_test)
shap.summary_plot(shap_values, X_test)

This helps validate that models are learning meaningful patterns (e.g., detecting unnatural conjunction usage).

Future Directions

1. Hybrid Approaches: Combine classical models with lightweight neural networks
2. Feature Evolution: Incorporate new LLM-specific metrics
3. Ensemble Systems: Create detection stacks that blend ML generations

While deep learning will dominate cutting-edge detection research, classical methods provide essential capabilities in deployment scenarios with limited compute resources. The best detection systems of the future will likely integrate both paradigms, using classical models for real-time filtering and deep learning for final classification.