DEV Community: Bahadir Kusat

What Is PEFT? A Guide to Parameter-Efficient Fine-Tuning

Bahadir Kusat — Mon, 20 Jul 2026 14:10:51 +0000

A technical guide comparing LoRA, QLoRA, rsLoRA, AdaLoRA, DoRA, IA³, prompt tuning, and adapter deployment workflows.

DEHA Research · July 16, 2026 · 18 min read

PEFT, or Parameter-Efficient Fine-Tuning, is a family of methods that adapts a large pretrained model to new tasks by training only a small subset of its parameters or lightweight components added to the model, rather than modifying all of its weights.

The objective is not merely to save GPU memory. PEFT also makes it possible to create small, portable, and task-specific adapters that share the same underlying base model.

In practice, the term PEFT is used in two different ways. First, it refers to the general family of methods that includes LoRA, IA³, and prompt tuning. Second, it refers to the open-source Hugging Face PEFT library, which implements these techniques alongside Transformers, Diffusers, and Accelerate.

This article first explains PEFT as an engineering approach, then examines the core concepts of the Hugging Face library and the decisions required for production deployment.

In Brief: What Is PEFT?

PEFT freezes most of the base model’s weights and trains only small adapters, scaling vectors, or learnable soft prompts.

Because the optimizer needs to store gradients and states for fewer parameters, training-memory requirements and checkpoint sizes are reduced.

However, PEFT is not a solution for poor-quality training data, and it does not guarantee the same performance as full fine-tuning for every task.

Why Was PEFT Developed?

During full fine-tuning, gradients and optimizer states must be maintained for every trainable weight in the model.

In a mixed-precision training run using AdamW, these states may consume several times more memory than the model weights themselves. Storing a complete model checkpoint for every individual task also becomes operationally expensive as model size increases.

PEFT preserves the general representations learned by the pretrained model while concentrating task-specific changes into a small parameter budget.

Hugging Face documentation states that this approach can substantially reduce the number of trainable parameters, computational requirements, and storage costs while achieving results close to full fine-tuning on many tasks.

The word close is important. Results depend on the model, task, dataset, PEFT method, and allocated parameter budget.

The Main Difference Between PEFT and Full Fine-Tuning Feature Full Fine-Tuning PEFT Parameters being trained The entire model or a large portion of it A small group of added or selected parameters Optimizer memory Very high Limited primarily to trainable parameters Checkpoint Complete model weights Usually a small adapter file Per-task deployment A separate full model may be required Adapters can be switched on the same base model Adaptation capacity Highest Limited by the selected method and parameter budget Base-model dependency May produce an independent full checkpoint Requires the correct base model and revision

PEFT does not mean that the model stops performing most computations.

Frozen layers do not require optimizer states, but the forward pass still runs through them, and a significant portion of the backward graph remains necessary to propagate gradients to the adapters.

Long-sequence activations, attention operations, and temporary kernel memory are still required.

PEFT may therefore make it possible to fine-tune a 7-billion-parameter model on a smaller GPU, but it does not make the maximum sequence length unlimited.

How Are PEFT Methods Classified?

Not every PEFT technique is a variation of LoRA.

The methods can be divided into four practical groups according to where and how they modify the model.

Weight- or Adapter-Based Methods

Methods such as LoRA, AdaLoRA, and DoRA represent weight updates using lightweight structures.

Activation Scaling

Methods such as IA³ use learnable vectors to scale attention and feed-forward activations.

Soft Prompting

Prompt tuning and prefix tuning add learnable continuous vectors to a frozen model.

Selective Tuning

These approaches train only selected existing parameters, such as biases, LayerNorm parameters, specific tokens, or subsets of model layers.

The method that trains the fewest parameters is not necessarily the best method.

When a task only requires steering capabilities that the model already possesses, a soft prompt may be sufficient.

When the goal is to teach a new output structure or produce a stronger behavioral change, LoRA applied to linear layers may provide greater flexibility.

IA³ may be considered when the available parameter and memory budget is extremely limited.

LoRA: The Most Widely Used PEFT Method

LoRA represents the update to a frozen weight matrix W as the product of two smaller matrices:

W′ = W + sBA

The rank r determines the narrow intermediate dimension between the two matrices.

For a linear layer with dimensions d_out × d_in, a full weight update requires:

d_out × d_in

trainable parameters.

LoRA adds approximately:

r × (d_in + d_out)

trainable parameters.

For example:

4096 × 4096 linear layer

Full weight:
16,777,216 parameters

LoRA with r = 16:
16 × (4096 + 4096) = 131,072 parameters

Approximately 128 times fewer trainable parameters
for this layer.

The total reduction depends on how many layers receive LoRA adapters and which modules are targeted.

The original LoRA study reported strong results across several tasks while substantially reducing the number of trainable parameters and the associated memory requirements.

The main LoRA configuration decisions include:

r,
lora_alpha,
target_modules,
dropout,
additional modules that must be saved.

Rank determines the adapter’s capacity, while alpha determines the scale of its contribution.

Is QLoRA a PEFT Method or a Quantization Method?

QLoRA combines two separate ideas.

The frozen base-model weights are loaded in 4-bit precision and temporarily dequantized to an appropriate compute precision during calculations. The trainable LoRA adapters remain in a higher precision.

Quantization therefore addresses how the base model is represented in memory, while LoRA determines which parameters are trained.

Simply loading a model in 4-bit precision is not fine-tuning.

Similarly, using LoRA without quantizing the base model is LoRA, not QLoRA.

QLoRA uses techniques such as:

NormalFloat 4-bit, or NF4,
double quantization,
paged optimizers.

Its purpose is to make it possible to adapt large models using more limited GPU resources.

Training with a 4-bit base model does not mean that the adapter checkpoint itself is stored in 4-bit precision.

Any quality difference introduced by quantization must also be measured separately for each model and task.

What Do rsLoRA, AdaLoRA, and DoRA Change? rsLoRA

Standard LoRA scales the adapter contribution using:

alpha / r

Rank-Stabilized LoRA, or rsLoRA, instead uses:

alpha / √r

This formulation is designed to prevent the learning signal from becoming excessively small at higher ranks.

It can be enabled in Hugging Face PEFT using:

use_rslora=True

The approach is particularly useful when comparing different ranks because it aims to prevent the standard scaling formula from confounding the effect of rank itself.

AdaLoRA

Standard LoRA generally assigns the same rank budget to most targeted matrices.

AdaLoRA evaluates the importance of different layers during training and distributes a limited parameter budget adaptively. It prunes singular values associated with less important updates.

The method is intended to use the available parameter budget more efficiently than uniform rank allocation, particularly when the budget is highly constrained.

DoRA

DoRA decomposes a weight into magnitude and directional components.

It uses LoRA to update the directional component while learning the magnitude component separately.

The objective is to reduce the difference in training dynamics between LoRA and full fine-tuning.

Research on DoRA has reported improvements over LoRA in language, vision, and multimodal tasks.

The trade-off is that its training compute and temporary memory consumption may be higher than those of vanilla LoRA.

IA³: Scaling Activations Instead of Updating Matrices

IA³ adds learnable scaling vectors to:

attention key activations,
attention value activations,
intermediate feed-forward activations.

Because it uses vectors instead of low-rank matrix updates, the number of trainable parameters can be even smaller than with LoRA.

The T-Few study introduced IA³ while demonstrating that parameter-efficient fine-tuning could outperform in-context learning in both accuracy and computational efficiency on certain few-shot tasks.

IA³ is attractive when extremely small adapters and rapid switching between tasks are required.

However, LoRA’s matrix-based adaptation capacity may be more flexible for complex behavioral transformations.

The decision should therefore be based not only on checkpoint size, but also on target-task evaluation and the serving architecture.

Prompt Tuning and Prefix Tuning

Soft-prompt methods do not generate human-readable prompt words.

Instead, they learn continuous vectors in the model’s embedding space through backpropagation.

Prompt Tuning

Prompt tuning adds learnable virtual tokens to the model input.

Research has shown that, as model scale increases, prompt tuning can close the performance gap with full-model tuning on some T5 tasks.

Prefix Tuning

Prefix tuning provides learnable prefix key and value representations to the attention mechanism in each Transformer layer.

The original study reported results close to full fine-tuning on table-to-text generation and summarization tasks while training approximately 0.1% of the model parameters.

The adapter files produced by these methods can be extremely small.

However, virtual tokens and prefixes may affect:

the amount of usable context,
the KV cache,
inference architecture.

A soft prompt is also not a natural-language prompt that can be transferred freely to a different tokenizer or a different base-model version.

Which PEFT Method Should Be Selected? Requirement First Candidate What Must Be Evaluated General instruction or formatting adaptation LoRA Target modules, rank, and alpha Adapting a large model on a single GPU QLoRA 4-bit quality difference and compute dtype Greater stability at high ranks rsLoRA Alpha and learning rate together Extremely limited parameter budget AdaLoRA or IA³ Real task capacity Reducing the quality gap between LoRA and full fine-tuning DoRA Additional compute and VRAM Very small per-task state Prompt or prefix tuning Model scale and context cost Adding new or frequently changing knowledge RAG first Information freshness and source requirements

PEFT is powerful for behavioral and task adaptation.

It should not be the default method for writing frequently changing organizational knowledge into model weights.

When information must remain current and its sources need to be shown, retrieval-augmented generation is often more appropriate.

PEFT and RAG may also be used together.

For example, an adapter can teach the desired answer structure while retrieval supplies current supporting evidence.

A Basic LoRA Workflow with Hugging Face PEFT from peft import LoraConfig, get_peft_model

config = LoraConfig(
task_type="CAUSAL_LM",
r=16,
lora_alpha=32,
lora_dropout=0.05,
target_modules=[
"q_proj",
"k_proj",
"v_proj",
"o_proj",
"gate_proj",
"up_proj",
"down_proj",
],
bias="none",
use_rslora=True,
)

model = get_peft_model(base_model, config)
model.print_trainable_parameters()

The names used in target_modules vary across model architectures and should not be copied blindly from a different model.

Before training begins, developers should inspect and record:

the list of trainable parameters,
the percentage of trainable parameters,
the data types of several adapter tensors,
the devices on which those tensors are located.

PEFT also supports adding, activating, and deleting adapters through its integration with Transformers.

When print_trainable_parameters() reports a much higher proportion than expected, the embedding layer, language-model head, or bias parameters may have been enabled unintentionally.

When the proportion is much lower than expected, the target-module names may not match the actual module names in the model.

A decreasing training loss does not, by itself, prove that the adapters were attached to the correct locations.

Saving and Loading Adapters and Base-Model Dependency

A PEFT checkpoint usually does not include the entire base model.

The adapter configuration describes:

the base-model identity,
the PEFT method and its settings,
the targeted modules.

The adapter weights contain only the small learned difference.

The following information should be versioned for production deployment:

base-model repository and exact revision or commit,
tokenizer files,
chat template,
additional special tokens,
PEFT and Transformers versions,
adapter configuration,
quantization configuration,
compute data type,
training-data version,
evaluation report,
licensing relationship.

Attaching an adapter to a base model with the same name but a different revision may damage behavior even when it does not produce a tensor-shape error.

When tokens have been added to the tokenizer, changes to the embedding or language-model head may also need to be stored.

In this situation, backing up only the adapter file is not sufficient.

Merging, Hotswapping, and Multiple Adapters

A LoRA adapter can be applied to the base weights dynamically at runtime or merged into them using:

merge_and_unload()

Merging removes the need for a separate adapter computation and produces a standalone model.

However, it reduces the ability to:

switch adapters dynamically,
share one base model across many adapters,
maintain small task-specific checkpoints.

Merging with a quantized model must be tested carefully with respect to data type and memory use.

Loading a separate adapter for each customer, language, or task on a shared base model can reduce storage requirements.

However, it introduces new operational risks, including:

adapter caching,
tenant authorization,
concurrent loading,
GPU-memory management,
accidentally selecting the wrong adapter.

Accepting an adapter identifier directly from a client without validating authorization could cause one customer’s behavioral adapter to be executed for another customer.

Different adapters can also be merged using weighted combinations.

However, two capabilities are not guaranteed to combine automatically or without quality loss. Their parameter-update directions may conflict.

PEFT provides model-merging approaches such as:

linear merging,
SVD-based merging,
TIES.

A merged model must be reevaluated independently on every source task.

Problems That PEFT Does Not Solve Poor Data

Training fewer parameters does not correct inaccurate, repetitive, contaminated, or leaked training data.

Hallucinations

An adapter does not guarantee factual accuracy. Retrieval may still be required for source-grounded information.

Long Context

PEFT does not automatically increase the context window or reduce the computational cost of attention.

A Poorly Matched Tokenizer

Adding LoRA adapters does not fully solve the problem of a tokenizer that is unsuitable for the target language.

Security

Malicious, unauthorized, or contaminated training data may introduce backdoors and unwanted behavioral changes.

Serving Cost

The inference cost of the base model largely remains in place.

PEFT may reduce the risk of catastrophic forgetting because fewer parameters are trainable, but it does not eliminate the risk.

An adapter can still steer the output of the base model very strongly.

The following should be compared with the adapter both enabled and disabled:

target-task performance,
general capabilities,
safety behavior,
multilingual performance,
long-context behavior.

Common PEFT Mistakes Copying target_modules from another training run without inspecting the model architecture. Starting training without printing and verifying the trainable parameters. Treating rank and alpha as the same concept or increasing both together with the learning rate without control. Failing to verify compute dtype, quantization dtype, and adapter dtype in QLoRA. Accidentally calculating loss over prompt or user tokens. Failing to version the base revision, tokenizer, and chat template together with the adapter. Sending the checkpoint with the lowest training loss directly to production. Failing to reevaluate the model after merging the adapter.
A Robust PEFT Experiment Plan Establish a baseline by measuring the base model’s zero-shot and few-shot performance. Divide the data into training, validation, and private test sets. Test the chat template and loss mask. Run a small initial LoRA experiment and record the number of trainable parameters and peak VRAM usage. Sweep the learning rate, rank, and target modules in a controlled sequence rather than changing them all simultaneously. When standard LoRA reaches its limits, compare rsLoRA, DoRA, or AdaLoRA under the same parameter or compute budget. Measure task success, general evaluations, safety, and latency with the adapter enabled and disabled. Benchmark merged and unmerged serving configurations using realistic traffic profiles. Lock the base revision, tokenizer, configuration, dataset version, and evaluation report into a single release manifest. Conclusion

PEFT is not a trick for training a large model “for free” on a small GPU.

It is a deliberate engineering trade-off among:

adaptation capacity,
memory use,
storage requirements,
deployment flexibility.

LoRA is a strong default starting point, but QLoRA, rsLoRA, AdaLoRA, DoRA, IA³, and soft-prompt methods target different bottlenecks.

The correct method should not be selected solely according to the percentage of trainable parameters. It should be selected according to real task performance and the production architecture.

For the complete model-development process, see our guide to training artificial intelligence models.

For details about learning rates, batches, VRAM management, and truncation, see our technical guide to LLM fine-tuning.

Frequently Asked Questions
What Does PEFT Stand For?

PEFT stands for Parameter-Efficient Fine-Tuning.

It adapts a model by training a small group of parameters rather than updating the entire model.

Are PEFT and LoRA the Same Thing?

No.

PEFT is the broader family of parameter-efficient adaptation techniques. LoRA is one of the most widely used methods within that family.

Other examples include:

IA³,
prefix tuning,
prompt tuning,
AdaLoRA,
DoRA.
What Is the Difference Between QLoRA and LoRA?

LoRA adds low-rank adapters to a frozen base model.

QLoRA also quantizes the base-model weights to 4-bit precision, reducing VRAM usage further.

Should a PEFT Adapter Be Merged?

When a single adapter will be used permanently, merging may simplify serving.

When multiple adapters or hotswapping are required on the same base model, keeping the adapter separate is more flexible.

Both configurations should be measured independently for quality and latency.

How Much VRAM Does PEFT Save?

There is no fixed percentage.

Gradient and optimizer-state memory may be reduced substantially, but memory is still required for:

base-model weights,
activations,
attention operations,
temporary tensors.

The actual saving depends on the model, PEFT method, rank, sequence length, batch size, and quantization configuration.

References
Hu, E. J. et al. (2021). LoRA: Low-Rank Adaptation of Large Language Models. arXiv:2106.09685.
Dettmers, T. et al. (2023). QLoRA: Efficient Finetuning of Quantized LLMs. arXiv:2305.14314.
Zhang, Q. et al. (2023). AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning. arXiv:2303.10512.
Liu, S.-Y. et al. (2024). DoRA: Weight-Decomposed Low-Rank Adaptation. arXiv:2402.09353.
Kalajdzievski, D. (2023). A Rank Stabilization Scaling Factor for Fine-Tuning with LoRA. arXiv:2312.03732.
Liu, H. et al. (2022). Few-Shot Parameter-Efficient Fine-Tuning Is Better and Cheaper than In-Context Learning. arXiv:2205.05638.
Li, X. L. and Liang, P. (2021). Prefix-Tuning: Optimizing Continuous Prompts for Generation. arXiv:2101.00190.
Lester, B. et al. (2021). The Power of Scale for Parameter-Efficient Prompt Tuning. arXiv:2104.08691.
Hugging Face. PEFT Documentation.

This article was prepared on July 16, 2026. The performance of each method depends on the selected model and task. Production decisions should be based on independent evaluation and realistic serving benchmarks.

Continue reading on DEVComunity:
https://dehayz.com/blog

LLM Fine-Tuning Guide: Full Fine-Tuning, LoRA, Learning Rate, and VRAM

Bahadir Kusat — Thu, 16 Jul 2026 21:30:33 +0000

From data preparation and tokenizer selection to pretraining, LoRA, RLHF, evaluation, and production monitoring, this guide covers the major stages involved in training an AI model.

Training an artificial intelligence model is not simply a matter of loading a dataset onto a GPU and running a few commands. A successful model requires a measurable objective, legally usable and carefully cleaned data, an architecture suited to the problem, controlled optimization, independent evaluation, and continuous monitoring after deployment.

In large language model development, a mistake in any one of these stages can waste millions of training examples and a significant amount of compute.

This guide explains the model development process primarily through the training of large language models. However, fundamental concepts such as dataset splitting, loss functions, overfitting, and evaluation also apply to computer vision, speech, and predictive models.

The goal is not to provide a single fixed recipe. Instead, it is to explain which training approach is appropriate for which problem and to clarify the cost difference between training a model from scratch and adapting an existing model.

In Brief: How Is an AI Model Trained?

First, the target task and success criteria are defined. Data is collected, reviewed for licensing and privacy, cleaned, and divided into training, validation, and test sets.

The model generates predictions from the input data. The difference between the prediction and the correct target is measured using a loss function. Backpropagation calculates how each parameter contributed to the error, and an optimization algorithm updates the parameters.

This process is repeated under controlled conditions until the model achieves acceptable results in independent tests and safety evaluations.

What Does Training a Model Actually Mean?

A neural network initially contains a large number of numerical parameters. During training, the model generates a prediction for a given input. This prediction is compared with the expected output, and the difference is measured using a loss function.

Backpropagation calculates how much each parameter contributed to the error. An optimization algorithm such as AdamW then updates the parameters in small steps intended to reduce the loss.

A training step can be summarized as follows:

data batch
↓
model prediction
↓
loss calculation
↓
backpropagation
↓
parameter update
↓
validation and logging

For language models, the most common pretraining objective is next-token prediction: given the preceding tokens, the model attempts to predict the next token.

In image classification, the target may be a class label. In speech recognition, the target is usually text. In regression, it may be a numerical value.

Training is therefore not a single algorithm. It is a general optimization process that uses different objectives depending on the problem being solved.

Choose the Right Method First: Not Every Project Should Start from Scratch

One of the most expensive mistakes in model development is selecting a training method before clearly defining the requirement.

A company that wants an assistant capable of answering questions from internal documents does not need to train a new foundation model. Similarly, training on billions of tokens is usually unnecessary when the goal is simply to create a brand-specific writing style.

Approach When to Use It Primary Cost or Risk
Prompting and tool use When the desired behavior is already within the capabilities of the existing model Lowest development cost, but consistency must be tested
RAG When current, private, or organization-specific information is required Retrieval quality, access control, and source attribution
SFT or LoRA When style, formatting, or task behavior must be changed persistently High-quality examples are required, and regressions may occur
Continued pretraining When a new language or specialized domain must be learned extensively Catastrophic forgetting, data-mixture problems, and high compute requirements
Pretraining from scratch When full control over the tokenizer, architecture, licensing, and model weights is required Highest data, engineering, infrastructure, and operational cost

LoRA reduces adaptation costs by freezing the original model weights and training smaller low-rank matrices. The original study showed that this method could dramatically reduce the number of trainable parameters at GPT-3 scale without introducing additional inference latency.

QLoRA further reduces memory requirements by representing the frozen base model in 4-bit precision.

Define the Objective and Success Criteria Before Training

Statements such as “the model should be good at Turkish” or “it should perform well in customer service” are not measurable objectives.

Instead, a project should define specific task groups, such as:

following instructions in Turkish,
summarizing long documents,
answering questions about product policies,
refusing unsafe requests correctly,
generating valid structured JSON,
using tools reliably,
citing retrieved sources accurately.

Each task should have clearly defined measurements. Depending on the application, these may include accuracy, source consistency, format validity, latency, inference cost, user success rate, and safety compliance.

It is also important to establish a baseline using a strong existing model before beginning training.

After training, the evaluation should measure not only improvements on the target task but also possible losses in general capabilities, performance differences across user groups, and changes in inference cost.

Otherwise, a model may improve on a particular benchmark while becoming worse in real-world use.

Training Data: Rights, Quality, and Representation Come Before Quantity

A robust data pipeline includes:

source inventory,
licensing and usage-right verification,
personal-data removal,
language detection,
quality filtering,
harmful-content policies,
deduplication,
dataset-mixture design.

A text being publicly accessible on the internet does not automatically mean that it can be used without restriction for model training.

The source license, terms of service, personal-data status, and applicable laws must be evaluated separately.

Duplicate data is not merely a waste of compute. Repeated examples can increase memorization and distort evaluation results when training data overlaps with benchmark or test data.

Research by Lee and colleagues showed that removing near-duplicate content from language-model datasets could achieve similar or better validation loss with fewer training steps.

A reliable data pipeline should include the following controls:

Source provenance: Record where each document came from, when it was collected, and under which license it may be used.
PII removal: Detect email addresses, phone numbers, identification numbers, confidential records, and other sensitive information before training.
Deduplication: Detect exact and near-duplicate content at both document and chunk level.
Language balance: Do not represent low-resource languages exclusively through translated content.
Test isolation: Prevent evaluation questions, answers, and close derivatives from entering the training pipeline.

Why Are the Tokenizer and Data Mixture Part of the Model Design?

A language model does not process text directly as words. It processes sequences of tokens.

When a tokenizer vocabulary is poorly suited to a particular language, the same meaning may require a much longer token sequence. This increases both training and inference costs while reducing the amount of useful content that fits within the model’s context window.

For agglutinative languages such as Turkish, tokenizer selection is therefore not a minor implementation detail. It is part of the data and architecture design.

A suitable tokenizer should represent common roots, suffixes, word forms, and domain-specific terminology efficiently. Subword methods such as Byte Pair Encoding generally provide better coverage than a purely word-level vocabulary.

However, some structures should remain atomic and should not be divided into multiple subword tokens. These may include:

ChatML control tokens such as <|im_start|> and <|im_end|>,
role markers,
tool-call delimiters,
end-of-sequence tokens,
structured output markers,
frequently used code expressions,
important domain-specific terms.

Preserving these structures as dedicated tokens can help the model learn message boundaries, conversation roles, tool-call formats, and structured generation more consistently.

The data mixture is just as important as the total number of tokens.

The proportions of web text, code, mathematics, academic material, conversational data, and domain-specific documents influence which capabilities the model develops.

The Llama 3 technical report presents dataset filtering, data-mixture selection through scaling experiments, and training on approximately 15 trillion multilingual tokens as interconnected parts of a single model-development system.

How Should Model Size, Token Budget, and Compute Be Planned?

A larger model is not always a better investment.

Scaling-law research by Kaplan and colleagues showed that model performance changes predictably with model size, dataset size, and compute.

The later Chinchilla study demonstrated that many large models had been undertrained relative to their parameter count. Under a fixed compute budget, model size and the number of training tokens must be scaled together.

In practice, teams should first run smaller pilot experiments and study their learning curves.

Before beginning a large training run, the following variables should be validated:

batch size,
learning rate,
warmup ratio,
learning-rate schedule,
sequence length,
optimizer configuration,
data-mixture proportions,
checkpoint frequency,
numerical precision.

The project should also estimate:

total token count,
approximate FLOPs,
GPU hours,
checkpoint size,
storage requirements,
network bandwidth,
evaluation cost,
expected failure and restart overhead.

Training cost is not limited to GPU rental. It also includes data preparation, failed experiments, storage, engineering work, evaluation, deployment, and continuous inference after the model enters production.

Which Technical Signals Should Be Monitored During Training?

Training loss alone is not sufficient.

When training loss decreases but validation loss does not, the model may be overfitting or memorizing the training data.

A sudden increase in gradient norm may indicate instability. Reduced token throughput may indicate an infrastructure bottleneck. Different loss patterns across data sources or languages may reveal a problem in the dataset mixture.

At minimum, the following signals should be recorded during training:

training and validation loss,
perplexity,
task-specific intermediate evaluations,
learning rate,
gradient norm,
weight norm,
numerical overflow and underflow events,
tokens processed per second,
GPU utilization,
memory consumption,
distributed communication time,
checkpoint duration,
sample and token distributions by language and source,
random seed,
code version,
dataset version,
all hyperparameters.

Mixed-precision and distributed training can reduce costs, but they also introduce new failure modes.

Checkpoints should be created regularly. Corrupted or incomplete checkpoints should be detected automatically, and the training pipeline should be able to resume safely without accidentally repeating or skipping large portions of the dataset.

From a Pretrained Model to an Assistant: SFT, Preference Training, and Safety

A pretrained language model learns statistical patterns in language, but it does not automatically behave like a safe and reliable assistant that follows user instructions.

During supervised fine-tuning, or SFT, the model is trained on carefully prepared instruction-and-response examples.

After SFT, developers may use preference ranking, reward modeling, reinforcement learning from human feedback, or other preference-optimization techniques.

The InstructGPT study showed that post-training with human feedback could make a smaller model more aligned with user preferences than a larger raw pretrained model.

Direct Preference Optimization, or DPO, introduced an alternative that directly optimizes the policy model from preference pairs without requiring a separate reward model and a complex reinforcement-learning loop.

Regardless of the selected method, preference data should distinguish between:

factual accuracy,
helpfulness,
safety,
relevance,
style,
instruction compliance.

When these criteria are mixed together carelessly, a model may learn that sounding confident or persuasive is more important than being correct.

Evaluation: A Benchmark Score Is Not the Same as Product Quality

A strong evaluation framework generally has three layers:

general-purpose benchmarks,
private tests designed for the target task,
realistic user scenarios.

Automated metrics provide scale and reproducibility. Human evaluation captures nuance. Red-team testing searches for vulnerabilities such as:

prompt injection,
sensitive-data leakage,
unsafe guidance,
hallucinated citations,
unauthorized tool use,
privilege escalation,
failure to follow access-control rules.

When benchmark questions have entered the training data, the model may memorize their answers instead of demonstrating genuine reasoning or generalization.

Research on data contamination in modern LLM benchmarks has shown that benchmark scores may overestimate real-world generalization performance.

To reduce this risk, evaluation pipelines may use:

time-based dataset cutoffs,
canary examples,
similarity searches,
unpublished test sets,
independently created adversarial cases.

Common Mistakes in Model Training Using Fine-Tuning to Solve the Wrong Problem

Do not fine-tune a model to solve a current-information problem that could be handled more effectively through retrieval-augmented generation.

Allowing the Model to See the Test Data

Do not select training checkpoints or repeatedly adjust hyperparameters based directly on final test-set results.

Failing to Track Data Sources

Without source provenance, it becomes difficult or impossible to manage licenses, deletion requests, contamination analysis, and dataset updates.

Looking Only at Average Performance

Average scores can hide poor performance in particular languages, domains, demographic groups, or high-risk scenarios.

Ignoring Base-Model Regressions

A model may improve on the target task while losing general knowledge, reasoning ability, safety behavior, multilingual performance, or formatting reliability.

Forgetting Serving Costs

A model may be technically trainable but too slow, memory-intensive, or expensive to operate in production.

A Realistic Model-Development Roadmap for Small Teams Define one user problem and a measurable acceptance test. Establish a baseline using a strong existing model, prompt engineering, and RAG. Classify failures as knowledge, behavior, tool-use, formatting, or safety problems. Prepare high-quality SFT data only when a persistent behavioral problem remains. Run a small LoRA or QLoRA experiment before considering full fine-tuning. Keep private evaluation, human assessment, and safety testing independent from training. Deploy through a limited canary release and monitor cost, latency, reliability, and user success in production.

At DEHA, model development is not treated as weight training alone.

Our Turkish-language model research is designed together with tool calling, retrieval with source attribution, workspace isolation, quality control, and measurable Turkish evaluation pipelines.

From the user’s perspective, value does not come from a benchmark table. It comes from completing the requested task correctly, reliably, and securely.

Full fine-tuning, LoRA rank and alpha selection, learning-rate scheduling, gradient accumulation, paged optimizers, and sequence-length configuration are covered in greater detail in our technical guide to LLM fine-tuning.

Conclusion

Training an artificial intelligence model is a broader engineering discipline than simply combining data, optimization, and GPUs.

The first question should be whether the problem genuinely requires training. If it does, data rights, data quality, tokenizer design, compute budgets, post-training, safety, and evaluation must be designed as parts of the same system.

The largest model or the longest training run cannot compensate for a poorly defined objective.

For most teams, the best starting point is not to train a new foundation model from scratch. A more practical approach is to build a measurable system on top of a strong base model, connect external knowledge through RAG, and apply efficient fine-tuning only when a clearly demonstrated behavioral gap remains.

Training from scratch becomes reasonable only when the strategic value of full control over the data, licensing, tokenizer, architecture, weights, infrastructure, and long-term inference exceeds the associated cost.

Frequently Asked Questions
What Is Required to Train an Artificial Intelligence Model?

A clearly defined task, legally usable and cleaned data, an appropriate base model or architecture, training infrastructure, an evaluation dataset, experiment tracking, and safety testing are required.

Should an LLM Be Trained from Scratch?

For most products, no.

Prompt engineering, RAG, and LoRA-based adaptation are generally faster and more economical. Training from scratch should be considered when there is a strategic requirement for full control over the tokenizer, model architecture, licensing, and weights.

What Is the Difference Between Fine-Tuning and RAG?

Fine-tuning changes a model’s behavior or task capability by updating its weights.

RAG retrieves current or private information from an external source at query time. It is generally more suitable when the primary requirement is keeping information accurate and up to date.

How Many GPUs Are Required for Model Training?

There is no single answer.

LoRA training for a relatively small model may be performed on one GPU. Pretraining a large foundation model from scratch may require hundreds or thousands of accelerators.

The required infrastructure depends on the model size, token count, sequence length, numerical precision, desired training duration, and distributed-training strategy.

References
Hoffmann, J. et al. (2022). Training Compute-Optimal Large Language Models. arXiv:2203.15556.
Dubey, A. et al. (2024). The Llama 3 Herd of Models. arXiv:2407.21783.
Lee, K. et al. (2021). Deduplicating Training Data Makes Language Models Better. arXiv:2107.06499.
Hu, E. J. et al. (2021). LoRA: Low-Rank Adaptation of Large Language Models. arXiv:2106.09685.
Dettmers, T. et al. (2023). QLoRA: Efficient Finetuning of Quantized LLMs. arXiv:2305.14314.
Ouyang, L. et al. (2022). Training Language Models to Follow Instructions with Human Feedback. arXiv:2203.02155.
Rafailov, R. et al. (2023). Direct Preference Optimization: Your Language Model Is Secretly a Reward Model. arXiv:2305.18290.
Deng, C. et al. (2023). Investigating Data Contamination in Modern Benchmarks for Large Language Models. arXiv:2311.09783.

This article was prepared on July 14, 2026. Technical concepts have been simplified where necessary to improve readability. The references point to primary research publications. Data licensing and personal-data requirements may require separate legal assessment depending on the project and jurisdiction.

Continue reading on DEVComunity:
https://dehayz.com/blog

How Is an Artificial Intelligence Model Trained? An LLM Training Guide

Bahadir Kusat — Thu, 16 Jul 2026 21:06:46 +0000

From data preparation and tokenizer selection to pretraining, LoRA, RLHF, evaluation, and production monitoring, this guide covers the major stages involved in training an AI model.

DEHA Research · July 14, 2026 · 18 min read

In large language model development, a mistake in any one of these stages can waste millions of training examples and a significant amount of compute.

In Brief: How Is an AI Model Trained?

First, the target task and success criteria are defined. Data is collected, reviewed for licensing and privacy, cleaned, and divided into training, validation, and test sets.

This process is repeated under controlled conditions until the model achieves acceptable results in independent tests and safety evaluations.

What Does Training a Model Actually Mean?

Backpropagation calculates how much each parameter contributed to the error. An optimization algorithm such as AdamW then updates the parameters in small steps intended to reduce the loss.

A training step can be summarized as follows:

data batch
↓
model prediction
↓
loss calculation
↓
backpropagation
↓
parameter update
↓
validation and logging

For language models, the most common pretraining objective is next-token prediction: given the preceding tokens, the model attempts to predict the next token.

In image classification, the target may be a class label. In speech recognition, the target is usually text. In regression, it may be a numerical value.

Training is therefore not a single algorithm. It is a general optimization process that uses different objectives depending on the problem being solved.

Choose the Right Method First: Not Every Project Should Start from Scratch

One of the most expensive mistakes in model development is selecting a training method before clearly defining the requirement.

QLoRA further reduces memory requirements by representing the frozen base model in 4-bit precision.

Define the Objective and Success Criteria Before Training

Statements such as “the model should be good at Turkish” or “it should perform well in customer service” are not measurable objectives.

Instead, a project should define specific task groups, such as:

It is also important to establish a baseline using a strong existing model before beginning training.

Otherwise, a model may improve on a particular benchmark while becoming worse in real-world use.

Training Data: Rights, Quality, and Representation Come Before Quantity

A robust data pipeline includes:

source inventory,
licensing and usage-right verification,
personal-data removal,
language detection,
quality filtering,
harmful-content policies,
deduplication,
dataset-mixture design.

A text being publicly accessible on the internet does not automatically mean that it can be used without restriction for model training.

The source license, terms of service, personal-data status, and applicable laws must be evaluated separately.

Duplicate data is not merely a waste of compute. Repeated examples can increase memorization and distort evaluation results when training data overlaps with benchmark or test data.

Research by Lee and colleagues showed that removing near-duplicate content from language-model datasets could achieve similar or better validation loss with fewer training steps.

A reliable data pipeline should include the following controls:

Why Are the Tokenizer and Data Mixture Part of the Model Design?

A language model does not process text directly as words. It processes sequences of tokens.

For agglutinative languages such as Turkish, tokenizer selection is therefore not a minor implementation detail. It is part of the data and architecture design.

However, some structures should remain atomic and should not be divided into multiple subword tokens. These may include:

Preserving these structures as dedicated tokens can help the model learn message boundaries, conversation roles, tool-call formats, and structured generation more consistently.

The data mixture is just as important as the total number of tokens.

The proportions of web text, code, mathematics, academic material, conversational data, and domain-specific documents influence which capabilities the model develops.

How Should Model Size, Token Budget, and Compute Be Planned?

A larger model is not always a better investment.

Scaling-law research by Kaplan and colleagues showed that model performance changes predictably with model size, dataset size, and compute.

In practice, teams should first run smaller pilot experiments and study their learning curves.

Before beginning a large training run, the following variables should be validated:

batch size,
learning rate,
warmup ratio,
learning-rate schedule,
sequence length,
optimizer configuration,
data-mixture proportions,
checkpoint frequency,
numerical precision.

The project should also estimate:

total token count,
approximate FLOPs,
GPU hours,
checkpoint size,
storage requirements,
network bandwidth,
evaluation cost,
expected failure and restart overhead.

Which Technical Signals Should Be Monitored During Training?

Training loss alone is not sufficient.

When training loss decreases but validation loss does not, the model may be overfitting or memorizing the training data.

At minimum, the following signals should be recorded during training:

Mixed-precision and distributed training can reduce costs, but they also introduce new failure modes.

From a Pretrained Model to an Assistant: SFT, Preference Training, and Safety

A pretrained language model learns statistical patterns in language, but it does not automatically behave like a safe and reliable assistant that follows user instructions.

During supervised fine-tuning, or SFT, the model is trained on carefully prepared instruction-and-response examples.

After SFT, developers may use preference ranking, reward modeling, reinforcement learning from human feedback, or other preference-optimization techniques.

The InstructGPT study showed that post-training with human feedback could make a smaller model more aligned with user preferences than a larger raw pretrained model.

Regardless of the selected method, preference data should distinguish between:

factual accuracy,
helpfulness,
safety,
relevance,
style,
instruction compliance.

When these criteria are mixed together carelessly, a model may learn that sounding confident or persuasive is more important than being correct.

Evaluation: A Benchmark Score Is Not the Same as Product Quality

A strong evaluation framework generally has three layers:

general-purpose benchmarks,
private tests designed for the target task,
realistic user scenarios.

Automated metrics provide scale and reproducibility. Human evaluation captures nuance. Red-team testing searches for vulnerabilities such as:

prompt injection,
sensitive-data leakage,
unsafe guidance,
hallucinated citations,
unauthorized tool use,
privilege escalation,
failure to follow access-control rules.

When benchmark questions have entered the training data, the model may memorize their answers instead of demonstrating genuine reasoning or generalization.

Research on data contamination in modern LLM benchmarks has shown that benchmark scores may overestimate real-world generalization performance.

To reduce this risk, evaluation pipelines may use:

time-based dataset cutoffs,
canary examples,
similarity searches,
unpublished test sets,
independently created adversarial cases.

Common Mistakes in Model Training Using Fine-Tuning to Solve the Wrong Problem

Do not fine-tune a model to solve a current-information problem that could be handled more effectively through retrieval-augmented generation.

Allowing the Model to See the Test Data

Do not select training checkpoints or repeatedly adjust hyperparameters based directly on final test-set results.

Failing to Track Data Sources

Without source provenance, it becomes difficult or impossible to manage licenses, deletion requests, contamination analysis, and dataset updates.

Looking Only at Average Performance

Average scores can hide poor performance in particular languages, domains, demographic groups, or high-risk scenarios.

Ignoring Base-Model Regressions

A model may improve on the target task while losing general knowledge, reasoning ability, safety behavior, multilingual performance, or formatting reliability.

Forgetting Serving Costs

A model may be technically trainable but too slow, memory-intensive, or expensive to operate in production.

A Realistic Model-Development Roadmap for Small Teams Define one user problem and a measurable acceptance test. Establish a baseline using a strong existing model, prompt engineering, and RAG. Classify failures as knowledge, behavior, tool-use, formatting, or safety problems. Prepare high-quality SFT data only when a persistent behavioral problem remains. Run a small LoRA or QLoRA experiment before considering full fine-tuning. Keep private evaluation, human assessment, and safety testing independent from training. Deploy through a limited canary release and monitor cost, latency, reliability, and user success in production.

At DEHA, model development is not treated as weight training alone.

Our Turkish-language model research is designed together with tool calling, retrieval with source attribution, workspace isolation, quality control, and measurable Turkish evaluation pipelines.

From the user’s perspective, value does not come from a benchmark table. It comes from completing the requested task correctly, reliably, and securely.

Conclusion

Training an artificial intelligence model is a broader engineering discipline than simply combining data, optimization, and GPUs.

The largest model or the longest training run cannot compensate for a poorly defined objective.

Frequently Asked Questions
What Is Required to Train an Artificial Intelligence Model?

Should an LLM Be Trained from Scratch?

For most products, no.

What Is the Difference Between Fine-Tuning and RAG?

Fine-tuning changes a model’s behavior or task capability by updating its weights.

RAG retrieves current or private information from an external source at query time. It is generally more suitable when the primary requirement is keeping information accurate and up to date.

How Many GPUs Are Required for Model Training?

There is no single answer.

LoRA training for a relatively small model may be performed on one GPU. Pretraining a large foundation model from scratch may require hundreds or thousands of accelerators.

The required infrastructure depends on the model size, token count, sequence length, numerical precision, desired training duration, and distributed-training strategy.

Explore DEHA

Experience a Turkish-first artificial intelligence workspace.

Use document analysis, web and academic search, article generation, presentations, and source-grounded conversations within a single workspace.

Continue reading on DEVComunity:
https://dehayz.com/blog

What Is Turkish-Language AI? Tokenizers, Training Data, and Language Model Development

Bahadir Kusat — Tue, 14 Jul 2026 21:46:54 +0000

A Turkish-language interface and an artificial intelligence system developed around the linguistic structure of Turkish are not the same thing. This article examines the difference in light of academic research.

Turkish-language AI is not simply software with Turkish menus or a system capable of answering questions in Turkish. More precisely, it refers to AI systems that represent Turkish text efficiently, are trained or adapted using natural Turkish data, account for the language’s morphology and usage contexts, and are evaluated through Turkish-specific benchmarks.

This distinction matters because the ability of a multilingual model to generate Turkish text does not necessarily mean that it processes Turkish as efficiently as English or performs reliably on tasks grounded in the cultural, institutional, and linguistic context of Türkiye.

Research indicates that tokenizer selection can affect training costs and downstream task performance, that data quality may be just as important as data volume, and that Turkish requires independent evaluation datasets with linguistic and cultural validity.

The Brief Answer: What Defines Turkish-Language AI?

Four layers should be considered together:

The tokenizer and vocabulary
Turkish training data
The model’s training or adaptation method
Turkish-specific evaluation

Improving any one of these layers can be beneficial. However, improving only one of them is not sufficient to demonstrate that a model genuinely understands Turkish.

What Is Turkish-Language AI, and What Is It Not?

In everyday usage, the term “Turkish AI” may refer to three different kinds of systems.

The first is a system in which a Turkish interface has been added to a model primarily trained in another language.

The second is a multilingual model that is capable of communicating in Turkish.

The third is a system that directly targets Turkish when making decisions about tokenization, training data, model adaptation, and evaluation.

From a technical perspective, the third definition is the most meaningful.

This distinction also separates the concepts of being “locally developed” and being “proficient in Turkish.” A model may have been developed in Türkiye without its Turkish performance ever being demonstrated through independent evaluations.

Conversely, a multinational model may perform well on some Turkish tasks because it was trained on substantial Turkish data and uses a tokenizer that represents Turkish efficiently.

The model’s origin should therefore not be treated as a substitute for measurable language proficiency.

What Is a Tokenizer, and Why Is It Critical for Turkish?

Language models do not process text directly as complete words. Text is first divided into numerical units called tokens.

Common tokenization methods such as Byte Pair Encoding, WordPiece, and Unigram construct a vocabulary by learning frequently occurring character sequences from training corpora. This process does not necessarily involve learning grammatical rules. It is generally based on statistical frequency.

Turkish is a highly productive agglutinative language. A single written word such as:

evlerinizdekilerden

can encode a root together with plural, possessive, locative, nominalizing, and ablative information.

A tokenizer does not always divide such structures along meaningful linguistic boundaries. As a result, the same root or suffix may be divided inconsistently across different words, and equivalent information may require longer token sequences in Turkish than in other languages.

A study by Ali et al. comparing 24 monolingual and multilingual models found that using English-centric tokenizers in multilingual models could substantially reduce downstream performance and increase training costs because of inefficient vocabulary allocation.

However, the study also provides an important warning: simple tokenizer metrics such as fertility, meaning the number of tokens per word, and parity do not reliably predict overall model quality on their own.

The conclusion should therefore not be that fewer tokens always produce a better model (Ali et al., 2024).

Why Does Token Efficiency Affect Users?

Cost: Many AI APIs charge according to the number of input and output tokens. A language that requires more tokens to represent the same information may therefore be more expensive to process.

Context capacity: When the same information occupies more tokens, less space remains within the model’s context window for documents, instructions, or conversation history.

Latency: Longer token sequences require more computational steps, particularly during output generation.

Linguistic representation: Token divisions that consistently conflict with morpheme boundaries may make it more difficult for a model to learn relationships between word roots and suffixes.

A large-scale study published in 2025 reported that performance differences in morphologically complex languages cannot be explained solely through tokenizer alignment. Dataset size, data composition, and evaluation methodology must also be considered together (Arnett et al., 2025).

The correct conclusion of the Turkish tokenization debate is therefore not that a morphology-aware tokenizer will solve every problem. Rather, tokenization should be understood as one of the fundamental design decisions in developing a Turkish-language model.

Why Is Turkish Training Data at Least as Important as the Tokenizer?

A language model does not learn a language by memorizing a grammar textbook. It learns from the patterns contained in its training data.

When Turkish examples are limited or consist mainly of machine-translated material, the model may learn everyday speech, professional terminology, formal correspondence, regional expressions, and cultural references unevenly.

Having a large amount of data is not sufficient on its own. Duplicated content, machine-generated text, corrupted character encoding, personal data, licensing problems, and domain imbalance can all reduce the quality of a training corpus.

A strong Turkish training corpus should contain a balanced mixture of natural texts from several domains, including:

News
Literature
Academic publications
Legislation and regulatory documents
Technical documentation
Conversational language
Profession-specific material
Texts from different social and cultural contexts

During data cleaning, Turkish characters, punctuation, named entities, and meaningful formatting should be preserved. Duplicated material should be reduced, and both the source and licensing status of each dataset should remain traceable.

Collecting personal or sensitive data indiscriminately is not only a quality problem. It also creates legal and ethical risks.

The SindBERT study published in 2026 illustrates this point. Although the large-scale model trained on 312 GB of Turkish text achieved strong results, the authors’ comparisons with smaller and more carefully curated models suggested that data quality and diversity may be more important than raw volume alone (Schmitt & Schweter, 2026).

What Challenges Does the Structure of Turkish Create for Language Models?

Turkish belongs to the Turkic language family and has highly productive agglutinative morphology.

New word forms can be created by adding several suffixes sequentially to a root. Vowel harmony and consonant alternations can cause the same suffix to appear in different surface forms.

Because person and subject information can often be inferred from verb conjugation, an explicit subject may be omitted from a sentence.

The basic word order is generally subject–object–verb. However, information structure, emphasis, and discourse context allow Turkish word order to remain relatively flexible.

These characteristics do not make Turkish impossible for AI systems to process. They do, however, require models to encounter sufficient and diverse examples, represent suffix sequences consistently, and be evaluated using more than translated English benchmarks.

The TurBLiMP study introduced a specialized benchmark consisting of automatically generated and expert-validated minimal pairs for evaluating Turkish grammar.

Unlike broad knowledge tests, TurBLiMP can directly measure linguistic phenomena such as:

Agreement
Case marking
Negation
Syntactic dependencies
Morphological compatibility

This makes it possible to evaluate Turkish grammatical competence more directly (Başar et al., 2025).

What Stages Are Involved in Developing a Turkish-Language Model?

Training a model entirely from scratch is not the only available option.

Depending on the project’s objectives, budget, data, and computational resources, four approaches may be used independently or together.

Pre-Training From Scratch

Both the tokenizer and the model are trained from the beginning using Turkish-dominant data.

This approach provides the greatest level of control, but it also requires the largest amount of data, computing infrastructure, engineering effort, and evaluation work.

Continual Pre-Training

An existing foundation model continues training on a clean Turkish corpus.

Continual pre-training can improve the model’s representation of the Turkish language, local knowledge, professional terminology, or a specific domain without requiring the entire model to be trained from the beginning.

Task Adaptation

Supervised fine-tuning or parameter-efficient methods such as LoRA can be used to teach Turkish instruction following, conversational behavior, output formats, or specialized tasks.

This stage should not be confused with language acquisition. A model may be trained to follow instructions in Turkish without having received enough Turkish data during pre-training to develop strong general language competence.

Retrieval-Based Augmentation

Current or organization-specific information can be supplied at inference time through search systems and retrieval-augmented generation rather than being embedded into the model through additional training.

This approach is particularly useful for:

Frequently changing information
Institutional documents
Internal company knowledge
Legislation
Academic sources
Product documentation

The MODA study showed that a modular approach separating continual pre-training on a Turkish web corpus from parameter-efficient task adaptation could outperform both base models and models trained only through instruction tuning on Turkish benchmarks (Bayar et al., 2026).

This result is particularly important because it emphasizes that language acquisition and task alignment are not the same process.

How Should Turkish-Language AI Be Evaluated?

A few fluent-looking responses do not constitute a scientific evaluation.

A Turkish-language model should be evaluated separately across several dimensions:

Grammar
General and local knowledge
Reasoning
Summarization
Safety
Instruction following
Long-context comprehension
Professional and technical domains
Cultural appropriateness

Benchmarks translated from English do not always preserve the educational, institutional, and cultural context of Türkiye.

TurkishMMLU addresses this need with more than 10,000 questions prepared by experts using the Turkish high-school curriculum. It is one of the first large-scale Turkish multitask evaluation benchmarks.

The researchers also note that automatically translated evaluations may contain errors and cultural biases (Yüksel et al., 2024).

TR-MMLU provides another Turkish evaluation framework containing 6,200 questions across 62 subject areas (Bayram et al., 2025).

Evaluation datasets must themselves be audited carefully. Surface-level characteristics such as morpheme count, subword count, or sentence length may influence model scores.

A morphology-aware study published in 2026 found that alignment between tokenizer boundaries and morphological boundaries may be broadly associated with model performance. However, this relationship alone does not establish causality (Başar & Bisazza, 2026).

Reliable evaluation therefore requires more than a single score or benchmark.

Applications of Turkish-Language AI

Language proficiency does not merely produce more natural conversations. It directly affects the quality of real-world AI products and workflows.

Accurately extracting entities and relationships from Turkish documents, preserving the semantic roles carried by suffixes in legal or public-sector texts, adjusting formality in customer communication, generating curriculum-aligned educational content, and synthesizing academic sources in natural Turkish all depend on strong language competence.

Document and PDF Analysis

A Turkish-capable system can answer questions, summarize content, locate evidence, and compare claims across long Turkish reports and documents.

Academic Research

It can synthesize international literature in Turkish while preserving genuine DOI records, citations, and source relationships.

Institutional Workspaces

It can maintain separate project, customer, regulatory, or departmental contexts without mixing information between them.

Voice and Visual Interaction

It can combine Turkish speech, images, documents, and written instructions within the same task.

Content Generation

It can produce articles, reports, and presentations while following both Turkish linguistic conventions and the structural requirements of the requested format.

What Should Users Consider When Choosing a Turkish AI System?

For most users, the number of model parameters is less informative than the answers to the following questions:

Has the system’s Turkish performance been measured through independent evaluations?
Does it provide real and verifiable links when sources are requested?
Can it preserve context in long Turkish documents?
Does it use search tools when current information is required?
Is user data used for model training?
Can it communicate uncertainty when the available evidence is insufficient?
Has it been evaluated across Turkish grammar, culture, and professional domains?
Can it separate different projects and information sources reliably?

DEHA’s approach is based on the same distinction.

Turkish-first model development is considered together with web and academic search, document-grounded Source Studio workspaces, persistent but isolated workspace memory, and article and presentation generation workflows.

The objective is not merely to generate Turkish sentences. It is to support an entire professional workflow conducted in Turkish from beginning to end.

Conclusion

Turkish-language AI is not a single model name or a marketing label.

It is an engineering and research field that covers how a tokenizer divides Turkish text, the quality and diversity of the training data, model adaptation methods, retrieval systems, and culturally valid evaluation benchmarks.

The agglutinative structure of Turkish increases the importance of this work. However, reducing model quality to token count alone would not be scientifically accurate.

A reliable Turkish-language system must consider several factors together:

Linguistic efficiency
Data quality and diversity
Task performance
Source reliability
Cultural validity
User privacy
Transparent evaluation

The future of Turkish-language AI lies in moving beyond the simple claim that “Turkish is supported” and toward measurable, transparent, independently evaluated, and properly sourced Turkish proficiency.

References
Ali, M., et al. (2024). Tokenizer Choice for LLM Training: Negligible or Crucial? Findings of NAACL 2024.
Arnett, C., et al. (2025). Why Do Language Models Perform Worse for Morphologically Complex Languages? COLING 2025.
Başar, E., et al. (2025). TurBLiMP: A Turkish Benchmark of Linguistic Minimal Pairs. EMNLP 2025.
Yüksel, A., et al. (2024). TurkishMMLU: Measuring Massive Multitask Language Understanding in Turkish. arXiv:2407.12402.
Bayram, M. A., et al. (2025). Setting Standards in Turkish NLP: TR-MMLU for Large Language Model Evaluation. arXiv:2501.00593.
Bayar, A. E., et al. (2026). Building a Turkish Large Language Model via Continual Pre-Training and Parameter-Efficient Adaptation. SIGTURK 2026.
Başar, E., & Bisazza, A. (2026). A Morphology-Aware Evaluation of Turkish Syntax in Large Language Models. SIGTURK 2026.
Toraman, Ç., et al. (2026). TurkBench: A Benchmark for Evaluating Turkish Large Language Models. arXiv:2601.07020.
Schmitt, R., & Schweter, S. (2026). SindBERT, the Sailor: Charting the Seas of Turkish NLP. SIGTURK 2026.

This article was last updated on July 12, 2026. DOI references point to publisher records, while arXiv references point to the corresponding preprint records. Preprints do not have the same publication status as peer-reviewed studies.

Continue reading on DEVComunity:
https://dehayz.com/blog

What Is GPT? A Practical Guide to Tokens, Transformers, Training, and Fine-Tuning

Bahadir Kusat — Tue, 14 Jul 2026 20:21:22 +0000

Artificial intelligence systems can now write articles, explain scientific concepts, generate software code, summarize documents, and participate in remarkably natural conversations. At the center of this development is a class of language models commonly associated with three letters: GPT.

Despite its widespread use, GPT is often described too simply. It is not merely a chatbot, a search engine, or a database containing prepared answers. GPT is a neural language model trained to process sequences of tokens and predict what should come next.

Understanding GPT therefore requires looking beyond the chat interface. We need to examine tokenization, Transformer architecture, pre-training, parameters, post-training, and the statistical process through which a model produces language.

What Does GPT Stand For?

GPT stands for Generative Pre-trained Transformer. Each word describes a fundamental part of the system.

Generative means that the model can produce new sequences, such as text, code, structured data, or other token-based outputs.

Pre-trained means that the model first learns general patterns from a large collection of data before it is adapted for specific tasks or conversational behavior.

Transformer refers to the neural-network architecture on which GPT is based.

The Transformer architecture was introduced by Vaswani and colleagues in the 2017 paper Attention Is All You Need. Unlike earlier sequence models that depended heavily on recurrent neural networks, the Transformer used attention mechanisms to process relationships between elements in a sequence more efficiently and in parallel.

The original GPT research applied generative pre-training to a Transformer-based language model. The central idea was to first train a general-purpose model on unlabelled text and then adapt it to downstream language tasks. This combination of large-scale pre-training and task-specific adaptation became one of the foundations of modern natural language processing.

GPT Does Not Read Text Directly

Before text can be processed by GPT, it must be converted into smaller units called tokens.

A token is not necessarily a complete word. Depending on the tokenizer, a token may represent:

A complete word
Part of a word
A punctuation mark
A number
A whitespace pattern
A byte or character sequence

For example, a tokenizer might represent a common word with one token while dividing an uncommon technical term into several subword tokens. The exact division depends on the tokenizer’s vocabulary and training method.

Subword tokenization methods became important because a fixed word-level vocabulary cannot efficiently represent every possible word, spelling variation, technical term, or newly created expression. Byte Pair Encoding, commonly abbreviated as BPE, was adapted for neural language processing to represent rare words as sequences of smaller subword units.

A simplified GPT processing pipeline looks like this:

User text
↓
Tokenizer
↓
Token IDs
↓
Token embeddings
↓
Transformer layers
↓
Probability distribution over the vocabulary
↓
Selected next token
↓
Generated text

After tokenization, each token is mapped to a numerical identifier. The model then converts these identifiers into vectors known as embeddings. These vectors provide the mathematical representations that the Transformer processes.

Tokenization is not a minor preprocessing detail. It affects context length, multilingual performance, numerical representation, generation speed, and the model’s ability to process domain-specific terminology.

The Central Objective: Predict the Next Token

At the core of a GPT model is a deceptively simple training objective: predict the next token from the tokens that came before it.

Given a sequence of tokens

[
x_1, x_2, x_3, \ldots, x_t,
]

the model estimates the probability of the next token:

[
P(x_t \mid x_1, x_2, \ldots, x_{t-1}).
]

During training, the model repeatedly compares its prediction with the actual next token in the training data. The difference between the prediction and the correct answer is measured through a loss function, commonly cross-entropy loss. The model’s parameters are then adjusted to reduce that error.

A simplified language-model training objective can be expressed as:

[
\mathcal{L} = -\sum_{t=1}^{T}\log P(x_t \mid x_{<t}).
]

This process is repeated across extremely large numbers of token sequences. Through next-token prediction, the model gradually learns grammatical structures, semantic relationships, writing patterns, factual associations, programming syntax, and recurring forms of reasoning found in its training data. The original GPT work formalized this autoregressive language-modeling objective using a multi-layer Transformer decoder.

When the model generates an answer, it uses the same basic mechanism. It calculates a probability distribution over its vocabulary, selects a token according to the decoding strategy, adds that token to the sequence, and repeats the process.

Input: "The capital of France is"

Prediction 1: " Paris"
New sequence: "The capital of France is Paris"

Prediction 2: "."
New sequence: "The capital of France is Paris."

Prediction 3: End of response

The model does not normally produce an entire paragraph in a single step. It generates the response sequentially, one token at a time.

How Self-Attention Works

The defining component of the Transformer is self-attention.

Self-attention allows each token representation to incorporate information from other relevant tokens in the sequence. Consider the sentence:

The programmer fixed the server because it had stopped responding.

To interpret the word “it,” the model must represent its relationship with earlier words such as “server.” Attention mechanisms help the model calculate these contextual relationships.

Within an attention layer, token representations are projected into three kinds of vectors:

Query
Key
Value

The standard scaled dot-product attention operation is expressed as:

\text{softmax}
\left(
\frac{QK^\top}{\sqrt{d_k}}
\right)V.
]

The query and key vectors determine how strongly different positions should attend to one another. The value vectors contain the information combined to create the resulting contextual representation.

Transformers generally use multi-head attention, meaning that several attention operations are performed in parallel. Different attention heads can learn to represent different kinds of relationships, although individual heads do not necessarily correspond to clean, human-defined linguistic rules.

GPT models use a causal attention mask. This prevents a token from accessing future tokens during ordinary autoregressive training. When predicting token (x_t), the model may use tokens before (x_t), but it cannot look ahead at the correct answer.

A GPT layer also contains feed-forward neural networks, residual connections, and normalization operations. By stacking many such layers, the model constructs increasingly contextual representations of the input sequence.

What Does Pre-Training Teach the Model?

During pre-training, a GPT model is exposed to a large corpus of tokenized data and optimized for next-token prediction.

The data may contain many forms of language, including prose, technical documents, conversations, educational material, and source code. The exact dataset, filtering procedure, and mixture vary between models.

Pre-training does not usually provide the model with an explicit database of facts or a manually designed grammar. Instead, the model learns distributed statistical representations through optimization.

For example, the model is not necessarily given a formal rule stating that a verb must agree with its subject. It encounters many examples in which grammatical agreement occurs and adjusts its parameters in ways that make grammatically consistent continuations more probable.

This training process also allows sufficiently capable models to perform tasks that were not represented as separate training objectives. GPT-3 demonstrated that a large autoregressive language model could perform many tasks through instructions or a small number of examples placed directly in the prompt, without additional gradient-based fine-tuning for each task. This behavior became known as zero-shot, one-shot, and few-shot learning.

More precisely, this is often called in-context learning. The model adapts its output according to patterns in the current context, but its underlying parameters are not normally updated during the conversation.

What Are Model Parameters?

Parameters are numerical values learned during training. They include the weights used by attention projections, feed-forward networks, embeddings, and other components of the model.

Parameters determine how information is transformed as it passes through the network. They are not individual facts that can normally be inspected as simple entries such as:

Parameter 8,217,491 = "Paris is the capital of France"

Knowledge is distributed across many parameters and internal representations.

Increasing the parameter count can increase a model’s capacity, but parameter count alone does not determine quality. Performance also depends on factors such as:

Training-data quantity and quality
Tokenizer design
Model architecture
Optimization procedure
Training compute
Context length
Post-training data
Evaluation methodology

Research on neural scaling laws found predictable relationships between language-model loss, model size, dataset size, and training compute across broad experimental ranges. However, these findings do not imply that simply increasing parameter count will automatically produce a more helpful or reliable assistant.

The InstructGPT experiments provide a useful example. Human evaluators preferred the outputs of a 1.3-billion-parameter instruction-tuned model over those of the much larger 175-billion-parameter GPT-3 base model on the researchers’ prompt distribution. This demonstrated the importance of post-training and alignment rather than parameter count alone.

A Base GPT Model Is Not Automatically a Chatbot

A model trained only with next-token prediction is generally called a base model.

Base models can complete text, imitate styles, answer some questions, and perform tasks through prompting. However, their fundamental objective is to continue sequences in statistically plausible ways. They are not automatically optimized to act as helpful conversational assistants.

Turning a base model into an instruction-following assistant usually requires additional post-training.

A simplified post-training pipeline may include:

Supervised Fine-Tuning

Human-written or curated examples are used to teach the model how to respond to instructions.

A training example might contain:

Instruction:
Explain photosynthesis to a twelve-year-old.

Desired response:
Plants use sunlight to convert water and carbon dioxide into...

The model is trained to make the desired response more probable when presented with similar instructions.

Preference Training

Several possible model responses are compared and ranked. These preferences provide information about which outputs are more helpful, accurate, clear, or safe.

Reinforcement Learning from Human Feedback

In the classical RLHF pipeline, preference comparisons are used to train a reward model. The language model is then optimized to produce responses receiving higher predicted rewards.

The InstructGPT study combined supervised demonstrations with ranked model outputs and reinforcement learning from human feedback. Its results showed that post-training could substantially improve instruction following and human preference ratings.

GPT and ChatGPT Are Not the Same Thing

GPT refers to the underlying family of generative Transformer models and the associated architectural and training approach.

ChatGPT is a conversational system designed to interact with users through dialogue. Its behavior depends not only on a language model but also on post-training, conversation formatting, system instructions, safety mechanisms, and—in some implementations—external tools.

OpenAI introduced ChatGPT as a dialogue-oriented sibling of InstructGPT, trained to respond conversationally and handle follow-up questions.

The distinction can be summarized as follows:

GPT:
The underlying generative language-model family.

ChatGPT:
A conversational product and system built around language models.

Similarly, large language model, or LLM, is a broader category. Not every LLM is a GPT model. Other language-model families may use different architectures, training procedures, tokenizers, licensing models, or multimodal components.

Fine-Tuning GPT Models

Pre-training produces a general-purpose model, but organizations often need models adapted to particular domains, languages, formats, or behaviors.

Full fine-tuning updates all or most of the model’s parameters. While this can be effective, it requires substantial GPU memory, storage, and computational resources for large models.

Parameter-efficient fine-tuning methods attempt to reduce these requirements.

LoRA

Low-Rank Adaptation, or LoRA, freezes the original model weights and introduces smaller trainable matrices into selected layers.

Instead of directly learning a complete weight update (\Delta W), LoRA approximates it using two lower-rank matrices:

[
\Delta W = BA,
]

where the selected rank is much smaller than the original matrix dimensions.

This significantly reduces the number of trainable parameters and makes it possible to store separate lightweight adapters for different tasks or domains. The original LoRA study reported competitive performance while substantially reducing trainable parameter counts and memory requirements compared with full fine-tuning.

QLoRA

QLoRA combines quantization with LoRA-based fine-tuning.

The base model is stored in a low-precision format—four-bit quantization in the original QLoRA formulation—while gradients are propagated into trainable LoRA adapters. The base model remains frozen.

This approach dramatically reduces memory requirements. The original QLoRA research demonstrated fine-tuning of a 65-billion-parameter model on a single 48 GB GPU while preserving performance comparable to a full 16-bit fine-tuning baseline in its experiments.

LoRA and QLoRA do not replace pre-training. They adapt an already pre-trained model by modifying a much smaller set of trainable values.

Does GPT Actually Understand Language?

This question does not have a universally accepted yes-or-no answer because the word “understand” can refer to several different things.

At the mechanistic level, GPT processes numerical token representations and learns a probability distribution over possible continuations. It does not experience language in the same biological and social manner as a human being.

However, reducing the model to “autocomplete” can also be misleading. To predict language accurately across many contexts, the model develops internal representations that can support translation, summarization, classification, code generation, question answering, and forms of multi-step problem solving.

The original GPT research found that generative pre-training produced representations transferable to several natural-language understanding tasks. GPT-3 later demonstrated broad task behavior through prompting and in-context examples.

A careful description is therefore:

GPT learns complex statistical and representational structures from data, producing behavior that can resemble linguistic understanding, but fluent output alone does not prove human-like comprehension or factual reliability.

This distinction matters because a model may generate a confident, grammatically perfect, and entirely incorrect answer.

Why Does GPT Hallucinate?

A hallucination occurs when a model generates information that is false, unsupported, or inconsistent with the available evidence.

This behavior is connected to the model’s fundamental objective. A language model is trained to produce probable continuations, not to guarantee that every generated statement has been independently verified.

If the model has incomplete information, conflicting patterns, or insufficient context, it may still generate a plausible-looking answer instead of remaining silent. Modern post-training can reduce this behavior, but it cannot completely eliminate it.

Research examining language-model hallucinations argues that generative errors arise from statistical properties of pre-training and can persist through post-training because standard evaluation and optimization procedures may reward guessing rather than uncertainty.

For this reason, GPT outputs should be verified when used in:

Medical decisions
Legal interpretation
Financial analysis
Academic research
Security-sensitive software
Current news and rapidly changing information

A language model can assist with these tasks, but linguistic confidence should never be treated as proof.

Context Windows and Temporary Information

GPT models operate on a limited sequence of tokens known as the context window.

The context may contain:

The user’s current message
Earlier messages in the conversation
System instructions
Retrieved documents
Tool outputs
Generated tokens

The model conditions its next-token predictions on the information available within this context. Information outside the active context is not automatically available unless the surrounding system retrieves or stores it separately.

This is why application-level systems often combine language models with retrieval-augmented generation, databases, search engines, vector stores, and external tools. These components do not become part of the GPT model itself; they provide relevant information to the model at inference time.

GPT Is a Model, Not the Entire AI System

A modern AI application may contain far more than a language model.

A production system can include:

User interface
↓
Authentication and access control
↓
Prompt and context management
↓
Retrieval or web search
↓
GPT or another language model
↓
Tool execution
↓
Safety and validation layers
↓
Response presented to the user

The quality of an AI product therefore depends not only on its base model but also on its data pipeline, retrieval system, memory architecture, tool integration, evaluation process, and application design.

A powerful model placed inside a poorly designed system can still produce unreliable results. Conversely, a smaller model supported by high-quality retrieval, structured workflows, and careful validation can outperform a larger general-purpose model on a narrowly defined task.

Conclusion

GPT is a generative neural language model based on the Transformer architecture. It converts text into tokens, represents those tokens as vectors, processes them through stacked attention and feed-forward layers, and generates output through repeated next-token prediction.

Its capabilities emerge from several interconnected components:

Tokenization
Transformer-based causal self-attention
Large-scale pre-training
Learned parameters
Prompt-based in-context learning
Supervised fine-tuning
Preference optimization
Efficient adaptation methods such as LoRA and QLoRA

The apparent simplicity of next-token prediction should not be confused with a simple system. When applied across sufficiently large and diverse datasets with substantial computational resources, this objective can produce models capable of performing a broad range of language and reasoning-related tasks.

At the same time, GPT remains a probabilistic model. It can generate inaccurate information, reproduce biases, misunderstand instructions, and express uncertainty poorly. Understanding both its architecture and its limitations is essential for using it responsibly.

GPT is not magic, and it is not merely a collection of prepared answers. It is a learned mathematical system that models patterns in sequences—and modern AI applications are built by combining that model with data, tools, infrastructure, and carefully designed human objectives.

Academic References

Brown, T. B., Mann, B., Ryder, N., et al. (2020). Language Models Are Few-Shot Learners. Advances in Neural Information Processing Systems, 33, 1877–1901.

Dettmers, T., Pagnoni, A., Holtzman, A., & Zettlemoyer, L. (2023). QLoRA: Efficient Finetuning of Quantized LLMs. Advances in Neural Information Processing Systems, 36.

Hu, E. J., Shen, Y., Wallis, P., et al. (2021). LoRA: Low-Rank Adaptation of Large Language Models. International Conference on Learning Representations.

Kalai, A. T., Nachum, O., Vempala, S. S., & Zhang, E. (2025). Why Language Models Hallucinate.

Kaplan, J., McCandlish, S., Henighan, T., et al. (2020). Scaling Laws for Neural Language Models.

Ouyang, L., Wu, J., Jiang, X., et al. (2022). Training Language Models to Follow Instructions with Human Feedback. Advances in Neural Information Processing Systems, 35, 27730–27744.

Radford, A., Narasimhan, K., Salimans, T., & Sutskever, I. (2018). Improving Language Understanding by Generative Pre-Training. OpenAI.

Sennrich, R., Haddow, B., & Birch, A. (2016). Neural Machine Translation of Rare Words with Subword Units. Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, 1715–1725.

Vaswani, A., Shazeer, N., Parmar, N., et al. (2017). Attention Is All You Need. Advances in Neural Information Processing Systems, 30.

Continue reading on DEVComunity:
https://dehayz.com/blog/gpt-nedir-nasil-calisir