What Is PEFT? A Guide to Parameter-Efficient Fine-Tuning

A technical guide comparing LoRA, QLoRA, rsLoRA, AdaLoRA, DoRA, IA³, prompt tuning, and adapter deployment workflows.

DEHA Research · July 16, 2026 · 18 min read

PEFT, or Parameter-Efficient Fine-Tuning, is a family of methods that adapts a large pretrained model to new tasks by training only a small subset of its parameters or lightweight components added to the model, rather than modifying all of its weights.

The objective is not merely to save GPU memory. PEFT also makes it possible to create small, portable, and task-specific adapters that share the same underlying base model.

In practice, the term PEFT is used in two different ways. First, it refers to the general family of methods that includes LoRA, IA³, and prompt tuning. Second, it refers to the open-source Hugging Face PEFT library, which implements these techniques alongside Transformers, Diffusers, and Accelerate.

This article first explains PEFT as an engineering approach, then examines the core concepts of the Hugging Face library and the decisions required for production deployment.

In Brief: What Is PEFT?

PEFT freezes most of the base model’s weights and trains only small adapters, scaling vectors, or learnable soft prompts.

Because the optimizer needs to store gradients and states for fewer parameters, training-memory requirements and checkpoint sizes are reduced.

However, PEFT is not a solution for poor-quality training data, and it does not guarantee the same performance as full fine-tuning for every task.

1. Why Was PEFT Developed?

During full fine-tuning, gradients and optimizer states must be maintained for every trainable weight in the model.

In a mixed-precision training run using AdamW, these states may consume several times more memory than the model weights themselves. Storing a complete model checkpoint for every individual task also becomes operationally expensive as model size increases.

PEFT preserves the general representations learned by the pretrained model while concentrating task-specific changes into a small parameter budget.

Hugging Face documentation states that this approach can substantially reduce the number of trainable parameters, computational requirements, and storage costs while achieving results close to full fine-tuning on many tasks.

The word close is important. Results depend on the model, task, dataset, PEFT method, and allocated parameter budget.

1. The Main Difference Between PEFT and Full Fine-Tuning Feature Full Fine-Tuning PEFT Parameters being trained The entire model or a large portion of it A small group of added or selected parameters Optimizer memory Very high Limited primarily to trainable parameters Checkpoint Complete model weights Usually a small adapter file Per-task deployment A separate full model may be required Adapters can be switched on the same base model Adaptation capacity Highest Limited by the selected method and parameter budget Base-model dependency May produce an independent full checkpoint Requires the correct base model and revision

PEFT does not mean that the model stops performing most computations.

Frozen layers do not require optimizer states, but the forward pass still runs through them, and a significant portion of the backward graph remains necessary to propagate gradients to the adapters.

Long-sequence activations, attention operations, and temporary kernel memory are still required.

PEFT may therefore make it possible to fine-tune a 7-billion-parameter model on a smaller GPU, but it does not make the maximum sequence length unlimited.

1. How Are PEFT Methods Classified?

Not every PEFT technique is a variation of LoRA.

The methods can be divided into four practical groups according to where and how they modify the model.

Weight- or Adapter-Based Methods

Methods such as LoRA, AdaLoRA, and DoRA represent weight updates using lightweight structures.

Activation Scaling

Methods such as IA³ use learnable vectors to scale attention and feed-forward activations.

Soft Prompting

Prompt tuning and prefix tuning add learnable continuous vectors to a frozen model.

Selective Tuning

These approaches train only selected existing parameters, such as biases, LayerNorm parameters, specific tokens, or subsets of model layers.

The method that trains the fewest parameters is not necessarily the best method.

When a task only requires steering capabilities that the model already possesses, a soft prompt may be sufficient.

When the goal is to teach a new output structure or produce a stronger behavioral change, LoRA applied to linear layers may provide greater flexibility.

IA³ may be considered when the available parameter and memory budget is extremely limited.

1. LoRA: The Most Widely Used PEFT Method

LoRA represents the update to a frozen weight matrix W as the product of two smaller matrices:

W′ = W + sBA

The rank r determines the narrow intermediate dimension between the two matrices.

For a linear layer with dimensions d_out × d_in, a full weight update requires:

d_out × d_in

trainable parameters.

LoRA adds approximately:

r × (d_in + d_out)

trainable parameters.

For example:

4096 × 4096 linear layer

Full weight:
16,777,216 parameters

LoRA with r = 16:
16 × (4096 + 4096) = 131,072 parameters

Approximately 128 times fewer trainable parameters
for this layer.

The total reduction depends on how many layers receive LoRA adapters and which modules are targeted.

The original LoRA study reported strong results across several tasks while substantially reducing the number of trainable parameters and the associated memory requirements.

The main LoRA configuration decisions include:

r,
lora_alpha,
target_modules,
dropout,
additional modules that must be saved.

Rank determines the adapter’s capacity, while alpha determines the scale of its contribution.

1. Is QLoRA a PEFT Method or a Quantization Method?

QLoRA combines two separate ideas.

The frozen base-model weights are loaded in 4-bit precision and temporarily dequantized to an appropriate compute precision during calculations. The trainable LoRA adapters remain in a higher precision.

Quantization therefore addresses how the base model is represented in memory, while LoRA determines which parameters are trained.

Simply loading a model in 4-bit precision is not fine-tuning.

Similarly, using LoRA without quantizing the base model is LoRA, not QLoRA.

QLoRA uses techniques such as:

NormalFloat 4-bit, or NF4,
double quantization,
paged optimizers.

Its purpose is to make it possible to adapt large models using more limited GPU resources.

Training with a 4-bit base model does not mean that the adapter checkpoint itself is stored in 4-bit precision.

Any quality difference introduced by quantization must also be measured separately for each model and task.

1. What Do rsLoRA, AdaLoRA, and DoRA Change? rsLoRA

Standard LoRA scales the adapter contribution using:

alpha / r

Rank-Stabilized LoRA, or rsLoRA, instead uses:

alpha / √r

This formulation is designed to prevent the learning signal from becoming excessively small at higher ranks.

It can be enabled in Hugging Face PEFT using:

use_rslora=True

The approach is particularly useful when comparing different ranks because it aims to prevent the standard scaling formula from confounding the effect of rank itself.

AdaLoRA

Standard LoRA generally assigns the same rank budget to most targeted matrices.

AdaLoRA evaluates the importance of different layers during training and distributes a limited parameter budget adaptively. It prunes singular values associated with less important updates.

The method is intended to use the available parameter budget more efficiently than uniform rank allocation, particularly when the budget is highly constrained.

DoRA

DoRA decomposes a weight into magnitude and directional components.

It uses LoRA to update the directional component while learning the magnitude component separately.

The objective is to reduce the difference in training dynamics between LoRA and full fine-tuning.

Research on DoRA has reported improvements over LoRA in language, vision, and multimodal tasks.

The trade-off is that its training compute and temporary memory consumption may be higher than those of vanilla LoRA.

1. IA³: Scaling Activations Instead of Updating Matrices

IA³ adds learnable scaling vectors to:

attention key activations,
attention value activations,
intermediate feed-forward activations.

Because it uses vectors instead of low-rank matrix updates, the number of trainable parameters can be even smaller than with LoRA.

The T-Few study introduced IA³ while demonstrating that parameter-efficient fine-tuning could outperform in-context learning in both accuracy and computational efficiency on certain few-shot tasks.

IA³ is attractive when extremely small adapters and rapid switching between tasks are required.

However, LoRA’s matrix-based adaptation capacity may be more flexible for complex behavioral transformations.

The decision should therefore be based not only on checkpoint size, but also on target-task evaluation and the serving architecture.

1. Prompt Tuning and Prefix Tuning

Soft-prompt methods do not generate human-readable prompt words.

Instead, they learn continuous vectors in the model’s embedding space through backpropagation.

Prompt Tuning

Prompt tuning adds learnable virtual tokens to the model input.

Research has shown that, as model scale increases, prompt tuning can close the performance gap with full-model tuning on some T5 tasks.

Prefix Tuning

Prefix tuning provides learnable prefix key and value representations to the attention mechanism in each Transformer layer.

The original study reported results close to full fine-tuning on table-to-text generation and summarization tasks while training approximately 0.1% of the model parameters.

The adapter files produced by these methods can be extremely small.

However, virtual tokens and prefixes may affect:

the amount of usable context,
the KV cache,
inference architecture.

A soft prompt is also not a natural-language prompt that can be transferred freely to a different tokenizer or a different base-model version.

1. Which PEFT Method Should Be Selected? Requirement First Candidate What Must Be Evaluated General instruction or formatting adaptation LoRA Target modules, rank, and alpha Adapting a large model on a single GPU QLoRA 4-bit quality difference and compute dtype Greater stability at high ranks rsLoRA Alpha and learning rate together Extremely limited parameter budget AdaLoRA or IA³ Real task capacity Reducing the quality gap between LoRA and full fine-tuning DoRA Additional compute and VRAM Very small per-task state Prompt or prefix tuning Model scale and context cost Adding new or frequently changing knowledge RAG first Information freshness and source requirements

PEFT is powerful for behavioral and task adaptation.

It should not be the default method for writing frequently changing organizational knowledge into model weights.

When information must remain current and its sources need to be shown, retrieval-augmented generation is often more appropriate.

PEFT and RAG may also be used together.

For example, an adapter can teach the desired answer structure while retrieval supplies current supporting evidence.

1. A Basic LoRA Workflow with Hugging Face PEFT from peft import LoraConfig, get_peft_model

config = LoraConfig(
task_type="CAUSAL_LM",
r=16,
lora_alpha=32,
lora_dropout=0.05,
target_modules=[
"q_proj",
"k_proj",
"v_proj",
"o_proj",
"gate_proj",
"up_proj",
"down_proj",
],
bias="none",
use_rslora=True,
)

model = get_peft_model(base_model, config)
model.print_trainable_parameters()

The names used in target_modules vary across model architectures and should not be copied blindly from a different model.

Before training begins, developers should inspect and record:

the list of trainable parameters,
the percentage of trainable parameters,
the data types of several adapter tensors,
the devices on which those tensors are located.

PEFT also supports adding, activating, and deleting adapters through its integration with Transformers.

When print_trainable_parameters() reports a much higher proportion than expected, the embedding layer, language-model head, or bias parameters may have been enabled unintentionally.

When the proportion is much lower than expected, the target-module names may not match the actual module names in the model.

A decreasing training loss does not, by itself, prove that the adapters were attached to the correct locations.

1. Saving and Loading Adapters and Base-Model Dependency

A PEFT checkpoint usually does not include the entire base model.

The adapter configuration describes:

the base-model identity,
the PEFT method and its settings,
the targeted modules.

The adapter weights contain only the small learned difference.

The following information should be versioned for production deployment:

base-model repository and exact revision or commit,
tokenizer files,
chat template,
additional special tokens,
PEFT and Transformers versions,
adapter configuration,
quantization configuration,
compute data type,
training-data version,
evaluation report,
licensing relationship.

Attaching an adapter to a base model with the same name but a different revision may damage behavior even when it does not produce a tensor-shape error.

When tokens have been added to the tokenizer, changes to the embedding or language-model head may also need to be stored.

In this situation, backing up only the adapter file is not sufficient.

1. Merging, Hotswapping, and Multiple Adapters

A LoRA adapter can be applied to the base weights dynamically at runtime or merged into them using:

merge_and_unload()

Merging removes the need for a separate adapter computation and produces a standalone model.

However, it reduces the ability to:

switch adapters dynamically,
share one base model across many adapters,
maintain small task-specific checkpoints.

Merging with a quantized model must be tested carefully with respect to data type and memory use.

Loading a separate adapter for each customer, language, or task on a shared base model can reduce storage requirements.

However, it introduces new operational risks, including:

adapter caching,
tenant authorization,
concurrent loading,
GPU-memory management,
accidentally selecting the wrong adapter.

Accepting an adapter identifier directly from a client without validating authorization could cause one customer’s behavioral adapter to be executed for another customer.

Different adapters can also be merged using weighted combinations.

However, two capabilities are not guaranteed to combine automatically or without quality loss. Their parameter-update directions may conflict.

PEFT provides model-merging approaches such as:

linear merging,
SVD-based merging,
TIES.

A merged model must be reevaluated independently on every source task.

1. Problems That PEFT Does Not Solve Poor Data

Training fewer parameters does not correct inaccurate, repetitive, contaminated, or leaked training data.

Hallucinations

An adapter does not guarantee factual accuracy. Retrieval may still be required for source-grounded information.

Long Context

PEFT does not automatically increase the context window or reduce the computational cost of attention.

A Poorly Matched Tokenizer

Adding LoRA adapters does not fully solve the problem of a tokenizer that is unsuitable for the target language.

Security

Malicious, unauthorized, or contaminated training data may introduce backdoors and unwanted behavioral changes.

Serving Cost

The inference cost of the base model largely remains in place.

PEFT may reduce the risk of catastrophic forgetting because fewer parameters are trainable, but it does not eliminate the risk.

An adapter can still steer the output of the base model very strongly.

The following should be compared with the adapter both enabled and disabled:

target-task performance,
general capabilities,
safety behavior,
multilingual performance,
long-context behavior.

1. Common PEFT Mistakes Copying target_modules from another training run without inspecting the model architecture. Starting training without printing and verifying the trainable parameters. Treating rank and alpha as the same concept or increasing both together with the learning rate without control. Failing to verify compute dtype, quantization dtype, and adapter dtype in QLoRA. Accidentally calculating loss over prompt or user tokens. Failing to version the base revision, tokenizer, and chat template together with the adapter. Sending the checkpoint with the lowest training loss directly to production. Failing to reevaluate the model after merging the adapter.
2. A Robust PEFT Experiment Plan Establish a baseline by measuring the base model’s zero-shot and few-shot performance. Divide the data into training, validation, and private test sets. Test the chat template and loss mask. Run a small initial LoRA experiment and record the number of trainable parameters and peak VRAM usage. Sweep the learning rate, rank, and target modules in a controlled sequence rather than changing them all simultaneously. When standard LoRA reaches its limits, compare rsLoRA, DoRA, or AdaLoRA under the same parameter or compute budget. Measure task success, general evaluations, safety, and latency with the adapter enabled and disabled. Benchmark merged and unmerged serving configurations using realistic traffic profiles. Lock the base revision, tokenizer, configuration, dataset version, and evaluation report into a single release manifest. Conclusion

PEFT is not a trick for training a large model “for free” on a small GPU.

It is a deliberate engineering trade-off among:

adaptation capacity,
memory use,
storage requirements,
deployment flexibility.

LoRA is a strong default starting point, but QLoRA, rsLoRA, AdaLoRA, DoRA, IA³, and soft-prompt methods target different bottlenecks.

The correct method should not be selected solely according to the percentage of trainable parameters. It should be selected according to real task performance and the production architecture.

For the complete model-development process, see our guide to training artificial intelligence models.

For details about learning rates, batches, VRAM management, and truncation, see our technical guide to LLM fine-tuning.

Frequently Asked Questions
What Does PEFT Stand For?

PEFT stands for Parameter-Efficient Fine-Tuning.

It adapts a model by training a small group of parameters rather than updating the entire model.

Are PEFT and LoRA the Same Thing?

No.

PEFT is the broader family of parameter-efficient adaptation techniques. LoRA is one of the most widely used methods within that family.

Other examples include:

IA³,
prefix tuning,
prompt tuning,
AdaLoRA,
DoRA.
What Is the Difference Between QLoRA and LoRA?

LoRA adds low-rank adapters to a frozen base model.

QLoRA also quantizes the base-model weights to 4-bit precision, reducing VRAM usage further.

Should a PEFT Adapter Be Merged?

When a single adapter will be used permanently, merging may simplify serving.

When multiple adapters or hotswapping are required on the same base model, keeping the adapter separate is more flexible.

Both configurations should be measured independently for quality and latency.

How Much VRAM Does PEFT Save?

There is no fixed percentage.

Gradient and optimizer-state memory may be reduced substantially, but memory is still required for:

base-model weights,
activations,
attention operations,
temporary tensors.

The actual saving depends on the model, PEFT method, rank, sequence length, batch size, and quantization configuration.

References
Hu, E. J. et al. (2021). LoRA: Low-Rank Adaptation of Large Language Models. arXiv:2106.09685.
Dettmers, T. et al. (2023). QLoRA: Efficient Finetuning of Quantized LLMs. arXiv:2305.14314.
Zhang, Q. et al. (2023). AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning. arXiv:2303.10512.
Liu, S.-Y. et al. (2024). DoRA: Weight-Decomposed Low-Rank Adaptation. arXiv:2402.09353.
Kalajdzievski, D. (2023). A Rank Stabilization Scaling Factor for Fine-Tuning with LoRA. arXiv:2312.03732.
Liu, H. et al. (2022). Few-Shot Parameter-Efficient Fine-Tuning Is Better and Cheaper than In-Context Learning. arXiv:2205.05638.
Li, X. L. and Liang, P. (2021). Prefix-Tuning: Optimizing Continuous Prompts for Generation. arXiv:2101.00190.
Lester, B. et al. (2021). The Power of Scale for Parameter-Efficient Prompt Tuning. arXiv:2104.08691.
Hugging Face. PEFT Documentation.

This article was prepared on July 16, 2026. The performance of each method depends on the selected model and task. Production decisions should be based on independent evaluation and realistic serving benchmarks.

Continue reading on DEVComunity:
https://dehayz.com/blog