DEV Community: Pruna AI

Pruna 0.3.2: More OSS Algos, More Ways to Optimize

Sara Han — Wed, 11 Mar 2026 09:00:00 +0000

It’s been almost a year since we open-sourced. Over that time, Pruna has grown quickly: more contributors, algorithms, families, tutorials, and optimized models. With v0.3.2, open-sourcing many more of these algorithms is the natural next step.

What Landed in 0.3.2

This release expands the ecosystem with support for a broad set of new algorithms and new algorithm families, improved compatibility across them, and a set of fixes that make the whole framework stronger.

New algorithms and families: Pruna 0.3.2 adds a broad new set of optimization building blocks to the OSS stack. This includes new compilers, kernels, pruners, and entire new algorithm families such as Decoders, Distillers, Enhancers, and Recoverers.
More than just new algos: The most important part of this release is not only the number of new algorithms, but how they fit into Pruna. 0.3.2 increases composability by allowing otherwise incompatible algorithms to be treated as compatible when they are applied to disjoint parts of a model.
More tutorials: The new release also brings more tutorials to help you learn how to make your model more efficient. So it makes it easier for you to discover what each method does, understand when to use it, and get started composing them in practice.
Pruning bugs and maintenance: This release is not only about new features, but it also includes important fixes and cleanup work that reinforce the core of Pruna. That includes pruning-related bug fixes, maintenance work across the codebase, and general improvements that make the new algorithms easier to use and more reliable in practice.

For more information, check the GitHub release here.

Meet the New Algorithms and Families

One of the biggest updates in 0.3.2 is the expansion of Pruna’s optimization core.

Expanding Existing Families

Compilers: ipex_llm and x_fast

These new compiler integrations expand the set of execution-level optimizations. You can use ipex-llm for PyTorch-based LLM inference on Intel CPUs and x-fast to speed up inference for any model using a combination of xformers, triton, cudnn, and torch tracing.
Kernels: ring_attn and sage_attn

This release introduces two important kernel-level additions. Ring attention brings distributed attention capabilities that help scale workloads across multiple devices, while sage attention adds a fast, memory-efficient attention kernel to the toolbox.
Pruner: padding_pruning

Padding pruning allows you to remove unnecessary padded computation. This is a targeted optimization that, while simple, still delivers efficiency gains.

# Usage example
from pruna import SmashConfig, smash

# Initialize the SmashConfig and configure the algorithms
smash_config = SmashConfig(["ring_attn", "torch_compile"])
# Configure the hyperparameters
smash_config.add({
    "torch_compile_target": "module_list"
})
# Optionally, add further compatible algorithms
smash_config.add(["qkv_diffusers", "padding_pruning"])

Introducing New Families

Decoders: zipar

Pruna now supports decoders to speed up autoregressive generation by changing the decoding strategy itself. These methods speed up autoregressive generation by making decoding more parallelizable.
Distillers: text_to_image_distillation_inplace_perp, text_to_image_distillation_lora, text_to_image_distillation_perp, hyper

Distillers make it easier to reduce inference costs by transferring behavior into smaller, more efficient variants.
Enhancers: img2img_denoise, realesrgan_upscale

Enhancers improve output quality after or alongside optimization. These methods are especially useful when the goal is not only faster inference, but also better final outputs.

Recoverers: text_to_image_distillation_inplace_perp, text_to_image_distillation_lora, text_to_image_distillation_perp, text_to_image_inplace_perp, text_to_image_lora, text_to_image_perp, text_to_text_inplace_perp, text_to_text_lora, text_to_text_perp

Recoverers make it possible to push compression more aggressively and then restore part of the lost quality afterward. This gives you a much more flexible optimization workflow, especially when combining quantization, pruning, or distillation with quality recovery steps.

# Usage example
from pruna import SmashConfig

smash_config = SmashConfig({
    # Quantize the model to 4-bits
    "diffusers_int8": {
        "weight_bits": 4
    },
    # Recover, allowing you to push quantization to lower bit rates without compromising quality
    "text_to_image_perp": {
        # you can increase or reduce 'batch_size' depending on your GPU, or use 'gradient_accumulation_steps' with it
        "batch_size": 8,
        "num_epochs": 4,
        "validate_every_n_epoch": 0.5 # run validation every half epoch
    }
})
# Attach a text-to-image dataset, used for recovery
smash_config.add_data("COCO")
smash_config.data.limit_datasets((256, 64, 1))

More Efficient Strategies

Diagram showcasing the current algorithm families supported by Pruna (10-03-2026)

So, instead of only asking “how do I make this model faster?”, you can now think in more advanced strategies like:

compress first, then recover quality
parallelize decoding instead of just reducing precision
distribute attention across devices
add post-processing quality enhancers
swap in better attention kernels
combine multiple compatible algorithms into a single pipeline

This makes Pruna more flexible not just as a collection of optimizations, but also as a system for easily combining them.

Try out Pruna 0.3.2, smash your model, and show us what combinations you come up with.

Enjoy the Quality and Efficiency!

Compress your own models with Pruna and give us a ⭐️ to bring you many more algos!
Stay up to date with the latest AI efficiency research on our blog, explore our materials collection, or dive into our courses.
Join the conversation and stay updated in our Discord community.

LLM Architectures Explained: What Powers Today’s Top Models

Sara Han — Wed, 04 Mar 2026 11:22:59 +0000

Large Language Models (LLMs) have rapidly taken the spotlight in a wide range of fields over the past few years. At Pruna, the focus has been clear: make these models smaller, faster, cheaper, and greener. To make this possible, the team has explored and provided different optimization techniques, from caching and model compilation to advanced quantization and beyond.

For an overview of AI model optimization techniques, see this blog.

However, these individual optimizations are just pieces of a much larger machine. We must lift the hood and examine the engine to understand how it works. This blog post will provide an overview, not attempting to cover every mathematical detail, but focusing on the main intuition, of the key architectures powering today’s language models: Autoregressive Models, State-Space Models, Diffusion-based Models, and Liquid Neural Networks.

Where It All Begins: Tokenizers and Embeddings

Before we dive into the intricate inner workings, it’s worth remembering that an LLM can’t “think” until it first “reads” your request, something it does through tokenization and embedding.

For example, if you ask, "How do I optimize a model?", the model doesn’t receive that sentence as you wrote it. Instead, first it's tokenized, i.e., the text is broken into smaller, more frequent chunks known as tokens. The process involves the following steps:

Text normalization, standardizing case and punctuation to ensure consistency.
Pre-tokenization, which breaks the text into rough chunks such as words or subwords.
The actual tokenization kicks in. This step can vary slightly between models depending on design choices: the tokenization method (most commonly Byte Pair Encoding, or BPE, and its variants), the vocabulary and special tokens that define the model’s “dictionary,” and the training data that influences how the tokenizer learns the patterns to split the input.

When it’s time to generate text, the model maps each token’s ID back to its original text fragment. But tokens alone aren’t enough — the model needs to understand their meaning and relationships, and work with numerical representations. That’s where embeddings come in. Each token ID is transformed into a high-dimensional vector that captures the meaning of the word based on how it was used in the training set. This is what allows LLMs to grasp intent, subtlety, and meaning far beyond basic definitions.

Token-by-Token: The Autoregressive Way

Many LLMs are autoregressive, i.e., they generate text by predicting the next token in a sequence one by one. The Transformer architecture powers most of today’s leading models.

Once we step into a transformer, we'll find a stack of transformer blocks. Each block processes the incoming token and passes the results to the next. At each block's heart, two operations occur: self-attention and a feedforward network.

The self-attention mechanism determines how important each token is relative to all others in the sequence. This process implies:

The model computes attention scores by multiplying the query vector of the current token with the key vectors of all other tokens.
After normalization, each score is used to weigh the corresponding value vector. The weighted sum of these values becomes the output of the attention layer.
When a query and key are a strong match — meaning they produce a high attention score — the associated value has a stronger influence on the final output.
Transformers use multi-head attention, i.e., multiple attention mechanisms ("heads") are run in parallel to increase the model's ability to capture different types of relationships. Each head focuses on different aspects of the input, combining their outputs to form a richer representation.

After the self-attention step, the output at each position is passed through a feedforward neural network, a stack of dense layers with non-linear activation functions like ReLU or GeLU. This helps the model detect complex patterns that attention alone might miss.

Finally, each sub-layer (self-attention and feed forward) is wrapped with residual connections and layer normalization, which helps stabilize the model and allows for deeper networks.

Source: https://arxiv.org/abs/1706.03762

To make the Transformer more efficient, several optimizations are often applied to different part of the transformer block:

Since the attention mechanism is typically the main computational bottleneck, various strategies have been focused on reducing its load:
- KV caching stores previously computed keys and values to speed up text generation significantly by avoiding redundant computations.
- Sparse Attention limits focus to a subset of tokens
- Sliding Window Attention restricts attention to the most recent tokens
- Flash Attention improves GPU memory usage and throughput
- Paged Attention manages KV caches more effectively for long sequences
- Multi-Query Attention (MQA) lowers computational cost by sharing keys and values across all attention heads.
Feed forward can be improved with another powerful approach, the Mixture of Experts (MoE). It replaces the traditional single feedforward block with multiple expert networks specialized in different patterns or topics, selectively activated through a gating mechanism. So, only a subset of them runs at a time, allowing the model to scale efficiently during training.

Thinking in States: A Different Way to Think About Sequences

While autoregressive models like Transformers generate text by predicting the next token based on all previously seen tokens, State Space Models (SSMs) take inspiration from physic. At a time, they map a continuous input sequence to a latent space representation and predict the output sequence.

Source: https://newsletter.maartengrootendorst.com/p/a-visual-guide-to-mamba-and-state

SSMs capture only the most relevant information in three different ways to represent the relationship between input, state, and output. Depending on the task, the stage of the process, or the type of data, it’s possible to switch between these representations, although it requires some advanced methods, to take advantage of the most efficient one for the problem at hand, maximizing performance.

Aspect	Continuous Representation	Recurrent Representation	Convolutional Representation
Core idea	Describes how the state changes smoothly over time.	Breaks time into steps, updating the current state based on the previous state and new input.	Updates the current state using a weighted history of previous states.
Advantages	Ideal for data with irregular or time-shifted sampling · Mathematically feasible analysis	Natural fit for sequences · Efficient inference	Local, interpretable features · Parallelizable training
Disadvantages	Very slow training and inference	Slow training · Gradient issues in too-long sequences	Inefficient in online/autoregressive use · Fixed context size
Suitability	Handling continuous data	Efficient inference	Fast training via parallelization

To handle the complexity of natural language, deep SSMs stack multiple state space layers and add non-linear transformations. In this setup, the SSM blocks handle dependencies across tokens in the sequence, while the non-linear layers capture dependencies across embedding dimensions. This division of labor allows the model to represent intricate language patterns while still benefiting from the efficiency of state-tracking mechanisms.

Removing the Noise: Diffusion LLMs

In the world of computer vision, one of the most groundbreaking advances in recent years has been diffusion models. The core idea is quite intuitive: start with an image and gradually add random noise over many steps until it turns into pure noise — resembling TV static or white noise. Then, train a model to reverse this process — step by step — by learning how to remove the noise and recover the original image (or generate a completely new one). Through this iterative denoising, the model learns the underlying patterns and structures of visual data, encoding that knowledge into a latent space, i.e., a map of all the possible images the model could generate, where each point represents a unique combination of learned features.

Similar principles have recently been explored in the context of language modeling, where researchers are adapting diffusion-based approaches to generate text. In this case, the process begins with a random noise representation, which is then gradually refined and “denoised” into a coherent sequence of tokens.

Unlike traditional autoregressive models that generate one token at a time, diffusion-based language models produce the entire sequence simultaneously (although they can also operate in a semi-autoregressive fashion by predicting blocks of tokens after blocks of tokens). This makes the process inherently parallelizable and potentially more efficient, especially during inference. In addition, as they consider the whole text structure simultaneously, they might be naturally better at logical reasoning and generating well-structured responses. Their ability to continuously refine output also holds promise for reducing hallucinations and minimizing errors.

Source: https://arxiv.org/abs/2502.09992

Overview at a Glance

Now that we’ve walked through the main architectures, it’s time to recap!

	Autoregressive LLMs	State-Space LLMs	Diffusion LLMs
Core Idea	Sequential token prediction via conditional probabilities.	Sequence modeling via state-space equations.	Iterative noise reduction.
Computational Cost	High	Low	High
Inference Speed	Slow-Medium	Fast	Medium-Fast
Long-context	Limited by memory	Designed for long sequences.	Limited by memory.
Interpretability	Medium	Medium	Low
Examples	GPT, LLaMA, Mistral	Mamba	LLaDA, Mercury Coder

While we’ve gone over the core ideas of these architectures, you should take into account that each can have other possible configurations depending on how encoding and decoding are designed for specific tasks.

What's Next?

In this blog post, we explored an overview of the main architectures behind today’s cutting-edge LLMs. Understanding these foundations is key to optimizing performance and choosing where to focus your efforts.

Enjoy the Quality and Efficiency!

Want to take it further?

Compress your own models with Pruna and give us a ⭐ to show your support!
Try our image and video models with just one click.
Stay up to date with the latest AI efficiency research on our blog, explore our materials collection, or dive into our courses.
Join the conversation and stay updated in our Discord community.

FLUX.2 [flex] Challenge

Sara Han — Fri, 30 Jan 2026 12:00:00 +0000

To celebrate together the release of FLUX.2 [flex] by Pruna AI in collaboration with Black Forest Labs. We’re launching the FLUX.2 [flex] Design Challenge! 🎨

Theme

Make an infographic of how to grow a black forest of plums

How to participate

1️⃣ Answer with a single creative infographic using FLUX.2 [flex] and the screenshot showcasing it. You can try it here: https://bfl.ai/models/flux-2

2️⃣ Follow the @PrunaAI and @bfl_ml account in X.

3️⃣ Mention us and add the hashtag #flux2flex.

Prize

Popularity + Judge’s evaluation

🥇 €150 🥈€100 🥉 €50

Dates

Until February 6th (23:59 CET)

Join the challenge, vote for your favorites, and inspire the community!

Read the rules in detail: https://www.pruna.ai/blog/flux2flex-challenge

Slashing torch.compile Warmup & LoRA Swapping Times with Pruna

Sara Han — Wed, 28 Jan 2026 15:07:53 +0000

PyTorch introduced torch.compile, a powerful feature that significantly boosts performance by compiling the models. However, it comes with a catch: the first run is very slow. That warmup delay can be a drag on development iteration and can lead to slower cold starts in production. If you’ve ever swapped a LoRA or made a small model change, you’ve probably noticed that frustrating pause before things get moving again. But what if you could dramatically reduce, or even eliminate, these warmup delays?

In this post, we'll dive into two practical techniques, powered by Pruna, to mitigate warmup times. We'll show you how to:

Using Pruna's portable compilation feature, eliminate the initial model warmup when deploying or reloading a model on a new machine (with identical hardware).
Achieve zero warmup when switching LoRAs (Low-Rank Adaptations) on an already optimized model.

Get ready to reclaim those precious seconds (or even minutes!) and make your torch.compile experience smoother than ever.

The Challenge: Understanding `torch.compile` Warmup

Before we dive into the solutions, let's briefly touch upon why torch.compile has a warmup phase. When you first invoke a model compiled with torch.compile, several things happen under the hood. PyTorch needs to:

Capture the computational graph: It traces the execution of your model to understand its structure.
Perform graph optimizations: The captured graph is then optimized for better performance.
Detect and fuse operators: The backend (such as Inductor) identifies which operations can be combined for faster execution.
Generate code: Optimized code (often CUDA kernels for GPUs or efficient CPU code) is generated by the chosen backend (like Inductor).
Compile the code: This generated code is then compiled into executable machine instructions.

This entire process, especially the code generation and compilation steps, can take a noticeable amount of time, ranging from seconds to minutes, depending on the model's complexity and the hardware. While this is a one-time cost for a given model shape and hardware (as the compiled artifacts are cached), it can be disruptive:

Start/Stop instances: When a new instance of an application starts (e.g., a serverless function or a new pod in Kubernetes), the first request might experience this long warmup, leading to poor user experience.
Switch instances: If you compile a model on one machine and then try to run it on another (even with identical hardware), the cache might not be directly usable, leading to another full warmup.
Switch model adapters: Swapping LoRAs or other adapters can alter the model graph triggering recompilation.
Development Iteration: Waiting for recompilation after minor code changes or restarting a kernel slows the development cycle.

Pruna offers elegant ways to mitigate these issues, as we'll see next.

Use Case 1: Eliminating Initial Warmup with Pruna's Portable Compilation

The Problem

Traditionally, running a compiled model on a new machine triggers a full compilation warmup, even if the hardware is identical. This can slow down processes, especially when deploying models to production or sharing them with others.

The Core Idea

Pruna makes compilation portable. It saves the required artifacts so they can be easily packaged with your model and reused on another machine (with the same hardware architecture and CUDA drivers) without needing to recompile from scratch. That way, the model will run fast right from the first inference.

The Benefits

Faster deployment: Skip the first-run delay when deploying pre-compiled models to production servers, especially serverless instances.
Easier collaboration: Share ready-to-run models with your team.
Smoother pipelines: Speed up CI/CD by avoiding repeated compilation.

How-to Use Pruna’s Portable Compilation

Let's walk through how to use this feature:

Load your model as normally: In our example, we use a Stable Diffusion pipeline from Diffusers.
Configure Pruna for Portable Compilation: This is where the magic happens. Create a SmashConfig object and configure torch_compile to be portable.
Smash the Model: Apply the configuration using smash().
Run and Save the Model: Run your model for the first time trigger compilation process, including the warmup. After that, just save your Pruna-smashed model, and it’ll be ready to use on any other machine.

import torch
from diffusers import StableDiffusionPipeline
from pruna import SmashConfig, smash

# Load the model
pipe = StableDiffusionPipeline.from_pretrained(
    "CompVis/stable-diffusion-v1-4",
    torch_dtype=torch.float16
).to("cuda")

# Configure torch.compile and combine it with other Pruna features, as caching
smash_config = SmashConfig(
    {
        "deepcache": {},
        "torch_compile": {"torch_compile_make_portable": True}
    }
)

# Smash the model
pipe = smash(pipe, smash_config=smash_config)

# Run the model for the first time
pipe("a photo of an astronaut riding a horse on mars")

# Save the smashed model, including its portable compilation artifacts
pipe.save_pretrained("smashed_sd_portable_model/")

Use case 2: Zero Warmup for LoRA Switching with Diffusers Hotswap and Pruna (`torch.compile`) Compatibility

The Problem

Low-Rank Adaptation (LoRA) is a game-changer for efficiently fine-tuning large models. It allows for quick adaptation by training only a small set of parameters.

A powerful workflow involves dynamically switching between different LoRAs on a base model to change its output on the fly—for instance, altering image styles in a generative model. However, a challenge arises when you combine it with compilation. Every LoRA swap can look like a graph change—triggering a long recompilation and wiping out the speed advantage.

The Core Idea

While Diffusers handles the mechanics of LoRA hotswapping, using Pruna with torch.compileand leveraging one of its cachers ensures that these Diffusers-driven LoRA swaps are efficient and don't cause recompilation warmups after the initial model compilation.

The Benefits

With Pruna and Diffusers together, you get flexible LoRA adaptation and high-performance execution with no warmup delays.

Instant LoRA swaps: Serve models that adapt to diverse user inputs by loading different LoRAs or applications requiring rapid switching between LoRA-defined styles or functionalities (e.g., in an image generation UI), without the latency of recompilation.
Efficient experimentation: Test multiple LoRAs quickly without waiting for recompiles.

How-to Leverage Diffusers Hotswap with Pruna for Zero Warmup

Let's walk through how this works:

Load the Base Model and Enable Diffusers LoRA Hotswapping.
Configure Pruna: Configure torch.compile and enable a cacher. In this example, we will be using the fora cacher, but others also maintain compatibility.
Smash the Model: Apply the configuration using smash().
Run the Model: Run the model for the first time triggering the torch.compile warmup for the base model and the current LoRA. Then, you’ll be ready to hotswap to a new LoRA

import torch
from diffusers import FluxPipeline
from pruna import SmashConfig, smash

# Load the base model and enable LoRA hotswapping
pipe = FluxPipeline.from_pretrained(
    "black-forest-labs/FLUX.1-dev",
    torch_dtype=torch.bfloat16
).to("cuda")
pipe.enable_lora_hotswap(target_rank=128) # target_rank is an example

# Load an initial LoRA
pipe.load_lora_weights("alvdansen/frosting_lane_flux") # Example LoRA

# Configure Pruna's `torch.compile` and `fora`
smash_config = SmashConfig(
    {
        "fora": {"fora_interval": 2, "fora_start_step": 2},
        "torch_compile": {}
    }
)
smash_config._prepare_saving = False # `False`for experimentation

# Smash the model
pipe = smash(
    model=pipe,
    smash_config=smash_config,
)

# Run the model for the first time
prompt ="a cat jumping in the air to catch a bird"
generator = torch.Generator("cpu").manual_seed(0)
pipe(prompt, num_inference_steps=28, generator=generator).images[0]

Comparing the Solutions: Portable Compilation vs. Pruna Cacher Compatibility

While we separately presented these use cases, they can be easily combined:

Use portable compilation to create a base smashed model (perhaps with a default LoRA and apply Pruna optimization that loads quickly on new instances.
Once loaded, pruna’s compatibility with hot-swapping would ensure that any subsequent LoRA hot swaps (managed by Diffusers) on that instance are also free of torch.compile warmup delays.

This combined approach would give you a fast cold start and adapter switching.

Conclusions: Reclaim Your Time with Pruna

The torch.compile warmup can slow down production workflows for cold starts and adapter switching. Pruna addresses these challenges with two key features:

Portable compilation (torch_compile_make_portable=True) removes first-run warmup when deploying to identical hardware, enabling immediate peak performance.
Diffusers' LoRA hotswapping with torch.compile and a Pruna cacher enables instant LoRA switching without recompilation delays.

For background on PyTorch's compilation and caching mechanisms, you might find the official PyTorch torch.compile Caching Tutorial insightful.

We hope this guide helps you optimize your torch.compile workflows. Happy coding!

Enjoy the Quality and Efficiency!

Want to take it further?

Compress your own models with Pruna and give us a ⭐ to show your support!
Stay up to date with the latest AI efficiency research on our blog, explore our materials collection, or dive into our courses.
Join the conversation and stay updated in our Discord community.

Measuring What Matters: Objective Metrics for Image Generation Assessment

Sara Han — Wed, 03 Dec 2025 18:05:47 +0000

🎊 Announcement: Try our performance models for free, P-image and P-image-Edit, here. Just 1 second without compromising the quality!

Generating high-quality visuals with state-of-the-art models is becoming increasingly accessible. Open-source models run on laptops, and cloud services turn text into images in seconds. These models are already reshaping industries like advertising, gaming, fashion, and science.

But creating images is the easy part. Judging their quality is much harder. Human feedback is slow, expensive, biased, and often inconsistent. Plus, quality has many faces: creativity, realism, and style don’t always align. Improving one can hurt another.

That’s why we need clear, objective metrics that capture quality, coherence, and originality. We’ll explore methods for evaluating image quality and comparing models with Pruna, beyond simply asking "does it look cool?"

Metrics Overview

There is no single correct way to categorize evaluation metrics, as a metric can belong to multiple categories depending on its usage and the data it evaluates. In our repository, all quality metrics can be computed in two modes: single and pairwise.

Single mode evaluates a model by comparing the generated images to input references or ground truth images, producing one score per model.
Pairwise mode compares two models by directly evaluating the generated images from each model together, producing a single comparative score for these two models.

This flexibility enables both absolute evaluations (assessing each model individually) and relative evaluations (direct comparisons between models).

On top of the evaluation modes, it also makes sense to think about metrics in terms of their evaluation criteria to provide structure and clarity. Our metrics fall into two overarching categories:

Efficiency Metrics: Measure the speed, memory usage, carbon emissions, energy, etc., usage of models during inference. At Pruna, we focus on making your models smaller, faster, cheaper, and greener, so evaluating your models using these efficiency metrics is a natural fit. However, because efficiency metrics are not specific to image generation tasks, we won't discuss them in detail in this blog post. If you'd like to learn more about these metrics, please refer to our documentation.
Quality Metrics: Measure generated images' intrinsic quality and alignment to intended prompts or references. These include:
- Distribution Alignment: How closely generated images resemble real-world distributions.
- Prompt Alignment: Semantic similarity between generated images and their intended prompts.
- Perceptual Alignment: Pixel-level or perceptual similarity between generated and reference images.

The table below summarizes the most common quality metrics available at Pruna, their categories, score ranges, and key limitations to help guide metric selection.

Metric	Measures	Category	Range (↑ higher is better/↓ lower is better)	Limitations
FID	Distributional similarity to real images	Distribution Alignment	0 to ∞ (↓)	Assumes Gaussianity, requires a large dataset, depends on a surrogate model
CMMD	CLIP-space distributional similarity	Distribution Alignment	0 to ∞ (↓)	Kernel choice affects results, depends on a surrogate model
CLIPScore	Image-text alignment	Prompt Alignment	0 to 100 (↑)	Insensitive to image quality, depends on a surrogate model
PSNR	Pixel-wise similarity	Perceptual Alignment	0 to ∞ (↑)	Not well perceptually aligned
SSIM	Structural similarity	Perceptual Alignment	-1 to 1 (↑)	Can be unstable for small input variations
LPIPS	Perceptual similarity	Perceptual Alignment	0 to 1 (↓)	depends on a surrogate model

Distribution Alignment Metrics

Distribution alignment metrics measure how closely generated images resemble real-world data distributions, comparing both low- and high-dimensional features. In pairwise mode, they compare outputs from different models to produce a single score that reflects relative image quality.

The generated image closely resembles the real one, and the distributions are well aligned, suggesting good quality.

The generated image is noticeably off, and the distributions differ significantly, which the metric captures as a mismatch.

Fréchet Inception Distance (FID): FID (introduced here) is one of the most popular metrics for evaluating how realistic AI-generated images are. It works by comparing the feature distribution of the reference images (e.g., real images) to the images generated by the model to evaluate.

Here’s how it works in a nutshell:

We take a pretrained surrogate model and pass both real and generated images through it. The pretrained surrogate model is usually the Inception v3, explaining the metric name.
The model turns each image into a feature embedding (a numerical summary of the image). We assume the embeddings from each set form a Gaussian distribution.
FID then measures the distance between the two distributions — the closer they are, the better.

A lower FID score indicates that the generated images are more similar to real ones, meaning better image quality.

Want the math?
FID is calculated as the Fréchet distance between two multivariate Gaussians:

$\text{FID} = |\mu_r - \mu_g|^2 + \text{Tr}(\Sigma_r + \Sigma_g - 2(\Sigma_r \Sigma_g)^{1/2})$
where:

$(μr,Σr)(\mu_r, \Sigma_r)$ are the mean and covariance of real image features,

$(μg,Σg)(\mu_g, \Sigma_g)$ are the mean and covariance of generated image features,denotes the trace of a matrix},

$(ΣrΣg)1/2(\Sigma_r \Sigma_g)^{1/2}$ is the geometric mean of the covariance matrices.

Clip Maximum-Mean-Discrepancy (CMMD): CMMD (introduced here) is another way to measure how close your generated images are to real ones. Like FID, it compares feature distributions, but instead of using Inception features, it uses embeddings from a pretrained CLIP model.

Here’s how it works:

We take a pretrained surrogate model and pass both real and generated images through it. The pretrained surrogate model is usually the CLIP.
The model turns each image into a feature embedding (a numerical summary of the image). We do not assume the embeddings from each set form a Gaussian distribution.
Use a kernel function (usually RBF) to compare how these distributions differ, without assuming they are Gaussian.

A lower CMMD score indicates that the feature distributions of generated images are more similar to those of real images, meaning better image quality.

Want the math?
CMMD is based on the Maximum Mean Discrepancy (MMD) and is computed as:

$\text{CMMD} = \mathbb{E}!\left[ k(\phi(x_r), \phi(x_r')) \right]+ \mathbb{E}!\left[ k(\phi(x_g), \phi(x_g')) \right]- 2\,\mathbb{E}!\left[ k(\phi(x_r), \phi(x_g)) \right]$
where:

$ϕ(xr),ϕ(xr′)\phi(x_r), \phi(x_r')$ are two independent real image embeddings extracted from CLIP.

$ϕ(xg),ϕ(xg′)\phi(x_g), \phi(x_g')$ are two independent generated image embeddings extracted from CLIP.

$k (x, y)$ is a positive definite kernel function that measures similarity between embeddings.

The expectations $E[⋅]\mathbb{E}[\cdot]$ are computed over multiple sample pairs.

Prompt Alignment Metrics

Prompt alignment metrics evaluate how well generated images match their input prompts, especially in text-to-image tasks. In pairwise mode, they instead measure semantic similarity between outputs from different models, shifting focus from prompt alignment to model agreement.

CLIPScore: CLIPScore (introduced here) tells you how well a generated image matches the text prompt that produced it. It uses a pretrained CLIP model, which maps both text and images into the same embedding space.

Here’s the idea:

Pass the image and its prompt through the surrograte CLIP model to get their embeddings.
Measure how close these two embeddings are. The closer they are, the better the alignment between the image and the prompt.

CLIPScore ranges from 0 to 100. A higher score means the image is more semantically aligned with the prompt. Note that this metric doesn’t assess visual quality, but rather the match in meaning.

Want the math?
Given an image $x$ and its corresponding text prompt $t$, CLIP Score is computed as:

$\text{CLIPScore} = \max!\left(100 \times \frac{\phi_I(x) \cdot \phi_T(t)} {|\phi_I(x)|\,|\phi_T(t)|},\; 0\right)$
where:

$ϕI(x)\phi_I(x)$ is the CLIP image embedding of the generated image.

$ϕT(t)\phi_T(t)$ is the CLIP text embedding of the associated prompt. CLIP Score ranges from 0 to 100, with higher scores indicating better alignment between an image and its corresponding text prompt. However, it can be insensitive to image quality, as it focuses on semantic similarity rather than visual fidelity.

Perceptual Alignment Metrics

Perceptual alignment metrics evaluate the perceptual quality and internal consistency of generated images. They compare pixel-level or feature-level differences between images. These metrics are often pairwise by nature, as comparing generated images with other generated images is more appropriate in certain cases, such as pixel-by-pixel comparisons.

Peak Signal-to-Noise Ratio (PSNR): PSNR measures the pixel-level similarity between a generated image and its reference (ground truth) image. It is widely used for evaluating image compression and restoration models.

A higher PSNR value indicates better image quality, but PSNR does not always correlate well with human perception.

Want the math?
PSNR is computed as:

$\text{PSNR} = 10 \times \log_{10}!\left(\frac{L^2}{\text{MSE}}\right)$
where:

$L$ is the maximum possible pixel value (e.g., 255 for an 8-bit image).

$MSE\text{MSE}$ (Mean Squared Error) is the average squared difference between pixel values.

Structural Similarity Index (SSIM): SSIM improves upon PSNR by comparing local patterns of pixel intensities instead of just raw pixel differences. It models human visual perception by considering luminance, contrast, and structure in small image patches

SSIM ranges from -1 to 1, where 1 indicates perfect similarity.

Want the math?
SSIM is often computed as:

$\text{SSIM}(x, y) = \frac{(2\mu_x\mu_y + C_1)(2\sigma_{xy} + C_2)} {(\mu_x^2 + \mu_y^2 + C_1)(\sigma_x^2 + \sigma_y^2 + C_2)}$
where:

$μx,μy\mu_x, \mu_y$ are the mean intensities of images $x$ and $y$ .

$σx2,σy2\sigma_x^2, \sigma_y^2$ are the variances.

$σxy\sigma_{xy}$ is the covariance between the images.

$C_1, C_2$ are small constants for stability.

**Learned Perceptual Image Patch Similarity (LPIPS): **LPIPS is a deep-learning-based metric that measures perceptual similarity between images using features from a pre-trained neural network (e.g., VGG, AlexNet). Unlike PSNR and SSIM, LPIPS captures high-level perceptual differences rather than pixel-wise differences.

Want the math?
LPIPS is computed as:

$\text{LPIPS}(x, y) = \sum_l w_l \,|F_l(x) - F_l(y)|_2^2$
where:

$F_l(x)$ and $F_l(y)$ are deep feature representations of images $x$ and $y$ from layer $l$ .

$w_l$ are learned weights that adjust the importance of each feature layer.

To illustrate how different distortions impact metric scores, let's look at the following example. The image below showcases various distortions applied to an original image and how metrics like SSIM, PSNR, and LPIPS react to these changes.

The results in the image illustrate how different types of distortions affect the scores given by these task-based metrics. Notably:

Blurred images tend to score higher in SSIM than in PSNR. This suggests that while fine details are lost, the overall structure and patterns of the image remain intact, which aligns with SSIM’s focus on structural consistency.
Pixelated images, on the other hand, maintain relatively high PSNR values but drop in SSIM ranking. This indicates that while pixel intensity differences remain small, the structural coherence of the image is significantly degraded—highlighting SSIM’s sensitivity to spatial relationships rather than just pixel-level accuracy.

These observations demonstrate why selecting the right metric is crucial. Each of the metrics captures different aspects of image quality, making them useful in different scenarios depending on the type of distortion and the perceptual quality being assessed.

Confidently evaluate AI models with the Evaluation Agent!

The evaluation framework in pruna consists of several key components:

Step 1: Define what you want to measure

Use the Task object to specify which quality metrics you'd like to compute. You can provide the metrics in three different ways depending on how much control you need.

from pruna.evaluation.task import Task
from pruna.data.pruna_datamodule import PrunaDataModule
from pruna.evaluation.metrics.metric_torch import TorchMetricWrapper

# Method 1: plain text from predefined options
evaluate_image_generation_task = Task("image_generation_quality", datamodule=PrunaDataModule.from_string('LAION256'))

# Method 2: list of metric names
metrics = ['clip_score', 'psnr']
evaluate_image_generation_task = Task(request = metrics, datamodule=PrunaDataModule.from_string('LAION256'))

# Method 3: list of metric instances
clip_score_metric = TorchMetricWrapper("clip_score", model_name_or_path = "openai/clip-vit-base-patch32")
psnr_metric = TorchMetricWrapper('psnr', base=2.0)
metrics = [clip_score_metric, psnr_metric]
evaluate_image_generation_task = Task(request= metrics, datamodule=PrunaDataModule.from_string('LAION256'))

Step 2: Run the Evaluation Agent

Pass your model to the EvaluationAgent and let it handle everything: running inference, computing metrics, and returning the final scores.

from pruna.evaluation.evaluation_agent import EvaluationAgent

eval_agent = EvaluationAgent(evaluate_image_generation_task)
results = eval_agent.evaluate(your_model)

As AI-generated images become more prevalent, evaluating their quality effectively is more important than ever. Whether you're optimizing for realism, accuracy, or perceptual similarity, selecting the right evaluation metric is key. With Pruna now open-source, you have the freedom to explore, customize, and even contribute new evaluation metrics to the community .

Our documentation and tutorials (here) provides a step-by-step guide on how to add your own metrics, making it easier than ever to tailor evaluations to your needs. Try it out today, contribute, and help shape the future of AI image evaluation!

Enjoy the Quality and Efficiency!

Compress your own models with Pruna and give us a ⭐ to show your support!
Try our models and endpoints in Replicate with just one click.
Stay up to date with the latest AI efficiency research on our blog, explore our materials collection, or dive into our courses.
Join the conversation and stay updated in our Discord community.

Introducing Pruna 0.3.0 - The Upgrade You’ve Been Waiting For

Sara Han — Mon, 17 Nov 2025 15:59:11 +0000

Today, we are excited to announce that we have released the long-awaited Pruna 0.3.0.

We’ve restructured our internal framework to make algorithm management more flexible and scalable, setting the stage for even more powerful algorithm support going forward.

Why the Refactor

In previous versions, certain algorithm groups — such as cachers or quantizers — were tightly coupled to the package’s structure. This rigid grouping made it difficult to introduce new types of algorithms or to combine them in flexible ways.

Starting with Pruna 0.3.0, we’ve reworked this system so that such classifications are no longer hard constraints. Instead, they now serve as supplementary metadata, enabling a more modular, composable, and future-proof design. This refactor lays the groundwork for integrating new optimization techniques and custom pipelines without structural limitations.

This is a ground refactorization that enables two things:

Instead of applying algorithms in a fixed way defined by their group, compression algorithms can be applied in flexible orders regardless of their group.
Instead of constraining one algorithm per group in the SmashConfig, multiple algorithms from the same group can be combined as long as they are marked as compatible.

What This Means for You

You don’t need to do anything special — just upgrade to the new version and you’ll be ready to go!

pip install --upgrade pruna

Once upgraded, everything will work out of the box. While we’ve slightly refined how configuration is defined (for the better!), the old interface would still be valid. You can find all the details in the next section.

What Changed

A More Flexible Algorithm Interface

This release introduces a more flexible configuration interface for algorithms.

Before, you had to define your SmashConfig step by step.

from pruna import SmashConfig

config = SmashConfig()
config["compiler"] = "torch_compile"
config["cacher"] = "deepcache"

Now, with this release, you can do it all in one line with a list of the algorithm names — faster and simpler.
```
from pruna import SmashConfig

config = SmashConfig(["torch_compile", "deepcache"])
```

A More Flexible Hyperparameters Interface

This release introduces a more flexible configuration interface for hyperparameters.

Before, if you needed to specify algorithm parameters, you no longer had to go through the tedious process of setting each one.

from pruna import SmashConfig

config = SmashConfig()
config["compiler"] = "torch_compile"
config["torch_compile_fullgraph"] = True
config["torch_compile_mode"] = "max-autotune"
config["quantizer"] = "hqq"
config["hqq_weight_bits"] = 4
config["hqq_compute_dtype"] = "torch.bfloat16"

Now, you can now use a dictionary-style configuration to define detailed, per-algorithm parameters all at once.

from pruna import SmashConfig

config = SmashConfig({
      "hqq":
          {
              "weight_bits": 4,
              "compute_dtype": "torch.bfloat16"
          },
      "torch_compile":
          {
          "fullgraph": True,
          "mode": "max-autotune"
          }
})

A More Flexible Algorithm Ordering and Compatibility

Another major change is how the algorithm application order is determined.

Previously, the execution sequence was dictated by the hierarchy of algorithm groups and a global ordering based on these groups. In 0.3.0, this has been replaced by a more atomic and declarative system: each algorithm now specifies its own compatibility rules and ordering constraints. If an algorithm is compatible with another one, it will now always specify in which order they can be executed.

This makes the algorithm pipeline more self-organizing, robust to new extensions, and capable of resolving valid combinations dynamically.

New documentation

To make sure you have everything you need, we’ve also updated our documentation. You can now easily find the latest guides and tutorials under the “Open Source” tab.

Get Started Now

Enjoy the Quality and Efficiency!

Compress your own models with Pruna and give us a ⭐ to show your support!
Try our endpoints in Replicate, Wiro or Segmind with just one click.
Stay up to date with the latest AI efficiency research on our blog, explore our materials collection, or dive into our courses.
Join the conversation and stay updated in our Discord community.

Effective Prompting for Generative Vision Models

Sara Han — Mon, 10 Nov 2025 18:14:43 +0000

It’s likely that you’ve used a vision model to generate an image recently, but ended up with somewhat questionable results. You might have blamed this on the model not working correctly (and maybe that’s true), but it could also be because you didn’t give it the proper instructions.

A vision model will only create what it’s asked to, and how you ask matters. Prompting isn’t just about describing what you see; it’s about guiding the model so it interprets your request correctly. Just one word can sometimes double its accuracy.

In this blog, we’ll cover the key principles for prompting your vision models more effectively, from good practices to the nuances of different use cases. Whether you’re a developer, designer, marketer, or beginner, this guide will help you achieve the results you’re looking for.

Where to Test Your Prompts

Before diving into how vision prompting works, let’s first look at where we can put it to the test. In this case, we’ll be using several endpoints available on Replicate, which we’ve optimized with Pruna to make them cheaper, faster, and more efficient. All of Pruna’s models are available here.

Prompting Good Practices

While there are nuances that can be applied to each use case, there are also several key principles that should always be kept in mind when prompting a model:

Give direction: State the goal, task, context, or desired style.
Be clear: Use precise, unambiguous language. You don’t need to describe every detail, just select the key words that matter most.
Split the work: If the goal is complex, break the prompt down into several chained steps.
Provide examples: If possible, include an example and reference it in your prompt.
Tune your prompts: Always review the output and refine your prompts based on the results to get better responses. Using a grid can be helpful.
Know your model: Review the model’s documentation or description. Some models support tags, parameters, or specific input formats that can significantly improve performance.

Prompting in Practice

From Words to Pictures

For image generation, you can craft the perfect prompt following a default structure: Subject + Subject’s Action + Style + Context.

Subject: Where is the focus of your image? It should be the main element of your image (person, object, animal, or scene).
Subject’s Action: What’s the subject doing? It should describe what the subject is doing or how it interacts with the environment.
Style: How is the image presented? It should specify the artistic direction or medium.
Context: How and where is it happening? It should include the background, lighting, atmosphere, mood, point of view, or colors.

When writing the prompt, make sure each element is descriptive and focused only on the specific element you want to generate, avoiding contradictions. If it's abstract or vague, it can lead to unpredictable results. For example, a prompt like “The best thing you can draw” is too ambiguous and might not produce anything appealing or coherent. Similarly, simply copying and pasting random text from the internet won’t work well — the model will struggle to extract a clear meaning or visual direction from it.

From Text or Image to Video

For video generation, we can use a similar structure as for image generation. However, some extra aspects should be considered: Subject + Subject’s Action + Environment + Shot Type + Style + Context.

Subject: Who or what is the main focus of your video? It should be the main element of your scene (person, object, animal).
Subject’s Action: What’s the subject doing? It should describe what the subject is doing or how it interacts with the environment.
Environment: Where is it happening? It should include the scene details surrounding the subject.
Shot Type: What’s the camera’s perspective or movement? It should describe the angle, trajectory, movement, and speed of the camera.
Style: How is the image presented? It should specify the artistic direction or medium.
Context: How is it happening? It should include the background, lighting, atmosphere, mood, point of view, or colors.

Editing Images

For image editing, we should introduce a new prompt structure: Task + Target + Edit Type + Preservation

Task: What do you want to accomplish? It should define the main goal of the edit.
Target: What specific element should be edited? It should identify the subject or area to modify.
Edit Type: How should the change be applied? It should describe the method, intensity, or style of the edit.
Preservation: What should remain unchanged? It should specify which parts of the image mustn’t change.

More Considerations

On one hand, even though most vision models have recently improved — with greater care taken in training data and design — different biases can persist. That’s why, when prompting, it’s important not to reinforce them. You can mitigate this by evaluating the outputs to ensure diversity and representation, and by providing more context and detail.

On the other hand, prompting in vision models raises a range of ethical questions that go beyond bias. Therefore, it’s essential to consider factors such as consent, authorship, data protection, and manipulation when using them.

What’s Next

In conclusion, this blog post provides a structured and straightforward guide to get started with prompting a vision model. So, you can generate an image or video, or edit an existing one to suit your needs.

Enjoy the Quality and Efficiency!

Want to take it further?

Compress your own models with Pruna and give us a ⭐ to show your support!
Stay up to date with the latest AI efficiency research on our blog, explore our materials collection, or dive into our courses.
Join the conversation and stay updated in our Discord community.

Making AI Models Faster, Cheaper, and Greener — Here’s How

Sara Han — Mon, 03 Nov 2025 13:11:27 +0000

In this blog, we present the key techniques to gain AI efficiency, meaning models that are:

Faster: Accelerate inference times through advanced optimization techniques
Smaller: Reduce model size while maintaining quality
Cheaper: Lower computational costs and resource requirements
Greener: Decrease energy consumption and environmental impact

For this, Pruna provides an open-source toolkit that simplifies scalable inference, requiring just a few lines of code to optimize your models in each of the mentioned aspects.

So first, let’s take a quick look at an overview of these techniques, and then we’ll dive deeper into each one.

Optimization Techniques

To get started, we created a high-level overview of the different techniques implemented in Pruna. This list can be further enriched; however, it provides a solid basis for your understanding.

Technique	Description	Impacts
Batching	Groups multiple inputs together to be processed simultaneously, improving computational efficiency and reducing overall processing time.	Speed (✅), Memory (❌), Accuracy (~)
Caching	Stores intermediate results of computations to speed up subsequent operations, reducing inference time by reusing previously computed results.	Speed (✅), Memory (⚠️), Accuracy (~)
Speculative Decoding	Speculative decoding speeds up AI text generation by having a small, fast model predict several tokens at once, which a larger model then verifies, creating an efficient parallel workflow.	Speed (✅), Memory (❌), Accuracy (⚠️)
Compilation	Compilation optimizes the model with instructions for specific hardware.	Speed (✅), Memory (➖), Accuracy (~)
Distillation	Trains a smaller, simpler model to mimic a larger, more complex model.	Speed (✅), Memory (✅), Accuracy (❌)
Quantization	Reduces the precision of weights and activations, lowering memory requirements.	Speed (✅), Memory (✅), Accuracy (❌)
Pruning	Removes less important or redundant connections and neurons, resulting in a sparser, more efficient network.	Speed (✅), Memory (✅), Accuracy (❌)
Recovering	Restores the performance of a model after compression.	Speed (⚠️), Memory (⚠️), Accuracy (🟢)

✅(improves), ➖(stays the same), ~(could worsen), ❌(worsens)

Technique requirements and constraints

Before we continue, note that each one of these techniques and their underlying implementation algorithms has specific requirements and constraints. Some techniques can only be applied on specific hardware, like GPUs, or models like LLMs or image generation models. Others might require a tokenizer, processor, or dataset to function. Lastly, not all techniques can be used interchangeably, and therefore have compatibility limitations.

The Optimization Techniques

We will now dive a bit deeper into different optimization techniques. Although we will dive a bit deeper into the various techniques and their underlying algorithms, we will not be going into the nitty-gritty details, and keep it high level, and for each technique, highlight one of the fundamental underlying algorithms that has been implemented in the Pruna library.

Batching AI model inference

Batching groups multiple inputs together to be processed simultaneously, improving computational efficiency and reducing overall processing time. Instead of processing one prompt at a time, the GPU processes multiple prompts in parallel, maximizing hardware utilization. This significantly increases throughput since modern GPUs are designed for parallel computation. Batching reduces the per-example computational overhead and allows for better distribution of fixed costs across multiple inputs, thus often increasing the throughput.

For batching, we implemented WhisperS2T, which works on top of Whisper models. It intelligently batches smaller speech segments and is designed to be faster than other implementations, boasting a 2.3X speed improvement over WhisperX and a 3X speed boost compared to HuggingFace Pipeline with FlashAttention 2 (Insanely Fast Whisper).

Caching intermediate results

Caching stores intermediate results of computations to speed up subsequent operations, reducing inference time by reusing previously computed results. For transformer-based LLMs, this typically involves storing key-value pairs from previous tokens to avoid redundant computation. When generating text token by token, each new token can reuse cached computations from previous tokens rather than recomputing the entire sequence. This dramatically improves inference efficiency, especially for long-context applications. However, caching goes beyond only saving KV computations and can be used in multiple places for LLMs and image generation models.

For caching, we implemented DeepCache, which works on top of diffuser models. DeepCache accelerates inference by leveraging the U-Net blocks of diffusion pipelines to reuse cached high-level features. The nice thing is that it is training-free and almost lossless, while accelerating models 2X to 5X.

Speculative decoding with parallelizing generation

Speculative decoding improves the efficiency of language model inference by parallelizing parts of the generation process. Instead of generating one token at a time, a smaller, faster draft model generates multiple candidate tokens in a single forward pass. The larger, more accurate model then verifies or corrects these tokens in parallel, allowing for faster token generation without significantly sacrificing output quality. This approach reduces the number of sequential steps required, thereby lowering overall latency and accelerating inference. It’s essential to note that the effectiveness of speculative decoding depends on the alignment between the draft and target models, as well as the chosen parameters, such as batch size and verification strategy.

For speculative decoding, we have not implemented any algorithms. Yet! Stay tuned to discover our future speculative decoding algorithms.

Compilation for specific hardware

Compilation optimizes the model for specific hardware by translating the high-level model operations into low-level hardware instructions. Compilers like NVIDIA TensorRT, Apache TVM, or Google XLA analyze the computational graph, fuse operations where possible, and generate optimized code for the target hardware. This process eliminates redundant operations, reduces memory transfers, and leverages hardware-specific acceleration features, resulting in faster inference times and lower latency. It is essential to note that each combination of model/hardware will have a different optimal compilation setup.

For compilation, we implemented Stable-fast, which works on top of diffuser models. Stable-fast is an optimization framework for Image-Gen models. It accelerates inference by fusing key operations into optimized kernels and converting diffusion pipelines into efficient TorchScript graphs.

Distillation for smaller models

Distillation trains a smaller, simpler model to mimic a larger, more complex model. The larger “teacher” model produces outputs that the smaller “student” model learns to replicate, effectively transferring knowledge while reducing computational requirements. This technique preserves much of the performance and capabilities of larger models while significantly reducing parameter count, memory usage, and inference time. Distillation can target specific capabilities of interest rather than general performance.

For distillation, we implemented Hyper-SD, which works on top of diffusion models. Hyper-SD is a distillation framework that segments the diffusion process into time-step groups to preserve and reformulate the ODE trajectory. By integrating human feedback and score distillation, it enables near-lossless performance with drastically fewer inference steps.

Quantization for lower precision

Quantization reduces the precision of weights and activations, lowering memory requirements by converting high-precision floating-point numbers (FP32/FP16) to lower-precision formats (INT8/INT4). It reduces model size, memory bandwidth requirements, and computational complexity. Modern quantization techniques, such as Post-Training Quantization (PTQ) and Quantization-Aware Training (QAT), minimize accuracy loss while achieving substantial efficiency gains. Hardware accelerators often have specialized support for low-precision arithmetic, further enhancing performance.

For quantization, we implemented Half-Quadratic Quantization (HQQ), which works on top of any model. HQQ utilizes fast and robust optimization techniques for on-the-fly quantization, eliminating the need for calibration data and making it applicable to any model. This algorithm is adapted explicitly for diffuser models.

Pruning away redundant neurons

Pruning removes less important or redundant connections and neurons, resulting in a sparser, more efficient network. Various pruning strategies exist, including magnitude-based pruning (removing the smallest weights) and lottery ticket hypothesis approaches (finding sparse subnetworks). Key design choices typically involve deciding which structure to prune (e.g., weight, neuron, blocks) and determining how to score structures (e.g., using weight magnitude, first-order, or second-order information). Pruning can significantly reduce model size (often by 80-90%) with minimal performance degradation when done carefully. Sparse models require specialized hardware or software support to realize computational gains.

For pruning, we implemented structured pruning, which works on top of any model. Structured pruning removes entire units like neurons, channels, or filters from a network, leading to a more compact and computationally efficient model while preserving a regular structure that standard hardware can easily optimize.

Recovering performance with training

Recovering is special since it allows for improving the performance of compressed models. After compression, it restores the performance of a model through techniques like finetuning or retraining. After aggressive pruning, models typically experience some performance degradation, which can be mitigated by additional training steps. This recovery phase allows the remaining parameters to adapt and compensate for the compression. Approaches for efficient recovery include learning rate rewinding, weight rewinding, and gradual pruning with recovery steps between pruning iterations. The recovery process helps achieve optimal trade-offs between model size and performance.

For recovering, we implemented text-to-text PERP, which works on top of text generation models. This recoverer is a general-purpose PERP recoverer for text-to-text models using norm, head, and bias finetuning and optionally HuggingFace’s LoRA. Similarly, we support text-to-image PERP for other image generation models.

What’s next?

This blog provided a brief introduction to each of these categories, but there are many more nuances, techniques, and implementations that we will highlight in upcoming blogs. The cool thing is that each of these techniques has been implemented in the open-source Pruna library and is ready for you to experiment with!

Enjoy the Quality and Efficiency!

Want to take it further?

Compress your own models with Pruna and give us a ⭐ to show your support!
Explore our materials collection, or dive into our courses.
Join the conversation and stay updated in our Discord community.

The AI efficiency framework from Pruna AI is now open-source

Bertrand Charpentier — Thu, 20 Mar 2025 12:24:56 +0000

I am Bertrand from Pruna AI. Together with John, Rayan, Stephan, we created Pruna AI to tackle challenges in AI model optimization. We’re a group of researchers in AI efficiency and reliability, originally from TUM.

Since we got so many times questions on how compression of AI models was working under the hood, we decided to open-source the pruna package with the help of all the Pruna AI team. As a whole, the pruna package is an AI efficiency framework that can be installed with pip install pruna to compress models, and thus save memory and compute power when running AI models for inference.

With open-sourcing, people can now inspect and contribute to the open code. Beyond the code, we provide detailed readme, tutorials, benchmarks, and documentation (https://docs.pruna.ai/en/stable/index.html) to make transparent compression, evaluation, and saving/loading/serving of AI models.

Beyond the open-source package, we commercially offer pruna_pro with advanced compression methods, recovery methods, and an optimization agent to unlock greater efficiency and productivity gains.

We are pleased to share this with you all. We would be glad to hear your thoughts and questions in the comments :)

DEV Community: Pruna AI

Pruna 0.3.2: More OSS Algos, More Ways to Optimize

What Landed in 0.3.2

Meet the New Algorithms and Families

Expanding Existing Families

Introducing New Families

More Efficient Strategies

Enjoy the Quality and Efficiency!

LLM Architectures Explained: What Powers Today’s Top Models

Where It All Begins: Tokenizers and Embeddings

Token-by-Token: The Autoregressive Way

Thinking in States: A Different Way to Think About Sequences

Removing the Noise: Diffusion LLMs

Overview at a Glance

What's Next?

Enjoy the Quality and Efficiency!

FLUX.2 [flex] Challenge

Slashing torch.compile Warmup & LoRA Swapping Times with Pruna

The Challenge: Understanding torch.compile Warmup

Use Case 1: Eliminating Initial Warmup with Pruna's Portable Compilation

The Problem

The Core Idea

The Benefits

How-to Use Pruna’s Portable Compilation

Use case 2: Zero Warmup for LoRA Switching with Diffusers Hotswap and Pruna (torch.compile) Compatibility

The Problem

The Core Idea

The Benefits

How-to Leverage Diffusers Hotswap with Pruna for Zero Warmup

Comparing the Solutions: Portable Compilation vs. Pruna Cacher Compatibility

Conclusions: Reclaim Your Time with Pruna

Enjoy the Quality and Efficiency!

Measuring What Matters: Objective Metrics for Image Generation Assessment

Metrics Overview

Distribution Alignment Metrics

Prompt Alignment Metrics

Perceptual Alignment Metrics

Confidently evaluate AI models with the Evaluation Agent!

Enjoy the Quality and Efficiency!

Introducing Pruna 0.3.0 - The Upgrade You’ve Been Waiting For

Why the Refactor

What This Means for You

What Changed

A More Flexible Algorithm Interface

A More Flexible Hyperparameters Interface

A More Flexible Algorithm Ordering and Compatibility

New documentation

Get Started Now

Enjoy the Quality and Efficiency!

Effective Prompting for Generative Vision Models

Where to Test Your Prompts

Prompting Good Practices

Prompting in Practice

From Words to Pictures

From Text or Image to Video

Editing Images

More Considerations

What’s Next

Enjoy the Quality and Efficiency!

Making AI Models Faster, Cheaper, and Greener — Here’s How

Optimization Techniques

Technique requirements and constraints

The Optimization Techniques

Batching AI model inference

Caching intermediate results

Speculative decoding with parallelizing generation

Compilation for specific hardware

Distillation for smaller models

Quantization for lower precision

Pruning away redundant neurons

Recovering performance with training

What’s next?

Enjoy the Quality and Efficiency!

The AI efficiency framework from Pruna AI is now open-source

The Challenge: Understanding `torch.compile` Warmup

Use case 2: Zero Warmup for LoRA Switching with Diffusers Hotswap and Pruna (`torch.compile`) Compatibility