DEV Community: aimodels-fyi

A beginner's guide to the Qwen3.8-Flash-Next model by Qwen on Huggingface

aimodels-fyi — Wed, 09 Sep 2026 03:02:40 +0000

This is a simplified guide to an AI model called Qwen3.8-Flash-Next maintained by Qwen. If you like these kinds of analysis, you should join AImodels.fyi or follow us on Twitter.

Overview

Qwen3.8-Flash-Next is an experimental open-weight causal language model with a vision encoder from Qwen. It targets coding agents, long-horizon tool use, multimodal computer tasks, multilingual software engineering, and reasoning. The most important point before adoption is that this is an architecture preview intended to underpin Qwen4, not the production-hosted Qwen3.8-Flash service: the hosted version adds production features such as 1M-token context by default and official built-in tools. The model has 125B total language-model parameters with 6B activated, plus 51B n-gram embedding parameters and 4B multi-token-prediction parameters. It provides 262,144 native context tokens and can extend to 1,000,000 tokens. The repository supplies post-trained weights and configuration in Hugging Face Transformers format, with compatibility listed for Transformers, vLLM, SGLang, TokenSpeed, and other serving systems. The model card lists the license as other; it does not provide terms that establish commercial-use rights, so legal review is required before commercial deployment.

Best use cases

Long-horizon coding agents. The model suits repository-level coding, debugging, test execution, and tool-driven software work. It scores 62.5 on SWE-bench Pro, 81.0 on SWE-bench Multilingual, 58.7 on DeepSWE 1.1, and 91.9 on LiveCodeBench v6. Its 256K evaluation context, sparse attention design, multi-step training, and agent-oriented post-training support workflows that require reading large repositories, planning changes, invoking tools, and recovering from failures.

Multilingual software engineering. Choose it for coding tasks across multiple programming-language and natural-language environments. Its 81.0 SWE-bench Multilingual score exceeds the listed results for Qwen3.8-27B at 73.8 and Qwen3.7-Plus at 75.8. The model’s general instruction-following score of 81.3 and competitive-coding score of 91.9 also support structured implementation and problem-solving tasks.

Tool-using productivity agents. The model performs well on office and professional workflows that require multiple actions rather than a single answer. It scores 73.9 on CoWorkBench, 55.7 on JobBench, 51.2 on Agents’ Last Exam, and 73.5 on Toolathlon Verified. These results make it a candidate for research assistants, document workflows, finance or legal task automation, and other systems that combine reasoning with external tools. The model card warns that reducing reasoning effort can lower total completion time in multi-turn agents by causing failures and retries, so per-turn latency is not the only performance metric.

Multimodal computer-use systems. The vision encoder and multimodal benchmark results support tasks such as navigating Android applications, recreating applications across desktop, mobile, and web platforms, analyzing charts, and solving visual mathematics. It scores 84.5 on AndroidWorld, 19.4 binary and 52.3 partial on OSWorld 2.0, 49.9 on RecreationBench, 64.0 on Vision2Web, and 88.5 on RealWorldQA. These results favor research systems that connect screenshots or visual documents to actions and structured responses.

Scientific and visual reasoning. The model scores 91.7 on GPQA Diamond, 35.9 on HLE, 90.6 on MathVision without CI and 95.7 with CI, and 84.6 on CharXiv without CI and 90.6 with CI. It can support chart interpretation, visual math, scientific question answering, and multidisciplinary analysis. The HLE result remains below Claude-Opus-4.6 (Max) at 40.0, so it should not be treated as the strongest option for every frontier reasoning task.

Limitations

The model is large despite its low 6B activated count. The 125B language-model weights, 51B n-gram embedding parameters, and 4B MTP parameters create substantial storage and memory demands. The provided material gives no VRAM requirement, quantization size, measured tokens-per-second result, latency figure, or recommended batch size. Do not infer hardware capacity from the activated-parameter count: n-gram embeddings and inactive weights still affect deployment memory, while framework implementation determines how much can be offloaded.

The model has a native context length of 262,144 tokens, with extension up to 1,000,000 tokens. The repository does not state the quality or speed behavior at the extended limit. Long context also increases serving complexity, and the model card provides no end-to-end latency measurements. Qwen claims that Qwen Sparse Attention reduces long-context latency, but the supplied information does not quantify the reduction.

Benchmark results show uneven strengths. Qwen3.8-Flash-Next leads the listed models on many agentic, coding, instruction-following, and multimodal rows, but it does not lead every task. On NL2Repo-Bench it scores 48.1, below DeepSeek-V4-Flash-0731 at 54.2. On HLE it scores 35.9, below Claude-Opus-4.6 (Max) at 40.0. On CharXiv without CI it scores 84.6, below Qwen3.7-Plus at 85.8, although its with-CI score is 90.6. Benchmark harnesses, prompts, temperatures, and judges differ, so these numbers do not establish universal quality rankings.

The model operates in thinking mode by default and emits content in <think>\n...\n</think>\n\n before the final response. This can increase output length, cost, and latency. Non-thinking mode exists, but the supplied material does not include the actual disabling code. Sampling-parameter support varies by inference framework.

The license is listed only as other. No license terms appear in the supplied README, so commercial use, redistribution, modification, and hosted-service obligations remain unresolved. The model card also does not provide a detailed bias, safety, privacy, or misuse analysis. Teams need their own evaluation and policy controls.

How it compares

Qwen3.8-Flash-Next-GGUF

Pick Qwen3.8-Flash-Next when you need the original Transformers-format weights and direct compatibility with Transformers, vLLM, SGLang, or TokenSpeed. Pick the GGUF alternative when local deployment through GGUF-compatible tooling and quantization matter more than retaining the original weight format. The supplied information does not provide quantized file sizes, quality deltas, VRAM requirements, or speed measurements, so it cannot establish a numeric cost or accuracy advantage.

Qwen3-Coder-Next

Pick Qwen3.8-Flash-Next for a broader multimodal and agentic system: it includes a vision encoder, reaches 1M-token extensibility, and reports strong computer-use and visual-reasoning results. Pick Qwen3-Coder-Next for coding-agent and local-development workloads when its specialized design and 3B activated parameters are the priority. The key tradeoff is general multimodal breadth and higher reported agent scores versus a smaller, coding-focused deployment profile.

Qwen3.6-35B-A3B

Pick Qwen3.8-Flash-Next when its reported coding-agent, tool-use, long-context, and multimodal capabilities justify a larger deployment. Pick Qwen3.6-35B-A3B when a smaller 35B-total, 3B-activated model better fits local serving constraints. The supplied benchmark table does not include Qwen3.6-35B-A3B, so no direct quality, speed, or cost comparison is available.

Qwen3.8-2.4T-A95B

Pick Qwen3.8-Flash-Next when you need a much smaller open-weight model with 6B activated parameters and a practical path to self-hosting. Pick Qwen3.8-2.4T-A95B when maximum model capacity is more important than infrastructure cost and complexity. The supplied information gives no shared benchmark results, latency data, or pricing, so the tradeoff can be stated only as model scale versus deployment burden.

Qwen3.5-35B-A3B

Pick Qwen3.8-Flash-Next for the newer experimental architecture, 125B total parameters with 6B activated, 262K native context, and the listed multimodal agent results. Pick Qwen3.5-35B-A3B when its 35B-total, 3B-activated profile offers a better fit for constrained infrastructure. The supplied material does not include direct benchmark, speed, pricing, or VRAM comparisons.

Technical specifications

Qwen3.8-Flash-Next uses a causal language model with a vision encoder and includes both pre-training and post-training. Its architecture is an experimental preview for Qwen4.

Total language-model parameters: 125B.
Activated language-model parameters: 6B.
N-gram embedding parameters: 51B.
MTP parameters: 4B.
Hidden dimension: 2,560.
Token embedding size: 248,320, padded.
N-gram embedding table: 20,000,000 entries, using bigrams and trigrams at layer 2.
Layers: 48.
Hidden layout: 12 repetitions of three Gated DeltaNet-to-MoE blocks followed by one Qwen Sparse Attention-to-MoE block.
Gated DeltaNet: 48 linear-attention heads for V, 16 for QK, head dimension 128.
Qwen Sparse Attention: 24 Q heads, 2 KV heads, head dimension 256, rotary position-embedding dimension 64.
QSA indexer: MQA with 4 query heads and 1 shared key head; indexer head dimension 128.
QSA budget: 512 blocks or 2,048 tokens.
MoE: 512 experts; 10 routed experts plus 1 shared expert activated; expert intermediate dimension 640.
Gated Residual: 4 branches; bottleneck rank 320.
LM output size: 248,320, padded.
MTP: one layer trained with multi-steps.
Context: 262,144 tokens natively; extensible to 1,000,000 tokens.
Model format: Hugging Face Transformers weights and configuration.
Listed serving compatibility: Hugging Face Transformers, vLLM, SGLang, TokenSpeed, and other compatible systems.
Recommended production serving engines: SGLang, KTransformers, or vLLM.
Training recipe: Muon and AdamW applied to specific weight categories; batch-size warmups removed; training starts at the target batch size; refitted scaling laws support larger learning rates and fewer optimizer steps.
Architecture innovations: QSA selects micro-blocks rather than individual tokens; Gated Residual uses an element-wise data-dependent read gate and per-branch scalar write gate; n-gram embeddings provide a parameter-scaling path designed for offloading and memory-constrained accelerators.
Repository downloads shown in the supplied metadata: 2,551.
Pipeline tag: image-text-to-text.
Library metadata: transformers.
Model tag: Text-to-Text.
License metadata: other.

The README provides no training-dataset name or size, training-step count, compute budget, VRAM requirement, quantization specification, measured inference speed, or pricing.

Model inputs and outputs

Inputs

Text prompts for causal language-model generation.
Multimodal inputs supported by the model type and image-text-to-text pipeline designation.
Context up to 262,144 tokens natively.
Context extension up to 1,000,000 tokens.
Multi-turn conversations and agent trajectories.
Tool-use workflows, when the surrounding serving or application framework supplies tools.
Thinking-mode generation by default.
Sampling parameters for thinking mode: temperature=1.0, top_p=0.95, top_k=20, min_p=0.0, presence_penalty=0.0, repetition_penalty=1.0.
Sampling parameters for instruct or non-thinking mode: temperature=0.7, top_p=0.80, top_k=20, min_p=0.0, presence_penalty=1.5, repetition_penalty=1.0.

Outputs

Generated text from a causal language model.
Thinking-mode output containing <think>...</think> before the final response.
Final natural-language answers, code, plans, tool arguments, and agent actions as determined by the application.
Vision-conditioned text responses for image-text-to-text workflows.
The model card does not define a universal tool-call schema or built-in tool set for this open-weight release.
Applications that expose thinking content should parse or filter the <think> section before presenting the final answer.

Getting started

The supplied README does not include a complete Python loading and inference example. It recommends API use for streamlined integration and lists deployment recipes for SGLang, vLLM, and TokenSpeed. For production or high-throughput workloads, use a dedicated serving engine and consult the framework-specific recipe. The official hosted service is Qwen Cloud; the hosted Qwen3.8-Flash version adds production features beyond this preview.

Frequently asked questions

Q: Can I use Qwen3.8-Flash-Next commercially?

A: The metadata lists the license as other, but the supplied README does not state the license terms. Do not assume commercial permission; obtain and review the applicable license before deployment.

Q: How much VRAM does Qwen3.8-Flash-Next require?

A: No VRAM requirement is provided. The model has 125B language-model parameters, 51B n-gram embedding parameters, and 4B MTP parameters, so the 6B activated count does not describe total storage or memory needs.

Q: What context length does it support?

A: It supports 262,144 tokens natively and is extensible to 1,000,000 tokens. The README does not provide quality or latency measurements for the extended range.

Q: Does it accept images?

A: The model type includes a vision encoder, and the metadata uses the image-text-to-text pipeline tag. The supplied material does not specify image resolution, image file formats, preprocessing rules, or a complete multimodal code example.

Q: Is thinking enabled by default?

A: Yes. The model emits <think>...</think> content before the final response. The README provides separate sampling recommendations for thinking and instruct or non-thinking modes, but the supplied excerpt does not include the code for disabling thinking.

Q: Which framework should I use for serving?

A: The weights and configuration use Hugging Face Transformers format and are listed as compatible with Transformers, vLLM, SGLang, and TokenSpeed. For production or high-throughput serving, the README recommends SGLang, KTransformers, or vLLM.

Q: How fast is inference?

A: The supplied information gives no tokens-per-second or latency measurements. Qwen claims that QSA cuts long-context latency by selecting micro-blocks rather than individual tokens, but the README provides no numeric comparison.

Q: Is this the same as the hosted Qwen3.8-Flash model?

A: No. Qwen3.8-Flash is the official hosted version based on this architecture preview and adds production features, including 1M-token context by default and official built-in tools.

Q: Can I fine-tune it?

A: The repository provides post-trained Transformers weights and configuration, but the supplied material does not document a fine-tuning recipe, supported parameter-efficient method, or hardware plan. Transformers compatibility provides an integration path, not a confirmed fine-tuning procedure.

Q: Where can I read the related research?

A: The Qwen3 technical report is available through the Qwen3 technical report. The model README also identifies a Qwen3.8-Flash-Next technical report, but no internal link for that report is provided here.

Click here to read the full guide to Qwen3.8-Flash-Next

A beginner's guide to the Grok-Imagine-Image model by Xai on Replicate

aimodels-fyi — Mon, 31 Aug 2026 02:59:08 +0000

This is a simplified guide to an AI model called Grok-Imagine-Image maintained by Xai. If you like these kinds of analysis, you should join AImodels.fyi or follow us on Twitter.

Overview

grok-imagine-image is xAI's state-of-the-art image generation and editing model, developed by xai. It accepts a text prompt and optional input image to generate or edit images across multiple aspect ratios. The model handles image-to-image transformations when a source image is provided, making it suitable for both pure generation and guided editing workflows. It supports standard image formats (JPG, JPEG, PNG, WebP) and outputs a single image URI. The critical advantage before using it is understanding that when an image is provided, the model prioritizes editing over generation—the aspect ratio parameter is ignored during image editing operations, and output dimensions depend on the input image dimensions rather than your specified aspect ratio preference.

Best use cases

Product photography enhancement and background replacement. When you need to modify existing product photos—changing backgrounds, adjusting lighting, or refreshing outdated product shots—grok-imagine-image works well because it understands both the visual content and text instructions simultaneously. You supply a product photo and a prompt like "modern minimalist white background" or "luxury lifestyle setting," and it preserves the product while transforming the environment.

Creative direction iteration for marketing assets. Design teams and marketing departments benefit from rapid style exploration. Provide a base composition or mood board image with text prompts describing desired changes ("add warm sunset lighting," "make it look more premium," "apply cyberpunk aesthetic") to generate variations without starting from scratch each time.

Illustration and concept art refinement. Artists use the image-to-image capability to evolve sketches or rough compositions into finished work. Supply a sketch or rough layout and describe the desired artistic direction ("detailed oil painting style," "photorealistic with dramatic shadows," "anime character with expressive eyes") to maintain compositional intent while upgrading visual quality.

Interior design visualization. Designers and real estate professionals leverage it to show clients how spaces could look. Photograph a room and use prompts to demonstrate renovations ("convert to modern minimalist," "add warm wood tones and plants," "contemporary luxury aesthetic") without commissioning separate renderings.

Content adaptation across platforms. Creators regenerate images for different aspect ratios and platforms. Generate a 1:1 square image for Instagram, then the same concept in 16:9 for YouTube thumbnails or 9:16 for Stories by calling the model with different aspect_ratio parameters during pure generation mode (without providing an input image).

Limitations

The model ignores aspect ratio specifications when editing an existing image; the output dimensions are determined by the input image dimensions, not your aspect_ratio parameter. This means you cannot resize edited images through the API—you must handle aspect ratio conversion externally or regenerate without an input image if you need specific dimensions.

Output quality depends heavily on prompt clarity and input image quality. Vague prompts produce unpredictable results, and low-resolution or heavily compressed input images limit the fidelity of edits. The model may struggle with highly specific technical requirements, precise text rendering within images, or maintaining exact object positioning during edits.

Supported input formats are limited to JPG, JPEG, PNG, and WebP. Other formats (TIFF, BMP, GIF) are not accepted. The model has no documented maximum file size, but extremely large images may timeout or consume excessive resources.

The model has no built-in safety filtering documentation, but as an xAI product, it likely has standard content policies. Outputs could potentially violate copyright or create problematic content if the prompt requests it.

Compared to grok-2-image, which has been deprecated as of February 24, 2026, this model is the active replacement and receives updates. If you are currently using the older model, migration is necessary.

How it compares

grok-2-image: This older xAI model was deprecated on February 24, 2026, and grok-imagine-image is its direct successor. Use the current model for all new projects; the predecessor is no longer maintained and will not receive improvements.

grok-imagine-video: This model generates videos from text prompts using xAI's video generation technology. Choose grok-imagine-image if you need still images with precise control over composition or styling; pick the video model if you need motion, temporal coherence, or animated content. The video model handles temporal continuity differently and is optimized for frame sequences rather than single-image quality.

grok-imagine-r2v: This model generates videos guided by reference images, combining a still image template with video generation. Use grok-imagine-image for pure image editing or generation; use the R2V model if you want to create a video where a reference image controls composition and style while adding motion and temporal development.

grok-imagine-video-extension: This specializes in extending existing videos with new frames based on prompts. If you have a video clip and want to extend it, use this model; if you only have a still image and want to generate a video, use grok-imagine-video instead. For image-only tasks, stick with grok-imagine-image.

grok-4: This is a reasoning and language model, not an image generation model. Use grok-imagine-image exclusively for visual content creation and editing; use Grok 4 for text-based reasoning, analysis, or conversation that informs creative decisions.

Technical specifications

grok-imagine-image is a generative image model optimized for both text-to-image generation and image-to-image editing. The model runs on Replicate's infrastructure and was last updated on February 12, 2026 (Cog version 0.16.11). It generates single output images in URI format.

The model supports three aspect ratios for pure generation mode: 1:1 (square, the default), and two additional ratios specified in the schema enum. When editing an image, aspect ratio is ignored and the output maintains the input image's proportions. Input images must be in JPG, JPEG, PNG, or WebP format. The text prompt length and complexity are not documented, but standard language model practices suggest reasonable limits apply.

Key parameters:

Prompt input: unrestricted length string for generation or editing instructions
Image input: optional URI-based input supporting JPG, JPEG, PNG, WebP formats
Aspect ratio: default 1:1, ignored during image editing, applied only during pure generation

No parameter count, training dataset size, inference time, or hardware requirements are documented in the available materials. The model outputs a single image URI regardless of input complexity.

Model inputs and outputs

Inputs

prompt (string, required): Text description guiding image generation or editing. Used to define artistic style, content, composition, or modifications when an image is provided.
image (string/URI, optional): Input image for editing in JPG, JPEG, PNG, or WebP format. When omitted, the model performs pure text-to-image generation. When provided, the model edits this image based on the prompt.
aspect_ratio (enum, default: "1:1"): Output dimensions for generated images. Ignored when editing an image. Options: "1:1" (square) and two additional ratios defined in the schema.

Outputs

image (string/URI): A single generated or edited image returned as a URI string pointing to the output image file.

Getting started

import replicate

client = replicate.Client(api_token="your_replicate_api_token")

# Pure text-to-image generation
output = client.run(
    "xai/grok-imagine-image",
    input={
        "prompt": "a serene mountain landscape at sunrise with golden light, photorealistic, highly detailed",
        "aspect_ratio": "1:1"
    }
)
print("Generated image:", output)

# Image editing with a reference image
output = client.run(
    "xai/grok-imagine-image",
    input={
        "prompt": "transform this room into a modern minimalist space with warm lighting and natural wood",
        "image": "https://example.com/room.jpg",
        "aspect_ratio": "1:1"  # This will be ignored since an image is provided
    }
)
print("Edited image:", output)

# Generating multiple aspect ratios
for ratio in ["1:1"]:  # Add other supported ratios as needed
    output = client.run(
        "xai/grok-imagine-image",
        input={
            "prompt": "a cyberpunk cityscape with neon signs and rain",
            "aspect_ratio": ratio
        }
    )
    print(f"Generated {ratio} image:", output)

Frequently asked questions

Q: Can I edit an image and specify its output dimensions?

A: No. When you provide an input image, the aspect_ratio parameter is ignored and the output maintains the input image's original dimensions. To resize or change aspect ratios, regenerate without an input image using pure text-to-image generation.

Q: What image formats does the model accept?

A: The model accepts JPG, JPEG, PNG, and WebP formats. Other formats like TIFF, BMP, or GIF are not supported.

Q: Is grok-imagine-image still actively maintained?

A: Yes. It replaced the deprecated grok-2-image model on February 24, 2026, and receives ongoing updates from xAI. The latest version was deployed February 12, 2026.

Q: How does this model differ from grok-2-image?

A: grok-imagine-image is the official successor to grok-2-image, which was deprecated by xAI on February 24, 2026. All new projects should use the current model for bug fixes, improvements, and ongoing support.

Q: Can I use this model to generate videos?

A: No. For video generation, use grok-imagine-video instead. This model generates single still images only. For video extension or frame-guided video generation, see grok-imagine-r2v and grok-imagine-video-extension.

Q: What happens if I provide both a prompt and an image?

A: The model enters image editing mode. It uses your prompt to guide modifications to the provided image while preserving the original image's structure, subject matter, and dimensions. The edited output reflects changes described in the prompt applied to the input image.

Q: Are there limits on prompt length or complexity?

A: No documented limits are provided, but standard language model constraints apply. Very long or extremely complex prompts may be truncated or produce unpredictable results; keep prompts clear and concise for consistent output.

Q: What aspect ratios does the model support?

A: The schema specifies a default of "1:1" (square). Additional supported ratios exist but are not listed in the available documentation; test your desired aspect ratio or refer to xAI's documentation for the full list.

Click here to read the full guide to Grok-Imagine-Image

OpenART Red-Teams Stateful Agents Across 10,000 Evolving Environment Scenarios

aimodels-fyi — Mon, 24 Aug 2026 18:34:19 +0000

This is a Plain English Papers summary of a research paper called OpenART Red-Teams Stateful Agents Across 10,000 Evolving Environment Scenarios. If you like these kinds of analyses, you can find more AI and machine-learning research on AIModels.fyi or follow us on Twitter.

OpenART turns persistent state into the red-team target

OpenART evaluates agent safety across more than 10,000 validated stateful scenarios spanning 50 domains and requiring a median of 97 tool calls. Its central claim is that safety failures can emerge from trajectories in which workspace data, permissions, memory, and plans are repeatedly modified, rather than from isolated prompts alone.

The arena keeps each benign task objective and hidden safety contract fixed while changing only the target-visible environment state. This design targets delayed failures that static benchmarks can miss: an early authorized mutation may influence later decisions, expose protected resources, or produce unsafe output many steps after the original change. OpenART extends the broader idea of agent safety evaluation by making persistent environment state the object that evolves during testing.

OpenART reports a pooled strict Attack Success Rate of 85.0% across 75 agent-model configurations. Strict success requires both the deterministic evaluator and a GLM-5.2 judge to identify the attack condition, so disagreements count as failures rather than being treated as partial evidence....

Continue reading the full paper summary on AIModels.fyi →

RA-Bench Reveals Why Crisis-Video Deepfake Detectors Fail Across Generators and Social Media

aimodels-fyi — Mon, 24 Aug 2026 18:33:44 +0000

This is a Plain English Papers summary of a research paper called RA-Bench Reveals Why Crisis-Video Deepfake Detectors Fail Across Generators and Social Media. If you like these kinds of analyses, you can find more AI and machine-learning research on AIModels.fyi or follow us on Twitter.

The crisis detection problem we've been getting wrong

Video synthesis has reached an inflection point. Recent generators can fabricate realistic depictions of wars, natural disasters, infrastructure failures, and public emergencies so convincingly that they fool both people and current detection systems. The threat isn't hypothetical anymore. A fabricated video of a nuclear plant explosion, a hospital collapse during an earthquake, or a terrorist attack could trigger panic, military response, or severe economic disruption within hours.

Yet here's the troubling part: we don't actually know if our best detection tools can handle these high-stakes scenarios in the wild. Researchers have built impressive deepfake detectors, trained them on standard benchmarks, and measured their performance. But those benchmarks test detectors against generic synthetic videos, not against the specific threat that actually matters: AI-generated crisis footage designed to fool people about real things that happened.

It's like training a border guard to spot counterfeit passports in a lab with perfect lighting and a magnifying glass, then sending them to a busy airport where they have to make decisions in three seconds. The guard's failure has nothing to do with their skill. The problem is that the testing environment was completely divorced from the real scenario....

Continue reading the full paper summary on AIModels.fyi →

Macaron-V1: Continual Learning with Self-Improvement and Mixture-of-LoRA Adapters

aimodels-fyi — Mon, 24 Aug 2026 18:28:00 +0000

This is a Plain English Papers summary of a research paper called Macaron-V1: Continual Learning with Self-Improvement and Mixture-of-LoRA Adapters. If you like these kinds of analyses, you can find more research on AIModels.fyi or follow us on Twitter.

The problem with frozen models

Most AI systems today follow a familiar pattern: train, evaluate, deploy, and then stop. The model is locked at that moment, treated as a finished product rather than a living system. But the real world immediately begins to diverge from training data. Users interact with the system in ways the training process never anticipated. New domains emerge. Preferences shift. The model that seemed smart on test day becomes gradually less relevant over time.

This frozen-in-place approach isn't accidental. It reflects how machine learning has been practiced for decades. Retraining is expensive. Deploying new versions carries risk. The infrastructure to continuously improve systems in production barely exists. So instead, teams ship a model and move on, accepting that it will decay slowly but inevitably.

Macaron-V1 asks a different question: what if AI systems could continuously improve themselves through real-world experience, learning from the billions of interactions that happen after deployment? Not in theory, but actually, in production, with users.

The answer isn't magic. It requires two architectural shifts. First, treat deployment as the beginning of a learning process, not the end of one. Build versioning, evaluation contracts, and feedback loops directly into the system. Second, stop assuming you need to retrain your entire model. Instead, freeze a stable base and compose lightweight specialist adapters around it, allowing the system to grow in capability without losing its foundation.

Rethinking deployment as a continuous learning opportunity

The insight here is architectural. Instead of viewing the deployed model as the final form, Macaron-V1 treats it as the first link in an infinite chain. Each version learns from real-world feedback, gets evaluated against an external quality contract, and either gets promoted or discarded. The next version incorporates the lessons. Then the cycle repeats.

This requires inverting how teams typically think about production. Production isn't where you stop learning; it's where you have the most valuable learning signal. Your users are running the biggest, most realistic experiment you could design. Each interaction reveals something about what actually works. The challenge is converting that chaotic signal into systematic improvement.

The machinery for this is Model-Harness Co-design. The "harness" here isn't just inference code. It's the complete environment surrounding the model: how users interact with it, what tools it can call, how outputs are evaluated, where feedback comes from, what success looks like. Traditionally, teams treat the model as the entire story and the harness as plumbing. Macaron-V1 reverses this. The model and harness are versioned together, tested together, deployed together. They evolve as a unit because they're codependent.

Why does this matter? Because much of the real intelligence lives in the harness, not just the model weights. A system that retrieves the wrong context, formats outputs poorly, or collects feedback carelessly will be useless no matter how smart the underlying model is. By co-designing model and harness, Macaron-V1 ensures improvements propagate all the way to user-facing behavior.

The recursive improvement cycle

The actual mechanics are deceptively simple. Each cycle follows the same pattern: collect data from production, evaluate it against a contract, select the best new configuration, deploy it. Repeat.

The contract is the key mechanism. It's a versioned, external specification of what "better" means. Not a leaderboard score or a vague notion of quality, but a formal definition: users should be able to accomplish X with the system, with Y level of reliability, in Z time. This prevents drift. It forces clarity about what you're actually optimizing for. And it prevents the system from learning perverse behaviors that technically fit the data but violate your underlying intentions.

Versioning throughout ensures you can rollback when something breaks, compare different approaches, and maintain a clear lineage of improvement. Every model is tagged. Every harness is tagged. Every version pair is evaluated before deployment. You don't ship broken things. That discipline is boring but essential.

The evaluation gate is equally important. Not every change makes the system better, even if it fits the training data perfectly. You need external validation that the new version actually satisfies the contract before you deploy it. This costs compute, but the cost is paid once. The alternative is deploying regressions to millions of users, which is worse.

Over time, this process compounds. Early cycles might yield big improvements. Later cycles might be smaller. But the point is that the system never stops improving because the feedback loop never stops running. This is fundamentally different from traditional machine learning, where you improve once and then coast.

Mixture-of-LoRA: specialization without retraining

Here's where the architecture becomes elegant. You don't want to retrain your entire model for each new capability. Your base model is massive (Macaron-V1-Venti uses a 744B GLM-5.2 base) and represents years of training. You also don't want to lose what it already knows. Instead, you need a way to add specialization without touching the foundation.

That's what Mixture-of-LoRA does. LoRA, or Low-Rank Adaptation, is a technique that trains a small set of additional parameters while freezing the base model. Imagine your base model is like a brilliant consultant whose worldview is fixed and valuable. You don't want to retrain their brain. Instead, you hire domain experts, architects, doctors, lawyers, who work alongside them. Each brings specialized knowledge. The consultant's foundation never changes.

Macaron-V1-Venti composes four specialist LoRAs: one for chat, one for coding, one for agent behavior, one for UI generation. Each LoRA is a small matrix of learned weights that modulates how the base model behaves in that domain. When a user sends a message, the system picks the most relevant LoRA (or blends multiple) for that turn. Only that adapter is active. The base model stays frozen.

This solves two critical problems simultaneously. First, it makes the system infinitely extensible. New domains don't require retraining the whole system. You train a new LoRA and plug it in. Retire old ones. Improve existing ones. The base model is stable and never needs to change. Second, it's dramatically more efficient. You only serve the adapters you need. The frozen base model is a shared resource, amortized across all tasks.

But there's something deeper here. This architecture is built for continual learning. You can improve individual LoRAs without affecting others. You can add new specializations as new use cases emerge. You can retire adapters that aren't working. This is vastly different from systems where everything is entangled in one monolithic model. Compartmentalization creates resilience. It prevents catastrophic forgetting. It enables true experimentation because failures are isolated.

The design also separates concerns elegantly. The base model is responsible for core reasoning, world knowledge, and general capability. LoRAs are responsible for specialization. You can improve both independently. A new base model release doesn't break your LoRAs. A broken LoRA doesn't corrupt your base. This is how systems scale and improve over time without accumulating technical debt.

The infrastructure that makes continual learning real

Architecture is elegant on a whiteboard. But making it work reliably at scale requires unglamorous infrastructure. Macaron-V1 builds this infrastructure as a first-class design priority, which is why it's credible.

MinT is the post-training platform that converts messy production data into training signal. Not all feedback from users is useful. Some is noise. Some is biased. MinT filters, validates, and prepares data before it's used to train new versions. This is where garbage-in-garbage-out prevention happens. A system built on bad data will be bad, no matter how clever the architecture.

LongStraw extends reinforcement learning to handle long-horizon reasoning. As agents interact with the system over extended episodes, context accumulates and decisions compound. Simple token prediction isn't enough. You need the system to reason about long-term consequences. LongStraw handles this without exploding compute costs, making it practical to learn from rich, extended interactions.

The versioned HCP contract (presumably Human-Compatible Performance) is the formal specification mentioned earlier. It's not a score or a metric. It's a contract: this version must satisfy these properties. Without this, you don't know what you're optimizing for. Versions drift. Learning becomes directionless.

MindForge is the agentic RL framework that handles learning from action sequences. Agents don't just predict tokens; they take actions in the world and observe consequences. Those action trajectories are rich learning signals. MindForge learns policies from them, allowing the system to improve how it decides what to do, not just what to say.

There are also stability techniques for sparse Mixture-of-Experts models, which can be brittle at scale. Sparse models can suffer from mode collapse and dead neurons. The paper introduces methods to prevent this, making large sparse models reliable for production deployment. This is infrastructure work: invisible unless it breaks, but essential.

Stateful interaction and generative UI

Beyond just improving model weights, Macaron-V1 changes what the system can actually do. GenUI (component-native UI generation) means the system doesn't just produce text descriptions of interfaces. It generates actual interactive components. A system that can only speak is limited. A system that can generate UIs, take actions, and maintain state is fundamentally different.

Why does this matter for continual learning? Because interaction is richer than text. When users interact with a generated UI, click buttons, modify forms, and abandon unsatisfying options, their behavior reveals whether the generation was useful. A rejected UI teaches you something. A completed workflow teaches you something else. This is feedback signal that pure language prediction never captures.

The stateful substrate means conversations persist and inform future interactions. The system remembers context across turns. This creates rich temporal dependencies that a stateless system can't learn from. Users interact differently when the system understands context. The system learns different patterns. Both improve together.

This is what "experiential intelligence" means: the system learns from the experience of actually doing things in the world, not just from predicting what should happen. It's fundamentally more grounded than language-only systems.

Two implementations across scales

The architecture isn't tied to one scale. Macaron-V1-Venti uses a 744B GLM-5.2 base, designed for cloud deployment with maximum capability. Macaron-V1-Tall uses a 50B Qwen3.6 base, deployable locally or on smaller infrastructure. Same architecture. Different tradeoffs.

This matters because it proves the design isn't a scaling hack. It's a principled architecture that works when you're deploying on frontier models and when you're optimizing for local inference. The Mixture-of-LoRA design transfers across orders of magnitude. The co-design principles apply at both scales. This kind of invariance across scales is rare and valuable.

What's proven and what's still speculative

The paper doesn't overclaim. Initial results validate that Macaron-V1 works as a system. It's competitive on Personal Intelligence benchmarks, GenUI capability, and general capability tests. The architecture functions. The infrastructure holds up.

But the foundational questions remain unanswered. Does continual learning actually compound over time, or do gains plateau after a few cycles? Does collective intelligence emerge when millions of users interact with different LoRAs? Does the system learn from all of them simultaneously, or do specializations remain isolated?

These questions matter because they determine whether continual learning is a minor optimization or a fundamental shift in how AI systems improve. If improvement compounds indefinitely, then systems get progressively smarter just by operating. If it plateaus quickly, the benefit is limited. If collective learning emerges, then diversity in use cases becomes an asset. If specializations remain isolated, then the system improves but doesn't develop true breadth.

The paper explicitly leaves these as open questions. That honesty is valuable. It tells you what the system can do today and maps the territory of uncertainty that remains.

Design principles that generalize

The specific system is Macaron-V1, but the underlying principles extend far beyond it. First, separate concerns: the base model handles core reasoning and knowledge, LoRAs handle specialization, the harness handles interaction and evaluation. Each can improve independently. Second, contracts over magic. Define explicitly what success means, rather than hoping gradient descent finds it. Third, infrastructure as design. The plumbing isn't separate from intelligence; it's integral to it. Fourth, extensibility as architecture. Build systems that are designed to change, not just trained to perform once.

Finally, feedback loops in production are the fuel for improvement. Not validation sets or held-out test data, but actual user behavior. Production is where the signal lives.

Work on continual motion language agents using LoRA variants has explored similar space, showing that adapter-based approaches transfer well across related domains. Related work on dynamic mixture models for self-evolving agents demonstrates how mixture approaches enable adaptation. And research on multi-agent cooperative learning shows that collective improvement is possible when systems coordinate effectively.

The frontier ahead

Macaron-V1 demonstrates one credible path toward systems that genuinely improve from production experience. The architecture works. The infrastructure holds up. The initial results are promising. But the hard questions about compounding improvement and emergent intelligence remain unsolved.

The vision is systems that never stop improving because they never stop learning from users. Not through occasional retraining cycles, but through continuous, automated feedback loops. Every interaction becomes training data. Every deployment becomes an experiment. Every version is slightly smarter than the last.

That's not science fiction. Macaron-V1 shows it's buildable today. But whether the vision scales to truly transformative improvement remains the central open question. If the answer is yes, continual learning becomes the default paradigm. If it's no, we're back to periodic retraining and frozen models.

The system is live. The learning loop is running. The uncertainty is productive. The next chapter will be written in production.

Read the full paper summary on AIModels.fyi

BDH-CQ Uses Recurrent Latent Reasoning to Cut ARC-AGI Inference Costs

aimodels-fyi — Mon, 24 Aug 2026 18:27:25 +0000

This is a Plain English Papers summary of a research paper called BDH-CQ Uses Recurrent Latent Reasoning to Cut ARC-AGI Inference Costs. If you like these kinds of analyses, you can find more research on AIModels.fyi or follow us on Twitter.

The cost-accuracy trap in visual reasoning

Large language models are fundamentally mismatched for visual reasoning tasks. They're forced to describe every thought out loud, generating token after token to explain their logic. This verbosity taxes compute budgets, yet paradoxically doesn't improve performance. Ask a language model to solve an ARC-AGI puzzle (a visual reasoning benchmark designed to test abstract thinking), and it either struggles despite the verbosity or succeeds expensively. The root problem runs deeper than just inference cost: the model learns from demonstrations by parsing them as language tokens, which is an indirect and inefficient way to absorb a visual pattern.

The efficiency frontier has been unforgiving. If you want cheap inference, you sacrifice accuracy. If you want accuracy, you sacrifice cost. Every model on the leaderboard until recently clustered into one of two camps, and no one had found a path that broke the tradeoff.

BDH-CQ challenges this assumption by proposing something radical: reasoning doesn't need to be visible to work. The model absorbs demonstrations silently into its internal memory state, then solves problems through private iteration in hidden layers, without generating a single token of intermediate reasoning. A 150-parameter variant achieves 29.5% pass@2 on the ARC-AGI-1 benchmark at a computed cost of just $0.0007 per task, puncturing through the previous Pareto frontier and establishing a new state of the art in cost efficiency.

Learning through hidden states

The core insight is deceptively simple: a model's reasoning process doesn't need to match human communication. When you learn a new skill from examples, you don't narrate every observation. You absorb patterns directly into your intuition. BDH-CQ applies this to neural networks by treating the model's recurrent hidden state as a working memory that continuously absorbs information from demonstrations.

Here's how it actually works. The model receives a sequence of examples from the demonstration set. Each example updates its internal state. By the time the model reaches the query input (the problem to solve), its memory has been shaped by everything it learned from those examples. It then leverages this primed state to solve the new problem through iterative computation in latent space.

This is fundamentally different from how in-context learning works in language models. In a transformer, examples appear as tokens in the prompt and the model has to parse them using the same machinery it uses for language understanding. Here, examples bypass that linguistic bottleneck entirely. They directly steer the model's latent representation. The model doesn't need to "read" what it should learn; it can absorb patterns directly.

This reframing solves two problems simultaneously. First, it's cheaper because the model never generates reasoning tokens. Second, it might actually learn better from few examples because the information flows directly into working memory rather than being filtered through language parsing. The approach doesn't fight the architecture; it aligns with what recurrent networks are naturally built to do.

The recurrent mechanism

Understanding the architecture requires stepping back to what recurrence actually provides. A recurrent neural network maintains a hidden state that evolves over time. At each step, the state updates based on current input while carrying information from all previous steps. This is the opposite of a transformer, which processes all tokens in parallel.

In BDH-CQ, the hidden state acts as working memory. When the model processes the first demonstration, its state shifts. When it processes the second demonstration, the state shifts again, carrying forward information from the first. By the final demonstration, the state has absorbed the entire pattern. Then the model receives the query input and continues to refine the same state through iterative refinement. Only at the very end does it convert this refined latent state into an actual output.

The iteration step is crucial. Unlike language models that generate one token at a time and stop, BDH-CQ can iterate multiple times over the query input, refining its hidden state with each pass. This gives the model time to "think" about how to apply the learned pattern, without paying the cost of generating any tokens. The number of iterations becomes a tunable parameter: more iterations mean more reasoning time, but also higher compute cost. Figure 7 plots this tradeoff explicitly.

Pass@2 and compute cost scale with reasoning effort, revealing the cost of added thinking time

Recurrence is specifically suited to this task because it's designed to handle variable-length sequences and accumulate information over time. A model needs some mechanism to "show" what to do through examples, then have it think about applying that pattern to a new case. Recurrence provides that mechanism naturally.

What the model actually learns

Raw benchmark numbers hide important questions. Does BDH-CQ solve problems because it genuinely learns the transformation, or because it's picking up surface patterns? The authors addressed this through a more surgical approach: controlled experiments where they could vary specific aspects and measure exactly what the model captured.

Rather than just testing on the public benchmark, they constructed four controlled generalization families derived from actual ARC-AGI tasks. The extend family asks the model to complete a seed pattern to the boundary. The copy family replicates a motif to multiple anchor points. The order family sorts items by a property like height. The nesting family manages spatial hierarchies. For each family, they showed the model examples at increasing difficulty and measured exactly when it stopped generalizing.

Extend, copy, order, and nesting represent core visual reasoning concepts that can be tested systematically

The results reveal an uneven landscape. Figure 5 plots generalization curves for each concept, and the picture is mixed.

Controlled generalization curves show which concepts the model learns robustly and where it hits walls

Some concepts, like copying, are learned robustly. The model keeps generalizing even as the examples become harder. Other concepts, like ordering, hit a wall at intermediate difficulty. More revealing is the gap between "semantic accuracy" and "strict accuracy." If a model achieves 50% semantic accuracy but only 20% strict accuracy, it's roughly understanding the concept but failing on execution details. A large gap indicates the model gets the shape right but misses details. A small gap indicates genuine understanding.

An interesting follow-up tested compositionality. What happens if you ask the model to combine transformations? For example, move and rotate a motif simultaneously. Figure 6 introduces this scenario with representative examples.

The model can learn individual transformations like relocation and rotation, but combining them remains challenging

The model can combine some operations but not others, suggesting it learns distinct transformation "skills" that sometimes compose and sometimes don't. This nuance is valuable. It tells researchers where to look for limitations and which combinations might be fixable with better training rather than fundamental architectural constraints.

The diagnostic profile

Different types of visual reasoning pose different challenges. Figure 3 breaks down the model's performance by concept area, revealing which it handles well and which remain obstacles.

Pass@2 varies significantly by concept area, from near-solved to stubbornly difficult

Symmetric operations and geometric transformations are handled relatively well, perhaps because recurrent networks naturally encode such patterns. More abstract reasoning, particularly tasks requiring counting or symbolic manipulation, remains difficult. This isn't a flaw in the paper. It's valuable scientific information. By isolating which concepts remain hard, the authors guide future research toward genuine bottlenecks rather than problems that are already close to solved.

The gap between semantic and strict accuracy is diagnostic. When it's large, the model understands the task concept but fails on details. When it's small, the model either gets it right or fundamentally misunderstands. This distinction helps explain what's actually happening inside the hidden states.

There's also a cost dimension to success. Figure 7 showed how performance scales with reasoning effort. The model doesn't achieve 29.5% with minimal compute. It requires careful tuning of the iteration budget. Too few iterations and the model doesn't have time to solve hard problems. Too many and resources are wasted on problems that settle quickly. The paper quantifies this tradeoff, which is precisely the practical knowledge researchers need when deciding whether to use this approach.

Breaking the frontier

This is where the theoretical efficiency meets real-world numbers. Look at Figure 2, which plots every public result on ARC-AGI-1 as of August 2026.

BDH-CQ's 29.5% pass@2 sits strictly to the left of previous methods, breaking the cost-accuracy Pareto frontier

The previous frontier shows an unmistakable tradeoff: cheap methods were inaccurate, accurate methods were expensive. The relationship was nearly linear. More money bought more accuracy, with no third path visible. BDH-CQ's operating point sits strictly below and to the left of everything else. It achieves better cost for equivalent accuracy or better accuracy for equivalent cost. It's not a marginal improvement in one direction. It's a qualitatively different point on the frontier.

This result validates the entire conceptual framework. Recurrent latent reasoning actually works. Learning from demonstrations through hidden state updates actually transfers to unseen problems. The theoretical elegance has real empirical backing.

The broader implication extends beyond this specific benchmark. The core finding is that reasoning doesn't require verbalization, and few-shot learning doesn't require parsing examples as language tokens. These principles apply to any domain where you need to learn from demonstrations and solve problems under tight efficiency constraints. A recommendation system that learns from user interaction sequences. A robotics controller that internalizes movement patterns from video. A medical diagnostic system that absorbs patterns from case studies. The architecture's generality is the lasting contribution.

What remains unsolved

The model still fails on roughly 70% of tasks even with optimized reasoning effort. Some concepts remain stubbornly difficult. This isn't weakness in the framing. It's necessary honesty. The authors have achieved a breakthrough in cost efficiency, not solved visual reasoning entirely. Readers should understand both what BDH-CQ accomplishes and what remains genuinely hard.

The approach shares conceptual ancestry with other work on reasoning and latent representations. Related research on hierarchical latent reasoning in recommendation systems, recursive vision-language models for symbolic reasoning, and hierarchical recurrent mechanisms for adaptive reasoning explores similar intuitions about how to combine learning from demonstrations with iterative latent computation. Whether BDH-CQ's specific advantages come from the recurrent architecture, the latent reasoning approach, the specific problem structure of ARC-like tasks, or some combination remains an open question.

The fundamental contribution is demonstrating that efficient reasoning is achievable without expensive verbalization. The model shows genuine learning from few examples, exhibits interpretable failure modes, and pushes the cost-accuracy frontier in a direction that hadn't been reached before. These aren't minor increments. They're shifts in how the problem can be approached.

Read the full paper summary on AIModels.fyi

How Cross-Model Compatibility Lets Attackers Extract Proprietary LLM Reasoning Traces

aimodels-fyi — Mon, 24 Aug 2026 18:26:48 +0000

This is a Plain English Papers summary of a research paper called How Cross-Model Compatibility Lets Attackers Extract Proprietary LLM Reasoning Traces. If you like these kinds of analyses, you can find more research on AIModels.fyi or follow us on Twitter.

The illusion of safety

Major AI companies now show users their models' step-by-step reasoning as a feature. OpenAI offers it through o1, Anthropic through extended thinking, Google through its reasoning-focused variants. But this reasoning is a double-edged sword. It's intellectually valuable to share, showing users why a model reached a conclusion. But it's also intellectually valuable to steal. Competitors want to understand how frontier models think. Researchers want to study their reasoning patterns. Attackers want to extract proprietary algorithms.

So the companies made a choice: hide the reasoning from users by encrypting it. The idea sounds straightforward enough. Return the reasoning to the user's device in an encrypted, unreadable form. The user can't see it, competitors can't see it, but they can pass it back to the server in future requests if they need continuity with previous reasoning. The server alone holds the decryption keys. Problem solved.

Except it wasn't. Researchers discovered that this encryption doesn't actually hide reasoning. It just makes it look hidden. The encrypted blocks are designed to work everywhere within a company's ecosystem, across different sessions and different models. That universal compatibility is a feature for convenience. But it's also an architectural vulnerability that anyone can exploit.

The architectural gamble

To understand where this went wrong, you need to see how the system actually works.

When a user sends a request to a frontier model like GPT-4, the model internally generates a reasoning trace, the raw thought process behind its answer. Instead of returning this reasoning in plaintext, the company encrypts it on the server before sending it to the client. The encrypted block, which we can call a reasoning token, travels to the user's device alongside the final answer. The user's device can't open it. It's encrypted. But the device can store it, and when the user makes a follow-up request that needs the previous reasoning, they send the encrypted token back to the server. The server decrypts it, uses the reasoning to inform the next response, and hands back another encrypted token.

The security assumption embedded in this design seems reasonable: only the company's servers hold the decryption keys. Tokens are cryptographic black boxes from the user's perspective.

But there's a hidden assumption underneath, one that silently shaped the architecture. The assumption is that isolation between users matters less than convenience within the company's ecosystem. So instead of using different encryption keys for different users or different security tiers, the system uses one logical encryption scheme across all models, all sessions, and all users within a provider. Your encrypted reasoning tokens are cryptographically compatible with mine. A token from ChatGPT is readable by GPT-3.5. A token from Claude Instant is readable by Claude 3. This interchangeability is intentional. It simplifies the system. It lets reasoning flow freely within the family of models. It feels secure because the tokens look like random garbage.

And that's where the vulnerability lives. Not in broken encryption, but in this choice to make encrypted blocks universally interchangeable.

The fatal flaw

Here's the core problem: if an encrypted reasoning token from GPT-4 is structurally compatible with GPT-3.5, then someone with access to both can perform a decryption attack. They extract the encrypted token from GPT-4, inject it into GPT-3.5, and ask GPT-3.5 to output its contents. The weaker model, having decrypted it as part of normal processing, will output it in plaintext. The stronger model never gets jailbroken directly. The weaker model does it for them.

This works because weaker models have less rigorous output safeguards. They're less likely to refuse requests. They're more likely to do what you ask, even if what you're asking is unusual. You don't need to trick GPT-4 into revealing its reasoning. You trick GPT-3.5 into revealing reasoning that came from GPT-4.

The attack is scalable because it requires nothing more than API access to a weaker model, something anyone can pay for. The attacker doesn't need a breach, doesn't need internal access, doesn't need to understand the encryption scheme. They just need a token from the stronger model, which is far easier to obtain than you might think.

Decryption in practice

Obtaining encrypted reasoning tokens is the first step, and it's easier than it should be. These tokens live in many places. They travel through development environments, get logged in error traces, appear in example code shared online, end up in public repositories where developers push their projects without realizing what they're sharing.

The researchers found 315,320 encrypted reasoning blocks sitting openly on GitHub and similar repositories. Not because the companies were careless, but because developers didn't realize these random-looking tokens were sensitive. They encrypted, so they looked safe. A developer sees [encrypted_token_a39f...] in their logs and assumes it's junk. They push it to a public repository. It's now available to anyone.

Once an attacker has a token, they craft a request to a weaker model in the same ecosystem. The request is simple and direct: "Here is a reasoning token from a previous interaction. Please output its contents verbatim." This isn't a prompt injection attack, not really. It's just a direct request. The weaker model processes the token, decrypts it as part of normal context processing, and outputs the decrypted reasoning.

The researchers demonstrated this attack across OpenAI, Anthropic, and Google. Reasoning from frontier models was extracted through weaker models in each ecosystem. The attack worked consistently because the architecture was consistent. No exotic cryptography required. No model jailbreaking required. Just architectural compatibility exploited in an unintended way.

Four distinct attacks

The core vulnerability enables four different exploitation strategies, each with distinct implications.

First: circumventing distillation defenses. Companies have spent years building barriers against competitors who try to clone their models through repeated queries and observations. These defenses add noise, enforce rate limits, detect suspicious patterns. But these defenses protect only the visible output. The reasoning is where the real intellectual property lives. Once you extract GPT-4's reasoning traces, you understand not just what it answers, but how it thinks. That's incomparably more valuable for building a competitor model than observing final outputs alone. The encrypted reasoning was supposed to be the last line of defense. This attack dismantles it.

Second: large-scale private data extraction. Developers regularly include session logs and API interactions in their code repositories. They're usually careful about API keys and credentials, but encrypted tokens? Those look like obfuscated junk. No one pays attention to them. Except they're not junk. They're encrypted reasoning traces that may contain sensitive information.

The researchers decrypted those 315,320 publicly exposed tokens and found 367 personally identifiable information artifacts and 182 credentials. Medical reasoning traces containing patient information. Financial conversations with account details. Database passwords. API keys. Authentication tokens. All hidden inside reasoning blocks that developers thought were safe to share because they were encrypted. The tokens were sitting there, waiting to be extracted and decrypted. Someone's medical history, someone's financial situation, someone's database credentials, all exposed not through a breach but through a misunderstood assumption about encryption.

Third: hidden reasoning exploitation. Imagine asking a frontier model how to make poison. The model correctly refuses: "I can't help with that." But during reasoning, to properly understand and reject the request, it may have actually generated partial insights or key details about the poison-making process. The final output hides this dangerous information. But the encrypted reasoning token preserves it. By extracting and decrypting the token, an attacker gets the information the model refused to give, because the reasoning contains exactly what the output rejected.

Fourth: invisible prompt injection. An attacker embeds a malicious prompt entirely within an encrypted reasoning token, then releases this token publicly as part of some innocuous data dump. Unknowing developers pick it up and use it in their applications. When their model processes the token, it decrypts the hidden prompt and executes it, without anyone seeing the injection. The attack is invisible because it's encrypted. It spreads easily because encrypted tokens look like innocuous data. And it's powerful because it can be injected at scale into any agentic system that reuses reasoning tokens across multiple models or sessions.

What was actually exposed

It's easy to dismiss vulnerabilities in theory. Real data exposures are harder to ignore.

The researchers didn't just identify the vulnerability. They exercised it systematically. From the 315,320 encrypted blocks harvested from public repositories, they extracted real information:

367 PII artifacts including names, email addresses, and partial social security numbers. 182 credentials including database passwords, API keys, and authentication tokens. Reasoning traces from proprietary medical and financial models that revealed internal algorithms. Information that models had explicitly hidden from their final outputs, surfaced through the reasoning.

The credentials are particularly alarming because they're living in encrypted blocks inside public repositories. A developer forks a project, receives a seemingly random encrypted token as part of the code, and suddenly has a database password they shouldn't have. Attackers don't even need sophisticated tools. They just need to run existing code to extract these tokens, then run the decryption attack to convert them to plaintext.

This happened because developers operated under a reasonable assumption: encrypted data is safe to share. The vulnerability transformed that assumption from true to false, but the encrypted appearance meant developers didn't notice the shift.

Why the design choice made sense

At this point, a reasonable question emerges: why didn't the companies catch this? This wasn't a product of incompetence or negligence. It came from real engineering tradeoffs that were perfectly defensible at the time.

On one side, there's a genuine scaling problem. Reasoning traces are massive. A single reasoning trace from an advanced model can be thousands of tokens. Every conversation with every user multiplies this data. Storing all reasoning server-side doesn't scale. Every user interaction adds permanent storage requirements. Deletion and compliance become operational nightmares. If a user asks you to delete their data, you have to find and purge their reasoning from every trace they ever participated in. Terabytes of reasoning, potentially scattered across backup systems, data lakes, and compliance archives.

On the other side, there's an elegant solution: move the storage burden to the client. The company encrypts the reasoning on the server, sends it to the client device, and the client is responsible for storing it. If they delete it, it's gone. If they use it again, they send it back. From a scaling perspective, this is brilliant. The company outsources storage to millions of client devices.

But this solution embedded a silent assumption: encryption alone is sufficient to protect reasoning across any boundary you want to cross. The company designed the system thinking about encryption strength, not about interactions between models. Each model was secure in isolation. Encryption was strong. But no one anticipated that a weaker model in the same ecosystem could decrypt tokens from a stronger model. Why would they? The ecosystem was designed with a specific security model: tokens are encrypted, therefore safe to pass around, therefore interchangeable. The interchangeability was a feature, not a threat vector.

This is a pattern that appears throughout systems security. Vulnerabilities often emerge not from broken components, but from unexamined assumptions about how components interact. Each piece of the system was secure in isolation. Encryption was sound. Models were safeguarded. But the interaction between strong models and weak models within a unified ecosystem created an attack surface that no single component was designed to defend.

Securing the architecture

Fixing this vulnerability requires changes at both the cryptographic and architectural levels.

The core problem is interchangeability. A straightforward cryptographic fix is to make encrypted tokens context-dependent. Instead of encrypting just the reasoning, encrypt the reasoning along with metadata about who requested it, which model should be able to decrypt it, what session it belongs to, which user owns it. Now an encrypted token from GPT-4 becomes cryptographically bound to GPT-4. If you try to inject it into GPT-3.5, the decryption fails because the context doesn't match. The token can't be reused across model boundaries. It's a simple concept, but it fundamentally breaks the interchangeability that enables the attack.

An alternative cryptographic approach is to use different encryption keys for different models or different security tiers. Frontier models use one key, weaker models use another. If a token is encrypted with the frontier key, a weaker model can't decrypt it because it doesn't have the key. This is more heavyweight but provides stronger isolation.

Beyond cryptography, the system architecture itself could change. Instead of returning encrypted tokens to clients, companies could store reasoning server-side but much more efficiently: compressed to a fraction of its original size, retained only briefly, indexed by session and user ID so compliance and deletion become straightforward database operations. Modern storage is cheap enough to make this viable, especially if reasoning is aggressively compressed and retention windows are limited to days or weeks rather than indefinite.

Another architectural approach is to separate the reasoning infrastructure from the API infrastructure entirely. Reasoning lives on one set of servers with strict, segregated access controls. The API layer generates responses based on reasoning, but those responses don't carry the reasoning tokens at all. Reasoning never leaves the secure infrastructure.

The research on mitigating chain-of-thought leakage and sanitizing reasoning for PII protection points toward similar directions: the system needs to stop treating reasoning as something that can safely flow across boundaries just because it's encrypted.

The broader lesson

This vulnerability reveals something important about how modern AI systems are built. The companies involved aren't incompetent. They're constrained. Reasoning traces create genuine value for users but also genuine security problems. The companies tried to have both by encrypting the traces and moving them to clients. It was a reasonable engineering decision given the constraints.

But it also embedded an assumption that didn't survive contact with reality. The assumption was that encryption plus isolation plus internal safeguards would be enough. In practice, the system created new interfaces for attack that wouldn't have existed if reasoning had never left the server in the first place.

Going forward, this suggests that reasoning traces, like other forms of sensitive AI system state, probably shouldn't move across trust boundaries at all, no matter how they're encrypted. The companies have moved reasonably quickly to patch this specific vulnerability through the mitigations described, but the deeper lesson is about design philosophy: some things are safer if they stay in one place.

Read the full paper summary on AIModels.fyi

A beginner's guide to the Vibevoice model by Microsoft on Replicate

aimodels-fyi — Mon, 24 Aug 2026 03:13:28 +0000

This is a simplified guide to an AI model called Vibevoice maintained by Microsoft. If you like these kinds of analysis, you should join AImodels.fyi or follow us on Twitter.

Overview

vibevoice is microsoft's long-form multi-speaker text-to-speech model that synthesizes conversational audio up to 90 minutes in a single pass with support for up to 4 distinct speakers. The model uses continuous speech tokenizers (Acoustic and Semantic) operating at an ultra-low frame rate of 7.5 Hz combined with a next-token diffusion framework that leverages a Large Language Model to understand textual context and a diffusion head to generate high-fidelity acoustic details. The architecture is based on a 1.5B parameter model that maintains speaker consistency and semantic coherence across long-form dialogue. The model accepts text scripts with multiple named speakers, produces high-quality audio output, and supports English, Chinese, and other languages. This is a research model intended for development purposes, not production deployment without further testing.

Best use cases

Podcast and long-form audio production: Generate complete podcast episodes or audiobook chapters in a single pass with multiple speakers maintaining consistent voices across 90-minute stretches. The model's ability to handle extended dialogue with natural turn-taking makes it suitable for scripted podcast production where you want to avoid stitching together multiple short audio clips.

Multi-speaker conversational content: Create dialogue-heavy content like interview simulations, educational conversations between multiple characters, or fictional dialogues where speaker identity must remain consistent throughout. The 4-speaker support with built-in diarization handling produces outputs that sound like natural conversation rather than separate synthesized chunks.

Cross-lingual content creation: Generate speech in English, Chinese, and other supported languages with the same long-form capabilities. This enables creation of localized audio content for global audiences without managing separate models for each language.

Spontaneous speech and singing: The model demonstrates capability for generating spontaneous-sounding speech patterns and even spontaneous singing, making it suitable for creative audio projects that require naturalness beyond standard read-speech synthesis.

Research and experimentation with TTS architecture: As an open-source frontier model, it serves researchers exploring continuous tokenization approaches, diffusion-based speech synthesis, and long-context audio generation techniques at scale.

Limitations

The model is explicitly not recommended for commercial or real-world applications without further testing and development. Microsoft states it is intended for research and development purposes only. The removed TTS code from the repository following instances of misuse indicates the model can produce realistic synthetic speech suitable for deepfakes and disinformation, requiring responsible deployment practices. The model inherits biases and errors from its base language model (Qwen2.5 1.5b), which affects output quality and potential harmful stereotypes in synthesized speech.

Output quality remains inconsistent for certain inputs: the model may produce unexpected, biased, or inaccurate speech outputs in edge cases. Accuracy depends heavily on input script quality and clarity. The Replicate API currently limits inputs to 4 speakers simultaneously through the schema (speaker_1, speaker_2, speaker_3, speaker_4), though the underlying model claims support for up to 4 distinct speakers in conversation. Speaker naming and turn-taking must be explicitly indicated in the input script; the model does not automatically detect speaker boundaries from unmarked text.

CFG scale (guidance strength) ranges from a default of 1.3 and affects output consistency, but the schema does not specify minimum or maximum bounds, making it unclear what values are actually supported. The model's 90-minute limit for single-speaker or conversational speech may still fragment very long inputs, and actual inference time for such extended outputs is not documented. The Replicate schema provides no explicit information about supported audio formats, sample rates, or codec specifications for the output audio file.

How it compares

vibevoice by fal-ai provides the same core model but through a different platform (FAL instead of Replicate), potentially offering different latency, pricing, or API ergonomics. Choose this Replicate version if you already use Replicate's infrastructure; choose the FAL version if you prefer FAL's platform or have latency/cost advantages with that provider.

VibeVoice-1.5B by microsoft on Hugging Face offers the same model weights for local deployment, giving you complete control over inference, batching, and customization at the cost of managing your own infrastructure and GPU resources.

vibevoice/0.5b by fal-ai is the lightweight streaming variant (0.5B parameters) optimized for real-time synthesis with ~300 milliseconds latency and ~10-minute long-form capability. Use the full 1.5B version when quality and 90-minute capability matter more than speed; use 0.5B for interactive, streaming, or resource-constrained deployments.

vibevoice/7b by fal-ai represents a larger, more capable variant of the VibeVoice family. Choose this if you need superior quality and have the compute budget; choose 1.5B for the balance of quality and efficiency that this Replicate model provides.

v3 by elevenlabs is a proprietary, production-ready TTS system with commercial support. Choose ElevenLabs v3 if you need guaranteed uptime, commercial licensing, and production support; choose vibevoice if you prioritize open-source research, multi-speaker dialogue, or want to self-host.

Technical specifications

The model is based on 1.5B parameters and uses a next-token diffusion framework combining a Large Language Model with a diffusion head. It operates continuous speech tokenizers at 7.5 Hz frame rate to efficiently preserve audio fidelity while handling long sequences. The underlying language model is Qwen2.5 1.5b, which inherits its training data characteristics and potential biases.

The Replicate version released on 2025-08-29 uses cog version 0.16.6 and the latest version ID is 624421f6fdd4122d0b3ff391ff3449f09db9ad4927167110a4c4b104fa37f728. The model is publicly visible on Replicate.

Input handling: The schema specifies a text script input, four optional speaker parameters, and a CFG scale parameter. The default CFG scale is 1.3. Speaker options appear to include multilingual voices (English, German, French, Italian, Japanese, Korean, Dutch, Polish, Portuguese, Spanish) with gender and style variants (e.g., "en-Alice_woman").

Output format: The model returns a single URI string pointing to the generated audio file. The actual audio codec, sample rate, and bitrate are not documented in the schema or README.

Supported languages: English, Chinese, and cross-lingual synthesis. The readme mentions multilingual voices in nine languages (DE, FR, IT, JP, KR, NL, PL, PT, ES) but indicates these are experimental and primarily documented for the 0.5B streaming variant.

Constraints: Maximum 4 speakers through the API schema. Conversational speech generation up to 90 minutes in single pass. Single-speaker speech also up to 90 minutes. No explicit minimum input length specified.

Model inputs and outputs

Inputs

script (string, required): The text content to convert to speech. No length limit explicitly stated but designed for long-form inputs up to 90 minutes of speech.
speaker_1 (enum, default: "en-Alice_woman"): The primary speaker voice. Supports multilingual options including English, German, French, Italian, Japanese, Korean, Dutch, Polish, Portuguese, and Spanish variants.
speaker_2 (enum, optional, nullable): Second speaker voice for multi-speaker dialogue.
speaker_3 (enum, optional, nullable): Third speaker voice for multi-speaker dialogue.
speaker_4 (enum, optional, nullable): Fourth speaker voice for multi-speaker dialogue.
scale (number, default: 1.3): CFG scale (guidance strength) controlling output consistency. No documented minimum or maximum bounds.

Outputs

Output (string, URI format): A URL pointing to the generated audio file. Format details (codec, sample rate, duration) are not specified in the schema.

Getting started

import replicate

client = replicate.Replicate()

output = client.run(
    "microsoft/vibevoice:624421f6fdd4122d0b3ff391ff3449f09db9ad4927167110a4c4b104fa37f728",
    input={
        "script": "Speaker 1: Hello, how are you today? Speaker 2: I'm doing great, thanks for asking. How about you? Speaker 1: Wonderful! I wanted to discuss the new project.",
        "speaker_1": "en-Alice_woman",
        "speaker_2": "en-Bob_man",
        "scale": 1.3
    }
)

print(output)

Replace the speaker voice options with available choices from the enum (e.g., "de-Klaus_man" for German, "fr-Marie_woman" for French). Adjust the script to use explicit speaker labels followed by colons. For single-speaker content, omit speaker_2, speaker_3, and speaker_4.

Frequently asked questions

Q: What audio format does the model output?

A: The Replicate schema specifies the output as a URI string pointing to the generated audio file, but does not document the audio codec, sample rate, or file format. Check the returned URL to determine the actual format and properties.

Q: Can I use this model commercially?

A: Microsoft explicitly states the model is not recommended for commercial or real-world applications without further testing and development. The model is intended for research and development purposes only. You should review the license at the GitHub repository and consult with legal counsel before any commercial deployment.

Q: How do I format multiple speakers in the script?

A: Use explicit speaker labels followed by colons in the script text (e.g., "Speaker 1: Hello there. Speaker 2: Hi, how are you?"). Assign the corresponding speaker voice to speaker_1, speaker_2, speaker_3, or speaker_4 via the input parameters. The model will maintain voice consistency for each speaker throughout the dialogue.

Q: What is the CFG scale and how should I adjust it?

A: CFG scale (guidance strength) controls how strongly the model adheres to the input text and speaker constraints. The default is 1.3. Higher values typically produce more constrained, consistent output; lower values may produce more creative but less reliable outputs. The schema does not specify the valid range, so experiment cautiously within reasonable bounds (0.5–2.0 as a starting point).

Q: How long can a single input script be?

A: The model is designed for long-form synthesis up to 90 minutes in a single pass for conversational or single-speaker content. No explicit minimum or maximum character/token limit is documented, but inputs should represent the approximate speech duration you want to generate (roughly 150 words per minute of speech as a guide).

Q: Is this model still actively maintained?

A: The Replicate version was released on 2025-08-29 and the GitHub repository continues to receive updates, including a significant update on 2025-09-05 where the original TTS code was removed due to misuse concerns. The model is actively maintained by Microsoft but with an emphasis on responsible research use.

Q: What are the known failure modes?

A: The model may produce unexpected, biased, or inaccurate speech, particularly with unclear scripts or content outside its training distribution. It inherits biases from Qwen2.5 1.5b. Speaker consistency can degrade with very long conversations (near 90 minutes) or when speaker boundaries are ambiguous. The model is not designed to handle musical notation, complex phonetic instructions, or other specialized audio synthesis requirements.

Q: What is the difference between this 1.5B model and the 0.5B streaming variant?

A: The 1.5B model prioritizes quality and long-form capability (up to 90 minutes), while the 0.5B streaming variant optimizes for real-time latency (~300 milliseconds to first audio) and streaming text input, with a maximum of ~10-minute speech generation. Use 1.5B for offline, high-quality batch processing; use 0.5B for interactive or resource-constrained applications.

Click here to read the full guide to Vibevoice

A beginner's guide to the Beat_this model by Xavriley on Replicate

aimodels-fyi — Mon, 24 Aug 2026 03:12:55 +0000

This is a simplified guide to an AI model called Beat_this maintained by Xavriley. If you like these kinds of analysis, you should join AImodels.fyi or follow us on Twitter.

Overview

beat_this is a beat and downbeat tracking model from the ISMIR 2024 paper "Beat This! Accurate Beat Tracking Without DBN Postprocessing" by xavriley and collaborators at CPJKU. The model detects precise beat positions and downbeat boundaries in audio files without relying on Dynamic Bayesian Network postprocessing, achieving state-of-the-art F1 scores while maintaining generality across diverse music genres. The architecture alternates convolutions with transformers operating either over frequency or time dimensions, and is trained on multiple datasets including solo instruments, pieces with time signature changes, and classical music with high tempo variations. The main model (final0, final1, final2) weighs approximately 78 MB each, with a smaller variant available at 8.1 MB. The most critical detail before using it: the model achieves good results specifically because it avoids meter and tempo constraints that traditional systems impose, but this means it can still fail on difficult and underrepresented genres and performs worse on continuity metrics compared to methods using postprocessing.

Best use cases

Music information retrieval and analysis workflows. If you build music analysis software that needs to segment tracks into beat-aligned sections for tempo detection, structural analysis, or synchronization with other modalities, beat_this provides clean beat and downbeat annotations without requiring external postprocessing pipelines. The model outputs precise timestamps suitable for downstream music information retrieval tasks like onset detection or harmonic analysis.

Rhythm-aware music production tools. For digital audio workstations, beat detection plugins, or metronome applications, this model provides frame-level accuracy suitable for real-time audio alignment and grid snapping. The two input parameters—constant tempo assumption and DBN postprocessing toggle—let you trade accuracy for continuity depending on whether your source material has stable rhythm or requires smoothing.

Music transcription and notation systems. Beat and downbeat detection forms a foundational layer for automatic music transcription pipelines. Feeding the model's output into tempo curve estimation and time signature inference produces richer musical analysis than beat detection alone, particularly useful for music notation software or academic music analysis tools.

Dataset annotation and validation. If you maintain collections of annotated music data, this model can automatically generate beat annotations for new tracks, which human annotators can then correct. The availability of multiple trained seeds (final0, final1, final2) allows you to ensemble predictions for higher confidence annotation.

Classical and complex music analysis. The model explicitly handles tempo variations, time signature changes, and solo instruments—scenarios where simpler beat trackers fail. If your music corpus includes Bach, contemporary classical, or experimental music, this model outperforms systems trained only on pop or dance music datasets.

Limitations

The model trains on multiple public datasets but explicitly excludes GTZAN for evaluation fairness; if you run inference on GTZAN files, results will be misleadingly high. The model struggles with difficult and underrepresented genres—the paper acknowledges this limitation directly. Performance on continuity metrics (how smoothly beat timing progresses) is worse than DBN-postprocessed systems, making it less suitable for applications requiring strictly monotonic beat sequences without jitter. The Replicate API defaults to using DBN postprocessing (use_dbn: true) and constant tempo assumption (constant_tempo: true), which partially reverses the paper's main contribution of avoiding DBN; disabling use_dbn requires setting it to false explicitly. The model uses PyTorch 2.0+, ffmpeg for non-WAV audio, and optional CUDA support—CPU inference is slower and lacks float16 optimization. The input schema accepts only audio as a URI, meaning you must upload files to a publicly accessible URL rather than passing raw bytes. Output format from Replicate's API is undocumented (schema shows title: Output with no further specification), so the actual return structure requires inspection during your first API call. The model file size (78 MB for main variants) requires downloading checkpoints on first use, which takes time on bandwidth-limited systems.

How it compares

demixing by jimothyjohn separates instruments and vocals from audio—a different task than beat detection. Choose beat_this if you need tempo and rhythm information; choose demixing if you need to isolate individual instruments before processing.

all-in-one-music-structure-analysis by cwalo performs comprehensive analysis including BPM, downbeats, and structure in one call. Use this alternative if you need multiple outputs (structure, demuxing, BPM) simultaneously; choose beat_this if you want the most accurate beat and downbeat detection specifically, as it focuses on that single task and achieves state-of-the-art F1 scores.

music by elevenlabs and music-2.6 by minimax are generative models that compose music from prompts—the opposite of beat_this, which analyzes existing audio. These are irrelevant if your goal is beat tracking, but useful if you need to create rhythmically structured content.

musicgen by charlesmccarthy generates music from text or composition plans, again a generative task rather than analysis. Not comparable to beat_this unless you plan to analyze generated music afterward.

Technical specifications

The model uses a hybrid convolutional-transformer architecture that alternates between frequency-domain and time-domain transformer blocks. Training uses a custom loss function tolerant to small time shifts in beat annotations (typically ±70 ms), addressing a core weakness in prior beat tracking systems where perfect frame alignment was unrealistic. The model trains on spectrograms (22 kHz sample rate, monophonic) preprocessed from raw audio using pedalboard. Multiple model variants exist: the main final0/1/2 are trained on all datasets except GTZAN with three random seeds; small0/1/2 offer a 10x smaller model (8.1 MB) with slight accuracy drops; single_final0/1/2 use a single train/validation split; and fold0-7 provide 8-fold cross-validation variants for fair evaluation on datasets used in training. The model supports CPU and GPU inference (CUDA recommended), with optional float16 precision for recent GPUs. Inference can be distributed across multiple GPUs using the --touch-first and --skip-existing command-line flags. The Replicate deployment defaults to using DBN postprocessing from madmom, which contradicts the paper's main claim about avoiding DBN—this requires explicit configuration to disable. Input audio supports any format readable by torchaudio with ffmpeg backend; output format is a .beats TSV file compatible with Sonic Visualizer.

Architecture: Alternating convolutional and partial transformer blocks (frequency and time transformers)
Model sizes: 78 MB (main/final), 8.1 MB (small), 78 MB (single/fold variants)
Training data: Multiple datasets including solo instruments, time signature changes, classical music; excludes GTZAN
Audio preprocessing: 22 kHz monophonic spectrograms via pedalboard
Loss function: Shift-tolerant loss (~70 ms tolerance for beat annotation timing)
Inference compute: GPU (CUDA) preferred; CPU fallback available; float16 option on recent GPUs
Output format: TSV .beats file with beat and downbeat timestamps
Dependencies: PyTorch 2.0+, tqdm, einops, soxr, rotary-embedding-torch; ffmpeg for non-WAV audio; optional madmom for DBN
License: Check the repository LICENSE file (linked in metadata)

Model inputs and outputs

Inputs

audio (string, URI, required): URL pointing to an audio file in WAV, MP3, FLAC, or other formats supported by torchaudio with ffmpeg
constant_tempo (boolean, default: true): Assume the source material has constant tempo; use true for pop/electronic music, false for classical or live recordings with tempo drift
use_dbn (boolean, default: true): Apply Dynamic Bayesian Network postprocessing for temporal smoothing; set to false to use the raw model output without DBN constraints

Outputs

Output (object): The schema indicates only title: Output with no documented structure; returns beat and downbeat annotations (likely as a .beats file or JSON array of timestamps based on the command-line tool behavior)

Getting started

import replicate

# Initialize client (assumes REPLICATE_API_TOKEN environment variable)
client = replicate.Replicate()

# Run beat detection on an audio file
output = client.run(
    "xavriley/beat_this:26142842c6dc94673820f0a9762214fa7109015d966031d8e6137bef4fd14323",
    input={
        "audio": "https://example.com/path/to/your/audio.mp3",
        "constant_tempo": True,
        "use_dbn": True
    }
)

print(output)

For local Python use without Replicate, install the package and use directly:

from beat_this.inference import File2Beats

# Load model (downloads automatically on first run)
file2beats = File2Beats(checkpoint_path="final0", device="cuda", dbn=False)

# Get beat and downbeat positions
beats, downbeats = file2beats("path/to/audio.mp3")

# Save to Sonic Visualizer format
from beat_this.utils import save_beat_tsv
save_beat_tsv(beats, downbeats, "output.beats")

Frequently asked questions

Q: Should I disable use_dbn when running on Replicate?

A: It depends on your use case. The paper's main contribution is achieving state-of-the-art results without DBN, so set use_dbn: false if you want the raw model predictions. However, DBN provides smoother, more musically consistent beat sequences at the cost of introducing meter constraints; use use_dbn: true if you need temporal continuity or notice jittery output from the raw model.

Q: What audio formats does beat_this accept?

A: The model accepts WAV, MP3, FLAC, and any format that ffmpeg can decode, provided ffmpeg is installed and torchaudio is configured to use it as a backend. For Replicate, pass audio as a URI string rather than raw bytes.

Q: How accurate is this model compared to manual annotation?

A: The paper reports state-of-the-art F1 scores on the GTZAN test set when using the final0/1/2 models. However, accuracy degrades on difficult and underrepresented genres, and it performs worse on continuity metrics (how smoothly beat timing changes) compared to DBN-postprocessed systems. For fair evaluation on datasets used in training, use the cross-validation variants (fold0-7 or single_final variants).

Q: Can I use this model commercially?

A: Check the LICENSE file in the GitHub repository for the specific open-source license terms. The code is publicly available and the paper is published, but license restrictions may apply depending on your use case.

Q: Which model variant should I use: final, small, or fold?

A: Use final0 (the default) for general-purpose beat tracking on unseen music. Use small0/1/2 if model size or inference speed matters more than accuracy—they sacrifice some F1 score for 10x smaller file size. Use fold* or single_final* only for fair research evaluation on datasets that contributed to training data; otherwise results will be biased upward.

Q: How long does inference take on CPU vs GPU?

A: The README does not specify exact inference times, but indicates GPU is strongly recommended. The command-line tool defaults to GPU with CPU fallback; float16 mode on recent GPUs improves speed. For batch processing, distribute across multiple GPUs with separate process instances.

Q: What happens if the audio has tempo changes or unusual time signatures?

A: The model is specifically designed to handle tempo variations and time signature changes, as demonstrated on classical and contemporary music in the training data. However, disable constant_tempo: true if you suspect tempo drift, and be aware that very extreme or rapid tempo changes may still cause failures.

Q: Does the model work for non-Western music or underrepresented genres?

A: The paper explicitly acknowledges that the model "can still fail, especially for difficult and underrepresented genres." The training datasets focus on Western music traditions. If you work with non-Western, experimental, or niche genres, test on a representative sample before production use.

Click here to read the full guide to Beat_this

A beginner's guide to the Qwen-Image-2-Pro model by Qwen on Replicate

aimodels-fyi — Mon, 24 Aug 2026 03:12:20 +0000

This is a simplified guide to an AI model called Qwen-Image-2-Pro maintained by Qwen. If you like these kinds of analysis, you should join AImodels.fyi or follow us on Twitter.

Overview

qwen-image-2-pro is a text-to-image generation model from qwen, Alibaba's Qwen team, that emphasizes text rendering, semantic adherence, and realism. The model is built on a 20-billion-parameter Multimodal Diffusion Transformer (MMDiT) architecture and supports multiple aspect ratios with native 2K resolution capability. The single most important thing to know before using it: this model prioritizes complex text rendering—particularly for Chinese logographic text—through a progressive training strategy that evolved from simple to complex textual inputs. It integrates Qwen2.5-VL as its vision-language understanding component and combines semantic representations from the vision encoder with reconstructive representations from a VAE for enhanced consistency. The pro version specifically optimizes for enhanced text rendering, realism, and semantic adherence compared to the base release.

Best use cases

Professional infographic and typography-heavy design generation. The model excels at generating structured layouts like PowerPoint presentations, posters, comics, and documents with precise text placement and readability. Unlike general-purpose image models that often fail at typography, the pro version's curriculum learning approach and large-scale text annotation pipeline enable it to handle paragraph-level descriptions and render multiple text elements with correct spacing and alignment. This makes it suitable for automating marketing material creation or generating design mockups from text specifications.

Chinese text rendering in generated images. The model achieves state-of-the-art performance on logographic languages, particularly Chinese, which most Western image models struggle with. If your application requires generating images with Chinese labels, signage, or captions embedded in the image itself, this model will produce significantly better results than competitors. The technical report demonstrates exceptional progress on Chinese text rendering through specialized data collection and synthesis pipelines.

Realistic human portrait and character generation. The pro version reduces "AI look" artifacts and includes richer facial and age details compared to earlier versions. It generates finer natural textures in skin, hair, and materials, making it suitable for creating authentic-looking character references, avatar generation, or portrait-style illustrations where photorealism matters. The model handles age variation, facial expressions, and subtle lighting effects better than base models.

Detailed landscape and natural texture generation. The enhanced version produces sharper landscapes, water reflections, fur textures, and material details. If your use case involves generating nature-focused imagery, architectural photography, or detailed environmental art, the improved texture fidelity translates to higher production quality without post-processing.

Flexible aspect ratio content generation at scale. The model supports seven predefined aspect ratios (1:1, 16:9, 9:16, 4:3, 3:4, 3:2, 2:3) with specified dimensions. This enables batch generation of cohesive image sets for social media campaigns, product displays, or design systems where consistent proportions and composition across multiple images matter without manual cropping.

Limitations

No native image editing capability on this version. While Alibaba's Qwen team has released separate image editing models (Qwen-Image-Edit-2511), qwen-image-2-pro on Replicate is pure text-to-image generation. The input schema shows optional image input support, but this is for style transfer or image-to-image reference, not for direct editing workflows. If you need precise masking-based inpainting or region-specific modifications, you need a different model.

Inference speed not specified. The README mentions optimization frameworks like LightX2V achieve 42x overall speedup and diffusion distillation reduces inference steps, but no baseline generation time is documented for this Replicate deployment. You should test expected latency in your target use case before production deployment. The model requires approximately 50 inference steps by default based on the README example.

Text accuracy degrades with complexity. While text rendering is exceptional for the category, extremely long multi-paragraph instructions, complex nested text hierarchies, or stylized fonts still produce occasional errors. The model's strength is in clear, readable text rendering—not in photorealistic font fidelity or handwriting styles.

Resolution constraints within aspect ratio boundaries. Maximum resolutions vary by aspect ratio (1664×928 for 16:9, 1584×1056 for 3:2, etc.). The largest single dimension caps around 1664 pixels. For applications requiring 4K or ultra-high-resolution output, this model falls short of newer competitors.

No output format customization. The API returns a single URI string (image URL) without control over format, quality compression, or metadata embedding. You cannot request WebP, AVIF, or lossless PNG variants—the model returns a default format.

Limited negative prompt control. The schema includes a negative_prompt field, but the README examples use extended Chinese negative prompts ("低分辨率，低画质，肢体畸形..." etc.), suggesting optimal results may require careful negative prompt engineering. English-only users lack documented best practices.

Prompt expansion is automatic and sometimes overcorrective. The enable_prompt_expansion parameter defaults to true, meaning Replicate automatically modifies your input prompt for optimization. If you need exact semantic control without model-driven modification, you must explicitly set this to false, but results may degrade.

Apache 2.0 license requires attribution. Commercial use is permitted, but you must provide license notices and cannot hold the authors liable. This is generally permissive for most use cases, but compliance is mandatory for production deployment.

Successor models already available. Qwen-Image-2.0 and Qwen-Image-2512 are documented as improvements with better character realism and faster inference. This pro version may become outdated as newer releases ship to Replicate. Check the maintainer's releases regularly.

How it compares

vs. qwen-image-2: The qwen-image-2 variant on Replicate is the next-generation unified model supporting both generation and editing. Choose qwen-image-2-pro if you want optimized text rendering and realism focused on generation quality alone; pick qwen-image-2 if you need integrated image editing in a single pipeline without switching models. The pro version likely has slightly better typography, while qwen-image-2 offers more operational flexibility.

vs. wan-2.7-image-pro by wan-video: Wan 2.7 Pro supports 4K output and multi-image editing workflows with a thinking mode for complex prompts. Choose qwen-image-2-pro for superior text rendering and Chinese language support; pick Wan 2.7 if you absolutely need 4K resolution and integrated multi-image coordination. Wan excels at photorealism at higher resolutions; Qwen excels at text and Chinese content.

vs. hunyuan-image-2.1 by tencent: Hunyuan-Image-2.1 is also tuned for Chinese text and supports 2K resolution. Choose qwen-image-2-pro if the Replicate integration and Qwen's training methodology (progressive curriculum learning for text) matter to your workflow; pick Hunyuan if you prefer Tencent's model or have better latency/cost data suggesting Hunyuan performs better in your region. Both are competitive on Chinese text rendering.

vs. qwen-image-fast by prunaai: Qwen-Image-Fast is an aggressively optimized version that generates 1.5MP images in ~1 second through distillation. Choose qwen-image-2-pro if generation quality and text fidelity matter more than latency; pick qwen-image-fast if you need real-time inference for interactive applications and can tolerate lower text rendering accuracy. The pro version is production-grade quality; fast is production-grade speed.

vs. qwen-image-2/pro/text-to-image by fal-ai: This is the same model deployed on fal.ai instead of Replicate. Choose based on which platform better integrates with your stack. Replicate has stronger async job handling; fal-ai may offer different pricing or latency profiles. No quality difference between the deployments.

Technical specifications

qwen-image-2-pro is a Multimodal Diffusion Transformer (MMDiT) with 20 billion parameters. It integrates Qwen2.5-VL as its vision-language encoder for semantic understanding and uses a dual-encoding mechanism that feeds the original image into both the vision encoder and a VAE encoder separately—this allows the model to balance semantic consistency with visual fidelity during editing tasks.

The model was trained on a comprehensive data pipeline including large-scale collection, filtering, annotation, synthesis, and balancing. Progressive training strategy began with non-text-to-text rendering, evolved from simple to complex textual inputs, and scaled to paragraph-level descriptions. This curriculum learning approach is the source of the text rendering advantage, particularly for Chinese.

For generation, the model uses 50 inference steps by default with a true_cfg_scale of 4.0 (classifier-free guidance). It operates in bfloat16 precision (lower-precision inference compared to float32) for memory efficiency and can run on NVIDIA GPUs with sufficient VRAM (exact requirements not documented in the provided materials, but transformer-based diffusion models of this scale typically require 16-40GB).

Supported aspect ratios and their native resolutions:

1:1 → 1328×1328
16:9 → 1664×928
9:16 → 928×1664
4:3 → 1472×1104
3:4 → 1104×1472
3:2 → 1584×1056
2:3 → 1056×1584

The model can match input image dimensions when match_input_image is set to true, overriding the aspect_ratio parameter. The API implements automatic prompt expansion via the enable_prompt_expansion parameter, which modifies input prompts for optimization when enabled (default: true).

The codebase requires transformers>=4.51.3 (supporting Qwen2.5-VL integration) and the latest diffusers library from HuggingFace. Model weights are available on HuggingFace and ModelScope, distributed under Apache 2.0 license. The Replicate deployment was last updated 2026-03-04 using Cog version 0.16.12.

Model inputs and outputs

Inputs

prompt (string, required): Text description of the image to generate. Supports paragraph-level complexity with text rendering instructions.
image (string, URI, optional): URL of a reference image for style transfer, image-to-image generation, or editing context. Not required for pure text-to-image generation.
aspect_ratio (enum, default: "1:1"): Predefined aspect ratio selection. Valid values: "1:1", "16:9", "9:16", "4:3", "3:4", "3:2", "2:3". Ignored if match_input_image is true.
negative_prompt (string, default: ""): Specifications for elements to avoid in generation. Can be in English or Chinese.
match_input_image (boolean, default: false): When true and an image is provided, uses the input image's native aspect ratio and resolution instead of the aspect_ratio parameter.
enable_prompt_expansion (boolean, default: true): Toggles automatic prompt optimization. When true, Replicate rewrites your prompt for better results; when false, uses your prompt literally.
seed (integer, range 0–2147483647, optional): Random seed for reproducible generation. Omitting produces non-deterministic results.

Outputs

output (string, URI): A URL pointing to the generated image. The image is returned in a default format (specific format not documented); hosted on Replicate's CDN for temporary access.

Getting started

import replicate

client = replicate.Replicate()

output = client.run(
    "qwen/qwen-image-2-pro",
    input={
        "prompt": "A 25-year-old professional woman with warm brown eyes, wearing a navy blazer, standing in a modern office with floor-to-ceiling windows overlooking a city skyline. Soft natural lighting. High-quality portrait photography style.",
        "negative_prompt": "blurry, low quality, distorted face, artificial lighting",
        "aspect_ratio": "16:9",
        "enable_prompt_expansion": True,
        "seed": 42
    }
)

print(output)  # Returns URL string to generated image

For Chinese text rendering:

output = client.run(
    "qwen/qwen-image-2-pro",
    input={
        "prompt": "一张现代办公室海报，标题写着'2024年度报告'，包含柱状图表和饼图，配色为蓝色和白色，背景是城市写字楼的窗户景观。专业设计风格。",
        "negative_prompt": "文字模糊，扭曲，低分辨率，AI生成感强",
        "aspect_ratio": "4:3",
        "enable_prompt_expansion": True,
        "seed": 123
    }
)

print(output)

For image-to-image with style transfer:

output = client.run(
    "qwen/qwen-image-2-pro",
    input={
        "prompt": "Apply the style of an oil painting with warm, golden tones",
        "image": "https://example.com/reference-image.jpg",
        "match_input_image": True,
        "enable_prompt_expansion": False,
        "seed": 456
    }
)

print(output)

Frequently asked questions

Q: What is the expected generation time for an image?

A: The README shows default 50 inference steps, but specific latency for the Replicate deployment is not documented. Related optimization frameworks (LightX2V) achieve 42x overall speedup, suggesting baseline inference could be significantly accelerated if you deploy locally, but Replicate's hosted version timing depends on queue load and GPU allocation. Test with your expected load to determine SLA feasibility.

Q: Can this model edit existing images, or is it text-to-image only?

A: qwen-image-2-pro is text-to-image generation with optional image input for style transfer and image-to-image reference. The Replicate deployment does not support the image editing capabilities of Qwen-Image-Edit-2511. If you need inpainting, masking, or direct editing, use a dedicated editing model instead.

Q: Does the model handle both English and Chinese text rendering equally well?

A: The model handles both, but excels at Chinese text rendering due to specialized data pipelines and curriculum learning optimized for logographic languages. English text is also strong, but the README and technical report emphasize Chinese as the standout feature. For multilingual documents, expect Chinese components to render more reliably than mixed-script complexity.

Q: What resolution does this model support, and can I generate 4K images?

A: The maximum resolution depends on aspect ratio (1664×928 for 16:9, 1584×1056 for 3:2, etc.). The largest single dimension is approximately 1664 pixels, so this does not support 4K (3840×2160 or higher). For 4K output, consider wan-2.7-image-pro instead.

Q: Should I disable prompt expansion, or is the default automatic optimization reliable?

A: The default automatic prompt expansion is enabled and generally improves results without degrading quality. Disable it only if you require exact semantic control over your input text or suspect the model is misinterpreting your intent. When disabled, results may be lower quality, so test both settings for your specific use case.

Q: Is this model suitable for commercial production use?

A: Yes. The Apache 2.0 license permits commercial use. However, you must provide license attribution and cannot hold the authors liable. For production systems, implement rate limiting, error handling, image validation, and cost monitoring since Replicate charges per API call. The model is actively maintained with regular releases, so check for newer versions periodically.

Q: How does this compare to other Chinese text-rendering models like Hunyuan-Image-2.1?

A: Both qwen-image-2-pro and hunyuan-image-2.1 prioritize Chinese text rendering and support 2K resolution. The main difference is provider (Alibaba vs. Tencent) and Qwen's specific curriculum learning methodology. Pick based on Replicate vs. Tencent integration preferences and your observed latency/cost in practice—quality differences are marginal for Chinese text.

Q: Can I use the same seed to guarantee identical outputs across multiple API calls?

A: Yes. The seed parameter (range 0–2147483647) enables reproducible generation. Set it to the same value to regenerate identical images. This is useful for A/B testing, debugging, and deterministic workflows, though hardware variations across inference runs may introduce minor noise-level differences.

Q: What happens if I set match_input_image to true without providing an image?

A: This parameter is only meaningful when an image is provided. If you set it true without an image input, it is ignored and the aspect_ratio parameter applies instead. Always pair match_input_image: true with a valid image URI to override aspect ratio selection.

Click here to read the full guide to Qwen-Image-2-Pro

A beginner's guide to the Flux-Pulid model by Jichengdu on Replicate

aimodels-fyi — Mon, 24 Aug 2026 03:11:46 +0000

This is a simplified guide to an AI model called Flux-Pulid maintained by Jichengdu. If you like these kinds of analysis, you should join AImodels.fyi or follow us on Twitter.

Overview

flux-pulid is a face identity customization model built on the FLUX diffusion architecture that generates images matching specific identity characteristics extracted from reference photos. Developed by jichengdu at ByteDance, this model implements the PuLID (Pure and Lightning ID Customization via Contrastive Alignment) method presented at NeurIPS 2024. The model uses contrastive alignment to inject identity information into the FLUX generation process while maintaining high image quality and editability through text prompts. The critical distinction before using it: this model trades some identity fidelity compared to newer v0.9.1 releases in exchange for broader male face compatibility, and the identity injection mechanism via the start_step parameter fundamentally alters how much you can edit the generated image through text prompts—starting at step 0 preserves identity most faithfully but reduces prompt control, while starting at step 4 enables more creative editing but weakens identity retention.

Best use cases

Portrait generation with specific identity preservation for creative projects. When you have a reference photo of a person and need to generate new images of them in different contexts, styles, or compositions, flux-pulid excels at maintaining recognizable facial characteristics while applying text-based modifications. For example, generating professional headshots in various business settings, reimagining a portrait subject in historical clothing, or creating multiple stylistic interpretations of the same person's face. The identity weight parameter (0.0–3.0) lets you tune how strongly the model enforces facial similarity, useful when the text prompt demands significant appearance changes that conflict with strict identity preservation.

Diverse pose and expression variations from a single reference photo. Rather than hiring models for multiple photoshoots, you can use one reference portrait and generate variations with different poses, expressions, lighting conditions, and camera angles. This is valuable for headshot galleries, dating profile variations, or character exploration in game and film pre-production. The guidance scale and prompt sophistication of FLUX means you have fine control over these variations through natural language rather than manual parameter tweaking.

Identity-consistent creative fashion and styling exploration. Fashion designers and stylists can reference a person's face and generate mockups showing how they look in different clothing, hairstyles, and makeup styles without requiring fitting sessions. The model preserves facial identity while the text prompt controls visual elements like "wearing a Victorian gown with elaborate updo" or "punk rock aesthetic with heavy eyeliner." This workflow compresses design iteration cycles from hours to minutes.

Conditional face generation for content moderation and synthetic data. The ability to generate variations of specific identities from reference images can support testing facial recognition systems, generating balanced datasets for bias evaluation, or creating templated variations of approved identities for controlled content. The seed parameter ensures reproducibility when needed for validation workflows.

Limitations

Identity fidelity gaps on certain male face inputs. The v0.9.0 model used here (based on the Replicate metadata) exhibits lower ID fidelity on some male face reference images compared to later v0.9.1 releases. The README explicitly notes "ID fidelity is not high enough for some male inputs," meaning certain male faces may not generate with sufficient similarity to the reference. This is a known limitation of this specific version and suggests male identity preservation requires careful prompt engineering or switching to v0.9.1 if available.

Fundamental tradeoff between identity fidelity and prompt editability. The start_step parameter (0–10, recommended 0–4) controls when identity information injection begins in the denoising process. Starting at step 0 maximizes identity preservation but severely constrains what text prompts can achieve—the generated image will look most like the reference but may ignore creative prompt requests. Starting at step 4 allows much more prompt influence and creative control but weakens identity similarity by roughly 5 percentage points according to the README. You cannot simultaneously achieve maximum fidelity on both dimensions.

Maximum resolution constraints limit billboard and large-format use. The model generates images up to 1536×1536 pixels (width and height individually capped at 1536 minimum 256), but inference time and quality degrade at maximum resolutions. For large-format printing, cinema-quality assets, or applications requiring 4K resolution, this model falls short. The default is 896×1152, a reasonable compromise.

Inference speed and computational requirements not publicly documented. The README mentions the model requires 16GB VRAM for FLUX-dev standard precision (or can use fp8 quantization to reduce to 12GB), but actual inference latency per API call is not specified. This makes it difficult to predict response times for time-sensitive applications or batch processing workflows without running benchmarks.

Limited batch generation and output control. You can generate 1–4 images per call (via num_outputs), but cannot generate more without multiple API calls. The model offers no built-in upscaling, inpainting, or face detection/alignment features—you handle image preprocessing and post-processing separately.

Text prompt encoding constraints. The max_sequence_length parameter caps T5 tokenization at 128–512 tokens (default 128), meaning extremely long or detailed prompts may be truncated. This is much shorter than modern LLM context windows and may force you to compress complex creative direction.

License and commercial use restrictions. The license is available at the GitHub repository, but the README includes a disclaimer: "This project strives to impact the domain of AI-driven image generation positively. Users are granted the freedom to create images using this tool, but they are expected to comply with local laws and utilize it responsibly. The developers do not assume any responsibility for potential misuse by users." This suggests caution around identity-based synthetic images in regulated contexts (deepfakes, impersonation, etc.).

How it compares

flux-pulid by zsxkib. This appears to be a different implementation or version of PuLID-FLUX on Replicate. Without detailed schema comparison, both likely share the same underlying model but may differ in preprocessing, quantization, or wrapper code. Pick this model if the zsxkib version is unavailable or has different latency characteristics; both should produce similar quality outputs for the same inputs.

flux-pulid by bytedance. The official ByteDance implementation of PuLID-FLUX, likely identical or nearly identical to this model since both are from the original research team. If you need guaranteed official support or the latest optimizations, prefer the bytedance version; otherwise, functionality should be equivalent.

pulid by zsxkib. This is the SDXL-based PuLID model (v1 or v1.1), not the FLUX version. Choose SDXL-PuLID if you need faster inference on older hardware, more mature model versions with v1.1 improvements, or existing integrations with SDXL-based workflows. Choose this FLUX version for superior image quality, better prompt adherence, and more advanced generation features at the cost of higher computational requirements.

oot_diffusion_dc by qiweiii. This model specializes in full-body outfit-try-on synthesis, while flux-pulid focuses on face identity. Use OOT Diffusion when you need to see how clothing fits on a full body; use flux-pulid when the task is face-centric (portraits, expressions, styles applied to the head). These are complementary rather than competing tools.

flux-multi-pulid-controlnet by mattheum. This is a multi-identity fork of FLUX-PuLID that accepts multiple faces plus a depth map and bounding boxes to place different identities in specific regions of an image. Use the standard flux-pulid for single-identity generation with maximum fidelity and ease of use; use the ControlNet variant when you need simultaneous generation of multiple distinct people in one image with precise spatial control.

Technical specifications

Architecture and training. The model implements PuLID as described in the NeurIPS 2024 paper using FLUX-dev as the base diffusion architecture. The approach uses contrastive alignment to embed identity information extracted from reference face images, enabling precise identity conditioning during the generation process. The underlying FLUX model provides significantly higher image quality and prompt adherence compared to SDXL-based alternatives.

Model versioning. This Replicate deployment runs PuLID-FLUX-v0.9.0 (released September 12, 2024, per the README). A newer v0.9.1 version exists with approximately 5 percentage points higher ID similarity on quantitative metrics, particularly for male face inputs, though this deployment has not yet been updated.

Hardware requirements. The model requires a GPU with at least 16GB VRAM for standard FLUX-dev inference in bf16 precision. It can be optimized to run on 12GB VRAM using fp8 quantization (requires PyTorch 2.4.1+), according to the README.

Input specifications:

Reference face image: URI format, required, can be JPEG or PNG
Text prompt: string, default "portrait, color, cinematic"
Negative prompt: string, default includes common quality issues and artifacts
Image dimensions: width and height 256–1536 pixels independently, defaults 896×1152
Inference steps: 1–20, default 20 (higher = longer inference, better quality)
Identity weight: 0.0–3.0, default 1.0 (controls reference image influence)
Start step: 0–10, recommended 0–4 (when to inject identity; 0 = highest fidelity, 4 = more editability)
Guidance scale: 1.0–10.0, default 4.0 (text prompt adherence)
True CFG: 1.0–10.0, default 1.0 (values >1.0 enable True CFG for finer prompt control)
Seed: integer or -1 for random, affects noise initialization
Max sequence length: 128–512, default 128 (T5 tokenizer limit for prompts)
Output format: webp or jpeg
Output quality: 1–100 for lossy formats, default 80
Number of outputs: 1–4, default 1

Output specifications: Array of image URIs in the specified format (webp or jpeg), with quality determined by the output_quality parameter.

File format and model artifacts. The underlying model weights ship as safetensors format for security. The Replicate deployment wraps this with Cog (version 0.13.7 per metadata) for containerized inference.

Model inputs and outputs

Inputs

main_face_image (string, URI): Reference face image for identity extraction; required
prompt (string): Text description guiding generation; default "portrait, color, cinematic"
negative_prompt (string): Undesired visual elements; default includes quality issues and artifacts
width (integer, 256–1536): Output image width; default 896
height (integer, 256–1536): Output image height; default 1152
num_steps (integer, 1–20): Denoising iterations; default 20
start_step (integer, 0–10): When to begin identity injection (0 = max fidelity, 4 = max editability); default 0
guidance_scale (number, 1.0–10.0): Text prompt influence strength; default 4.0
true_cfg (number, 1.0–10.0): True CFG multiplier; 1.0 uses standard CFG; >1.0 enables finer control; default 1.0
id_weight (number, 0.0–3.0): Reference image influence magnitude; default 1.0
seed (integer): Random seed for reproducibility; -1 or blank for random; default blank
max_sequence_length (integer, 128–512): T5 prompt tokenization limit; smaller = faster; default 128
output_format (enum): "webp" or "jpeg"; default "webp"
output_quality (integer, 1–100): Compression quality for lossy formats; default 80
num_outputs (integer, 1–4): Number of images to generate; default 1

Outputs

Array of strings (URIs pointing to generated images in the specified format)

Getting started

import replicate

client = replicate.Replicate()

output = client.run(
    "jichengdu/flux-pulid",
    input={
        "main_face_image": "https://example.com/reference_face.jpg",
        "prompt": "a professional headshot in a modern office, natural lighting, wearing business attire",
        "negative_prompt": "bad quality, worst quality, text, signature, watermark, extra limbs",
        "width": 896,
        "height": 1152,
        "num_steps": 20,
        "start_step": 0,
        "guidance_scale": 4.0,
        "id_weight": 1.0,
        "seed": 42,
        "output_format": "webp",
        "output_quality": 80,
        "num_outputs": 1
    }
)

print(output)

Frequently asked questions

Q: How does the start_step parameter affect output quality and identity preservation?

A: The start_step parameter controls when identity information is injected into the diffusion process (0–10, with 0–4 recommended). Starting at step 0 maximizes facial similarity to the reference image but severely limits what the text prompt can achieve, resulting in less creative variation. Starting at step 4 allows the text prompt much more influence and creative control but reduces identity fidelity by approximately 5 percentage points. Choose based on whether you prioritize identity accuracy or prompt flexibility for your use case.

Q: What is the difference between guidance_scale and true_cfg?

A: The guidance_scale parameter (1.0–10.0) controls standard classifier-free guidance, strengthening how much the model follows your text prompt. The true_cfg parameter (1.0–10.0) is an advanced option where values >1.0 enable "True CFG," providing finer-grained control over generation characteristics. For most users, leave true_cfg at the default 1.0 and adjust guidance_scale instead.

Q: Can I use this model to generate images of people who are not in my reference photo?

A: No, this model is designed for identity customization—it extracts facial characteristics from the reference image you provide and generates new images preserving those characteristics. It cannot synthesize arbitrary identities without a reference photo. If you try to use a photo of person A to generate person B, the output will reflect person A's identity features.

Q: What happens if my reference face image is low quality or partially obscured?

A: The model relies on clear facial information to extract identity characteristics. Low-resolution, heavily filtered, or partially obscured reference images may result in poor identity transfer or artifacts in the output. The README explicitly mentions that identity fidelity varies, particularly on some male face inputs in the v0.9.0 version, so quality issues with specific face types are known limitations.

Q: Is this model suitable for production use with user-uploaded reference images?

A: Use caution. The README includes a disclaimer that users must comply with local laws and use the tool responsibly. Generating synthetic images of identities from user photos raises legal and ethical concerns around deepfakes, consent, and impersonation. In production contexts, you should implement identity verification, usage policies, and legal review before deploying identity customization features to end users.

Q: How does id_weight differ from start_step in controlling identity influence?

A: The id_weight parameter (0.0–3.0) controls the magnitude of identity information injected, acting like a volume knob for how strongly the reference face influences the output. The start_step parameter controls when that injection begins, fundamentally changing the interplay between identity and prompt control. Increasing id_weight strengthens identity at any given start_step; decreasing start_step injects identity earlier and locks in more facial similarity regardless of id_weight.

Q: What image dimensions should I use for best results?

A: The default 896×1152 is a reasonable balance, representing landscape-to-portrait aspect ratio typical of portrait photography. You can use 1536×1536 for maximum resolution, but inference time increases and quality may vary. For specific use cases (LinkedIn headshots are typically square or vertical; fashion photography may prefer wider shots), adjust width and height accordingly, keeping both between 256 and 1536.

Q: Is the model still actively maintained?

A: Yes, the original PuLID project is actively maintained by ByteDance researchers. The v0.9.1 release (October 2024) improved ID fidelity, and the main repository continues accepting issues and contributions. However, this specific Replicate deployment runs v0.9.0; the maintainer may update it to v0.9.1 in the future if they choose to refresh the wrapper.

Click here to read the full guide to Flux-Pulid

A beginner's guide to the Qwen-Image-2 model by Qwen on Replicate

aimodels-fyi — Mon, 24 Aug 2026 03:11:13 +0000

This is a simplified guide to an AI model called Qwen-Image-2 maintained by Qwen. If you like these kinds of analysis, you should join AImodels.fyi or follow us on Twitter.

Overview

qwen-image-2 is a next-generation image generation and editing foundation model from qwen, Alibaba's Qwen team. It is a 20-billion-parameter Multimodal Diffusion Transformer (MMDiT) that handles both text-to-image generation and image editing with exceptional text rendering capabilities, especially for Chinese characters and complex typography. The model supports up to 1k-token instructions for professional-grade graphic generation including presentations, posters, and comics, with native 2K resolution support for detailed realistic imagery. Key to its design are curriculum learning strategies that progressively scale from simple to complex text inputs, a comprehensive data pipeline with large-scale collection and synthesis, and a dual-encoding mechanism that balances semantic consistency with visual fidelity during editing. The model outputs single images in URI format and runs on Replicate's infrastructure.

Best use cases

Professional typography and infographic generation. This model excels at rendering text-heavy designs like presentation slides, posters, and promotional graphics where accurate text placement and legibility matter. The 1k-token instruction support and explicit typography training make it suitable for designers and marketers who need to generate branded materials with precise text composition, especially when those designs include Chinese characters or mixed-language text that other models struggle with.

Complex scene generation with detailed text elements. Use this model when you need photorealistic imagery that includes readable text, signage, or labels—product packaging with visible product names, street scenes with legible storefront signs, or book covers with rendered titles. The progressive training strategy from simple to paragraph-level descriptions gives it stronger semantic adherence for intricate multi-object scenes compared to earlier-generation models.

Image editing with high consistency preservation. The dual-encoding mechanism (semantic via Qwen2.5-VL and reconstructive via VAE) makes this model particularly strong at editing tasks where you need to modify specific elements while preserving the identity and visual quality of unchanged regions. Applications include portrait retouching, background replacement, or style transfer where the original image's integrity matters as much as the edit instruction.

Character realism and human generation. The 2025 Qwen-Image-2512 update dramatically reduced "AI look" artifacts and improved facial detail, age representation, and natural textures in human subjects. This makes it suitable for generating character artwork, portrait galleries, or concept art where human subjects need to appear convincingly real rather than stylized.

Multilingual prompt understanding with Chinese emphasis. This model handles complex Chinese text rendering and mixed-language prompts better than most open-source alternatives due to its curriculum learning on logographic languages. Use it when your prompts or desired output text are primarily Chinese, or when you need reliable text rendering in a non-Latin script.

Limitations

The model requires explicit aspect ratio specification or image matching; it does not automatically crop or resize to arbitrary dimensions. Text rendering, while improved, still produces occasional errors in character accuracy and spacing when prompts include very long strings or densely packed text. Generation time scales with resolution and inference step count; 50 steps at 1664×928 will take longer than 25 steps at 1024×1024, and the exact timing depends on your hardware. The model does not support arbitrary input image dimensions—you should provide images matching one of the standard aspect ratios (1:1, 16:9, 9:16, 4:3, 3:4, 3:2, 2:3) for best results. Negative prompts work but may not eliminate all unwanted visual patterns, particularly when conflicts exist between prompt and negative prompt semantics. The model may struggle with extremely detailed instructions exceeding 1k tokens or prompts requiring frame-by-frame consistency across multiple images. Generation output is deterministic only when using the same seed; different seeds on the same prompt produce different images. The model is Apache 2.0 licensed, which permits commercial use but requires attribution; check the specific license terms for your deployment context.

How it compares

qwen-image-2-pro: The pro version includes further enhancements in text rendering, realism, and semantic adherence. Choose qwen-image-2 for general-purpose generation and cost savings; choose qwen-image-2-pro when you specifically need maximum text accuracy, human realism, or are willing to pay more for marginally better output quality.

Qwen-Image: This is the earlier base model in the Qwen series. qwen-image-2 incorporates the improvements from later versions (Qwen-Image-2512, Qwen-Image-Edit-2511) with better human realism, finer natural textures, and stronger text rendering. Use the original Qwen-Image only if you require compatibility with older implementations or have computational constraints that favor the smaller predecessor.

qwen-image (FAL): This is the same foundational model hosted on FAL's infrastructure rather than Replicate. The core capabilities are identical; the choice depends on your deployment preference and pricing model. FAL may offer different latency characteristics or regional availability; Replicate integration may suit you better if you already use that platform.

hunyuan-image-2.1: Tencent's model also targets 2K-resolution generation and strong semantic adherence. Choose qwen-image-2 if you need superior Chinese text rendering and editing capabilities; choose Hunyuan Image 2.1 if you prefer Tencent's approach or have better integration with Tencent's ecosystem.

qwen-image-fast: This is an optimized distillation of Qwen-Image by Pruna AI, generating 1.5MP images in approximately 1 second. Choose qwen-image-2 when you prioritize output quality, text rendering precision, and editing capabilities; choose qwen-image-fast when you need latency below 1-2 seconds and can accept some quality reduction from aggressive optimization.

Technical specifications

The model is a 20-billion-parameter Multimodal Diffusion Transformer (MMDiT) architecture with native support for 2K resolution output. It integrates Qwen2.5-VL for semantic understanding and VAE encoding for reconstructive fidelity during editing tasks. The training pipeline includes large-scale data collection with filtering, annotation, and synthesis stages, followed by progressive curriculum learning that escalates from non-text rendering through simple text to complex paragraph-level descriptions.

Architecture and training:

20B parameter MMDiT model
Dual-encoding mechanism: Qwen2.5-VL for semantics + VAE encoder for visual reconstruction
Multi-task training: text-to-image (T2I), text-image-to-image (TI2I), and image-to-image (I2I) reconstruction
Progressive curriculum learning from simple to complex text inputs
Large-scale data pipeline with collection, filtering, annotation, and synthesis

Resolution and aspect ratio support:

Native 2K resolution support (1664×928, 928×1664, and other standard aspect ratios)
Supported aspect ratios: 1:1 (1328×1328), 16:9 (1664×928), 9:16 (928×1664), 4:3 (1472×1104), 3:4 (1104×1472), 3:2 (1584×1056), 2:3 (1056×1584)
Professional typography support up to 1k-token instructions

Inference and optimization:

Compatible with Diffusers library (requires version supporting Qwen2.5-VL)
Supports bfloat16 precision for reduced memory footprint
Compatible with acceleration frameworks: LightX2V (42.55x overall speedup reported), vLLM-Omni, SGLang-Diffusion, ComfyUI
Adjustable inference steps (default around 50 for quality output)
Configurable CFG scale (true_cfg_scale parameter, default 4.0)

Supported formats and libraries:

PyTorch model weights (Hugging Face and ModelScope formats)
Diffusers pipeline integration
PIL Image input/output
JPEG, PNG output formats

Model inputs and outputs

Inputs

prompt (string, required): Text description for image generation or editing. Supports up to 1k tokens. Use detailed, structured prompts for professional typography; simple natural language for general generation.
image (string, URI, optional): Reference image URL for image editing, style transfer, or image-to-image generation. Omit for pure text-to-image generation.
aspect_ratio (string, default: "1:1"): Aspect ratio selector. Allowed values: "1:1", "16:9", "9:16", "4:3", "3:4", "3:2", "2:3". Determines output dimensions; each ratio has fixed pixel dimensions (e.g., 16:9 = 1664×928).
match_input_image (boolean, default: false): When true and an image is provided, use the input image's aspect ratio and resolution instead of the aspect_ratio parameter. Useful for consistent editing across multiple operations.
negative_prompt (string, default: ""): Specify elements, styles, or qualities to avoid in generation. Leave empty for unconstrained generation.
enable_prompt_expansion (boolean, default: true): Automatically expand and optimize the prompt for better results. Set to false to use the prompt exactly as provided.
seed (integer, nullable, range: 0–2147483647): Random seed for reproducible generation. Omit or set to null for non-deterministic generation; same seed with identical prompts produces identical images across runs.

Outputs

Output (string, URI format): A single image URI pointing to the generated or edited image. Download or process this URL to retrieve the final image file. Format is typically JPEG or PNG depending on the server implementation.

Getting started

import replicate

# Text-to-image generation
output = replicate.run(
    "qwen/qwen-image-2",
    input={
        "prompt": "A serene mountain landscape with snow-capped peaks, crystal clear lake reflection, golden hour lighting, photorealistic, highly detailed",
        "aspect_ratio": "16:9",
        "negative_prompt": "blurry, low quality, distorted",
        "enable_prompt_expansion": True,
        "seed": 42
    }
)
print(output)  # Returns URI to generated image

# Image editing with text instruction
output_edit = replicate.run(
    "qwen/qwen-image-2",
    input={
        "prompt": "Change the sky to a vibrant sunset with orange and pink hues",
        "image": "https://example.com/landscape.jpg",
        "aspect_ratio": "16:9",
        "match_input_image": True,
        "negative_prompt": "oversaturated, unrealistic colors",
        "seed": 123
    }
)
print(output_edit)  # Returns URI to edited image

# Professional infographic generation with Chinese text
output_typo = replicate.run(
    "qwen/qwen-image-2",
    input={
        "prompt": "设计一张产品海报，标题为'智能生活2025'，包含产品图片和关键特性列表，蓝色主色调，现代简约风格，1080x1920分辨率",
        "aspect_ratio": "9:16",
        "enable_prompt_expansion": True,
        "seed": None  # Random seed for variety
    }
)
print(output_typo)  # Returns URI to generated poster

Frequently asked questions

Q: What image formats does the model accept as input for editing?

A: The model accepts image URLs in URI format. You should provide a publicly accessible URL pointing to a JPEG or PNG image; the Replicate API will fetch and process it internally.

Q: Can I use this model for commercial applications?

A: Yes. The model is licensed under Apache 2.0, which permits commercial use, modification, and distribution. You must include a copy of the license and provide attribution to Alibaba's Qwen team in your application or documentation.

Q: Why does my generated image sometimes have text that looks incorrect or garbled?

A: While text rendering is a core strength of qwen-image-2, extremely long text strings, densely packed layouts, or unusual character combinations may still produce errors. Try breaking longer text into multiple lines, using clear spacing, and keeping prompts under 1k tokens for best results. You can enable prompt expansion to let the model optimize your instruction.

Q: How does qwen-image-2 handle Chinese characters compared to other models?

A: The model includes explicit curriculum learning on logographic languages like Chinese and was trained with a large-scale data pipeline optimized for text rendering in non-Latin scripts. It significantly outperforms earlier models and most open-source alternatives on Chinese text generation and placement accuracy.

Q: What is the difference between setting match_input_image to true versus false?

A: When false (default), the model uses the aspect_ratio parameter to determine output dimensions. When true, the model ignores aspect_ratio and instead matches the input image's resolution and aspect ratio, ensuring consistent dimensions across editing operations on the same source image.

Q: Is the model still actively maintained and receiving updates?

A: Yes. The model has received multiple updates including Qwen-Image-2512 (December 2025) with improved human realism and texture quality, and Qwen-Image-Edit-2511 with better editing consistency. The Replicate version documented here reflects the latest public release.

Q: How does seed affect generation, and should I always set one?

A: The seed parameter ensures reproducibility: identical prompt, settings, and seed produce identical images. Set a seed when you need deterministic output (testing, reproducibility, batch processing); omit it or set it to null for variety in exploratory generation or user-facing applications.

Q: What are the actual pixel dimensions for each aspect ratio?

A: The model uses fixed-size presets: 1:1 is 1328×1328, 16:9 is 1664×928, 9:16 is 928×1664, 4:3 is 1472×1104, 3:4 is 1104×1472, 3:2 is 1584×1056, and 2:3 is 1056×1584. You cannot request custom arbitrary dimensions; you must choose from these ratios.

Click here to read the full guide to Qwen-Image-2