Back to Articles Fine-tune video and image models at scale with NVIDIA NeMo Automodel and 🤗 Diffusers Enterprise + Article Published July 17, 2026 Upvote 6 Pranav Prashant Thombre pthombre Follow nvidia linnan wang linnanwang Follow nvidia Alexandros Koumparoulis akoumpa Follow nvidia Wenwen Gao wgao2021 Follow nvidia Sylendran Arunagiri Sylendran95 Follow nvidia Bernard Nguyen bernardwin Follow nvidia A joint post from NVIDIA and Hugging Face. Special thanks to Sayak Paul from Hugging Face for their contributions to the integration work and for co-authoring this blog. Diffusion models power some of the most exciting open-source releases of the last two years — such as FLUX.1-dev for text-to-image and Wan 2.1 and HunyuanVideo for text-to-video. The 🤗 Diffusers library has become the de facto home for these models, giving researchers and builders a single, consistent interface for inference, adaptation, and pipeline composition. In addition, training and fine-tuning diffusion models are also on the rise, requiring utilities that offer memory-efficient sharding, latent caching, multiresolution bucketing, and configurations that scale gracefully from one GPU to hundreds. To cater to these technical demands, we offer the NVIDIA NeMo Automodel open-source library. Today, we're highlighting the collaboration between NVIDIA and Hugging Face that brings production-grade, distributed diffusion training to any Diffusers-format model on the Hugging Face Hub — with no checkpoint conversion and no model rewrites for any new model. The integration is documented in the Diffusers training guide and is fully open source under Apache 2.0. Table of contents What is NeMo Automodel? Supported diffusion models What this collaboration unlocks A look at the fine-tuning workflow 1. Pre-encode the dataset 2. Launch training with the existing FLUX YAML 3. Generate from the fine-tuned checkpoint 4. Performance Other Finetuned/LoRA examples Try it today Coming next: Pythonic recipe APIs Resources What is NeMo Automodel? NeMo Automodel is an open-source PyTorch DTensor-native training library, part of the NVIDIA NeMo framework, built around two design principles that matter for the Diffusers ecosystem: Hugging Face native. Point pretrained_model_name_or_path at any Diffusers model ID on the Hub and start training. NeMo Automodel uses Diffusers model classes (e.g. WanTransformer3DModel) for loading and Diffusers pipelines (WanPipeline) for generation. Checkpoints round-trip cleanly back into the Diffusers ecosystem. One program, any scale. The recipes and training scripts can be easily modified to suit training at any scale. Parallelism is a configuration choice, not a code rewrite — switch between FSDP2, tensor parallel, expert parallel, context parallel, and pipeline parallel by declaring configurations, not rewriting models. AutoModel currently supports flow-matching models only. Under the hood, it uses flow matching as the training objective, with latent-space training (via pre-encoded VAE outputs) and multiresolution bucketed dataloading to accelerate throughput. Supported diffusion models NeMo Automodel integration ships with ready-to-use fine-tuning recipes for the open diffusion models below. The list reflects the recipes currently in examples/diffusion/finetune. Model Hugging Face ID Task Parameters LoRA recipe Wan 2.1 T2V 1.3B / 14B Wan-AI/Wan2.1-T2V-1.3B-Diffusers Wan-AI/Wan2.1-T2V-14B-Diffusers Text-to-Video 1.3B (fits on a single 40GB A100) / 14B Yes Wan 2.2 T2V A14B Wan-AI/Wan2.2-T2V-A14B-Diffusers Text-to-Video 27B total (MoE), 14B active per step No FLUX.1-dev black-forest-labs/FLUX.1-dev Text-to-Image 12B Yes FLUX.2-dev black-forest-labs/FLUX.2-dev Text-to-Image 32B Yes HunyuanVideo 1.5 hunyuanvideo-community/HunyuanVideo-1.5-Diffusers-720p_t2v Text-to-Video 13B Yes Qwen-Image Qwen/Qwen-Image Text-to-Image 20B (MMDiT) Yes What this collaboration unlocks For Diffusers users, the practical gains break down into a few concrete capabilities. No checkpoint conversion. Pretrained weights from the Hub work out of the box. There's no separate "training format" to convert to, then convert back. Your fine-tuned checkpoint loads directly into a DiffusionPipeline for inference, or back to the Hub for sharing. Downstream tools — quantization, compilation, LoRA adapters, custom samplers — all keep working. Fast path to new model support. When a new diffusion model lands in Diffusers, enabling it in NeMo Automodel takes a small, contained code addition — a data preprocessing handler and a model adapter — rather than a full custom training script. The rest of the recipe stack (FSDP2, bucketed dataloading, checkpointing, generation) carries over unchanged, and the same YAML-driven workflow applies. Full and parameter-efficient fine-tuning. Both full fine-tuning and LoRA-style PEFT are supported, so you can choose between maximum quality (full FT on a large cluster) or maximum efficiency (LoRA on a single node). The same recipe structure handles both. Scalable training that goes beyond what built-in scripts offer. NeMo Automodel adds sharding schemes such as FSDP2, tensor, context, and pipeline parallelisms, multi-node orchestration (SLURM today, Kubernetes coming), and multiresolution bucketing. These capabilities make training larger models like FLUX.1-dev (12B) and HunyuanVideo (13B) possible. A look at the fine-tuning workflow In this section, we walk through the typical workflow for fine-tuning any of the supported models. The recommended way to install Automodel is the NeMo Automodel Docker container (nvcr.io/nvidia/nemo-automodel:26.06), which ships with PyTorch, TransformerEngine, and other CUDA-compiled dependencies pre-built. Alternatively, install with pip3 install nemo-automodel or from source (pip3 install git+https://github.com/NVIDIA-NeMo/Automodel.git); see the installation guide for all options. This guide walks through a full-transformer fine-tune of FLUX.1-dev on the 78-card Rider–Waite tarot dataset, then generating from the resulting checkpoint. It reuses the checked-in YAML configs and applies run-specific settings as command-line overrides, so no new config files are required. 1. Pre-encode the dataset The diffusion recipe consumes cached VAE latents and text embeddings instead of encoding source images during every training step. Stream the 78 Rider–Waite images directly from Hugging Face and distribute preprocessing across all visible GPUs:

uv run --locked --no-default-groups \ --extra diffusion \ --extra diffusion-media \ python -m tools.diffusion.preprocessing_multiprocess image \ --dataset_name multimodalart/1920-raider-waite-tarot-public-domain \ --dataset_media_column image \ --dataset_caption_column caption \ --dataset_streaming \ --max_images 78 \ --output_dir /cache/flux_tarot \ --processor flux \ --model_name black-forest-labs/FLUX.1-dev \ --max_pixels 245760

The captions already contain the trtcrd trigger token. With this pixel budget and the dataset's portrait aspect ratio, preprocessing assigns the samples to the 384×640 bucket used by the showcase run. For image training, preprocessing produces .pt cache files and sharded metadata:

/cache/flux_tarot/ ├── 384x640/ │ ├── .pt │ └── ... ├── metadata_shard_0000.json ├── metadata.json └── _hf_dataset/ └── images/

2. Launch training with the existing FLUX YAML Use examples/diffusion/finetune/flux_t2i_flow.yaml directly. The YAML already selects FLUX.1-dev, full transformer fine-tuning, the FLUX flow-matching adapter, an effective batch size of 32, and eight-way FSDP2. Supply the tarot-specific paths and settings as command-line overrides:

uv run --locked --no-default-groups --extra diffusion \ torchrun --nproc-per-node=8 \ examples/diffusion/finetune/finetune.py \ -c examples/diffusion/finetune/flux_t2i_flow.yaml \ --model.transformer_engine_fp8 false \ --data.dataloader.cache_dir /cache/flux_tarot \ --data.dataloader.base_resolution '[384,640]' \ --lr_scheduler.lr_decay_style constant \ --lr_scheduler.lr_warmup_steps 20 \ --step_scheduler.max_steps 200 \ --step_scheduler.ckpt_every_steps 50 \ --checkpoint.checkpoint_dir /tmp/flux_tarot/checkpoints/full \ --checkpoint.save_consolidated true \ --seed 2026

The run produces checkpoints at steps 50, 100, 150, and 200. The final checkpoint is labeled epoch_66_step_199; the label is zero-based even though it represents the completed 200th optimizer step. 3. Generate from the fine-tuned checkpoint Use the existing FLUX generation YAML and point model.checkpoint at the complete training checkpoint:

uv run --locked --no-default-groups --extra diffusion \ python examples/diffusion/generate/generate.py \ -c examples/diffusion/generate/configs/generate_flux.yaml \ --model.checkpoint /tmp/flux_tarot/checkpoints/full/epoch_66_step_199 \ --inference.height 640 \ --inference.width 384 \ --inference.prompts '["a trtcrd of an astronaut tending a rose garden on Mars, \"the gardener\""]' \ --output.output_dir /tmp/flux_tarot/generations/full/step_200 \ --seed 2026

Include trtcrd to invoke the learned tarot style. For a control comparison, keep the seed and scene fixed but omit the trigger:

uv run --locked --no-default-groups --extra diffusion \ python examples/diffusion/generate/generate.py \ -c examples/diffusion/generate/configs/generate_flux.yaml \ --model.checkpoint /tmp/flux_tarot/checkpoints/full/epoch_66_step_199 \ --inference.height 640 \ --inference.width 384 \ --inference.prompts '["an astronaut tending a rose garden on Mars, \"the gardener\""]' \ --output.output_dir /tmp/flux_tarot/generations/control \ --seed 2026

Results At step 200, the triggered astronaut prompts retain their requested content while acquiring a cream, red, and black vintage palette, heavy ink contours, flat color fields, aged-paper tones, and allegorical card composition. The untriggered astronaut remains photographic, demonstrating that the learned effect is substantially associated with trtcrd rather than replacing the base model globally. Prompt (seed) Baseline Fine-tuned (step 200) Astronaut tending a rose garden on Mars (seed 2026) Astronaut in a moonlit garden on Mars (seed 2028) Astronaut planting a sapling beneath two moons (seed 2029) 4. Performance All measurements were collected on one node with 8× NVIDIA H100 80GB GPUs. Results are means ± sample standard deviation over three steady-state 10-step windows. Text-to-image — 512×512(see if seconds can be aligned) Model Training Parallelism GBS / LBS Step time Images/s Images/s/GPU Peak allocated/GPU FLUX.1-dev Full FSDP2 32 / 4 0.902 ± 0.039 s 35.51 ± 1.55 4.44 ± 0.19 63.88 GiB FLUX.1-dev LoRA r64 DDP 48 / 6 0.894 ± 0.008 s 53.73 ± 0.48 6.72 ± 0.06 67.43 GiB Qwen-Image Full FSDP2 40 / 5 0.974 ± 0.075 s 41.21 ± 3.06 5.15 ± 0.38 53.55 GiB Qwen-Image LoRA r64 DDP 24 / 3 0.515 ± 0.006 s 46.63 ± 0.54 5.83 ± 0.07 66.33 GiB Text-to-video — 512×512×49 frames Each sample is one 49-frame video clip. Model Training GBS / LBS Activation checkpointing Step time Clips/s Clips/s/GPU Peak allocated/GPU Wan 2.1 1.3B Full 8 / 1 Off 0.942 ± 0.038 s 8.50 ± 0.35 1.06 ± 0.04 6.09 GiB Wan 2.1 14B Full 8 / 1 On 3.798 ± 0.017 s 2.107 ± 0.006 0.263 ± 0.006 33.35 GiB Wan 2.1 14B LoRA r64 16 / 2 On 7.585 ± 0.014 s 2.110 ± 0.000 0.263 ± 0.000 24.07 GiB Wan 2.2 A14B, high-noise Full 8 / 1 On 4.628 ± 0.031 s 1.730 ± 0.010 0.217 ± 0.006 23.57 GiB HunyuanVideo 1.5 Full 8 / 1 On 5.926 ± 0.046 s 1.350 ± 0.010 0.170 ± 0.000 15.90 GiB HunyuanVideo 1.5 LoRA r64 8 / 1 On 5.575 ± 0.006 s 1.433 ± 0.006 0.180 ± 0.000 10.58 GiB Measurement details Hardware: 8× H100 80GB HBM3, fully NVLink-connected. Image dataset: lambda/naruto-blip-captions, 256 cached samples. Video dataset: svjack/Lelouch_Vi_Britannia_FramePack_First_Last_Frame_Video_Captioned, 112 cached samples. Full batches were enforced with drop_last=true. Checkpoint writes were disabled. Step time includes dataloading, forward, backward, gradient clipping, optimizer, and scheduler work. Memory is peak PyTorch CUDA allocator usage, not total NVML device memory. Other Finetuned/LoRA examples The results from fine-tuning and LoRA showcase the power of NeMo Automodel for domain specialization. For instance, fine-tuning the Wan 2.1 model on a Ghibli video dataset successfully adapted the output style, demonstrated by a noticeable change in a flower's appearance compared to the baseline. Baseline: Finetuned on Ghibli's videos: We also observed the distinct impact of using LoRA, where applying the adapter to Wan 2.1 caused the video to adopt a characteristic Ghibli style, particularly visible in the highlighting of characters' eyes. No LoRA: LoRA: These examples, including those for FLUX.2, confirm that users can achieve both maximum quality via full fine-tuning and maximum efficiency via LoRA-style PEFT, tailoring the output to specific stylistic domains. Try it today Learn more about the integration and find more fine-tuning examples in the NeMo Automodel documentation Coming next: Pythonic recipe APIs YAML is a strong fit for reproducible configuration, especially for teams that want files they can check in, review, and reuse, but many teams also need a programmatic interface. In an upcoming NeMo Automodel release, we plan to surface the diffusion recipes through a fully typed Pythonic API as well. Users will be able to compose the same model, data, optimizer, PEFT/LoRA, parallelism, checkpointing, and generation pieces directly from Python. The Pythonic path is intended to make the recipes easier to use from existing training code, notebooks, and experiment workflows, and to offer a first-class Pythonic interface alongside the YAML quick-start path. Resources NeMo Automodel GitHub Diffusion Fine-Tuning Guide Diffusion Dataset Preparation NeMo Automodel for Transformers (LLM/VLM fine-tuning) Models mentioned in this article 6 Datasets mentioned in this article 3 More from this author NVIDIA Nemotron 3 Embed Ranks #1 Overall on RTEB, Advancing Agentic Retrieval 35 July 16, 2026 Data for Agents 22 July 8, 2026 Community EditPreview Upload images, audio, and videos by dragging in the text input, pasting, or clicking here. Tap or paste here to upload images Comment · Sign up or log in to comment Upvote 6 Models mentioned in this article 6 Datasets mentioned in this article 3

🔥 인기 글 랭킹

Masked Diffusion Language Models are Strong and Steerable Text-Based World Models for Agentic RL

RAIL Guard: Closing the Evaluation-to-Remediation Gap in Responsible AI for LLM Agents

SelKV: Selective KV Cache Merging with Per-Token Merge-or-Drop and Attention Compensation

Some Large Language Models Exhibit Consistent Risk Attitudes

PlanFlip: Attacking Multi-Agent LLM Systems via Planning-Phase Prompt Injection

Rater State Bias in RLHF Preference Data: An Audit Framework

Democratizing AI with Small Language Models: Structured Benchmarking and Parameter-Efficient Fine-Tuning for Local Deployment

Design and Validation of a Lightweight 1D CNN for Affective Touch Classification in Soft Plush Companions

Deterministic Replay for AI Agent Systems

A Survey on GNN-based Link Prediction: Techniques, Applications, and Challenges

Accurate and Efficient Long-Term Memory for LLM Agents

Generative Ontology Induction: Domain-Agnostic Schema Discovery from Document Corpora Using Large Language Models

It Takes 8 Tokens: Weak-to-Strong Off-Policy RL via Auxiliary Branches

PPO-HSC: An Exploratory Reinforcement Learning Framework Based on Wide-Area Policy Coverage Optimization

Symbolic Augmentation Closes a Canonical-Equivalence Blind Spot in Neural Fact-Checkers

ColGraphRAG: Late-Interaction Evidence Retrieval for Multimodal GraphRAG

JUMP: Single-Pass Membership Inference on Fine-Tuned Diffusion Language Models

Generalist AI Control: Towards Multi-purpose Adaptive Algorithms

A Survey on the Verification of Reinforcement Learning Policies

Shapley Context Pruning: A Cooperative Game Perspective for Context Reranking and Pruning

China’s AI models have Trump’s AI world at war with itself

Introducing Cosmos 3 Edge

AI is more likely than humans to form biases when hiring

Logic, Optimization, and Artificial Intelligence

MGDT: MLLM-Guided Diffusion Transformer with Relation-Adaptive Mixture-of-Experts for Multimodal Knowledge Graph Completion

AnovaX: A Local, Multi-Agent Voice Assistant with LLM Planning, Typed Executors, and Adaptive Recovery

Do Coding Agents Need Executable World Models, Simplification, and Verification to Solve ARC-AGI-3?

Causal-Audit: Explicit and Auditable Graph-based Reasoning via Target-Aware Causal Chain Construction

A Critical Analysis of Trustworthy AI Tools, Mark Frameworks, and the Implementation Chasms

Knowledge-Centric Agents for Workflow Generation

Beyond a Joke: Multi-Angle Reasoning for Detecting and Explaining Harmful Humor in Memes

NeurOWL: An LLM-Based Neural-symbolic Framework for Incomplete OWL Ontology Reasoning

Behavioral Controllability of Agentic Models for Information Extraction: From Fixed Workflows to Reflective Agents

SeerGuard: A Safety Framework for Mobile GUI Agents via World Model Prediction

GraphDx: A Cost-Aware Knowledge-Enhanced Multi-Agent Framework for Sequential Diagnosis

Precise but Uncoupled: Reviewer Precision Does Not Guarantee Critique Uptake in Multi-Agent Math Reasoning

DrawingVQA: A Real-World Benchmark for Multi-Depth Visual-Textual Reasoning on Construction Drawings

From Black Box to Executable Logic: Explainable Reinforcement Learning through Prolog Expert Systems

Cura 1T: Specialized Model for Agentic Healthcare

AgentFAIR: A Multi-Agent Collaborative Framework for FAIRness Evaluation of Geospatial Datasets

ToolVerse: Unlocking Massive Environments and Long-Horizon Tasks for Agentic Reinforcement Learning

S1-Omni: A Unified Multimodal Reasoning Model for Scientific Understanding, Prediction, and Generation

Neuro-Symbolic AI for LEED compliance: Document-Centric Benchmarking, Deterministic Numeric Checking, and When Multimodal Hurts

Fine-tune video and image models at scale with NVIDIA NeMo Automodel and 🤗 Diffusers

The risk of weather data sabotage is rising

NVIDIA Nemotron 3 Embed Ranks #1 Overall on RTEB, Advancing Agentic Retrieval

Newer Models, Same Advantage

AI-Native Insurance for Agentic AI: Pricing, Underwriting, and End-to-End Automation

LAPO: Leave-One-Turn Attribution for Self-Generated Process Rewards in Multi-Turn Search Reasoning

Oracle Agent Memory as an Enterprise Memory Substrate for Long-Horizon AI Agents