LTX Video 2 Quickstart¶

In this example, we'll train an LTX Video 2 LoRA using the LTX-2 video/audio VAEs and the matching Gemma text encoder for the selected LTX-2 version.

Hardware requirements¶

LTX Video 2 is a heavy 19B model. It combines: 1. Gemma3 / Gemma4: The text encoder. LTX-2.0 and LTX-2.3 use Gemma3; LTX-2.5 uses Gemma4. 2. LTX-2 Video VAE (plus the Audio VAE when conditioning on audio). 3. 19B Video Transformer: A large DiT backbone.

This setup is VRAM-intensive, and the VAE pre-caching step can spike memory usage.

• Note: The initial VAE pre-caching step can require more VRAM. You may need CPU offloading or a larger GPU just for the caching phase.
• Tip: Set "offload_during_startup": true in your config.json to ensure the VAE and text encoder are not loaded to the GPU at the same time, which significantly reduces pre-caching memory pressure.
• System RAM: 64GB+ is recommended for larger runs; more RAM helps with caching.

Observed performance and memory (field reports)¶

• Baseline settings: 480p, 17 frames, batch size 2 (minimal video length/resolution).
• RamTorch (incl. text encoder): ~13 GB VRAM used on an AMD 7900XTX.
• NVIDIA 3090/4090/5090+ should see similar or better VRAM headroom.
• No offload (int8 TorchAO): ~29-30 GB VRAM used; 32 GB hardware recommended.
• Peak system RAM: ~46 GB when loading the bf16 Gemma text encoder then quantizing to int8 (~32 GB VRAM).
• Peak system RAM: ~34 GB when loading bf16 LTX-2 transformer then quantizing to int8 (~30 GB VRAM).
• No offload (full bf16): ~48 GB VRAM required for model training without any offload enabled.
• Throughput:
• ~8 sec/step on A100-80G SXM4 (no compile).
• ~16 sec/step on 7900XTX (local run).
• ~30 min for 200 steps on A100-80G SXM4.

⚠️ Test regional compilation before keeping it enabled. dynamo_use_regional_compilation is not a universal speedup for LTX Video 2 and can reduce steady-state throughput even after compilation has warmed up. Compare eager and compiled runs with the same fixed batch, frame count, and resolution, and judge them by several post-warmup training steps rather than the initial compilation steps.

Prerequisites¶

Ensure Python 3.12 is installed.

python --version

Installation¶

pip install 'simpletuner[cuda]'

# CUDA 13 / Blackwell users (NVIDIA B-series GPUs)
pip install 'simpletuner[cuda13]' --extra-index-url https://download.pytorch.org/whl/cu130

See INSTALL.md for advanced installation options.

Setting up the environment¶

Web interface¶

simpletuner server

Manual configuration¶

Run the helper script:

simpletuner configure

Or copy the example and edit manually:

cp config/config.json.example config/config.json

Configuration parameters¶

Key settings for LTX Video 2:

• model_family: ltxvideo2
• model_flavour: dev (default), dev-fp4, dev-fp8, 2.3-dev, 2.3-distilled, 2.5-dev, or 2.5-distilled.
• pretrained_model_name_or_path: Lightricks/LTX-2, dg845/LTX-2.3-Diffusers, dg845/LTX-2.3-Distilled-Diffusers, Lightricks/LTX-2.5, or a local .safetensors file.
• train_batch_size: 1. Do not increase this unless you have an A100/H100.
• validation_resolution:
• 512x768 is a safe default for testing.
• 720x1280 (720p) is possible but heavy.
• validation_num_video_frames: Must be compatible with VAE compression (4x).
• For 5s (at ~12-24fps): Use 61 or 49.
• Formula: (frames - 1) % 4 == 0.
• validation_guidance: 5.0.
• ltx2_validation_audio_guidance: Optional separate audio CFG scale for validation. Leave unset to reuse validation_guidance for audio; for LTX-2.5 AV validation, ComfyUI's dual-CFG node defaults are video CFG 3.0 and audio CFG 7.0.
• ltx2_validation_pipeline_mode: Keep trained-stage for normal validation. Use spatial-upscale to run the LTX-2 two-stage spatial upscaler path: half-resolution latent generation, spatial latent upscaling, then full-resolution re-denoising.
• spatial-upscale requires the requested validation_resolution to be divisible by 64.
• Optional overrides: ltx2_validation_spatial_upsampler_model and ltx2_validation_spatial_upsampler_filename. Defaults are Lightricks/LTX-2.3 and ltx-2.3-spatial-upscaler-x2-1.1.safetensors.
• frame_rate: Default is 25.

LTX-2 2.0 flavours ship as a single .safetensors checkpoint that includes the transformer, video VAE, audio VAE, and vocoder. For LTX-2.3, SimpleTuner loads the matching Diffusers repo selected by model_flavour (2.3-dev or 2.3-distilled). For LTX-2.5, SimpleTuner assumes the Diffusers repo path is Lightricks/LTX-2.5 and routes the text encoder through Gemma4. If you load a local single-file LTX-2.5 checkpoint, keep model_flavour set to 2.5-dev or 2.5-distilled; SimpleTuner will use Lightricks/LTX-2.5 as the component/config source unless you override it with a local path.

Dual CFG for LTX-2 AV validation uses the normal conditional and unconditional predictions and applies separate scales to the video and audio latent streams. It does not add an extra model pass by itself. Extra guidance features such as STG, modality guidance, or reference-audio identity guidance each add their own additional pass when enabled.

Optional: VRAM optimizations¶

If you need more VRAM headroom: - Musubi block swap: Set musubi_blocks_to_swap (try 4-8) and optionally musubi_block_swap_device (default cpu) to stream the last transformer blocks from CPU. Expect lower throughput but lower peak VRAM. - VAE patch convolution: Set --vae_enable_patch_conv=true to enable temporal chunking in the LTX-2 VAE; expect a small speed hit but lower peak VRAM. - VAE temporal roll: Set --vae_enable_temporal_roll=true for more aggressive temporal chunking (larger speed hit). - VAE tiling: Set --vae_enable_tiling=true to tile VAE encode/decode for large resolutions.

Optional: CREPA temporal regularizer¶

To reduce flicker and keep subjects stable across frames: - In Training → Loss functions, enable CREPA. - Recommended starting values: Block Index = 8, Weight = 0.5, Adjacent Distance = 1, Temporal Decay = 1.0. - Keep the default vision encoder (dinov2_vitg14, size 518) unless you need a smaller one (dinov2_vits14 + 224). - Requires network (or a cached torch hub) to fetch DINOv2 weights the first time. - Only enable Drop VAE Encoder if you are training entirely from cached latents; otherwise leave it off.

Advanced Experimental Features¶

[
  {
    "id": "my-video-dataset",
    "type": "local",
    "dataset_type": "video",
    "instance_data_dir": "datasets/videos",
    "caption_strategy": "textfile",
    "resolution": 512,
    "video": {
        "num_frames": 61,
        "min_frames": 61,
        "frame_rate": 25,
        "bucket_strategy": "aspect_ratio"
    },
    "repeats": 10
  },
  {
    "id": "text-embeds",
    "type": "local",
    "dataset_type": "text_embeds",
    "default": true,
    "cache_dir": "cache/text/ltxvideo2",
    "disabled": false
  }
]

"audio": {
    "auto_split": true,
    "sample_rate": 16000,
    "channels": 1,
    "duration_interval": 3.0,
    "allow_zero_audio": false
}

mkdir -p datasets/videos
</details>

# Place .mp4 / .mov files here.
# Place corresponding .txt files with same filename for captions.

wandb login
huggingface-cli login

simpletuner train

{
  "tread_config": {
    "routes": [
      {
        "selection_ratio": 0.5,
        "start_layer_idx": 2,
        "end_layer_idx": -2
      }
    ]
  }
}

{
  "base_model_precision": "int8-quanto",
  "checkpoint_step_interval": 100,
  "data_backend_config": "config/ltx2/multidatabackend.json",
  "disable_benchmark": true,
  "dynamo_mode": "",
  "evaluation_type": "none",
  "hub_model_id": "simpletuner-ltxvideo2-19b-t2v-lora-test",
  "learning_rate": 0.00006,
  "lr_warmup_steps": 50,
  "lycoris_config": "config/lycoris_config.json",
  "max_grad_norm": 0.1,
  "max_train_steps": 200,
  "minimum_image_size": 0,
  "model_family": "ltxvideo2",
  "model_flavour": "dev",
  "model_type": "lora",
  "num_train_epochs": 0,
  "offload_during_startup": true,
  "optimizer": "adamw_bf16",
  "output_dir": "output/examples/ltxvideo2-19b-t2v.peft-lora",
  "override_dataset_config": true,
  "ramtorch": true,
  "ramtorch_text_encoder": true,
  "report_to": "none",
  "resolution": 480,
  "scheduled_sampling_reflexflow": false,
  "seed": 42,
  "skip_file_discovery": "",
  "tracker_project_name": "lora-training",
  "tracker_run_name": "example-training-run",
  "train_batch_size": 2,
  "vae_batch_size": 1,
  "vae_enable_patch_conv": true,
  "vae_enable_slicing": true,
  "vae_enable_temporal_roll": true,
  "vae_enable_tiling": true,
  "validation_disable": true,
  "validation_disable_unconditional": true,
  "validation_guidance": 5,
  "validation_num_inference_steps": 40,
  "validation_num_video_frames": 81,
  "validation_prompt": "🟫 is holding a sign that says hello world from ltxvideo2",
  "validation_resolution": "768x512",
  "validation_seed": 42,
  "validation_using_datasets": false
}

[
  {
    "id": "my-audio-dataset",
    "type": "local",
    "dataset_type": "audio",
    "instance_data_dir": "datasets/audio",
    "caption_strategy": "textfile",
    "audio": {
      "sample_rate": 16000,
      "channels": 2,
      "duration_interval": 3.0,
      "truncation_mode": "beginning"
    },
    "repeats": 10
  },
  {
    "id": "text-embeds",
    "type": "local",
    "dataset_type": "text_embeds",
    "default": true,
    "cache_dir": "cache/text/ltxvideo2",
    "disabled": false
  }
]

{
  "validation_adapter_config": [
    { "label": "canny", "path": "Lightricks/LTX-2-19b-IC-LoRA-Canny-Control:ltx-2-19b-ic-lora-canny-control.safetensors" },
    { "label": "pose", "path": "Lightricks/LTX-2-19b-IC-LoRA-Pose-Control:ltx-2-19b-ic-lora-pose-control.safetensors" }
  ]
}