ERNIE-Image

Z-Image LoRA Training with DiffSynth-Studio: Dataset, Parameters, and Limits

Apr 28, 2026

Z-Image LoRA Training Hero

Abstract: LoRA is a parameter-efficient adaptation method for diffusion models. This guide covers dataset preparation, the official DiffSynth-Studio training entry point, and checkpoint evaluation. Settings depend on the chosen dataset and runtime, and Turbo training has additional acceleration constraints.

Published: 2026-04-28
Reading time: about 10 minutes
Difficulty: Intermediate

This is an external training workflow. The ERNIE-Image image generator can be used for ordinary image generation, but this site does not provide hosted Z-Image LoRA training or promise that a trained adapter can be loaded here.

1. What Is LoRA? Why Does Z-Image Need It

LoRA (Low-Rank Adaptation) adds trainable low-rank adapter parameters while the base model remains the reference model. It can reduce the number of parameters updated during adaptation, but it is not equivalent to full fine-tuning and does not guarantee a particular file size, quality level, or VRAM requirement.

The official Z-Image repository describes Z-Image as the foundation model intended for fine-tuning and downstream development. Its model table lists Z-Image-Turbo as a distilled model with fine-tunability marked N/A.

1.1 Base and Turbo are different training choices

DiffSynth-Studio's official Z-Image model documentation explains that direct training of Z-Image-Turbo can quickly make it lose its acceleration capability. If Turbo is trained with the standard SFT path, inference may need to use the non-accelerated path or one of the documented differential/distillation approaches.

For a first adaptation, use the base Z-Image example and evaluate the adapter before considering Turbo-specific workflows. Do not describe Turbo as retaining its original acceleration after arbitrary LoRA training.

1.2 What LoRA can and cannot promise

An adapter can help a model learn a subject, style, or recurring visual pattern, but results depend on the data, captions, target modules, training schedule, and inference weight. A successful training run is not proof of identity consistency, generalization, or compatibility with every Z-Image checkpoint.

2. Dataset Preparation: Selection, Cropping, and Captioning

Dataset quality is part of the experiment. Keep the images relevant to the intended subject or style, remove accidental duplicates and unrelated watermarks, and keep the captions consistent with the content you want the model to learn.

2.1 Image selection

There is no official universal image count, resolution, crop ratio, or subject-occupancy threshold for every LoRA dataset. Select enough varied examples to represent the target while keeping the subject or style identifiable. Hold out a few images or prompts for evaluation where possible, and record how the set was assembled.

2.2 Preprocessing

Use the dimensions accepted by the training script and make the preprocessing reproducible. Do not turn a sample value in a script into a site-wide recommendation. Check aspect ratio, image decoding, captions, and metadata before starting the run.

2.3 Captions and trigger words

Captions should describe the visual content that should remain controllable. A trigger token can be used when the chosen workflow supports it, but its spelling and use must be consistent between the dataset and evaluation prompts. Compare captions that include and omit optional attributes rather than assuming that one captioning recipe works for every subject or style.

Regularization data and captioning strategies are optional design choices. If they are used, document their source and the reason for including them; the official example does not establish a universal regularization ratio or weight.

3. Training Parameters and the Official Example

The values in an example script are starting points for that script, not guaranteed recommendations. Learning rate, epochs, rank, target modules, resolution, repetition, and worker count should be reviewed against the dataset and hardware.

The official DiffSynth-Studio example is Z-Image.sh at commit df3333283609ff06a5a7b6809df9c9ae73ceb7a6. It invokes the repository's real training entry point, examples/z_image/model_training/train.py:

modelscope download --dataset DiffSynth-Studio/diffsynth_example_dataset --include "z_image/Z-Image/*" --local_dir ./data/diffsynth_example_dataset

accelerate launch examples/z_image/model_training/train.py \
  --dataset_base_path data/diffsynth_example_dataset/z_image/Z-Image \
  --dataset_metadata_path data/diffsynth_example_dataset/z_image/Z-Image/metadata.csv \
  --max_pixels 1048576 \
  --dataset_repeat 50 \
  --model_id_with_origin_paths "Tongyi-MAI/Z-Image:transformer/*.safetensors,Tongyi-MAI/Z-Image-Turbo:text_encoder/*.safetensors,Tongyi-MAI/Z-Image-Turbo:vae/diffusion_pytorch_model.safetensors" \
  --learning_rate 1e-4 \
  --num_epochs 5 \
  --remove_prefix_in_ckpt "pipe.dit." \
  --output_path "./models/train/Z-Image_lora" \
  --lora_base_model "dit" \
  --lora_target_modules "to_q,to_k,to_v,to_out.0,w1,w2,w3" \
  --lora_rank 32 \
  --use_gradient_checkpointing \
  --dataset_num_workers 8

This example downloads a public dataset and model files. Review the download sizes and the requirements of the chosen runtime before executing it. Read the current train.py argument definitions before changing the example.

The example uses a particular dataset path, model-origin mapping, target-module list, and starting values. Keep those values tied to the cited repository revision. They are not a promise that the same settings fit another dataset or GPU.

4. Optimizer and Memory Choices

Choose an optimizer and memory strategy from the options exposed by the exact DiffSynth-Studio revision, then compare runs with the same dataset, seed, precision, and evaluation prompts.

Gradient checkpointing, worker count, offload, and batch behavior change the memory and time profile. Record them with the command. Do not report a generic percentage improvement or hardware threshold without a measurement log.

5. Training Workflow: Step by Step

Step 1: Prepare and inspect the dataset

Use the dataset layout expected by the cited DiffSynth-Studio example. Verify that every metadata entry resolves to the intended image and caption, and inspect a sample after any crop or resize operation.

Step 2: Review the command for the target model

Start from the official script and change one group of settings at a time. Confirm whether the run targets base Z-Image or Turbo, and keep the output directory tied to the model revision and dataset revision.

Step 3: Launch the official entry point

Use accelerate launch .../train.py as shown above.

Step 4: Monitor the run

Track loss, checkpoint creation, memory use, interruptions, and the exact command. A falling loss alone does not prove that the adapter generalizes or that the base model's intended behavior is preserved.

Step 5: Evaluate checkpoints

Use the same prompts and inference settings for each checkpoint, then add prompts that differ from the training examples. Check subject or style consistency, prompt controllability, artifacts, and behavior when the adapter is disabled. Keep the best checkpoint based on the intended task rather than on a single loss value.

6. Quality Monitoring and Evaluation

Evaluate at least these dimensions:

  • Subject or style consistency: the learned concept should appear when requested.
  • Prompt controllability: scene, pose, lighting, and other non-target attributes should remain changeable.
  • Artifacts: inspect repeated textures, distorted details, color blocks, and anatomy errors.
  • Generalization: use scenes and compositions that are absent from the training set.
  • Base-model behavior: compare with the adapter disabled and check that the inference model matches the training target.

Keep the prompt set, seed, inference steps, adapter weight, model revision, and checkpoint path in the evaluation record. Avoid turning one successful sample into a universal quality claim.

7. Common Troubleshooting

Overfitting

Symptoms include outputs that repeat the training composition, reduced prompt control, or artifacts that become stronger in later checkpoints. Review captions and dataset diversity, compare earlier checkpoints, and adjust one training parameter at a time.

Underfitting

If the adapter has little visible effect, verify that the correct adapter is loaded, the trigger or caption convention matches, the target modules are supported, and the checkpoint was written completely. Only then consider changing the schedule or capacity.

Turbo acceleration changes

For Z-Image-Turbo, direct training can remove the original distilled acceleration. Follow the official Turbo guidance and test the documented inference path after training. Do not promise the original Turbo speed for a custom adapter.

Loading or memory errors

Check the model-origin paths, companion components, framework revision, target modules, and device backend. Reduce the workload only after recording the original command and error. A memory workaround for one device is not a compatibility guarantee for another.

Summary

  1. Use the official DiffSynth-Studio train.py entry point and keep the example tied to its repository revision.
  2. Treat the example values as starting points, not universal settings or hardware promises.
  3. Use base Z-Image for the clearest fine-tuning path; direct Turbo training may sacrifice its acceleration.
  4. Evaluate checkpoints with fixed prompts plus held-out prompts, and record the complete runtime context.
  5. This site provides image generation links for the resulting workflow, not hosted Z-Image LoRA training.

References

ERNIE-Image Team

Z-Image LoRA Training with DiffSynth-Studio: Dataset, Parameters, and Limits | Blog