ERNIE-Image

ERNIE-Image Multi-LoRA Fusion Complete Guide: The Art of Combining Styles, Characters, and Effects

Jul 29, 2026

ERNIE-Image Multi-LoRA Fusion Complete Guide: The Art of Combining Styles, Characters, and Effects

Published: 2026-07-29
Tags: ERNIE-Image, LoRA Fusion, Multi-LoRA, ComfyUI, TIES, DARE, SLERP

From Single LoRA to Multi-LoRA

In July 2026, CivitAI officially added ERNIE-Image to its native LoRA training ecosystem. Within weeks, hundreds of community LoRAs emerged — from realistic portraits to anime styles, brand visuals to specific concepts. The barrier to training a LoRA has dropped to 1000 Buzz (~$1) and 30 images.

But the real challenge isn't training — it's combining.

A great image often requires precise control of multiple elements simultaneously: a character's appearance, the atmosphere, and a specific style filter. A single LoRA can only solve one problem. Multi-LoRA composition — stacking style, character, and effect LoRAs together — is the key to unlocking ERNIE-Image's full potential.

The core challenge lies in understanding ERNIE-Image's DiT (Diffusion Transformer) architecture and ComfyUI's three modes of multi-LoRA combination.

How LoRA Works on DiT Architecture

ERNIE-Image uses a single-stream DiT architecture, sharing a similar diffusion transformer design with FLUX and SD3.5. LoRA acts on the Transformer blocks within the model — self-attention, cross-attention, and feed-forward layers.

Each LoRA is essentially a lightweight correction matrix applied to the original weights:

W' = W + α · BA

Where A and B are low-rank matrices and α is the scaling coefficient. When combining multiple LoRAs, these corrections can be chained or linearly merged. The key lies in proper weight allocation and conflict avoidance.

DiT Block Structure

ERNIE-Image's DiT consists of multiple Transformer blocks, each containing:

  • Self-attention: The image's self-attention mechanism (q, k, v projections)
  • Cross-attention: Cross-attention between text prompt and image
  • Feed-forward (MLP): Feed-forward network (w1, w2, w3)

Different LoRA types affect different blocks: style LoRAs predominantly influence self-attention and MLP layers, while character LoRAs depend more on cross-attention prompt response.

Three Modes of Multi-LoRA Composition

Mode 1: Chain Connection (Simplest)

ComfyUI's native approach. Chain Load LoRA nodes end-to-end:

Checkpoint → Load LoRA (Style) → Load LoRA (Character) → Load LoRA (Effect) → Sampler

Each LoRA node has two core parameters:

  • strength_model (-100 to 100): Controls visual feature strength — details, texture, lighting
  • strength_clip (-100 to 100): Controls prompt/trigger word response strength

Recommended Starting Points:

LoRA Type strength_model strength_clip
Character 0.8-1.0 0.6-0.8
Style 0.7-0.9 0.7-0.9
Concept 0.6-0.8 0.8-1.0

Weight Allocation Strategies:

  • Two LoRAs: Set each at 0.5-0.6 for balanced effects
  • Three+ LoRAs: Primary at ~0.8, secondary at ~0.3-0.5
  • Character + Style: Character 0.8-1.0, Style 0.5-0.7

The advantage of chaining is simplicity — no extra nodes needed. The downside is that weight balancing becomes difficult beyond 3 LoRAs, and effects can conflict.

Mode 2: Linear Merging (ModelMergeSimple)

ComfyUI's built-in ModelMergeSimple node can proportionally merge multiple LoRA weights into one. Unlike chaining, this method avoids exponential weight decay:

merged_weight = w1 * LoRA_1 + w2 * LoRA_2 + w3 * LoRA_3
# where w1 + w2 + w3 = 1.0

Practical tip: Use the Z-Image LoRA Merger community node (compatible with ERNIE-Image), supporting multiple normalization modes:

  • normalize: Normalize by target_strength², preserving proportions
  • average: Equal weight division, simple but imprecise
  • sqrt_scale: Scale by 1/√n, mathematically sound for independent effects
  • linear_decay: Full weight for primary LoRA, decreasing progressively (1, 1/2, 1/3...)

Mode 3: Advanced Merge Algorithms (LoRA Power-Merger)

For users seeking optimal results, the community provides LoRA Power-Merger (ComfyUI custom node) supporting 8+ advanced merge algorithms:

TIES-Merging (Recommended)
From the paper "TIES-Merging: Resolving Interference When Merging Models." Core idea: only keep parameter changes where LoRAs agree, discarding conflicting parts.

  • Parameters: density 0.7-0.9
  • Best for: Combining stylistically different LoRAs
  • Advantage: Reduces conflicts, preserves core features

DARE
From the paper "Language Models are Super Mario." Randomly drops most delta parameters, then scales remaining ones.

  • Parameters: density 0.9 (discards 90% of delta parameters)
  • Best for: Same-type LoRAs (e.g., two character LoRAs)
  • Advantage: Drastically reduces conflicts, good for same-domain merging

SLERP (Spherical Linear Interpolation)
Interpolates weights on a sphere, preserving geometric structure.

  • Parameters: t 0-1 (0 favors first LoRA, 1 favors second)
  • Best for: Two-LoRA scenarios
  • Advantage: Smooth transitions, ideal for style transfer

Practical Case Study: Creating a Custom Portrait Style

Scenario

Goal: Generate a character portrait with a specific artistic style. We need:

  1. A realistic character LoRA (for facial features)
  2. A watercolor style LoRA (for visual texture)
  3. A lighting effect LoRA (for mood)

Step 1: Test Each LoRA Individually

Before combining, fix seed and prompt, test each LoRA's individual effect. Record each LoRA's optimal strength range.

Fixed parameters:
  seed: 42
  prompt: "portrait of a woman, soft lighting, detailed face"
  steps: 50
  cfg: 4.0
  ERNIE-Image base model

Test results:

  • Character LoRA: face features sharpest at 0.8
  • Watercolor LoRA: optimal texture at 0.7, oversaturated above 0.9
  • Lighting LoRA: effective at 0.5 without distortion

Step 2: Chain Combination

Chain in order: "Character → Style → Effect":

Character LoRA strength_model=0.8, strength_clip=0.7
  → Watercolor LoRA strength_model=0.7, strength_clip=0.7
    → Lighting LoRA strength_model=0.5, strength_clip=0.6

Step 3: Fine-tune Weights

If a feature is too dominant (e.g., watercolor completely masks the character), two adjustment options:

  1. Lower that LoRA's strength_model (0.7 → 0.5)
  2. Raise character LoRA's strength_clip (0.7 → 0.9)

Rule of thumb: If the character's face is being over-stylized, adjust strength_clip before strength_model. strength_clip controls prompt responsiveness — raising it helps trigger words "pierce through" style layers more effectively.

Step 4: Merge Optimization (Advanced)

For better consistency, try TIES merging:

  1. Load three LoRAs via PM LoRA Power Stacker
  2. PM TIES node with density=0.8
  3. PM LoRA Merger using TIES method
  4. Apply merged LoRA to model

Common Issues and Solutions

Issue 1: No effect after stacking

Cause: Missing trigger word or insufficient strength_clip.

Solution:

  • Confirm the LoRA requires a trigger word (check model card)
  • Set strength_clip to 0.8-1.0
  • Verify LoRA compatibility with the base model

Issue 2: Face distortion or feature loss

Cause: Auxiliary LoRAs overpower the character LoRA.

Solution:

  • Place character LoRA first in the chain
  • Lower auxiliary LoRA strength_model (below 0.5)
  • Test character LoRA alone first, then add auxiliary LoRAs gradually

Issue 3: Oversaturation or style conflict

Cause: Two LoRAs affect the same parameters simultaneously.

Solution:

  • Use DARE merging to discard conflicting deltas
  • Or use PM LoRA Modifier for block-level control:
    • Style LoRA → limit to attention layers
    • Character LoRA → limit to MLP layers

Issue 4: Diminishing returns in chain

Cause: Chaining is essentially "matryoshka" stacking — distant LoRAs get diluted.

Solution:

  • Use ModelMergeSimple instead of chaining
  • Or use LoRA Power-Merger's SLERP for consistency
  • Limit to 4 or fewer LoRAs

ERNIE-Image DiT-Specific Considerations

Differences from SDXL/SD1.5

ERNIE-Image's DiT architecture differs fundamentally from SDXL's UNet:

  • Different block naming: DiT uses diffusion_model.blocks.N.self_attn.q instead of SDXL's input_blocks.N.1. Block-level control tools from the SDXL era need adjusted layer mapping.
  • CFG differences: ERNIE-Image Base recommends CFG=4.0 (50 steps), Turbo recommends CFG=1.0 (8 steps). In multi-LoRA scenarios, fine-tuning CFG can alleviate LoRA conflicts.
  • VAE compatibility: ERNIE-Image uses FLUX's VAE (flux2-vae.safetensors), incompatible with SDXL. Cross-architecture LoRAs cannot be mixed.

GGUF Quantization

GGUF-quantized ERNIE-Image fully supports multi-LoRA:

# GGUF Multi-LoRA Setup
base_model: ernie-image-turbo-Q8_0.gguf
lora_1: character_style.safetensors (strength: 0.8)
lora_2: lighting_effect.safetensors (strength: 0.4)
steps: 8 (Turbo) or 50 (Base)

Quantization precision loss has minimal impact on multi-LoRA composition, since LoRA operates as a low-rank differential — orthogonal to the model's quantization precision.

Summary

ERNIE-Image multi-LoRA fusion is a "composition art" — not simple numerical stacking, but understanding each LoRA's characteristics, their interactions, and DiT architecture's peculiarities.

Three core recommendations:

  1. Test alone, then combine: Always confirm each LoRA's optimal individual parameters first
  2. Less is more: Best results with 3 LoRAs or fewer; use merge algorithms beyond 4
  3. DiT is different: ERNIE-Image's block structure diverges from SDXL — adjust block-level control accordingly

CivitAI's ERNIE-Image LoRA ecosystem is growing rapidly. Mastering multi-LoRA composition means you are no longer limited by a single LoRA's capability boundary — you can freely combine styles, characters, and effects in a modular fashion, creating a truly unique visual language.

References

ERNIE-Image Team