fix for multi gpu training #247

ddPn08 · 2023-03-02T15:25:32Z

Fixes for multi-GPU training with train_network.py.
I haven't fully tested it yet, so I'm submitting it as a draft.
Open it when everything is tested.

It would be helpful if you could let me know if you have noticed anything.

train_network.py のマルチGPUトレーニングの修正です。
まだ完全にテストしていないのでドラフトです。正常に動作したら開きます。

何かお気づきの点ありましたら教えていただけると助かります。

kohya-ss · 2023-03-02T22:51:42Z

Thank you for this! I forgot is_main_process everywhere... It is hard to find time to prepare a multi-gpu environment, but I will check the script.

ありがとうございます。is_main_process が必要でしたね……。マルチGPU環境を用意してテストする時間がなかなか取れないのですが、スクリプトについて私も確認します。

Panchovix · 2023-03-04T03:22:01Z

I have 2 4090s, so I can test this to train a LoRA. Have I to add a parameter to the training, or do I have to select multi-gpu in accelerate config?

ddPn08 · 2023-03-04T05:39:06Z

@Panchvzluck
Very helpful.
Run accelerate config and answer:
Which type of machine are you using? -> multi-GPU
How many GPU(s) should be used for distributed training? -> <Number of GPUs>

Other than that, the default settings are fine.

The problem that cannot be tested is that the model output during training is partially corrupted.
(If you run networks/check_lora_weights.py, some will contain nan)

https://github.com/kohya-ss/sd-scripts/pull/247/files#diff-62cf7de156b588b9acd7af26941d7bb189368221946c8b5e63f69df5cda56f39R581-R584
I think this fix fixes it, but I haven't been able to test it.

Panchovix · 2023-03-04T16:44:14Z

When trying to train, I get this issue (I'm on Windows 10)

Traceback (most recent call last):
  File "G:\acc\sd-scripts\train_network.py", line 642, in <module>
    train(args)
  File "G:\acc\sd-scripts\train_network.py", line 108, in train
    accelerator, unwrap_model = train_util.prepare_accelerator(args)
  File "G:\acc\sd-scripts\library\train_util.py", line 1965, in prepare_accelerator
    accelerator = Accelerator(gradient_accumulation_steps=args.gradient_accumulation_steps, mixed_precision=args.mixed_precision,
  File "G:\acc\sd-scripts\venv\lib\site-packages\accelerate\accelerator.py", line 308, in __init__
    self.state = AcceleratorState(
  File "G:\acc\sd-scripts\venv\lib\site-packages\accelerate\state.py", line 150, in __init__
    torch.distributed.init_process_group(backend="nccl", **kwargs)
  File "G:\acc\sd-scripts\venv\lib\site-packages\torch\distributed\distributed_c10d.py", line 602, in init_process_group
    default_pg = _new_process_group_helper(
  File "G:\acc\sd-scripts\venv\lib\site-packages\torch\distributed\distributed_c10d.py", line 727, in _new_process_group_helper
    raise RuntimeError("Distributed package doesn't have NCCL " "built in")
RuntimeError: Distributed package doesn't have NCCL built in
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 1436) of binary: G:\acc\sd-scripts\venv\Scripts\python.exe

My accelerate configs were:

This Machine
multi-GPU
How many different machines will you use (use more than 1 for multi-node training)?: 1
Do you wish to optimize your script with torch dynamo?[yes/NO]: no
Do you want to use DeepSpeed? [yes/NO]: no
Do you want to use FullyShardedDataParallel? [yes/NO]: no
Do you want to use Megatron-LM ? [yes/NO]: no
How many GPU(s) should be used for distributed training?: 2
What GPU(s) (by id) should be used for training on this machine as a comma-seperated list? [all]:0,1 (tried with all and had the same issue)
fp16

Also, as another question (for the meanwhile), it is possible to run training on separated cards at the same time? So for example, train LoRA 1 on GPU 1 and LoRA 2 on GPU 2.

I tried with separate venvs/folders, but it seems both use the same accelerate config from Users\user/.cache\huggingface\accelerate\default_config.yaml

EDIT: Just found you can use --multi_gpu to use multiple GPUs without the need to change the accelerator config.

For example, accelerate launch --num_cpu_threads_per_process 32 --multi_gpu train_network.py

Now, it gives the same issue mentioned above RuntimeError: Distributed package doesn't have NCCL built in

EDIT2: You can specify which GPU to use with --gpu_ids, so for example
accelerate launch --gpu_ids 0 --num_cpu_threads_per_process 16 train_network.py uses GPU 0
accelerate launch --gpu_ids 1 --num_cpu_threads_per_process 16 train_network.py uses GPU 1
and so on

EDIT3: it seems on Windows you have to use gloo instead of NCCL, but I'm not sure how to do that.

ddPn08 · 2023-03-05T09:46:51Z

After looking into it, it seems to be an error that occurs in the windows environment.
Can you add the code below to the top of train_network.py and try it?

import os
os.environ["PL_TORCH_DISTRIBUTED_BACKEND"] = "gloo"

Panchovix · 2023-03-05T16:11:07Z

@ddPn08 At the end, managed to fix it by replacing all "nccl" string values into "gloo" in sd-scripts\venv\Lib\site-packages\accelerate\state.py

I tried

import os
os.environ["PL_TORCH_DISTRIBUTED_BACKEND"] = "gloo"

before and sadly that doesn't fix it.

Then, started to train a LoRA, and for some reason it seems to be duplicated? It is that a expected behaviour?

Also, my settings were

$learning_rate = 0.0005
$lr_warmup_ratio = 0.05
$train_batch_size = 5
$num_epochs = 2
$save_every_n_epochs=1
$scheduler="cosine_with_restarts"
$network_dim=128
$text_encoder_lr=1e-5
$unet_lr=0.00025
$mts = [int]($total / $train_batch_size * $num_epochs)
$lr_warmup_steps = [math]::Round($mts * $lr_warmup_ratio)

accelerate launch --num_cpu_threads_per_process 16 train_network.py `
    --keep_tokens=1 `
    --cache_latents `
    --enable_bucket `
    --max_data_loader_n_workers=1 `
    --persistent_data_loader_workers `
    --shuffle_caption `
    --pretrained_model_name_or_path=$ckpt `
    --train_data_dir=$image_dir `
    --reg_data_dir=$reg_dir `
    --resolution=768,768 `
    --output_dir=$output `
    --train_batch_size=$train_batch_size `
    --lr_scheduler=$scheduler `
    --learning_rate=$learning_rate `
    --lr_warmup_steps=$lr_warmup_steps `
    --max_train_steps=$mts `
    --use_8bit_adam `
    --xformers `
    --mixed_precision=bf16 `
    --save_every_n_epochs=$save_every_n_epochs `
    --seed=31337 `
    --save_precision=bf16 `
    --logging_dir="" `
    --caption_extension=.txt `
    --save_model_as=safetensors `
    --network_module=networks.lora `
    --text_encoder_lr=$text_encoder_lr `
    --unet_lr=$unet_lr `
    --network_dim=$network_dim `
    --network_alpha=$network_dim `
    --clip_skip=2 `

But then it does 4 epochs, andl the output look like this

Measuring folders:
        monagenshin: 20 repeats * 132 images = 2640
Total images with repeats: 2640       
Max training steps 2640 / 5 * 2 = 1056
NOTE: Redirects are currently not supported in Windows or MacOs.
[W ..\torch\csrc\distributed\c10d\socket.cpp:558] [c10d] The client socket has failed to connect to [kubernetes.docker.internal]:29500 (system error: 10049 - La direcci≤n solicitada no es vßlida en este contexto.).
[W ..\torch\csrc\distributed\c10d\socket.cpp:558] [c10d] The client socket has failed to connect to [kubernetes.docker.internal]:29500 (system error: 10049 - La direcci≤n solicitada no es vßlida en este contexto.).
prepare tokenizerprepare tokenizer

Use DreamBooth method.
Use DreamBooth method.
prepare images.
found directory D:\Mona training\20_monagenshin contains 132 image files
prepare images.
found directory D:\Mona training\20_monagenshin contains 132 image files
2640 train images with repeating.2640 train images with repeating.

0 reg images.
0 reg images.no regularization images / 正則化画像が見つかりませんでした

no regularization images / 正則化画像が見つかりませんでした
[Dataset 0]
  batch_size: 5
  resolution: (768, 768)
  enable_bucket: True
  min_bucket_reso: 256
  max_bucket_reso: 1024
  bucket_reso_steps: 64
  bucket_no_upscale: False

  [Subset 0 of Dataset 0]
    image_dir: "D:\Mona training\20_monagenshin"
    image_count: 132
    num_repeats: 20
    shuffle_caption: True
    keep_tokens: 1
    caption_dropout_rate: 0.0
    caption_dropout_every_n_epoches: 0
    caption_tag_dropout_rate: 0.0
    color_aug: False
    flip_aug: False
    face_crop_aug_range: None
    random_crop: False
    is_reg: False
    class_tokens: monagenshin
    caption_extension: .txt

[Dataset 0]
  batch_size: 5
  resolution: (768, 768)
  enable_bucket: True
  min_bucket_reso: 256
  max_bucket_reso: 1024
  bucket_reso_steps: 64
  bucket_no_upscale: False

  [Subset 0 of Dataset 0]
    image_dir: "D:\Mona training\20_monagenshin"
    image_count: 132
    num_repeats: 20
    shuffle_caption: True
    keep_tokens: 1
    caption_dropout_rate: 0.0
    caption_dropout_every_n_epoches: 0
    caption_tag_dropout_rate: 0.0
    color_aug: False
    flip_aug: False
    face_crop_aug_range: None
    random_crop: False
    is_reg: False
    class_tokens: monagenshin
    caption_extension: .txt



[Dataset 0][Dataset 0]

loading image sizes.loading image sizes.

100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 132/132 [00:02<00:00, 46.34it/s] 

make buckets
make buckets
number of images (including repeats) / 各bucketの画像枚数（繰り返し回数を含む）
bucket 0: resolution (512, 1024), count: 140
number of images (including repeats) / 各bucketの画像枚数（繰り返し回数を含む）
bucket 1: resolution (576, 960), count: 600bucket 0: resolution (512, 1024), count: 140

bucket 1: resolution (576, 960), count: 600bucket 2: resolution (576, 1024), count: 180

bucket 3: resolution (640, 896), count: 1460bucket 2: resolution (576, 1024), count: 180

bucket 4: resolution (704, 832), count: 100bucket 3: resolution (640, 896), count: 1460

bucket 4: resolution (704, 832), count: 100bucket 5: resolution (768, 768), count: 40

bucket 5: resolution (768, 768), count: 40
bucket 6: resolution (832, 704), count: 20bucket 6: resolution (832, 704), count: 20

bucket 7: resolution (896, 640), count: 80
bucket 7: resolution (896, 640), count: 80bucket 8: resolution (1024, 512), count: 20

bucket 8: resolution (1024, 512), count: 20
mean ar error (without repeats): 0.024412930218512426
mean ar error (without repeats): 0.024412930218512426
prepare accelerator
prepare accelerator
[W ..\torch\csrc\distributed\c10d\socket.cpp:558] [c10d] The client socket has failed to connect to [kubernetes.docker.internal]:29500 (system error: 10049 - La direcci≤n solicitada no es vßlida en este contexto.).
[W ..\torch\csrc\distributed\c10d\socket.cpp:558] [c10d] The client socket has failed to connect to [kubernetes.docker.internal]:29500 (system error: 10049 - La direcci≤n solicitada no es vßlida en este contexto.).
[W ..\torch\csrc\distributed\c10d\socket.cpp:558] [c10d] The client socket has failed to connect to [kubernetes.docker.internal]:29500 (system error: 10049 - La direcci≤n solicitada no es vßlida en este contexto.).
[W ..\torch\csrc\distributed\c10d\socket.cpp:558] [c10d] The client socket has failed to connect to [kubernetes.docker.internal]:29500 (system error: 10049 - La direcci≤n solicitada no es vßlida en este contexto.).
Using accelerator 0.15.0 or above.Using accelerator 0.15.0 or above.

load StableDiffusion checkpoint
load StableDiffusion checkpoint
loading u-net:loading u-net:  <All keys matched successfully><All keys matched successfully>

loading vae: loading vae:<All keys matched successfully> 
<All keys matched successfully>
Some weights of the model checkpoint at openai/clip-vit-large-patch14 were not used when initializing CLIPTextModel: ['vision_model.encoder.layers.15.mlp.fc2.weight', 'vision_model.encoder.layers.14.self_attn.out_proj.bias', 'vision_model.encoder.layers.20.self_attn.k_proj.weight', 'vision_model.encoder.layers.17.self_attn.v_proj.bias', 'vision_model.encoder.layers.15.self_attn.k_proj.weight', 'vision_model.encoder.layers.15.self_attn.v_proj.bias', 'vision_model.encoder.layers.8.mlp.fc2.weight', 'vision_model.encoder.layers.18.mlp.fc1.weight', 'vision_model.encoder.layers.22.mlp.fc2.bias', 'vision_model.encoder.layers.15.self_attn.k_proj.bias', 'vision_model.encoder.layers.1.mlp.fc1.bias', 'vision_model.encoder.layers.5.self_attn.v_proj.weight', 'vision_model.encoder.layers.5.mlp.fc1.bias', 'vision_model.encoder.layers.3.mlp.fc1.weight', 'vision_model.encoder.layers.14.self_attn.k_proj.weight', 'vision_model.encoder.layers.11.self_attn.v_proj.weight', 'vision_model.encoder.layers.4.self_attn.v_proj.bias', 'vision_model.encoder.layers.17.mlp.fc2.bias', 'vision_model.encoder.layers.0.self_attn.v_proj.bias', 'vision_model.encoder.layers.2.layer_norm2.bias', 'vision_model.encoder.layers.14.layer_norm2.bias', 'vision_model.encoder.layers.23.mlp.fc2.weight', 'vision_model.encoder.layers.13.mlp.fc1.weight', 'vision_model.encoder.layers.14.self_attn.v_proj.bias', 'vision_model.encoder.layers.10.mlp.fc1.bias', 'vision_model.encoder.layers.11.self_attn.out_proj.weight', 'vision_model.encoder.layers.13.mlp.fc2.weight', 'vision_model.encoder.layers.8.layer_norm1.weight', 'vision_model.encoder.layers.18.mlp.fc2.weight', 'vision_model.encoder.layers.8.mlp.fc1.weight', 'vision_model.encoder.layers.1.layer_norm1.bias', 'vision_model.encoder.layers.0.layer_norm2.weight', 'vision_model.encoder.layers.8.layer_norm1.bias', 'vision_model.encoder.layers.5.layer_norm1.bias', 'vision_model.encoder.layers.10.layer_norm1.weight', 'vision_model.encoder.layers.10.layer_norm2.weight', 'vision_model.encoder.layers.9.layer_norm1.bias', 'vision_model.encoder.layers.15.mlp.fc2.bias', 'vision_model.encoder.layers.19.self_attn.q_proj.bias', 'vision_model.encoder.layers.20.layer_norm2.bias', 'vision_model.encoder.layers.0.self_attn.q_proj.weight', 'vision_model.encoder.layers.16.layer_norm1.weight', 'vision_model.encoder.layers.2.mlp.fc2.weight', 'vision_model.encoder.layers.10.layer_norm1.bias', 'vision_model.encoder.layers.23.self_attn.q_proj.weight', 'vision_model.encoder.layers.13.mlp.fc2.bias', 'vision_model.encoder.layers.16.self_attn.v_proj.weight', 'vision_model.encoder.layers.0.self_attn.q_proj.bias', 'vision_model.encoder.layers.10.self_attn.q_proj.bias', 'vision_model.encoder.layers.7.mlp.fc2.weight', 'vision_model.encoder.layers.10.self_attn.q_proj.weight', 'vision_model.encoder.layers.13.self_attn.out_proj.weight', 'vision_model.encoder.layers.2.self_attn.out_proj.bias', 'vision_model.encoder.layers.4.layer_norm1.weight', 'vision_model.encoder.layers.16.mlp.fc1.bias', 'vision_model.encoder.layers.19.self_attn.k_proj.bias', 'vision_model.encoder.layers.22.layer_norm2.bias', 'vision_model.encoder.layers.17.self_attn.q_proj.weight', 'vision_model.encoder.layers.5.layer_norm2.bias', 'vision_model.encoder.layers.21.mlp.fc1.weight', 'vision_model.encoder.layers.6.layer_norm2.bias', 'vision_model.encoder.layers.4.layer_norm2.bias', 'vision_model.encoder.layers.18.self_attn.out_proj.weight', 'vision_model.encoder.layers.9.self_attn.v_proj.bias', 'vision_model.encoder.layers.9.mlp.fc1.bias', 'vision_model.encoder.layers.12.self_attn.k_proj.bias', 'vision_model.encoder.layers.6.mlp.fc2.bias', 'vision_model.encoder.layers.7.self_attn.q_proj.bias', 'vision_model.encoder.layers.22.self_attn.out_proj.weight', 'vision_model.encoder.layers.17.mlp.fc2.weight', 'vision_model.encoder.layers.12.self_attn.out_proj.bias', 'vision_model.encoder.layers.15.layer_norm2.weight', 'vision_model.encoder.layers.16.layer_norm1.bias', 'vision_model.encoder.layers.14.layer_norm1.bias', 'vision_model.encoder.layers.0.layer_norm1.bias', 'vision_model.encoder.layers.10.mlp.fc2.bias', 'vision_model.encoder.layers.8.mlp.fc2.bias', 'vision_model.encoder.layers.22.self_attn.v_proj.bias', 'vision_model.encoder.layers.12.mlp.fc1.bias', 'vision_model.encoder.layers.4.self_attn.out_proj.weight', 'vision_model.encoder.layers.16.mlp.fc2.bias', 'vision_model.encoder.layers.4.layer_norm1.bias', 'vision_model.encoder.layers.12.layer_norm2.weight', 'vision_model.encoder.layers.0.mlp.fc2.weight', 'vision_model.encoder.layers.23.mlp.fc2.bias', 'vision_model.encoder.layers.13.self_attn.out_proj.bias', 'vision_model.encoder.layers.0.self_attn.v_proj.weight', 'vision_model.encoder.layers.4.self_attn.k_proj.weight', 'vision_model.encoder.layers.22.self_attn.q_proj.bias', 'vision_model.encoder.layers.22.layer_norm1.weight', 'vision_model.encoder.layers.23.layer_norm1.bias', 'vision_model.encoder.layers.2.self_attn.k_proj.bias', 'vision_model.encoder.layers.2.layer_norm1.weight', 'vision_model.encoder.layers.5.self_attn.q_proj.bias', 'vision_model.encoder.layers.2.mlp.fc1.weight', 'vision_model.encoder.layers.7.layer_norm1.bias', 'vision_model.encoder.layers.14.mlp.fc2.weight', 'vision_model.encoder.layers.10.mlp.fc2.weight', 'vision_model.encoder.layers.23.self_attn.out_proj.bias', 'vision_model.encoder.layers.14.self_attn.k_proj.bias', 'vision_model.encoder.layers.6.mlp.fc2.weight', 'vision_model.encoder.layers.21.self_attn.q_proj.bias', 'vision_model.encoder.layers.12.mlp.fc2.weight', 'vision_model.encoder.layers.8.layer_norm2.weight', 'vision_model.encoder.layers.10.self_attn.k_proj.bias', 'vision_model.encoder.layers.18.layer_norm1.bias', 'vision_model.encoder.layers.2.self_attn.q_proj.weight', 'vision_model.encoder.layers.12.self_attn.out_proj.weight', 'vision_model.encoder.layers.17.mlp.fc1.bias', 'vision_model.encoder.layers.15.self_attn.q_proj.weight', 'vision_model.encoder.layers.8.self_attn.k_proj.bias', 'vision_model.encoder.layers.19.self_attn.out_proj.weight', 'vision_model.encoder.layers.13.layer_norm2.bias', 'vision_model.encoder.layers.7.mlp.fc2.bias', 'vision_model.encoder.layers.4.self_attn.k_proj.bias', 'vision_model.encoder.layers.22.mlp.fc1.bias', 'vision_model.encoder.layers.5.self_attn.out_proj.bias', 'vision_model.encoder.layers.11.self_attn.k_proj.weight', 'vision_model.encoder.layers.7.mlp.fc1.weight', 'vision_model.encoder.layers.16.self_attn.v_proj.bias', 'vision_model.encoder.layers.3.layer_norm2.weight', 'vision_model.encoder.layers.3.self_attn.q_proj.weight', 'vision_model.encoder.layers.0.mlp.fc1.bias', 'vision_model.encoder.layers.19.mlp.fc1.weight', 'vision_model.encoder.layers.1.layer_norm2.bias', 'vision_model.encoder.layers.15.self_attn.out_proj.weight', 'vision_model.encoder.layers.13.self_attn.k_proj.bias', 'vision_model.encoder.layers.9.self_attn.q_proj.weight', 'vision_model.encoder.layers.1.mlp.fc2.bias', 'vision_model.encoder.layers.12.self_attn.v_proj.weight', 'vision_model.encoder.layers.20.self_attn.k_proj.bias', 'vision_model.encoder.layers.7.self_attn.q_proj.weight', 'vision_model.encoder.layers.3.self_attn.k_proj.weight', 'vision_model.encoder.layers.14.mlp.fc2.bias', 'vision_model.encoder.layers.7.layer_norm1.weight', 'vision_model.encoder.layers.1.self_attn.k_proj.bias', 
'vision_model.encoder.layers.14.mlp.fc1.weight', 'vision_model.encoder.layers.22.self_attn.k_proj.bias', 'vision_model.encoder.layers.7.layer_norm2.weight', 'vision_model.encoder.layers.6.self_attn.v_proj.bias', 'vision_model.encoder.layers.20.mlp.fc2.weight', 'vision_model.encoder.layers.11.self_attn.v_proj.bias', 'vision_model.encoder.layers.5.layer_norm2.weight', 'vision_model.encoder.layers.19.self_attn.v_proj.weight', 'vision_model.encoder.layers.4.self_attn.q_proj.weight', 'vision_model.encoder.layers.12.self_attn.v_proj.bias', 'vision_model.encoder.layers.18.self_attn.q_proj.weight', 'vision_model.encoder.layers.18.self_attn.v_proj.weight', 'vision_model.encoder.layers.23.self_attn.v_proj.weight', 'vision_model.encoder.layers.23.mlp.fc1.weight', 'vision_model.encoder.layers.21.self_attn.v_proj.bias', 'vision_model.encoder.layers.23.layer_norm1.weight', 'vision_model.encoder.layers.18.self_attn.k_proj.bias', 'vision_model.encoder.layers.1.self_attn.out_proj.bias', 'vision_model.encoder.layers.1.self_attn.k_proj.weight', 'vision_model.encoder.layers.17.self_attn.k_proj.weight', 'vision_model.encoder.layers.0.layer_norm2.bias', 'vision_model.encoder.layers.8.self_attn.v_proj.weight', 'vision_model.encoder.layers.1.self_attn.v_proj.weight', 'vision_model.encoder.layers.15.self_attn.out_proj.bias', 'vision_model.encoder.layers.16.mlp.fc1.weight', 'vision_model.encoder.layers.16.self_attn.out_proj.weight', 'vision_model.encoder.layers.11.self_attn.out_proj.bias', 'vision_model.encoder.layers.23.layer_norm2.bias', 'vision_model.encoder.layers.1.layer_norm1.weight', 'vision_model.encoder.layers.15.self_attn.q_proj.bias', 'vision_model.encoder.layers.17.self_attn.q_proj.bias', 'vision_model.encoder.layers.0.mlp.fc2.bias', 'vision_model.encoder.layers.3.self_attn.k_proj.bias', 'vision_model.encoder.layers.16.self_attn.q_proj.bias', 'vision_model.post_layernorm.weight', 'vision_model.encoder.layers.1.mlp.fc1.weight', 'vision_model.encoder.layers.15.layer_norm1.bias', 'vision_model.encoder.layers.21.layer_norm2.weight', 'vision_model.encoder.layers.6.self_attn.out_proj.weight', 'vision_model.embeddings.position_ids', 'vision_model.encoder.layers.18.self_attn.k_proj.weight', 'vision_model.encoder.layers.22.self_attn.q_proj.weight', 'vision_model.encoder.layers.22.layer_norm1.bias', 'vision_model.encoder.layers.23.mlp.fc1.bias', 'vision_model.encoder.layers.8.self_attn.v_proj.bias', 'vision_model.encoder.layers.6.self_attn.q_proj.bias', 'vision_model.encoder.layers.21.mlp.fc2.bias', 'vision_model.encoder.layers.4.mlp.fc2.bias', 'visual_projection.weight', 'vision_model.encoder.layers.11.self_attn.q_proj.weight', 'vision_model.encoder.layers.2.self_attn.v_proj.bias', 'vision_model.encoder.layers.2.self_attn.out_proj.weight', 'vision_model.embeddings.class_embedding', 'vision_model.encoder.layers.5.self_attn.v_proj.bias', 'vision_model.encoder.layers.2.self_attn.q_proj.bias', 'vision_model.encoder.layers.20.layer_norm1.weight', 'vision_model.encoder.layers.11.self_attn.q_proj.bias', 'vision_model.encoder.layers.13.self_attn.v_proj.weight', 'vision_model.encoder.layers.10.self_attn.v_proj.weight', 'vision_model.encoder.layers.18.self_attn.v_proj.bias', 'vision_model.encoder.layers.20.mlp.fc2.bias', 'vision_model.encoder.layers.9.self_attn.q_proj.bias', 'vision_model.encoder.layers.0.self_attn.k_proj.weight', 'vision_model.encoder.layers.6.layer_norm1.weight', 'vision_model.encoder.layers.6.layer_nSome weights of the model checkpoint at openai/clip-vit-large-patch14 were not used when initializing CLIPTextModel: ['vision_model.encoder.layers.12.mlp.fc1.bias', 'visual_projection.weight', 'vision_model.encoder.layers.20.self_attn.v_proj.weight', 'vision_model.encoder.layers.8.self_attn.q_proj.weight', 'vision_model.encoder.layers.2.layer_norm1.weight', 'vision_model.encoder.layers.21.self_attn.q_proj.bias', 'vision_model.encoder.layers.12.mlp.fc2.weight', 'vision_model.encoder.layers.20.self_attn.k_proj.weight', 'vision_model.encoder.layers.3.mlp.fc1.weight', 'vision_model.encoder.layers.0.self_attn.q_proj.bias', 'vision_model.encoder.layers.17.layer_norm2.weight', 'vision_model.encoder.layers.11.self_attn.q_proj.bias', 'vision_model.encoder.layers.21.layer_norm2.bias', 'vision_model.encoder.layers.12.self_attn.v_proj.bias', 'vision_model.embeddings.class_embedding', 'vision_model.encoder.layers.8.self_attn.k_proj.weight', 'vision_model.encoder.layers.22.self_attn.k_proj.weight', 'vision_model.encoder.layers.16.layer_norm1.bias', 'vision_model.encoder.layers.3.mlp.fc1.bias', 'vision_model.encoder.layers.5.self_attn.k_proj.bias', 'vision_model.encoder.layers.0.self_attn.v_proj.bias', 'vision_model.post_layernorm.bias', 'vision_model.encoder.layers.18.self_attn.out_proj.bias', 'vision_model.encoder.layers.11.self_attn.v_proj.weight', 'vision_model.encoder.layers.13.layer_norm1.weight', 'vision_model.embeddings.position_ids', 'vision_model.encoder.layers.9.mlp.fc2.weight', 'vision_model.encoder.layers.21.mlp.fc1.bias', 'vision_model.encoder.layers.3.self_attn.k_proj.bias', 'vision_model.encoder.layers.3.self_attn.v_proj.bias', 'vision_model.encoder.layers.6.self_attn.v_proj.weight', 'vision_model.encoder.layers.4.mlp.fc1.bias', 'vision_model.encoder.layers.9.self_attn.k_proj.weight', 'vision_model.encoder.layers.6.self_attn.k_proj.bias', 'vision_model.encoder.layers.3.layer_norm1.weight', 'vision_model.encoder.layers.8.self_attn.v_proj.bias', 'vision_model.encoder.layers.5.layer_norm2.weight', 'vision_model.encoder.layers.18.self_attn.q_proj.bias', 'vision_model.encoder.layers.10.mlp.fc2.bias', 'vision_model.encoder.layers.6.mlp.fc1.weight', 'vision_model.encoder.layers.16.self_attn.out_proj.bias', 'vision_model.encoder.layers.6.layer_norm2.bias', 'vision_model.encoder.layers.23.layer_norm1.bias', 'vision_model.encoder.layers.15.self_attn.q_proj.bias', 'vision_model.encoder.layers.2.mlp.fc1.bias', 'vision_model.encoder.layers.7.layer_norm1.weight', 'vision_model.encoder.layers.16.mlp.fc1.bias', 'vision_model.encoder.layers.6.layer_norm2.weight', 'vision_model.encoder.layers.14.self_attn.out_proj.bias', 'vision_model.post_layernorm.weight', 'vision_model.encoder.layers.13.self_attn.v_proj.weight', 'vision_model.encoder.layers.14.self_attn.v_proj.bias', 'vision_model.encoder.layers.14.self_attn.out_proj.weight', 'vision_model.encoder.layers.12.layer_norm2.weight', 'vision_model.encoder.layers.21.layer_norm2.weight', 'vision_model.encoder.layers.4.self_attn.out_proj.bias', 'vision_model.encoder.layers.16.mlp.fc2.bias', 'vision_model.encoder.layers.13.self_attn.out_proj.weight', 'vision_model.encoder.layers.15.self_attn.k_proj.bias', 'vision_model.encoder.layers.20.self_attn.q_proj.weight', 'vision_model.encoder.layers.15.self_attn.out_proj.weight', 'vision_model.encoder.layers.18.self_attn.q_proj.weight', 'vision_model.encoder.layers.3.self_attn.out_proj.bias', 'vision_model.encoder.layers.4.layer_norm1.weight', 'vision_model.encoder.layers.8.mlp.fc1.weight', 'vision_model.encoder.layers.14.layer_norm1.weight', 'vision_model.encoder.layers.19.self_attn.v_proj.bias', 'vision_model.encoder.layers.22.layer_norm1.weight', 'vision_model.encoder.layers.20.mlp.fc1.weight', 'vision_model.encoder.layers.1.layer_norm2.weight', 'vision_model.encoder.layers.15.mlp.fc2.bias', 'vision_model.encoder.layers.20.self_attn.out_proj.weight', 'vision_model.encoder.layers.17.self_attn.v_proj.bias', 'vision_model.encoder.layers.22.mlp.fc1.bias', 'vision_model.encoder.layers.22.layer_norm2.bias', 'vision_model.encoder.layers.11.self_attn.k_proj.weight', 'vision_model.encoder.layers.0.layer_norm1.bias', 'vision_model.encoder.layers.22.mlp.fc2.weight', 'vision_model.encoder.layers.2.layer_norm2.weight', 'vision_model.encoder.layers.5.self_attn.out_proj.bias', 'vision_model.encoder.layers.14.layer_norm1.bias', 'vision_model.encoder.layers.0.mlp.fc2.weight', 'vision_model.encoder.layers.16.layer_norm2.bias', 'vision_model.encoder.layers.8.self_attn.out_proj.weight', 'vision_model.encoder.layers.3.layer_norm2.weight', 'vision_model.encoder.layers.1.layer_norm2.bias', 'vision_model.encoder.layers.13.layer_norm2.bias', 'vision_model.encoder.layers.19.mlp.fc2.weight', 'vision_model.encoder.layers.19.self_attn.out_proj.weight', 'vision_model.encoder.layers.19.layer_norm2.bias', 'vision_model.encoder.layers.2.layer_norm2.bias', 'vision_model.encoder.layers.15.layer_norm1.bias', 'vision_model.encoder.layers.23.self_attn.q_proj.bias', 'vision_model.encoder.layers.3.mlp.fc2.weight', 'vision_model.encoder.layers.3.layer_norm2.bias', 'vision_model.encoder.layers.12.self_attn.out_proj.weight', 'vision_model.encoder.layers.0.self_attn.out_proj.weight', 'vision_model.encoder.layers.1.mlp.fc1.weight', 'vision_model.encoder.layers.0.self_attn.q_proj.weight', 'vision_model.encoder.layers.14.self_attn.q_proj.bias', 'vision_model.encoder.layers.8.layer_norm2.bias', 'vision_model.encoder.layers.8.mlp.fc1.bias', 'vision_model.encoder.layers.12.layer_norm1.bias', 'vision_model.encoder.layers.3.self_attn.out_proj.weight', 'vision_model.encoder.layers.16.self_attn.k_proj.bias', 'vision_model.encoder.layers.18.self_attn.v_proj.weight', 'vision_model.encoder.layers.2.self_attn.v_proj.weight', 'vision_model.encoder.layers.16.self_attn.out_proj.weight', 'vision_model.encoder.layers.19.self_attn.k_proj.bias', 'vision_model.encoder.layers.10.self_attn.k_proj.weight', 'vision_model.encoder.layers.19.mlp.fc2.bias', 'vision_model.encoder.layers.17.layer_norm1.weight', 'vision_model.encoder.layers.9.mlp.fc2.bias', 'vision_model.encoder.layers.23.self_attn.out_proj.bias', 'vision_model.encoder.layers.2.mlp.fc1.weight', 'vision_model.encoder.layers.17.self_attn.out_proj.bias', 'vision_model.encoder.layers.14.self_attn.v_proj.weight', 'vision_model.encoder.layers.19.layer_norm1.bias', 'vision_model.encoder.layers.22.layer_norm2.weight', 'vision_model.encoder.layers.2.self_attn.q_proj.bias', 'vision_model.encoder.layers.5.self_attn.v_proj.bias', 'vision_model.encoder.layers.5.self_attn.v_proj.weight', 'vision_model.encoder.layers.3.self_attn.q_proj.bias', 'vision_model.encoder.layers.6.layer_norm1.bias', 'vision_model.encoder.layers.17.self_attn.out_proj.weight', 'vision_model.encoder.layers.8.mlp.fc2.bias', 'vision_model.encoder.layers.3.layer_norm1.bias', 'vision_model.encoder.layers.20.layer_norm1.weight', 'vision_model.encoder.layers.1.mlp.fc2.bias', 'vision_model.encoder.layers.2.self_attn.q_proj.weight', 'vision_model.encoder.layers.5.self_attn.out_proj.weight', 'vision_model.encoder.layers.17.self_attn.q_proj.weight', 'vision_model.encoder.layers.10.mlp.fc1.weight', 'vision_model.encoder.layers.7.mlp.fc2.weight', 'vision_model.embeddings.position_embedding.weight', 'text_projection.weight', 'vision_model.encoder.layers.15.layer_norm2.bias', 'vision_model.encoder.layers.8.self_attn.v_proj.weight', 'vision_model.encoder.layers.10.self_attn.out_proj.weight', 'vision_model.encoder.layers.7.self_attn.q_proj.weight', 'vision_model.encoder.layers.2.mlp.fc2.bias', 'vision_model.encoder.layers.5.self_attn.q_proj.weight', 'vision_model.encoder.layers.14.self_attn.k_proj.bias', 'vision_model.encoder.layers.1.self_attn.v_proj.bias', 'vision_model.encoder.layers.23.self_attn.v_proj.weight', 'vision_model.encoder.layers.6.layer_norm1.weight', 'vision_model.encoder.layers.18.mlp.fc1.weight', 'vision_model.encoder.layers.22.self_attn.v_proj.weight', 'vision_model.encoder.layers.9.self_attn.v_proj.bias', 'vision_model.encoder.layers.7.self_attn.v_proj.bias', 'vision_model.encoder.layers.19.self_attn.out_proj.bias', 'vision_model.encoder.layers.6.mlp.fc2.bias', 'vision_model.encoder.layers.18.layer_norm1.weight', 'vision_model.encoder.layers.21.self_attn.k_proj.bias', 'vision_model.encoder.layers.19.self_attn.k_proj.weight', 'vision_model.encoder.layers.11.self_attn.v_proj.bias', 'vision_model.encoder.layers.4.layer_norm2.bias', 'vision_model.encoder.layers.10.self_attn.k_proj.bias', 'vision_model.encoder.layers.10.self_attn.v_proj.weight', 'vision_model.encoder.layers.22.layer_norm1.bias', 'vision_model.encoder.layers.6.mlp.fc1.bias', 'vision_model.encoder.layers.7.self_attn.k_proj.bias', 'vision_model.encoder.layers.11.self_attn.k_proj.bias', 'vision_model.encoder.layers.23.mlp.fc1.weight', 'vision_model.encoder.layers.17.mlp.fc2.weight', 'vision_model.encoder.layers.0.layer_norm1.weight', 'vision_model.encoder.layers.11.self_attn.q_proj.weight', 'vision_model.encoder.layers.17.mlp.fc2.bias', 'vision_model.encoder.layers.4.self_attn.out_proj.weight', 'vision_model.encoder.layers.20.mlp.fc2.weight', 'vision_model.encoder.layers.23.self_attn.v_proj.bias', 'vision_model.encoder.layers.1.mlp.fc2.weight', 'vision_model.encoder.layers.9.mlp.fc1.weight', 'vision_model.encoder.layers.1.self_attn.k_proj.weight', 'vision_model.encoder.layers.18.self_attn.v_proj.bias', 'vision_model.encoder.layers.15.layer_norm1.weight', 'vision_model.encoder.layers.6.self_attn.k_proj.weight', 'vision_model.encoder.layers.9.layer_norm2.bias', 'vision_model.encoder.layers.16.self_attn.v_proj.bias', 'vision_model.encoder.layers.9.self_attn.k_proj.bias', 'vision_model.encoder.layers.22.self_attn.q_proj.weight', 'vision_model.encoder.layers.17.self_attn.k_proj.weight', 'vision_model.encoder.layers.5.mlp.fc1.bias', 'vision_model.encoder.layers.14.mlp.fc2.bias', 'vision_model.encoder.layers.2.layer_norm1.bias', 'vision_model.encoder.layers.9.self_attn.out_proj.bias', 'vision_model.encoder.layers.7.mlp.fc1.weight', 'vision_model.encoder.layers.23.self_attn.out_proj.weight', 'vision_model.encoder.layers.7.layer_norm2.weight', 'vision_model.encoder.layers.13.self_attn.out_proj.bias', 'vision_model.encoder.layers.15.mlp.fc2.weight', 'vision_model.encoder.layers.18.layer_norm2.weight', 'vision_model.encoder.layers.15.self_attn.k_proj.weight', 'vision_model.encoder.layers.21.layer_norm1.bias', 'vision_model.encoder.layers.18.self_attn.k_proj.bias', 'vision_model.encoder.layers.4.self_attn.k_proj.weight', 'vision_model.encoder.layers.4.layer_norm1.bias', 'vision_model.encoder.layers.3.self_attn.q_proj.weight', 'vision_model.encoder.layers.13.layer_norm2.weight', 'vision_model.encoder.layers.22.self_attn.v_proj.bias', 'vision_model.encoder.loading text encoder:loading text encoder:  <All keys matched successfully><All keys matched successfully>

Replace CrossAttention.forward to use xformers
Replace CrossAttention.forward to use xformers
[Dataset 0]
caching latents.
  0%|                                                                                                                                                                                                                                      | 0/132 [00:00<?, ?it/s[ 
Dataset 0]
caching latents.
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 132/132 [00:24<00:00,  5.43it/s] 
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 132/132 [00:24<00:00,  5.43it/s] 
import network module: networks.lora
import network module: networks.lora
create LoRA for Text Encoder: 72 modules.
create LoRA for Text Encoder: 72 modules.
create LoRA for U-Net: 192 modules.
enable LoRA for text encoder
enable LoRA for U-Net
prepare optimizer, data loader etc.

===================================BUG REPORT===================================
Welcome to bitsandbytes. For bug reports, please submit your error trace to: https://github.com/TimDettmers/bitsandbytes/issues
For effortless bug reporting copy-paste your error into this form: https://docs.google.com/forms/d/e/1FAIpQLScPB8emS3Thkp66nvqwmjTEgxp8Y9ufuWTzFyr9kJ5AoI47dQ/viewform?usp=sf_link
================================================================================
CUDA SETUP: Loading binary H:\sd-scripts\venv\lib\site-packages\bitsandbytes\libbitsandbytes_cuda116.dll...
create LoRA for U-Net: 192 modules.
enable LoRA for text encoder
enable LoRA for U-Net
prepare optimizer, data loader etc.

===================================BUG REPORT===================================
Welcome to bitsandbytes. For bug reports, please submit your error trace to: https://github.com/TimDettmers/bitsandbytes/issues
For effortless bug reporting copy-paste your error into this form: https://docs.google.com/forms/d/e/1FAIpQLScPB8emS3Thkp66nvqwmjTEgxp8Y9ufuWTzFyr9kJ5AoI47dQ/viewform?usp=sf_link
================================================================================
CUDA SETUP: Loading binary H:\sd-scripts\venv\lib\site-packages\bitsandbytes\libbitsandbytes_cuda116.dll...
use 8-bit AdamW optimizer | {}use 8-bit AdamW optimizer | {}

running training / 学習開始
  num train images * repeats / 学習画像の数×繰り返し回数: 2640
  num reg images / 正則化画像の数: 0
  num batches per epoch / 1epochのバッチ数: 264
  num epochs / epoch数: 4
  batch size per device / バッチサイズ: 5
  gradient accumulation steps / 勾配を合計するステップ数 = 1
  total optimization steps / 学習ステップ数: 1056
steps:   0%|                                                                                                                                                                                                                              | 0/1056 [00:00<?, ?it/s]epoch 1/4
steps:   6%|████████████                                                                                                                                                                                            | 64/1056 [01:14<19:19,  1.17s/it, loss=0.0997]

Both cards seem to use ~20GB of VRAM each, and so then it would be effectively "faster" to run a training instance per GPU for different LoRAs than use both cards for 1 LoRA.

Now, I checked the 4 epochs with networks/check_lora_weights.py and got no nans, so that's fixed. Example of one (but all of them look similar)

(venv) PS H:\sd-scripts\networks> py .\check_lora_weights.py "monav13_768-000002.safetensors"
loading: monav13_768-000002.safetensors
number of LoRA modules: 528
lora_te_text_model_encoder_layers_0_mlp_fc1.lora_down.weight,0.018040349707007408,2.2444874048233032e-07
lora_te_text_model_encoder_layers_0_mlp_fc1.lora_up.weight,0.00013709600898437202,7.821654435247183e-10
lora_te_text_model_encoder_layers_0_mlp_fc2.lora_down.weight,0.00901611428707838,7.264316082000732e-08
lora_te_text_model_encoder_layers_0_mlp_fc2.lora_up.weight,0.00013321918959263712,4.71482053399086e-09
lora_te_text_model_encoder_layers_0_self_attn_k_proj.lora_down.weight,0.017995638772845268,8.475035429000854e-08
lora_te_text_model_encoder_layers_0_self_attn_k_proj.lora_up.weight,0.0001373447448713705,3.92901711165905e-09
lora_te_text_model_encoder_layers_0_self_attn_out_proj.lora_down.weight,0.017999175935983658,1.3224780559539795e-07
lora_te_text_model_encoder_layers_0_self_attn_out_proj.lora_up.weight,0.00012994608550798148,4.6566128730773926e-09
lora_te_text_model_encoder_layers_0_self_attn_q_proj.lora_down.weight,0.018037037923932076,1.4901161193847656e-07
lora_te_text_model_encoder_layers_0_self_attn_q_proj.lora_up.weight,0.0001352666731690988,3.3760443329811096e-09
lora_te_text_model_encoder_layers_0_self_attn_v_proj.lora_down.weight,0.018024059012532234,5.811452865600586e-07
lora_te_text_model_encoder_layers_0_self_attn_v_proj.lora_up.weight,0.00012715467892121524,7.0140231400728226e-09
lora_te_text_model_encoder_layers_10_mlp_fc1.lora_down.weight,0.018016288056969643,5.611218512058258e-08
lora_te_text_model_encoder_layers_10_mlp_fc1.lora_up.weight,0.00014655150880571455,2.673914423212409e-10
lora_te_text_model_encoder_layers_10_mlp_fc2.lora_down.weight,0.009020697325468063,2.9685907065868378e-08
lora_te_text_model_encoder_layers_10_mlp_fc2.lora_up.weight,0.0001443356741219759,1.8189894035458565e-09
lora_te_text_model_encoder_layers_10_self_attn_k_proj.lora_down.weight,0.018065841868519783,7.636845111846924e-08
lora_te_text_model_encoder_layers_10_self_attn_k_proj.lora_up.weight,0.00016515696188434958,5.1513779908418655e-09
lora_te_text_model_encoder_layers_10_self_attn_out_proj.lora_down.weight,0.01798878237605095,2.980232238769531e-07
lora_te_text_model_encoder_layers_10_self_attn_out_proj.lora_up.weight,0.00014796659525018185,5.056790541857481e-10
lora_te_text_model_encoder_layers_10_self_attn_q_proj.lora_down.weight,0.018030038103461266,1.8905848264694214e-07
lora_te_text_model_encoder_layers_10_self_attn_q_proj.lora_up.weight,0.00016095208411570638,1.4406396076083183e-09
lora_te_text_model_encoder_layers_10_self_attn_v_proj.lora_down.weight,0.018055664375424385,8.614733815193176e-08
lora_te_text_model_encoder_layers_10_self_attn_v_proj.lora_up.weight,0.0001436993625247851,1.127773430198431e-09
lora_te_text_model_encoder_layers_11_mlp_fc1.lora_down.weight,0.018052801489830017,4.842877388000488e-08
lora_te_text_model_encoder_layers_11_mlp_fc1.lora_up.weight,0.0,0.0
lora_te_text_model_encoder_layers_11_mlp_fc2.lora_down.weight,0.009011094458401203,5.587935447692871e-09
lora_te_text_model_encoder_layers_11_mlp_fc2.lora_up.weight,0.0,0.0
lora_te_text_model_encoder_layers_11_self_attn_k_proj.lora_down.weight,0.018093997612595558,5.587935447692871e-08
lora_te_text_model_encoder_layers_11_self_attn_k_proj.lora_up.weight,0.0,0.0
lora_te_text_model_encoder_layers_11_self_attn_out_proj.lora_down.weight,0.018091218546032906,6.332993507385254e-07
lora_te_text_model_encoder_layers_11_self_attn_out_proj.lora_up.weight,0.0,0.0
lora_te_text_model_encoder_layers_11_self_attn_q_proj.lora_down.weight,0.0179747287184,2.60770320892334e-08
lora_te_text_model_encoder_layers_11_self_attn_q_proj.lora_up.weight,0.0,0.0
lora_te_text_model_encoder_layers_11_self_attn_v_proj.lora_down.weight,0.01803145371377468,3.725290298461914e-08
lora_te_text_model_encoder_layers_11_self_attn_v_proj.lora_up.weight,0.0,0.0
lora_te_text_model_encoder_layers_1_mlp_fc1.lora_down.weight,0.018040960654616356,5.140900611877441e-07
lora_te_text_model_encoder_layers_1_mlp_fc1.lora_up.weight,0.00013872658018954098,1.9372237147763371e-10
lora_te_text_model_encoder_layers_1_mlp_fc2.lora_down.weight,0.009020853787660599,2.270098775625229e-08
lora_te_text_model_encoder_layers_1_mlp_fc2.lora_up.weight,0.0001281834556721151,3.2741809263825417e-10
lora_te_text_model_encoder_layers_1_self_attn_k_proj.lora_down.weight,0.018021561205387115,1.043081283569336e-07
lora_te_text_model_encoder_layers_1_self_attn_k_proj.lora_up.weight,0.0001312968524871394,1.9063008949160576e-09
lora_te_text_model_encoder_layers_1_self_attn_out_proj.lora_down.weight,0.018023988232016563,8.96397978067398e-09
lora_te_text_model_encoder_layers_1_self_attn_out_proj.lora_up.weight,0.00013290181232150644,8.949427865445614e-10
lora_te_text_model_encoder_layers_1_self_attn_q_proj.lora_down.weight,0.018035173416137695,5.327165126800537e-07
lora_te_text_model_encoder_layers_1_self_attn_q_proj.lora_up.weight,0.0001310379448113963,2.9685907065868378e-09
lora_te_text_model_encoder_layers_1_self_attn_v_proj.lora_down.weight,0.018083620816469193,1.30385160446167e-07
lora_te_text_model_encoder_layers_1_self_attn_v_proj.lora_up.weight,0.0001249411579919979,1.5861587598919868e-09
lora_te_text_model_encoder_layers_2_mlp_fc1.lora_down.weight,0.01801864244043827,9.73232090473175e-08
lora_te_text_model_encoder_layers_2_mlp_fc1.lora_up.weight,0.0001400903711328283,1.4370016288012266e-10
lora_te_text_model_encoder_layers_2_mlp_fc2.lora_down.weight,0.009034804999828339,5.960464477539063e-08
lora_te_text_model_encoder_layers_2_mlp_fc2.lora_up.weight,0.00012830895138904452,2.710294211283326e-10
lora_te_text_model_encoder_layers_2_self_attn_k_proj.lora_down.weight,0.01805754192173481,2.039596438407898e-07
lora_te_text_model_encoder_layers_2_self_attn_k_proj.lora_up.weight,0.00013034019502811134,5.180481821298599e-09
lora_te_text_model_encoder_layers_2_self_attn_out_proj.lora_down.weight,0.018101584166288376,4.023313522338867e-07
lora_te_text_model_encoder_layers_2_self_attn_out_proj.lora_up.weight,0.00013767550990451127,6.821210263296962e-11
lora_te_text_model_encoder_layers_2_self_attn_q_proj.lora_down.weight,0.018108101561665535,1.471489667892456e-07
lora_te_text_model_encoder_layers_2_self_attn_q_proj.lora_up.weight,0.00012771716865245253,2.0372681319713593e-09
lora_te_text_model_encoder_layers_2_self_attn_v_proj.lora_down.weight,0.018094537779688835,3.8370490074157715e-07
lora_te_text_model_encoder_layers_2_self_attn_v_proj.lora_up.weight,0.00013133385800756514,1.9063008949160576e-09
lora_te_text_model_encoder_layers_3_mlp_fc1.lora_down.weight,0.01803683675825596,6.332993507385254e-07
lora_te_text_model_encoder_layers_3_mlp_fc1.lora_up.weight,0.00014205188199412078,3.5652192309498787e-10
lora_te_text_model_encoder_layers_3_mlp_fc2.lora_down.weight,0.009019613265991211,1.51805579662323e-07
lora_te_text_model_encoder_layers_3_mlp_fc2.lora_up.weight,0.00013397443399298936,8.076312951743603e-10
lora_te_text_model_encoder_layers_3_self_attn_k_proj.lora_down.weight,0.0180459376424551,4.805624485015869e-07
lora_te_text_model_encoder_layers_3_self_attn_k_proj.lora_up.weight,0.00013662416313309222,8.207280188798904e-09
lora_te_text_model_encoder_layers_3_self_attn_out_proj.lora_down.weight,0.01800958253443241,1.6298145055770874e-07
lora_te_text_model_encoder_layers_3_self_attn_out_proj.lora_up.weight,0.00013536405458580703,3.6816345527768135e-09
lora_te_text_model_encoder_layers_3_self_attn_q_proj.lora_down.weight,0.018056919798254967,1.210719347000122e-07
lora_te_text_model_encoder_layers_3_self_attn_q_proj.lora_up.weight,0.0001355230197077617,5.6461431086063385e-09
lora_te_text_model_encoder_layers_3_self_attn_v_proj.lora_down.weight,0.018026428297162056,1.0654330253601074e-06
lora_te_text_model_encoder_layers_3_self_attn_v_proj.lora_up.weight,0.00012870371574535966,2.6047928258776665e-09
lora_te_text_model_encoder_layers_4_mlp_fc1.lora_down.weight,0.01797945611178875,1.9650906324386597e-07
lora_te_text_model_encoder_layers_4_mlp_fc1.lora_up.weight,0.0001420131156919524,1.4770193956792355e-09
lora_te_text_model_encoder_layers_4_mlp_fc2.lora_down.weight,0.00902443565428257,2.0023435354232788e-08
lora_te_text_model_encoder_layers_4_mlp_fc2.lora_up.weight,0.0001378466549795121,4.5693013817071915e-09
lora_te_text_model_encoder_layers_4_self_attn_k_proj.lora_down.weight,0.01801932044327259,4.172325134277344e-07
lora_te_text_model_encoder_layers_4_self_attn_k_proj.lora_up.weight,0.00013925637176726013,3.292370820418e-10
lora_te_text_model_encoder_layers_4_self_attn_out_proj.lora_down.weight,0.018072664737701416,3.46451997756958e-07
lora_te_text_model_encoder_layers_4_self_attn_out_proj.lora_up.weight,0.0001369359524687752,8.221832104027271e-10
lora_te_text_model_encoder_layers_4_self_attn_q_proj.lora_down.weight,0.01806602068245411,3.557652235031128e-07
lora_te_text_model_encoder_layers_4_self_attn_q_proj.lora_up.weight,0.00014328767429105937,2.3137545213103294e-09
lora_te_text_model_encoder_layers_4_self_attn_v_proj.lora_down.weight,0.018012141808867455,2.0582228899002075e-07
lora_te_text_model_encoder_layers_4_self_attn_v_proj.lora_up.weight,0.0001257303956663236,8.003553375601768e-10
lora_te_text_model_encoder_layers_5_mlp_fc1.lora_down.weight,0.018032366409897804,1.6391277313232422e-07
lora_te_text_model_encoder_layers_5_mlp_fc1.lora_up.weight,0.0001430324773536995,1.34718902700115e-11
lora_te_text_model_encoder_layers_5_mlp_fc2.lora_down.weight,0.009011360816657543,1.2747477740049362e-08
lora_te_text_model_encoder_layers_5_mlp_fc2.lora_up.weight,0.00013930741988588125,1.716671249596402e-11
lora_te_text_model_encoder_layers_5_self_attn_k_proj.lora_down.weight,0.018069777637720108,1.8510036170482635e-08
lora_te_text_model_encoder_layers_5_self_attn_k_proj.lora_up.weight,0.00013022894563619047,7.385096978396177e-10
lora_te_text_model_encoder_layers_5_self_attn_out_proj.lora_down.weight,0.01802656054496765,5.587935447692871e-07
lora_te_text_model_encoder_layers_5_self_attn_out_proj.lora_up.weight,0.00013893010327592492,7.057678885757923e-10
lora_te_text_model_encoder_layers_5_self_attn_q_proj.lora_down.weight,0.018032647669315338,4.4889748096466064e-07
lora_te_text_model_encoder_layers_5_self_attn_q_proj.lora_up.weight,0.00012832116044592112,8.985807653516531e-10
lora_te_text_model_encoder_layers_5_self_attn_v_proj.lora_down.weight,0.018096325919032097,6.146728992462158e-07
lora_te_text_model_encoder_layers_5_self_attn_v_proj.lora_up.weight,0.0001316393754677847,1.4042598195374012e-09
lora_te_text_model_encoder_layers_6_mlp_fc1.lora_down.weight,0.018103135749697685,5.438923835754395e-07
lora_te_text_model_encoder_layers_6_mlp_fc1.lora_up.weight,0.00014824719983153045,8.776623872108757e-11
lora_te_text_model_encoder_layers_6_mlp_fc2.lora_down.weight,0.009032030589878559,1.387670636177063e-07
lora_te_text_model_encoder_layers_6_mlp_fc2.lora_up.weight,0.00014503275451716036,4.94765117764473e-09
lora_te_text_model_encoder_layers_6_self_attn_k_proj.lora_down.weight,0.0180209893733263,3.650784492492676e-07
lora_te_text_model_encoder_layers_6_self_attn_k_proj.lora_up.weight,0.00013061946083325893,6.257323548197746e-10
lora_te_text_model_encoder_layers_6_self_attn_out_proj.lora_down.weight,0.018056385219097137,2.5890767574310303e-07
lora_te_text_model_encoder_layers_6_self_attn_out_proj.lora_up.weight,0.00014197490236256272,1.0277290130034089e-10
lora_te_text_model_encoder_layers_6_self_attn_q_proj.lora_down.weight,0.018064947798848152,4.190951585769653e-08
lora_te_text_model_encoder_layers_6_self_attn_q_proj.lora_up.weight,0.00013472615682985634,4.627509042620659e-09
lora_te_text_model_encoder_layers_6_self_attn_v_proj.lora_down.weight,0.018080277368426323,2.905726432800293e-07
lora_te_text_model_encoder_layers_6_self_attn_v_proj.lora_up.weight,0.00013334567483980209,1.0011717677116394e-08
lora_te_text_model_encoder_layers_7_mlp_fc1.lora_down.weight,0.017963655292987823,1.909211277961731e-07
lora_te_text_model_encoder_layers_7_mlp_fc1.lora_up.weight,0.00014824671961832792,8.640199666842818e-11
lora_te_text_model_encoder_layers_7_mlp_fc2.lora_down.weight,0.009028312750160694,6.845220923423767e-08
lora_te_text_model_encoder_layers_7_mlp_fc2.lora_up.weight,0.00014558923430740833,5.034962669014931e-09
lora_te_text_model_encoder_layers_7_self_attn_k_proj.lora_down.weight,0.018087947741150856,3.655441105365753e-08
lora_te_text_model_encoder_layers_7_self_attn_k_proj.lora_up.weight,0.00014552648644894361,1.2951204553246498e-09
lora_te_text_model_encoder_layers_7_self_attn_out_proj.lora_down.weight,0.01803925819694996,1.6298145055770874e-07
lora_te_text_model_encoder_layers_7_self_attn_out_proj.lora_up.weight,0.00014431349700316787,5.384208634495735e-10
lora_te_text_model_encoder_layers_7_self_attn_q_proj.lora_down.weight,0.017967727035284042,2.9243528842926025e-07
lora_te_text_model_encoder_layers_7_self_attn_q_proj.lora_up.weight,0.00014080297842156142,1.2078089639544487e-09
lora_te_text_model_encoder_layers_7_self_attn_v_proj.lora_down.weight,0.018000425770878792,2.980232238769531e-07
lora_te_text_model_encoder_layers_7_self_attn_v_proj.lora_up.weight,0.00013895421579945832,7.494236342608929e-10
lora_te_text_model_encoder_layers_8_mlp_fc1.lora_down.weight,0.018030283972620964,6.845220923423767e-08
lora_te_text_model_encoder_layers_8_mlp_fc1.lora_up.weight,0.00014756708696950227,2.8558133635669947e-10
lora_te_text_model_encoder_layers_8_mlp_fc2.lora_down.weight,0.009026797488331795,2.759043127298355e-08
lora_te_text_model_encoder_layers_8_mlp_fc2.lora_up.weight,0.00014561554417014122,2.735760062932968e-09
lora_te_text_model_encoder_layers_8_self_attn_k_proj.lora_down.weight,0.018077656626701355,8.307397365570068e-07
lora_te_text_model_encoder_layers_8_self_attn_k_proj.lora_up.weight,0.00013431234401650727,1.4115357771515846e-09
lora_te_text_model_encoder_layers_8_self_attn_out_proj.lora_down.weight,0.018044564872980118,2.1606683731079102e-07
lora_te_text_model_encoder_layers_8_self_attn_out_proj.lora_up.weight,0.00014453777112066746,2.091837814077735e-10
lora_te_text_model_encoder_layers_8_self_attn_q_proj.lora_down.weight,0.01801876537501812,1.0337680578231812e-07
lora_te_text_model_encoder_layers_8_self_attn_q_proj.lora_up.weight,0.00013505601964425296,4.874891601502895e-10
lora_te_text_model_encoder_layers_8_self_attn_v_proj.lora_down.weight,0.018025098368525505,7.227063179016113e-07
lora_te_text_model_encoder_layers_8_self_attn_v_proj.lora_up.weight,0.00013827478687744588,1.9063008949160576e-09
lora_te_text_model_encoder_layers_9_mlp_fc1.lora_down.weight,0.018085436895489693,3.3760443329811096e-08
lora_te_text_model_encoder_layers_9_mlp_fc1.lora_up.weight,0.0001491605071350932,1.77351466845721e-10
lora_te_text_model_encoder_layers_9_mlp_fc2.lora_down.weight,0.00902534555643797,5.704350769519806e-08
lora_te_text_model_encoder_layers_9_mlp_fc2.lora_up.weight,0.0001452104770578444,9.89530235528946e-10
lora_te_text_model_encoder_layers_9_self_attn_k_proj.lora_down.weight,0.018031517043709755,4.302710294723511e-07
lora_te_text_model_encoder_layers_9_self_attn_k_proj.lora_up.weight,0.0001365664356853813,3.2378011383116245e-10
lora_te_text_model_encoder_layers_9_self_attn_out_proj.lora_down.weight,0.018059201538562775,1.1594966053962708e-07
lora_te_text_model_encoder_layers_9_self_attn_out_proj.lora_up.weight,0.00014530531188938767,3.841705620288849e-09
lora_te_text_model_encoder_layers_9_self_attn_q_proj.lora_down.weight,0.018049487844109535,1.0207295417785645e-06
lora_te_text_model_encoder_layers_9_self_attn_q_proj.lora_up.weight,0.00013503286754712462,4.452886059880257e-09
lora_te_text_model_encoder_layers_9_self_attn_v_proj.lora_down.weight,0.018044980242848396,2.2258609533309937e-07
lora_te_text_model_encoder_layers_9_self_attn_v_proj.lora_up.weight,0.00014355081657413393,4.045432433485985e-09
lora_unet_down_blocks_0_attentions_0_proj_in.lora_down.weight,0.02695436403155327,6.668269634246826e-07
lora_unet_down_blocks_0_attentions_0_proj_in.lora_up.weight,0.002571584191173315,8.381903171539307e-08
lora_unet_down_blocks_0_attentions_0_proj_out.lora_down.weight,0.027157718315720558,1.6689300537109375e-06
lora_unet_down_blocks_0_attentions_0_proj_out.lora_up.weight,0.0029680312145501375,1.2048985809087753e-08
lora_unet_down_blocks_0_attentions_0_transformer_blocks_0_attn1_to_k.lora_down.weight,0.027177834883332253,4.976987838745117e-06
lora_unet_down_blocks_0_attentions_0_transformer_blocks_0_attn1_to_k.lora_up.weight,0.002566937357187271,1.2200325727462769e-07
lora_unet_down_blocks_0_attentions_0_transformer_blocks_0_attn1_to_out_0.lora_down.weight,0.027201276272535324,2.8908252716064453e-06
lora_unet_down_blocks_0_attentions_0_transformer_blocks_0_attn1_to_out_0.lora_up.weight,0.0025127928238362074,6.07222318649292e-07
lora_unet_down_blocks_0_attentions_0_transformer_blocks_0_attn1_to_q.lora_down.weight,0.02758965454995632,2.339482307434082e-06
lora_unet_down_blocks_0_attentions_0_transformer_blocks_0_attn1_to_q.lora_up.weight,0.0028262478299438953,2.9802322387695312e-08
lora_unet_down_blocks_0_attentions_0_transformer_blocks_0_attn1_to_v.lora_down.weight,0.027384048327803612,1.0505318641662598e-06
lora_unet_down_blocks_0_attentions_0_transformer_blocks_0_attn1_to_v.lora_up.weight,0.0022325031459331512,1.1990778148174286e-08
lora_unet_down_blocks_0_attentions_0_transformer_blocks_0_attn2_to_k.lora_down.weight,0.018077990040183067,4.284083843231201e-07
lora_unet_down_blocks_0_attentions_0_transformer_blocks_0_attn2_to_k.lora_up.weight,0.002570380922406912,4.1211023926734924e-08
lora_unet_down_blocks_0_attentions_0_transformer_blocks_0_attn2_to_out_0.lora_down.weight,0.027754565700888634,2.2351741790771484e-06
lora_unet_down_blocks_0_attentions_0_transformer_blocks_0_attn2_to_out_0.lora_up.weight,0.003073151456192136,2.2444874048233032e-07
lora_unet_down_blocks_0_attentions_0_transformer_blocks_0_attn2_to_q.lora_down.weight,0.028050264343619347,8.195638656616211e-08
lora_unet_down_blocks_0_attentions_0_transformer_blocks_0_attn2_to_q.lora_up.weight,0.0028040024917572737,5.3318217396736145e-08
lora_unet_down_blocks_0_attentions_0_transformer_blocks_0_attn2_to_v.lora_down.weight,0.018335694447159767,5.923211574554443e-07
lora_unet_down_blocks_0_attentions_0_transformer_blocks_0_attn2_to_v.lora_up.weight,0.0017004006076604128,1.8335413187742233e-09
lora_unet_down_blocks_0_attentions_0_transformer_blocks_0_ff_net_0_proj.lora_down.weight,0.027218323200941086,2.950429916381836e-06
lora_unet_down_blocks_0_attentions_0_transformer_blocks_0_ff_net_0_proj.lora_up.weight,0.003334182081744075,4.220055416226387e-09
lora_unet_down_blocks_0_attentions_0_transformer_blocks_0_ff_net_2.lora_down.weight,0.014180133119225502,5.844049155712128e-08
lora_unet_down_blocks_0_attentions_0_transformer_blocks_0_ff_net_2.lora_up.weight,0.0031400718726217747,1.564621925354004e-07
lora_unet_down_blocks_0_attentions_1_proj_in.lora_down.weight,0.027042517438530922,1.7462298274040222e-09
lora_unet_down_blocks_0_attentions_1_proj_in.lora_up.weight,0.002659635152667761,7.264316082000732e-08
lora_unet_down_blocks_0_attentions_1_proj_out.lora_down.weight,0.02726522646844387,2.086162567138672e-06
lora_unet_down_blocks_0_attentions_1_proj_out.lora_up.weight,0.002751108491793275,2.735760062932968e-08
lora_unet_down_blocks_0_attentions_1_transformer_blocks_0_attn1_to_k.lora_down.weight,0.027254438027739525,3.725290298461914e-07
lora_unet_down_blocks_0_attentions_1_transformer_blocks_0_attn1_to_k.lora_up.weight,0.002590487478300929,1.5832483768463135e-08
lora_unet_down_blocks_0_attentions_1_transformer_blocks_0_attn1_to_out_0.lora_down.weight,0.027013886719942093,5.327165126800537e-07
lora_unet_down_blocks_0_attentions_1_transformer_blocks_0_attn1_to_out_0.lora_up.weight,0.0023098986130207777,8.556526154279709e-09
lora_unet_down_blocks_0_attentions_1_transformer_blocks_0_attn1_to_q.lora_down.weight,0.02738323248922825,2.175569534301758e-06
lora_unet_down_blocks_0_attentions_1_transformer_blocks_0_attn1_to_q.lora_up.weight,0.0025748249609023333,3.306195139884949e-08
lora_unet_down_blocks_0_attentions_1_transformer_blocks_0_attn1_to_v.lora_down.weight,0.02730870619416237,6.668269634246826e-07
lora_unet_down_blocks_0_attentions_1_transformer_blocks_0_attn1_to_v.lora_up.weight,0.0020623819436877966,9.313225746154785e-08
lora_unet_down_blocks_0_attentions_1_transformer_blocks_0_attn2_to_k.lora_down.weight,0.01807185262441635,7.562339305877686e-07
lora_unet_down_blocks_0_attentions_1_transformer_blocks_0_attn2_to_k.lora_up.weight,0.0023828200064599514,3.073364496231079e-08
lora_unet_down_blocks_0_attentions_1_transformer_blocks_0_attn2_to_out_0.lora_down.weight,0.027984697371721268,1.695007085800171e-07
lora_unet_down_blocks_0_attentions_1_transformer_blocks_0_attn2_to_out_0.lora_up.weight,0.002583096269518137,3.5925040720030665e-11
lora_unet_down_blocks_0_attentions_1_transformer_blocks_0_attn2_to_q.lora_down.weight,0.027813399210572243,3.557652235031128e-07
lora_unet_down_blocks_0_attentions_1_transformer_blocks_0_attn2_to_q.lora_up.weight,0.0029099357780069113,1.1641532182693481e-07
lora_unet_down_blocks_0_attentions_1_transformer_blocks_0_attn2_to_v.lora_down.weight,0.018278274685144424,5.289912223815918e-07
lora_unet_down_blocks_0_attentions_1_transformer_blocks_0_attn2_to_v.lora_up.weight,0.001896234811283648,8.381903171539307e-08
lora_unet_down_blocks_0_attentions_1_transformer_blocks_0_ff_net_0_proj.lora_down.weight,0.02673858031630516,3.904104232788086e-06
lora_unet_down_blocks_0_attentions_1_transformer_blocks_0_ff_net_0_proj.lora_up.weight,0.0031289278995245695,3.5390257835388184e-08
lora_unet_down_blocks_0_attentions_1_transformer_blocks_0_ff_net_2.lora_down.weight,0.01412285678088665,4.563480615615845e-08
lora_unet_down_blocks_0_attentions_1_transformer_blocks_0_ff_net_2.lora_up.weight,0.0029206632170826197,9.968061931431293e-10
lora_unet_down_blocks_1_attentions_0_proj_in.lora_down.weight,0.01990528404712677,1.0943040251731873e-07
lora_unet_down_blocks_1_attentions_0_proj_in.lora_up.weight,0.003504107939079404,1.7229467630386353e-08
lora_unet_down_blocks_1_attentions_0_proj_out.lora_down.weight,0.0201072059571743,8.800998330116272e-08
lora_unet_down_blocks_1_attentions_0_proj_out.lora_up.weight,0.0037280949763953686,1.0943040251731873e-08
lora_unet_down_blocks_1_attentions_0_transformer_blocks_0_attn1_to_k.lora_down.weight,0.019837165251374245,2.3096799850463867e-07
lora_unet_down_blocks_1_attentions_0_transformer_blocks_0_attn1_to_k.lora_up.weight,0.0033914882224053144,1.5459954738616943e-07
lora_unet_down_blocks_1_attentions_0_transformer_blocks_0_attn1_to_out_0.lora_down.weight,0.019823584705591202,1.0884832590818405e-08
lora_unet_down_blocks_1_attentions_0_transformer_blocks_0_attn1_to_out_0.lora_up.weight,0.0032685776241123676,1.885928213596344e-08
lora_unet_down_blocks_1_attentions_0_transformer_blocks_0_attn1_to_q.lora_down.weight,0.019978944212198257,3.5762786865234375e-07
lora_unet_down_blocks_1_attentions_0_transformer_blocks_0_attn1_to_q.lora_up.weight,0.003491665469482541,6.612390279769897e-08
lora_unet_down_blocks_1_attentions_0_transformer_blocks_0_attn1_to_v.lora_down.weight,0.019804764539003372,6.370246410369873e-07
lora_unet_down_blocks_1_attentions_0_transformer_blocks_0_attn1_to_v.lora_up.weight,0.002987623680382967,1.434236764907837e-07
lora_unet_down_blocks_1_attentions_0_transformer_blocks_0_attn2_to_k.lora_down.weight,0.018183166161179543,2.9243528842926025e-07
lora_unet_down_blocks_1_attentions_0_transformer_blocks_0_attn2_to_k.lora_up.weight,0.0025926497764885426,1.964508555829525e-09
lora_unet_down_blocks_1_attentions_0_transformer_blocks_0_attn2_to_out_0.lora_down.weight,0.019946657121181488,6.817281246185303e-07
lora_unet_down_blocks_1_attentions_0_transformer_blocks_0_attn2_to_out_0.lora_up.weight,0.0035691331140697002,2.1141022443771362e-07
lora_unet_down_blocks_1_attentions_0_transformer_blocks_0_attn2_to_q.lora_down.weight,0.02002926915884018,2.905726432800293e-07
lora_unet_down_blocks_1_attentions_0_transformer_blocks_0_attn2_to_q.lora_up.weight,0.0029271915555000305,1.2048985809087753e-08
lora_unet_down_blocks_1_attentions_0_transformer_blocks_0_attn2_to_v.lora_down.weight,0.018199225887656212,1.0104849934577942e-07
lora_unet_down_blocks_1_attentions_0_transformer_blocks_0_attn2_to_v.lora_up.weight,0.002294327598065138,1.1816155165433884e-08
lora_unet_down_blocks_1_attentions_0_transformer_blocks_0_ff_net_0_proj.lora_down.weight,0.020112747326493263,2.905726432800293e-07
lora_unet_down_blocks_1_attentions_0_transformer_blocks_0_ff_net_0_proj.lora_up.weight,0.003754063742235303,2.852175384759903e-09
lora_unet_down_blocks_1_attentions_0_transformer_blocks_0_ff_net_2.lora_down.weight,0.010669637471437454,6.51925802230835e-08
lora_unet_down_blocks_1_attentions_0_transformer_blocks_0_ff_net_2.lora_up.weight,0.0037681148387491703,4.237517714500427e-08
lora_unet_down_blocks_1_attentions_1_proj_in.lora_down.weight,0.020103517919778824,1.1222437024116516e-07
lora_unet_down_blocks_1_attentions_1_proj_in.lora_up.weight,0.003576788352802396,5.727633833885193e-08
lora_unet_down_blocks_1_attentions_1_proj_out.lora_down.weight,0.020279815420508385,4.377216100692749e-07
lora_unet_down_blocks_1_attentions_1_proj_out.lora_up.weight,0.0036853314377367496,1.862645149230957e-08
lora_unet_down_blocks_1_attentions_1_transformer_blocks_0_attn1_to_k.lora_down.weight,0.019921908155083656,1.043081283569336e-06
lora_unet_down_blocks_1_attentions_1_transformer_blocks_0_attn1_to_k.lora_up.weight,0.003457531798630953,7.683411240577698e-08
lora_unet_down_blocks_1_attentions_1_transformer_blocks_0_attn1_to_out_0.lora_down.weight,0.0198792926967144,3.3155083656311035e-07
lora_unet_down_blocks_1_attentions_1_transformer_blocks_0_attn1_to_out_0.lora_up.weight,0.003342119511216879,1.4028046280145645e-08
lora_unet_down_blocks_1_attentions_1_transformer_blocks_0_attn1_to_q.lora_down.weight,0.02002725377678871,9.89530235528946e-09
lora_unet_down_blocks_1_attentions_1_transformer_blocks_0_attn1_to_q.lora_up.weight,0.003444915171712637,1.1292286217212677e-08
lora_unet_down_blocks_1_attentions_1_transformer_blocks_0_attn1_to_v.lora_down.weight,0.019783079624176025,1.1622905731201172e-06
lora_unet_down_blocks_1_attentions_1_transformer_blocks_0_attn1_to_v.lora_up.weight,0.002905451226979494,5.2619725465774536e-08
lora_unet_down_blocks_1_attentions_1_transformer_blocks_0_attn2_to_k.lora_down.weight,0.01809518225491047,5.066394805908203e-07
lora_unet_down_blocks_1_attentions_1_transformer_blocks_0_attn2_to_k.lora_up.weight,0.0025085732340812683,1.7229467630386353e-08
lora_unet_down_blocks_1_attentions_1_transformer_blocks_0_attn2_to_out_0.lora_down.weight,0.01979825086891651,2.2910535335540771e-07
lora_unet_down_blocks_1_attentions_1_transformer_blocks_0_attn2_to_out_0.lora_up.weight,0.003094414249062538,1.0069925338029861e-08
lora_unet_down_blocks_1_attentions_1_transformer_blocks_0_attn2_to_q.lora_down.weight,0.02009187638759613,1.296401023864746e-06
lora_unet_down_blocks_1_attentions_1_transformer_blocks_0_attn2_to_q.lora_up.weight,0.003225622233003378,1.7508864402770996e-07
lora_unet_down_blocks_1_attentions_1_transformer_blocks_0_attn2_to_v.lora_down.weight,0.018244387581944466,4.284083843231201e-07
lora_unet_down_blocks_1_attentions_1_transformer_blocks_0_attn2_to_v.lora_up.weight,0.0018309354782104492,1.146690919995308e-08
lora_unet_down_blocks_1_attentions_1_transformer_blocks_0_ff_net_0_proj.lora_down.weight,0.02011146768927574,4.507601261138916e-07
lora_unet_down_blocks_1_attentions_1_transformer_blocks_0_ff_net_0_proj.lora_up.weight,0.0037767954636365175,8.381903171539307e-09
lora_unet_down_blocks_1_attentions_1_transformer_blocks_0_ff_net_2.lora_down.weight,0.01067089382559061,3.655441105365753e-08
lora_unet_down_blocks_1_attentions_1_transformer_blocks_0_ff_net_2.lora_up.weight,0.0037953450810164213,1.0849907994270325e-07
lora_unet_down_blocks_2_attentions_0_proj_in.lora_down.weight,0.014840138144791126,4.516914486885071e-08
lora_unet_down_blocks_2_attentions_0_proj_in.lora_up.weight,0.004110939800739288,2.898741513490677e-08
lora_unet_down_blocks_2_attentions_0_proj_out.lora_down.weight,0.015024006366729736,1.9278377294540405e-07
lora_unet_down_blocks_2_attentions_0_proj_out.lora_up.weight,0.0041526309214532375,7.776543498039246e-08
lora_unet_down_blocks_2_attentions_0_transformer_blocks_0_attn1_to_k.lora_down.weight,0.014431332238018513,4.302710294723511e-07
lora_unet_down_blocks_2_attentions_0_transformer_blocks_0_attn1_to_k.lora_up.weight,0.00384822441264987,1.8510036170482635e-08
lora_unet_down_blocks_2_attentions_0_transformer_blocks_0_attn1_to_out_0.lora_down.weight,0.014645631425082684,1.2852251529693604e-07
lora_unet_down_blocks_2_attentions_0_transformer_blocks_0_attn1_to_out_0.lora_up.weight,0.003956165164709091,2.6309862732887268e-08
lora_unet_down_blocks_2_attentions_0_transformer_blocks_0_attn1_to_q.lora_down.weight,0.014606975018978119,7.003545761108398e-07
lora_unet_down_blocks_2_attentions_0_transformer_blocks_0_attn1_to_q.lora_up.weight,0.003912101499736309,1.51805579662323e-07
lora_unet_down_blocks_2_attentions_0_transformer_blocks_0_attn1_to_v.lora_down.weight,0.014483565464615822,1.6577541828155518e-07
lora_unet_down_blocks_2_attentions_0_transformer_blocks_0_attn1_to_v.lora_up.weight,0.0036822992842644453,3.236345946788788e-08
lora_unet_down_blocks_2_attentions_0_transformer_blocks_0_attn2_to_k.lora_down.weight,0.018123844638466835,7.338821887969971e-07
lora_unet_down_blocks_2_attentions_0_transformer_blocks_0_attn2_to_k.lora_up.weight,0.0030060589779168367,2.0489096641540527e-08
lora_unet_down_blocks_2_attentions_0_transformer_blocks_0_attn2_to_out_0.lora_down.weight,0.014399806037545204,7.450580596923828e-08
lora_unet_down_blocks_2_attentions_0_transformer_blocks_0_attn2_to_out_0.lora_up.weight,0.0036295210011303425,3.3527612686157227e-08
lora_unet_down_blocks_2_attentions_0_transformer_blocks_0_attn2_to_q.lora_down.weight,0.014460697770118713,9.12696123123169e-08
lora_unet_down_blocks_2_attentions_0_transformer_blocks_0_attn2_to_q.lora_up.weight,0.003116987645626068,5.727633833885193e-08
lora_unet_down_blocks_2_attentions_0_transformer_blocks_0_attn2_to_v.lora_down.weight,0.018280966207385063,1.601874828338623e-07
lora_unet_down_blocks_2_attentions_0_transformer_blocks_0_attn2_to_v.lora_up.weight,0.0021629869006574154,4.540197551250458e-08
lora_unet_down_blocks_2_attentions_0_transformer_blocks_0_ff_net_0_proj.lora_down.weight,0.015107055194675922,2.759043127298355e-08
lora_unet_down_blocks_2_attentions_0_transformer_blocks_0_ff_net_0_proj.lora_up.weight,0.004155454691499472,3.958120942115784e-09
lora_unet_down_blocks_2_attentions_0_transformer_blocks_0_ff_net_2.lora_down.weight,0.0081787733361125,8.672941476106644e-09
lora_unet_down_blocks_2_attentions_0_transformer_blocks_0_ff_net_2.lora_up.weight,0.004172121174633503,1.3329554349184036e-08
lora_unet_down_blocks_2_attentions_1_proj_in.lora_down.weight,0.01492051500827074,3.6135315895080566e-07
lora_unet_down_blocks_2_attentions_1_proj_in.lora_up.weight,0.004144246224313974,1.3783574104309082e-07
lora_unet_down_blocks_2_attentions_1_proj_out.lora_down.weight,0.014962760731577873,2.0116567611694336e-07
lora_unet_down_blocks_2_attentions_1_proj_out.lora_up.weight,0.00413044448941946,3.6961864680051804e-09
lora_unet_down_blocks_2_attentions_1_transformer_blocks_0_attn1_to_k.lora_down.weight,0.014559021219611168,3.390014171600342e-07
lora_unet_down_blocks_2_attentions_1_transformer_blocks_0_attn1_to_k.lora_up.weight,0.004026948474347591,1.8812716007232666e-07
lora_unet_down_blocks_2_attentions_1_transformer_blocks_0_attn1_to_out_0.lora_down.weight,0.01464794296771288,2.682209014892578e-07
lora_unet_down_blocks_2_attentions_1_transformer_blocks_0_attn1_to_out_0.lora_up.weight,0.003902283962816,1.0477378964424133e-08
lora_unet_down_blocks_2_attentions_1_transformer_blocks_0_attn1_to_q.lora_down.weight,0.014640028588473797,5.029141902923584e-07
lora_unet_down_blocks_2_attentions_1_transformer_blocks_0_attn1_to_q.lora_up.weight,0.003989217337220907,6.51925802230835e-08
lora_unet_down_blocks_2_attentions_1_transformer_blocks_0_attn1_to_v.lora_down.weight,0.014479085803031921,3.213062882423401e-08
lora_unet_down_blocks_2_attentions_1_transformer_blocks_0_attn1_to_v.lora_up.weight,0.003644642187282443,4.0745362639427185e-08
lora_unet_down_blocks_2_attentions_1_transformer_blocks_0_attn2_to_k.lora_down.weight,0.01821483112871647,4.3958425521850586e-07
lora_unet_down_blocks_2_attentions_1_transformer_blocks_0_attn2_to_k.lora_up.weight,0.003092742059379816,4.2782630771398544e-09
lora_unet_down_blocks_2_attentions_1_transformer_blocks_0_attn2_to_out_0.lora_down.weight,0.014537018723785877,3.0174851417541504e-07
lora_unet_down_blocks_2_attentions_1_transformer_blocks_0_attn2_to_out_0.lora_up.weight,0.003901376621797681,3.306195139884949e-08
lora_unet_down_blocks_2_attentions_1_transformer_blocks_0_attn2_to_q.lora_down.weight,0.014459410682320595,1.664739102125168e-08
lora_unet_down_blocks_2_attentions_1_transformer_blocks_0_attn2_to_q.lora_up.weight,0.003393967170268297,1.909211277961731e-08
lora_unet_down_blocks_2_attentions_1_transformer_blocks_0_attn2_to_v.lora_down.weight,0.018299439921975136,5.611218512058258e-08
lora_unet_down_blocks_2_attentions_1_transformer_blocks_0_attn2_to_v.lora_up.weight,0.0024652371648699045,3.119930624961853e-08
lora_unet_down_blocks_2_attentions_1_transformer_blocks_0_ff_net_0_proj.lora_down.weight,0.015249811112880707,3.501772880554199e-07
lora_unet_down_blocks_2_attentions_1_transformer_blocks_0_ff_net_0_proj.lora_up.weight,0.004228560719639063,2.255546860396862e-09
lora_unet_down_blocks_2_attentions_1_transformer_blocks_0_ff_net_2.lora_down.weight,0.008172180503606796,3.213062882423401e-08
lora_unet_down_blocks_2_attentions_1_transformer_blocks_0_ff_net_2.lora_up.weight,0.004205022938549519,3.329478204250336e-08
lora_unet_mid_block_attentions_0_proj_in.lora_down.weight,0.014782065525650978,2.3562461137771606e-07
lora_unet_mid_block_attentions_0_proj_in.lora_up.weight,0.004151907749474049,2.954038791358471e-09
lora_unet_mid_block_attentions_0_proj_out.lora_down.weight,0.01476508378982544,2.905726432800293e-07
lora_unet_mid_block_attentions_0_proj_out.lora_up.weight,0.004211650229990482,6.612390279769897e-08
lora_unet_mid_block_attentions_0_transformer_blocks_0_attn1_to_k.lora_down.weight,0.014340388588607311,3.03611159324646e-07
lora_unet_mid_block_attentions_0_transformer_blocks_0_attn1_to_k.lora_up.weight,0.0036132303066551685,3.189779818058014e-08
lora_unet_mid_block_attentions_0_transformer_blocks_0_attn1_to_out_0.lora_down.weight,0.014541861601173878,1.2386590242385864e-07
lora_unet_mid_block_attentions_0_transformer_blocks_0_attn1_to_out_0.lora_up.weight,0.0038885201793164015,3.183231456205249e-10
lora_unet_mid_block_attentions_0_transformer_blocks_0_attn1_to_q.lora_down.weight,0.014439122751355171,1.3504177331924438e-07
lora_unet_mid_block_attentions_0_transformer_blocks_0_attn1_to_q.lora_up.weight,0.0036812410689890385,1.1757947504520416e-08
lora_unet_mid_block_attentions_0_transformer_blocks_0_attn1_to_v.lora_down.weight,0.014366865158081055,4.470348358154297e-08
lora_unet_mid_block_attentions_0_transformer_blocks_0_attn1_to_v.lora_up.weight,0.003668074030429125,4.6566128730773926e-08
lora_unet_mid_block_attentions_0_transformer_blocks_0_attn2_to_k.lora_down.weight,0.018337924033403397,4.1676685214042664e-08
lora_unet_mid_block_attentions_0_transformer_blocks_0_attn2_to_k.lora_up.weight,0.003167518647387624,3.4226104617118835e-08
lora_unet_mid_block_attentions_0_transformer_blocks_0_attn2_to_out_0.lora_down.weight,0.014471413567662239,2.477318048477173e-07
lora_unet_mid_block_attentions_0_transformer_blocks_0_attn2_to_out_0.lora_up.weight,0.0037700063548982143,2.2118911147117615e-09
lora_unet_mid_block_attentions_0_transformer_blocks_0_attn2_to_q.lora_down.weight,0.014390707015991211,5.471520125865936e-08
lora_unet_mid_block_attentions_0_transformer_blocks_0_attn2_to_q.lora_up.weight,0.003617155598476529,2.8172507882118225e-08
lora_unet_mid_block_attentions_0_transformer_blocks_0_attn2_to_v.lora_down.weight,0.018257958814501762,9.5367431640625e-07
lora_unet_mid_block_attentions_0_transformer_blocks_0_attn2_to_v.lora_up.weight,0.0028656679205596447,1.5133991837501526e-08
lora_unet_mid_block_attentions_0_transformer_blocks_0_ff_net_0_proj.lora_down.weight,0.014957894571125507,3.0174851417541504e-07
lora_unet_mid_block_attentions_0_transformer_blocks_0_ff_net_0_proj.lora_up.weight,0.004033668898046017,1.5788828022778034e-09
lora_unet_mid_block_attentions_0_transformer_blocks_0_ff_net_2.lora_down.weight,0.00775900250300765,8.149072527885437e-09
lora_unet_mid_block_attentions_0_transformer_blocks_0_ff_net_2.lora_up.weight,0.004361285362392664,1.1292286217212677e-08
lora_unet_up_blocks_1_attentions_0_proj_in.lora_down.weight,0.015101635828614235,4.3958425521850586e-07
lora_unet_up_blocks_1_attentions_0_proj_in.lora_up.weight,0.004290766082704067,6.075424607843161e-10
lora_unet_up_blocks_1_attentions_0_proj_out.lora_down.weight,0.015163403935730457,9.220093488693237e-08
lora_unet_up_blocks_1_attentions_0_proj_out.lora_up.weight,0.004286089446395636,9.022187441587448e-09
lora_unet_up_blocks_1_attentions_0_transformer_blocks_0_attn1_to_k.lora_down.weight,0.014674095436930656,2.5494955480098724e-08
lora_unet_up_blocks_1_attentions_0_transformer_blocks_0_attn1_to_k.lora_up.weight,0.004155351314693689,5.005858838558197e-08
lora_unet_up_blocks_1_attentions_0_transformer_blocks_0_attn1_to_out_0.lora_down.weight,0.014695746824145317,3.14321368932724e-08
lora_unet_up_blocks_1_attentions_0_transformer_blocks_0_attn1_to_out_0.lora_up.weight,0.003989169839769602,3.795139491558075e-08
lora_unet_up_blocks_1_attentions_0_transformer_blocks_0_attn1_to_q.lora_down.weight,0.01474658865481615,8.009374141693115e-08
lora_unet_up_blocks_1_attentions_0_transformer_blocks_0_attn1_to_q.lora_up.weight,0.004220888018608093,1.2200325727462769e-07
lora_unet_up_blocks_1_attentions_0_transformer_blocks_0_attn1_to_v.lora_down.weight,0.014611957594752312,6.07222318649292e-07
lora_unet_up_blocks_1_attentions_0_transformer_blocks_0_attn1_to_v.lora_up.weight,0.0038074911572039127,8.568167686462402e-08
lora_unet_up_blocks_1_attentions_0_transformer_blocks_0_attn2_to_k.lora_down.weight,0.018199743703007698,1.0356307029724121e-06
lora_unet_up_blocks_1_attentions_0_transformer_blocks_0_attn2_to_k.lora_up.weight,0.00291853747330606,2.223532646894455e-08
lora_unet_up_blocks_1_attentions_0_transformer_blocks_0_attn2_to_out_0.lora_down.weight,0.014474332332611084,2.942979335784912e-07
lora_unet_up_blocks_1_attentions_0_transformer_blocks_0_attn2_to_out_0.lora_up.weight,0.0034121046774089336,3.026798367500305e-08
lora_unet_up_blocks_1_attentions_0_transformer_blocks_0_attn2_to_q.lora_down.weight,0.014505851082503796,6.548361852765083e-09
lora_unet_up_blocks_1_attentions_0_transformer_blocks_0_attn2_to_q.lora_up.weight,0.0034947185777127743,1.6298145055770874e-08
lora_unet_up_blocks_1_attentions_0_transformer_blocks_0_attn2_to_v.lora_down.weight,0.018251260742545128,8.083879947662354e-07
lora_unet_up_blocks_1_attentions_0_transformer_blocks_0_attn2_to_v.lora_up.weight,0.0021495800465345383,4.7031790018081665e-08
lora_unet_up_blocks_1_attentions_0_transformer_blocks_0_ff_net_0_proj.lora_down.weight,0.015575339086353779,2.1886080503463745e-08
lora_unet_up_blocks_1_attentions_0_transformer_blocks_0_ff_net_0_proj.lora_up.weight,0.004480196628719568,4.773028194904327e-09
lora_unet_up_blocks_1_attentions_0_transformer_blocks_0_ff_net_2.lora_down.weight,0.00830579362809658,3.5390257835388184e-08
lora_unet_up_blocks_1_attentions_0_transformer_blocks_0_ff_net_2.lora_up.weight,0.004507940262556076,1.4260876923799515e-09
lora_unet_up_blocks_1_attentions_1_proj_in.lora_down.weight,0.01523263193666935,1.3690441846847534e-07
lora_unet_up_blocks_1_attentions_1_proj_in.lora_up.weight,0.004378103651106358,1.4435499906539917e-07
lora_unet_up_blocks_1_attentions_1_proj_out.lora_down.weight,0.015352305956184864,4.400499165058136e-08
lora_unet_up_blocks_1_attentions_1_proj_out.lora_up.weight,0.0044911103323102,7.078051567077637e-08
lora_unet_up_blocks_1_attentions_1_transformer_blocks_0_attn1_to_k.lora_down.weight,0.014887978322803974,8.987262845039368e-08
lora_unet_up_blocks_1_attentions_1_transformer_blocks_0_attn1_to_k.lora_up.weight,0.004499340429902077,1.6065314412117004e-08
lora_unet_up_blocks_1_attentions_1_transformer_blocks_0_attn1_to_out_0.lora_down.weight,0.014935034327208996,2.812594175338745e-07
lora_unet_up_blocks_1_attentions_1_transformer_blocks_0_attn1_to_out_0.lora_up.weight,0.004152772482484579,4.470348358154297e-08
lora_unet_up_blocks_1_attentions_1_transformer_blocks_0_attn1_to_q.lora_down.weight,0.014943686313927174,3.92901711165905e-09
lora_unet_up_blocks_1_attentions_1_transformer_blocks_0_attn1_to_q.lora_up.weight,0.0045045772567391396,1.4808028936386108e-07
lora_unet_up_blocks_1_attentions_1_transformer_blocks_0_attn1_to_v.lora_down.weight,0.014718247577548027,5.289912223815918e-07
lora_unet_up_blocks_1_attentions_1_transformer_blocks_0_attn1_to_v.lora_up.weight,0.00403695460408926,7.0140231400728226e-09
lora_unet_up_blocks_1_attentions_1_transformer_blocks_0_attn2_to_k.lora_down.weight,0.01826159469783306,4.991888999938965e-07
lora_unet_up_blocks_1_attentions_1_transformer_blocks_0_attn2_to_k.lora_up.weight,0.0030614815186709166,2.898741513490677e-08
lora_unet_up_blocks_1_attentions_1_transformer_blocks_0_attn2_to_out_0.lora_down.weight,0.014614465646445751,4.284083843231201e-08
lora_unet_up_blocks_1_attentions_1_transformer_blocks_0_attn2_to_out_0.lora_up.weight,0.003915694076567888,6.664777174592018e-09
lora_unet_up_blocks_1_attentions_1_transformer_blocks_0_attn2_to_q.lora_down.weight,0.014484372921288013,1.3969838619232178e-07
lora_unet_up_blocks_1_attentions_1_transformer_blocks_0_attn2_to_q.lora_up.weight,0.0035729638766497374,1.501757651567459e-08
lora_unet_up_blocks_1_attentions_1_transformer_blocks_0_attn2_to_v.lora_down.weight,0.01835586503148079,5.21540641784668e-07
lora_unet_up_blocks_1_attentions_1_transformer_blocks_0_attn2_to_v.lora_up.weight,0.0022668992169201374,5.238689482212067e-09
lora_unet_up_blocks_1_attentions_1_transformer_blocks_0_ff_net_0_proj.lora_down.weight,0.016280265524983406,2.3469328880310059e-07
lora_unet_up_blocks_1_attentions_1_transformer_blocks_0_ff_net_0_proj.lora_up.weight,0.004894702695310116,3.073364496231079e-08
lora_unet_up_blocks_1_attentions_1_transformer_blocks_0_ff_net_2.lora_down.weight,0.00868183746933937,5.611218512058258e-08
lora_unet_up_blocks_1_attentions_1_transformer_blocks_0_ff_net_2.lora_up.weight,0.004991691093891859,2.8172507882118225e-08
lora_unet_up_blocks_1_attentions_2_proj_in.lora_down.weight,0.015224930830299854,5.052424967288971e-08
lora_unet_up_blocks_1_attentions_2_proj_in.lora_up.weight,0.00433324882760644,5.820766091346741e-08
lora_unet_up_blocks_1_attentions_2_proj_out.lora_down.weight,0.015327557921409607,1.2759119272232056e-07
lora_unet_up_blocks_1_attentions_2_proj_out.lora_up.weight,0.004356320481747389,3.3760443329811096e-08
lora_unet_up_blocks_1_attentions_2_transformer_blocks_0_attn1_to_k.lora_down.weight,0.014565430581569672,1.6205012798309326e-07
lora_unet_up_blocks_1_attentions_2_transformer_blocks_0_attn1_to_k.lora_up.weight,0.004031130112707615,9.022187441587448e-09
lora_unet_up_blocks_1_attentions_2_transformer_blocks_0_attn1_to_out_0.lora_down.weight,0.014722181484103203,3.050081431865692e-08
lora_unet_up_blocks_1_attentions_2_transformer_blocks_0_attn1_to_out_0.lora_up.weight,0.0038998376112431288,1.3271346688270569e-08
lora_unet_up_blocks_1_attentions_2_transformer_blocks_0_attn1_to_q.lora_down.weight,0.014786151237785816,1.7601996660232544e-07
lora_unet_up_blocks_1_attentions_2_transformer_blocks_0_attn1_to_q.lora_up.weight,0.004221697803586721,7.392372936010361e-09
lora_unet_up_blocks_1_attentions_2_transformer_blocks_0_attn1_to_v.lora_down.weight,0.014489014632999897,2.3469328880310059e-07
lora_unet_up_blocks_1_attentions_2_transformer_blocks_0_attn1_to_v.lora_up.weight,0.0035727887880057096,1.4784745872020721e-08
lora_unet_up_blocks_1_attentions_2_transformer_blocks_0_attn2_to_k.lora_down.weight,0.018137652426958084,4.805624485015869e-07
lora_unet_up_blocks_1_attentions_2_transformer_blocks_0_attn2_to_k.lora_up.weight,0.00292794406414032,9.313225746154785e-09
lora_unet_up_blocks_1_attentions_2_transformer_blocks_0_attn2_to_out_0.lora_down.weight,0.01439357828348875,9.033828973770142e-08
lora_unet_up_blocks_1_attentions_2_transformer_blocks_0_attn2_to_out_0.lora_up.weight,0.003157215192914009,3.003515303134918e-08
lora_unet_up_blocks_1_attentions_2_transformer_blocks_0_attn2_to_q.lora_down.weight,0.014468813315033913,1.4156103134155273e-07
lora_unet_up_blocks_1_attentions_2_transformer_blocks_0_attn2_to_q.lora_up.weight,0.003281589597463608,3.026798367500305e-08
lora_unet_up_blocks_1_attentions_2_transformer_blocks_0_attn2_to_v.lora_down.weight,0.01828121580183506,6.379559636116028e-08
lora_unet_up_blocks_1_attentions_2_transformer_blocks_0_attn2_to_v.lora_up.weight,0.0019631446339190006,1.979060471057892e-08
lora_unet_up_blocks_1_attentions_2_transformer_blocks_0_ff_net_0_proj.lora_down.weight,0.01593686453998089,3.562308847904205e-08
lora_unet_up_blocks_1_attentions_2_transformer_blocks_0_ff_net_0_proj.lora_up.weight,0.004727794323116541,1.5599653124809265e-08
lora_unet_up_blocks_1_attentions_2_transformer_blocks_0_ff_net_2.lora_down.weight,0.008502433076500893,2.6193447411060333e-08
lora_unet_up_blocks_1_attentions_2_transformer_blocks_0_ff_net_2.lora_up.weight,0.004718913231045008,1.525040715932846e-08
lora_unet_up_blocks_2_attentions_0_proj_in.lora_down.weight,0.02039734646677971,1.0756775736808777e-07
lora_unet_up_blocks_2_attentions_0_proj_in.lora_up.weight,0.003920417744666338,8.288770914077759e-08
lora_unet_up_blocks_2_attentions_0_proj_out.lora_down.weight,0.02058623917400837,1.601874828338623e-07
lora_unet_up_blocks_2_attentions_0_proj_out.lora_up.weight,0.004109619185328484,6.100162863731384e-08
lora_unet_up_blocks_2_attentions_0_transformer_blocks_0_attn1_to_k.lora_down.weight,0.02010199800133705,1.1324882507324219e-06
lora_unet_up_blocks_2_attentions_0_transformer_blocks_0_attn1_to_k.lora_up.weight,0.0037793994415551424,1.648440957069397e-07
lora_unet_up_blocks_2_attentions_0_transformer_blocks_0_attn1_to_out_0.lora_down.weight,0.01996254362165928,3.334134817123413e-07
lora_unet_up_blocks_2_attentions_0_transformer_blocks_0_attn1_to_out_0.lora_up.weight,0.0034865750931203365,4.0046870708465576e-08
lora_unet_up_blocks_2_attentions_0_transformer_blocks_0_attn1_to_q.lora_down.weight,0.020086273550987244,2.2258609533309937e-07
lora_unet_up_blocks_2_attentions_0_transformer_blocks_0_attn1_to_q.lora_up.weight,0.003771562594920397,1.257285475730896e-07
lora_unet_up_blocks_2_attentions_0_transformer_blocks_0_attn1_to_v.lora_down.weight,0.02002520114183426,2.682209014892578e-07
lora_unet_up_blocks_2_attentions_0_transformer_blocks_0_attn1_to_v.lora_up.weight,0.003351055784150958,6.332993507385254e-08
lora_unet_up_blocks_2_attentions_0_transformer_blocks_0_attn2_to_k.lora_down.weight,0.018248070031404495,1.1734664440155029e-07
lora_unet_up_blocks_2_attentions_0_transformer_blocks_0_attn2_to_k.lora_up.weight,0.0031843450851738453,6.577465683221817e-09
lora_unet_up_blocks_2_attentions_0_transformer_blocks_0_attn2_to_out_0.lora_down.weight,0.019932204857468605,1.2293457984924316e-06
lora_unet_up_blocks_2_attentions_0_transformer_blocks_0_attn2_to_out_0.lora_up.weight,0.00280439923517406,1.2078089639544487e-09
lora_unet_up_blocks_2_attentions_0_transformer_blocks_0_attn2_to_q.lora_down.weight,0.02014869451522827,2.7939677238464355e-08
lora_unet_up_blocks_2_attentions_0_transformer_blocks_0_attn2_to_q.lora_up.weight,0.0031433100812137127,2.5727786123752594e-08
lora_unet_up_blocks_2_attentions_0_transformer_blocks_0_attn2_to_v.lora_down.weight,0.018199333921074867,1.123407855629921e-08
lora_unet_up_blocks_2_attentions_0_transformer_blocks_0_attn2_to_v.lora_up.weight,0.0019254256039857864,3.888271749019623e-08
lora_unet_up_blocks_2_attentions_0_transformer_blocks_0_ff_net_0_proj.lora_down.weight,0.020642025396227837,8.530914783477783e-07
lora_unet_up_blocks_2_attentions_0_transformer_blocks_0_ff_net_0_proj.lora_up.weight,0.004087388515472412,5.384208634495735e-09
lora_unet_up_blocks_2_attentions_0_transformer_blocks_0_ff_net_2.lora_down.weight,0.010870339348912239,3.748573362827301e-08
lora_unet_up_blocks_2_attentions_0_transformer_blocks_0_ff_net_2.lora_up.weight,0.004098108038306236,4.260800778865814e-08
lora_unet_up_blocks_2_attentions_1_proj_in.lora_down.weight,0.020257659256458282,2.0954757928848267e-07
lora_unet_up_blocks_2_attentions_1_proj_in.lora_up.weight,0.00378410704433918,1.2514647096395493e-08
lora_unet_up_blocks_2_attentions_1_proj_out.lora_down.weight,0.02043549343943596,1.5832483768463135e-07
lora_unet_up_blocks_2_attentions_1_proj_out.lora_up.weight,0.004034148063510656,3.3760443329811096e-08
lora_unet_up_blocks_2_attentions_1_transformer_blocks_0_attn1_to_k.lora_down.weight,0.020250357687473297,1.1082738637924194e-07
lora_unet_up_blocks_2_attentions_1_transformer_blocks_0_attn1_to_k.lora_up.weight,0.003890378400683403,7.450580596923828e-08
lora_unet_up_blocks_2_attentions_1_transformer_blocks_0_attn1_to_out_0.lora_down.weight,0.01999465562403202,1.564621925354004e-07
lora_unet_up_blocks_2_attentions_1_transformer_blocks_0_attn1_to_out_0.lora_up.weight,0.003531880211085081,2.4330802261829376e-08
lora_unet_up_blocks_2_attentions_1_transformer_blocks_0_attn1_to_q.lora_down.weight,0.0202980674803257,7.636845111846924e-07
lora_unet_up_blocks_2_attentions_1_transformer_blocks_0_attn1_to_q.lora_up.weight,0.004010376520454884,3.2223761081695557e-07
lora_unet_up_blocks_2_attentions_1_transformer_blocks_0_attn1_to_v.lora_down.weight,0.020029837265610695,1.1101365089416504e-06
lora_unet_up_blocks_2_attentions_1_transformer_blocks_0_attn1_to_v.lora_up.weight,0.0034101582132279873,5.122274160385132e-09
lora_unet_up_blocks_2_attentions_1_transformer_blocks_0_attn2_to_k.lora_down.weight,0.018234336748719215,1.825392246246338e-07
lora_unet_up_blocks_2_attentions_1_transformer_blocks_0_attn2_to_k.lora_up.weight,0.0032917261123657227,8.754432201385498e-08
lora_unet_up_blocks_2_attentions_1_transformer_blocks_0_attn2_to_out_0.lora_down.weight,0.02026018127799034,7.189810276031494e-07
lora_unet_up_blocks_2_attentions_1_transformer_blocks_0_attn2_to_out_0.lora_up.weight,0.003019471652805805,2.176966518163681e-08
lora_unet_up_blocks_2_attentions_1_transformer_blocks_0_attn2_to_q.lora_down.weight,0.020182013511657715,1.424923539161682e-07
lora_unet_up_blocks_2_attentions_1_transformer_blocks_0_attn2_to_q.lora_up.weight,0.0036568560171872377,1.2456439435482025e-08
lora_unet_up_blocks_2_attentions_1_transformer_blocks_0_attn2_to_v.lora_down.weight,0.018172411248087883,3.986060619354248e-07
lora_unet_up_blocks_2_attentions_1_transformer_blocks_0_attn2_to_v.lora_up.weight,0.0017911562463268638,7.916241884231567e-08
lora_unet_up_blocks_2_attentions_1_transformer_blocks_0_ff_net_0_proj.lora_down.weight,0.02080499567091465,5.029141902923584e-07
lora_unet_up_blocks_2_attentions_1_transformer_blocks_0_ff_net_0_proj.lora_up.weight,0.004240636248141527,7.974449545145035e-09
lora_unet_up_blocks_2_attentions_1_transformer_blocks_0_ff_net_2.lora_down.weight,0.010819272138178349,9.080395102500916e-08
lora_unet_up_blocks_2_attentions_1_transformer_blocks_0_ff_net_2.lora_up.weight,0.003969248849898577,1.0710209608078003e-08
lora_unet_up_blocks_2_attentions_2_proj_in.lora_down.weight,0.020098024979233742,1.5366822481155396e-07
lora_unet_up_blocks_2_attentions_2_proj_in.lora_up.weight,0.0037281070835888386,7.4214767664670944e-09
lora_unet_up_blocks_2_attentions_2_proj_out.lora_down.weight,0.0204204972833395,3.1851232051849365e-07
lora_unet_up_blocks_2_attentions_2_proj_out.lora_up.weight,0.004058992024511099,3.166496753692627e-08
lora_unet_up_blocks_2_attentions_2_transformer_blocks_0_attn1_to_k.lora_down.weight,0.01990589313209057,4.917383193969727e-07
lora_unet_up_blocks_2_attentions_2_transformer_blocks_0_attn1_to_k.lora_up.weight,0.0035659833811223507,1.548323780298233e-08
lora_unet_up_blocks_2_attentions_2_transformer_blocks_0_attn1_to_out_0.lora_down.weight,0.019910871982574463,1.691281795501709e-06
lora_unet_up_blocks_2_attentions_2_transformer_blocks_0_attn1_to_out_0.lora_up.weight,0.0034882426261901855,1.4062970876693726e-07
lora_unet_up_blocks_2_attentions_2_transformer_blocks_0_attn1_to_q.lora_down.weight,0.020060839131474495,4.1909515857696533e-07
lora_unet_up_blocks_2_attentions_2_transformer_blocks_0_attn1_to_q.lora_up.weight,0.0037677984219044447,8.707866072654724e-08
lora_unet_up_blocks_2_attentions_2_transformer_blocks_0_attn1_to_v.lora_down.weight,0.019664114341139793,3.259629011154175e-07
lora_unet_up_blocks_2_attentions_2_transformer_blocks_0_attn1_to_v.lora_up.weight,0.00309186615049839,2.735760062932968e-08
lora_unet_up_blocks_2_attentions_2_transformer_blocks_0_attn2_to_k.lora_down.weight,0.018263397738337517,1.1362135410308838e-07
lora_unet_up_blocks_2_attentions_2_transformer_blocks_0_attn2_to_k.lora_up.weight,0.0035008310806006193,7.636845111846924e-08
lora_unet_up_blocks_2_attentions_2_transformer_blocks_0_attn2_to_out_0.lora_down.weight,0.02030911110341549,1.3113021850585938e-06
lora_unet_up_blocks_2_attentions_2_transformer_blocks_0_attn2_to_out_0.lora_up.weight,0.00339595228433609,6.984919309616089e-08
lora_unet_up_blocks_2_attentions_2_transformer_blocks_0_attn2_to_q.lora_down.weight,0.02030547894537449,1.6763806343078613e-07
lora_unet_up_blocks_2_attentions_2_transformer_blocks_0_attn2_to_q.lora_up.weight,0.003981409594416618,1.2665987014770508e-07
lora_unet_up_blocks_2_attentions_2_transformer_blocks_0_attn2_to_v.lora_down.weight,0.0183580219745636,2.514570951461792e-07
lora_unet_up_blocks_2_attentions_2_transformer_blocks_0_attn2_to_v.lora_up.weight,0.0020988541655242443,6.658956408500671e-08
lora_unet_up_blocks_2_attentions_2_transformer_blocks_0_ff_net_0_proj.lora_down.weight,0.020962035283446312,5.681067705154419e-08
lora_unet_up_blocks_2_attentions_2_transformer_blocks_0_ff_net_0_proj.lora_up.weight,0.00431962963193655,1.979060471057892e-09
lora_unet_up_blocks_2_attentions_2_transformer_blocks_0_ff_net_2.lora_down.weight,0.010867650620639324,4.336470738053322e-09
lora_unet_up_blocks_2_attentions_2_transformer_blocks_0_ff_net_2.lora_up.weight,0.004008754156529903,3.725290298461914e-08
lora_unet_up_blocks_3_attentions_0_proj_in.lora_down.weight,0.027033526450395584,2.7418136596679688e-06
lora_unet_up_blocks_3_attentions_0_proj_in.lora_up.weight,0.0029628153424710035,1.0244548320770264e-07
lora_unet_up_blocks_3_attentions_0_proj_out.lora_down.weight,0.027453552931547165,4.6193599700927734e-06
lora_unet_up_blocks_3_attentions_0_proj_out.lora_up.weight,0.0032100253738462925,2.0227162167429924e-09
lora_unet_up_blocks_3_attentions_0_transformer_blocks_0_attn1_to_k.lora_down.weight,0.0277244932949543,7.404014468193054e-08
lora_unet_up_blocks_3_attentions_0_transformer_blocks_0_attn1_to_k.lora_up.weight,0.0032874152529984713,3.371387720108032e-07
lora_unet_up_blocks_3_attentions_0_transformer_blocks_0_attn1_to_out_0.lora_down.weight,0.02709488943219185,1.6614794731140137e-06
lora_unet_up_blocks_3_attentions_0_transformer_blocks_0_attn1_to_out_0.lora_up.weight,0.002774429740384221,1.0617077350616455e-07
lora_unet_up_blocks_3_attentions_0_transformer_blocks_0_attn1_to_q.lora_down.weight,0.02776627242565155,1.2218952178955078e-06
lora_unet_up_blocks_3_attentions_0_transformer_blocks_0_attn1_to_q.lora_up.weight,0.0032871596049517393,1.8044374883174896e-08
lora_unet_up_blocks_3_attentions_0_transformer_blocks_0_attn1_to_v.lora_down.weight,0.027111876755952835,8.23289155960083e-07
lora_unet_up_blocks_3_attentions_0_transformer_blocks_0_attn1_to_v.lora_up.weight,0.0024682015646249056,7.867129170335829e-11
lora_unet_up_blocks_3_attentions_0_transformer_blocks_0_attn2_to_k.lora_down.weight,0.01811121217906475,1.0989606380462646e-07
lora_unet_up_blocks_3_attentions_0_transformer_blocks_0_attn2_to_k.lora_up.weight,0.002588347066193819,1.525040715932846e-08
lora_unet_up_blocks_3_attentions_0_transformer_blocks_0_attn2_to_out_0.lora_down.weight,0.02779335156083107,1.601874828338623e-07
lora_unet_up_blocks_3_attentions_0_transformer_blocks_0_attn2_to_out_0.lora_up.weight,0.002785659162327647,5.634501576423645e-08
lora_unet_up_blocks_3_attentions_0_transformer_blocks_0_attn2_to_q.lora_down.weight,0.02783801034092903,4.6566128730773926e-07
lora_unet_up_blocks_3_attentions_0_transformer_blocks_0_attn2_to_q.lora_up.weight,0.003032614942640066,8.335337042808533e-08
lora_unet_up_blocks_3_attentions_0_transformer_blocks_0_attn2_to_v.lora_down.weight,0.01823175884783268,7.413327693939209e-07
lora_unet_up_blocks_3_attentions_0_transformer_blocks_0_attn2_to_v.lora_up.weight,0.0019040617626160383,2.5727786123752594e-08
lora_unet_up_blocks_3_attentions_0_transformer_blocks_0_ff_net_0_proj.lora_down.weight,0.027120143175125122,1.2516975402832031e-06
lora_unet_up_blocks_3_attentions_0_transformer_blocks_0_ff_net_0_proj.lora_up.weight,0.0033633157145231962,4.3655745685100555e-09
lora_unet_up_blocks_3_attentions_0_transformer_blocks_0_ff_net_2.lora_down.weight,0.014175185933709145,2.3189932107925415e-07
lora_unet_up_blocks_3_attentions_0_transformer_blocks_0_ff_net_2.lora_up.weight,0.0030258100014179945,9.872019290924072e-08
lora_unet_up_blocks_3_attentions_1_proj_in.lora_down.weight,0.02675911784172058,4.4517219066619873e-07
lora_unet_up_blocks_3_attentions_1_proj_in.lora_up.weight,0.002562910784035921,8.987262845039368e-08
lora_unet_up_blocks_3_attentions_1_proj_out.lora_down.weight,0.027119960635900497,3.129243850708008e-07
lora_unet_up_blocks_3_attentions_1_proj_out.lora_up.weight,0.002791489940136671,3.4226104617118835e-08
lora_unet_up_blocks_3_attentions_1_transformer_blocks_0_attn1_to_k.lora_down.weight,0.027324240654706955,4.5634806156158447e-07
lora_unet_up_blocks_3_attentions_1_transformer_blocks_0_attn1_to_k.lora_up.weight,0.0027338664513081312,1.9744038581848145e-07
lora_unet_up_blocks_3_attentions_1_transformer_blocks_0_attn1_to_out_0.lora_down.weight,0.027069538831710815,9.164214134216309e-07
lora_unet_up_blocks_3_attentions_1_transformer_blocks_0_attn1_to_out_0.lora_up.weight,0.002174451481550932,6.100162863731384e-08
lora_unet_up_blocks_3_attentions_1_transformer_blocks_0_attn1_to_q.lora_down.weight,0.02737056091427803,1.1026859283447266e-06
lora_unet_up_blocks_3_attentions_1_transformer_blocks_0_attn1_to_q.lora_up.weight,0.0025771278887987137,3.3527612686157227e-07
lora_unet_up_blocks_3_attentions_1_transformer_blocks_0_attn1_to_v.lora_down.weight,0.027162957936525345,3.11434268951416e-06
lora_unet_up_blocks_3_attentions_1_transformer_blocks_0_attn1_to_v.lora_up.weight,0.0018519219011068344,2.0489096641540527e-07
lora_unet_up_blocks_3_attentions_1_transformer_blocks_0_attn2_to_k.lora_down.weight,0.01810164377093315,8.940696716308594e-07
lora_unet_up_blocks_3_attentions_1_transformer_blocks_0_attn2_to_k.lora_up.weight,0.0025861079338937998,5.3085386753082275e-08
lora_unet_up_blocks_3_attentions_1_transformer_blocks_0_attn2_to_out_0.lora_down.weight,0.02767060324549675,7.711350917816162e-07
lora_unet_up_blocks_3_attentions_1_transformer_blocks_0_attn2_to_out_0.lora_up.weight,0.002754218876361847,3.4924596548080444e-08
lora_unet_up_blocks_3_attentions_1_transformer_blocks_0_attn2_to_q.lora_down.weight,0.02774946019053459,3.557652235031128e-07
lora_unet_up_blocks_3_attentions_1_transformer_blocks_0_attn2_to_q.lora_up.weight,0.002821526490151882,4.353933036327362e-08
lora_unet_up_blocks_3_attentions_1_transformer_blocks_0_attn2_to_v.lora_down.weight,0.018118008971214294,2.207234501838684e-07
lora_unet_up_blocks_3_attentions_1_transformer_blocks_0_attn2_to_v.lora_up.weight,0.0018078137654811144,6.984919309616089e-08
lora_unet_up_blocks_3_attentions_1_transformer_blocks_0_ff_net_0_proj.lora_down.weight,0.027068305760622025,6.295740604400635e-07
lora_unet_up_blocks_3_attentions_1_transformer_blocks_0_ff_net_0_proj.lora_up.weight,0.00323273497633636,9.255018085241318e-09
lora_unet_up_blocks_3_attentions_1_transformer_blocks_0_ff_net_2.lora_down.weight,0.013996893540024757,5.699694156646729e-07
lora_unet_up_blocks_3_attentions_1_transformer_blocks_0_ff_net_2.lora_up.weight,0.0027002221904695034,2.998858690261841e-07
lora_unet_up_blocks_3_attentions_2_proj_in.lora_down.weight,0.02696603536605835,1.7210841178894043e-06
lora_unet_up_blocks_3_attentions_2_proj_in.lora_up.weight,0.0023644238244742155,4.44706529378891e-08
lora_unet_up_blocks_3_attentions_2_proj_out.lora_down.weight,0.027057301253080368,6.48200511932373e-07
lora_unet_up_blocks_3_attentions_2_proj_out.lora_up.weight,0.002060717437416315,2.828892320394516e-08
lora_unet_up_blocks_3_attentions_2_transformer_blocks_0_attn1_to_k.lora_down.weight,0.027243101969361305,3.7439167499542236e-07
lora_unet_up_blocks_3_attentions_2_transformer_blocks_0_attn1_to_k.lora_up.weight,0.0028194591868668795,1.2014061212539673e-07
lora_unet_up_blocks_3_attentions_2_transformer_blocks_0_attn1_to_out_0.lora_down.weight,0.02755863592028618,1.7881393432617188e-06
lora_unet_up_blocks_3_attentions_2_transformer_blocks_0_attn1_to_out_0.lora_up.weight,0.0024019803386181593,1.0550138540565968e-09
lora_unet_up_blocks_3_attentions_2_transformer_blocks_0_attn1_to_q.lora_down.weight,0.02731284871697426,9.834766387939453e-07
lora_unet_up_blocks_3_attentions_2_transformer_blocks_0_attn1_to_q.lora_up.weight,0.002743664663285017,1.0151416063308716e-07
lora_unet_up_blocks_3_attentions_2_transformer_blocks_0_attn1_to_v.lora_down.weight,0.027485251426696777,1.3187527656555176e-06
lora_unet_up_blocks_3_attentions_2_transformer_blocks_0_attn1_to_v.lora_up.weight,0.002071769442409277,3.096647560596466e-08
lora_unet_up_blocks_3_attentions_2_transformer_blocks_0_attn2_to_k.lora_down.weight,0.018074778839945793,3.241002559661865e-07
lora_unet_up_blocks_3_attentions_2_transformer_blocks_0_attn2_to_k.lora_up.weight,0.0020063056144863367,1.3504177331924438e-07
lora_unet_up_blocks_3_attentions_2_transformer_blocks_0_attn2_to_out_0.lora_down.weight,0.02762470208108425,4.598405212163925e-09
lora_unet_up_blocks_3_attentions_2_transformer_blocks_0_attn2_to_out_0.lora_up.weight,0.0024353754706680775,2.7241185307502747e-08
lora_unet_up_blocks_3_attentions_2_transformer_blocks_0_attn2_to_q.lora_down.weight,0.027875807136297226,1.773238182067871e-06
lora_unet_up_blocks_3_attentions_2_transformer_blocks_0_attn2_to_q.lora_up.weight,0.0023595443926751614,9.546056389808655e-08
lora_unet_up_blocks_3_attentions_2_transformer_blocks_0_attn2_to_v.lora_down.weight,0.01829538121819496,7.860362529754639e-07
lora_unet_up_blocks_3_attentions_2_transformer_blocks_0_attn2_to_v.lora_up.weight,0.0016200493555516005,8.66129994392395e-08
lora_unet_up_blocks_3_attentions_2_transformer_blocks_0_ff_net_0_proj.lora_down.weight,0.027170980349183083,2.9355287551879883e-06
lora_unet_up_blocks_3_attentions_2_transformer_blocks_0_ff_net_0_proj.lora_up.weight,0.0032687659841030836,1.280568540096283e-08
lora_unet_up_blocks_3_attentions_2_transformer_blocks_0_ff_net_2.lora_down.weight,0.014133786782622337,1.103617250919342e-07
lora_unet_up_blocks_3_attentions_2_transformer_blocks_0_ff_net_2.lora_up.weight,0.0025939960032701492,3.655441105365753e-08

And it does give the expected results.

So the output is not corrupted!

ddPn08 · 2023-03-06T06:41:50Z

thank you! Some duplication of output is normal.

ddPn08 · 2023-03-08T00:49:01Z

I got it working so I opened a PR.
Regarding the log duplication, it looks like I'll have to change the whole code, so I might open another PR.

動作するようになったのでPRを開きました。
ログの複製に関してはコード全体を変更しないといけなさそうなのでまた別のPRを開くかもしれません。

itskey-rsa · 2023-03-08T01:33:09Z

I am using 4 GPUs on Windows 10. Although the training process runs smoothly, I encounter an error when attempting to use the safetensors file generated by the Checkpoint to try txt2image.

activating extra network lora with arguments [<modules.extra_networks.ExtraNetworkParams object at 0x000002C2817C10C0>]: Exception Traceback (most recent call last): File "C:\Users\Administrator\stable-diffusion-webui\modules\extra_networks.py", line 75, in activate extra_network.activate(p, extra_network_args) File "C:\Users\Administrator\stable-diffusion-webui\extensions-builtin\Lora\extra_networks_lora.py", line 23, in activate lora.load_loras(names, multipliers) File "C:\Users\Administrator\stable-diffusion-webui\extensions-builtin\Lora\lora.py", line 151, in load_loras lora = load_lora(name, lora_on_disk.filename) File "C:\Users\Administrator\stable-diffusion-webui\extensions-builtin\Lora\lora.py", line 83, in load_lora sd = sd_models.read_state_dict(filename) File "C:\Users\Administrator\stable-diffusion-webui\modules\sd_models.py", line 217, in read_state_dict pl_sd = safetensors.torch.load_file(checkpoint_file, device=device) File "C:\Users\Administrator\stable-diffusion-webui\venv\lib\site-packages\safetensors\torch.py", line 98, in load_file with safe_open(filename, framework="pt", device=device) as f: Exception: Error while deserializing header: InvalidHeaderDeserialization

ddPn08 · 2023-03-08T01:47:32Z

@HayashiItsuki
Is it an error when using a model trained using my fork?

kohya-ss · 2023-03-08T11:00:19Z

Thank you for opening the PR! It looks good. I will wait a while for a comment from HayashiItsuki.

ありがとうございます！　良さそうに見えます。 @HayashiItsuki 氏のコメントを少しだけ待ちたいと思います。

itskey-rsa · 2023-03-08T21:51:35Z

Sorry for the delay, I had to redo it 3 times, but all were successful and I was able to use the model without any problems. (I think the reason the first one didn't succeed may be due to my environment)

sashaok123 · 2023-04-29T10:19:50Z

@ddPn08 I still can't figure out how to run training on multiple video cards? Where to prescribe parameters and which ones?

Panchovix · 2023-07-26T23:50:49Z

Sorry for the old dump, but since SDXL came, multiGPU training is a really big plus.

I have 2x4090, and using --multi_gpu seems to work, but I'm not sure if I should set the epoch to half the size to what a single GPU training will do? Assuming batch size 1,

For example, if I set epoch amount to 6 and 5000 total steps, it will effectively do 6 epochs with 5000 steps.

If I set epoch amount to 6, total steps to 5000 and multiGPU (2 GPUs), I get 12 epochs but the same 5000 total steps.

It is training 2 times per 1 step, when using 2 GPUs?

littleyeson · 2024-03-27T12:33:01Z

20:29:46-877134 INFO accelerate launch --gpu_ids="0,1" --multi_gpu --num_cpu_threads_per_process=4
"C:\AI\kohya_ss/sd-scripts/sdxl_train_network.py" --bucket_no_upscale --bucket_reso_steps=64
--cache_latents --enable_bucket --min_bucket_reso=512 --max_bucket_reso=2048
--gradient_checkpointing --learning_rate="0.0002" --logging_dir="C:/AI/train/Restaurant/log"
--network_weights="C:/AI/sd-webui/models/Lora/rest1024_xllora_v4_1.safetensors"
--lr_scheduler="cosine_with_restarts" --lr_scheduler_num_cycles="1"
--max_data_loader_n_workers="0" --max_grad_norm="1" --resolution="1024,1024"
--max_train_steps="13320" --min_snr_gamma=5 --mixed_precision="fp16" --network_alpha="64"
--network_dim=128 --network_module=networks.lora --no_half_vae --noise_offset=0.0372
--optimizer_type="AdamW8bit" --output_dir="C:/AI/train/Restaurant/model"
--output_name="rest1024_xllora_v5"
--pretrained_model_name_or_path="C:/AI/sd-webui/models/Stable-diffusion/realvisxlV40_v40Bakedva
e.safetensors" --save_every_n_epochs="1" --save_model_as=safetensors --save_precision="fp16"
--text_encoder_lr=2e-05 --train_batch_size="1" --train_data_dir="C:/AI/train/Restaurant/image"
--unet_lr=0.0002 --vae="C:/AI/sd-webui/models/VAE/sdxl_vae.safetensors" --xformers
--sample_sampler=dpm_2 --sample_prompts="C:/AI/train/Restaurant/model\sample\prompt.txt"
--sample_every_n_epochs=1 --sample_every_n_steps=1000
[2024-03-27 20:29:54,278] torch.distributed.elastic.multiprocessing.redirects: [WARNING] NOTE: Redirects are currently not supported in Windows or MacOs.
[W socket.cpp:663] [c10d] The client socket has failed to connect to [api2.enscape3d.com]:29500 (system error: 10049 - 在其上下文中，该请求的地址无效。).

Traceback (most recent call last):
File "C:\AI\kohya_ss\sd-scripts\sdxl_train_network.py", line 184, in
trainer.train(args)
File "C:\AI\kohya_ss\sd-scripts\train_network.py", line 228, in train
accelerator = train_util.prepare_accelerator(args)
File "C:\AI\kohya_ss\sd-scripts\library\train_util.py", line 4031, in prepare_accelerator
INFO 0 reg images. train_util.py:1511
WARNING no regularization images / 正則化画像が見つかりませんでした train_util.py:1516
accelerator = Accelerator(
File "C:\AI\kohya_ss\venv\lib\site-packages\accelerate\accelerator.py", line 371, in init
self.state = AcceleratorState(
File "C:\AI\kohya_ss\venv\lib\site-packages\accelerate\state.py", line 758, in init
PartialState(cpu, **kwargs)
File "C:\AI\kohya_ss\venv\lib\site-packages\accelerate\state.py", line 217, in init
torch.distributed.init_process_group(backend=self.backend, **kwargs)
File "C:\AI\kohya_ss\venv\lib\site-packages\torch\distributed\c10d_logger.py", line 74, in wrapper
func_return = func(*args, **kwargs)
File "C:\AI\kohya_ss\venv\lib\site-packages\torch\distributed\distributed_c10d.py", line 1148, in init_process_group
default_pg, _ = _new_process_group_helper(
File "C:\AI\kohya_ss\venv\lib\site-packages\torch\distributed\distributed_c10d.py", line 1268, in _new_process_group_helper
raise RuntimeError("Distributed package doesn't have NCCL built in")
RuntimeError: Distributed package doesn't have NCCL built in

How to fix it?

FurkanGozukara · 2024-04-18T10:17:25Z

multi gpu broken kaggle training anyone has any guess how to fix?

#1272

ddPn08 marked this pull request as draft March 2, 2023 15:26

fix for multi gpu training

87846c0

ddPn08 force-pushed the dev branch from ea57508 to 87846c0 Compare March 8, 2023 00:46

ddPn08 marked this pull request as ready for review March 8, 2023 00:47

Merge branch 'dev' into dev

458173d

kohya-ss merged commit 00a9d73 into kohya-ss:dev Mar 10, 2023

minienglish1 mentioned this pull request Mar 13, 2023

Multi-gpu training with accelerate does not look to work as expected, Dreambooth stalls d8ahazard/sd_dreambooth_extension#1053

Closed

kohya-ss mentioned this pull request Mar 21, 2023

window server multi-GPU don't working #299

Open

This was referenced Mar 31, 2023

[Request] Fix multi-gpu training for train_db.py and fine_tune.py #359

Closed

[Bug] LORA train.py fails in DDP multi-gpu mode with --network_train_unet_only flag because of None #365

Closed

Isotr0py mentioned this pull request May 13, 2023

Fix DDP issues and Support DDP for all training scripts #448

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix for multi gpu training #247

fix for multi gpu training #247

ddPn08 commented Mar 2, 2023

kohya-ss commented Mar 2, 2023

Panchovix commented Mar 4, 2023

ddPn08 commented Mar 4, 2023 •

edited

Loading

Panchovix commented Mar 4, 2023 •

edited

Loading

ddPn08 commented Mar 5, 2023

Panchovix commented Mar 5, 2023

ddPn08 commented Mar 6, 2023

ddPn08 commented Mar 8, 2023

itskey-rsa commented Mar 8, 2023

ddPn08 commented Mar 8, 2023

kohya-ss commented Mar 8, 2023

itskey-rsa commented Mar 8, 2023

sashaok123 commented Apr 29, 2023

Panchovix commented Jul 26, 2023 •

edited

Loading

littleyeson commented Mar 27, 2024

FurkanGozukara commented Apr 18, 2024

fix for multi gpu training #247

fix for multi gpu training #247

Conversation

ddPn08 commented Mar 2, 2023

kohya-ss commented Mar 2, 2023

Panchovix commented Mar 4, 2023

ddPn08 commented Mar 4, 2023 • edited Loading

Panchovix commented Mar 4, 2023 • edited Loading

ddPn08 commented Mar 5, 2023

Panchovix commented Mar 5, 2023

ddPn08 commented Mar 6, 2023

ddPn08 commented Mar 8, 2023

itskey-rsa commented Mar 8, 2023

ddPn08 commented Mar 8, 2023

kohya-ss commented Mar 8, 2023

itskey-rsa commented Mar 8, 2023

sashaok123 commented Apr 29, 2023

Panchovix commented Jul 26, 2023 • edited Loading

littleyeson commented Mar 27, 2024

FurkanGozukara commented Apr 18, 2024

ddPn08 commented Mar 4, 2023 •

edited

Loading

Panchovix commented Mar 4, 2023 •

edited

Loading

Panchovix commented Jul 26, 2023 •

edited

Loading