How multi-gpu latent caching works?

[FurkanGozukara]

Currently I am trainning on Kaggle with 15 train images and 100 repeat and thus 1500 reg images

Actually ideally I would like each GPU to get different set of reg images but i doubt that the system is designed that way

So currently I see that both gpu is caching same images?

I think at the very least each GPU can cache a portion of the reg images pool

How it works exactly right now @kohya-ss

[DKnight54]

At the moment, caching of latents is only done on the main process, ie on one GPU only. I do not know if there is any issues that would prevent using accelerate to split the load, but I do not understand enough of caching logic to say for sure.

[tristanwqy]

try to split your data instead :)

[kohya-ss]

As DKnight54 wrote, currently the caching of latents is done on the main process. It doesn't seem to be easy to distribute the images for multiple GPUs with accelerate.

So the easy way is splitting data to multiple dataset, as tristanwqy wrote. And then we can run tools/cache_latents.py for each dataset and GPU.

[FurkanGozukara]

As DKnight54 wrote, currently the caching of latents is done on the main process. It doesn't seem to be easy to distribute the images for multiple GPUs with accelerate.

So the easy way is splitting data to multiple dataset, as tristanwqy wrote. And then we can run tools/cache_latents.py for each dataset and GPU.

i think that would be nice