Loading CLIP Models

[WASasquatch]

By the features list am I to assume we can load, like, the new big CLIP models and use them in place of packages clip models with models? Kinda want to know before I spend 3 hours downloading one (if it doesn't fail... rural 4g net)

[WASasquatch]

So I'm still curious about this. I tried ViT-H-14, and it doesn't work. I tried the OpenCLIP version and the PyTorch version, and it just seems to produces texture noise on any sampler settings.

I'd be cool if we could use this much larger new CLIP models.

[BlenderNeko]

You're going to have to be careful which CLIP you try on which version of SD, SD 2.x has been trained with OpenCLIP Vit-H so that should work. If it doesn't there's probably a bug somewhere. But if you're tying to use OpenCLIP on SD 1.5 it's like trying to see if your Chinees friend speaks Swedish (he doesn't).

[WASasquatch]

You're going to have to be careful which CLIP you try on which version of SD, SD 2.x has been trained with OpenCLIP Vit-H so that should work. If it doesn't there's probably a bug somewhere. But if you're tying to use OpenCLIP on SD 1.5 it's like trying to see if your Chinees friend speaks Swedish (he doesn't).

Maybe you didn't use it right. This has been a thing for awhile with CLIP Guided Stable Diffusion community pipeline. Though it did have a prompt weight bug for awhile. But it's fun to work with, and you can get really good fine details out of it.

It's based on Disco Diffusion type CLIP Guidance, which was the most popular image generation tool to use local before SD was a thing.

[WASasquatch]

Though definitely not as fast as normal pipeline.

[BlenderNeko]

Have not read the code but my guess is that they use the visual part of the CLIP model as a guidance signal instead of the text part as a simple encoder. That can indeed work regardless of whatever model you use for the guidance signal (apart from some caveats i wont go into here). When you load a CLIP model in comfy it expects that CLIP model to just be used as an encoder of the prompt. using external models as guidance is not (yet?) a thing in comfy.

[WASasquatch]

Have not read the code but my guess is that they use the visual part of the CLIP model as a guidance signal instead of the text part as a simple encoder. That can indeed work regardless of whatever model you use for the guidance signal (apart from some caveats i wont go into here). When you load a CLIP model in comfy it expects that CLIP model to just be used as an encoder of the prompt. using external models as guidance is not (yet?) a thing in comfy.

Makes sense. I still think it would be cool to play around with all the CLIP models. I actually really wanted to add this to our service, but because of our dedication to weighting, long prompts, and inpainting and img2img it didn't make sense to add (would have meant limiting and changing up whole front end when in use).

[comfyanonymous]

It's to load these for example: https://huggingface.co/runwayml/stable-diffusion-v1-5/blob/main/text_encoder/model.safetensors

Some rare checkpoints come without CLIP weights.

[WASasquatch]

Oh ok. Is there any plans to make this work like the CLIP Guided Stable Diffusion notebooks/scripts? That is, be able to load OpenCLIP models? (I don't really know anything about the PyTorch versions so just name dropping OpenCLIP)