Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

IP_adapter_face_plus_face Error layers #447

Open
MATEMATNKx opened this issue Nov 27, 2024 · 2 comments
Open

IP_adapter_face_plus_face Error layers #447

MATEMATNKx opened this issue Nov 27, 2024 · 2 comments

Comments

@MATEMATNKx
Copy link

I have full code

import torch
from diffusers import StableDiffusionXLPipeline
from PIL import Image

from ip_adapter import IPAdapterPlusXL
from ip_adapter.custom_pipelines import StableDiffusionXLCustomPipeline
base_model_path = "stabilityai/stable-diffusion-xl-base-1.0"
device = "cuda"
# load SDXL pipeline
pipe = StableDiffusionXLCustomPipeline.from_pretrained(
    base_model_path,
    torch_dtype=torch.float16,
    add_watermarker=False,
)

image_encoder_path = "sdxl_models/image_encoder"
ip_ckpt = "sdxl_models/ip-adapter-plus-face_sdxl_vit-h.bin" # a experimental version
# load ip-adapter
ip_model = IPAdapterPlusXL(pipe, image_encoder_path, ip_ckpt, device, num_tokens=16)

RuntimeError: Error(s) in loading state_dict for Resampler:
size mismatch for proj_in.weight: copying a param with shape torch.Size([1280, 1280]) from checkpoint, the shape in current model is torch.Size([1280, 1664]).

Directory:
image
image

Few week ago i tried to load https://huggingface.co/h94/IP-Adapter-FaceID/blob/main/ip-adapter-faceid-plusv2_sdxl.bin
but i have errors in layers to

@MATEMATNKx
Copy link
Author

Full error

RuntimeError                              Traceback (most recent call last)
Cell In[5], line 4
      2 ip_ckpt = "sdxl_models/ip-adapter-plus-face_sdxl_vit-h.bin" # a experimental version
      3 # load ip-adapter
----> 4 ip_model = IPAdapterPlusXL(pipe, image_encoder_path, ip_ckpt, device, num_tokens=16)

File /usr/local/lib/python3.11/dist-packages/ip_adapter/ip_adapter.py:84, in IPAdapter.__init__(self, sd_pipe, image_encoder_path, ip_ckpt, device, num_tokens)
     81 # image proj model
     82 self.image_proj_model = self.init_proj()
---> 84 self.load_ip_adapter()

File /usr/local/lib/python3.11/dist-packages/ip_adapter/ip_adapter.py:135, in IPAdapter.load_ip_adapter(self)
    133 else:
    134     state_dict = torch.load(self.ip_ckpt, map_location="cpu")
--> 135 self.image_proj_model.load_state_dict(state_dict["image_proj"])
    136 ip_layers = torch.nn.ModuleList(self.pipe.unet.attn_processors.values())
    137 ip_layers.load_state_dict(state_dict["ip_adapter"])

File /usr/local/lib/python3.11/dist-packages/torch/nn/modules/module.py:2215, in Module.load_state_dict(self, state_dict, strict, assign)
   2210         error_msgs.insert(
   2211             0, 'Missing key(s) in state_dict: {}. '.format(
   2212                 ', '.join(f'"{k}"' for k in missing_keys)))
   2214 if len(error_msgs) > 0:
-> 2215     raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
   2216                        self.__class__.__name__, "\n\t".join(error_msgs)))
   2217 return _IncompatibleKeys(missing_keys, unexpected_keys)

RuntimeError: Error(s) in loading state_dict for Resampler:
	size mismatch for proj_in.weight: copying a param with shape torch.Size([1280, 1280]) from checkpoint, the shape in current model is torch.Size([1280, 1664]).
	```

@panqihe-zjut
Copy link

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants