Stack error

[Charlesliu77]

the image size of inputs are different, i got the error below when using the dynamic_s2 preprocess method:
RuntimeError: stack expects each tensor to be equal size, but got [2560, 3584] at entry 0 and [3072, 3584] at entry 1.

[bfshi]

Hi @Charlesliu77, could you point to which line of the code this issue happens at?

[Charlesliu77]

Hi @Charlesliu77, could you point to which line of the code this issue happens at?

llava_arch.py: line 378
image_features = torch.stack(image_features, dim=0)
the input image in different size after dynamic_s2 and token processing can't stack together

[bfshi]

Hi, can you try replacing this line with

if all([feature.shape[0] == image_features[0].shape[0] for feature in image_features]):
    image_features = torch.stack(image_features, dim=0)

[Charlesliu77]

Hi, can you try replacing this line with

if all([feature.shape[0] == image_features[0].shape[0] for feature in image_features]):
    image_features = torch.stack(image_features, dim=0)

Thanks a lot, it works, but i have another question about the model verison, what's the difference between Nvila and Nvila-lite?

[bfshi]

NVILA-Lite is designed is to optimize the efficiency over NVILA while maintaining a competitive performance. The main differences between NVILA-Lite and NVILA include that NVILA-Lite uses 3x3 downsample instead of 2x2 in the mm projector, and NVILA-Lite uses dynamic res instead of dynamic s2. We will update more details about NVILA-Lite in our next version of the preprint. Stay tuned!