New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

convert_nt_to_hf is broken for non-interleaved RoPE #241

Open

gritukan opened this issue Nov 15, 2024 · 1 comment

gritukan commented Nov 15, 2024

nanotron/examples/llama/convert_nanotron_to_hf.py

Lines 34 to 39 in 51ca40b

    
           def interleave(w: torch.Tensor): 
        
               w_new = [] 
        
               for head_w in w.split(d_qk): 
        
                   head_w = head_w.view(d_qk // 2, 2, -1).transpose(0, 1).reshape(d_qk, -1) 
        
                   w_new.append(head_w) 
        
               return torch.cat(w_new)

This code should not be executed if the model uses non-interleaved RoPE.

Contributor

TJ-Solergibert commented Nov 23, 2024

Hi, you can check #174 too 👀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment