Not a bug, but some questions.

[Kevin-naticl]

Great work! About why it works i have some ideas, but IDK if i am right, So here i am.
My perspective:

Thought: Why is it that even though SVF has fewer trainable parts, it is still more effective?

From the perspective of linear transformation, among the three matrices derived from Singular Value Decomposition (SVD), the most crucial one is the middle matrix. The first and third matrices are essentially just rotations in the linear transformation. The diagonal matrix in the middle actually determines the scaling of weights across various dimensions. SVF can directly alter min(m,n) dimensions.

Secondly, during LoRA fine-tuning, it appears that there are (m+n)*r' parameters. However, if these two matrices are also subjected to SVD, in reality, when adjusting the most critical scaling dimensions, only r'+r' dimensions are actually available for use.

THX so much.

[floatingbigcat]

Hi @Kevin-naticl , Thank you for your interset and sharing your thought.

I think you are right. The information that the LoRA finetuning process can bring is essentially upper bounded by the number of RANK. Meanwhile, the SVF tunes the singular value matrix, it actually introduces full-rank information. Which we believe is where the effectiveness root from.