Difference between deepmd transfer learning and fine-tuning

[rajnichahal]

Hi,
I wonder how deepmd transfer learning is different from the fine-tuning option.

Upon reading the documentation for transfer learning, it says that "the parameters of the whole embedding net and the hidden layers of the fitting net can be fixed, and only the parameters of the output layer of the fitting net are trainable". My understanding is that it is the case with fine-tuning as well.

For deepmd, both options are taking a pretrained model and are updating the PES based on a new smaller dataset. So, which one should I go with?

Thanks a lot!

[iProzd]

Hello @rajnichahal , in this context, the concept of transfer learning is included within finetuning, where finetuning has a broader functionality and wider applicability:

1. The typical use case for transfer learning is when you have a model trained on a specific dataset, and you want to use the same dataset (or one with a very similar structure) but with a different functional, to train a new model. In this case, you can use transfer learning, which fixes the parameters of the descriptor and the most part of fitting net, and only trains the final output layer.
2. However, if there is a significant difference between these two datasets, this falls within the scope of finetuning. The idea of finetuning is to pre-train a more general model representation on a larger scale dataset (e.g., covering the periodic table of elements), and then finetune it on another completely different dataset (often only related by elements). During finetuning, the energy bias of the data will be recalculated, and you can manually specify which part of the network parameters to train or fix. (By default, all parameters are not fixed.) And it is expected that the previous pre-training will be helpful during finetuning, reducing the amount of new data required to obtain a PES with high accuracy.

In summary, the functionality of finetuning includes that of transfer learning. Transfer learning is only applicable when your datasets before and after are almost identical and you just want to change the functional. In other scenarios, it is recommended to use finetuning.

[rajnichahal]

Thank you for your response, @iProzd!
Got it. Thanks for making it clear.

Although, I am wondering what does it mean by 'fixes the parameters of the descriptor and the most part of fitting net'? I was thinking that it will only take the descriptor and fitting net parameters that were set for pre-trained model.