Inquiry: Scaling the Lr

[rb876]

Hi,
it is not clear to me how the Lr is scaled throughout the learning.

Many Thanks

[henripal]

Here:

sgld/sgld/sgld/trainer.py

Line 94 in 60bd564

optimizer.step(model_desc['lr']/(epoch//model_desc['lr_epoch'] + 1))

Contrary to pytorch optimizer, the SGLD optimizer takes in an optional lr as argument!

[rb876]

Oh thanks, and how do you scale the learning rate with respect to the number of data points ? Looking at ChunyuanLI implementation (ChunyuanLI/pSGLD#2) and at gmarceaucaron implementation (https://github.com/gmarceaucaron/natural-langevin-dynamics-for-neural-networks) they scale the langevin noise with respect to the number of data points in the training (Ntrain) or the square root of Ntrain to allow convergence. Do you do something similar ?

[henripal]

@rb876 and I discusssing this by email - closing the issue!