You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
We are currently testing the correctness of the implementation differently for each neural network, depending on how easy it is to access the original implementation and what reasonable tests can be performed. In doing so we also install the reference implementations in the CI conda environment.
I think we should remove the development tests and the dependencies that the comparison with the original implementation involves. After implementing a neural network, the final test should compare the implementation to the original implementation by transferring weights and biases and ensuring that the output is the same. This seems like a large time investment but can be done in a day. This is then the only test that we keep in the CI --- we test that given the weights and biases, we get a specific output for the energies/forces and other properties of interest.
I'm adding to this based on the discussion today (with Marcus and Ian).
Per Ian's suggestion, keep the tests in place, and just skip them with a description like ("legacy test that will fail due to dependencies").
Create new tests that just use hardcoded outputs from the original. Would it be better to just save these as .npz files to make the tests cleaner and more flexible?
We should probably create a separate folder that contains a python file for generating the hardcoded values (with an appropriate conda environment file) and even some simple tests to validate library output against these files. The idea of having the tests in here as well would be that if they do come out with a new version of schnetpack, this would (assuming no massive architecture changes, just changing dependencies) allow us to validate no substantive changes in the output have occurred. Since our original implementations were based on their code, if a new release includes a bug fix, we want to be able to catch it!.
We are currently testing the correctness of the implementation differently for each neural network, depending on how easy it is to access the original implementation and what reasonable tests can be performed. In doing so we also install the reference implementations in the CI conda environment.
I think we should remove the development tests and the dependencies that the comparison with the original implementation involves. After implementing a neural network, the final test should compare the implementation to the original implementation by transferring weights and biases and ensuring that the output is the same. This seems like a large time investment but can be done in a day. This is then the only test that we keep in the CI --- we test that given the weights and biases, we get a specific output for the energies/forces and other properties of interest.
@ArnNag and @chrisiacovella , what is your opinion on this?
The text was updated successfully, but these errors were encountered: