Issue with continuous-value-predicting LSTM results

[jdiaz4302]

As a first step in using these tools, I am trying to get training set influence for a small LSTM (~1000 weights) and toy-sized data set (train_n and test_n = 100).

After making very few adjustments (i.e., changing the nll_loss to mse_loss), I can get results running the following

ptif.calc_influence_function.calc_influence_single(
    model,
    train_loader,
    test_loader,
    np.argmax(y_test).item(), # trying to get influence regarding the most extreme set instance
    gpu = -1,
    recursion_depth = 1,  # setting this up to 5 also works but takes longer, anything above 5 seems memory-prohibitive
    r = 1 # setting this up to 5 also works but takes longer, anything above 5 seems memory-prohibitive
)

The results are consistent between multiple calls to the function, but they correlate very poorly with leave-one-out training results.

Do you know if I am using this incorrectly or if there is any fixable reason why the implementation may perform poorly for an LSTM predicting continuous values (most examples that I see are using CNN architectures for image classification)

If useful, my fork containing those minimal changes can be found here - https://github.com/jdiaz4302/pytorch_influence_functions