Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Nans found in input #18

Open
Reham-Osama opened this issue Jul 23, 2019 · 2 comments
Open

Nans found in input #18

Reham-Osama opened this issue Jul 23, 2019 · 2 comments

Comments

@Reham-Osama
Copy link

When I try to train the baseline the following warning is logged many times:
"Warning: NaN or Inf found in input tensor."
Note that it happens only during training, doesn't happen during validation.

Is there something wrong in the baseline configurations or code that causes this issue to appear so frequently?
@alontalmor
Copy link
Collaborator

Hi can you please send the command you used when this happened ? thanks

@Reham-Osama
Copy link
Author

Reham-Osama commented Jul 24, 2019 •
edited
Loading

This is the command I used
"python -m allennlp.run train baseline/MRQA_BERTbase.jsonnet -s output_dir --include-package mrqa_allennlp"

@ZHO9504 ZHO9504 mentioned this issue Aug 6, 2019
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@Reham-Osama @alontalmor