Efficient Tensor conversion from list of numpy arrays #1071
Conversation
|
Would you mind undoing the version bump? Usually I'll only do a bump after a few changes, and have a corresponding release when I do it. |
|
Absolutely no problem. Actually, I did the version bump because GitHub Actions reported a cancellation but no failure and wanted to retest. Will undo the commit. |
|
Thanks! |
|
|
||
|
|
||
| if isinstance(value, list) and all(isinstance(v, numpy.ndarray) for v in value): | ||
| value = numpy.array(value) |
There was a problem hiding this comment.
This doesn't seem to have a return statement in all cases?
There was a problem hiding this comment.
It's saying that if value is a list of numpy arrays, cast value as a single numpy array. L63 will then take that new numpy array and operate on it, but more efficiently than if it were a list of arrays.
|
Did this PR break the build? |
|
No. There are some unit tests that stochastically raise NaNs. When I manually re-run them they run successfully. 100% of the time these same tests run successfully on my local machine so I'm not sure what their source is. |
While playing around with pomegranate v1 I run into this warning:
/home/josalhor/Desktop/data-synt/venv/lib/python3.11/site-packages/pomegranate/_utils.py:62: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at ../torch/csrc/utils/tensor_new.cpp:261.)I think my pull requests fixes that warning and should speed up the casting.