Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Rank of updates array (1) must be greater than or equal to inner-most dimension of indices array (29) #285

Open
wsnoble opened this issue Jan 29, 2024 · 3 comments
Open

Rank of updates array (1) must be greater than or equal to inner-most dimension of indices array (29) #285

wsnoble opened this issue Jan 29, 2024 · 3 comments
Labels
bug Something isn't working macos

Comments

@wsnoble
Copy link
Contributor

wsnoble commented Jan 29, 2024

On an M2 Mac, I am getting the error shown below. I do have the fallback variable set properly:

% echo $PYTORCH_ENABLE_MPS_FALLBACK
1

I am using the following version:

Casanovo: 4.0.1
Depthcharge: 0.2.3
Lightning: 2.1.3
PyTorch: 2.1.2

The input MGF file is 151009_exo4_4.short.mgf.txt

% casanovo sequence 151009_exo4_4.short.mgf 
WARNING: Dataloader multiprocessing is currently not supported on Windows or MacOS; using only a single thread.
Seed set to 454
INFO: Model weights file /Users/wnoble/Library/Caches/casanovo/casanovo_massivekb_v4_0_0.ckpt retrieved from local cache
INFO: Casanovo version 4.0.1
INFO: Sequencing peptides from:
INFO:   151009_exo4_4.short.mgf
GPU available: True (mps), used: True
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs
INFO: Reading 1 files...
151009_exo4_4.short.mgf: 2000spectra [00:00, 20073.48spectra/s]
WARNING: Skipped 1282 spectra with invalid precursor info
Predicting DataLoader 0:   0%|                            | 0/1 [00:00<?, ?it/s]WARNING: UserWarning: The operator 'aten::_nested_tensor_from_mask_left_aligned' is not currently supported on the MPS backend and will fall back to run on the CPU. This may have performance implications. (Triggered internally at /Users/runner/work/pytorch/pytorch/pytorch/aten/src/ATen/mps/MPSFallback.mm:13.)
WARNING: UserWarning: The operator 'aten::_nested_tensor_from_mask_left_aligned' is not currently supported on the MPS backend and will fall back to run on the CPU. This may have performance implications. (Triggered internally at /Users/runner/work/pytorch/pytorch/pytorch/aten/src/ATen/mps/MPSFallback.mm:13.)
/AppleInternal/Library/BuildRoots/0032d1ee-80fd-11ee-8227-6aecfccc70fe/Library/Caches/com.apple.xbs/Sources/MetalPerformanceShaders/MPSNDArray/Kernels/MPSNDArrayGatherND.mm:256: failed assertion `Rank of updates array (1) must be greater than or equal to inner-most dimension of indices array (29)'
zsh: abort      casanovo sequence 151009_exo4_4.short.mgf
/opt/homebrew/Caskroom/miniconda/base/envs/casanovo/lib/python3.10/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '
@wsnoble wsnoble added the bug Something isn't working label Jan 29, 2024
@bittremieux bittremieux mentioned this issue Feb 10, 2024
Closed
@wsnoble
Copy link
Contributor Author

wsnoble commented Feb 17, 2024

Update: this error does not appear when I don't use the GPU.

@Lilferrit Lilferrit self-assigned this Aug 6, 2024
@Lilferrit Lilferrit removed their assignment Sep 17, 2024
@loganjohnson0
Copy link

Excited to try out Casanovo and thanks for the great documentation. I also have a similar error, which seems to impact M series Macs in other pipelines as well, likely related to lack of memory or memory allocation. My process was killed almost immediately after the start of the search on the example mgf file from the documetation. Looking forward to your future developments!

Apple M2 Pro
16 GB Memory 
MacOS Sequoia 15.1

(casanovo_env) MacBook-Pro-2:~ logan.johnson$ conda list python
# packages in environment at /Users/logan.johnson/opt/anaconda3/envs/casanovo_env:
#
# Name                    Version                   Build  Channel
python                    3.10.15              hce00570_1  

(casanovo_env) MacBook-Pro-2:~ logan.johnson$ echo $PYTORCH_ENABLE_MPS_FALLBACK
1
(casanovo_env) MacBook-Pro-2:~ logan.johnson$ casanovo sequence /Users/logan.johnson/Downloads/sample_preprocessed_spectra.mgf
WARNING: Dataloader multiprocessing is currently not supported on Windows or MacOS; using only a single thread.
Seed set to 454
INFO: Model weights file /Users/logan.johnson/Library/Caches/casanovo/casanovo_v4_2_0_v4_2_0.ckpt retrieved from local cache
INFO: Casanovo version 4.2.1
INFO: Sequencing peptides from:
INFO:   /Users/logan.johnson/Downloads/sample_preprocessed_spectra.mgf
GPU available: True (mps), used: True
TPU available: False, using: 0 TPU cores
HPU available: False, using: 0 HPUs
INFO: Reading 1 files...
/Users/logan.johnson/Downloads/sample_preprocessed_spectra.mgf: 128spectra [00:00, 13661.88spectra/s]
Predicting DataLoader 0:   0%|                                                                                     | 0/1 [00:00<?, ?it/s]WARNING: UserWarning: The operator 'aten::_nested_tensor_from_mask_left_aligned' is not currently supported on the MPS backend and will fall back to run on the CPU. This may have performance implications. (Triggered internally at /Users/runner/work/pytorch/pytorch/pytorch/aten/src/ATen/mps/MPSFallback.mm:13.)
WARNING: UserWarning: The operator 'aten::_nested_tensor_from_mask_left_aligned' is not currently supported on the MPS backend and will fall back to run on the CPU. This may have performance implications. (Triggered internally at /Users/runner/work/pytorch/pytorch/pytorch/aten/src/ATen/mps/MPSFallback.mm:13.)
/AppleInternal/Library/BuildRoots/4b66fb3c-7dd0-11ef-b4fb-4a83e32a47e1/Library/Caches/com.apple.xbs/Sources/MetalPerformanceShaders/MPSNDArray/Kernels/MPSNDArrayGatherND.mm:255: failed assertion `Rank of updates array (1) must be greater than or equal to inner-most dimension of indices array (29)'
Abort trap: 6
(casanovo_env) MacBook-Pro-2:~ logan.johnson$ /Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '

casanovo_20241112120802.log

@wsnoble
Copy link
Contributor Author

wsnoble commented Nov 13, 2024

FWIW, I upgraded to v4.2.1 and ran the test above on my M2, but I get a different error. I'm on Python 3.10.12.

>  casanovo sequence sample_preprocessed_spectra.mgf
WARNING: Dataloader multiprocessing is currently not supported on Windows or MacOS; using only a single thread.
Seed set to 454
INFO: Downloading model weights file /Users/wnoble/Library/Caches/casanovo/casanovo_v4_2_0_v4_2_0.ckpt from https://github.com/Noble-Lab/casanovo/releases/download/v4.2.0/casanovo_v4_2_0.ckpt
100%|████████████████████████████████████████| 542M/542M [02:20<00:00, 4.06MB/s]
INFO: Casanovo version 4.2.1
INFO: Sequencing peptides from:
INFO:   sample_preprocessed_spectra.mgf
GPU available: True (mps), used: True
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs
WARNING: /opt/homebrew/Caskroom/miniconda/base/envs/casanovo/lib/python3.10/site-packages/lightning/pytorch/utilities/migration/utils.py:55: The loaded checkpoint was produced with Lightning v2.2.1, which is newer than your current Lightning version: v2.1.3

WARNING: /opt/homebrew/Caskroom/miniconda/base/envs/casanovo/lib/python3.10/site-packages/lightning/pytorch/utilities/migration/utils.py:55: The loaded checkpoint was produced with Lightning v2.2.1, which is newer than your current Lightning version: v2.1.3

INFO: Reading 1 files...
sample_preprocessed_spectra.mgf: 0spectra [00:00, ?spectra/s]
Traceback (most recent call last):
  File "/opt/homebrew/Caskroom/miniconda/base/envs/casanovo/bin/casanovo", line 8, in <module>
    sys.exit(main())
  File "/opt/homebrew/Caskroom/miniconda/base/envs/casanovo/lib/python3.10/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/opt/homebrew/Caskroom/miniconda/base/envs/casanovo/lib/python3.10/site-packages/rich_click/rich_command.py", line 126, in main
    rv = self.invoke(ctx)
  File "/opt/homebrew/Caskroom/miniconda/base/envs/casanovo/lib/python3.10/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/opt/homebrew/Caskroom/miniconda/base/envs/casanovo/lib/python3.10/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/opt/homebrew/Caskroom/miniconda/base/envs/casanovo/lib/python3.10/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/opt/homebrew/Caskroom/miniconda/base/envs/casanovo/lib/python3.10/site-packages/casanovo/casanovo.py", line 143, in sequence
    runner.predict(peak_path, output)
  File "/opt/homebrew/Caskroom/miniconda/base/envs/casanovo/lib/python3.10/site-packages/casanovo/denovo/model_runner.py", line 160, in predict
    test_index = self._get_index(peak_path, False, "")
  File "/opt/homebrew/Caskroom/miniconda/base/envs/casanovo/lib/python3.10/site-packages/casanovo/denovo/model_runner.py", line 394, in _get_index
    return Index(index_fname, filenames, valid_charge=valid_charge)
  File "/opt/homebrew/Caskroom/miniconda/base/envs/casanovo/lib/python3.10/site-packages/depthcharge/data/hdf5.py", line 104, in __init__
    self.add_file(ms_file)
  File "/opt/homebrew/Caskroom/miniconda/base/envs/casanovo/lib/python3.10/site-packages/depthcharge/data/hdf5.py", line 192, in add_file
    parser.read()
  File "/opt/homebrew/Caskroom/miniconda/base/envs/casanovo/lib/python3.10/site-packages/depthcharge/data/parsers.py", line 98, in read
    self.mz_arrays = np.concatenate(self.mz_arrays).astype(np.float64)
  File "<__array_function__ internals>", line 200, in concatenate
ValueError: need at least one array to concatenate

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working macos
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@Lilferrit @wsnoble @loganjohnson0