Error when loading template_metrics extension

[jonpedros]

I've been getting some errors when computing/loading extensions from a SortingAnalyzer containig a template_metrics extension. I calculate extensions from a dict as follows:

phy_exts = {"random_spikes": {},
            "waveforms": {},
            "templates":{},
            "noise_levels": {},
            "principal_components": {"n_components":5, "mode":"by_channel_local"},
            "template_similarity": {"method": "cosine_similarity"},
            "spike_amplitudes": {},
            "quality_metrics": {},
            "template_metrics": {}}

analyzer_obj.compute(phy_exts)

After successfully computing waveforms, fitting PCAs and projecting waveforms, I get these errors:

c:\Users\systemses\Anaconda3\envs\si_env\Lib\site-packages\spikeinterface\core\sortinganalyzer.py:1924: UserWarning: Found no run_info file for template_metrics, extension should be re-computed.
  warnings.warn(f"Found no run_info file for {self.extension_name}, extension should be re-computed.")
c:\Users\systemses\Anaconda3\envs\si_env\Lib\site-packages\spikeinterface\core\sortinganalyzer.py:1931: UserWarning: Found no run_info file for template_metrics, extension should be re-computed.
  warnings.warn(f"Found no run_info file for {self.extension_name}, extension should be re-computed.")
c:\Users\systemses\Anaconda3\envs\si_env\Lib\site-packages\spikeinterface\core\sortinganalyzer.py:2004: UserWarning: Found no data for template_metrics, extension should be re-computed.
  warnings.warn(f"Found no data for {self.extension_name}, extension should be re-computed.")

After which it will proceed to calculate spike_amplitudes successfully but then hang indefinitely.
I couldn't manually delete the folder with the waveforms extension, so code seemed to have hang there.

[chrishalcrow]

Hello,
Thanks for posting the issue. I can't reproduce the error. Here was my attempt to reproduce:

import spikeinterface.full as si

rec, sort = si.generate_ground_truth_recording()
analyzer_obj = si.create_sorting_analyzer(recording=rec, sorting=sort, format="binary_folder", folder="test_analyzer")

phy_exts = {"random_spikes": {},
            "waveforms": {},
            "templates":{},
            "noise_levels": {},
            "principal_components": {"n_components":5, "mode":"by_channel_local"},
            "template_similarity": {"method": "cosine_similarity"},
            "spike_amplitudes": {},
            "quality_metrics": {},
            "template_metrics": {}}

analyzer_obj.compute(phy_exts)

analyzer_2 = si.load_sorting_analyzer("test_analyzer")
analyzer_2.compute(phy_exts)

I get the warning, but the spike amplitudes are successfully computed, then it computes the pc_metrics. It'd be good if I could reproduce your problem. Do you know which extensions were successfully computed for the analyzer that you're loading?

I'll look into the warning, which seems like it shouldn't be there.

[zm711]

Hey Chris are you on windows? Maybe I should test. There can some folder release issues with Windows that doesn't happen on Mac/Linux (which I think you mostly test from). I'm not free until tomorrow, but happy to also troubleshoot then unless you have a windows computer to work on :)

[jonpedros]

If I disable computing of both template_metrics and quality_metrics, code runs fine. If I disable only template_metrics, code still hangs indefinitely but I get this error at some point:

[c:\Users\systemses\Anaconda3\envs\si_env\Lib\site-packages\spikeinterface\qualitymetrics\misc_metrics.py:908](file:///C:/Users/systemses/Anaconda3/envs/si_env/Lib/site-packages/spikeinterface/qualitymetrics/misc_metrics.py:908): UserWarning: Some units have too few spikes : amplitude_cutoff is set to NaN
  warnings.warn(f"Some units have too few spikes : amplitude_cutoff is set to NaN")
[c:\Users\systemses\Anaconda3\envs\si_env\Lib\site-packages\numpy\core\_methods.py:206](file:///C:/Users/systemses/Anaconda3/envs/si_env/Lib/site-packages/numpy/core/_methods.py:206): RuntimeWarning: Degrees of freedom <= 0 for slice
  ret = _var(a, axis=axis, dtype=dtype, out=out, ddof=ddof,
[c:\Users\systemses\Anaconda3\envs\si_env\Lib\site-packages\numpy\core\_methods.py:163](file:///C:/Users/systemses/Anaconda3/envs/si_env/Lib/site-packages/numpy/core/_methods.py:163): RuntimeWarning: invalid value encountered in divide
  arrmean = um.true_divide(arrmean, div, out=arrmean,
[c:\Users\systemses\Anaconda3\envs\si_env\Lib\site-packages\numpy\core\_methods.py:198](file:///C:/Users/systemses/Anaconda3/envs/si_env/Lib/site-packages/numpy/core/_methods.py:198): RuntimeWarning: invalid value encountered in divide
  ret = ret.dtype.type(ret / rcount)

Provided cone snippet throws warnings but indeed runs until pc_metric calculation, but is interrupted by the following error:

---------------------------------------------------------------------------
PermissionError                           Traceback (most recent call last)
Cell In[1], [line 19](vscode-notebook-cell:?execution_count=1&line=19)
     [16](vscode-notebook-cell:?execution_count=1&line=16) analyzer_obj.compute(phy_exts)
     [18](vscode-notebook-cell:?execution_count=1&line=18) analyzer_2 = si.load_sorting_analyzer("test_analyzer")
---> [19](vscode-notebook-cell:?execution_count=1&line=19) analyzer_2.compute(phy_exts)

File c:\Users\systemses\Anaconda3\envs\si_env\Lib\site-packages\spikeinterface\core\sortinganalyzer.py:1328, in SortingAnalyzer.compute(self, input, save, extension_params, verbose, **kwargs)
   [1326](file:///C:/Users/systemses/Anaconda3/envs/si_env/Lib/site-packages/spikeinterface/core/sortinganalyzer.py:1326)     params_, job_kwargs = split_job_kwargs(kwargs)
   [1327](file:///C:/Users/systemses/Anaconda3/envs/si_env/Lib/site-packages/spikeinterface/core/sortinganalyzer.py:1327)     assert len(params_) == 0, "Too many arguments for SortingAnalyzer.compute_several_extensions()"
-> [1328](file:///C:/Users/systemses/Anaconda3/envs/si_env/Lib/site-packages/spikeinterface/core/sortinganalyzer.py:1328)     self.compute_several_extensions(extensions=input, save=save, verbose=verbose, **job_kwargs)
   [1329](file:///C:/Users/systemses/Anaconda3/envs/si_env/Lib/site-packages/spikeinterface/core/sortinganalyzer.py:1329) elif isinstance(input, list):
   [1330](file:///C:/Users/systemses/Anaconda3/envs/si_env/Lib/site-packages/spikeinterface/core/sortinganalyzer.py:1330)     params_, job_kwargs = split_job_kwargs(kwargs)

File c:\Users\systemses\Anaconda3\envs\si_env\Lib\site-packages\spikeinterface\core\sortinganalyzer.py:1443, in SortingAnalyzer.compute_several_extensions(self, extensions, save, verbose, **job_kwargs)
   [1441](file:///C:/Users/systemses/Anaconda3/envs/si_env/Lib/site-packages/spikeinterface/core/sortinganalyzer.py:1441) for extension_name in sorted_extensions.keys():
   [1442](file:///C:/Users/systemses/Anaconda3/envs/si_env/Lib/site-packages/spikeinterface/core/sortinganalyzer.py:1442)     for child in _get_children_dependencies(extension_name):
-> [1443](file:///C:/Users/systemses/Anaconda3/envs/si_env/Lib/site-packages/spikeinterface/core/sortinganalyzer.py:1443)         self.delete_extension(child)
   [1445](file:///C:/Users/systemses/Anaconda3/envs/si_env/Lib/site-packages/spikeinterface/core/sortinganalyzer.py:1445) extensions_with_pipeline = {}
   [1446](file:///C:/Users/systemses/Anaconda3/envs/si_env/Lib/site-packages/spikeinterface/core/sortinganalyzer.py:1446) extensions_without_pipeline = {}

File c:\Users\systemses\Anaconda3\envs\si_env\Lib\site-packages\spikeinterface\core\sortinganalyzer.py:1616, in SortingAnalyzer.delete_extension(self, extension_name)
   [1613](file:///C:/Users/systemses/Anaconda3/envs/si_env/Lib/site-packages/spikeinterface/core/sortinganalyzer.py:1613) if self.format != "memory" and self.has_extension(extension_name):
   [1614](file:///C:/Users/systemses/Anaconda3/envs/si_env/Lib/site-packages/spikeinterface/core/sortinganalyzer.py:1614)     # need a reload to reset the folder
   [1615](file:///C:/Users/systemses/Anaconda3/envs/si_env/Lib/site-packages/spikeinterface/core/sortinganalyzer.py:1615)     ext = self.load_extension(extension_name)
-> [1616](file:///C:/Users/systemses/Anaconda3/envs/si_env/Lib/site-packages/spikeinterface/core/sortinganalyzer.py:1616)     ext.delete()
   [1618](file:///C:/Users/systemses/Anaconda3/envs/si_env/Lib/site-packages/spikeinterface/core/sortinganalyzer.py:1618) # remove from dict
   [1619](file:///C:/Users/systemses/Anaconda3/envs/si_env/Lib/site-packages/spikeinterface/core/sortinganalyzer.py:1619) self.extensions.pop(extension_name, None)

File c:\Users\systemses\Anaconda3\envs\si_env\Lib\site-packages\spikeinterface\core\sortinganalyzer.py:2306, in AnalyzerExtension.delete(self)
   [2302](file:///C:/Users/systemses/Anaconda3/envs/si_env/Lib/site-packages/spikeinterface/core/sortinganalyzer.py:2302) def delete(self):
   [2303](file:///C:/Users/systemses/Anaconda3/envs/si_env/Lib/site-packages/spikeinterface/core/sortinganalyzer.py:2303)     """
   [2304](file:///C:/Users/systemses/Anaconda3/envs/si_env/Lib/site-packages/spikeinterface/core/sortinganalyzer.py:2304)     Delete the extension from the folder or zarr and from the dict.
   [2305](file:///C:/Users/systemses/Anaconda3/envs/si_env/Lib/site-packages/spikeinterface/core/sortinganalyzer.py:2305)     """
-> [2306](file:///C:/Users/systemses/Anaconda3/envs/si_env/Lib/site-packages/spikeinterface/core/sortinganalyzer.py:2306)     self._delete_extension_folder()
   [2307](file:///C:/Users/systemses/Anaconda3/envs/si_env/Lib/site-packages/spikeinterface/core/sortinganalyzer.py:2307)     self.params = None
   [2308](file:///C:/Users/systemses/Anaconda3/envs/si_env/Lib/site-packages/spikeinterface/core/sortinganalyzer.py:2308)     self.run_info = self._default_run_info_dict()

File c:\Users\systemses\Anaconda3\envs\si_env\Lib\site-packages\spikeinterface\core\sortinganalyzer.py:2292, in AnalyzerExtension._delete_extension_folder(self)
   [2290](file:///C:/Users/systemses/Anaconda3/envs/si_env/Lib/site-packages/spikeinterface/core/sortinganalyzer.py:2290)     extension_folder = self._get_binary_extension_folder()
   [2291](file:///C:/Users/systemses/Anaconda3/envs/si_env/Lib/site-packages/spikeinterface/core/sortinganalyzer.py:2291)     if extension_folder.is_dir():
-> [2292](file:///C:/Users/systemses/Anaconda3/envs/si_env/Lib/site-packages/spikeinterface/core/sortinganalyzer.py:2292)         shutil.rmtree(extension_folder)
   [2294](file:///C:/Users/systemses/Anaconda3/envs/si_env/Lib/site-packages/spikeinterface/core/sortinganalyzer.py:2294) elif self.format == "zarr":
   [2295](file:///C:/Users/systemses/Anaconda3/envs/si_env/Lib/site-packages/spikeinterface/core/sortinganalyzer.py:2295)     import zarr

File c:\Users\systemses\Anaconda3\envs\si_env\Lib\shutil.py:787, in rmtree(path, ignore_errors, onerror, dir_fd)
    [785](file:///C:/Users/systemses/Anaconda3/envs/si_env/Lib/shutil.py:785)     # can't continue even if onerror hook returns
    [786](file:///C:/Users/systemses/Anaconda3/envs/si_env/Lib/shutil.py:786)     return
--> [787](file:///C:/Users/systemses/Anaconda3/envs/si_env/Lib/shutil.py:787) return _rmtree_unsafe(path, onerror)

File c:\Users\systemses\Anaconda3\envs\si_env\Lib\shutil.py:634, in _rmtree_unsafe(path, onerror)
    [632](file:///C:/Users/systemses/Anaconda3/envs/si_env/Lib/shutil.py:632)             os.unlink(fullname)
    [633](file:///C:/Users/systemses/Anaconda3/envs/si_env/Lib/shutil.py:633)         except OSError:
--> [634](file:///C:/Users/systemses/Anaconda3/envs/si_env/Lib/shutil.py:634)             onerror(os.unlink, fullname, sys.exc_info())
    [635](file:///C:/Users/systemses/Anaconda3/envs/si_env/Lib/shutil.py:635) try:
    [636](file:///C:/Users/systemses/Anaconda3/envs/si_env/Lib/shutil.py:636)     os.rmdir(path)

File c:\Users\systemses\Anaconda3\envs\si_env\Lib\shutil.py:632, in _rmtree_unsafe(path, onerror)
    [630](file:///C:/Users/systemses/Anaconda3/envs/si_env/Lib/shutil.py:630) else:
    [631](file:///C:/Users/systemses/Anaconda3/envs/si_env/Lib/shutil.py:631)     try:
--> [632](file:///C:/Users/systemses/Anaconda3/envs/si_env/Lib/shutil.py:632)         os.unlink(fullname)
    [633](file:///C:/Users/systemses/Anaconda3/envs/si_env/Lib/shutil.py:633)     except OSError:
    [634](file:///C:/Users/systemses/Anaconda3/envs/si_env/Lib/shutil.py:634)         onerror(os.unlink, fullname, sys.exc_info())

PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'test_analyzer\\extensions\\waveforms\\waveforms.npy'

[chrishalcrow]

I managed to debug the errors, I think! This could be the cause of the hanging, or maybe not. I would appreciate it if you could try out the newly linked PR #3478 - thanks!

[jonahpearl]

I couldn't manually delete the folder with the waveforms extension, so code seemed to have hang there.

See #3348 — maybe the same issue? (Not that we have a solution yet, but in case you want to follow along.) And I'm able to compute quality + template metrics tho with no issues even tho I do usually run into the waveform folder issue — and that's on Linux — so I don't think that's the issue here.