Time dimension in audio datasets after applying ToFrame transform

[msbouanane]

Hello,

I am trying to train an SNN using the SHD and NTIDIGITS datasets. Since binning is necessary to specify the temporal resolution
of the spiking input, I am trying to use ToFrame to slice the data along constant time. However when I pass it the DataLoader, no time dimension appears in the event tensors. Here are 2 screenshots on how ToFrame transform behave differently when applied to a vision dataset (NMNIST) and audio dataset (SHD or NTIDIGITS):

How to overcome this problem?

[biphasic]

Hey,
The problem is a different one: you first apply a downsample transform with default arguments, which will divide all timestamps by 1000, turning them from microseconds to miliseconds. Then, you apply the ToFrame transform with a time window of 1000. But most recordings are under 1s long! That's why you essentially binned all events of a recording into 1 frame. I recommend to always look at the raw data before transforming to frames. This short piece of code works:

import tonic
from torch.utils.data import DataLoader

transform = tonic.transforms.ToFrame(sensor_size=tonic.datasets.SHD.sensor_size, time_window=1000)

trainset = tonic.datasets.SHD(save_to='./data', train=True, transform=transform)

raster, targets = trainset[0]
print(raster.shape)

(700, 1, 700)

Your code also works, you just have to adjust the time_window parameter.

PS: if you post code instead of images, then I can easily copy it and re-run your examples, which makes it easier for me.

[msbouanane]

Thank you very much! Actually yes I did realize that I should not use both transforms together. However, I would like if you have any recommendation regarding the time_window for SHD. I used 1000 but there was no learning and it took an unbelievably long time to train on Google Colab. It took 3h 39m to train 40 epochs and accuracy was stagnated at around 20% since the 12th epoch. (I am using snnTorch)

And when I use ToFrame with NTIDIGITS, I keep getting this error:

frame_transform = tonic.transforms.ToFrame(sensor_size=tonic.datasets.NTIDIGITS.sensor_size, time_window=1000)

trainset = tonic.datasets.NTIDIGITS(save_to='./data', transform=frame_transform, train=True)
trainloader = DataLoader(trainset, batch_size=128, collate_fn=tonic.collation.PadTensors())

event_tensor, target = next(iter(trainloader))
print(event_tensor.shape)

ValueError: too many dimensions 'str'

[biphasic]

Personally I used this paper as a reference. Check out Table 1 in the supplementary material: https://www.biorxiv.org/content/10.1101/2021.03.22.436372v2
They use 250 time steps max. So basically a time window of 4000 (microseconds).
As for the NTIDIGITS, this should not be happening. I'll open an issue for that, thanks for pointing it out!

[mosfe]

Hello, the dataset of ' tonic.datasets.NTIDIGITS' has not been found in the current version, is it been deleted?

[biphasic]

yes I deleted it after I looked into the dataset and found some samples missing and targets inconsistent. I wrote to Shi-Chi Liu about it but that didn't clear things up