Code Crashing in Pedometer

[XOnanoSmartfoam]

I am trying to work through your code and I cannot get past line 79 in data_utils.py
p = Pedometer(gx=acc_x, gy=acc_y, gz=acc_z, sr=sampling_rate)

It gives the following errors:

AssertionError Traceback (most recent call last)
File c:\Users\jakem\AppData\Local\Programs\Python\Python310\lib\site-packages\pandas\core\internals\construction.py:969, in _finalize_columns_and_data(content, columns, dtype)
968 try:
--> 969 columns = _validate_or_indexify_columns(contents, columns)
970 except AssertionError as err:
971 # GH#26429 do not raise user-facing AssertionError

File c:\Users\jakem\AppData\Local\Programs\Python\Python310\lib\site-packages\pandas\core\internals\construction.py:1017, in _validate_or_indexify_columns(content, columns)
1015 if not is_mi_list and len(columns) != len(content): # pragma: no cover
1016 # caller's responsibility to check for this...
-> 1017 raise AssertionError(
1018 f"{len(columns)} columns passed, passed data had "
1019 f"{len(content)} columns"
1020 )
1021 elif is_mi_list:
1022
1023 # check if nested list column, length of each sub-list should be equal

AssertionError: 3 columns passed, passed data had 61773 columns

The above exception was the direct cause of the following exception:

ValueError Traceback (most recent call last)
Cell In[12], line 2
1 #Training Set
----> 2 X, Y_disp, Y_head, Y_pos, x0_list, y0_list, size_of_each, x_vel, y_vel, head_s, head_c, X_orig = import_oxiod_dataset(type_flag = 2,
3 useMagnetometer = True, useStepCounter = True, AugmentationCopies = 0,
4 dataset_folder = f,
5 sub_folders = ['handbag/','handheld/','pocket/','running/','slow_walking/','trolley/'],
6 sampling_rate = sampling_rate,
7 window_size = window_size, stride = stride, verbose=False)

File c:\Users\jakem\OneDrive - xonano.com\Documents\Python Scripts\TinyOdom_OxIDOD\data_utils.py:81, in import_oxiod_dataset(type_flag, useMagnetometer, useStepCounter, AugmentationCopies, dataset_folder, sub_folders, sampling_rate, window_size, stride, verbose)
78 mag_z = cur_train['Mag_Z'].to_numpy().reshape((acc_x.shape[0],1))
80 if(useStepCounter):
---> 81 p = Pedometer(gx=acc_x, gy=acc_y, gz=acc_z, sr=sampling_rate, data=None)
82 step_count, step_locations = p.get_steps()
83 loc = np.zeros(cur_train.shape[0])

File c:\Users\jakem\AppData\Local\Programs\Python\Python310\lib\site-packages\pydometer_init_.py:54, in Pedometer.init(self, gx, gy, gz, sr, data)
52 self.sr = 1.0/secs_per_sample
53 elif gx is not None and gy is not None and gz is not None:
---> 54 self.data = pd.DataFrame([gx, gy, gz], columns=['gx','gy','gz'])
55 else:
56 raise ValueError("Must provide data, or gx/gy/gz.")

File c:\Users\jakem\AppData\Local\Programs\Python\Python310\lib\site-packages\pandas\core\frame.py:746, in DataFrame.init(self, data, index, columns, dtype, copy)
744 if columns is not None:
745 columns = ensure_index(columns)
--> 746 arrays, columns, index = nested_data_to_arrays(
747 # error: Argument 3 to "nested_data_to_arrays" has incompatible
748 # type "Optional[Collection[Any]]"; expected "Optional[Index]"
749 data,
750 columns,
751 index, # type: ignore[arg-type]
752 dtype,
753 )
754 mgr = arrays_to_mgr(
755 arrays,
756 columns,
(...)
759 typ=manager,
760 )
761 else:

File c:\Users\jakem\AppData\Local\Programs\Python\Python310\lib\site-packages\pandas\core\internals\construction.py:510, in nested_data_to_arrays(data, columns, index, dtype)
507 if is_named_tuple(data[0]) and columns is None:
508 columns = ensure_index(data[0]._fields)
--> 510 arrays, columns = to_arrays(data, columns, dtype=dtype)
511 columns = ensure_index(columns)
513 if index is None:

File c:\Users\jakem\AppData\Local\Programs\Python\Python310\lib\site-packages\pandas\core\internals\construction.py:875, in to_arrays(data, columns, dtype)
872 data = [tuple(x) for x in data]
873 arr = _list_to_arrays(data)
--> 875 content, columns = _finalize_columns_and_data(arr, columns, dtype)
876 return content, columns

File c:\Users\jakem\AppData\Local\Programs\Python\Python310\lib\site-packages\pandas\core\internals\construction.py:972, in _finalize_columns_and_data(content, columns, dtype)
969 columns = validate_or_indexify_columns(contents, columns)
970 except AssertionError as err:
971 # GH#26429 do not raise user-facing AssertionError
--> 972 raise ValueError(err) from err
974 if len(contents) and contents[0].dtype == np.object:
975 contents = _convert_object_array(contents, dtype=dtype)

ValueError: 3 columns passed, passed data had 61773 columns

[swapnilsayansaha]

Ya it's an issue with pydometer package. Go to line 54 (your debug log says this is where the error is) of the following file: c:\Users\jakem\AppData\Local\Programs\Python\Python310\lib\site-packages\pydometer_init_.py

Replace these lines:

        elif gx is not None and gy is not None and gz is not None:
            self.data = pd.DataFrame([gx, gy, gz], columns=['gx', 'gy', 'gz'])

with these lines:

        elif gx is not None and gy is not None and gz is not None:
            self.data = pd.DataFrame(np.concatenate((gx.reshape(gx.shape[0],1),gy.reshape(gy.shape[0],1),gz.reshape(gz.shape[0],1)),axis=1), columns=['gx','gy','gz'])

That should solve the issue.

[XOnanoSmartfoam]

Thank you very much @swapnilsayansaha!

[XOnanoSmartfoam]

@swapnilsayansaha, Do you have a list of the datasets that were used in the training, Validation, and Test sets? It seems as though there are some paths that are incomplete and only reference the folder not the file. I could just run it with random data used in each but I would like to try and replicate your results to verify it is working correctly.

Thank you very much in advance for your help!

[swapnilsayansaha]

Check the paper: https://dl.acm.org/doi/pdf/10.1145/3534594. We used OxIOD, RoNIN, AQUALOC, EuROC MAV, and GunDog datasets. You can get the dataset download instructions in dataset_download_and_splits folder in each of the 5 folders.

I don't exactly remember the splits I used. I do know that at least for the RoNiN dataset, I used the original splits provided in the RoNIN dataset. Also for all the datasets, I think I gave example split file names in dataset_download_and_splits folders in each dataset folder. Follow the instructions in those folders to properly arrange the directories of each of the 5 datasets.