Unable to reproduce results

[ghost]

Hi Akara,

I'm unable to reproduce the results as mentioned in the documentation of this repository.

Below are the steps that I have followed,

• python download_sleepedf.py
• python prepare_sleepedf.py
• python trainer.py --db sleepedf --gpu 0 --from_fold 0 --to_fold 19
• python predict.py --config_file config/sleepedf.py --model_dir out_sleepedf/train --output_dir out_sleepedf/predict --log_file out_sleepedf/predict.log --use-best

And following is final results section from predict.log file:

=== Overall ===
W: 10197
N1: 2804
N2: 17799
N3: 5703
REM: 7717
n=44220, acc=33.1, mf1=30.2
Confusion Matrix
[[3501 2895 1495 964 1342]
[ 280 834 953 270 467]
[1549 4599 6773 2761 2117]
[ 490 1388 1462 1746 617]
[ 735 2139 1741 1314 1788]]
Total: 44220
Number of samples from each class: [10197. 2804. 17799. 5703. 7717.]
Accuracy: 33.1
Macro F1-Score: 30.2
Per-class Precision: 53.4 7.0 54.5 24.7 28.2
Per-class Recall: 34.3 29.7 38.1 30.6 23.2
Per-class F1-Score: 41.8 11.4 44.8 27.4 25.5

Please let me know if there are configurations that have to be setup to reproduce the results.

[akaraspt]

@shyam-lab Can u show me the versions of all dependencies that you have installed in your environment? This can be done using pip freeze or pipreqs.

[ghost]

This has been evaluated in Amazon ec2 instance (g3s.xlarge) which comes pre-built with deep learning packages. And as mentioned in the repository, we have performed the run the code within conda environment.

ubuntu@ip-172-31-28-90:/tinysleepnet$ conda activate tinysleepnet
(tinysleepnet) ubuntu@ip-172-31-28-90:/tinysleepnet$ pip freeze
absl-py==0.12.0
astor==0.8.1
cached-property==1.5.2
certifi==2020.12.5
cycler==0.10.0
gast==0.4.0
grpcio==1.37.0
h5py==3.1.0
importlib-metadata==3.10.1
joblib==1.0.1
Keras-Applications==1.0.8
Keras-Preprocessing==1.1.2
kiwisolver==1.3.1
Markdown==3.3.4
matplotlib==3.3.4
mne==0.18.2
mock==4.0.3
numpy==1.19.5
pandas==1.1.5
Pillow==8.2.0
protobuf==3.15.8
pyEDFlib==0.1.19
pyparsing==2.4.7
python-dateutil==2.8.1
pytz==2021.1
scikit-learn==0.24.1
scipy==1.5.4
six==1.15.0
tensorboard==1.13.1
tensorflow-estimator==1.13.0
tensorflow-gpu==1.13.1
termcolor==1.1.0
threadpoolctl==2.1.0
typing-extensions==3.7.4.3
Werkzeug==1.0.1
wget==3.2
zipp==3.4.1

[quwei913]

Hey guys, has this issue been solved? I can't reproduce the result as well. My environment is as below:

(tiny) wequ0318@gpux4:/share/wequ0318/tinysleepnet$ conda list|grep cudnn
cudnn 7.6.5 cuda10.0_0
(tiny) wequ0318@gpux4:/share/wequ0318/tinysleepnet$ pip freeze
absl-py @ file:///opt/conda/conda-bld/absl-py_1639803114343/work
astor==0.8.1
certifi==2021.5.30
cycler==0.11.0
dataclasses @ file:///tmp/build/80754af9/dataclasses_1614363715916/work
dm-tree==0.1.6
future==0.17.1
gast @ file:///tmp/build/80754af9/gast_1637837472536/work
grpcio @ file:///tmp/build/80754af9/grpcio_1597424460877/work
h5py @ file:///tmp/build/80754af9/h5py_1593454121459/work
importlib-metadata @ file:///tmp/build/80754af9/importlib-metadata_1631916693255/work
install==1.3.4
Keras-Applications @ file:///tmp/build/80754af9/keras-applications_1594366238411/work
Keras-Preprocessing @ file:///tmp/build/80754af9/keras-preprocessing_1612283640596/work
kiwisolver==1.3.1
Markdown @ file:///tmp/build/80754af9/markdown_1614363833670/work
matplotlib==3.0.2
mkl-fft==1.3.0
mkl-random==1.1.1
mkl-service==2.3.0
mne==0.18.2
mock @ file:///tmp/build/80754af9/mock_1607622725907/work
numpy @ file:///tmp/build/80754af9/numpy_and_numpy_base_1603487797006/work
pandas==0.24.1
protobuf==3.17.2
pyEDFlib==0.1.19
pyparsing==3.0.9
python-dateutil==2.8.2
pytz==2022.1
scikit-learn==0.20.3
scipy==1.2.1
six @ file:///tmp/build/80754af9/six_1644875935023/work
tensorboard==1.13.1
tensorflow==1.13.1
tensorflow-estimator==1.13.0
tensorflow-probability==0.13.0
termcolor==1.1.0
typing_extensions @ file:///opt/conda/conda-bld/typing_extensions_1647553014482/work
Werkzeug @ file:///opt/conda/conda-bld/werkzeug_1645628268370/work
zipp @ file:///tmp/build/80754af9/zipp_1633618647012/work

[Lakshya-CB]

try this :

python download_sleepedf.py
python prepare_sleepedf.py
python trainer.py --db sleepedfx --gpu 0 --from_fold 0 --to_fold 9

Hats off man. good work. There is silly mistake :
your-download_sleepedf.py - actually downloads dataset from here:
https://www.physionet.org/files/sleep-edfx/1.0.0/sleep-cassette/

and to train on the whole dataset you need to load the parameters from config/sleepedfx.py
If you load the config/sleepedf.py only 20 subjects are picked. hence lower accuracy

Well done by budaaar!! good work honestly <3

[yangyuqing15715165798]

try this :

python download_sleepedf.py python prepare_sleepedf.py python trainer.py --db sleepedfx --gpu 0 --from_fold 0 --to_fold 9

Hats off man. good work. There is silly mistake : your-download_sleepedf.py - actually downloads dataset from here: https://www.physionet.org/files/sleep-edfx/1.0.0/sleep-cassette/

and to train on the whole dataset you need to load the parameters from config/sleepedfx.py If you load the config/sleepedf.py only 20 subjects are picked. hence lower accuracy

Well done by budaaar!! good work honestly <3
Do you have this problem? How to solve it
signals = psg_f.readSignal(select_ch_idx).reshape(-1, n_epoch_samples)
ValueError: cannot reshape array of size 7950000 into shape (90000)

[liu-yuyun]

Did you solve this problem? I checked the logs during the training process and found that several folds ended training early, and the resulting confusion matrix was incorrectly classified into the same class

[The-crucified]

@yangyuqing15715165798
Do you have this problem? How to solve it
signals = psg_f.readSignal(select_ch_idx).reshape(-1, n_epoch_samples)
ValueError: cannot reshape array of size 7950000 into shape (90000)

I modified this line of code. signals = psg_f.readSignal(select_ch_idx).reshape(-1, n_epoch_samples)
To signals = psg_f.readSignal(select_ch_idx).reshape(-1, int(sampling_rate)).
But I don't know if it's right. if you have solved this problem, can you tell me the method.