Low mIoU on Cityscapes

[fabvio]

Hi, thanks for sharing your code. I trained the model on the cityscapes dataset (btw, you missed a self here) without code edits and I can get only 68% mIoU. Do you have any pretrained models, or can you describe your training strategy? I trained on a single Tesla v100 GPU, with lr = 0.007 and batch size 8.
Thanks in advance

[jfzhang95]

What is the backbone you used in your model?
Actually I did not train the model on cityscapes dataset.

[fabvio]

The backbone is Resnet, as I have seen in another issue that Xception may have some problems. Am I right?

[jfzhang95]

Yes, previous code miss bn after depth wise cnn. I have fixed it. But I do not have pretrained aligned xception model right now. So I dont recommend you use this backbone.

[fabvio]

I think that the low performances could be related to the issue #20, because in your code you fill to 0, and not to ignore_index, causing the network to learn street features on the black part of the image.

[jfzhang95]

Hi, it is an error in previous branch, but in current branch, I have already fixed it.

[fabvio]

Are you sure about that? In the cityscapes dataset here you are referring to the method I linked before, with 0 filling, so you are actually telling your network that all the padding you have belongs to a class. This is on master, tell me if I got something wrong.

[jfzhang95]

Hi, I think you are right.
When training model on cityscapes dataset, mask should be padded to 255 instead of 0. Otherwise it will affect final performance.

[fabvio]

I will test the model trained with the updated code and report the performances to you. Right now, I am training with really resized images (512x256), batch size 16 and lr=0.007.

[fabvio]

Update on the results: very low mIoU (around 0.6). I think there is some additional problem with the code, because I trained also on another dataset (bdd100k) and the results are equally bad. Or maybe my training parameters are wrong, but they are similar to yours...

[ghost]

Same here... low mIoU (around 0.6), base_size = 1024, crop=600

[jfzhang95]

I tried to train ResNet based model on cityscapes dataset. My training setting is batch_size 16 (8 TITAN XP GPUs), base_size and crop_size 796, lr 0.01, epoch 200. Currently on epochs 50, and best mIoU is 0.65. So I think we need to tune training parameters in order to reach higher mIoU.

[jfzhang95]

Maybe you could explore pytorch-encoding or drn repos to try their training setting and data augmentation methods on Cityscapes dataset. I believe their code could help you.

[guanxiongsun]

Same Low IOU on cityscapes, 200 epoch, 8 GPUs, BS = 16, lr 0.01, base_size 1024, crop_size 769. The Final result is 67.98%

[jfzhang95]

Hi, all

I just noticed an issue in ESPNet. It seems like Cityscapes dataset script uses weighted mIoU. So maybe you can evaluate mIoU using the official script instead of script implemented in this repo.

[wyfeng1020]

But the use of whether general or weighted mIou should not cause such a large performance gap.

[herleeyandi]

@jfzhang95 @fabvio How to do a single inference?, I have train in cityscapes I got only 62% mIoU. After I am doing testing I got this result. Why the result seems very bad? I have doing the normalization first before feeding to the network like substracted mean and unit variance.

[yazhe2017]

Hi, thanks for the great repo. I try to train the cityscapes dataset with the suggested parameters. However, the performance of network is weird. The epoch loss decrease all the way. However, it is weird to note that the miou first rise then descend and the highest miou is 57.5%. Could you please try to give a suggestion on this? Thanks a lot.
My parameters setting is 4 v100 GPUs, bs=16, base_size=513, crop_size=513, backbone=resnet lr=0.01. I made additional trying with setting as 8 v100 GPUs, base_size=1024, crop_size=769. However, the performances for both experiments are similar.
By the way, I use the pytorch provided model (Imagenet) as the pretrained backbone.

[GeoffreyChen777]

This repo use nn.Dropout instead of nn.Dropout2d. Maybe this is one reason.

[EchoAmor]

This repo use nn.Dropout instead of nn.Dropout2d. Maybe this is one reason.

Hello ,have u solve this problem yet? can you share your results? I also only got 68% mIOU,and dont know how to improve . Thank you very much! @GeoffreyChen777

[yazhe2017]

I found the problem for my case probably caused by the function of SynchronizedBN. When I turn of the SynBN and use the nn.BatchNorm2d, I have the highest performance 72% in val dataset without coco pretrained. However, it is weird to note that the miou descends in the middle of training. By the way, my setting is bs=4, base_size=796, crop_size=796, backbone=resnet lr=0.01 1GPU(v100). The current miou is still far away from the state-of-the-art performance, maybe there are other problems.

[sreeragh-ar]

@jfzhang95

Hi, all

I just noticed an issue in ESPNet. It seems like Cityscapes dataset script uses weighted mIoU. So maybe you can evaluate mIoU using the official script instead of script implemented in this repo.

Trained the model on Cityscapes for 100 epochs, evaluated using official Cityscapes script. Obtained mIoU of 0.694.

Evaluation using the repo

The best mIoU obtained was 0.6816.

I used the whole image i.e, 1024*2048 during evaluation. The mIoU improved slightly to 0.6943

Evaluation using official Cityscapes scripts
Installed
pip install cityscapesScripts.
Saved the predictions (used input size 1024*2048) as images (after converting train ids to valid class ids) in a folder.
And used csEvalPixelLevelSemanticLabeling command.

The mIoU obtained was 0.694

[ShunZuo-AI]

What is the backbone you used in your model? Actually I did not train the model on cityscapes dataset.

the metrics Acc np.diag(self.confusion_matrix) / self.confusion_matrix.sum(axis=1) may be wrong,please check it.