Question about result on DAVIS16

[yongliu20]

I want ask why I get this result using your pre-trained model? Thanks!

[hkchengrex]

That doesn't look right... Can I see some output images?

[hkchengrex]

Specifically, you can compare them with our pre-computed results.

[yongliu20]

The results of 'blackswan' are these:

[yongliu20]

I also think this J&F result is not right but I can not find out what went wrong...

[hkchengrex]

"blackswan" will almost always be good because it is so easy... Can you download our pre-computed results and see if there are any differences? I want to check whether the problem lies in generation or evaluation.

[yongliu20]

Ok, I will do it now

[yongliu20]

There are indeed some differences between them but very small. And I test your pre-computed results, the J&F is:

The evaluation method which I use is from davis.

[yongliu20]

I find that my result is as same as the D16_s012. And the above result belongs to D16_s012_notop.

[hkchengrex]

1. I just evaluated again and I got the same reported number (90.9 for D16_s012_notop). To check that my evaluation code is correct, I downloaded STM's results (https://github.com/seoungwugoh/STM) (mapped from 0~1 to 0~255) and evaluated them in my environment. I got the same result as the reported number in STM. So most likely your evaluation has some problems.
2. Visually, the results also look closer to 90 than 75.
3. I also found that I swapped two rows (D16_s02, D16_s012_notop) in the table in README. I will fix that, thanks for the hint.

DAVIS 2016 evaluation code is not very well maintained. It was not easy to get it right for me back then... I recalled there are some discussion threads about a proper implementation. I am going to check on those and get back to you.

[hkchengrex]

Here: davisvideochallenge/davis2017-evaluation#4
Hope it helps.

You can always check the numbers with ours/STM's.
I might open source my own evaluation code later.

[yongliu20]

Well, I have solved this problem by modifying your eval_davis_2016.py, like this:

[hkchengrex]

Well yeah if your evaluation script expects 0/1 outputs... The ground truths in DAVIS 2016 are 0/255 so I'm sticking with that. Glad that it has been fixed.

[hkchengrex]

Hmm I think I can actually modify the code a bit to make both happy. Gonna do that. Thanks.

[yongliu20]

My previous problem is that the output values of the foreground pixels are not same. And thanks for your help!