MNN batch inference time not more efficient than single image

[danielbr33]

You can use C++ API to do it. How ever, it's not more efficient to use multi batch for inference.

Why is multi batch computing inefficient using MNN?

Is this true for all operators?

Originally posted by @mingyunzzu in #673

Can somebody explain why inference with batch isn't more efficient in MNN? When I run detection on single image it takes 7 miliseconds and when I run on batch of 32 images it takes 8 miliseconds per image. This is only the time of inference measured by time of runSession without preparing images and postprocessing. What can I use to reach better results?

[jxt1234]

1. The issue's bug has been resolved. Now batch inference time depend on the compute flops of device. If one image has reach the peak flops, then batch image will not more efficient.
2. The get the device's compute peak flops, can use ./run_test.out speed/MatMulBConst
3. Normally GPU has more flops than CPU, you can use opencl instead of CPU to forward.