including startup time makes the results misleading

[stevengj]

In Julia, and perhaps in some of the other languages, you are mostly measuring the one-time startup cost, rather than the cost of actually doing the mathematical calculation. So the results are misleading as a guide for speed of realistic number-crunching tasks, since compute-heavy problems inevitably run for long enough that the startup time is negligible (whether or not you run them as a script). In comparison, your benchmark runs in less than a second, so startup overhead is artificially inflated.

In Julia's case, this startup cost is especially heavy because it includes a one-time cost of compiling the function you are benchmarking. I see that you also have a "Julia (AOT Compiled)" test which removes a big chunk of this startup cost by using StaticCompiler.jl (although this is not a typical way to run Julia). But any benchmark that includes startup cost is misleading as a guide to numerical performance. Why include any misleading results?

Recommendation: do the timing within each language's script (all languages offer high-resolution timers), and print out the elapsed compute time for calling f(rounds) only. In Julia's case, call the function once (e.g. with rounds=1) before beginning the timing, to trigger its compilation. That way, you are only reporting the actual compute cost. (Better yet, run the timing in loop a few times and report the minimum time, which cuts down on noise.)

(An alternative would be to increase the cost of the benchmark so that it takes tens of seconds to run or more, so that the startup time becomes more negligible like in a more realistic problem. But it's better — more reliable, and faster — to do timing inside the benchmark code itself.)

[Moelf]

#22