Compiled vs interpreted language?

[sloanlance]

I've read, and @johnkerl recently explained to me, that Miller was originally written in C. Then, in the last few years, it was rewritten using Go. I've only recently started developing with Go (see #1366), but I already like it better than C. Therefore, the rewrite was a good move.

My favorite programming language is Python, but I also like PHP, to some degree Perl, and various other languages. Overall, I enjoy interpreted languages more than compiled ones. I used to use C and Java in my work, but I'm very happy I no longer use those on a regular basis.

I think that makes my bias clear. 😉

My questions are…

1. Is there a reason Miller is written in a compiled language?
   • Is it to offer better performance when processing large data sets?
2. Has there been an attempt to implement Miller in an interpreted language?
   • If there has been, was it rejected for poor performance?
   • Was the attempt recent? Depending on how long ago it was, the most recent computers and interpreters may perform much better than those of a few years ago.

I ask these questions largely for academic purposes. I don't expect anyone to suddenly port Miller to Python.

[johnkerl]

Hi @sloanlance -- some thoughts ...

TL;DR:

1. Is there a reason Miller is written in a compiled language?
   • Is it to offer better performance when processing large data sets? -- Yes
2. Has there been an attempt to implement Miller in an interpreted language?
   • If there has been, was it rejected for poor performance? -- Yes
   • Was the attempt recent? -- Eight years ago
   • Depending on how long ago it was, the most recent computers and interpreters may perform much better than those of a few years ago. -- Indeed, maybe so!

More info:

• This is a bit dated but still true: https://miller.readthedocs.io/en/latest/why/
• Older:
  • https://johnkerl.org/miller-docs-by-release/5.9.0/whyc.html
  • https://johnkerl.org/miller-docs-by-release/5.9.0/whyc-details.html
• My experience with Python was (and still is) it doesn't scale at all well to gigabyte datasets
  • One thing that can help is accelerating key parts in C++ and using pybind11. We do this where I currently work
  • I didn't want to do this with Miller
  • Candidly, working with multiplatform compiled binaries with PyPI, Conda, etc is nice for the end user but prohibitive for the time I have available as a spare-time open-source developer/author/owner/maintainer -- going this route is a non-starter for me
• I did do some very early on profilings -- it was not recent; 2015, and maybe some re-runs back in 2019 or so
  • Details here https://johnkerl.org/miller-docs-by-release/5.9.0/whyc-details.html
  • Python numbers not shown there -- I don't know why not
  • https://github.com/johnkerl/miller/blob/main/python/sketch.py was some sketch code from 2015 (unfortunately, this is Python 2)
• If I had it to do all over again, I'd have chosen Rust over Go
  • I was a year deep into the Go port and -- very unwisely -- didn't do gigascale perf-testing until the very end.
  • I had to do some awkward workarounds regarding memory allocation and goroutine scheduling that I'd rather not have had to have done
  • Unfortunately, I don't have it to do all over again -- the C-to-Go port was over a year and I don't have it in me to do a Go-to-Rust port
  • Moving forward I really like nushell ... if I were to do it all over again I'd probably integrate with the nushell project rather than porting Miller-as-is to a third implementation language ...

[janxkoci]

If I had it to do all over again, I'd have chosen Rust over Go

I know you won't (anytime soon), but I vote for Julia 😃 It looks like a scripting language, but thanks to multiple dispatch, advanced typing system, and JIT compiler it achieves performance comparable to C.

[johnkerl]

@janxkoci I've heard that Julia is indeed intended to solve the "best of both worlds" problem -- also I've heard (second-hand) about start-up-time issues ... I should research this a bit more, out of curiosity if nothing else ...

[janxkoci]

Awesome! 😁

About the slow startups - I was already looking into it a bit and I think for something like miller you could precompile binaries with PackageCompiler 😉

PS: I also highly recommend looking at this explanation of Julia's speed and its trade-offs.

[johnkerl]

P.S.

I think that makes my bias clear. 😉

Got it! 😎

I have two biases ...

One is: for ease of use I want interpreted languages. Which is one reason I enjoy using Miller. And Python can give me this too.

The second bias is, for internal implementation details I want raw performance, and in my experience Python cannot give me this -- except with the unacceptable-to-me overhead of Python-with-C++ and all its package-ownership challenges.

So it seems the best way to go is to use languages like C++, Go, Rust (first bias) as the internal implementation with which to build languages that have a nice interpreted UX (second bias).

[johnkerl]

I did do some very early on profilings -- it was not recent; 2015, and maybe some re-runs back in 2019 or so

• Details here https://johnkerl.org/miller-docs-by-release/5.9.0/whyc-details.html
• Python numbers not shown there -- I don't know why not
• https://github.com/johnkerl/miller/blob/main/python/sketch.py was some sketch code from 2015 (unfortunately, this is Python 2)

@sloanlance in #1372 I updated the old 2015 sketch to Python 3 and re-ran some numbers -- TL;DR it's about 4-5x slower than Miller-in-Go at least on the tests below.

8M-line test-data file:

$ wc -l ~/tmp/data-big.csv
 8164850 /Users/kerl/tmp/data-big.csv

$ head ~/tmp/data-big.csv
a,b,i,x,y
wye,hat,0,0.063880031026843948,0.631681468166262583
zee,eks,1,0.432733176011933562,0.946489232567366834
zee,zee,2,0.319151693638019096,0.982514974187367995
zee,wye,3,0.588725880993868134,0.715881863942314789
hat,hat,4,0.785373189852281106,0.767201847288385919
zee,hat,5,0.508234222656224977,0.892580183172868691
eks,hat,6,0.383808686576694591,0.672595294505634933
zee,wye,7,0.729025047146261262,0.723967387212425151
pan,pan,8,0.709860992948459946,0.633708484765937774

Time a CSV cut:

$ python sketch.py --cut x,b --icsv --ocsv < ~/tmp/data-big.csv > foo
TIME IN SECONDS 15.731

$ mlr --csv cut -of x,b < ~/tmp/data-big.csv > foo2
TIME IN SECONDS 3.730

$ wc -l foo foo2
 8164850 foo
 8164850 foo2
 16329700 total

$ tail foo foo2
==> foo <==
0.942294761045973206,pan
0.858856721249807942,pan
0.725672185758907462,hat
0.150233794539344406,pan
0.148802299587429632,eks
0.274454841052393816,pan
0.101953339344800137,hat
0.999582775402619861,pan
0.278848832137346681,pan
0.754386172981181269,eks

==> foo2 <==
0.942294761045973206,pan
0.858856721249807942,pan
0.725672185758907462,hat
0.150233794539344406,pan
0.148802299587429632,eks
0.274454841052393816,pan
0.101953339344800137,hat
0.999582775402619861,pan
0.278848832137346681,pan
0.754386172981181269,eks

Time DKVP to XTAB with cut:

python sketch.py --cut x,b --oxtab < ~/tmp/data/big.dkvp > bar
TIME IN SECONDS 23.045

$ mlr --oxtab cut -of x,b < ~/tmp/data-big.dkvp > bar2
TIME IN SECONDS 6.205

$ wc -l bar bar2
 24494547 bar
 24494546 bar2
 48989093 total

$ tail bar bar2
==> bar <==

x 0.999582775402619861
b pan

x 0.278848832137346681
b pan

x 0.754386172981181269
b eks


==> bar2 <==
b hat

x 0.999582775402619861
b pan

x 0.278848832137346681
b pan

x 0.754386172981181269
b eks