Measure performance of tebako packages against plain Ruby scripts

[maxirmx]

Tebako benchmarking
Tebako needs benchmarking environment that can be run against the ruby-packer and ocra
compute time (tebako architecture vs ruby-packer)
read/write time (due to DwarFS instead of SqashFS)

and although to support improvements like tamatebako/tebako#74

Originally posted by @maxirmx in tamatebako/tebako#45 (comment)

[ronaldtse]

The following issues are dependencies:

• Implement support for Ruby 3.0, 3.1 (except MSys) tebako#88
• Implement support for Ruby 3.0, 3.1, 3.2 for MSys / Windows tebako#90

[maxirmx]

We would like to run benchmarking for three configurations:

• Simple gem without native extensions ** fontist ??? or one of its components**
• Simple gem with native extension emf2svg
• Complex application metanorma

For each configuration we will compare timing of loacally installed gem on Ubuntu, MacOs and Windows against tebako/aibika package.

[ronaldtse]

Simple gem without native extensions

fontist uses the excavate gem which uses a lot of native extensions.

coradoc is in pure Ruby and has good specs.

Let's use coradoc?

Simple but slightly more complex gem that depends on native extensions

vectory uses the emf2svg gem so it uses "some" native extensions, maybe it is a good candidate for "in between "simple gem with native extension" and "complex application".

[maxirmx]

Revised plan

• "Hello, world" script
• Coradoc
• Vectory; Emf2svg
• Metanorma

[maxirmx]

Vectory gem has not been released to rubygems. There is version 0.1.0 only and it does nothing.
It can be installed from git source of course but I found that tebako does not support such gems: tamatebako/tebako#119
So I will test against emf2svg for now

[ronaldtse]

Ping @alexeymorozov to release vectory.

[alexeymorozov]

vectory is released (v0.2.0). Thanks to @CAMOBAP!

[ronaldtse]

Thanks @maxirmx @alexeymorozov and @CAMOBAP !

[maxirmx]

Hi, @ronaldtse @CAMOBAP

Any comments are much appreciated: Benchmarking results

[CAMOBAP]

@maxirmx just to to be on the same page (and make sure that I got the result's right) even with "decompression" tamatebako perform really well, right?

How do you think it it make sense to benchmark on windows/linux too?

P.S. If need I can help with benchmarking on darwin-x86_64

[ronaldtse]

How do you think it it make sense to benchmark on windows/linux too?

Definitely the next step!

Thank you @maxirmx on the awesome results!

Can you please help write a blog post on these benchmark results?

[ronaldtse]

One of the things we wish to do is to avoid writing to/reading from the FS for native extensions, so that we can use libdwarfs in memory...

[maxirmx]

@maxirmx just to to be on the same page (and make sure that I got the result's right) even with "decompression" tamatebako perform really well, right?

How do you think it it make sense to benchmark on windows/linux too?

P.S. If need I can help with benchmarking on darwin-x86_64

Well, I read these results in a following way:

• Tebako package parses Ruby code faster then native interpreter. I think that reading from memory / decompressing is faster then reading from disk, even from SSD
• External file processing is not noticebly slower for Tebako package despite of additional abstraction layer introduced by Tebako
• Tebako package initialization adds a 'penalty' time needed to read a fairly big file that conatains Ruby standard lib. The application not necessarily uses the whole library but we still need to map it to application memory address space on startup.
• Any native extension or other native code like Java jars in tebako add initialization time because we are decompressing them to temp and loading again

@ronaldtse, I will do a blog post out of these thoughts,

We do not have Tebako Windows version. I tried twice with Msys but coud not resolve linking issues. I am going to try native (Microsoft). If it works I will do benchmarking. Testing Aibika (Ocra) does not make sense imho. On the client side it is just a self extracting archieve.

I have darwin-x86_64 box, thank you for an idea, I will run benchmarking on it as well.
I do not have Linux box will do tests on Windows Subsystem for Linux

[CAMOBAP]

@maxirmx maybe you also can share/add some information about tools and environment preparation?

I never do such benchmarking myself and have some abstract understanding that we need isolate process somehow (reduce effect of other processes) and reduce impact of memory cache somehow

[maxirmx]

I believe that for metanorma the following statement is true:
Processing with tebako package will take the same time as original code plus 3-4 seconds

[maxirmx]

@maxirmx maybe you also can share/add some information about tools and environment preparation?

I never do such benchmarking myself and have some abstract understanding that we need isolate process somehow (reduce effect of other processes) and reduce impact of memory cache somehow

OS kernel measures resources

On Unix-like sytems there is time utility that gives you 'true' time
Just add system time and user time

The trick is to call /usr/bin/time because bash has its own time command that provides different statistics

[maxirmx]

All credits related to excellent package performance go to @mhx