Improve performance by increasing the buffer size to 20K #88
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Can also start at 1K like before, and then grow to 20K after we've done a few loops (and after that even further): that way memory usage stays small for small files, but is still fast for larger files. I don't really know what your expectations for memory usage are. 20K is nothing for any desktop system, but can be relatively a lot for embedded systems, I guess? |
|
In my view, TOML is primary meant for configuration. Most TOML files probably have less than 100 lines. 1K seems very small, and I don't have objections to grow it to around 16k. |
Reading a 5M file takes about 13 seconds. For reference: Python takes about 1.2s for the same file, Go 0.4s, and C++ about 0.25s. So there's a lot to gain! One reason is that when reading the file in memory it only allocates 1000 bytes at a time; a simple measurement with this: 1k.toml 0.00s 10k.toml 0.00s 100k.toml 0.01s 1M.toml 0.52s 5M.toml 12.97s 10M.toml 74.58s And some measurements for other values: 2K 1k.toml 0.00s 10k.toml 0.01s 100k.toml 0.01s 1M.toml 0.36s 5M.toml 9.30s 10M.toml 53.68s 5K 1k.toml 0.00s 10k.toml 0.00s 100k.toml 0.00s 1M.toml 0.27s 5M.toml 7.02s 10M.toml 44.81s 10K 1k.toml 0.00s 10k.toml 0.00s 100k.toml 0.01s 1M.toml 0.25s 5M.toml 6.27s 10M.toml 43.05s 15K 1k.toml 0.00s 10k.toml 0.00s 100k.toml 0.01s 1M.toml 0.23s 5M.toml 5.96s 10M.toml 39.19s 20K 1k.toml 0.00s 10k.toml 0.00s 100k.toml 0.01s 1M.toml 0.24s 5M.toml 5.66s 10M.toml 39.41s 25K 1k.toml 0.00s 10k.toml 0.00s 100k.toml 0.00s 1M.toml 0.21s 5M.toml 5.77s 10M.toml 38.10s 50K 1k.toml 0.00s 10k.toml 0.00s 100k.toml 0.00s 1M.toml 0.21s 5M.toml 5.38s 10M.toml 33.91s I set it to 16K as the performance benefits drop off after that, at least on my system, and it's still quite little memory, but can also use another value if you prefer – 5K or even 2K already make a difference. (the rest of the performance is mostly in the strcmp()s in check_key() by the way, that loops over all the keys for every key it finds, which is why larger files get so drastically slower).
arp242
force-pushed
the
buf
branch
2 times, most recently
from
4a84f1f to
1c3c2db
Compare
|
Yes, I agree. It's just that it would be nice to also make it work reasonably well with larger files, and seemed a simple enough change. I changed it to 16K now. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Reading a 5M file takes about 13 seconds. For reference: Python takes about 1.2s for the same file, Go 0.4s, and C++ about 0.25s. So there's a lot to gain!
One reason is that when reading the file in memory it only allocates 1000 bytes at a time; a simple measurement with this:
And some measurements for other values:
I set it to 20K as the performance benefits drop off after that, at least on my system, and it's still quite little memory, but can also use another value if you prefer – 5K or even 2K already make a difference.
(the rest of the performance is mostly in the strcmp()s in check_key() by the way, that loops over all the keys for every key it finds, which is why larger files get so drastically slower).