RAM usage

[jungle-boogie]

Hello,

@mholt and I discussed that this plug-in seems to use up a very large amount of
RAM when caddy search is enabled in the Caddyfile and/or when caddy is
compiled with caddy-search.

Here's an example of just starting caddy in the forground and crashing a short
amount of time.

% sudo ./caddy
Activating privacy features... done.
example.com:2020
example.com:2015
example.com:http
zsh: killed     sudo ./caddy
2:14.50 real, 13.971 user, 11.862 sys;  page: 50739 hard/728910 soft, swap: 0, I/O: 7690/1419
Mem: 743832KB (5441KB shared + 642KB data/stack = 6083KB), VCSW: 61653 IVCSW: 18582

Here's another example:

 zsh: killed     sudo ./caddy
3:23.89 real, 18.940 user, 16.209 sys;  page: 90625 hard/1087429 soft, swap: 0, I/O: 4590/1911
Mem: 749608KB (5434KB shared + 641KB data/stack = 6075KB), VCSW: 113115 IVCSW: 25478

The Caddyfile is very bare and only contained search, no regexs were used. The directory caddy is serving is 53MB but there are 4 mp3s that should be excluded from caddy-search. There's also 25MB of images that caddy-search should not be indexing.

Using caddy-search from master on freebsd -release 10.2 with Caddy 0.8 beta 4.

[joeblew99]

+1. I am also needing to use Caddy with Bleve. The Ram issue looks like a killer for production deployments.

[nicolinuxfr]

I'm currently trying caddy-search on a larger big blog (1250 posts or so) and on my Mac (MBP mid 2012 with 8 G of RAM), it seems fine.

The caddy process uses around 300 Mo of RAM and the index file for the search is around 270 Mo, so I guess it's not too much. Or am I wrong ?

[jungle-boogie]

so I guess it's not too much. Or am I wrong ?

I think it's way too much. Will your caddy process eventually terminate if it runs out of RAM?

[mholt]

We'll look into this. It could have something to do with the underlying bleve library.

[nicolinuxfr]

It was guess too : Bleve creates a database and stores a lot (maybe all) in RAM. I don't think it's unexpected.

[mholt]

Perhaps there's a way to minimize RAM usage, but ultimately the trade-off is this: we access the database on disk for each search query or we keep the database in memory for faster searches.

[nicolinuxfr]

Exactly and I think RAM is better.

[pedronasser]

@jungle-boogie @joeblew99 @nicolinuxfr @mholt
I've made a lot of changes recently.
Can you check if the memory allocation has been reduced?

[mholt]

For the caddy site, it went down from ~17 MB to ~15 MB. On a larger directory with many more subfolders I see memory usage at ~64 MB. So it seems a little better? Have we determined what is actually using the memory?

[ghost]

Are the above changes implemented in the version being auto-build by the caddy download service? I just downloaded a fresh caddy today to diagnose a server crash I was experiencing, and on a Pi2 I just get a "cannot allocate memory" crash when I try to launch caddy with search enabled.

[mholt]

It may still be too much for the Pi -- sorry about that. I haven't had time to work on this plugin myself in a long time...

[pepa65]

It would be great if there was a configuration option to either keep all in RAM (for machines with lots of RAM) or to store it on disk (for RAM-starved machines).