GetResponses* - memory problems / optimization

[m-delfino]

GetResponses* methods cause out of memory problems on small instances when downloading responses for large studies (a combination of survey size + number of participants + number of responses).

The main reason is the generation of the entire json/csv file in memory:

study-service/pkg/grpc/service/data_export.go

Line 484 in 168ac7c

buf := new(bytes.Buffer)

The json/csv generation should be done in chunks to avoid out of memory errors, as final buffer is already streamed, eg:

study-service/pkg/grpc/service/data_export.go

Line 382 in 168ac7c

return StreamFile(stream, buf)

[phev8]

You can very easily download the data in chunks as well. While to methods let you download the whole dataset, you can also adjust your query parameters to download data that actually fit in memory.
Or if you really plan to download the whole dataset, you need to use servers with larger memory.

Since there are existing workarounds that should work well, this is a low prio issue for now.