Output file size with DiffusionUniformKh

[vbalza]

When running OceanParcels, we have noticed that the output zarr when adding a diffusion kernel (DiffusionUniformKh) can be >10x larger relative to when we don't add diffusion. We tried investigating why the diffusion simulation output files are a lot larger but the contents appear identical once the file is loaded into an xarray DataSet. The hidden metadata files also look very similar—if not identical. Does anyone have a sense of why this may be the case?

We are attaching a notebook with a simple example showcasing this disparity in output file sizes. Thanks!
diffusion.zip

[JamiePringle]

I am traveling and can't open your example. But I suspect the issue has to do with how compression works. In particular, if you are running a bunch of particles starting at the same location, when you save the output the compression in zarr will greatly reduce the size of the output file. But when you add diffusion, they will take different paths, and will no longer be compressed. This could also be an issue if you started all the particles at the same depth, and without diffusion they stay there, but with diffusion, they spread out. This will also make compression less effective.

Another think that could trip up compression is if the size of the chunking changes between the two runs.

To figure out what is going, open the files with zarr, not xarray...

import zarr
dataIn=zarr.open('2010.zarr','r')

now use the ".info" attribute to figure out what is going with each variable by typing dataIn.VariableName.info, for example

In [4]: dataIn.z.info
Out[4]: 
Name               : /z
Type               : zarr.core.Array
Data type          : float32
Shape              : (551334054, 33)
Chunk shape        : (50000, 10)
Order              : C
Read-only          : True
Compressor         : Blosc(cname='lz4', clevel=5, shuffle=SHUFFLE, blocksize=0)
Store type         : zarr.storage.DirectoryStore
No. bytes          : 72776095128 (67.8G)
No. bytes stored   : 356437282 (339.9M)
Storage ratio      : 204.2
Chunks initialized : 44108/44108

You will see Chunk shape, which tells you how it is actually chunked on disk, and No.bytes, the number of bytes the uncompressed data would take up, and No. bytes stored, the space it takes up on disk. Helpfully, it also reports the "Storage ratio", which is the ratio o the two sizes. I suspect this is what you will see change between the two runs, especially for the vertical coordinate.

Also check that the "Compressor" has not changed.

Also, be sure your Chunk shape has not changed to something silly.

I hope this helps,
Jamie

p.s. The compression in this case is very dramatic, because the depth was fixed for this run. A more representative example is this horizontal coordinate:

In [5]: dataIn.lon.info
Out[5]: 
Name               : /lon
Type               : zarr.core.Array
Data type          : float32
Shape              : (551334054, 33)
Chunk shape        : (50000, 10)
Order              : C
Read-only          : True
Compressor         : Blosc(cname='lz4', clevel=5, shuffle=SHUFFLE, blocksize=0)
Store type         : zarr.storage.DirectoryStore
No. bytes          : 72776095128 (67.8G)
No. bytes stored   : 51141226004 (47.6G)
Storage ratio      : 1.4
Chunks initialized : 44108/44108