You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I'm trying to create fsspec files for netCDF 3 datasets on a local filesystem, but am encountering an error during the json reference file creation process:
AsyncGroup.create_array() got an unexpected keyword argument 'compression'
The error is coming from the netCDF3.py module NetCDF3toZarr translate method in the following block:
compression is not included in the create_dataset parameter list according to the zarr documentation, but compressor is, so fixing the argument in this and other create_dataset calls should presumably solve the issue.
Steps to reproduce:
def write_fsspec(fs_read, input_file, output_dir):
with fs_read.open(input_file) as infile:
print(f"Running kerchunk generation for {input_file}...")
chunks = kerchunk.netCDF3.NetCDF3ToZarr(infile, inline_threshold=300)
file_name = os.path.basename(input_file)
file_name = file_name.replace('.nc', '.json')
out_file_name = output_dir + '/' + file_name
with open(out_file_name, "wb") as f:
f.write(json.dumps(chunks.translate()).encode()) # call to netCDF3.py originates here
print(f"Finished writing {out_file_name}")
return out_file_name
dir_path = config['input_dir']
dir_path += '**/*.nc'
file_paths = glob.glob(dir_path, recursive=True)
fs_read = fsspec.filesystem('local')
temp_dir = TemporaryDirectory(prefix=config['output_dir'])
output_files = [write_fsspec(fs_read, f, temp_dir.name) for f in file_paths]
The text was updated successfully, but these errors were encountered:
What version of zarr do you have? You may need to downgrade to <3.0. We know that there is some work needed in this repo to make it compatible with the newely-released major version of zarr.
@martindurant Downgrading to Zarr v2.18.4 fixed the issue, though I am now encountering other errors. I'm not sure if these are fsspec or file metadata problems (dataset does not adhere to cmip conventions), but I'll open a different issue if it's something else in fsspec.
I'm trying to create fsspec files for netCDF 3 datasets on a local filesystem, but am encountering an error during the json reference file creation process:
The error is coming from the netCDF3.py module NetCDF3toZarr translate method in the following block:
I'm using Kerchunk v0.2.7.
compression
is not included in thecreate_dataset
parameter list according to the zarr documentation, butcompressor
is, so fixing the argument in this and other create_dataset calls should presumably solve the issue.Steps to reproduce:
The text was updated successfully, but these errors were encountered: