endStep stepUnits in subhourly wrf grib2 output

[emfdavid]

Enable reading grib2 files with subhourly data from the WRF model such as gs://high-resolution-rapid-refresh/hrrr.20230928/conus/hrrr.t00z.wrfsubhf01.grib2

The error: WrongStepUnitError(Wrong units for step (step must be integer))
Is now caught and logged. The bad coordinate attribute is dropped.

2023-10-04T17:40:23.741Z MainProcess MainThread WARNING:grib2-to-zarr:Ignoring coordinate 'step' for varname 'dswrf', raises: eccodes.WrongStepUnitError(Wrong units for step (step must be integer))

[martindurant]

(ref #363 )

[emfdavid]

(ref #363 )

@dcherian Do you have any suggestions on how to resolve this based on your work last week (very excited by the way!)?
My read is that the HRRR grib2 attributes/coords are actually an invalid type according to eccodes... but I am not sure.
My solution was to just catch and log the exception and move on, but would love to solve the actual problem if possible?

[dcherian]

Have you asked over at ecCodes? I think this needs to be solved there.

Can you link to a file that exhibits this problem?

[emfdavid]

Can you link to a file that exhibits this problem?

gs://high-resolution-rapid-refresh/hrrr.20230928/conus/hrrr.t00z.wrfsubhf01.grib2

[emfdavid]

Have you asked over at ecCodes? I think this needs to be solved there.

@martindurant will you merge a PR that catches and logs exception till we can get this sorted?

[emfdavid]

I will try upgrading to the latest eccodes and then file a bug if this is still an issue...

[martindurant]

will you merge a PR that catches and logs exception till we can get this sorted?

Certainly. The HDF module already does this.

[dcherian]

xref ecmwf/cfgrib#335

[emfdavid]

xref ecmwf/cfgrib#335

Thank you for finding those - I don't see anything here that is likely to get fixed quickly here in eccodes - do you?

[emfdavid]

No easily accessible channel to follow in eccodes as to when this will be resolved.
One day we will upgrade eccodes and just stop getting the warnings...

[emfdavid]

Is there a good way to copy out a few messages from a larger grib2 file to make a test file?

[martindurant]

Absolutely, this is what kerchunk.grib._split_file does, e.g., as used in scan_grib:

    with fsspec.open(url, "rb", **storage_options) as f:
        for offset, size, data in _split_file(f, skip=skip):

[emfdavid]

Should be ready to go now?

[martindurant]

Something changed in numcodecs entrypoints?

[emfdavid]

I am so confused - that can't be related to my changes right?

[martindurant]

Perhaps pin to numcodecs<=0.11, and we'll figure it out in a separate PR

[martindurant]

OK, done!
We'll come back to the numcodecs thing...