Sentinel-2C support

[vincentschut]

Sentinel-2C was launched some months ago, and the first images are coming in. However, when trying to run force-l2ps on them , it displays this error: "unknown/unsupported Sentinel-2! Parsing metadata failed."

We do run the latest FORCE version (3.7.12), and the exact same workflow works with S2A/B and Landsat.

Could we have S2C support please?
Thanks!

Vincent.

[davidfrantz]

Hi Vincent,

it is already there. Well, in the develop branch: bb84991

Cheers,
David

[vincentschut]

Ah, sorry, I missed that. We rather use a released version in our production system, if possible. Any chance you could do a release soon?

[vincentschut]

ESA just announced that per January 21, S2C will replace S2A: https://dataspace.copernicus.eu/node/2039. Would it be possible to have a new release of FORCE before that date? If not, please let me know, than I can see if we temporarily could use the develop branch.

[davidfrantz]

I will try to release it by the end of the month. I need to resolve at least one incompatibility between software components.

However, the Level 2 Processing System in dev can be regarded as stable; thus, feel free to use it already.

[vincentschut]

Thanks, we'll prepare for using the develop branch, in case we need S2C before an official release.

[jakimowb]

@davidfrantz thanks for adding S2C support with bb84991. However, it seems that bash/force-level1-csd.sh still misses support S2C and LND09 (or LC09).

[davidfrantz]

True. But see #334

Landsat should be disabled entirely in that script as the GCS doesn't hold Collection 2 (unless anything changed in between)

I will take a look whether S2C can formally be added and whether I can disable LND*.

[davidfrantz]

@jakimowb: 1ade2c3

Nevertheless, #334 is still blocking real usage

[jakimowb]

Actually I am developing an enhancement to https://github.com/vudongpham/CDSE_Sentinel2_downloader to
add the -f paramter that is already know from Landsat links:

-f | --forcelogs
Path to FORCE log file directory (Level-2 processing logs, directory will be searched recursively)
Links will only be generated for products that haven't been processed by FORCE yet.

This way the CDSE S2 downloader may be used more easily as to workaround download S2 from CDSE instead google.

[pjonsson]

Is #334 about writing a Python implementation that downloads directly from CDSE?

If it is, my question is why a clean re-implementation instead of collaborating with an existing project that already provides this functionality? I can see why a generic CDSE download project would not merge contributions that adds --forcelogs since that only targets a fraction of their users, but I'm sure they would happily merge a contribution that makes it possible to download Sentinel-2C if that isn't already supported.

[jakimowb]

@pjonsson good questions. Is there any CDSE library/tool that you would recommend to use? The CDSE Sentinel2 downloader was the first I looked into, it worked out smoothly and is easy to addapt to the FORCE / FORCE schedule workflows. But I agree to that taking up a tool with a larger community in background has benefits as well.

[pjonsson]

@jakimowb my only experience after the Copernicus Open Access Hub closed in late 2023 is with CDSETool. I was not the one who scouted for alternatives to sentinelsat (the package we used to download from the Copernicus Open Access Hub) or updated the code to use CDSETool, but my recollection of the discussions we had was that there weren't many other alternatives at the end of 2023. After that, we kept using CDSETool because the package authors either fixed bugs themselves or accepted my pull requests, so we never spent any time on looking for alternatives since we had something that worked for our needs.

Since I have quite a few pull requests merged in CDSETool by now, you should not consider my perspective unbiased. We don't use the CLI of CDSETool so I can't comment on that, but we use it as a library and have downloaded a couple of hundred thousand products and the only real problems we've had have been with the remote servers and faulty products, and neither of those are the fault of CDSETool. I did get a response from the Copernicus support saying they had made some adjustments to their servers which did seem to reduce the intermittent errors we had, and I had some pull requests merged in CDSETool that makes it retry (a lot) when the Copernicus servers misbehave so you usually get the product downloaded (eventually) even when the servers aren't well.

We're ~15 months from October 2023 so there might be a dozens of good third-party CDSE downloaders available out there by now, it might be worth a couple of hours of looking around and see if something else out there suits your needs better.

Regardless of what you choose, I don't see any third-party CDSE downloader project merging the functionality that --forcelogs does, so if you go the route of using a third-party project I assume you will need to have some (thin) wrapping around it for the Force-specific functionality. But making that Force-specific wrapping will still be quite a bit less effort than to implement that+something that gives you a reliable download from the Copernicus servers.