S3 listing ignores "IsTruncated = true" property

[wyang007]

In some cases, I see that Davix dir listing ignores true and when listing a bucket with a CEPH S3 storage. Here is an example:

DAVIX(body): Read block (910 bytes):
[<?xml version="1.0" encoding="UTF-8"?><ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Name>rubin-sts</Name><Prefix>LSSTCam/</Prefix><MaxKeys>1000</MaxKeys><Delimiter>/</Delimiter><IsTruncated>true</IsTruncated><CommonPrefixes><Prefix>LSSTCam/20230526/</Prefix></CommonPrefixes><CommonPrefixes><Prefix>LSSTCam/20230531/</Prefix></CommonPrefixes><CommonPrefixes><Prefix>LSSTCam/20230601/</Prefix></CommonPrefixes><CommonPrefixes><Prefix>LSSTCam/20230604/</Prefix></CommonPrefixes><CommonPrefixes><Prefix>LSSTCam/20230606/</Prefix></CommonPrefixes><CommonPrefixes><Prefix>LSSTCam/20230612/</Prefix></CommonPrefixes><CommonPrefixes><Prefix>LSSTCam/20230613/</Prefix></CommonPrefixes><CommonPrefixes><Prefix>LSSTCam/20230614/</Prefix></CommonPrefixes><CommonPrefixes><Prefix>LSSTCam/20230615/</Prefix></CommonPrefixes><Marker></Marker><NextMarker>LSSTCam/20230615/</NextMarker></ListBucketResult>]

[mpatrascoiu]

Hello,

The initial pull-request (#124) handled the IsTruncated + NextMarker listing scenario. However, it was incomplete when the NextMarker propery was missing.

A pull-request (#127) was started on this, but provided only a partial solution. Further discussion was carried via e-mail.

The objective is to also handle correctly the S3 listing scenario when the IsTruncated flag is present, but the NextMarker property is missing. Although rare, it is allowed by the S3 specification: https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjects.html

In such cases, in the next listing request, Davix should provided as NextMarker the last listed item, which can be either the last item in Contents or CommonPrefixes. This was addressed in the following commit:

• [15ef1a3]: Handle the scenario where the listing is truncated but the "NextMarker" key is missing from the server response

The next version (Davix v0.8.8) will contain this fix.
Closing this ticket.

Cheers,
Mihai