Provider tool "ls" or "dir" to list folder objects breaks when Cloud Storage bucket objects >1,000

[KennyCCK]

Documentation:

• https://cloud.google.com/tools/powershell/docs/quickstart#navigate_with_powershell_provider

Summary:

• Google Cloud SDK 322.0.0, Cloud Tools for Powershell (latest)
• It appears that there is a consistent issue with "ls" or "dir" when trying to list objects of a folder (eg. PS gs:\bucketX> ls folderB) when there are over 1,000 objects stored prior to the target folder.
• Sample output:

PS gs:\bucketTest> dir


   ParentPath: bucketTest

Name                Size ContentType TimeCreated Updated
----                ---- ----------- ----------- -------
A
B
C

PS gs:\bucketTest> dir B
dir : Cannot find path 'bucketTest\B' because it does not exist.
At line:1 char:1
+ dir B
+ ~~~~~~~~~~
    + CategoryInfo          : ObjectNotFound: (bucketTest\B:String) [Get-ChildItem], ItemNotFoundException
    + FullyQualifiedErrorId : PathNotFound,Microsoft.PowerShell.Commands.GetChildItemCommand

Debug/Troubleshoot details:

• I managed to execute the tool in verbose mode via Visual Studio and tracked where the problem is in the codes. The first logic entry [1], and next the issue is caused by the this line [2] as it constantly returns "false".
• _objectMap appears to be an array holding the result of the call to storage.objects.list REST API [3] which limits the object arrays up to 1,000 maximum
• As the objects array are alphabetically ordered, this appears to be the reason why if there are 1,000 objects prior the target folder, the _objectMap array would not contain the target folder object hence the .ContainsKey(objectName) always return "false"

Reproduction steps:

1. Create bucketTest with folder "A", "B", and "C"
2. Upload 1,010 objects to folder "A"
3. Open Powershell (assuming necessary configs are done)
4. Enter cd gs:\
5. PS gs:\> cd bucketTest
6. PS gs:\bucketTest> ls B (ItemNotFoundException expected to occur here)

References:

[1]

google-cloud-powershell/Google.PowerShell/Provider/GoogleCloudStorageProvider.cs

Line 390 in b0549f9

bool objectExists = model.ObjectExists(gcsPath.ObjectPath);

[2]

google-cloud-powershell/Google.PowerShell/Provider/BucketModel.cs

Line 108 in b0549f9

return _objectMap[objectName] != null;

[3] https://cloud.google.com/storage/docs/json_api/v1/objects/list#maxResults

Current Workaround

The following works fine and offer similar object listing results for Cloud Storage bucket folder:

• Cmdlets (eg. Get-GcsObject)
• gsutil ls