Unable to authenticate with web identity

[kwohlfahrt]

I am trying to authenticate to S3 with a web identity token file, as created by the AWS pod identity webhook.

The following snipped reproduces the issue:

import s3fs
fs = s3fs.S3FileSystem()
fs.ls("foo") # Fails with `AttributeError`

This is the traceback of the error:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.12/site-packages/fsspec/asyn.py", line 118, in wrapper
    return sync(self.loop, func, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/fsspec/asyn.py", line 103, in sync
    raise return_result
  File "/usr/local/lib/python3.12/site-packages/fsspec/asyn.py", line 56, in _runner
    result[0] = await coro
                ^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/s3fs/core.py", line 1023, in _ls
    files = await self._lsdir(path, refresh, versions=versions)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/s3fs/core.py", line 744, in _lsdir
    async for c in self._iterdir(
  File "/usr/local/lib/python3.12/site-packages/s3fs/core.py", line 775, in _iterdir
    await self.set_session()
  File "/usr/local/lib/python3.12/site-packages/s3fs/core.py", line 507, in set_session
    self.key = credentials.access_key
               ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/botocore/credentials.py", line 409, in access_key
    self._refresh()
  File "/usr/local/lib/python3.12/site-packages/botocore/credentials.py", line 502, in _refresh
    self._protected_refresh(is_mandatory=is_mandatory_refresh)
  File "/usr/local/lib/python3.12/site-packages/botocore/credentials.py", line 518, in _protected_refresh
    metadata = self._refresh_using()
               ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/botocore/credentials.py", line 665, in fetch_credentials
    return self._get_cached_credentials()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/botocore/credentials.py", line 675, in _get_cached_credentials
    response = self._get_credentials()
               ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/botocore/credentials.py", line 909, in _get_credentials
    return client.assume_role_with_web_identity(**kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'ClientCreatorContext' object has no attribute 'assume_role_with_web_identity'

I had installed s3fs with just pip install s3fs, it resulted in these specific versions:

# pip freeze
aiobotocore==2.8.0
aiohttp==3.9.1
aioitertools==0.11.0
aiosignal==1.3.1
attrs==23.1.0
botocore==1.33.1
frozenlist==1.4.0
fsspec==2023.12.0
idna==3.6
jmespath==1.0.1
multidict==6.0.4
python-dateutil==2.8.2
s3fs==2023.12.0
setuptools==69.0.1
six==1.16.0
urllib3==2.0.7
wheel==0.41.3
wrapt==1.16.0
yarl==1.9.3

Using a plain botocore client seems to work (and also works without an AccessDenied error when using a bucket I have access to):

import botocore.session
session = botocore.session.get_session()
client = session.create_client('s3')
client.list_objects_v2(Bucket="foo") # Fails with 'AccessDenied', as expected

[mpayne-coveo]

We're having the same issue using s3fs==2023.12.0

[martindurant]

I may push out a quick release if this seems widespread

[martindurant]

Fixed in #832