Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Using web identity token file credentials fails with AttributeError #833

Closed
akumor opened this issue Dec 5, 2023 · 5 comments
Closed

Using web identity token file credentials fails with AttributeError #833

akumor opened this issue Dec 5, 2023 · 5 comments

Comments

@akumor
Copy link

akumor commented Dec 5, 2023

Using version 2023.12.0 of s3fs I am seeing an issue with AWS credentials which I can reproduce by using a web identity token file for the credentials and running

Python 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import s3fs
>>> s3 = s3fs.S3FileSystem()
>>> s3fs.ls('some-bucket')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/tmp/virtualenv/lib/python3.10/site-packages/fsspec/asyn.py", line 118, in wrapper
    return sync(self.loop, func, *args, **kwargs)
  File "/tmp/virtualenv/lib/python3.10/site-packages/fsspec/asyn.py", line 103, in sync
    raise return_result
  File "/tmp/virtualenv/lib/python3.10/site-packages/fsspec/asyn.py", line 56, in _runner
    result[0] = await coro
  File "/tmp/virtualenv/lib/python3.10/site-packages/s3fs/core.py", line 1023, in _ls
    files = await self._lsdir(path, refresh, versions=versions)
  File "/tmp/virtualenv/lib/python3.10/site-packages/s3fs/core.py", line 744, in _lsdir
    async for c in self._iterdir(
  File "/tmp/virtualenv/lib/python3.10/site-packages/s3fs/core.py", line 775, in _iterdir
    await self.set_session()
  File "/tmp/virtualenv/lib/python3.10/site-packages/s3fs/core.py", line 507, in set_session
    self.key = credentials.access_key
  File "/tmp/virtualenv/lib/python3.10/site-packages/botocore/credentials.py", line 406, in access_key
    self._refresh()
  File "/tmp/virtualenv/lib/python3.10/site-packages/botocore/credentials.py", line 499, in _refresh
    self._protected_refresh(is_mandatory=is_mandatory_refresh)
  File "/tmp/virtualenv/lib/python3.10/site-packages/botocore/credentials.py", line 515, in _protected_refresh
    metadata = self._refresh_using()
  File "/tmp/virtualenv/lib/python3.10/site-packages/botocore/credentials.py", line 662, in fetch_credentials
    return self._get_cached_credentials()
  File "/tmp/virtualenv/lib/python3.10/site-packages/botocore/credentials.py", line 672, in _get_cached_credentials
    response = self._get_credentials()
  File "/tmp/virtualenv/lib/python3.10/site-packages/botocore/credentials.py", line 906, in _get_credentials
    return client.assume_role_with_web_identity(**kwargs)
AttributeError: 'ClientCreatorContext' object has no attribute 'assume_role_with_web_identity'

I am using aiobotocore==2.7.0 fsspec==2023.12.0 and aiohttp==3.9.1

@0x26res
Copy link

0x26res commented Dec 5, 2023

I'm having similar issues:

  • 'ClientCreatorContext' object has no attribute 'exceptions'
  • 'ClientCreatorContext' object has no attribute 'get_role_credentials'

@martindurant the only change I can find on the path that would have changed the session is here https://github.com/fsspec/s3fs/blame/3f5a8d799fced7732d733775903e2b0d876df1a9/s3fs/core.py#L502

        if self.session is None:
            self.session = aiobotocore.session.AioSession(**self.kwargs)

But it don't think it explains the issue we are experiencing.

@martindurant
Copy link
Member

Fixed in #835 (reversion), will release today. Feel free to test.

@0x26res
Copy link

0x26res commented Dec 5, 2023

@martindurant I can confirm 2023.12.1 is working. Should 2023.12.0 be yanked?

Thanks again for fixing.

@martindurant
Copy link
Member

Should 2023.12.0 be yanked?

Yes, OK (although no one will accidentally install it)

@akumor
Copy link
Author

akumor commented Dec 5, 2023

I can confirm 2023.12.1 has fixed the issue for me too. Thank you 🙇

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants