Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

asyncio error code does not handle connection lost properly #930

Open
Skylion007 opened this issue Jan 12, 2025 · 0 comments
Open

asyncio error code does not handle connection lost properly #930

Skylion007 opened this issue Jan 12, 2025 · 0 comments

Comments

@Skylion007
Copy link

When using s3fs, I saw this concerning message in the logs:

Future exception was never retrieved
future: <Future finished exception=ConnectionError('Connection lost')>
aiohttp.client_exceptions.ClientOSError: [Errno 32] Broken pipe

The above exception was the direct cause of the following exception:

ConnectionError: Connection lost

Luckily, I was using s3fs to output model checkpoints for training a deep learning model, and the next checkpoint succesfully saved, however, we should really probably properly handle this exception by either forwarding it to the caller, or alternatively, engaging in a low level retry to restore the connection and resume the upload.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant