Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Async requests #175

Merged
merged 7 commits into from
Apr 24, 2024
Merged

Async requests #175

merged 7 commits into from
Apr 24, 2024

Conversation

JR-1991
Copy link
Member

@JR-1991 JR-1991 commented Mar 1, 2024

Changes this PR introduces

This pull request adds async support to pyDataverse based on the previously merged switch from requests to httpx. Asynchronous requests significantly improve pyDataverse's performance by using concurrent requests. The implementation provides a seamless usage to work either sync or async by the usage of a context manager.

Small benchmark

  • 100 calls to api/info/version to demo.dataverse.org
  • Synchronous --> ~35s
  • Asynchronous --> ~0.5s 🏎️

Overview

  • Added _sync_request and _async_request base methods
  • Added Context manager to API to enable async requests
  • Small test to verify async requests are working

Example

Synchronous

native_api = NativeApi("https://demo.dataverse.org")
response = native_api.get_info_version()

Asynchronous

import asyncio
from pyDataverse.api import NativeApi


async def run_async(native_api):
    async with native_api:
        tasks = [native_api.get_info_version() for _ in range(100)]
        responses = await asyncio.gather(*tasks)

if __name__ == "__main__":
    native_api = NativeApi("https://demo.dataverse.org")
    asyncio.run(run_async(native_api))

@JR-1991 JR-1991 added type:feature New feature pkg:api api related activities prio:medium status:for review Waiting to be reviewd labels Mar 1, 2024
@JR-1991 JR-1991 self-assigned this Mar 1, 2024
@pdurbin pdurbin mentioned this pull request Mar 4, 2024
47 tasks
@pdurbin pdurbin added this to the 0.3.2 milestone Mar 4, 2024
Copy link

@haeussma haeussma left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tested the example with python=3.11.

Notices missing dependency urllib3 which is not specified in the requirements but imported in src/pyDataverse/__init__.py.

Except for that the example works as expected.

Copy link

@haeussma haeussma left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tested the example with python=3.11.

Notices missing dependency urllib3 which is not specified in the requirements but imported in src/pyDataverse/__init__.py.

Except for that the example works as expected.

@JR-1991
Copy link
Member Author

JR-1991 commented Apr 22, 2024

@haeussma thanks for the review! I have added urllib3 to the list of dependencies.

@JR-1991 JR-1991 merged commit 072acb7 into master Apr 24, 2024
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
pkg:api api related activities prio:medium status:for review Waiting to be reviewd type:feature New feature
Projects
Development

Successfully merging this pull request may close these issues.

3 participants