Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for async calls to the ohsome api #106

Closed
SlowMo24 opened this issue Feb 28, 2023 · 2 comments
Closed

Add support for async calls to the ohsome api #106

SlowMo24 opened this issue Feb 28, 2023 · 2 comments
Labels
enhancement New feature or request

Comments

@SlowMo24
Copy link
Collaborator

First code-snipped by @matthiasschaub

import json
from json import JSONDecodeError

import httpx


async def query_ohsome_api(url: str, data: dict) -> dict:
    """Query the ohsome API.

    A custom connection timeout is set since the ohsome API can take a long time to
    send an answer (< 10 minutes).

    Raises:
        httpx.HTTPStatusError: In case of 4xx and 5xx response status codes
        JSONDecodeError: In case of timeout during streaming the response
    """
    async with httpx.AsyncClient(timeout=httpx.Timeout(300, read=660)) as client:
        response = await client.post(
            url,
            data=data,
            headers={"user-agent": "Teapot"},
        )
    response.raise_for_status()
    try:
        return json.loads(response.content)
    except JSONDecodeError:
        print(
            "Ohsome API returned invalid JSON after streaming of the response. "
            + "The reason is a timeout of the ohsome API."
        )
        raise
@matthiasschaub
Copy link
Contributor

matthiasschaub commented Feb 28, 2023

For reference see how OQT implements this function dee here.

We extended error handling on above solution. Current code is:

async def query_ohsome_api(url: str, data: dict) -> dict:
    """Query the ohsome API.

    A custom connection timeout is set since the ohsome API can take a long time to
    send an answer (< 10 minutes).

    Raises:
        OhsomeApiError: In case of any response except 2xx status codes or invalid
            response due to timeout during streaming.
    """
    headers = {"user-agent": get_config_value("user_agent")}
    # 660s timeout for reading, and a 300s timeout elsewhere.
    async with httpx.AsyncClient(timeout=httpx.Timeout(300, read=660)) as client:
        resp = await client.post(url, data=data, headers=headers)
    try:
        resp.raise_for_status()
    except httpx.HTTPStatusError as error:
        # TODO: Make this more general if issue is closed
        # https://github.com/GIScience/ohsome-api/issues/288
        try:
            message = error.response.json()["message"]
        except KeyError:
            message = error.response.json()["error"]
        raise OhsomeApiError("Querying the ohsome API failed! " + message) from error
    try:
        return geojson.loads(resp.content)
    except JSONDecodeError as error:
        raise OhsomeApiError(
            "Ohsome API returned invalid GeoJSON after streaming of the response. "
            + "The reason is a timeout of the ohsome API."
        ) from error

@redfrexx redfrexx added the enhancement New feature or request label Nov 7, 2023
@SlowMo24
Copy link
Collaborator Author

ok, so I had a look into it. Tbh, at the time of writing the issue, I just heard that OQAPI was not using ohsome-py because of lack of async and thought we should change that. Also at that time async and parallel were synonymes for me...

But now I think the situation is fine, both libraries strive for different things. There is not sufficient high requirement to support async in ohsome-py, so I will close this. If we ever need it we could use e.g. httpx as a drop-in replacement.

I've also created #135 as a replacement.

@SlowMo24 SlowMo24 reopened this Nov 20, 2023
@SlowMo24 SlowMo24 closed this as not planned Won't fix, can't repro, duplicate, stale Nov 20, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

3 participants