fix: don't deadlock the runtime when fetching MSK auth token

[emef]

the previous code introduced a deadlock on the main tokio runtime (joining a thread that was running a blocking call on the runtime).

the only solution I found to this was to create a new temporary tokio runtime so that we can run the async aws code inside the mandatory non-async function. this function is only called when a kafka client using MSK auth needs to fetch a new oauth token, so it should be pretty infrequent.

[mwylde]

This seems reasonable for now, but I think a better approach would be to run a separate task that fetches and refreshes the oauth token, and stores it in an Arc<Mutex<_>> which is read from the generate_oauth_token metho. Ideally we wouldn't ever be blocking the async thread with a blocking method like this.

The real fix is to change the API in librdkafka (for which there's an open issue: fede1024/rust-rdkafka#727) but the upstream project is somewhat dormant at the moment.

[mattforbes-gr]

This seems reasonable for now, but I think a better approach would be to run a separate task that fetches and refreshes the oauth token, and stores it in an Arc<Mutex<_>> which is read from the generate_oauth_token metho. Ideally we wouldn't ever be blocking the async thread with a blocking method like this.

The real fix is to change the API in librdkafka (for which there's an open issue: fede1024/rust-rdkafka#727) but the upstream project is somewhat dormant at the moment.

yeah I agree w/ all this. I think that the rdkafka library is calling the generate_oauth_token function from c so I'm sure it wouldn't be any easy thing to make async-friendly. the background refresh is the next best thing but I just wanted to get unblocked. if I have some time free up, I can take a pass at cleaning that up