Fix interruptions in ASR while resources are being respawned #12
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Additionally: * cover with tests * switch to playwright for testing * fallback to speaking, if audio is not available (streaming URL will be ignored)
This bugfix removes respawning of ASR and TTS machines. Instead, the parent machine (`speechstate.ts`) generates new tokens and communicates them to ASR and TTS via NEW_TOKEN event. ASR doesn’t need to be pre-initiated, so only the context gets updated for a new token. New token is then used for a new ASR request. TTS *needs* to be pre-initiated, therefore on a new token it generates new instances of Web Speech API synthesiser and utterance. They are then used for the next TTS request. Appropriate tests are added which attempt to renew tokens while the system is recognising or speaking. Additionally, a new setting, `newTokenInterval` is added to configure how often new tokens will be requested (mainly for testing purposes), defaulting to 30 seconds. Unfortunately, the tests are not exactly fair because the old token doesn’t expire when the new one is released, but I tried to make sure that the new token is actually used.
vladmaraev
force-pushed
the
bugfix/no-more-respawning
branch
from
9bb77f3 to
8f63a2f
Compare
fredrik-talkamatic
approved these changes
Dec 9, 2024
There was a problem hiding this comment.
With my limited understanding: Looks good to me.
Only thing is that I think that we could coordinate what the locales should look like. In the python world they look like "xx_XX", and in the MS and JS world, they seem to look like "xx-XX". TS is converting them from "-" to "_" which the TTS cache is converting back again to "-". Let's talk about that.
vladmaraev
added a commit
that referenced
this pull request
Dec 9, 2024
* Play audio This change enables TTS to play audio. The audio is sent as part of SPEAK event (`audioURL` parameter). If the resource is not available, it falls back to normal synthesis (`utterance` parameter). Additionally, for streamed content, the `cache` parameter containing a URL can be provided with the SPEAK event. This URL will then be first called to check whether cache exists and then, (true) make another request to get an audio file to play or (false) fallback to normal TTS. * Fix interruptions in ASR while resources are being respawned This bugfix removes respawning of ASR and TTS machines. Instead, the parent machine (`speechstate.ts`) generates new tokens and communicates them to ASR and TTS via NEW_TOKEN event. ASR doesn’t need to be pre-initiated, so only the context gets updated for a new token. New token is then used for a new ASR request. TTS *needs* to be pre-initiated, therefore on a new token it generates new instances of Web Speech API synthesiser and utterance. They are then used for the next TTS request. Appropriate tests are added which attempt to renew tokens while the system is recognising or speaking. Additionally, a new setting, `newTokenInterval` is added to configure how often new tokens will be requested (mainly for testing purposes), defaulting to 30 seconds. Unfortunately, the tests are not exactly fair because the old token doesn’t expire when the new one is released, but I tried to make sure that the new token is actually used.
4 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This bugfix removes respawning of ASR and TTS machines. Instead, the
parent machine (
speechstate.ts) generates new tokens andcommunicates them to ASR and TTS via NEW_TOKEN event.
ASR doesn’t need to be pre-initiated, so only the context gets updated
for a new token. New token is then used for a new ASR request.
TTS needs to be pre-initiated, therefore on a new token it generates
new instances of Web Speech API synthesiser and utterance. They are
then used for the next TTS request.
Appropriate tests are added which attempt to renew tokens while the
system is recognising or speaking. Additionally, a new setting,
newTokenIntervalis added to configure how often new tokens will berequested (mainly for testing purposes), defaulting to 30
seconds. Unfortunately, the tests are not exactly fair because the old
token doesn’t expire when the new one is released, but I tried to
make sure that the new token is actually used.
This PR also includes changes from #11 (playing sound files and TTS caching).