QNABOT-BEDROCK-EMBEDDINGS-AND-LLM - Tags wrapping responses

[billsnyder]

In version v0.1.13 of the QNABOT-BEDROCK-EMBEDDINGS-AND-LLM plugin, responses returned from Bedrock come back wrapped in tags. In my case I am using the current suggested prompt which contains <question></question> tags in it and the responses come back wrapped in tags. Switching the prompt to a different set of tags results in the responses coming back wrapped in the corresponding tags so it would appear that the tags from the prompt are leaking through to the response. I am testing with model: anthropic.claude-v2. The result of these tags being present means they show up in text based responses and they also result in an error for any Polly based responses since Polly sees them as bad SSML markup. The error you get in that situation is: Invalid Bot Configuration: Encountered an exception when making Amazon Polly service call, Invalid SSML request (Service: AmazonPolly; Status Code: 400; Error Code: InvalidSsmlException; Request ID: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx; Proxy: null). I was running an earlier version of the plugin and this issue appeared after upgrading to version v0.1.13

[michaelin-96]

Can confirm this is an issue with model anthropic.claude-v2. Until this is fixed, should those using QNABOT-BEDROCK-EMBEDDINGS-AND-LLM stick with a different model? I haven't seen this issue while using anthropic.claude-instant-v1.

[rstrahan]

Thanks both for reporting this! Are either of you willing to submit a pull request with a fix? If it's not easily fixed by manipulating the prompt, another simple, but hopefully effective, method could be to strip the unwanted tags from the response in the lambda function before returning the response back to QnAbot. Fork the repo, modify the code in your fork, build/publish locally/test, and then commit/push back to your fork and submit a PR.. I can quickly accept and republish the plugins when I have that. Tx!!

[chanbi-aws]

Any updates on this? Using anthropic.claude-instant-v1 works but the results are not good compared with Claude v2.

[rstrahan]

@michaelin-96 Are you able to send a PR with a fix this week? If not, I'll try to find someone else to look at it.. Tx.

[michaelin-96]

@rstrahan Hey Bob, I don't have the bandwidth to look at this currently. Sorry.

[PercDev23]

Just curious if adding Claude 3 support will also potentially resolve tag wrapping issue. Very interested to use this with haiku.