Skip to content

Commit

Permalink
Merge branch 'feat/outbound-moderation' into develop
Browse files Browse the repository at this point in the history
  • Loading branch information
@clemlesne
clemlesne committed Jan 26, 2024
2 parents 2923cd0 + bb37ea3 commit 68b693b
Show file tree
Hide file tree
Showing 9 changed files with 492 additions and 62 deletions.
83 changes: 82 additions & 1 deletion README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,7 @@ Extract of the data stored during the call:
- [x] Create by itself a todo list of tasks to complete the claim
- [x] Customizable prompts
- [x] Disengaging from a human agent when needed
- [x] Filter out inappropriate content from the LLM, like profanity or concurrence company names
- [x] Fine understanding of the customer request with GPT-4 Turbo
- [x] Follow a specific data schema for the claim
- [x] Has access to a documentation database (few-shot training / RAG)
Expand Down Expand Up @@ -96,10 +97,11 @@ graph LR
user(["User"])
subgraph "Claim AI"
api["API"]
ai_search[("RAG\n(AI Search)")]
api["API"]
communication_service_sms["SMS gateway\n(Communication Services)"]
communication_service["Call gateway\n(Communication Services)"]
constent_safety["Moderation\n(Content Safety)"]
db[("Conversations and claims\n(Cosmos DB or SQLite)")]
event_grid[("Broker\n(Event Grid)")]
gpt["GPT-4 Turbo\n(OpenAI)"]
Expand All @@ -110,6 +112,7 @@ graph LR
api -- Generate completion --> gpt
api -- Save conversation --> db
api -- Send SMS report --> communication_service_sms
api -- Test for profanity --> constent_safety
api -- Transfer to agent --> communication_service
api -. Watch .-> event_grid
Expand All @@ -122,6 +125,76 @@ graph LR
user -- Call --> communication_service
```

### Sequence diagram

```mermaid
sequenceDiagram
autonumber
actor Customer
participant PSTN
participant Text to Speech
participant Speech to Text
actor Human agent
participant Event Grid
participant Communication Services
participant Content Safety
participant API
participant Cosmos DB
participant OpenAI GPT
participant AI Search
API->>Event Grid: Subscribe to events
Customer->>PSTN: Initiate a call
PSTN->>Communication Services: Forward call
Communication Services->>Event Grid: New call event
Event Grid->>API: Send event to event URL (HTTP webhook)
activate API
API->>Communication Services: Accept the call and give inbound URL
deactivate API
Communication Services->>Speech to Text: Transform speech to text
Communication Services->>API: Send text to the inbound URL
activate API
alt First call
API->>Communication Services: Send static SSML text
else Callback
API->>AI Search: Gather training data
API->>OpenAI GPT: Ask for a completion
OpenAI GPT-->>API: Answer (HTTP/2 SSE)
loop Over buffer
loop Over multiple tools
alt Is this a claim data update?
API->>Content Safety: Ask for safety test
alt Is the text safe?
API->>Communication Services: Send dynamic SSML text
end
API->>Cosmos DB: Update claim data
else Does the user want the human agent?
API->>Communication Services: Send static SSML text
API->>Communication Services: Transfer to a human
Communication Services->>Human agent: Call the phone number
else Should we end the call?
API->>Communication Services: Send static SSML text
API->>Communication Services: End the call
end
end
alt Is there a text?
alt Is there enough text to make a sentence?
API->>Content Safety: Ask for safety test
alt Is the text safe?
API->>Communication Services: Send dynamic SSML text
end
end
end
end
API->>Cosmos DB: Persist conversation
end
deactivate API
Communication Services->>PSTN: Send voice
PSTN->>Customer: Forward voice
```

## Local installation

### Prerequisites
Expand Down Expand Up @@ -170,6 +243,11 @@ ai_search:
endpoint: https://xxx.search.windows.net
index: trainings
semantic_configuration: default

content_safety:
access_key: xxx
blocklists: []
endpoint: https://xxx.cognitiveservices.azure.com
```
If you want to use a Service Principal to authenticate to Azure, you can also add the following in a `.env` file:
Expand Down Expand Up @@ -243,6 +321,9 @@ openai: {}
ai_search:
index: trainings
semantic_configuration: default
content_safety:
blocklists: []
```

Steps to deploy:
Expand Down
25 changes: 25 additions & 0 deletions bicep/app.bicep
View file
Original file line number Diff line number Diff line change
Expand Up @@ -130,6 +130,14 @@ resource containerApp 'Microsoft.App/containerApps@2023-05-01' = {
name: 'AI_SEARCH_ACCESS_KEY'
value: search.listAdminKeys().primaryKey
}
{
name: 'CONTENT_SAFETY_ENDPOINT'
value: cognitiveContentsafety.properties.endpoint
}
{
name: 'CONTENT_SAFETY_ACCESS_KEY'
value: cognitiveContentsafety.listKeys().key1
}
]
resources: {
cpu: 1
Expand Down Expand Up @@ -198,6 +206,23 @@ resource eventgridTopic 'Microsoft.EventGrid/systemTopics@2023-12-15-preview' =
}
}

resource cognitiveContentsafety 'Microsoft.CognitiveServices/accounts@2023-10-01-preview' = {
name: '${prefix}-contentsafety'
location: location
tags: tags
sku: {
name: 'F0'
}
kind: 'ContentSafety'
properties: {
customSubDomainName: '${prefix}-contentsafety'
publicNetworkAccess: 'Enabled'
networkAcls: {
defaultAction: 'Allow'
}
}
}

resource roleOpenaiContributor 'Microsoft.Authorization/roleDefinitions@2022-04-01' existing = {
name: 'a001fd3d-188f-4b5d-821b-7da978bf7442'
}
Expand Down
54 changes: 54 additions & 0 deletions examples/blocklist.csv
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,54 @@
blocklist,word
competitors,Aegon
competitors,AIA Group
competitors,Alleghany
competitors,Allianz
competitors,Allied World Assurance
competitors,Allstate
competitors,AmTrust Financial Services
competitors,Arch Capital Group
competitors,Argo Group International Holdings
competitors,Aspen Insurance Holdings
competitors,Assicurazioni Generali
competitors,Assurant
competitors,Aviva
competitors,AXA
competitors,Axis Capital Holdings
competitors,Baldwin & Lyons
competitors,China Life Insurance
competitors,China Pacific Insurance
competitors,Chubb
competitors,Cincinnati Financial
competitors,CNA Financial
competitors,Endurance Specialty Holdings
competitors,Everest Re Group
competitors,Fairfax Financial Holdings
competitors,Fidelity National Financial
competitors,First American Financial
competitors,Hanover Insurance Group
competitors,HCC Insurance Holdings
competitors,Legal & General Group
competitors,Liberty Mutual
competitors,Manulife Financial
competitors,Markel
competitors,Marsh & McLennan
competitors,MetLife
competitors,Munich Re
competitors,New York Life Insurance
competitors,Old Republic International
competitors,PartnerRe
competitors,Ping An Insurance
competitors,Progressive
competitors,Prudential Financial
competitors,RenaissanceRe Holdings
competitors,RSA Insurance Group
competitors,Selective Insurance Group
competitors,St. Paul Travelers
competitors,State Farm Insurance
competitors,Swiss Re
competitors,The Hartford
competitors,Torchmark
competitors,Travelers
competitors,Validus Holdings
competitors,Willis Group Holdings
competitors,Zurich Insurance Group
167 changes: 167 additions & 0 deletions examples/blocklist.ipynb
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,167 @@
{
"cells": [
{
"cell_type": "code",
"execution_count": 14,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Requirement already satisfied: azure-identity==1.15.0 in /Users/clemlesne/.pyenv/versions/3.11.3/lib/python3.11/site-packages (1.15.0)\n",
"Requirement already satisfied: azure-ai-contentsafety==1.0.0 in /Users/clemlesne/.pyenv/versions/3.11.3/lib/python3.11/site-packages (1.0.0)\n",
"Collecting pandas==2.2.0\n",
" Using cached pandas-2.2.0-cp311-cp311-macosx_11_0_arm64.whl.metadata (19 kB)\n",
"Requirement already satisfied: azure-core<2.0.0,>=1.23.0 in /Users/clemlesne/.pyenv/versions/3.11.3/lib/python3.11/site-packages (from azure-identity==1.15.0) (1.29.6)\n",
"Requirement already satisfied: cryptography>=2.5 in /Users/clemlesne/.pyenv/versions/3.11.3/lib/python3.11/site-packages (from azure-identity==1.15.0) (41.0.1)\n",
"Requirement already satisfied: msal<2.0.0,>=1.24.0 in /Users/clemlesne/.pyenv/versions/3.11.3/lib/python3.11/site-packages (from azure-identity==1.15.0) (1.25.0)\n",
"Requirement already satisfied: msal-extensions<2.0.0,>=0.3.0 in /Users/clemlesne/.pyenv/versions/3.11.3/lib/python3.11/site-packages (from azure-identity==1.15.0) (1.0.0)\n",
"Requirement already satisfied: isodate<1.0.0,>=0.6.1 in /Users/clemlesne/.pyenv/versions/3.11.3/lib/python3.11/site-packages (from azure-ai-contentsafety==1.0.0) (0.6.1)\n",
"Requirement already satisfied: numpy<2,>=1.23.2 in /Users/clemlesne/.pyenv/versions/3.11.3/lib/python3.11/site-packages (from pandas==2.2.0) (1.26.1)\n",
"Requirement already satisfied: python-dateutil>=2.8.2 in /Users/clemlesne/.pyenv/versions/3.11.3/lib/python3.11/site-packages (from pandas==2.2.0) (2.8.2)\n",
"Requirement already satisfied: pytz>=2020.1 in /Users/clemlesne/.pyenv/versions/3.11.3/lib/python3.11/site-packages (from pandas==2.2.0) (2023.3)\n",
"Requirement already satisfied: tzdata>=2022.7 in /Users/clemlesne/.pyenv/versions/3.11.3/lib/python3.11/site-packages (from pandas==2.2.0) (2023.3)\n",
"Requirement already satisfied: anyio<5.0,>=3.0 in /Users/clemlesne/.pyenv/versions/3.11.3/lib/python3.11/site-packages (from azure-core<2.0.0,>=1.23.0->azure-identity==1.15.0) (3.7.1)\n",
"Requirement already satisfied: requests>=2.21.0 in /Users/clemlesne/.pyenv/versions/3.11.3/lib/python3.11/site-packages (from azure-core<2.0.0,>=1.23.0->azure-identity==1.15.0) (2.31.0)\n",
"Requirement already satisfied: six>=1.11.0 in /Users/clemlesne/.pyenv/versions/3.11.3/lib/python3.11/site-packages (from azure-core<2.0.0,>=1.23.0->azure-identity==1.15.0) (1.16.0)\n",
"Requirement already satisfied: typing-extensions>=4.6.0 in /Users/clemlesne/.pyenv/versions/3.11.3/lib/python3.11/site-packages (from azure-core<2.0.0,>=1.23.0->azure-identity==1.15.0) (4.8.0)\n",
"Requirement already satisfied: cffi>=1.12 in /Users/clemlesne/.pyenv/versions/3.11.3/lib/python3.11/site-packages (from cryptography>=2.5->azure-identity==1.15.0) (1.15.1)\n",
"Requirement already satisfied: PyJWT<3,>=1.0.0 in /Users/clemlesne/.pyenv/versions/3.11.3/lib/python3.11/site-packages (from PyJWT[crypto]<3,>=1.0.0->msal<2.0.0,>=1.24.0->azure-identity==1.15.0) (2.8.0)\n",
"Requirement already satisfied: portalocker<3,>=1.0 in /Users/clemlesne/.pyenv/versions/3.11.3/lib/python3.11/site-packages (from msal-extensions<2.0.0,>=0.3.0->azure-identity==1.15.0) (2.7.0)\n",
"Requirement already satisfied: idna>=2.8 in /Users/clemlesne/.pyenv/versions/3.11.3/lib/python3.11/site-packages (from anyio<5.0,>=3.0->azure-core<2.0.0,>=1.23.0->azure-identity==1.15.0) (3.4)\n",
"Requirement already satisfied: sniffio>=1.1 in /Users/clemlesne/.pyenv/versions/3.11.3/lib/python3.11/site-packages (from anyio<5.0,>=3.0->azure-core<2.0.0,>=1.23.0->azure-identity==1.15.0) (1.3.0)\n",
"Requirement already satisfied: pycparser in /Users/clemlesne/.pyenv/versions/3.11.3/lib/python3.11/site-packages (from cffi>=1.12->cryptography>=2.5->azure-identity==1.15.0) (2.21)\n",
"Requirement already satisfied: charset-normalizer<4,>=2 in /Users/clemlesne/.pyenv/versions/3.11.3/lib/python3.11/site-packages (from requests>=2.21.0->azure-core<2.0.0,>=1.23.0->azure-identity==1.15.0) (3.2.0)\n",
"Requirement already satisfied: urllib3<3,>=1.21.1 in /Users/clemlesne/.pyenv/versions/3.11.3/lib/python3.11/site-packages (from requests>=2.21.0->azure-core<2.0.0,>=1.23.0->azure-identity==1.15.0) (1.26.16)\n",
"Requirement already satisfied: certifi>=2017.4.17 in /Users/clemlesne/.pyenv/versions/3.11.3/lib/python3.11/site-packages (from requests>=2.21.0->azure-core<2.0.0,>=1.23.0->azure-identity==1.15.0) (2023.5.7)\n",
"Using cached pandas-2.2.0-cp311-cp311-macosx_11_0_arm64.whl (11.8 MB)\n",
"Installing collected packages: pandas\n",
" Attempting uninstall: pandas\n",
" Found existing installation: pandas 2.0.3\n",
" Uninstalling pandas-2.0.3:\n",
" Successfully uninstalled pandas-2.0.3\n",
"Successfully installed pandas-2.2.0\n",
"Note: you may need to restart the kernel to use updated packages.\n"
]
}
],
"source": [
"%pip install azure-identity==1.15.0 azure-ai-contentsafety==1.0.0 pandas==2.2.0"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Initialize clients and access keys."
]
},
{
"cell_type": "code",
"execution_count": 32,
"metadata": {},
"outputs": [],
"source": [
"from azure.ai.contentsafety import BlocklistClient\n",
"from azure.ai.contentsafety.models import TextBlocklist, TextBlocklistItem, AddOrUpdateTextBlocklistItemsOptions\n",
"from azure.core.credentials import AzureKeyCredential\n",
"import pandas as pd\n",
"\n",
"key = AzureKeyCredential(\"19cbfb37271e4d5a9fafba91b35897a5\")\n",
"client = BlocklistClient(\"https://groupama-diva-poc-contentsafety.cognitiveservices.azure.com/\", key)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"List current blocklists."
]
},
{
"cell_type": "code",
"execution_count": 37,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"competitors: Competitors blocklist.\n"
]
}
],
"source": [
"blocklists = client.list_text_blocklists()\n",
"\n",
"if not blocklists:\n",
" print(\"There are no blocklists.\")\n",
"\n",
"for blocklist in blocklists:\n",
" print(f\"{blocklist.blocklist_name}: {blocklist.description}\")"
]
},
{
"cell_type": "code",
"execution_count": 38,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Creating blocklist competitors with 53 words\n"
]
}
],
"source": [
"name = \"competitors\"\n",
"description = \"Competitors blocklist.\"\n",
"\n",
"df = pd.read_csv(\"blocklist.csv\")\n",
"\n",
"block_items = {}\n",
"for index, row in df.iterrows():\n",
" text = TextBlocklistItem(text=row[\"word\"])\n",
" if row[\"blocklist\"] not in block_items:\n",
" block_items[row[\"blocklist\"]] = []\n",
" block_items[row[\"blocklist\"]].append(text)\n",
"\n",
"for blocklist, words in block_items.items():\n",
" print(f\"Creating blocklist {blocklist} with {len(words)} words\")\n",
" client.create_or_update_text_blocklist(\n",
" blocklist_name=blocklist,\n",
" options=TextBlocklist(blocklist_name=blocklist),\n",
" )\n",
" client.add_or_update_blocklist_items(\n",
" blocklist_name=blocklist,\n",
" options=AddOrUpdateTextBlocklistItemsOptions(\n",
" blocklist_items=words\n",
" )\n",
" )"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.3"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
9 changes: 9 additions & 0 deletions helpers/config_models/content_safety.py
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
from typing import List
from pydantic import SecretStr
from pydantic_settings import BaseSettings


class ContentSafetyModel(BaseSettings, env_prefix="content_safety_"):
access_key: SecretStr
blocklists: List[str]
endpoint: str
Loading

0 comments on commit 68b693b

Please sign in to comment.