From 9dfcbe83d046e365aaefaa2f9c9bf2c8366cd235 Mon Sep 17 00:00:00 2001 From: r12f Date: Thu, 16 Nov 2023 00:12:01 +0000 Subject: [PATCH 1/9] temp checkin. --- .../dataplane/dash-custom-data-plane-app.md | 15 +++++++++++++++ 1 file changed, 15 insertions(+) create mode 100644 documentation/dataplane/dash-custom-data-plane-app.md diff --git a/documentation/dataplane/dash-custom-data-plane-app.md b/documentation/dataplane/dash-custom-data-plane-app.md new file mode 100644 index 000000000..2b809e10c --- /dev/null +++ b/documentation/dataplane/dash-custom-data-plane-app.md @@ -0,0 +1,15 @@ +# DASH custom data plane app + +## Custom data plane app overview + +![DASH data path overview](../general/images/dash-data-path-overview.svg) + +## Initialization + +### Host interface creation + +### RSS queue support + +Switch attributes. + +## Hybrid mode From cb5ed87c9f2c0a97a430831510ec8da39f75056c Mon Sep 17 00:00:00 2001 From: r12f Date: Fri, 17 Nov 2023 22:44:05 +0000 Subject: [PATCH 2/9] byo data plane app design. --- .../dataplane/dash-byo-data-plane-app.md | 142 ++++++++++++++++++ .../dataplane/dash-custom-data-plane-app.md | 15 -- .../images/dash-byo-data-plane-app.svg | 4 + .../images/dash-default-data-plane-app.svg | 4 + 4 files changed, 150 insertions(+), 15 deletions(-) create mode 100644 documentation/dataplane/dash-byo-data-plane-app.md delete mode 100644 documentation/dataplane/dash-custom-data-plane-app.md create mode 100644 documentation/dataplane/images/dash-byo-data-plane-app.svg create mode 100644 documentation/dataplane/images/dash-default-data-plane-app.svg diff --git a/documentation/dataplane/dash-byo-data-plane-app.md b/documentation/dataplane/dash-byo-data-plane-app.md new file mode 100644 index 000000000..be27e7ca0 --- /dev/null +++ b/documentation/dataplane/dash-byo-data-plane-app.md @@ -0,0 +1,142 @@ +# DASH BYO (Bring-Your-Own) data plane app + +1. [1. Data plane app](#1-data-plane-app) +2. [2. BYO data plane app](#2-byo-data-plane-app) + 1. [2.1. Work flow](#21-work-flow) + 2. [2.2. DASH management role and ASIC programming](#22-dash-management-role-and-asic-programming) + 3. [2.3. Flow management](#23-flow-management) + 4. [2.4. RSS support](#24-rss-support) +3. [3. SAI API design](#3-sai-api-design) + 1. [3.1. SAI switch attribute updates](#31-sai-switch-attribute-updates) + 2. [3.2. DASH pipeline programming APIs](#32-dash-pipeline-programming-apis) +4. [4. Future work](#4-future-work) + +## 1. Data plane app + +Data plane app is one of the most important pieces in DASH pipeline. It is mainly responsible for: + +- Flow management, such as flow creation, deletion, resimulation, etc. +- Work with ASIC to complete the table lookup and packet transformations. Depends on ASIC capability, if any operation could not be completed inside ASIC, it can be postponed to data plane app and completed in software. +- ASIC management, such as initializing and programming match stage, etc. +- and more. + +![DASH data plane app](./images/dash-default-data-plane-app.svg) + +By default, DASH technology providers will provide their own data plane app. This is usually closed source and can satisfy the DASH community requirement. However, in some cases, you might want to use your own data plane app due to some reasons, such as: private protocol support or support special logic in packet handling that is not supported by DASH technology providers. In this case, you can enable DASH BYO data plane app. + +## 2. BYO data plane app + +BYO data plane app is essentially a DPDK application that directly interacts with the data plane netdev. + +When BYO data plane app is enabled, the data plane app provided by technology providers will be either disabled or running without tied to the data plane netdev and only act as part of SAI API implementation if needed. With this setup, DASH users can start to use that device to run their own data plane app and process the packets. + +After the change, at a high level, the system architecture will be look like below: + +![DASH BYO data plane app](./images/dash-byo-data-plane-app.svg) + +### 2.1. Work flow + +The work flow of enabling the BYO data plane app will be like below: + +```mermaid +sequenceDiagram + +participant User +participant SAI +participant IDPA as DASH Inbox data plane app +participant BYODPA as BYO data plane app +participant netdev + +User->>SAI: SAI create switch
with controller role
and settings +SAI->>IDPA: Configurate inbox
data plane app +User->>SAI: Get netdev name
as switch attribute +User->>BYODPA: Launch and configurate BYO data plane app +BYODPA->>SAI: SAI create switch with worker role +BYODPA->>netdev: Initialize on top of netdev +``` + +After initialization, the data plane app will be able to: + +- Receive/Send packets from/to netdev +- Use SAI API to program the ASIC + +### 2.2. DASH management role and ASIC programming + +There are 2 roles ASIC management roles for DASH: + +- Controller: When calling create switch with controller role, it will setup the ASIC, configurate the inbox data plane app, etc. +- Worker: When calling create switch with worker role, it will only initialize the SDK for calling SAI APIs. + +This allows the data plane app also be able to program the ASIC, so that data plane app can implement features such as: flow management, match stage entry eviction, etc. + +> **NOTE**: +> +> Please bare in mind that - Although some APIs can be accessed by both roles, such as creating match stage entries, but this is not a good practice, because the ASIC state will be overwritten by the last caller without any synchronization and knowledge from the other side. So, please make sure that the APIs are only called by one side. + +### 2.3. Flow management + +One of the most important responsibility of data plane app is flow management. Essentially, whenever a packet that cannot be handled by the hardware flow table, it will run through the DASH pipeline and sent to the data plane app. The data plane app will need to decide if a flow needs to be created, deleted or resimulated, and how. + +In DASH-SAI APIs, we have provided a set of APIs to help manage the flows. Please refer to DASH Flow API design doc for more details. + +### 2.4. RSS support + +RSS is a frequently used feature in data plane app, which allows the packets to be distributed among different worker threads to increase throughput. + +To enable and configurate RSS, we don't have any extra APIs in SAI, and BYO data plane app can follow the standard way in DPDK to achieve this. + +## 3. SAI API design + +### 3.1. SAI switch attribute updates + +```c +typedef enum _sai_dash_management_role_t { + /** + * @brief Controller role. + */ + SAI_DASH_MANAGEMENT_ROLE_CONTROLLER, + + /** + * @brief Worker role. Should only be used by BYO data plane app. + */ + SAI_DASH_MANAGEMENT_ROLE_WORKER, +} sai_dash_management_role_t; + +typedef enum _sai_switch_attr_extensions_t { + // ... + + + /** + * @brief DASH management role. + * + * @type sai_dash_management_role_t + * @flags CREATE_ONLY + */ + SAI_SWITCH_ATTR_DASH_MANAGEMENT_ROLE, + + /** + * @brief Inbox data plane app worker count. (-1 = Disable, 0 = Auto, > 0 = Specified count) + * + * @type sai_int32_t + * @flags CREATE_AND_SET + * @default 0 + */ + SAI_SWITCH_ATTR_DASH_INBOX_DATA_PLANE_APP_WORKER_COUNT, + + /** + * @brief BYO data plane app netdev name. + * + * @type sai_s8_list_t + * @flags READ_ONLY + */ + SAI_SWITCH_ATTR_DASH_BYO_DATA_PLANE_APP_NETDEV_NAME, +} sai_switch_attr_extensions_t; +``` + +### 3.2. DASH pipeline programming APIs + +Please refer to the DASH-SAI APIs for more details: . + +## 4. Future work + +- Hybrid data plane app, where both inbox and BYO data plane app are enabled at the same time. diff --git a/documentation/dataplane/dash-custom-data-plane-app.md b/documentation/dataplane/dash-custom-data-plane-app.md deleted file mode 100644 index 2b809e10c..000000000 --- a/documentation/dataplane/dash-custom-data-plane-app.md +++ /dev/null @@ -1,15 +0,0 @@ -# DASH custom data plane app - -## Custom data plane app overview - -![DASH data path overview](../general/images/dash-data-path-overview.svg) - -## Initialization - -### Host interface creation - -### RSS queue support - -Switch attributes. - -## Hybrid mode diff --git a/documentation/dataplane/images/dash-byo-data-plane-app.svg b/documentation/dataplane/images/dash-byo-data-plane-app.svg new file mode 100644 index 000000000..39559ae65 --- /dev/null +++ b/documentation/dataplane/images/dash-byo-data-plane-app.svg @@ -0,0 +1,4 @@ + + + +
Hardware
Hardware
ASIC
ASIC
Port
Port
 data plane netdev
 data plane netdev
Kernel Space
Kernel Space
ASIC Driver
ASIC Driver
User Space
User Space
BYO Data Plane App
BYO Data Plane App
lcore_0
lcore_0
RxQ
RxQ
TxQ
TxQ
Logic
Logic
lcore_1
lcore_1
RxQ
RxQ
TxQ
TxQ
Logic
Logic
lcore_n
lcore_n
RxQ
RxQ
TxQ
TxQ
Logic
Logic
...
...
DASH SAI APIs
DASH SAI APIs
Technology Provider ASIC SDK
Technology Provider ASIC SDK
Data Plane App
Data Plane App
ASIC SDK
ASIC SDK
Controller
Cont...
Control Path
Control...
Data Path
Data Pa...Text is not SVG - cannot display \ No newline at end of file diff --git a/documentation/dataplane/images/dash-default-data-plane-app.svg b/documentation/dataplane/images/dash-default-data-plane-app.svg new file mode 100644 index 000000000..09ee27aac --- /dev/null +++ b/documentation/dataplane/images/dash-default-data-plane-app.svg @@ -0,0 +1,4 @@ + + + +
Hardware
Hardware
ASIC
ASIC
Port
Port
 data plane netdev
 data plane netdev
Kernel Space
Kernel Space
ASIC Driver
ASIC Driver
User Space
User Space
Technology Provider ASIC SDK
Technology Provide...
Technology Provider Data Plane App
Technology Provider Data Plane App
lcore_0
lcore_0
RxQ
RxQ
TxQ
TxQ
Logic
Logic
lcore_1
lcore_1
RxQ
RxQ
TxQ
TxQ
Logic
Logic
lcore_n
lcore_n
RxQ
RxQ
TxQ
TxQ
Logic
Logic
...
...
DASH SAI APIs
DASH SAI APIs
Controller
Cont...
Control Path
Control...
Data Path
Data Pa...Text is not SVG - cannot display \ No newline at end of file From 9ef0a62fee43b7cbc6014f83795f5a86536e0bee Mon Sep 17 00:00:00 2001 From: r12f Date: Fri, 17 Nov 2023 23:13:18 +0000 Subject: [PATCH 3/9] fix spellcheck. --- .wordlist.txt | 3 +++ documentation/dataplane/dash-byo-data-plane-app.md | 8 ++++---- 2 files changed, 7 insertions(+), 4 deletions(-) diff --git a/.wordlist.txt b/.wordlist.txt index 032da645f..fc1b07b8e 100644 --- a/.wordlist.txt +++ b/.wordlist.txt @@ -66,6 +66,7 @@ buildimage BulkSyncDone BW bw +BYO bz callout cd @@ -360,6 +361,7 @@ natted NAT'ing nb nd +netdev netdevs Nexthop nexthop @@ -488,6 +490,7 @@ roadmap routable RPC RPCs +RSS RST RSTPackets RSTs diff --git a/documentation/dataplane/dash-byo-data-plane-app.md b/documentation/dataplane/dash-byo-data-plane-app.md index be27e7ca0..696da69e9 100644 --- a/documentation/dataplane/dash-byo-data-plane-app.md +++ b/documentation/dataplane/dash-byo-data-plane-app.md @@ -48,9 +48,9 @@ participant BYODPA as BYO data plane app participant netdev User->>SAI: SAI create switch
with controller role
and settings -SAI->>IDPA: Configurate inbox
data plane app +SAI->>IDPA: configure inbox
data plane app User->>SAI: Get netdev name
as switch attribute -User->>BYODPA: Launch and configurate BYO data plane app +User->>BYODPA: Launch and configure BYO data plane app BYODPA->>SAI: SAI create switch with worker role BYODPA->>netdev: Initialize on top of netdev ``` @@ -64,7 +64,7 @@ After initialization, the data plane app will be able to: There are 2 roles ASIC management roles for DASH: -- Controller: When calling create switch with controller role, it will setup the ASIC, configurate the inbox data plane app, etc. +- Controller: When calling create switch with controller role, it will setup the ASIC, configure the inbox data plane app, etc. - Worker: When calling create switch with worker role, it will only initialize the SDK for calling SAI APIs. This allows the data plane app also be able to program the ASIC, so that data plane app can implement features such as: flow management, match stage entry eviction, etc. @@ -83,7 +83,7 @@ In DASH-SAI APIs, we have provided a set of APIs to help manage the flows. Pleas RSS is a frequently used feature in data plane app, which allows the packets to be distributed among different worker threads to increase throughput. -To enable and configurate RSS, we don't have any extra APIs in SAI, and BYO data plane app can follow the standard way in DPDK to achieve this. +To enable and configure RSS, we don't have any extra APIs in SAI, and BYO data plane app can follow the standard way in DPDK to achieve this. ## 3. SAI API design From f03c328cf825c7b8429d202f74b62d5668103ff7 Mon Sep 17 00:00:00 2001 From: r12f Date: Tue, 28 Nov 2023 23:25:24 +0000 Subject: [PATCH 4/9] More details on how the 2 roles works. --- .../dataplane/dash-byo-data-plane-app.md | 83 +++++++++++++++---- .../images/dash-byo-data-plane-app.svg | 2 +- 2 files changed, 66 insertions(+), 19 deletions(-) diff --git a/documentation/dataplane/dash-byo-data-plane-app.md b/documentation/dataplane/dash-byo-data-plane-app.md index 696da69e9..d03782a34 100644 --- a/documentation/dataplane/dash-byo-data-plane-app.md +++ b/documentation/dataplane/dash-byo-data-plane-app.md @@ -2,10 +2,11 @@ 1. [1. Data plane app](#1-data-plane-app) 2. [2. BYO data plane app](#2-byo-data-plane-app) - 1. [2.1. Work flow](#21-work-flow) - 2. [2.2. DASH management role and ASIC programming](#22-dash-management-role-and-asic-programming) - 3. [2.3. Flow management](#23-flow-management) - 4. [2.4. RSS support](#24-rss-support) + 1. [2.1. DASH management role and system overview](#21-dash-management-role-and-system-overview) + 2. [2.2. Initialization work flow](#22-initialization-work-flow) + 3. [2.3. ASIC programming work flow](#23-asic-programming-work-flow) + 4. [2.4. Flow management](#24-flow-management) + 5. [2.5. RSS support](#25-rss-support) 3. [3. SAI API design](#3-sai-api-design) 1. [3.1. SAI switch attribute updates](#31-sai-switch-attribute-updates) 2. [3.2. DASH pipeline programming APIs](#32-dash-pipeline-programming-apis) @@ -30,11 +31,29 @@ BYO data plane app is essentially a DPDK application that directly interacts wit When BYO data plane app is enabled, the data plane app provided by technology providers will be either disabled or running without tied to the data plane netdev and only act as part of SAI API implementation if needed. With this setup, DASH users can start to use that device to run their own data plane app and process the packets. -After the change, at a high level, the system architecture will be look like below: +### 2.1. DASH management role and system overview + +With BYO data plane app, from our customer's prespective, we will have 2 sources to program the ASIC, such as creating match stage entries or managing flow entries: + +1. DASH users can explicitly program the ASIC via DASH SAI API proxy. A typical scenario is programming a new SDN policy. +2. BYO data plane app can also program the ASIC. A typical scenario is creating flow entries. + +Even further, these 2 sources sometimes might need to work together in certain scenarios. For example, when a VNET mapping is updated, we need to update the VNET mapping entry as well as trigging flow resimulation to update all the flow entries that are related to this VNET mapping. + +Hence we need a design to avoid the same set of SAI APIs being called in 2 different processes accidentally and causing problems, such as managing the stage entries, as the last caller will overwrite the ASIC state without any synchronization and knowledge from the other side. + +To solve this problem, we are introducing 2 roles in DASH: + +- Controller, which is responsible for initializing the ASIC and envirnoments. +- Worker, which is responsible for processing the packets of new flows, programming the match stage entries, managing the flows and more. + +At a high level, the system architecture will be look like below: ![DASH BYO data plane app](./images/dash-byo-data-plane-app.svg) -### 2.1. Work flow +To explain how these 2 roles works together, we will dive into the initialization and ASIC programming work flow here. + +### 2.2. Initialization work flow The work flow of enabling the BYO data plane app will be like below: @@ -42,30 +61,58 @@ The work flow of enabling the BYO data plane app will be like below: sequenceDiagram participant User -participant SAI +participant SP as SAI Proxy +participant SC as SAI (Controller) participant IDPA as DASH Inbox data plane app participant BYODPA as BYO data plane app +participant SW as SAI (Worker) participant netdev -User->>SAI: SAI create switch
with controller role
and settings -SAI->>IDPA: configure inbox
data plane app -User->>SAI: Get netdev name
as switch attribute +note over User,netdev: Controller initialization +User->>SP: SAI create switch
with controller role
and settings +SP->>SC: Forward request to SAI +SC->>SC: Initialize card
envirionment +SC->>IDPA: Configure inbox
data plane app +User->>SP: Get netdev name
as switch attribute +SP->>SC: Get netdev name
as switch attribute +note over User,netdev: BYO data plane app initialization User->>BYODPA: Launch and configure BYO data plane app -BYODPA->>SAI: SAI create switch with worker role +BYODPA->>SW: SAI create switch with worker role +SW->>SP: Connect to proxy for handling stage entry and flow management request BYODPA->>netdev: Initialize on top of netdev ``` -After initialization, the data plane app will be able to: +After initialization, the BYO data plane app will be able to: - Receive/Send packets from/to netdev - Use SAI API to program the ASIC -### 2.2. DASH management role and ASIC programming +> **NOTE**: +> +> Since BYO data plane app and DASH controller is essentially provided by our customer, so there are a few more things that BYO data plane app could do, but not listed in the diagram: +> +> 1. Controller can directly call into BYO data plane app for managing private features that is not provided by DASH. +> 2. BYO data plane app could call into the platform-dependent APIs (ASIC SDK) directly for managing the ASIC. However, by doing so, it also loses the portability of the BYO data plane app. -There are 2 roles ASIC management roles for DASH: +### 2.3. ASIC programming work flow -- Controller: When calling create switch with controller role, it will setup the ASIC, configure the inbox data plane app, etc. -- Worker: When calling create switch with worker role, it will only initialize the SDK for calling SAI APIs. +```mermaid +sequenceDiagram + +participant User +participant SP as SAI Proxy +participant BYODPA as BYO data plane app +participant SW as SAI (Worker) + +note over User,SW: User update a mapping entry +User->>SP: SAI create mapping entry +SP->>BYODPA: Forward request to worker +BYODPA->>SW: Update mapping entry +BYODPA->>SW: Trigger flow resimulation or other actions if needed. + +note over User,SW: BYO data plane app update a flow entry +BYODPA->>SW: SAI update flow entry +``` This allows the data plane app also be able to program the ASIC, so that data plane app can implement features such as: flow management, match stage entry eviction, etc. @@ -73,13 +120,13 @@ This allows the data plane app also be able to program the ASIC, so that data pl > > Please bare in mind that - Although some APIs can be accessed by both roles, such as creating match stage entries, but this is not a good practice, because the ASIC state will be overwritten by the last caller without any synchronization and knowledge from the other side. So, please make sure that the APIs are only called by one side. -### 2.3. Flow management +### 2.4. Flow management One of the most important responsibility of data plane app is flow management. Essentially, whenever a packet that cannot be handled by the hardware flow table, it will run through the DASH pipeline and sent to the data plane app. The data plane app will need to decide if a flow needs to be created, deleted or resimulated, and how. In DASH-SAI APIs, we have provided a set of APIs to help manage the flows. Please refer to DASH Flow API design doc for more details. -### 2.4. RSS support +### 2.5. RSS support RSS is a frequently used feature in data plane app, which allows the packets to be distributed among different worker threads to increase throughput. diff --git a/documentation/dataplane/images/dash-byo-data-plane-app.svg b/documentation/dataplane/images/dash-byo-data-plane-app.svg index 39559ae65..0d3316091 100644 --- a/documentation/dataplane/images/dash-byo-data-plane-app.svg +++ b/documentation/dataplane/images/dash-byo-data-plane-app.svg @@ -1,4 +1,4 @@ -
Hardware
Hardware
ASIC
ASIC
Port
Port
 data plane netdev
 data plane netdev
Kernel Space
Kernel Space
ASIC Driver
ASIC Driver
User Space
User Space
BYO Data Plane App
BYO Data Plane App
lcore_0
lcore_0
RxQ
RxQ
TxQ
TxQ
Logic
Logic
lcore_1
lcore_1
RxQ
RxQ
TxQ
TxQ
Logic
Logic
lcore_n
lcore_n
RxQ
RxQ
TxQ
TxQ
Logic
Logic
...
...
DASH SAI APIs
DASH SAI APIs
Technology Provider ASIC SDK
Technology Provider ASIC SDK
Data Plane App
Data Plane App
ASIC SDK
ASIC SDK
Controller
Cont...
Control Path
Control...
Data Path
Data Pa...Text is not SVG - cannot display \ No newline at end of file +
Hardware
Hardware
ASIC
ASIC
Port
Port
 data plane netdev
 data plane netdev
Kernel Space
Kernel Space
ASIC Driver
ASIC Driver
User Space
User Space
BYO Data Plane App
BYO Data Plane App
lcore_0
lcore_0
Logic
Logic
lcore_1
lcore_1
Logic
Logic
lcore_n
lcore_n
Logic
Logic
RxQ
RxQ
RxQ
RxQ
TxQ
TxQ
TxQ
TxQ
RxQ
RxQ
TxQ
TxQ
...
...
DASH SAI API Handler
DASH SAI API...
DASH SAI APIs
(Worker Role)
DASH SAI APIs...
DASH SAI API Proxy
DASH SAI API Proxy
Technology Provider ASIC SDK
Technology Provider ASIC SDK
Data Plane App
Data Plane App
ASIC SDK
ASIC SDK
DASH SAI APIs
(Controller Role)
DASH SAI APIs...
Controller
Cont...
Control Path
Control...
Data Path
Data Pa...Text is not SVG - cannot display \ No newline at end of file From 47fe405e352fcc033ccb36da1c5e4e90f7a58f24 Mon Sep 17 00:00:00 2001 From: r12f Date: Tue, 28 Nov 2023 23:28:08 +0000 Subject: [PATCH 5/9] Minor update on missing content. --- documentation/dataplane/dash-byo-data-plane-app.md | 10 ++++------ 1 file changed, 4 insertions(+), 6 deletions(-) diff --git a/documentation/dataplane/dash-byo-data-plane-app.md b/documentation/dataplane/dash-byo-data-plane-app.md index d03782a34..83fcbc3bc 100644 --- a/documentation/dataplane/dash-byo-data-plane-app.md +++ b/documentation/dataplane/dash-byo-data-plane-app.md @@ -96,6 +96,10 @@ After initialization, the BYO data plane app will be able to: ### 2.3. ASIC programming work flow +With this setup, whenever we need to program the ASIC with new entries, we will forward the request to the data plane app. + +Here is the example that shows how the users updates an existing mapping entry with flow resimulation, as well as how a BYO data plane app updates a flow entry: + ```mermaid sequenceDiagram @@ -114,12 +118,6 @@ note over User,SW: BYO data plane app update a flow entry BYODPA->>SW: SAI update flow entry ``` -This allows the data plane app also be able to program the ASIC, so that data plane app can implement features such as: flow management, match stage entry eviction, etc. - -> **NOTE**: -> -> Please bare in mind that - Although some APIs can be accessed by both roles, such as creating match stage entries, but this is not a good practice, because the ASIC state will be overwritten by the last caller without any synchronization and knowledge from the other side. So, please make sure that the APIs are only called by one side. - ### 2.4. Flow management One of the most important responsibility of data plane app is flow management. Essentially, whenever a packet that cannot be handled by the hardware flow table, it will run through the DASH pipeline and sent to the data plane app. The data plane app will need to decide if a flow needs to be created, deleted or resimulated, and how. From 51facf16a22df405b8b307a2872038d8f97b9e16 Mon Sep 17 00:00:00 2001 From: r12f Date: Tue, 28 Nov 2023 23:31:40 +0000 Subject: [PATCH 6/9] fix spellchecks. --- documentation/dataplane/dash-byo-data-plane-app.md | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/documentation/dataplane/dash-byo-data-plane-app.md b/documentation/dataplane/dash-byo-data-plane-app.md index 83fcbc3bc..8fd70e75d 100644 --- a/documentation/dataplane/dash-byo-data-plane-app.md +++ b/documentation/dataplane/dash-byo-data-plane-app.md @@ -33,18 +33,18 @@ When BYO data plane app is enabled, the data plane app provided by technology pr ### 2.1. DASH management role and system overview -With BYO data plane app, from our customer's prespective, we will have 2 sources to program the ASIC, such as creating match stage entries or managing flow entries: +With BYO data plane app, from our customer's perspective, we will have 2 sources to program the ASIC, such as creating match stage entries or managing flow entries: 1. DASH users can explicitly program the ASIC via DASH SAI API proxy. A typical scenario is programming a new SDN policy. 2. BYO data plane app can also program the ASIC. A typical scenario is creating flow entries. -Even further, these 2 sources sometimes might need to work together in certain scenarios. For example, when a VNET mapping is updated, we need to update the VNET mapping entry as well as trigging flow resimulation to update all the flow entries that are related to this VNET mapping. +Even further, these 2 sources sometimes might need to work together in certain scenarios. For example, when a VNET mapping is updated, we need to update the VNET mapping entry as well as triggering flow resimulation to update all the flow entries that are related to this VNET mapping. Hence we need a design to avoid the same set of SAI APIs being called in 2 different processes accidentally and causing problems, such as managing the stage entries, as the last caller will overwrite the ASIC state without any synchronization and knowledge from the other side. To solve this problem, we are introducing 2 roles in DASH: -- Controller, which is responsible for initializing the ASIC and envirnoments. +- Controller, which is responsible for initializing the ASIC and environments. - Worker, which is responsible for processing the packets of new flows, programming the match stage entries, managing the flows and more. At a high level, the system architecture will be look like below: @@ -75,6 +75,7 @@ SC->>SC: Initialize card
envirionment SC->>IDPA: Configure inbox
data plane app User->>SP: Get netdev name
as switch attribute SP->>SC: Get netdev name
as switch attribute + note over User,netdev: BYO data plane app initialization User->>BYODPA: Launch and configure BYO data plane app BYODPA->>SW: SAI create switch with worker role @@ -108,14 +109,14 @@ participant SP as SAI Proxy participant BYODPA as BYO data plane app participant SW as SAI (Worker) -note over User,SW: User update a mapping entry +note over User,SW: User updates a mapping entry User->>SP: SAI create mapping entry SP->>BYODPA: Forward request to worker BYODPA->>SW: Update mapping entry BYODPA->>SW: Trigger flow resimulation or other actions if needed. -note over User,SW: BYO data plane app update a flow entry -BYODPA->>SW: SAI update flow entry +note over User,SW: BYO data plane app updates a flow entry +BYODPA->>SW: SAI updates flow entry ``` ### 2.4. Flow management From bb05e98f60ca35b9b6bdbe576d962b8227058e50 Mon Sep 17 00:00:00 2001 From: r12f Date: Thu, 30 Nov 2023 01:45:41 +0000 Subject: [PATCH 7/9] Add notification when BYO data plane app is launched. --- .../dataplane/dash-byo-data-plane-app.md | 21 +++++++++++++++++-- 1 file changed, 19 insertions(+), 2 deletions(-) diff --git a/documentation/dataplane/dash-byo-data-plane-app.md b/documentation/dataplane/dash-byo-data-plane-app.md index 8fd70e75d..33670ac05 100644 --- a/documentation/dataplane/dash-byo-data-plane-app.md +++ b/documentation/dataplane/dash-byo-data-plane-app.md @@ -65,6 +65,7 @@ participant SP as SAI Proxy participant SC as SAI (Controller) participant IDPA as DASH Inbox data plane app participant BYODPA as BYO data plane app +participant SH as SAI Handler participant SW as SAI (Worker) participant netdev @@ -78,8 +79,12 @@ SP->>SC: Get netdev name
as switch attribute note over User,netdev: BYO data plane app initialization User->>BYODPA: Launch and configure BYO data plane app -BYODPA->>SW: SAI create switch with worker role -SW->>SP: Connect to proxy for handling stage entry and flow management request +BYODPA->>SH: Initialize +activate SH +SH->>SW: SAI create switch
with worker role +SH->>SP: Connect to proxy for handling stage entry and flow management request +SP->>User: Send BYO data plane app
ready notification +deactivate SH BYODPA->>netdev: Initialize on top of netdev ``` @@ -148,6 +153,9 @@ typedef enum _sai_dash_management_role_t { SAI_DASH_MANAGEMENT_ROLE_WORKER, } sai_dash_management_role_t; +typedef void (*sai_switch_dash_byo_data_plane_app_ready_notification_fn)( + _In_ sai_object_id_t switch_id); + typedef enum _sai_switch_attr_extensions_t { // ... @@ -176,6 +184,15 @@ typedef enum _sai_switch_attr_extensions_t { * @flags READ_ONLY */ SAI_SWITCH_ATTR_DASH_BYO_DATA_PLANE_APP_NETDEV_NAME, + + /** + * @brief Vendor specific path name of the firmware to load. + * + * @type sai_switch_dash_byo_data_plane_app_ready_notification_fn + * @flags CREATE_ONLY + * @default NULL + */ + SAI_SWITCH_ATTR_DASH_BYO_DATA_PLANE_APP_READY_NOTIFY, } sai_switch_attr_extensions_t; ``` From f1c5612b3c1a1aad040a6edfdd12c220c5aee262 Mon Sep 17 00:00:00 2001 From: r12f Date: Thu, 30 Nov 2023 01:54:56 +0000 Subject: [PATCH 8/9] Addressing comment from the community call today. --- documentation/dataplane/dash-byo-data-plane-app.md | 1 + 1 file changed, 1 insertion(+) diff --git a/documentation/dataplane/dash-byo-data-plane-app.md b/documentation/dataplane/dash-byo-data-plane-app.md index 33670ac05..ddd0c93d4 100644 --- a/documentation/dataplane/dash-byo-data-plane-app.md +++ b/documentation/dataplane/dash-byo-data-plane-app.md @@ -99,6 +99,7 @@ After initialization, the BYO data plane app will be able to: > > 1. Controller can directly call into BYO data plane app for managing private features that is not provided by DASH. > 2. BYO data plane app could call into the platform-dependent APIs (ASIC SDK) directly for managing the ASIC. However, by doing so, it also loses the portability of the BYO data plane app. +> 3. Since BYO data plane app is essentially a DPDK-based application, it can also use DPDK APIs to manage the netdev and the ASIC, e.g., setting up RSS options, etc. ### 2.3. ASIC programming work flow From 55ff4e8e0edb41079765ff33b65f08ba5fc164d7 Mon Sep 17 00:00:00 2001 From: r12f Date: Thu, 30 Nov 2023 01:55:47 +0000 Subject: [PATCH 9/9] minor fix. --- documentation/dataplane/dash-byo-data-plane-app.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/documentation/dataplane/dash-byo-data-plane-app.md b/documentation/dataplane/dash-byo-data-plane-app.md index ddd0c93d4..d55c8989f 100644 --- a/documentation/dataplane/dash-byo-data-plane-app.md +++ b/documentation/dataplane/dash-byo-data-plane-app.md @@ -187,7 +187,7 @@ typedef enum _sai_switch_attr_extensions_t { SAI_SWITCH_ATTR_DASH_BYO_DATA_PLANE_APP_NETDEV_NAME, /** - * @brief Vendor specific path name of the firmware to load. + * @brief Data plane app ready notification callback function. * * @type sai_switch_dash_byo_data_plane_app_ready_notification_fn * @flags CREATE_ONLY