diff --git a/chapter6.adoc b/chapter6.adoc index 81be3f9..2c8664d 100644 --- a/chapter6.adoc +++ b/chapter6.adoc @@ -309,7 +309,10 @@ operations requested through `control.OP`. {bits: 24, name: 'SRC_ID'}, {bits: 8, name: 'IOMMU_ID (WARL)'}, {bits: 8, name: 'SDID (WARL)'}, - {bits: 16, name: 'WPRI'} + {bits: 4, name: 'SRL'}, + {bits: 4, name: 'SML'}, + {bits: 4, name: 'SQRID'}, + {bits: 4, name: 'WPRI'} ], config:{lanes: 8, hspace:1024}} .... @@ -436,6 +439,13 @@ rule. The `SDID` field identifies the supervisor domain whose memory is accessed by this transaction. +The `SRL` and `SML` fields along with `operand-1.SSM` field are used to determine +the effective `RCID` and `MCID` provided by the IOMMU for device originated +requests. The determination of the effective `RCID` and `MCID` is as specified +by <>. The `SQRID` identifies the QRI for requests originating from the +devices associated with the SD and accompanies the `RCID` and `MCID` in the +requests made by the device to the QRI. + [[OP-1]] === Operand 1 register (`operand-1`) @@ -450,7 +460,8 @@ operations requested through `control.OP`. {bits: 4, name: 'MTT_MODE (WARL)'}, {bits: 1, name: 'PPNV (WARL)'}, {bits: 1, name: 'S (WARL)'}, - {bits: 4, name: 'WPRI'}, + {bits: 1, name: 'SSM'}, + {bits: 3, name: 'WPRI'}, {bits: 44, name: 'PPN'}, {bits: 10, name: 'WPRI'} ], config:{lanes: 8, hspace:1024}} diff --git a/chapter9.adoc b/chapter9.adoc new file mode 100644 index 0000000..fb870ce --- /dev/null +++ b/chapter9.adoc @@ -0,0 +1,401 @@ +[[chapter9]] +[[Smsdqos]] +== Capacity and Bandwidth QoS Register Interface (CBQRI) for Supervisor Domains + +Quality of Service (QoS) is defined as the minimal end-to-end performance +that is guaranteed in advance by a service level agreement (SLA) to a +workload. A workload may be a single application, a group of applications, +a virtual machine, a group of virtual machines, or a combination of those. + +=== Introduction to CBQRI and Ssqosid +The RISC-V Capacity and Bandwidth QoS Register Interface (CBQRI) cite:[CBQRI] +specification specifies mechanisms to a) associate QoS identifiers with +requests made by workloads to resources controlled by capacity and bandwidth +controllers, b) memory-mapped register interface for capacity allocation and +capacity usage monitoring in capacity controllers in shared resources such as +shared caches, c) memory-mapped register interface for bandwidth allocation +and bandwidth usage monitoring in bandwidth controllers in shared resources +such as memory controllers. + +The Ssqosid extension cite:[SSQOSID] provides a read/write S/HS-mode register +(`srmcfg`) to configure QoS Identifiers to be used with requests made by the +hart to shared resources. The CBQRI specification specifies the IOMMU extension +to associate QoS Identifiers with requests made by devices to shared resources. + +=== Security objective + +With supervisor domains, the resource accesses from an SD or the RDSM must not +be observable by entities that are not within their TCB using the resource usage +monitors. Similarly, the resource allocations for an SD or the RDSM must not be +influenced by entities outside their TCB. + +To support this security objective, the following capabilities are required: + +* *Multiple QoS register interfaces (QRIs) in the capacity and bandwidth + controllers within the SoC.* + + ** The RDSM may allocate a QRI exclusively to an SD or may mediate access to a + QRI among multiple SDs. A non-ISA supervisor domain QRI extension is + specified in <> to provide this capability. + +* *Associating a QRI ID (QRID) with requests originating from SDs and RDSM.* + + ** The QRID, along with the `RCID` and `AT`, is used to identify the + resource allocation configurations in a capacity or bandwidth controller. + The QRID, along with the `MCID`, is used to identify the ID of the + counter used to monitor resource usage in a capacity or bandwidth + controller. An ISA extension named Smqosid is specified in <> + to provide this capability. + +[[SMQOSID]] +=== Smqosid - Supervisor Domain Quality-of-Service Identifiers + +The Smqosid introduces a read/write M-mode register (`mrmcfg`) to configure +QoS identifiers for requests made by a hart with effective privilege mode equal +to M. + +The `mrmcfg` register is an MXLEN-bit read/write register used to configure a +Resource Control ID (`RCID`), Monitoring Counter ID (`MCID`), and a Machine QRID +(`MQRID`). The `MQRID, `RCID`, and `MCID` are WARL fields. The register is +formatted as shown in <> when MXLEN=64 and <> when MXLEN=32. +The CSR number is TBA. + +The `MQRID`, `RCID`, and `MCID` values accompany each request made with effective +privilege mode equal to M by the hart to shared resource controllers. The `MQRID` +and `RCID` values are used to determine the resource allocations (e.g., cache +occupancy limits, memory bandwidth limits, etc.) to enforce. The `MQRID` and +`MCID` values are used to identify a counter to monitor resource usage. + +[NOTE] +==== +A suggested reset value of 0 for the `RCID` and `MQRID` fields within the +`mrmcfg` register aligns with the default operational behavior of resource +controllers, which is to associate all capacity and resources with `RCID=0`. This +setup ensures a seamless initial state that conforms to expected default resource +allocation policies. The reset value for the `MCID` field, being potentially +implementation-defined, does not directly impact core functionality and may +vary based on specific system requirements. + +The `MQRID` is typically selected once at startup by the RDSM to designate the +QRI utilized by the RDSM itself. +==== + +When Smqosid is implemented, the configurations in the `srmcfg` register +introduced by Ssqosid extension only apply to requests made with effective +privilege modes less than M. + +[[MRMCFG64]] +.Machine Resource Management Configuration (`mrmcfg`) register for MXLEN=64 + +[wavedrom, , ] +.... +{reg: [ + {bits: 12, name: 'RCID'}, + {bits: 4, name: 'WPRI'}, + {bits: 12, name: 'MCID'}, + {bits: 4, name: 'MQRID'}, + {bits: 32, name: 'WPRI'}, +], config:{lanes: 2, hspace:1024}} +.... + +[[MRMCFG32]] +.Machine Resource Management Configuration (`mrmcfg`) register for MXLEN=32 + +[wavedrom, , ] +.... +{reg: [ + {bits: 12, name: 'RCID'}, + {bits: 4, name: 'WPRI'}, + {bits: 12, name: 'MCID'}, + {bits: 4, name: 'MQRID'}, +], config:{lanes: 1, hspace:1024}} +.... + +The Smqosid extension enables the scenario where two or more security domains +share a common set of capacity and bandwidth controllers. In this setup, access +to the QRI of these controllers is restricted to RDSM, which then provides +mediated access for these security domains to configure capacity or bandwidth +allocation and read the corresponding monitoring counters. A common QRID is +associated with requests from these domains, and the RDSM allocates sets of +`RCID` and `MCID` for each domain's use. Supervisor domains are then restricted +to selecting a `RCID` and `MCID` in `srmcfg` from the allocated range. Although +the RDSM could limit access to `srmcfg` to prevent direct domain configuration, +enabling domains to program `srmcfg` directly is preferred for performance +reasons. + +To facilitate delegation of `srmcfg` to the SD, the Smqosid extension introduces +`SRL` (supervisor-RCID-length) and `SML` (supervisor-MCID-length) fields in the +`msdcfg` register to configure the usable range of `RCID` and `MCID`. These +fields are WARL, with legal values from 0 to 12. Furthermore, a +supervisor-srmcfg-mode (`SSM`) field is introduced, with legal values of 0 and +1. The `SSM`, along with `SRL` and `SML` settings, determine the range of `RCID` +and `MCID` values an SD can program into the `srmcfg` register. The RDSM is +expected to configure these fields such that each SD can select from a disjoint +range of values for `RCID` and `MCID`. + +.`msdcfg` register + +[wavedrom, , ] +.... +{reg: [ + {bits: 6, name: 'SDICN'}, + {bits: 1, name: 'SSM'}, + {bits: 1, name: 'SDEDBGALW'}, + {bits: 1, name: 'SDETRCALW'}, + {bits: 11, name: 'WPRI'}, + {bits: 4, name: 'SRL'}, + {bits: 4, name: 'SML'}, + {bits: 4, name: 'SQRID'}, +], config:{lanes: 4, hspace:1024}} +.... + +The `srmcfg` CSR is provided with a privilege mode dependent read and write +logic which enables the RDSM to configure supervisor domain specific prefixes for +`RCID` and `MCID` fields. These prefixes remain invisible to supervisor domains +which can only configure `RCID` and `MCID` suffixes within the boundaries set by +RDSM in `msdcfg`. + +When `SSM` is 0, the `SRL` configuration enables the RDSM to configure a `RCID` +prefix value in the `srmcfg.RCID` field while allowing the supervisor domain to +program the low order `SRL` bits of the field. When `SSM` is 1, the supervisor +domain may only program a `RCID` value in the range 0 through +`2^RCIDLEN^ - 2^SRL^ - 1` where `RCIDLEN` is the number of implemented bits of +the `RCID` field. + +When `SSM` is 0, the `SML` configuration enables the RDSM to configure a `MCID` +prefix value in the `srmcfg.MCID` field while allowing the supervisor domain to +program the low order `SML` bits of the field. When `SSM` is 1, the supervisor +domain may only program a `MCID` value in the range 0 through +`2^MCIDLEN^ - 2^SML^ - 1` where `MCIDLEN` is the number of implemented bits of +the `MCID` field. + +When the `srmcfg` CSR is read at privilege modes less than M, the value returned +for the `RCID` and `MCID` fields of the register is computed as follows: + +.`RCID` and `MCID` read value compuation +[listing] +---- +SRL_MASK = (1 << SRL) - 1 +if SSM == 1 + RCID-value = srmcfg.RCID & SRL_MASK +else + RCID-value = srmcfg.RCID +endif + +SML_MASK = (1 << SML) - 1 +if SSM == 1 + MCID-value = srmcfg.MCID & SML_MASK +else + MCID-value = srmcfg.MCID +endif +---- + +<<< + +On a write to the `srmcfg` CSR at privilege modes less than M, the value +stored in the `RCID` and `MCID` fields of the register are computed as +follows: + +.`RCID` and `MCID` write value processing +[listing] +---- +SRL_MASK = (1 << SRL) - 1 +if SSM == 1 + srmcfg.RCID = (srmcfg.RCID & ~SRL_MASK) | (RCID-value & SRL_MASK) +else + if ((RCID-value & ~SRL_MASK) | SRL_MASK) != ((1 << RCIDLEN) - 1) + srmcfg.RCID = RCID-value + endif +endif + +SML_MASK = (1 << SML) - 1 +if SSM == 1 + srmcfg.MCID = (srmcfg.MCID & ~SML_MASK) | (MCID-value & SML_MASK) +else + if ((MCID-value & ~SML_MASK) | SML_MASK) != ((1 << MCIDLEN) - 1) + srmcfg.MCID = MCID-value + endif +endif +---- + +[NOTE] +==== +Consider a QRI that supports 32 RCIDs and is mediated by RDSM between two SDs. +The RDSM may allocate 24 RCIDs to the first SD and 8 RCIDs to the second SD. The +`SRL` and `SSM` configurations used by the RDSM to support this use case are as +follows: + +* The RDSM configures `SRL` to 3 and `SSM` to 0 for first SD. This allows the + first SD to select `RCID` values 0 through 23 in `srmcfg`. + +* The RDSM configures `SRL` to 3 and `SSM` to 1 for the second SD and programs + the `srmcfg` with a value of 24. The second SD is allowed to program the low + order 3 bits of `srmcfg.RCID`, thereby selecting `RCID` values between 24 and 31. + When this SD reads `srmcfg.RCID`, the value returned consist of the low 3 bits, + with all upper bits set to 0. +==== + +The `RCID` and `MCID` values that accompany requests, when made with an effective +privilege mode equal to M are stored in the `mrmcfg` register. Conversely, for +requests made with effective privilege modes less than M, these values are +stored in the `srmcfg` register. + +The Smqosid extension introduces a `SQRID` field in the `msdcfg` register to +hold the QRID for requests made by the hart with effective privilege mode less +than M respectively. The `SQRID` value along with the `RCID` and `MCID` values +accompanies each such request made by the hart to the shared resource +controllers. The `SQRID` and `RCID` values are used to determine the resource +allocations (e.g., cache occupancy limits, memory bandwidth limits, etc.) to +enforce. The `SQRID` and `MCID` values are used to identify a counter to monitor +resource usage. + +[NOTE] +==== +The `SQRID` may be updated during the process of switching SDs, to denote the +QRI allocated to that specific SD by the RDSM. +==== + +[[SDQRI]] +=== Supervisor Domain QoS Register interfaces (QRI) + +Capacity and bandwidth controllers that support supervisor domains provide one +or more memory-mapped QoS register interfaces (QRI). A QRI may be made +exclusively accessible to a supervisor domain or the RDSM may mediate access to +the QRI using an SBI. The RDSM can control access to the QRI from supervisor +domains using MTT and/or PMP. The RDSM controls access to the QRI from devices +using IOMTT and/or IOPMP. + +The number of `RCID` and `MCID` supported by the controllers for each QRI need +not be identical. For maximal flexibility in allocation of `RCID` and `MCID` +values, it is recommended that the number of `RCID` and `MCID` supported for a +given QRID be identical in all capacity and bandwidth controllers in the +system. + +The capacity and bandwidth controllers use the configurations that were +established for the `RCID` and `AT` in the request through the QRI corresponding +to the QRID in the request. Likewise the counters to count resource usage are +selected using `MCID` and the QRID in the request and a QRI can be used to +access counters associated with the corresponding QRID. + +By default all resources in the capacity and bandwidth controllers may be +allocated using any of the QRI. The controllers may optionally support +reservation of resources for use by a QRI. When such reservation is supported +the capacity or bandwidth reserved for a QRI may only be used by requests that +have the corresponding QRID. Supporting resource reservation capability allows +effective partitioning of the shared resources among SDs sharing the resources. + +To reserve capacity to a QRI, the capacity controllers support a new operation +called `CONFIG_QRI_LIMIT` (`OP=4`). The capacity to be reserved is specified +using the `cc_block_mask` register. The `AT` and `RCID` fields of the +`cc_alloc_ctl` register are ignored by this operation. The mask specified in +`cc_block_mask` for this operation must have a contiguous run of 1s and an +implementation may require the mask to reserve at least one capacity block; +else the operation will fail with `STATUS=5`. The `CONFIG_QRI_LIMIT` operation +may be requested once following reset. If the operation is requested again then +the operation will fail with `STATUS=2`. On successful completion of the +operation, the `cc_capabilities.NCBLKS` shall update to a value that is the +number of 1 bits in value held in the `cc_block_mask` and only bits `NCBLKS-1:0` +are writable in `cc_block_mask`. + +[NOTE] +==== +The `CONFIG_QRI_LIMIT` operation is a one time operation to allow the RDSM to +configure the capacity limit for a QRI before passing through the QRI to the +associated SD. An SD may then allocate capacity for RCIDs from within this limit +established by the RDSM. +==== + +<<< + +[NOTE] +==== +Let's consider a cache with `NCBLKS=8`. In this example, this cache supports two +QRIs with QRID of 0 and 1. The `CONFIG_QRI_LIMIT` operation is used to reserve +two capacity blocks numbered 7 and 6 for use by `QRID=0`. The `CONFIG_QRI_LIMIT` +operation is used to reserve six capacity blocks numbered 0 through 5 for use by +QRID=1. The SD that uses the QRID=0 is thus limited to selecting a 2 bit +capacity block mask where the bit 0 of the mask maps to capacity block 6 and bit +1 to capacity block 7. The SD that uses QRID=1 is limited to selecting a 6 bit +capacity block mask where the mask bits 0 through 5 map to the correspondingly +numbered capacity blocks. Both SDs in this example, have configured `RCID=5` +with 1 capacity block for requests with access-type `AT=0`. The effective +capacity block allocation in the controller is as follows: + +[width=100%] +[%header, cols="6,^1,^1,^1,^1,^1,^1,^1,^1"] +|=== +| | 7 | 6 | 5 | 4 | 3 | 2 | 1 | 0 +| `QRID=0`, `RCID=5`, `AT=0` | `0` | `1` | `0` | `0` | `0` | `0` | `0` | `0` +| `QRID=1`, `RCID=5`, `AT=0` | `0` | `0` | `0` | `0` | `0` | `1` | `0` | `0` +|=== +==== + +To reserve bandwidth to a QRI, the bandwidth controllers support a new operation +called `CONFIG_QRI_LIMIT` (`OP=4`). The `AT` and `RCID` fields of the +`bc_alloc_ctl` register are ignored by this operation. Likewise, the `sharedAT` +and `useShared` fields of the `bc_bw_alloc` register are ignored by this +operation. The bandwidth to be reserved is specified using the `Rbwb` field of +the `bc_bw_alloc` register and `Mweight`, when not equal to 0, represents a +proportional share of non-reserved or unused bandwidth that may be used by the +RCIDs associated with this QRI. When the `Mweight` is equal to 0, the `Rbwb` is +a hart limit and the RCIDs associated with this QRI are not eligible to use +unused or non-reserved bandwidth. The `CONFIG_QRI_LIMIT` operation may be +requested once following reset. If the operation is requested again then the +operation will fail with `STATUS=2`. On successful completion of the operation, +the `bc_capabilities.MRBWB` shall update to a value specified in `Rbwb`. When +`Mweight` for a QRI is not 0, then a two level weighted sharing of unused or +non-reserved bandwidth occurs. When the `Mweight` parameter for a QRI is not set +to 0, the amount of unused bandwidth allocated to QRID=x during contention +with another QRI that is also permitted to use unused bandwidth is determined +by dividing the `Mweight` of QRID=q by the sum of the `Mweight` of all other +contending `QRIs`. This ratio `Pq` is determined by <>. This weight share +of the unused bandwidth made available to a QRI is then shared among the +contending RCIDs of that QRI using the weights configured for the RCIDs. + +[latexmath#eq-1,reftext="equation ({counter:eqs})"] +++++ +\begin{equation} +Pq = \frac{Mweight_{q}}{\sum_{q=1}^{q=n} Mweight_{q}} +\end{equation} +++++ + +[NOTE] +==== +Consider a bandwidth controller that supports two QRIs. For brevity, this +example controller does not support bandwidth allocation by access-type `AT`. +In this example, the QRID=0 has been configured with `Rbwb` of 100 bandwidth +units and QRID=1 has been configured with `Rbwb` of 50 bandwidth units. The +`Mweight` configured for the two QRIs is 16, i.e., they equally share unused +bandwidth. + + + +Each QRI in this example is used to configure bandwidth limits for `RCID=5` and +`RCID=6` where each RCID has been allocated 10 units of reserved bandwidth and +configured with weights 50 and 25, respectively. + + + +With this configuration `RCID=5` receives 2/3 of the unused bandwidth made +available to the QRI and `RCID=6` receives 1/3 of the unused bandwidth made +available to the QRI when they both contend for the unused bandwidth. + + + +The effective configurations in the bandwidth controller are as follows: + +[width=100%] +[%header, cols="4,^2,^2,^2,^2"] +|=== +| | `RCID Rbwb` | `RCID Mweight` | `QRI Rbwb` | `QRI Mweight` +| `QRID=0`, `RCID=5` | `10` | `50` | `100` | `16` +| `QRID=0`, `RCID=6` | `10` | `25` | `100` | `16` +| `QRID=1`, `RCID=5` | `10` | `50` | `50` | `16` +| `QRID=1`, `RCID=6` | `10` | `25` | `50` | `16` +|=== +==== + +[NOTE] +==== +The bandwidth enforcement is typically work-conserving, meaning that it allows +unused bandwidth to be used by QRIs enabled to use it even if they have consumed +their `Rbwb`. + +When contending for unused bandwidth, the weighted share is typically +computed among the QRIs that are actively generating requests in that +accounting interval and have a non-zero weight programmed. +==== diff --git a/example.bib b/example.bib index 0844807..23f7e6d 100644 --- a/example.bib +++ b/example.bib @@ -23,3 +23,13 @@ @electronic{ExtDbg title = {RISC-V Debug Specification}, url = {https://github.com/riscv/riscv-debug-spec} } + +@electronic{CBQRI, + title = {RISC-V Capacity and Bandwidth Controller QoS Register Interface}, + url = {https://github.com/riscv-non-isa/riscv-cbqri} +} + +@electronic{SSQOSID, + title = {RISC-V Quality-of-Service (QoS) Identifiers}, + url = {https://github.com/riscv/riscv-ssqosid} +} diff --git a/header.adoc b/header.adoc index f8fc127..02cb994 100644 --- a/header.adoc +++ b/header.adoc @@ -68,6 +68,7 @@ include::chapter4.adoc[] include::chapter6.adoc[] include::chapter7.adoc[] include::chapter8.adoc[] +include::chapter9.adoc[] //the index must precede the bibliography //include::index.adoc[]