test(test-driver): [NET-1738] Increase max_concurrent_requests in s…

…ystem test agent to 10_000 (#715)

We create the agent in the test driver
[here](https://github.com/dfinity/ic/blob/3909a2cfe50518d54d217d4aa5918578591beb9c/rs/tests/driver/src/util.rs#L788)
to run workloads in system tests. The agent can [by default only submit
50 requests
concurrently](https://github.com/dfinity/agent-rs/blob/7e9489a65aba3e6993d8b96f6ef09e34a50837c1/ic-agent/src/agent/agent_config.rs#L20),
which is a problem if we want to execute high RPS ingress message
workloads against the synchronous call endpoint. It is a problem because
in the worst case, we might have roughly the size of the ingress pool,
~10_000, requests concurrently to one replica, which all take 10s to
resolve. In this scenario where each requests take 10s, the agent would
only be able to send 50 requests every 10 seconds.

To prevent this config value in the agent from being the bottleneck we
need to increase the max_concurrent_requests to roughly 10_000. This is
not a problem as the replicas now have H/2 enabled to allow multiplexing
of requests over a single TCP connection.

closes NET-1738

rs/tests/driver/src/util.rs

@@ -78,6 +78,9 @@ pub const MESSAGE_CANISTER_WASM: &[u8] = include_bytes!("message.wasm");
 pub const CFG_TEMPLATE_BYTES: &[u8] =
     include_bytes!("../../../../ic-os/components/ic/ic.json5.template");
 
+// Requests are multiplexed over H2 requests.
+pub const MAX_CONCURRENT_REQUESTS: usize = 10_000;
+
 pub fn get_identity() -> ic_agent::identity::BasicIdentity {
     ic_agent::identity::BasicIdentity::from_pem(IDENTITY_PEM.as_bytes())
         .expect("Invalid secret key.")
@@ -794,6 +797,7 @@ pub async fn agent_with_client_identity(
     let a = Agent::builder()
         .with_transport(transport)
         .with_identity(identity)
+        .with_max_concurrent_requests(MAX_CONCURRENT_REQUESTS)
         // Ingresses are created with the system time but are checked against the consensus time.
         // Consensus time is the time that is in the last finalized block. Consensus time might lag
         // behind, for example when the subnet has many modes and the progress of consensus is