no available connections cause by concurrency request limit bug

[crazycs520]

Bug Report

#1098 introduced concurrency request limit.

client-go/internal/client/client_batch.go

Line 407 in 03bbadb

if hasHighPriorityTask || c.available() > 0 {

client-go/internal/client/client_batch.go

Lines 569 to 571 in 03bbadb

	func (c *batchCommandsClient) available() int64 {
	return c.maxConcurrencyRequestLimit.Load() - c.sent.Load()
	}

Following is the general situation about the upper code:

hasHighPriorityTask = false  // most request don't have high priority.
c.maxConcurrencyRequestLimit = math.MaxInt64 // by default
c.sent >= 0

In general, the above code works fine. But it will have a bug with the data race in #1222 :

• the data race will cause the c.sent to be less than 0
• then it will cause c.available() computation overflow
• then c.available() > 0 will be false.
• then have no available connections issue, and the request could not be sent, resulting in a QPS drop to 0. See issue in qps drop to zero after inject one of tikv failure last for 10mins with enable tidb enable-forwarding pingcap/tidb#51728.